Title
#fleet
Martin Pöhlmann

Martin Pöhlmann

11/22/2021, 12:55 PM
Now to the problem: I have one windows machine (virtual box) that refuses to "talk" to fleet. Honestly I do not know where the error is. Fleet shows the machine (with some basic info but as "never fetched"). But I cannot run queries. This is also the same if I reset the osquery db on the machine. And even more curious I cannot delete the machine from fleet. The error of the HTTP request in chrome dev tools shows as follows:
{
  "message": "unsupported Scan, storing driver.Value type <nil> into type *time.Time",
  "errors": [
    {
      "name": "base",
      "reason": "unsupported Scan, storing driver.Value type <nil> into type *time.Time"
    }
  ]
}
Any ideas?
Lucas Rodriguez

Lucas Rodriguez

11/22/2021, 12:57 PM
As for the error log above, it looks like a bug, we'll try to reproduce on our end and get back to you.
1:00 PM
Were you able to find osquery logs in the VM?
1:00 PM
(to troubleshoot)
1:18 PM
Also, did you upgrade from a previous version of fleet? If so, any warnings about upgrades when starting fleet?
Martin Pöhlmann

Martin Pöhlmann

11/22/2021, 2:02 PM
Yes, I've updated from 4.3.1. I think the upgrade logs are gone, but I did not remember anything special. When I start fleet the following shows up:
fleet_1          | {"component":"crons","cron":"cleanups","details":"looping through ids: running visitFunc for queries: getting user_time p50 for query 1: timestamp: 2021-11-22T13:45:38+01:00: Error 1064: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'row_number, mm.* FROM (\n\t\tSELECT d.scheduled_query_id, d.user_time, d.executions' at line 4","err":"aggregating query stats","level":"error","ts":"2021-11-22T12:45:38.017855521Z"}
fleet_1          | {"component":"crons","cron":"cleanups","details":"looping through ids: running visitFunc for scheduled_queries: getting user_time p50 for scheduled_query 4: timestamp: 2021-11-22T13:45:38+01:00: Error 1064: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'row_number, mm.* FROM (\n\t\tSELECT d.scheduled_query_id, d.user_time, d.executions' at line 4","err":"aggregating scheduled query stats","level":"error","ts":"2021-11-22T12:45:38.019472394Z"}
2:02 PM
And the strange thing is that I do not have any logs/files under C:\ProgramFiles\osquery\logs on the machine
Lucas Rodriguez

Lucas Rodriguez

11/22/2021, 2:05 PM
OK, did you run
fleet prepare db
before running the new version of fleet?
Martin Pöhlmann

Martin Pöhlmann

11/22/2021, 2:05 PM
sure, that's run everytime the container starts
2:06 PM
command: sh -c "/usr/bin/fleet prepare db --no-prompt && /usr/bin/fleet serve"
2:07 PM
also the machine was added AFTER the upgrade
Lucas Rodriguez

Lucas Rodriguez

11/22/2021, 2:08 PM
OK, let me check with the team
Martin Pöhlmann

Martin Pöhlmann

11/22/2021, 2:09 PM
cool, thx
Lucas Rodriguez

Lucas Rodriguez

11/22/2021, 2:12 PM
QQ: MySQL server and version?
Martin Pöhlmann

Martin Pöhlmann

11/24/2021, 6:28 PM
@Lucas Rodriguez 8.0.26 - I'm seeing the same error now for another newly onboarded machine
Lucas Rodriguez

Lucas Rodriguez

11/24/2021, 6:33 PM
Hi Martin! Another user hit the same error, we have an issue and are working on a fix: https://github.com/fleetdm/fleet/issues/3095
Martin Pöhlmann

Martin Pöhlmann

11/24/2021, 6:37 PM
Thanks. I subscribed to the ticket. Will the clients resume to work normally after the fix? If there is need for manual intervention (which I have no problem with) it's okay to mention that in the ticket.
Lucas Rodriguez

Lucas Rodriguez

11/24/2021, 6:41 PM
I think there are two issues,1. One is the fleet side bug (unsupported Scan). 2. The other one is some issue on the osquery side, are you running osquery manually on the VM? (could you get logs?)
Martin Pöhlmann

Martin Pöhlmann

11/24/2021, 10:34 PM
no, I run it as service (osquery MSI installer). The logs folder is empty and I did not find anything in win eventlog. Are there any other places to look for?
9:21 PM
@Lucas Rodriguez let us continue discussion on GitHub (https://github.com/fleetdm/fleet/issues/3095) if you do not mind. I've added our osquery config there. If there is anything else I could provide, please let me know.
Lucas Rodriguez

Lucas Rodriguez

11/29/2021, 9:22 PM
Hi Martin I just saw your comment, great timing, I've started working on the issue today.
Martin Pöhlmann

Martin Pöhlmann

11/29/2021, 9:22 PM
Good to hear 🙂
9:23 PM
It's already late in 🇩🇪 , but if you need anything, I'll provide tomorrow
Lucas Rodriguez

Lucas Rodriguez

11/29/2021, 9:23 PM
There are a few related issues we are working on:1. Proper Orbit logs on Windows 2. Allow setting a "platform" to policies.
Martin Pöhlmann

Martin Pöhlmann

11/29/2021, 9:23 PM
we're not using orbit
9:24 PM
(maybe that'S also part of the issue)
Lucas Rodriguez

Lucas Rodriguez

11/29/2021, 9:24 PM
Not sure, we do support vanilla osquery.
9:24 PM
I'll continue troubleshooting and ask questions in the issue.
9:24 PM
Again, thanks a lot for the detailed comment.
Martin Pöhlmann

Martin Pöhlmann

11/29/2021, 9:24 PM
Cool, thanks.
9:25 PM
I'll have another look in the fleet server logs while I am on it
9:47 PM
so far the only suspicious status logs (delivered to server) are regarding the policies. I'm wondering if we could auto-detect the machines to send queries to? This is already the case for the query editor as it suggests where the query may run on.
9:47 PM
but as 1st step, manual selection would be more than okay
Lucas Rodriguez

Lucas Rodriguez

11/29/2021, 9:51 PM
Correct, we will allow configuring platform for policies and run them as live queries to test them out first (coming soon).
f

Flngen Flugen

11/30/2021, 2:22 PM
We're having this same issue with fleetdm/fleet:main image on k8s and osquery 5.0.1
Lucas Rodriguez

Lucas Rodriguez

11/30/2021, 5:52 PM
Hi Flngen! We are working on a fix for this that we'll try to include in fleet 4.6.2. (ETA: some day this week, hopefully Thursday).
Martin Pöhlmann

Martin Pöhlmann

12/02/2021, 9:03 AM
@Lucas Rodriguez I've updated to 6.4.2 just now. The hosts that were throwing this error still show the same error 🙂 will it take some time to get these updated? I also cannot delete these hosts, nor force-refresh these via the API (same error)
Lucas Rodriguez

Lucas Rodriguez

12/02/2021, 12:33 PM
Hi Martin!, can you double check the version in top right ->My Account -> fleet version should show up in the bottom right.
12:34 PM
Also, do you have access to the MySQL database? To help us troubleshoot better.
Martin Pöhlmann

Martin Pöhlmann

12/02/2021, 12:47 PM
12:48 PM
I think I can somehow hop on the docker mysql container
Lucas Rodriguez

Lucas Rodriguez

12/02/2021, 1:03 PM
OK, if possible, please run the following query
SELECT * FROM fleet.hosts h LEFT JOIN fleet.host_seen_times hst ON h.id = hst.host_id
(feel free to not include any sensitive data like hostnames)
Martin Pöhlmann

Martin Pöhlmann

12/02/2021, 1:32 PM
See PM
Lucas Rodriguez

Lucas Rodriguez

12/02/2021, 1:45 PM
Thanks, one more thing, could you also send us the server logs? (those surrounds a scan error you see on the browser)