Channels
#fleet
Title
# fleet
m
Mystery Incorporated
10/08/2021, 3:13 PMHey guys did something change with 4.4 and fetching? I have my fetch time to 27 minutes but as you can see I got some wacky fetch times (and most of these hosts are online 24/7)
Copy code
osquery:
detail_update_interval: 27mt
Tomas Touceda
10/08/2021, 3:16 PMwe added jitter of 10% of the interval to prevent hosts to spike the server, you can set it 0 if you prefer
j
Jocelyn Bothe
10/08/2021, 3:18 PMwouldn't a 10% jitter be max 2.7 minutes for a 27 min update interval? they're showing a discrepancy of hours and days
t
Tomas Touceda
10/08/2021, 3:19 PMright, if you could share fleet serve debug logs, that might shine some light as to what might be happening
m
Mystery Incorporated
10/08/2021, 3:22 PMhmm yea that's defs not 10% jitter I'll look at the logs
I tried to do a manual fetch and it's just stuck indefinitely fetching
j
Jocelyn Bothe
10/08/2021, 3:29 PMdo live queries work for these hosts?
m
Mystery Incorporated
10/08/2021, 3:36 PMYes live query and the usual scheduled queries
@Tomas Touceda because OSQuery just spews so many errors into my status logs about being unable to access registry globs and can't resolve azuread users etc I'm struggling to see anything relevent in the log
t
Tomas Touceda
10/08/2021, 3:42 PMhow about the fleet serve logs?
m
Mystery Incorporated
10/08/2021, 3:44 PMwhere are they?
t
Tomas Touceda
10/08/2021, 3:45 PMthey are the logs that fleet serve command writes, so it's stderr and then wherever they go from there
m
Mystery Incorporated
10/08/2021, 3:45 PM@Tomas Touceda
Copy code
{
"component": "crons",
"cron": "vulnerabilities",
"err": "getting cpes for: increase-memory-limit: fts5: syntax error near \".\"",
"level": "error",
"software->cpe": "error translating to CPE, skipping...",
"ts": "2021-10-08T15:37:07.933390548Z"
}Also
Copy code
{
"component": "http",
"err": "read auth token: reading from websocket: sockjs: session not in open state",
"msg": "failed to read auth token",
"ts": "2021-10-08T15:34:45.315116969Z"
}t
Tomas Touceda
10/08/2021, 3:47 PMcould you share a bigger potion of the log?
m
Mystery Incorporated
10/08/2021, 3:47 PMit's syslog so it's liked peppered through syslog
maybe if you know a grep command I could use
t
Tomas Touceda
10/08/2021, 3:48 PMthose two lines you've shared don't tell me much, I would need to see a bigger chunk to get a clearer look
you can look for
erringest-errm
Mystery Incorporated
10/08/2021, 3:51 PMyea I looked and there isn't anything else
just cron complaining about syntax and the websocket error can' see anything else I did a grep "fleet"
well cat /var/log/syslog | grep "fleet"
I did it with err and ingest-err like you suggest, err just gives me the same results as "fleet" and ingest-err turns up nothing.
t
Tomas Touceda
10/08/2021, 3:55 PMI suggest running fleet serve with debug logging enabled and sending logs in a way that you might be able to share them
m
Mystery Incorporated
10/08/2021, 3:56 PMI would like to log to a dedicated file instead of syslog if possibl;e yea
t
Tomas Touceda
10/08/2021, 3:57 PMnot sure how you're running fleet, fleet logs to stderr, you should be able to pipe it to a file
m
Mystery Incorporated
10/08/2021, 3:58 PMI running it in systemd unit
Copy code
ExecStart=/usr/local/bin/fleet serve --config /home/terrance/fleet/config.ymlso I should just do like a > /var/log/fleet.log if I want?
t
Tomas Touceda
10/08/2021, 4:00 PMm
Mystery Incorporated
10/08/2021, 4:06 PMOk I put these in
Copy code
StandardOutput=/var/log/fleet_standard.log
StandardError=/var/log/fleet_error.logmade the files, gave the user ownership, nothing being wrote to the files
t
Tomas Touceda
10/08/2021, 4:08 PMthat SO link suggests a different approach, through syslog
m
Mystery Incorporated
10/08/2021, 4:09 PMoh it's already writing to syslog I want to write to own file
ok kcool by reading further down that SO article I see I need file: in the path
it works now
writing to own log
Copy code
{"component":"crons","cron":"vulnerabilities","databases-path":"/mnt/fleetvuln","level":"info","ts":"2021-10-08T16:10:36.864309515Z"}
{"component":"crons","cron":"vulnerabilities","level":"info","periodicity":"1h0m0s","ts":"2021-10-08T16:10:36.864402755Z"}
{"address":"127.0.0.1:2498","msg":"listening","transport":"https","ts":"2021-10-08T16:10:36.883492137Z"}They should be in stdout not stderr shouldn't they?
t
Tomas Touceda
10/08/2021, 4:12 PMplease add --logging_debug to your fleet serve call, so we get more verbose logging
m
Mystery Incorporated
10/08/2021, 4:15 PMyep I did nopw
Copy code
{
"host": "xxxx",
"level": "debug",
"msg": "host reported software with empty name",
"source": "programs",
"ts": "2021-10-08T16:14:26.936622816Z",
"version": "5.5.0.6704"
} Copy code
{
"component": "http",
"err": [
"failed to save host software: insert software: Error 1054: Unknown column 'bundle_identifier' in 'field list'"
],
"ip_addr": "127.0.0.1:42184",
"level": "debug",
"method": "POST",
"took": "98.103886ms",
"ts": "2021-10-08T16:14:25.29209961Z",
"uri": "/api/v1/osquery/distributed/write",
"x_for_ip_addr": "x.x.x.x"
}Lots of those errors about 'bundle_identifier'
I did upgrade from 4.3.2 to 4.4 and i did do a prepare db before first launch
@Tomas Touceda I ran the Alter table command you gave ryan above that fixed it so I can only guess that a DB migration is missing in the 4.4 tag
z
zwass
10/08/2021, 5:05 PM
Thanks folks, looks like I made a mistake when cutting the 4.3.2 release so anyone upgrading 4.3.2->4.4.0 would be effected. We'll cut a 4.4.1 shortly that will fix this -- the manual fix discussed above also works (and the 4.4.1 fix will be compatible with that)
4.4.1 is published with the necessary fix. No need to update if you've already resolved this.
👍 1
m
Mystery Incorporated
10/09/2021, 2:08 AMThank you @zwass
13 Views