Title
#fleet
Mystery Incorporated

Mystery Incorporated

10/08/2021, 3:13 PM
Hey guys did something change with 4.4 and fetching? I have my fetch time to 27 minutes but as you can see I got some wacky fetch times (and most of these hosts are online 24/7)
osquery:
  detail_update_interval: 27m
Tomas Touceda

Tomas Touceda

10/08/2021, 3:16 PM
we added jitter of 10% of the interval to prevent hosts to spike the server, you can set it 0 if you prefer
Jocelyn Bothe

Jocelyn Bothe

10/08/2021, 3:18 PM
wouldn't a 10% jitter be max 2.7 minutes for a 27 min update interval? they're showing a discrepancy of hours and days
Tomas Touceda

Tomas Touceda

10/08/2021, 3:19 PM
right, if you could share fleet serve debug logs, that might shine some light as to what might be happening
Mystery Incorporated

Mystery Incorporated

10/08/2021, 3:22 PM
hmm yea that's defs not 10% jitter I'll look at the logs
3:24 PM
I tried to do a manual fetch and it's just stuck indefinitely fetching
Jocelyn Bothe

Jocelyn Bothe

10/08/2021, 3:29 PM
do live queries work for these hosts?
Mystery Incorporated

Mystery Incorporated

10/08/2021, 3:36 PM
Yes live query and the usual scheduled queries
3:39 PM
@Tomas Touceda because OSQuery just spews so many errors into my status logs about being unable to access registry globs and can't resolve azuread users etc I'm struggling to see anything relevent in the log
Tomas Touceda

Tomas Touceda

10/08/2021, 3:42 PM
how about the fleet serve logs?
Mystery Incorporated

Mystery Incorporated

10/08/2021, 3:44 PM
where are they?
Tomas Touceda

Tomas Touceda

10/08/2021, 3:45 PM
they are the logs that fleet serve command writes, so it's stderr and then wherever they go from there
Mystery Incorporated

Mystery Incorporated

10/08/2021, 3:45 PM
@Tomas Touceda
{
  "component": "crons",
  "cron": "vulnerabilities",
  "err": "getting cpes for: increase-memory-limit: fts5: syntax error near \".\"",
  "level": "error",
  "software->cpe": "error translating to CPE, skipping...",
  "ts": "2021-10-08T15:37:07.933390548Z"
}
3:46 PM
Also
3:46 PM
{
  "component": "http",
  "err": "read auth token: reading from websocket: sockjs: session not in open state",
  "msg": "failed to read auth token",
  "ts": "2021-10-08T15:34:45.315116969Z"
}
Tomas Touceda

Tomas Touceda

10/08/2021, 3:47 PM
could you share a bigger potion of the log?
Mystery Incorporated

Mystery Incorporated

10/08/2021, 3:47 PM
it's syslog so it's liked peppered through syslog
3:48 PM
maybe if you know a grep command I could use
Tomas Touceda

Tomas Touceda

10/08/2021, 3:48 PM
those two lines you've shared don't tell me much, I would need to see a bigger chunk to get a clearer look
3:48 PM
you can look for
err
or
ingest-err
Mystery Incorporated

Mystery Incorporated

10/08/2021, 3:51 PM
yea I looked and there isn't anything else
3:52 PM
just cron complaining about syntax and the websocket error can' see anything else I did a grep "fleet"
3:52 PM
well cat /var/log/syslog | grep "fleet"
3:55 PM
I did it with err and ingest-err like you suggest, err just gives me the same results as "fleet" and ingest-err turns up nothing.
Tomas Touceda

Tomas Touceda

10/08/2021, 3:55 PM
I suggest running fleet serve with debug logging enabled and sending logs in a way that you might be able to share them
Mystery Incorporated

Mystery Incorporated

10/08/2021, 3:56 PM
I would like to log to a dedicated file instead of syslog if possibl;e yea
Tomas Touceda

Tomas Touceda

10/08/2021, 3:57 PM
not sure how you're running fleet, fleet logs to stderr, you should be able to pipe it to a file
Mystery Incorporated

Mystery Incorporated

10/08/2021, 3:58 PM
I running it in systemd unit
3:59 PM
ExecStart=/usr/local/bin/fleet serve --config /home/terrance/fleet/config.yml
3:59 PM
so I should just do like a > /var/log/fleet.log if I want?
Mystery Incorporated

Mystery Incorporated

10/08/2021, 4:06 PM
Ok I put these in
StandardOutput=/var/log/fleet_standard.log
StandardError=/var/log/fleet_error.log
4:06 PM
made the files, gave the user ownership, nothing being wrote to the files
Tomas Touceda

Tomas Touceda

10/08/2021, 4:08 PM
that SO link suggests a different approach, through syslog
Mystery Incorporated

Mystery Incorporated

10/08/2021, 4:09 PM
oh it's already writing to syslog I want to write to own file
4:11 PM
ok kcool by reading further down that SO article I see I need file: in the path
4:11 PM
it works now
4:11 PM
writing to own log
4:11 PM
{"component":"crons","cron":"vulnerabilities","databases-path":"/mnt/fleetvuln","level":"info","ts":"2021-10-08T16:10:36.864309515Z"}
{"component":"crons","cron":"vulnerabilities","level":"info","periodicity":"1h0m0s","ts":"2021-10-08T16:10:36.864402755Z"}
{"address":"127.0.0.1:2498","msg":"listening","transport":"https","ts":"2021-10-08T16:10:36.883492137Z"}
4:12 PM
They should be in stdout not stderr shouldn't they?
Tomas Touceda

Tomas Touceda

10/08/2021, 4:12 PM
please add --logging_debug to your fleet serve call, so we get more verbose logging
Mystery Incorporated

Mystery Incorporated

10/08/2021, 4:15 PM
yep I did nopw
4:15 PM
{
  "host": "xxxx",
  "level": "debug",
  "msg": "host reported software with empty name",
  "source": "programs",
  "ts": "2021-10-08T16:14:26.936622816Z",
  "version": "5.5.0.6704"
}
4:16 PM
{
  "component": "http",
  "err": [
    "failed to save host software: insert software: Error 1054: Unknown column 'bundle_identifier' in 'field list'"
  ],
  "ip_addr": "127.0.0.1:42184",
  "level": "debug",
  "method": "POST",
  "took": "98.103886ms",
  "ts": "2021-10-08T16:14:25.29209961Z",
  "uri": "/api/v1/osquery/distributed/write",
  "x_for_ip_addr": "x.x.x.x"
}
4:18 PM
Lots of those errors about 'bundle_identifier'
4:19 PM
I did upgrade from 4.3.2 to 4.4 and i did do a prepare db before first launch
4:23 PM
@Tomas Touceda I ran the Alter table command you gave ryan above that fixed it so I can only guess that a DB migration is missing in the 4.4 tag
zwass

zwass

10/08/2021, 5:05 PM
Thanks folks, looks like I made a mistake when cutting the 4.3.2 release so anyone upgrading 4.3.2->4.4.0 would be effected. We'll cut a 4.4.1 shortly that will fix this -- the manual fix discussed above also works (and the 4.4.1 fix will be compatible with that)
12:25 AM
4.4.1 is published with the necessary fix. No need to update if you've already resolved this.
Mystery Incorporated

Mystery Incorporated

10/09/2021, 2:08 AM
Thank you @zwass