https://github.com/osquery/osquery logo
Powered by
Title
m

Mystery Incorporated

10/08/2021, 3:13 PM
Hey guys did something change with 4.4 and fetching? I have my fetch time to 27 minutes but as you can see I got some wacky fetch times (and most of these hosts are online 24/7) 
osquery:
  detail_update_interval: 27m
t

Tomas Touceda

10/08/2021, 3:16 PM
we added jitter of 10% of the interval to prevent hosts to spike the server, you can set it 0 if you prefer
j

Jocelyn Bothe

10/08/2021, 3:18 PM
wouldn't a 10% jitter be max 2.7 minutes for a 27 min update interval? they're showing a discrepancy of hours and days
t

Tomas Touceda

10/08/2021, 3:19 PM
right, if you could share fleet serve debug logs, that might shine some light as to what might be happening
m

Mystery Incorporated

10/08/2021, 3:22 PM
hmm yea that's defs not 10% jitter I'll look at the logs
I tried to do a manual fetch and it's just stuck indefinitely fetching
j

Jocelyn Bothe

10/08/2021, 3:29 PM
do live queries work for these hosts?
m

Mystery Incorporated

10/08/2021, 3:36 PM
Yes live query and the usual scheduled queries
@Tomas Touceda because OSQuery just spews so many errors into my status logs about being unable to access registry globs and can't resolve azuread users etc I'm struggling to see anything relevent in the log
t

Tomas Touceda

10/08/2021, 3:42 PM
how about the fleet serve logs?
m

Mystery Incorporated

10/08/2021, 3:44 PM
where are they?
t

Tomas Touceda

10/08/2021, 3:45 PM
they are the logs that fleet serve command writes, so it's stderr and then wherever they go from there
m

Mystery Incorporated

10/08/2021, 3:45 PM
@Tomas Touceda 
{
  "component": "crons",
  "cron": "vulnerabilities",
  "err": "getting cpes for: increase-memory-limit: fts5: syntax error near \".\"",
  "level": "error",
  "software->cpe": "error translating to CPE, skipping...",
  "ts": "2021-10-08T15:37:07.933390548Z"
}
Also
{
  "component": "http",
  "err": "read auth token: reading from websocket: sockjs: session not in open state",
  "msg": "failed to read auth token",
  "ts": "2021-10-08T15:34:45.315116969Z"
}
t

Tomas Touceda

10/08/2021, 3:47 PM
could you share a bigger potion of the log?
m

Mystery Incorporated

10/08/2021, 3:47 PM
it's syslog so it's liked peppered through syslog
maybe if you know a grep command I could use
t

Tomas Touceda

10/08/2021, 3:48 PM
those two lines you've shared don't tell me much, I would need to see a bigger chunk to get a clearer look
you can look for 
err
or 
ingest-err
m

Mystery Incorporated

10/08/2021, 3:51 PM
yea I looked and there isn't anything else
just cron complaining about syntax and the websocket error can' see anything else I did a grep "fleet"
well cat /var/log/syslog | grep "fleet"
I did it with err and ingest-err like you suggest, err just gives me the same results as "fleet" and ingest-err turns up nothing.
t

Tomas Touceda

10/08/2021, 3:55 PM
I suggest running fleet serve with debug logging enabled and sending logs in a way that you might be able to share them
m

Mystery Incorporated

10/08/2021, 3:56 PM
I would like to log to a dedicated file instead of syslog if possibl;e yea
t

Tomas Touceda

10/08/2021, 3:57 PM
not sure how you're running fleet, fleet logs to stderr, you should be able to pipe it to a file
m

Mystery Incorporated

10/08/2021, 3:58 PM
I running it in systemd unit
ExecStart=/usr/local/bin/fleet serve --config /home/terrance/fleet/config.yml
so I should just do like a > /var/log/fleet.log if I want?
t

Tomas Touceda

10/08/2021, 4:00 PM
https://stackoverflow.com/questions/37585758/how-to-redirect-output-of-systemd-service-to-a-file might help
m

Mystery Incorporated

10/08/2021, 4:06 PM
Ok I put these in 
StandardOutput=/var/log/fleet_standard.log
StandardError=/var/log/fleet_error.log
made the files, gave the user ownership, nothing being wrote to the files
t

Tomas Touceda

10/08/2021, 4:08 PM
that SO link suggests a different approach, through syslog
m

Mystery Incorporated

10/08/2021, 4:09 PM
oh it's already writing to syslog I want to write to own file
ok kcool by reading further down that SO article I see I need file: in the path
it works now
writing to own log
{"component":"crons","cron":"vulnerabilities","databases-path":"/mnt/fleetvuln","level":"info","ts":"2021-10-08T16:10:36.864309515Z"}
{"component":"crons","cron":"vulnerabilities","level":"info","periodicity":"1h0m0s","ts":"2021-10-08T16:10:36.864402755Z"}
{"address":"127.0.0.1:2498","msg":"listening","transport":"https","ts":"2021-10-08T16:10:36.883492137Z"}
They should be in stdout not stderr shouldn't they?
t

Tomas Touceda

10/08/2021, 4:12 PM
please add --logging_debug to your fleet serve call, so we get more verbose logging
m

Mystery Incorporated

10/08/2021, 4:15 PM
yep I did nopw
{
  "host": "xxxx",
  "level": "debug",
  "msg": "host reported software with empty name",
  "source": "programs",
  "ts": "2021-10-08T16:14:26.936622816Z",
  "version": "5.5.0.6704"
}
{
  "component": "http",
  "err": [
    "failed to save host software: insert software: Error 1054: Unknown column 'bundle_identifier' in 'field list'"
  ],
  "ip_addr": "127.0.0.1:42184",
  "level": "debug",
  "method": "POST",
  "took": "98.103886ms",
  "ts": "2021-10-08T16:14:25.29209961Z",
  "uri": "/api/v1/osquery/distributed/write",
  "x_for_ip_addr": "x.x.x.x"
}
Lots of those errors about 'bundle_identifier'
I did upgrade from 4.3.2 to 4.4 and i did do a prepare db before first launch
@Tomas Touceda I ran the Alter table command you gave ryan above that fixed it so I can only guess that a DB migration is missing in the 4.4 tag
z

zwass

10/08/2021, 5:05 PM
Thanks folks, looks like I made a mistake when cutting the 4.3.2 release so anyone upgrading 4.3.2->4.4.0 would be effected. We'll cut a 4.4.1 shortly that will fix this -- the manual fix discussed above also works (and the 4.4.1 fix will be compatible with that)
4.4.1 is published with the necessary fix. No need to update if you've already resolved this.
👍 1
m

Mystery Incorporated

10/09/2021, 2:08 AM
Thank you @zwass
12 Views
#fleetJoin Slack
Powered by