https://github.com/osquery/osquery logo
Join Slack
Powered by
Hey guys did something change with 4.4 and fetchin...
# fleet
m
Hey guys did something change with 4.4 and fetching? I have my fetch time to 27 minutes but as you can see I got some wacky fetch times (and most of these hosts are online 24/7)
Copy code
osquery:
  detail_update_interval: 27m
t
we added jitter of 10% of the interval to prevent hosts to spike the server, you can set it 0 if you prefer
j
wouldn't a 10% jitter be max 2.7 minutes for a 27 min update interval? they're showing a discrepancy of hours and days
t
right, if you could share fleet serve debug logs, that might shine some light as to what might be happening
m
hmm yea that's defs not 10% jitter I'll look at the logs
I tried to do a manual fetch and it's just stuck indefinitely fetching
j
do live queries work for these hosts?
m
Yes live query and the usual scheduled queries
@Tomas Touceda because OSQuery just spews so many errors into my status logs about being unable to access registry globs and can't resolve azuread users etc I'm struggling to see anything relevent in the log
t
how about the fleet serve logs?
m
where are they?
t
they are the logs that fleet serve command writes, so it's stderr and then wherever they go from there
m
@Tomas Touceda
Copy code
{
  "component": "crons",
  "cron": "vulnerabilities",
  "err": "getting cpes for: increase-memory-limit: fts5: syntax error near \".\"",
  "level": "error",
  "software->cpe": "error translating to CPE, skipping...",
  "ts": "2021-10-08T15:37:07.933390548Z"
}
Also
Copy code
{
  "component": "http",
  "err": "read auth token: reading from websocket: sockjs: session not in open state",
  "msg": "failed to read auth token",
  "ts": "2021-10-08T15:34:45.315116969Z"
}
t
could you share a bigger potion of the log?
m
it's syslog so it's liked peppered through syslog
maybe if you know a grep command I could use
t
those two lines you've shared don't tell me much, I would need to see a bigger chunk to get a clearer look
you can look for 
err
or 
ingest-err
m
yea I looked and there isn't anything else
just cron complaining about syntax and the websocket error can' see anything else I did a grep "fleet"
well cat /var/log/syslog | grep "fleet"
I did it with err and ingest-err like you suggest, err just gives me the same results as "fleet" and ingest-err turns up nothing.
t
I suggest running fleet serve with debug logging enabled and sending logs in a way that you might be able to share them
m
I would like to log to a dedicated file instead of syslog if possibl;e yea
t
not sure how you're running fleet, fleet logs to stderr, you should be able to pipe it to a file
m
I running it in systemd unit
Copy code
ExecStart=/usr/local/bin/fleet serve --config /home/terrance/fleet/config.yml
so I should just do like a > /var/log/fleet.log if I want?
t
m
Ok I put these in
Copy code
StandardOutput=/var/log/fleet_standard.log
StandardError=/var/log/fleet_error.log
made the files, gave the user ownership, nothing being wrote to the files
t
that SO link suggests a different approach, through syslog
m
oh it's already writing to syslog I want to write to own file
ok kcool by reading further down that SO article I see I need file: in the path
it works now
writing to own log
Copy code
{"component":"crons","cron":"vulnerabilities","databases-path":"/mnt/fleetvuln","level":"info","ts":"2021-10-08T16:10:36.864309515Z"}
{"component":"crons","cron":"vulnerabilities","level":"info","periodicity":"1h0m0s","ts":"2021-10-08T16:10:36.864402755Z"}
{"address":"127.0.0.1:2498","msg":"listening","transport":"https","ts":"2021-10-08T16:10:36.883492137Z"}
They should be in stdout not stderr shouldn't they?
t
please add --logging_debug to your fleet serve call, so we get more verbose logging
m
yep I did nopw
Copy code
{
  "host": "xxxx",
  "level": "debug",
  "msg": "host reported software with empty name",
  "source": "programs",
  "ts": "2021-10-08T16:14:26.936622816Z",
  "version": "5.5.0.6704"
}
Copy code
{
  "component": "http",
  "err": [
    "failed to save host software: insert software: Error 1054: Unknown column 'bundle_identifier' in 'field list'"
  ],
  "ip_addr": "127.0.0.1:42184",
  "level": "debug",
  "method": "POST",
  "took": "98.103886ms",
  "ts": "2021-10-08T16:14:25.29209961Z",
  "uri": "/api/v1/osquery/distributed/write",
  "x_for_ip_addr": "x.x.x.x"
}
Lots of those errors about 'bundle_identifier'
I did upgrade from 4.3.2 to 4.4 and i did do a prepare db before first launch
@Tomas Touceda I ran the Alter table command you gave ryan above that fixed it so I can only guess that a DB migration is missing in the 4.4 tag
z
Thanks folks, looks like I made a mistake when cutting the 4.3.2 release so anyone upgrading 4.3.2->4.4.0 would be effected. We'll cut a 4.4.1 shortly that will fix this -- the manual fix discussed above also works (and the 4.4.1 fix will be compatible with that)
4.4.1 is published with the necessary fix. No need to update if you've already resolved this.
👍 1
m
Thank you @zwass
15 Views
Open in Slack
PreviousNext
Powered by