https://github.com/osquery/osquery logo
Join Slack
Powered by
I have a number of Launcher clients that are stuck...
# kolide
d
I have a number of Launcher clients that are stuck running osquery 4.5.1, because the (updated) downloaded osqueryd is failing with a 
segementation fault
Copy code
{
  "binary": "/usr/local/launcher/bin/osqueryd-updates/1652735666/osqueryd",
  "binaryName": "osqueryd",
  "caller": "findnew.go:207",
  "fullBinaryPath": "/usr/local/launcher/bin/osqueryd",
  "level": "error",
  "msg": "not executable. Skipping",
  "reason": "signal: segmentation fault",
  "ts": "2022-06-02T14:04:21.625141787Z",
  "updateDir": "/usr/local/launcher/bin/osqueryd-updates"
}
I have removed the files in 
/usr/local/launcher/bin/osqueryd-updates/
and restarted launcher. I see that the autoupdate process is kicked off:
Copy code
{
  "binaryName": "osqueryd",
  "caller": "autoupdate.go:165",
  "level": "debug",
  "msg": "Created Updater",
  "stagingPath": "/var/launcher/securityonion/osqueryd-staging",
  "ts": "2022-06-02T19:44:59.357491281Z",
  "updater": "osqueryd",
  "updatesDirectory": "/usr/local/launcher/bin/osqueryd-updates"
}
But nothing appears to change. Any thoughts on how to troubleshoot this further?
s
There’s a lot to unpack here.
Why is 
/usr/local/launcher/bin/osqueryd-updates/1652735666/osqueryd
corrupt? Did something happen on your client side?
“Stuck” is a bit odd — launcher doesn’t really get stuck per se. What it does is download updates when they happen, and then try to run them. If it can run a file on local disk, it’ll skip that.
But…. launcher only runs updates when they happen. It has no real concept of what version is should run, just whether or not there has been a change. One way to force it to notice and re-update is to remove the local tuf store. This is less scary than it sounds — launcher will rebuild it from what’s baked into the binary. On your machines, that’s probably 
var/launcher/securityonion/osqueryd-tuf
. rm that and restart launcher
d
Thanks for the thoughts, still working on this. Removing the local tuf store & restarting launcher does not appear to force it to re-update, but it could be a timing issue. Double-checking some things.
Copy code
{
  "binaryName": "osqueryd",
  "caller": "findnew.go:230",
  "fullBinaryPath": "/usr/local/launcher/bin/osqueryd",
  "level": "debug",
  "msg": "no updates found",
  "ts": "2022-06-03T20:26:44.163520332Z",
  "updateDir": "/usr/local/launcher/bin/osqueryd-updates"
}
s
How long did you wait after restarting launcher? The updaters default to a 30 or 60 minut deay
d
ya, thats what I mean RE: Timing. Had an issue with the test system I was working with and it rebooted, so not sure if it hit that 60min timer yet
ah looks like osqueryd has an update delay of 90min
s
60 and 90 I guess. Seems believable.
d
Still not sure the root cause of the issue
14 Views
Open in Slack
PreviousNext
Powered by