Title
#kolide
defensivedepth

defensivedepth

06/02/2022, 7:55 PM
I have a number of Launcher clients that are stuck running osquery 4.5.1, because the (updated) downloaded osqueryd is failing with a
segementation fault
{
  "binary": "/usr/local/launcher/bin/osqueryd-updates/1652735666/osqueryd",
  "binaryName": "osqueryd",
  "caller": "findnew.go:207",
  "fullBinaryPath": "/usr/local/launcher/bin/osqueryd",
  "level": "error",
  "msg": "not executable. Skipping",
  "reason": "signal: segmentation fault",
  "ts": "2022-06-02T14:04:21.625141787Z",
  "updateDir": "/usr/local/launcher/bin/osqueryd-updates"
}
I have removed the files in
/usr/local/launcher/bin/osqueryd-updates/
and restarted launcher. I see that the autoupdate process is kicked off:
{
  "binaryName": "osqueryd",
  "caller": "autoupdate.go:165",
  "level": "debug",
  "msg": "Created Updater",
  "stagingPath": "/var/launcher/securityonion/osqueryd-staging",
  "ts": "2022-06-02T19:44:59.357491281Z",
  "updater": "osqueryd",
  "updatesDirectory": "/usr/local/launcher/bin/osqueryd-updates"
}
But nothing appears to change. Any thoughts on how to troubleshoot this further?
s

seph

06/03/2022, 2:17 AM
There’s a lot to unpack here.
2:18 AM
Why is
/usr/local/launcher/bin/osqueryd-updates/1652735666/osqueryd
corrupt? Did something happen on your client side?
2:18 AM
“Stuck” is a bit odd — launcher doesn’t really get stuck per se. What it does is download updates when they happen, and then try to run them. If it can run a file on local disk, it’ll skip that.
2:22 AM
But…. launcher only runs updates when they happen. It has no real concept of what version is should run, just whether or not there has been a change. One way to force it to notice and re-update is to remove the local tuf store. This is less scary than it sounds — launcher will rebuild it from what’s baked into the binary. On your machines, that’s probably
var/launcher/securityonion/osqueryd-tuf
. rm that and restart launcher
defensivedepth

defensivedepth

06/03/2022, 8:42 PM
Thanks for the thoughts, still working on this. Removing the local tuf store & restarting launcher does not appear to force it to re-update, but it could be a timing issue. Double-checking some things.
8:44 PM
{
  "binaryName": "osqueryd",
  "caller": "findnew.go:230",
  "fullBinaryPath": "/usr/local/launcher/bin/osqueryd",
  "level": "debug",
  "msg": "no updates found",
  "ts": "2022-06-03T20:26:44.163520332Z",
  "updateDir": "/usr/local/launcher/bin/osqueryd-updates"
}
s

seph

06/03/2022, 8:47 PM
How long did you wait after restarting launcher? The updaters default to a 30 or 60 minut deay
defensivedepth

defensivedepth

06/03/2022, 8:49 PM
ya, thats what I mean RE: Timing. Had an issue with the test system I was working with and it rebooted, so not sure if it hit that 60min timer yet
8:53 PM
ah looks like osqueryd has an update delay of 90min
s

seph

06/03/2022, 8:54 PM
60 and 90 I guess. Seems believable.
defensivedepth

defensivedepth

06/03/2022, 8:55 PM
Still not sure the root cause of the issue