https://github.com/osquery/osquery logo
Join Slack
Powered by
Hi team!! I was wondering if I could get some supp...
# fleet
b
Hi team!! I was wondering if I could get some support with a weird issue I'm having. The issue is that occasionally different scripts that I run on different hosts from Fleet will fail at completely random unrelated points and then immediately restart. I cannot find a single log line or anything that shows why the script is failing or being terminated but I can confirm it because I can see it with my eyes and that's it. I would guess the fleet script and it's child processes are being terminated as the script spawns some other processes that are getting closed and restarted (They span IBM Notifier popups). Once they are terminated, I'm guessing Fleet tries to re-run the script as if the machine had just rebooted Would really love some help digging into this issue. I feel like I've turned the macs inside out trying to find out what is happening but I'm not as familiar with the logging and script execution as the Fleet team is. Fleet version is v4.80.2, but has been happening since I'd guess around Fleet v4.77 on MacOS 26.1-3
k
Hey @Billy H! If the process running the script got terminated, that would definitely lead to the behavior you're seeing. Since the script never returns a value, Fleet assumes it did not run and runs it again.
b
@Kathy Satterlee Ok I'm glad to hear my hypothesis holds some merit! Any idea how I'd dig into logs to see why it's being terminated?
k
Do you also see the 
orbit
process restart?
Just want to make sure nothing's getting grumpy at it for something that's happening in the script.
b
well it's hard to tell (and I'll be honest I'm not sure how to check if orbit is restarting). This behavior is sporadic (1-2 times a day on my test machines) so I don't know when it's gonna happen
k
Does it consistently happen every time that script runs? If so, you could manually run the script and then check how long the orbit service has been running.
b
Unfortunately, no it does not. Its not even the same script. It happens to a bunch of scripts. It'll happen to the same script at different points in the execution. That being said next time I see it crash I'll check to see how long orbit has been running for
k
If it isn't happening consistently with the same script, it sounds like it may be unrelated to the actual script. Perhaps the osquery watchdog is kicking in and orbit is restarting... can you grab the orbit/osquery logs from a host that has had this behavior appear? https://fleetdm.com/guides/fleet-troubleshooting-for-it-admins#finding-fleetd-logs
b
Will do!
ok I can confirm that orbit is restarting when the scripts are restarting!
getting logs now
@Kathy Satterlee I've got the sanitized logs attached here. Looks like orbit rebooted at 3:44 PM (15:44ish in the logs) and I think specifically I'm seeing some errors here starting at line 652 in the log file:
Copy code
I0302 15:44:51.220070 1876733952 sqlite_util.cpp:303] DBManager contention: opening transient SQLite database
W0302 15:44:51.738408 1843736576 watcher.cpp:425] osqueryd worker (1288) stopping: Memory limits exceeded: 211582976 bytes (limit is 200MB)
I0302 15:44:51.738933 73310784 dispatcher.cpp:149] Thread: 0x2045ea240 requesting a stop
I0302 15:44:51.738967 73310784 dispatcher.cpp:156] Service: 0x6000012440e8 has been interrupted
I0302 15:44:51.738983 73310784 dispatcher.cpp:156] Service: 0x128704098 has been interrupted
I0302 15:44:51.739027 73310784 dispatcher.cpp:156] Service: 0x1287041a8 has been interrupted
I0302 15:44:51.739040 73310784 dispatcher.cpp:156] Service: 0x600001c4c018 has been interrupted
I0302 15:44:51.739051 73310784 dispatcher.cpp:156] Service: 0x128614538 has been interrupted
I0302 15:44:51.739059 73310784 dispatcher.cpp:156] Service: 0x600001c5c018 has been interrupted
I0302 15:44:51.739070 73310784 dispatcher.cpp:156] Service: 0x600001048638 has been interrupted
I0302 15:44:51.739080 73310784 dispatcher.cpp:122] Thread: 0x2045ea240 requesting a join
I0302 15:44:51.739104 73310784 dispatcher.cpp:140] Service thread: 0x600002b00000 has joined
I0302 15:44:51.739214 73310784 dispatcher.cpp:140] Service thread: 0x600002b55a60 has joined
I0302 15:44:51.739250 1868705792 extensions.cpp:244] Extension UUID 60519 shutdown request failed
and then again here at 794
Copy code
W0302 15:44:56.485975 1843736576 watcher.cpp:463] osqueryd worker (1288) could not be stopped. Sending kill signal.
I0302 15:44:56.537364 73310784 dispatcher.cpp:156] Service: 0x600003d5c018 has been interrupted
orbit.stderr_sanitized.log
this fleet issue may be related: https://github.com/fleetdm/fleet/issues/39927
u
Based on the timing we're seeing on those, I think you may be on the right track there.
b
got it! Should I add my logs and notes to that github issue?
@Kathy Satterlee looking at that issue and given that it is slated for 4.83.0, is there a temporary fix that can be applied in the meantime?
u
Unfortunately, there isn't one that I'm aware of. I'll definitely give you a yell if anything comes to mind!
👍🏻 1
3 Views
Open in Slack
PreviousNext
Powered by