https://github.com/osquery/osquery logo
Join Slack
Powered by
The GitHub macOS runners look very horked, and I d...
# core
s
The GitHub macOS runners look very horked, and I don’t even know how to debug that
I’m not very hopeful, but I opened a support ticket. https://support.github.com/ticket/personal/0/3055697
Which was supposed to be in the osquery org, but looks like it’s personal? That’s just a straight up MS bug
s
What issues are you seeing? We have some TLS tests failing which I'm trying to look into.
s
It’s currently taking over 20 minutes for a runner to even pick up a job.
Screenshot 2024-10-18 at 08.22.35.png
s
We have multiple workflow runs still building
and I think we have a lower limit of concurrent jobs for macOS
s
You think maybe it’s just an overall osquery concurrancy issue?
s
yeah
s
All 4 of those were waiting for over 20 minutes
But that doesn’t disagree — maybe the other running actions were doing stuff
s
Yeah I think you merged 3 PRs or so one after the other
s
Sure but that was 12 hours ago. That screen shot was 10 minutes ago
s
The job I linked is from 11 hours ago
and it is still going, because with 3 PRs and scheduled builds it's easy to take that long
s
Oh, I had to re-run it. So it’s caught up in the UI chaos of of 11h ago vs this mornings rerun
It had failed sometime, and I reran it this morning
https://github.com/organizations/osquery/settings/actions/hosted-runners is interesting — it shows current active jobs and implies there’s a limit of 60
s
Yes but there should be a lower concurrency limit per platform
you cannot spin up 60 macOS runners
image.png
5 macOS runners
s
Ah, that’s the thing I was just looking for. So we have a limit of 5. Which means any PR or merge is going to use that
Thank you for finding it
s
(here btw)
s
For the 5.14.0 build.. 4 of the macOS jobs failed spuriously last night. I reran those this morning, 3 passed, but 1 failed again. I’ll rerun it, of course, but it will take awhile. I think the GH runners are a little underpowered, and combined that with our slow builds and it’s rough. (Not anyone’s fault really, just kind the state we’re in)
s
Well so, they were already failing in the scheduled build since several days ago, I thought we would hold off on getting a fix for the tests, otherwise it's totally random if they pass. Granted I suspect at this point it's because we have been keeping the python version on macOS runner fixed, and also with the x64 architecture selected. The builds started to fail when the runner image has been updated (so maybe a macOS update was applied?)
s
No idea
I opted to cut the release, on the theory that either it would work well enough, or we’d cut a 5.14.1
s
I think part of the slowness can be reduced a bit because most of the time for the macOS runners is spent on the test for the unified_log table which is particularly slow, especially after having to do the workaround of looping through all of the entries to avoid a leak, even if a 
max_rows
has been passed.
A part from the max_rows, the test doesn't pass a timestamp in several queries, so the table brings up the complete list of logs, and then it has to loop through all of them. We can try to play with a reasonable timestamp limit, where 
max_rows
is applied (since that's the test) because there are enough logs, but not that many that the leak prevention makes it take several minutes.
Technically passing the timestamp changes the tests slightly... but... it seems a bit eccessive otherwise
s
Really, most of that CI job is in that test? Ouch
s
yep, takes 15mins
Also the http tests failures do not help, since osquery has retries in some cases
s
Still haven’t gotten a working CI run. We might need to path that to get 5.14 out
Yeah.. This mac test has failed many times, so I think we’re going to have to think harder. No release for us yet.
s
As far as I can see the real issue is that the python server listens the port, and so the C++ code goes ahead in doing the test, but the http server (HTTPServer class in python) is not actually initialized, and takes 10s+. Not all tests have retries and so they fail (or before increasing the python server timeout, the C++ part would retry but the server would close). Probably something to do with Sequoia network permissions stuff.
I'm trying to add a logic in the python server start that checks if the server is actually ready to serve.
https://github.com/osquery/osquery/pull/8452 testing on my CI this should do. A bit hacky, ideally we would know why it takes 40s+ for the internal HTTP server to start serving in Sequoia. I don't have it installed yet, would need to prepare a VM with it.
s
I think with the work from Stefano, and myself, CI is in good enough shape we can cut a 5.14.1 🤞
I might have a theory for how to make 
unified_log
better
14 Views
Open in Slack
PreviousNext
Powered by