Title
#eclecticiq-polylogyx-extension
s

Shane Sanborn

03/01/2022, 7:31 PM
Hi team, is there anyway for the extension and osquery not to use as many resources. Our osquery configuration before we added the extension with its query was lower than 5% cpu on average, but now with it its has gone up to average around 40% combined and will spike higher. We are wondering if there is any method to lower this. Any help is appreciated
h

himanshu

03/07/2022, 4:32 AM
could you share more details which process is taking how much cpu when you see
40% combined
usage. at the same time, please share osquery.flags and osquery.conf being used.
o

OpenPlgx

03/07/2022, 9:35 AM
Would also be great to know the filters. @Shane Sanborn are you seeing telemetry data that you think can be whitelisted thru the filters?
s

Shane Sanborn

03/07/2022, 2:50 PM
The two process I am referring to are the plgx extension and the osqueryd. It seems like they shot up this high during the execution of the queries. Also, I have added a lot of filters to lower the data we are sending, so all of the "noisy" data that is left is data that is necessary.
2:52 PM
Also just for reference om how much data is sending.. some of our older osquery queries still send more data then the newly added plgx ones.
h

himanshu

03/07/2022, 4:23 PM
to start with, could you review
plgx_win.conf
and spread interval of scheduled queries to a higher time value.. most of the queries are scheduled for
"interval": 30
which may be eating up CPU.
o

OpenPlgx

03/07/2022, 4:42 PM
A couple of quick optimizations I can recommend:1. since you are apply constraints on some of the queries in plgx_win.conf, why not push those constraints as filters itself (include or exclude)? 2. Spreading the intervals (as already suggested) will help 3. Remote_thread is very noisy event. Would you want to consider going for an 'inclusion' based approach (on target process like lsass or others) instead of an 'exclusion' based? 4. file_time_stomp I believe is very chatty event too and the filter is very wide (e.g. *.exe). Would you want to tighten it to executables in certain folders e.g. 'downloads' or 'users'?
s

Shane Sanborn

03/08/2022, 3:03 PM
remote_thread isn't actually sending too much data with the filters we have on it just an fyi and also for file_time_stomp.. I'm not sure if the
*.exe
filter is even working because I am seeing data come in that has a target_path that doesn't include that.. do I have to provide like an exclude all filter for this to work or?
h

himanshu

03/13/2022, 5:27 PM
there is no separate filter implemented as
win_file_timestomp_events
.
win_file_events
filter should work for file timestomp events.
s

Shane Sanborn

03/28/2022, 2:34 PM
hey @himanshu I have worked on the filters and now have trimmed down the amount of events coming in and the osqueryd.exe process doesnt use as much resources, but the plgx process is using a lot of resources when it first starts up.. do you have any advice on this?
3:07 PM
also some more info, the extension process seems to spike on every osquery restart as well
h

himanshu

03/31/2022, 12:47 PM
is it only on osquery start up? if yes, the extension must be doing some initialization steps including registering itself with osquery, setting up filters and other config.
s

Shane Sanborn

03/31/2022, 7:55 PM
It spikes on all osquery restarts at around 50% cpu but it does look like it spikes higher on the first start up..
8:00 PM
Just for evidence.. the one with the low cpu is showing osquery not taking ab resources anymore and the one with the spike in the beginning is plgx on start up but it will spike like that on every restart as well
h

himanshu

04/02/2022, 11:45 AM
what is it that is scheduled every 10 minutes in your configuration? can you please check
s

Shane Sanborn

04/05/2022, 2:42 PM
the osquery spike that happens every 10 mins is fine for me because its such a low percentage spike only going up to like 4% but its caused by one of largest queries that we have been using for years..
2:44 PM
more info on this is that this is still be caused by some of our plgx queries not the actually restart.. the only reason it wasn't being shown in our testing was because the intervals were really long, I'm going to do some messing around with the intervals to see what queries exactly are causing the spikes..
h

himanshu

04/05/2022, 5:45 PM
ok. please let us know what are those plgx queries
s

Shane Sanborn

04/12/2022, 6:01 PM
Hi Team, I have done some testing and nothing with changing the queries seem to help and even took off all the queries.. also I took off all the filters just to see and it was still just as high which I guess is expected.
6:02 PM
Also just fyi the only process that is high is the plgx_extension process.. the actually osquery process remains normal for the current configs I posted.. could I get any advice on how to conitue please?
o

OpenPlgx

04/13/2022, 4:15 AM
So the spike you are seeing is what? ~50% on restart...is that correct? I mean, its only at the start time the spike is..and that's the current problem definition. Is that right?
s

Shane Sanborn

04/13/2022, 1:53 PM
No oh so I should clarify I thought the spike was on restart but it seems to be triggered on intervals.. even when I have no queries running that are related to the extension
o

OpenPlgx

04/13/2022, 4:31 PM
hmm..thats concerning for sure. For the sake of test, would you try filters such that all events from all categories are excluded and the extension is not collecting anything...I am curious to know if this is something being triggered by an osquery action or something in the event collection path.
s

Shane Sanborn

04/13/2022, 4:49 PM
okay will try that today
6:31 PM
Hey @OpenPlgx I put exclude all on my filters and ran this with queries and without the queries for this extension... results were the same for both, the cpu went back to normal and there were no spikes!
6:33 PM
I assume my filters aren't set properly I guess and there are too many events getting queried but not sure how to either fix my filters if they aren't working or if I need to add more filters, could you provide more assistance when you are free please
7:34 PM
I have more info now after more testing.. so it seems when I have my
win_image_load_events
in there, thats when the spike is occuring
o

OpenPlgx

04/15/2022, 3:15 AM
Interesting. Let me look at your config. I believe you had shared it before.
3:16 AM
Well, but the good news seems like we are getting somewhere 🙂