Hi, OpenPlgx! 1. We already filter all what is possible to. There is one more problem - new agents not registering and old are not receiving new config from server. 2. We use shallow config.

Can you please share the server specs and the server resource utilization(avg cpu and ram usage). Also, how many endpoints you have enrolled?

Agent works on Intel Xeon with 2 virtual cores and 4 Gb of RAM. CPU is about 11% in average, RAM about 60%. 9 agent deployed, but only 3 left after control server reinstall. Unfortunately i cannot provide

Last seen

due to 6 of 9 agent are not appeared.

I mean the ESP application server specs and resource usage. Last seen details for any one of the system would be fine

Oh, okay. Server also have 2 virtual cores of Intel Xeon Platinum 8000 and 16 GB RAM. One of agents is on screenshot

Can you please confirm few things to us to identify why the newly enrolled agents are not showing in ESP UI? 1. The server ip in the agent machine's osquery flags file(C:\Program Files\plgx_osquery\osquery.flags) with the flag

--tls_hostname

is same as your control server(ESP)? 2. Certificate(plgx-esp/nginx/certificate.crt) of the server is matching with the certificate you are trying to enroll the agent?

Hi, @Kishore Arava! 1. It's very embarrasing, but no - there was localhost. Now it works, thanks. There left one more question - memory leaks or smth like that causing RDP fails.

Yes. Might be. Hope agents are reading the updated config from server now, Based on the last screenshot you shared.

Is the memory usage coming down in time? I mean is it a spike..i dont think there is a leak here..it could be due to high event load on the system

Memory usage going like this in time. And sometimes server going unresponsive. Unfortunately i couldn't look at server in failed state due to failed RDP. Now we exclude all we can to still be secured with.

I can agree that 120 Mb is a a little high (and that also seemed like a spike rather than being consistent) but not terrible for a server work load..what kind of physical RAM you have on these server machines running the agent? I am surprised that a 120 Mb RAM is causing RDP failures

Unfortunately i couldn't tell about physical RAM because all our servers in virtual environment. 120 Mb is not maximum. Two days ago i see 486 Mb.

I understand. I was curious to know if these were transient conditions or were they steady state...for e.g. as I read, few of your servers (endpoints) were not connected with the ESP-server due to an error in the flagsfile. But this wouldnt stop the agent from collecting/caching the data in the meantime and the moment connectivity would resume, it would try to pump all the data back to ESP-Server causing a temporary spike.. is that what we are seeing here (or any such transient condition), is what I am trying to understand.., other reason could be the volume of activity which can be controlled thru tuning event filters specific to your environment, but for us to guide you on that would require you to be able to share some data..

What kind of a data do you need?

Recent Activity data on the system exhibiting the persistent high RAM usage + the plgx_event_filters config

There is almost no activity on most RAM consuming machine because of no user activity except of constantly working Process Explorer, which is excluded in agent configuration. Can exclusions count affect on a load?

This error is usually benign (it comes from osquery) but I think what might be happening on this system is for some reason, the benign error is acting up...are you seeing many such errors, as if it were in a loop?

Yes, there is a bunch of such errors. Procdump didn't work, by the way 😞

Without having access to data, it is indeed difficult to suggest options...here is one thing you could try to get rid of this restart loop (which I believe is happening due to some reason osquery being acting up on this system)

Hi, @OpenPlgx! Thanks, i will check this now

this means agent hasn't received server port from Options UI

custom_plgx_ServerPort

. Can you run plgx_osqueryd from command line as below and share any errors/warnings you see on CLI from polylogyx osquery extension? 1. run 'sc stop plgx_osqueryd' 2. in osquery.flags file, rename the osquery db file name to something else, say

--database_path=C:\Program Files\plgx_osquery\osquery_temp.db

3. from CLI, run 'plgx_osqueryd.exe --flagfile osquery.flags --verbose' 4. Note any warnings/errors.

@Michael, these errors shouldn't cause a functional issue...We can bury them later...but lets see if you still get the high RAM/CPU usage that you mentioned about (or the osquery agent start/stop situation)

Hi, guys! Thanks for replies, i very appreciate it. So, on the server where i manually stopped all, removed vast and vastnw, then started plgx_osqueryd service has failed after about 2 days of work. On the server was no activity at all. In current state it still doesn't work. Memory consumed for 97%. It looks like memory leak, but i can't catch anything. All that i've got - without Polylogyx ESP or Polymon all works well

@Michael, the images you have added seems to suggest the memory outage is in windows defender and/or FireFox

Hi, OpenPlgx! Yes, agent taking 7 Mb cause it's dead. This is only one case, i haven't seen such error on DC servers, for example, which is Core server without Firefox 🙂 I can clean sensitive data from config and share it, if you need to.

what do you mean its dead? Its showing in the taskmgr, right?

I think it's dead cause it doesn't collect any information - last log events was about two days ago.

ok, so no queries are active..

It's confusing me too. From my point of view it looks like memory leak.

leak where? if it was leak in the agent, taskmgr would show up it in agent..so its not a leak

No, errors about VRAM are not appearing anywhere, this is only case

yes, but can I request something simpler before?

Sure, np. What should i do?

unistall the agent, completely from the server showing up the issue

Okay, wait a minute, please

(plgx_cpt being the tool you must have downloaded to install the agent, from the server)

Done. Do you need the log?

no, looks good

Done

oh, you are running from powershell, then perhaps: 1. Get-Service plgx_osqueryd 2. Get-Service vast 3. Get-Service vastnw

done

can you adjust the osquery.conf to queries of your interest? (you wont be able to run any query for win_* _events table yet.)

Okay. It take some time cause we have several packs which contains both standard osquery and plgx stuff,

cool

Let me review them for sensitive data and i will share

sure

Great. So along with the .conf file, can you also share what queries/query packs you were running?

Sure. Here it is

it doesn't have any queries/packs for polylogyx tables?

no, it doesn't

but you get into situation when you enable queries on PolyLogyx tables, right?

Looks like. Most consuming process was

plgx_osqueryd.exe

that is just vanilla osquery renamed to plgx_osqueryd

Do you mean these?

1. Download the latest extension binary by cloning: https://github.com/polylogyx/osq-ext-bin 2. stop the osquery service on your system 3. move the file plgx_win_extension.ext.exe, extensions.load, osquery.flags in c:\program files\osquery 4. Adjust your osquery.conf to apply all the filters (based on the above) 5. from an admin command prompt, run notepad and change osquery.flags file's following line: --database_path=C:\Program Files\osquery\osquery.db to --database_path=C:\Program Files\osquery\osquery1.db 6. restart the osquery service 7. from the admin prompt, check to see vast/vasntw service are running. In case not, stop & start the osquery service again

Thanks, i will

great, lets see if you hit the memory usage issue

We do not use Defender, and exclusions in our anti-virus are implemented

Your earlier images/screen shots showed MsMpEng.exe running and consuming the highest RAM in the system ...MsMpEng.exe is the Windows Defender Engine

Oh, yes, my bad. Defender is working on the one of the test servers. Now we testing on servers without Defender

that is indeed surprising ..there is really no diff in what we did manually vs what would have happened thru the CPT tool

I understand, but all what happens have no logic at all. On tests before massive deployment all works perfectly. Then was deployment and problems with RAM and RDP have appeared. Two freaking weeks of collecting information and tries to understand what's going on and now - this. All just works again. I think about vast and vastnw - they are slightly relatives to Sysmon driver, and Sysmon once cause same problems, but long ago. Could it be possible?

vast/vastnw are indeed sysmon-ish drivers....

No, it's stop collect, i know details in other tab. When i send message, it doesn't works for about 1.5 hours. Process - alive, but no new events

The events are getting collected in the event log all the time (as long as the process is alive in memory). Certain events might get dropped (depending on the filter conditions) but there is no documented way to stop collecting the events...