Hi guys, Could someone please help me to connect osquery agents on Windows Server 2019 with fleet? Here are some points: • I have same configs and deploy scripts both on problem servers and on basic Windows 10 VMs (works fine); • osquery agents successfully passes enrolling, I see it in logs and I see agents in UI, but they are in state "Never fetched" and mostly in "Offline" state; • I checked the debug output of osquery daemon and see only one possible issue:

W0207 18:02:24.934172 4352 watcher.cpp:391] osqueryd worker (560) stopping: Maximum sustainable CPU utilization limit exceeded: 18

close after executing

fleet_detail_query_software_windows

• Adding

--disable_watchdog=false --watchdog_delay=120 --watchdog_level=0 --watchdog_memory_limit=400 --watchdog_utilization_limit=21

was with no luck; And now I have no thoughts..

All of the failing hosts are Windows Server?

Right, Microsoft Windows Server 2019 Standard 10.0 and Microsoft Windows Server 2019 Datacenter 10.0

hi there, one thing you can try to narrow it all down to the software inventory query is disabling software inventory:

---
apiVersion: v1
kind: config
spec:
  host_settings:
    enable_software_inventory: false

if you apply that yaml, fleet will stop sending the software inventory queries to the hosts

Hi Tomas, Honestly, I tried to minimise query packs for these hosts. As they are Domain Controllers I expect too much ntfs or socket events. I attached the debug output of osquery daemon. You can see all applied queries. I have same result with disabled "process_all" query pack. My fleet installation is about 1k hosts, only 4 labels and no policies at all.

we have pending work to distribute the queries better for cases like this, I do notice you're running 5.1.0, would you be able to try 4.9.0? I have seen in the past issues with 5.x (in linux, though), but it would be good to discard some osquery issue

I dont have debug output for 4.9.0, but it was the same problem with it. I hopped upgrade will fix it 😞

ok, well that clears that doubt

Oh, I tried to disable software inventory as you recommended and it helped to resolve the issue! I also tried to set

enable_host_users = false

as that query returns about 1800 domain users, but it looks like that the issue only with

enable_software_inventory

that's great! so either the software inventory query pushed things over the edge, or it's taking too long. Would you be able to run this in one of the windows hosts:

WITH cached_users AS (SELECT * FROM users)
SELECT
  name AS name,
  version AS version,
  'Program (Windows)' AS type,
  'programs' AS source
FROM programs
UNION
SELECT
  name AS name,
  version AS version,
  'Package (Python)' AS type,
  'python_packages' AS source
FROM python_packages
UNION
SELECT
  name AS name,
  version AS version,
  'Browser plugin (IE)' AS type,
  'ie_extensions' AS source
FROM ie_extensions
UNION
SELECT
  name AS name,
  version AS version,
  'Browser plugin (Chrome)' AS type,
  'chrome_extensions' AS source
FROM cached_users CROSS JOIN chrome_extensions USING (uid)
UNION
SELECT
  name AS name,
  version AS version,
  'Browser plugin (Firefox)' AS type,
  'firefox_addons' AS source
FROM cached_users CROSS JOIN firefox_addons USING (uid)
UNION
SELECT
  name AS name,
  version AS version,
  'Package (Chocolatey)' AS type,
  'chocolatey_packages' AS source
FROM chocolatey_packages
UNION
SELECT
  name AS name,
  version AS version,
  'Package (Atom)' AS type,
  'atom_packages' AS source
FROM cached_users CROSS JOIN atom_packages USING (uid)
UNION
SELECT
  name AS name,
  version AS version,
  'Package (Python)' AS type,
  'python_packages' AS source
FROM python_packages;

to see if that's taking too long on its own?

So, users query returns me 1800 results (all domain users except computers). Software inventory uses

SELECT * FROM users

which returns ~3500 results (domain users + domain computers) and it increases CPU usage. I think this SELECT should be as in users query.

ah, interesting find!

If you do

select * from users

on that DC, do all of the users have a value for the

directory

column?

No, it looks like

directory

field exists only for accounts with successful local login. I attached redacted results with filled

directory

, all domain users looks like as in the last two lines.

hi there! this issue is still ongoing

Also encountered this, on both Windows 2016/2019 running osqueryd 5.2.2, but only on domain controllers though. No more issue after setting

enable_software_inventory: false

Same here on multiple Windows DCs. Disabled software_inventory & enable_host_users and things are stable