I noticed that I am unable to open couple of hosts within fl

Question

I noticed that I am unable to open couple of hosts within fleetdm showing Unable to load host Please try again Showing as online though Live Queries also not working Had a look at the reverseproxy logs and found bazillion of HTTP 500 for POST to api v1 osquery distributed write But some other hosts are working fine live queries also working fine and they are located on the same network as the ones which are not working so I dont suspect any networking firewalling proxy issue

zwass · Answer

Hmm is there any more information about the 500 errors in the Fleet server logs Which version of Fleet are you running Can you run osqueryd manually on one of the effected hosts with ` verbose tls dump` and see if there s any information about what the actual error is

Daniel Weeber · Answer

Anything I should have a look for in special

Daniel Weeber · Answer

Only thing I notice so far is ``` error internal error load host additional load pack stats timestamp 2021 12 29T21 16 45+01 00 sql Scan error on column index 10 name last executed unsupported Scan storing driver Value type u003cnil u003e into type time Time ```

zwass · Answer

Which Fleet version is this IIRC we fixed this issue or something similar recently

Daniel Weeber · Answer

4 6 2 Will upgrade to 4 7 0 now

zwass · Answer

Please lmk if that resolves it I think we may have fixed this issue in 4 7 0

Daniel Weeber · Answer

Working Host is opening up again in fleetdm gui and also executing live queries But its painfully slow get openssl versions is taking like 1 2mins

Daniel Weeber · Answer

Exact query is ```SELECT name AS name version AS version deb packages AS source FROM deb packages WHERE name LIKE openssl UNION SELECT name AS name version AS version apt sources AS source FROM apt sources WHERE name LIKE openssl UNION SELECT name AS name version AS version rpm packages AS source FROM rpm packages WHERE name LIKE openssl ```

Daniel Weeber · Answer

just tried via osqueryi on local machine result is instant

Daniel Weeber · Answer

Even with distributed interval=1 its slow af

zwass · Answer

Also instant with `sudo osqueryi` on local machine

zwass · Answer

Are other hosts also slow Or just the previously effected ones

Daniel Weeber · Answer

yes same query on target machine is instant via osqueryi

Daniel Weeber · Answer

Nope presumably all of them

zwass · Answer

How s the CPU doing on your Fleet servers Redis MySQL

Daniel Weeber · Answer

Seems bored

Daniel Weeber · Answer

Webinterface of fleetdm is rapid fast

Daniel Weeber · Answer

Query of linux windows and macs my macbook is painfully slow just tested all 3 mac seems a bit faster but not much about 75 seconds for ```SELECT FROM system info ```

Tomas Touceda · Answer

hi there is that CPU graph for the fleet instance could you tell me a bit more about your setup eg what redis and mysql are you using how is CPU memory in there

Daniel Weeber · Answer

Sorry for late reply VM on big virtualization cluster flash only SAN and so on redis and mysql is local everything is blazing fast expect executing queries on hosts even on hosts in the same subnet as the fleet server i just checked again after my last message did not use fleet over christmas new years same problem but a bit different if I execute a query get openssl versions for the first time it takes about 10 seconds slow but could be okay if I then click on run again right afterwards it takes over 100 seconds

Daniel Weeber · Answer

and yes this was the CPU graph of the fleet vm

zwass · Answer

Did we already look at the configured `distributed interval` for the hosts You can see it on the host detail page

Daniel Weeber · Answer

Distributed interval 1 min

zwass · Answer

With that distributed interval up to 60s would be totally normal depending on where in the interval the host was when the live query started

Daniel Weeber · Answer

But I have distributed interval=10 in my osquery flags

zwass · Answer

Maybe you have it set differently in your agent options in Fleet

Daniel Weeber · Answer

You re right wow Just remove that line and apply Or do I have to restart the remote osquery agents

Daniel Weeber · Answer

Just removed the global agent config still 1min

Daniel Weeber · Answer

Okay removing the line restarting remote osqueryd and refetching now its showing 10sec

Daniel Weeber · Answer

Thank you

zwass · Answer

Sounds like things are working as expected now Or are you still seeing unexpectedly long delays