Regarding the problem with querying lots of hosts

Question

< nyanshak>Regarding the problem with querying lots of hosts I ve found my problem The problem was with Redis I noticed in the logs that Redis was dropping the client s subscription a few minutes after I launched a live query and that s what caused it to stuck I had to increase the output buffer limit in the Redis configuration and that fixed the problem

zwass · Answer

Glad to hear you worked that out How high is the throughput of results you are working with

nyanshak · Answer

I upgraded ElastiCache node type to try to get a larger buffer size for Redis but it didn t help at all In my case it doesn t even really look like redis is doing anything as the CPU network traffic is so low

nyanshak · Answer

Didn t fix my problem with querying

nyanshak · Answer

< daniel319b> what redis version other config do you have

nyanshak · Answer

< zwass>

daniel319b · Answer

< zwass> hmm do you mean the throughput in redis

daniel319b · Answer

< nyanshak> Yeah I noticed that the CPU is very low on my redis as well I m using 3 fleet servers behind an NLB 1 MySQL server latest version 1 Redis server latest version I played around with sql max connections I think its around 2000 now The Redis output buffer size limit is 512MB

daniel319b · Answer

Have you checked your Redis logs

daniel319b · Answer

It s still not working as fluidly or fast as I d like but it gets the job done

nyanshak · Answer

I m using elasticache and haven t checked redis logs but I did bump the node size to where output buffer size was much larger max size supported by amazon but didn t see any improvement

nyanshak · Answer

< daniel319b> how many hosts do you have connected

nyanshak · Answer

and what are your settings for config refresh and distributed interval

daniel319b · Answer

I have about 9K hosts I use the default values When you query do you select a label or All Hosts Because I ve noticed when I run a query on a label its slower

nyanshak · Answer

all hosts

nyanshak · Answer

the query i m running should be straightforward as well select uuid from osquery info

daniel319b · Answer

Whats your sql max conns And how much RAM CPU do your servers have Maybe you need to upgrade

daniel319b · Answer

Oh never mind I saw the github issue

nyanshak · Answer

Yeah It s a db m5 12xlarge and for fleet instances c5 4xlarge

nyanshak · Answer

I ve tried max conns at 50 256 512 2048

nyanshak · Answer

didn t seem to make much of a difference