Channels
doorman
infrastructure
random
zercurity
community-feeds
fleet-dev
code-review
queryhub
apple-silicon
carving
tls
fim
goquery
zentral
aws
querycon
golang
zeek
file-carving
fuzzing
auditing-warroom
linen-dev
fleetosquery
plugins
jobs
arm-architecture
darkbytes
process-auditing
uptycs
android_tests
selfgroup
vendor-feeds
fleet
eclecticiq-polylogyx-extension
ebpf
website
core
general
macos
kolide
osctrl
extensions
foundation
sql
officehours
linux
windows
Title
b
beatus
09/15/2020, 7:35 PM👋 Would anyone have some recommendations on where to look further on this?
2020-09-15T19:30:47.771465407Z {"component":"service","err":"failed to ingest result: campaign waiting for listener","ip_addr":"x","level":"debug","method":"SubmitDistributedQueryResults","took":"8.342794ms","ts":"2020-09-15T19:30:47.771281978Z","x_for_ip_addr":"x"}z
Zach Zeid
09/15/2020, 9:05 PMdo you have
--tls_dumpdo you have both distributed read/write endpoints configured?
z
zwass
09/16/2020, 4:47 PMAre you getting any live query results? It's not unreasonable to see a few of these errors if you have a very high frequency of hosts checking in. Under normal circumstances the host will just try to submit results again in a few seconds and by then the listener will be attached.
b
beatus
09/16/2020, 7:38 PM--tls_dumpzwass - it seems like they're completely broken
c
crimsonknave
09/16/2020, 7:42 PMWe have never gotten live queries to work reliably. This started when we moved to a 'proper' redis database managed by another team. Before we had one per cluster in a multicluster kubernetes deployement.
It was the distributed write endpoint that was failing when I was debugging this weekend.
We have 6500 online hosts and see about 460 fails every 10 seconds.
distributed_intervaldistributed_tls_max_attemptsSpecifically, our load balancer is seeing 500 errors from
/api/v1/distributed/writez
zwass
09/16/2020, 8:04 PMWhich version of fleet are each of you running?
That scale should be easily supported by Fleet 3.x
Fleet does run live queries to update the host "details" even if you are not running any manually.
b
beatus
09/16/2020, 8:13 PMwe upgraded to 3.1 last week
c
crimsonknave
09/16/2020, 8:38 PMI found out the 'fun' way that's how Fleet knows that hosts are online when I turned the setting off to try and stop the errors 😆
Should the
distributed_query_campaignsdeleted0We were able to resolve this by removing the live query entries in redis, the distributed_query_campaigns in the database and restarting osqueryd.
However, in our production cluster any live query fails and triggers this.. I'm going to write up an issue for that.