Jon O'Brien
03/07/2019, 4:59 PMfleetctl
not returning any results and updated it today with some more info... I'm afraid I'm new to fleetctl
so I could well be doing something wrong but afaik it seems to be setup correctly. Unfortunately I don't get any error messages just a blank line with no results - any suggestions or guidance on how to get more debug info (--debug
doesn't show anything) that could point me in the direction of what might be going wrong would be gratefully appreciated!
Thanks in advance!zwass
Jon O'Brien
03/07/2019, 5:47 PMzwass
Jon O'Brien
03/07/2019, 5:51 PMzwass
Jon O'Brien
03/07/2019, 5:52 PMzwass
ralph
03/07/2019, 10:32 PMzwass
ralph
03/07/2019, 10:41 PMzwass
Jon O'Brien
03/08/2019, 12:31 PMfleetctl
- i'm sure I've seen in some issues an amount of debug info being returned after running a command very similar to mine which returns nothing?zwass
fleetctl --version
)? Do you see any logs on the server indicating that the client connected? Any errors?Jon O'Brien
03/08/2019, 4:50 PMfleetctl - version 2.0.2
branch: master
revision: 8ca0358bf28173685815b79d8683a4239d629a14
build date: 2019-01-18T00:39:59Z
build user: zwass
go version: go1.11.3
Do you mean logs on the fleet server? If so, not sure where I'd find them (sorry)?zwass
Jon O'Brien
03/08/2019, 7:52 PMzwass
fleetctl get label 'All Hosts'
? Also, a screenshot of the Fleet dashboard page sidebard with the labels. Like this:Jon O'Brien
03/08/2019, 7:56 PM$ fleetctl get label 'All Hosts'
apiVersion: v1
kind: label
spec:
ID: 0
description: All hosts which have enrolled in Kolide
label_type: 1
name: All Hosts
query: select 1;
$
zwass
select 1
(which should include all hosts) and query against that label?Jon O'Brien
03/11/2019, 11:57 AMfleetctl
fails, the results are being written to /var/log/kolide/status.log
on each of the 3 load balanced kolide servers.6% responded (99% online) | 170/2680 targeted hosts (170/171 online)
$ fleetctl get label --context my_context 'test label'
apiVersion: v1
kind: label
spec:
ID: 0
description: ""
label_type: 0
name: test label
query: select 1
$
fleetctl
will execute and immediately fail:
[----------]$ fleetctl query --timeout 25m --query 'select name,version from os_version' --labels 'All Hosts' --exit
⠋ [----------]$
[----------]$ fleetctl query --timeout 25m --query 'select name,version from os_version' --labels 'All Hosts'
0% responded (0% online) | 0/0 targeted hosts (0/0 online)
[----------]$
On closer inspection of the /var/log/kolide/status.log
on each of the 3 load balanced kolide servers I can see that data is actually being captured - it's just not being returned to the fleetctl
call.
Is there a timing setting that might be causing fleetctl
to drop out before any results have been captured?zwass
Jon O'Brien
03/14/2019, 10:38 AMfleetctl
seems to work more reliably with just a small number of hosts online (even through our load balancer). It appears to be more of an issue when we have a couple thousand hosts online.zwass
Jon O'Brien
03/18/2019, 10:19 AMfleetctl
.
As a workaround, we have reduced the number of fleet servers down to 1 and early testing appears to indicate that fleetctl
is working reliably now.fleetctl
fails:
{"log":"{\"err\":\"sending: sockjs: session not in open state\",\"msg\":\"error writing to channel\",\"ts\":\"2019-03-24T04:08:13.081053265Z\"}\n","stream":"stderr","time":"2019-03-24T04:08:13.08115086Z"}
{"log":"{\"component\":\"service\",\"err\":null,\"ip_addr\":\"192.168.20.3:60635\",\"method\":\"SubmitDistributedQueryResults\",\"took\":\"8.070283ms\",\"ts\":\"2019-03-24T04:08:13.086910133Z\"}\n","stream":"stderr","time":"2019-03-24T04:08:13.087039196Z"}
{"log":"{\"err\":\"write status\",\"msg\":\"error updating status\",\"ts\":\"2019-03-24T04:08:13.322847484Z\"}\n","stream":"stderr","time":"2019-03-24T04:08:13.32299169Z"}
Any idea what might be causing this?zwass