Title
#fleet
a

Artem

05/06/2022, 10:34 AM
Hello everyone! Is it possible to increase logging verbosity for FleetDM to MySQL connections? Or may be some tips to check something? We still see such errors in our
journalctl -u fleetdm
log without exact understanding of reason:
May 06 10:24:55 fleet-01.test.tech fleet[3448836]: {"component":"http","err":"timestamp: 2022-05-06T10:24:48Z: error in query ingestion || timestamp: 2022-05-06T10:24:52Z: error in query ingestion || create transaction: timestamp: 2022-05-06T10:24:55Z: context canceled || save host with id 369: timestamp: 2022-05-06T10:24:55Z: context canceled","ingestion-err":"ingesting query users: update host users: create transaction: timestamp: 2022-05-06T10:24:52Z: context canceled","ip_addr":"172.10.11.10","level":"error","method":"POST","took":"22.51394562s","ts":"2022-05-06T10:24:55.778064954Z","uri":"/api/v1/osquery/distributed/write","x_for_ip_addr":"172.10.11.10"}
May 06 10:25:01 fleet-01.test.tech fleet[3448836]: {"component":"http","err":"timestamp: 2022-05-06T10:24:58Z: error in query ingestion || timestamp: 2022-05-06T10:25:01Z: error in query ingestion || timestamp: 2022-05-06T10:25:01Z: error in query ingestion || timestamp: 2022-05-06T10:25:01Z: error in query ingestion || timestamp: 2022-05-06T10:25:01Z: error in query ingestion || timestamp: 2022-05-06T10:25:01Z: error in query ingestion || getting app config: selecting app config: timestamp: 2022-05-06T10:25:01Z: context canceled","ingestion-err":"ingest detail query: selecting app config: timestamp: 2022-05-06T10:25:01Z: context canceled","ip_addr":"172.10.11.11","level":"error","method":"POST","took":"19.280912956s","ts":"2022-05-06T10:25:01.630667525Z","uri":"/api/v1/osquery/distributed/write","x_for_ip_addr":"172.10.11.11"}
May 06 10:25:03 fleet-01.test.tech fleet[3448836]: {"component":"http","err":"timestamp: 2022-05-06T10:24:58Z: error in query ingestion || create transaction: timestamp: 2022-05-06T10:25:03Z: context canceled || save host with id 403: timestamp: 2022-05-06T10:25:03Z: context canceled","ingestion-err":"ingesting query software_linux: update host software: insert software: timestamp: 2022-05-06T10:24:58Z: context canceled","ip_addr":"172.10.11.12","level":"error","method":"POST","took":"20.692362396s","ts":"2022-05-06T10:25:03.53958792Z","uri":"/api/v1/osquery/distributed/write","x_for_ip_addr":"172.10.11.12"}
We don’t see any errors or high load on MySQL. At the same time we started to see software inventory and vulnerabilities data. Visually it looks like this information is updated correctly according to intervals.
Michal Nicpon

Michal Nicpon

05/06/2022, 3:55 PM
"context canceled" usually indicates a timeout of some sort. You can enable some extra logging for mysql by setting the environment variable
FLEET_ENABLE_DEV_SQL_INTERCEPTOR=true
4:52 PM
Note, this is only intended to be used for development, not production use
Lucas Rodriguez

Lucas Rodriguez

05/06/2022, 4:53 PM
Hi folks, something to check: those three "context canceled" errors shown here could be a timeout somewhere in the chain (TLS terminator, hosts themselves?), looks like the three requests are of ~20s (see
took
).
a

Artem

05/08/2022, 9:54 AM
@Michal Nicpon hello! Could you please tell me, what kind of logs should I see after enabling this env? It looks like there is no additional data now in
journalctl -u fleetdm -f
logs
Michal Nicpon

Michal Nicpon

05/09/2022, 3:17 PM
I forgot to mention that these logs are emitted at the debug, so you need to enable debug logging. See https://fleetdm.com/docs/deploying/configuration#logging-debug. Note that the logs may contain sensitive information
Lucas Rodriguez

Lucas Rodriguez

05/09/2022, 3:19 PM
Also, fleet does not directly log to syslog (unless the administrator has configured it to do so by redirecting output). I believe Fleet logs to
stderr
by default.
a

Artem

05/12/2022, 9:20 AM
Hello @Michal Nicpon! I did clear steps you described, but sorry, I still don’t see any new strings of data 😞 FLEET_ENABLE_DEV_SQL_INTERCEPTOR=true variable didn’t give me anything. If you have any other idea to check, it would be great! Thank you!
Michal Nicpon

Michal Nicpon

05/12/2022, 4:38 PM
when configured, you should see logs like
level=debug ts=2022-05-12T16:37:31.338234946Z component=sql-interceptor duration=214.011µs query="SELECT count(*) as hosts_count, hm.version FROM host_munki_info hm JOIN hosts h ON (h.id = hm.host_id) WHERE h.team_id = ? AND hm.deleted_at IS NULL GROUP BY hm.version" args={1} err=null
Did you make sure to enable debug logging? Also, what version of fleet are you using?
Lucas Rodriguez

Lucas Rodriguez

05/12/2022, 5:41 PM
How many scheduled queries do you have? and how many hosts?. (One guess we have is that scheduled query stats aggregation could be causing DB issues.)
a

Artem

05/18/2022, 8:39 PM
@Michal Nicpon hello! Sorry for very long delay time from my side, I had to check some stuff with DevOps team, but have no results. I enabled debug loggin and also updated our FleetDM to current 4.14.0 version. And I still can’t enable logs with
component=sql-interceptor
I use such cmdline to manually start Fleet with SQL Interceptor and get logs to stdout
FLEET_ENABLE_DEV_SQL_INTERCEPTOR=true /usr/local/bin/fleet serve --config /etc/fleetdm/fleetdm.yml 2>&1
And I don’t see
component=sql-interceptor
logs you mentioned before. But there are lots of
component=http
and
component=crons
. I thank about mistake in my cmdline structure but saw that another option for json looging works fine
FLEET_LOGGING_JSON=true /usr/local/bin/fleet serve --config /etc/fleetdm/fleetdm.yml 2>&1
Maybe I should start FleetDM in dev mode some way?
8:42 PM
@Lucas Rodriguez hello! We have approximately 500 hosts connected. And we still have 0 scheduled queries, currently we only use ad-hoc. We still implementing FleetDM and try to solve such problems.
Michal Nicpon

Michal Nicpon

05/18/2022, 9:34 PM
Did you also enable debug logging using
FLEET_LOGGING_DEBUG=true
?
9:35 PM
The full command would be
FLEET_ENABLE_DEV_SQL_INTERCEPTOR=true FLEET_LOGGING_DEBUG=true /usr/local/bin/fleet serve --config /etc/fleetdm/fleetdm.yml 2>&1
a

Artem

05/19/2022, 8:09 AM
@Michal Nicpon sorry, it doesn’t work for me. I tried using
FLEET_ENABLE_DEV_SQL_INTERCEPTOR=true FLEET_LOGGING_DEBUG=true /usr/local/bin/fleet serve --config /etc/fleetdm/fleetdm.yml 2>&1 | grep -v component=http
for 1-2 hours, but didn’t see
component=sql-interceptor
data.
9:42 AM
@Michal Nicpon is it obligatory to set
--dev
argument to use FLEET_ENABLE_DEV_SQL_INTERCEPTOR=true?
12:31 PM
I also see some difference in logging behavior if I set
--dev
option with same mysql settings. P.S. I checked MySQL credentials via mysql command and had success connection.
Tomas Touceda

Tomas Touceda

05/19/2022, 12:40 PM
hi there. Given that you have no scheduled queries, let's try disabling software inventory and host users and see if that helps. Could you tell me a bit more about your infrastructure? eg: how many fleet instances, what resources does fleet, redis, and mysql have
Michal Nicpon

Michal Nicpon

05/19/2022, 3:44 PM
--dev
is meant for developers, and sets mysql credentials to some hardcoded values for local development
a

Artem

05/20/2022, 5:35 AM
@Michal Nicpon hi! So I need to mention, the sql-interceptor does not work for me. I don’t know why. Have you tested this mechanism in the new version of the FleetDM? Are you sure that it really works now?
Tomas Touceda

Tomas Touceda

05/20/2022, 1:38 PM
@Artem the sql interceptor was not designed to be used in production environments, there might be quirks we haven't figured out
a

Artem

05/20/2022, 1:42 PM
@Tomas Touceda okay, got it, thanks! I will try to migrate fleet’s mysql db to dedicated empty host and catch the difference in behavior.
Tomas Touceda

Tomas Touceda

05/20/2022, 1:55 PM
all right, also the better we understand your infra, the better we can help troubleshoot: https://osquery.slack.com/archives/C01DXJL16D8/p1652964013906089?thread_ts=1651833264.541349&cid=C01DXJL16D8