Hello everyone! Is it possible to increase logging verbosity for FleetDM to MySQL connections? Or may be some tips to check something? We still see such errors in our

journalctl -u fleetdm

log without exact understanding of reason:

May 06 10:24:55 fleet-01.test.tech fleet[3448836]: {"component":"http","err":"timestamp: 2022-05-06T10:24:48Z: error in query ingestion || timestamp: 2022-05-06T10:24:52Z: error in query ingestion || create transaction: timestamp: 2022-05-06T10:24:55Z: context canceled || save host with id 369: timestamp: 2022-05-06T10:24:55Z: context canceled","ingestion-err":"ingesting query users: update host users: create transaction: timestamp: 2022-05-06T10:24:52Z: context canceled","ip_addr":"172.10.11.10","level":"error","method":"POST","took":"22.51394562s","ts":"2022-05-06T10:24:55.778064954Z","uri":"/api/v1/osquery/distributed/write","x_for_ip_addr":"172.10.11.10"}
May 06 10:25:01 fleet-01.test.tech fleet[3448836]: {"component":"http","err":"timestamp: 2022-05-06T10:24:58Z: error in query ingestion || timestamp: 2022-05-06T10:25:01Z: error in query ingestion || timestamp: 2022-05-06T10:25:01Z: error in query ingestion || timestamp: 2022-05-06T10:25:01Z: error in query ingestion || timestamp: 2022-05-06T10:25:01Z: error in query ingestion || timestamp: 2022-05-06T10:25:01Z: error in query ingestion || getting app config: selecting app config: timestamp: 2022-05-06T10:25:01Z: context canceled","ingestion-err":"ingest detail query: selecting app config: timestamp: 2022-05-06T10:25:01Z: context canceled","ip_addr":"172.10.11.11","level":"error","method":"POST","took":"19.280912956s","ts":"2022-05-06T10:25:01.630667525Z","uri":"/api/v1/osquery/distributed/write","x_for_ip_addr":"172.10.11.11"}
May 06 10:25:03 fleet-01.test.tech fleet[3448836]: {"component":"http","err":"timestamp: 2022-05-06T10:24:58Z: error in query ingestion || create transaction: timestamp: 2022-05-06T10:25:03Z: context canceled || save host with id 403: timestamp: 2022-05-06T10:25:03Z: context canceled","ingestion-err":"ingesting query software_linux: update host software: insert software: timestamp: 2022-05-06T10:24:58Z: context canceled","ip_addr":"172.10.11.12","level":"error","method":"POST","took":"20.692362396s","ts":"2022-05-06T10:25:03.53958792Z","uri":"/api/v1/osquery/distributed/write","x_for_ip_addr":"172.10.11.12"}

We don’t see any errors or high load on MySQL. At the same time we started to see software inventory and vulnerabilities data. Visually it looks like this information is updated correctly according to intervals.

"context canceled" usually indicates a timeout of some sort. You can enable some extra logging for mysql by setting the environment variable

FLEET_ENABLE_DEV_SQL_INTERCEPTOR=true

Hi folks, something to check: those three "context canceled" errors shown here could be a timeout somewhere in the chain (TLS terminator, hosts themselves?), looks like the three requests are of ~20s (see

took

).

@Michal Nicpon hello! Could you please tell me, what kind of logs should I see after enabling this env? It looks like there is no additional data now in

journalctl -u fleetdm -f

logs

I forgot to mention that these logs are emitted at the debug, so you need to enable debug logging. See https://fleetdm.com/docs/deploying/configuration#logging-debug. Note that the logs may contain sensitive information

Also, fleet does not directly log to syslog (unless the administrator has configured it to do so by redirecting output). I believe Fleet logs to

stderr

by default.

Hello @Michal Nicpon! I did clear steps you described, but sorry, I still don’t see any new strings of data 😞 FLEET_ENABLE_DEV_SQL_INTERCEPTOR=true variable didn’t give me anything. If you have any other idea to check, it would be great! Thank you!

when configured, you should see logs like

level=debug ts=2022-05-12T16:37:31.338234946Z component=sql-interceptor duration=214.011µs query="SELECT count(*) as hosts_count, hm.version FROM host_munki_info hm JOIN hosts h ON (h.id = hm.host_id) WHERE h.team_id = ? AND hm.deleted_at IS NULL GROUP BY hm.version" args={1} err=null

Did you make sure to enable debug logging? Also, what version of fleet are you using?

How many scheduled queries do you have? and how many hosts?. (One guess we have is that scheduled query stats aggregation could be causing DB issues.)

@Michal Nicpon hello! Sorry for very long delay time from my side, I had to check some stuff with DevOps team, but have no results. I enabled debug loggin and also updated our FleetDM to current 4.14.0 version. And I still can’t enable logs with

component=sql-interceptor

I use such cmdline to manually start Fleet with SQL Interceptor and get logs to stdout

FLEET_ENABLE_DEV_SQL_INTERCEPTOR=true /usr/local/bin/fleet serve --config /etc/fleetdm/fleetdm.yml 2>&1

And I don’t see

component=sql-interceptor

logs you mentioned before. But there are lots of

component=http

and

component=crons

. I thank about mistake in my cmdline structure but saw that another option for json looging works fine

FLEET_LOGGING_JSON=true /usr/local/bin/fleet serve --config /etc/fleetdm/fleetdm.yml 2>&1

Maybe I should start FleetDM in dev mode some way?

Did you also enable debug logging using

FLEET_LOGGING_DEBUG=true

?

@Michal Nicpon sorry, it doesn’t work for me. I tried using

FLEET_ENABLE_DEV_SQL_INTERCEPTOR=true FLEET_LOGGING_DEBUG=true /usr/local/bin/fleet serve --config /etc/fleetdm/fleetdm.yml 2>&1 | grep -v component=http

for 1-2 hours, but didn’t see

component=sql-interceptor

data.

hi there. Given that you have no scheduled queries, let's try disabling software inventory and host users and see if that helps. Could you tell me a bit more about your infrastructure? eg: how many fleet instances, what resources does fleet, redis, and mysql have

--dev

is meant for developers, and sets mysql credentials to some hardcoded values for local development

@Michal Nicpon hi! So I need to mention, the sql-interceptor does not work for me. I don’t know why. Have you tested this mechanism in the new version of the FleetDM? Are you sure that it really works now?

@Artem the sql interceptor was not designed to be used in production environments, there might be quirks we haven't figured out

@Tomas Touceda okay, got it, thanks! I will try to migrate fleet’s mysql db to dedicated empty host and catch the difference in behavior.

all right, also the better we understand your infra, the better we can help troubleshoot: https://osquery.slack.com/archives/C01DXJL16D8/p1652964013906089?thread_ts=1651833264.541349&cid=C01DXJL16D8