Hi everyone, i'm currently investigating a reason for an excessive memory utilization coming from Fleet and i was wondering where i can find the logs. It doesn't seem to be in /var/log

Hi @Joe ! By default, Fleet logs to the console with

stdout

and

stderr

.

Ah perfect, thank you!

Any time! Let me know if anything pops up that looks interesting/odd.

Whenever we start the service, memory utilization spikes and then the service fails to start

Have there been any recent changes on your end (Fleet upgrade, etc) What version of Fleet are you using? How much memory is allocated?

No changes have been made so far. We're running v4.17.1 and 8GB is allocated for the fleet server

I'd definitely recommend updating Fleet to the latest version, there are some performance enhancements along the way 🙂 That being said, “context canceled” errors usually mean that the connection to the database is taking too long and timing out. Can you run the following on your database?

show engine innodb status;
show processlist;

I tried running this under root but i still keep getting an access denied

MariaDB [kolide]> show engine innodb status;
ERROR 1227 (42000): Access denied; you need (at least one of) the PROCESS privilege(s) for this operation

MariaDB [kolide]> SHOW processlist;
+----+--------+-----------+--------+---------+------+-------+------------------+----------+
| Id | User   | Host      | db     | Command | Time | State | Info             | Progress |
+----+--------+-----------+--------+---------+------+-------+------------------+----------+
| 13 | kolide | localhost | kolide | Query   |    0 | init  | SHOW processlist |    0.000 |
+----+--------+-----------+--------+---------+------+-------+------------------+----------+
1 row in set (0.00 sec)

One thing that stands out there is that you're using MariaDB. Just as a note, that isn't supported and we'd really recommend using MySQL. Just wanted to make sure... are you working with @Terra over on this thread?: https://osquery.slack.com/archives/C01DXJL16D8/p1665682341784749 The details here are similar enough that I want to make sure these were two separate Fleet setups.

Yes i am

Good deal. we'll keep the conversation going over in that thread just so I don't keep telling y'all to do the same thing 🙂