Title
#fleet
j

Joe

10/13/2022, 2:08 PM
Hi everyone, i'm currently investigating a reason for an excessive memory utilization coming from Fleet and i was wondering where i can find the logs. It doesn't seem to be in /var/log
Kathy Satterlee

Kathy Satterlee

10/13/2022, 2:30 PM
Hi @Joe ! By default, Fleet logs to the console with
stdout
and
stderr
.
j

Joe

10/13/2022, 2:38 PM
Ah perfect, thank you!
Kathy Satterlee

Kathy Satterlee

10/13/2022, 3:10 PM
Any time! Let me know if anything pops up that looks interesting/odd.
j

Joe

10/13/2022, 3:15 PM
Whenever we start the service, memory utilization spikes and then the service fails to start
3:16 PM
Do you by chance have any clue what might be causing it?
Kathy Satterlee

Kathy Satterlee

10/13/2022, 3:30 PM
Have there been any recent changes on your end (Fleet upgrade, etc) What version of Fleet are you using? How much memory is allocated?
j

Joe

10/13/2022, 3:32 PM
No changes have been made so far. We're running v4.17.1 and 8GB is allocated for the fleet server
Kathy Satterlee

Kathy Satterlee

10/13/2022, 3:46 PM
I'd definitely recommend updating Fleet to the latest version, there are some performance enhancements along the way 🙂 That being said, “context canceled” errors usually mean that the connection to the database is taking too long and timing out. Can you run the following on your database?
show engine innodb status;
show processlist;
j

Joe

10/13/2022, 5:46 PM
I tried running this under root but i still keep getting an access denied
MariaDB [kolide]> show engine innodb status;
ERROR 1227 (42000): Access denied; you need (at least one of) the PROCESS privilege(s) for this operation
MariaDB [kolide]> SHOW processlist;
+----+--------+-----------+--------+---------+------+-------+------------------+----------+
| Id | User   | Host      | db     | Command | Time | State | Info             | Progress |
+----+--------+-----------+--------+---------+------+-------+------------------+----------+
| 13 | kolide | localhost | kolide | Query   |    0 | init  | SHOW processlist |    0.000 |
+----+--------+-----------+--------+---------+------+-------+------------------+----------+
1 row in set (0.00 sec)
Kathy Satterlee

Kathy Satterlee

10/13/2022, 7:00 PM
One thing that stands out there is that you're using MariaDB. Just as a note, that isn't supported and we'd really recommend using MySQL. Just wanted to make sure... are you working with @Terra over on this thread?: https://osquery.slack.com/archives/C01DXJL16D8/p1665682341784749 The details here are similar enough that I want to make sure these were two separate Fleet setups.
j

Joe

10/13/2022, 7:21 PM
Yes i am
Kathy Satterlee

Kathy Satterlee

10/13/2022, 8:44 PM
Good deal. we'll keep the conversation going over in that thread just so I don't keep telling y'all to do the same thing 🙂