https://github.com/osquery/osquery logo
#fleet
Title
# fleet
a

Ahmed

08/20/2021, 12:26 PM
Hi Team, I’m getting this error when clicking on any host after the upgrade to latest version of fleet.
Copy code
500: Oh, something went wrong.
Please file an issue if you believe this is a bug.

Error 1305: FUNCTION fleetdb.JSON_ARRAYAGG does not exist
t

Tomas Touceda

08/20/2021, 12:26 PM
hi Ahmed, sorry to hear you're having trouble. I'm going to guess you have MariaDB 10.5.4 or lower?
a

Ahmed

08/20/2021, 12:27 PM
Copy code
mysql --version
mysql  Ver 15.1 Distrib 10.4.11-MariaDB, for Linux (x86_64) using readline 5.1
t

Tomas Touceda

08/20/2021, 12:28 PM
I recommend you upgrade to 10.5.10 or later
👍 1
m

Mystery Incorporated

08/20/2021, 1:38 PM
@Tomas Touceda this prompted me to click on a host in the UI and my UI hangs and then the server crashes
t

Tomas Touceda

08/20/2021, 1:39 PM
could you provide me with fleet logs and possibly the network requests in the site before it hangs?
m

Mystery Incorporated

08/20/2021, 1:40 PM
I'll try, but it seems it's actually crashed my server entirely so I'm trying to bring it back up
@Tomas Touceda my mysql database won't even start anymore I think it killed it
t

Tomas Touceda

08/20/2021, 2:28 PM
if you share logs from the fleet server and maybe your database, we can dive deeper in the issue
m

Mystery Incorporated

08/21/2021, 12:46 AM
Yep just trying to get the mysql server back up then I can start fleet and observe and collect logs.
@Tomas Touceda got mysql restored, got everything started and waited for the vuln thing to happen, it's totally killed fleet and the server is crippled I can barely type a character when ssh in, I'll see if I can find any logs
@Tomas Touceda
Copy code
Aug 22 01:27:18   fleet[726]: {"component":"crons","cron":"vulnerabilities","err":"getting cpes for: npm: fts5: syntax error near \".\"","level":"error","software-\u003ecpe":"error translating to CPE, skipping...","ts":"2021-08-21T15:27:18.112656123Z"}
Aug 22 01:27:18   fleet[726]: {"component":"crons","cron":"vulnerabilities","err":"getting cpes for: npm: fts5: syntax error near \".\"","level":"error","software-\u003ecpe":"error translating to CPE, skipping...","ts":"2021-08-21T15:27:18.113840108Z"}
Aug 22 01:28:50   fleet[726]: 2021/08/22 01:28:50 timeout waiting for SETTINGS frames from 127.0.0.1:3258
Aug 22 01:30:58   fleet[726]: 2021/08/22 01:30:58 timeout waiting for SETTINGS frames from 127.0.0.1:3554
Aug 22 01:37:26   fleet[726]: 2021/08/22 01:37:26 timeout waiting for SETTINGS frames from 127.0.0.1:4198
Aug 22 01:42:37   fleet[726]: 2021/08/22 01:42:37 http: TLS handshake error from 127.0.0.1:4654: write tcp 127.0.0.1:2498->127.0.0.1:4654: i/o timeout
Aug 22 01:42:40   fleet[726]: 2021/08/22 01:42:40 http2: server: error reading preface from client 127.0.0.1:4642: read tcp 127.0.0.1:2498->127.0.0.1:4642: i/o timeout
Aug 22 01:42:44   fleet[726]: 2021/08/22 01:42:44 http: TLS handshake error from 127.0.0.1:4668: write tcp 127.0.0.1:2498->127.0.0.1:4668: write: broken pipe
Aug 22 01:42:48   fleet[726]: 2021/08/22 01:42:48 http: TLS handshake error from 127.0.0.1:4672: write tcp 127.0.0.1:2498->127.0.0.1:4672: write: broken pipe
Aug 22 01:43:43   fleet[726]: 2021/08/22 01:43:40 http: TLS handshake error from 127.0.0.1:4680: write tcp 127.0.0.1:2498->127.0.0.1:4680: write: broken pipe
Aug 22 01:43:49   fleet[726]: 2021/08/22 01:43:49 http: TLS handshake error from 127.0.0.1:4700: write tcp 127.0.0.1:2498->127.0.0.1:4700: write: broken pipe
Aug 22 01:44:41   fleet[726]: 2021/08/22 01:44:41 http: TLS handshake error from 127.0.0.1:4708: write tcp 127.0.0.1:2498->127.0.0.1:4708: i/o timeout
Aug 22 01:44:47   fleet[726]: 2021/08/22 01:44:47 http: TLS handshake error from 127.0.0.1:4716: write tcp 127.0.0.1:2498->127.0.0.1:4716: write: broken pipe
Aug 22 01:44:49   fleet[726]: 2021/08/22 01:44:49 http: TLS handshake error from 127.0.0.1:4714: write tcp 127.0.0.1:2498->127.0.0.1:4714: write: broken pipe
Aug 22 01:44:50   fleet[726]: 2021/08/22 01:44:50 http: TLS handshake error from 127.0.0.1:4726: write tcp 127.0.0.1:2498->127.0.0.1:4726: write: broken pipe
Aug 22 01:46:43   fleet[726]: 2021/08/22 01:45:34 http: TLS handshake error from 127.0.0.1:4710: write tcp 127.0.0.1:2498->127.0.0.1:4710: write: broken pipe
Aug 22 01:46:43   fleet[726]: 2021/08/22 01:46:22 http: TLS handshake error from 127.0.0.1:4736: read tcp 127.0.0.1:2498->127.0.0.1:4736: i/o timeout
Aug 22 01:46:43   fleet[726]: 2021/08/22 01:46:34 http: TLS handshake error from 127.0.0.1:4728: write tcp 127.0.0.1:2498->127.0.0.1:4728: i/o timeout
Aug 22 05:37:47   fleet[726]: 2021/08/22 01:47:53 http: TLS handshake error from 127.0.0.1:4748: read tcp 127.0.0.1:2498->127.0.0.1:4748: i/o timeout
Aug 22 05:37:47   fleet[726]: 2021/08/22 01:49:00 http: TLS handshake error from 127.0.0.1:4758: read tcp 127.0.0.1:2498->127.0.0.1:4758: i/o timeout
Aug 22 05:37:47   fleet[726]: 2021/08/22 01:49:39 http: TLS handshake error from 127.0.0.1:4750: write tcp 127.0.0.1:2498->127.0.0.1:4750: i/o timeout
Aug 22 05:37:47   fleet[726]: 2021/08/22 01:50:21 http: TLS handshake error from 127.0.0.1:4756: write tcp 127.0.0.1:2498->127.0.0.1:4756: i/o timeout
Aug 22 05:37:47   fleet[726]: 2021/08/22 01:50:33 http: TLS handshake error from 127.0.0.1:4738: write tcp 127.0.0.1:2498->127.0.0.1:4738: i/o timeout
Aug 22 05:37:47   fleet[726]: 2021/08/22 01:51:27 http: TLS handshake error from 127.0.0.1:4774: read tcp 127.0.0.1:2498->127.0.0.1:4774: i/o timeout
Aug 22 05:37:47   fleet[726]: 2021/08/22 01:54:32 http: TLS handshake error from 127.0.0.1:4760: write tcp 127.0.0.1:2498->127.0.0.1:4760: i/o timeout
Aug 22 05:37:47   fleet[726]: 2021/08/22 01:55:00 http: TLS handshake error from 127.0.0.1:4768: write tcp 127.0.0.1:2498->127.0.0.1:4768: i/o timeout
Aug 22 05:37:47   fleet[726]: 2021/08/22 01:56:02 http: TLS handshake error from 127.0.0.1:4792: read tcp 127.0.0.1:2498->127.0.0.1:4792: i/o timeout
Aug 22 05:37:47   fleet[726]: 2021/08/22 01:56:12 http: TLS handshake error from 127.0.0.1:4770: write tcp 127.0.0.1:2498->127.0.0.1:4770: i/o timeout
Aug 22 05:37:47   fleet[726]: 2021/08/22 01:56:17 http: TLS handshake error from 127.0.0.1:4786: write tcp 127.0.0.1:2498->127.0.0.1:4786: i/o timeout
Aug 22 05:37:47   fleet[726]: 2021/08/22 01:56:29 http: TLS handshake error from 127.0.0.1:4776: write tcp 127.0.0.1:2498->127.0.0.1:4776: i/o timeout
Aug 22 05:37:47   fleet[726]: 2021/08/22 01:58:14 http: TLS handshake error from 127.0.0.1:4808: read tcp 127.0.0.1:2498->127.0.0.1:4808: i/o timeout
Aug 22 05:37:47   fleet[726]: 2021/08/22 01:58:40 http: TLS handshake error from 127.0.0.1:4806: read tcp 127.0.0.1:2498->127.0.0.1:4806: i/o timeout
Aug 22 05:37:47   fleet[726]: 2021/08/22 01:59:33 http: TLS handshake error from 127.0.0.1:4790: write tcp 127.0.0.1:2498->127.0.0.1:4790: i/o timeout
Aug 22 05:37:47   fleet[726]: 2021/08/22 02:00:08 http: TLS handshake error from 127.0.0.1:4788: write tcp 127.0.0.1:2498->127.0.0.1:4788: i/o timeout
Aug 22 05:37:47   fleet[726]: 2021/08/22 02:00:14 http: TLS handshake error from 127.0.0.1:4810: read tcp 127.0.0.1:2498->127.0.0.1:4810: i/o timeout
Aug 22 05:37:47   fleet[726]: 2021/08/22 02:01:54 http: TLS handshake error from 127.0.0.1:4818: read tcp 127.0.0.1:2498->127.0.0.1:4818: i/o timeout
Aug 22 05:37:47   fleet[726]: 2021/08/22 02:02:05 http: TLS handshake error from 127.0.0.1:4822: read tcp 127.0.0.1:2498->127.0.0.1:4822: i/o timeout
Aug 22 05:37:47   fleet[726]: 2021/08/22 02:03:42 http: TLS handshake error from 127.0.0.1:4824: read tcp 127.0.0.1:2498->127.0.0.1:4824: i/o timeout
Aug 22 05:37:47   fleet[726]: 2021/08/22 02:03:41 http: TLS handshake error from 127.0.0.1:4794: write tcp 127.0.0.1:2498->127.0.0.1:4794: i/o timeout
Aug 22 05:37:47   fleet[726]: 2021/08/22 02:04:33 http: TLS handshake error from 127.0.0.1:4826: read tcp 127.0.0.1:2498->127.0.0.1:4826: i/o timeout
Aug 22 05:37:47   fleet[726]: 2021/08/22 02:05:37 http: TLS handshake error from 127.0.0.1:4828: read tcp 127.0.0.1:2498->127.0.0.1:4828: i/o timeout
Aug 22 05:37:47   fleet[726]: 2021/08/22 02:09:23 http: TLS handshake error from 127.0.0.1:4830: read tcp 127.0.0.1:2498->127.0.0.1:4830: i/o timeout
Aug 22 05:37:47   fleet[726]: 2021/08/22 02:13:08 http: TLS handshake error from 127.0.0.1:4838: read tcp 127.0.0.1:2498->127.0.0.1:4838: i/o timeout
Aug 22 05:37:47   fleet[726]: 2021/08/22 02:15:11 http: TLS handshake error from 127.0.0.1:4800: write tcp 127.0.0.1:2498->127.0.0.1:4800: i/o timeout
Aug 22 05:37:40   systemd[1]: fleet.service: Main process exited, code=killed, status=9/KILL
Aug 22 05:37:40   systemd[1]: fleet.service: Failed with result 'signal'.
And when I restart fleet after it crashed it says: fleet[10461]: {"component":"crons","cron":"vulnerabilities","err":"getting cpes for: increase-memory-limit: fts5: syntax error near \".\"","level":"error","software-\u003ecpe":"error translating to CPE, skipping...","ts":"2021-08-22T010717.030214237Z"}
I feel like it ate all my RAM and that's what's killing my server
I'd say that's exactly what's happened, it's consuming RAM to the point where everything dies even my server dies and turns off
29 Views