Title
#fleet
a

Ahmed

08/20/2021, 12:26 PM
Hi Team, I’m getting this error when clicking on any host after the upgrade to latest version of fleet.
500: Oh, something went wrong.
Please file an issue if you believe this is a bug.

Error 1305: FUNCTION fleetdb.JSON_ARRAYAGG does not exist
Tomas Touceda

Tomas Touceda

08/20/2021, 12:26 PM
hi Ahmed, sorry to hear you're having trouble. I'm going to guess you have MariaDB 10.5.4 or lower?
a

Ahmed

08/20/2021, 12:27 PM
mysql --version
mysql  Ver 15.1 Distrib 10.4.11-MariaDB, for Linux (x86_64) using readline 5.1
Tomas Touceda

Tomas Touceda

08/20/2021, 12:28 PM
I recommend you upgrade to 10.5.10 or later
Mystery Incorporated

Mystery Incorporated

08/20/2021, 1:38 PM
@Tomas Touceda this prompted me to click on a host in the UI and my UI hangs and then the server crashes
Tomas Touceda

Tomas Touceda

08/20/2021, 1:39 PM
could you provide me with fleet logs and possibly the network requests in the site before it hangs?
Mystery Incorporated

Mystery Incorporated

08/20/2021, 1:40 PM
I'll try, but it seems it's actually crashed my server entirely so I'm trying to bring it back up
2:25 PM
@Tomas Touceda my mysql database won't even start anymore I think it killed it
Tomas Touceda

Tomas Touceda

08/20/2021, 2:28 PM
if you share logs from the fleet server and maybe your database, we can dive deeper in the issue
Mystery Incorporated

Mystery Incorporated

08/21/2021, 12:46 AM
Yep just trying to get the mysql server back up then I can start fleet and observe and collect logs.
2:09 AM
@Tomas Touceda got mysql restored, got everything started and waited for the vuln thing to happen, it's totally killed fleet and the server is crippled I can barely type a character when ssh in, I'll see if I can find any logs
1:01 AM
@Tomas Touceda
1:01 AM
Aug 22 01:27:18   fleet[726]: {"component":"crons","cron":"vulnerabilities","err":"getting cpes for: npm: fts5: syntax error near \".\"","level":"error","software-\u003ecpe":"error translating to CPE, skipping...","ts":"2021-08-21T15:27:18.112656123Z"}
Aug 22 01:27:18   fleet[726]: {"component":"crons","cron":"vulnerabilities","err":"getting cpes for: npm: fts5: syntax error near \".\"","level":"error","software-\u003ecpe":"error translating to CPE, skipping...","ts":"2021-08-21T15:27:18.113840108Z"}
Aug 22 01:28:50   fleet[726]: 2021/08/22 01:28:50 timeout waiting for SETTINGS frames from 127.0.0.1:3258
Aug 22 01:30:58   fleet[726]: 2021/08/22 01:30:58 timeout waiting for SETTINGS frames from 127.0.0.1:3554
Aug 22 01:37:26   fleet[726]: 2021/08/22 01:37:26 timeout waiting for SETTINGS frames from 127.0.0.1:4198
Aug 22 01:42:37   fleet[726]: 2021/08/22 01:42:37 http: TLS handshake error from 127.0.0.1:4654: write tcp 127.0.0.1:2498->127.0.0.1:4654: i/o timeout
Aug 22 01:42:40   fleet[726]: 2021/08/22 01:42:40 http2: server: error reading preface from client 127.0.0.1:4642: read tcp 127.0.0.1:2498->127.0.0.1:4642: i/o timeout
Aug 22 01:42:44   fleet[726]: 2021/08/22 01:42:44 http: TLS handshake error from 127.0.0.1:4668: write tcp 127.0.0.1:2498->127.0.0.1:4668: write: broken pipe
Aug 22 01:42:48   fleet[726]: 2021/08/22 01:42:48 http: TLS handshake error from 127.0.0.1:4672: write tcp 127.0.0.1:2498->127.0.0.1:4672: write: broken pipe
Aug 22 01:43:43   fleet[726]: 2021/08/22 01:43:40 http: TLS handshake error from 127.0.0.1:4680: write tcp 127.0.0.1:2498->127.0.0.1:4680: write: broken pipe
Aug 22 01:43:49   fleet[726]: 2021/08/22 01:43:49 http: TLS handshake error from 127.0.0.1:4700: write tcp 127.0.0.1:2498->127.0.0.1:4700: write: broken pipe
Aug 22 01:44:41   fleet[726]: 2021/08/22 01:44:41 http: TLS handshake error from 127.0.0.1:4708: write tcp 127.0.0.1:2498->127.0.0.1:4708: i/o timeout
Aug 22 01:44:47   fleet[726]: 2021/08/22 01:44:47 http: TLS handshake error from 127.0.0.1:4716: write tcp 127.0.0.1:2498->127.0.0.1:4716: write: broken pipe
Aug 22 01:44:49   fleet[726]: 2021/08/22 01:44:49 http: TLS handshake error from 127.0.0.1:4714: write tcp 127.0.0.1:2498->127.0.0.1:4714: write: broken pipe
Aug 22 01:44:50   fleet[726]: 2021/08/22 01:44:50 http: TLS handshake error from 127.0.0.1:4726: write tcp 127.0.0.1:2498->127.0.0.1:4726: write: broken pipe
Aug 22 01:46:43   fleet[726]: 2021/08/22 01:45:34 http: TLS handshake error from 127.0.0.1:4710: write tcp 127.0.0.1:2498->127.0.0.1:4710: write: broken pipe
Aug 22 01:46:43   fleet[726]: 2021/08/22 01:46:22 http: TLS handshake error from 127.0.0.1:4736: read tcp 127.0.0.1:2498->127.0.0.1:4736: i/o timeout
Aug 22 01:46:43   fleet[726]: 2021/08/22 01:46:34 http: TLS handshake error from 127.0.0.1:4728: write tcp 127.0.0.1:2498->127.0.0.1:4728: i/o timeout
Aug 22 05:37:47   fleet[726]: 2021/08/22 01:47:53 http: TLS handshake error from 127.0.0.1:4748: read tcp 127.0.0.1:2498->127.0.0.1:4748: i/o timeout
Aug 22 05:37:47   fleet[726]: 2021/08/22 01:49:00 http: TLS handshake error from 127.0.0.1:4758: read tcp 127.0.0.1:2498->127.0.0.1:4758: i/o timeout
Aug 22 05:37:47   fleet[726]: 2021/08/22 01:49:39 http: TLS handshake error from 127.0.0.1:4750: write tcp 127.0.0.1:2498->127.0.0.1:4750: i/o timeout
Aug 22 05:37:47   fleet[726]: 2021/08/22 01:50:21 http: TLS handshake error from 127.0.0.1:4756: write tcp 127.0.0.1:2498->127.0.0.1:4756: i/o timeout
Aug 22 05:37:47   fleet[726]: 2021/08/22 01:50:33 http: TLS handshake error from 127.0.0.1:4738: write tcp 127.0.0.1:2498->127.0.0.1:4738: i/o timeout
Aug 22 05:37:47   fleet[726]: 2021/08/22 01:51:27 http: TLS handshake error from 127.0.0.1:4774: read tcp 127.0.0.1:2498->127.0.0.1:4774: i/o timeout
Aug 22 05:37:47   fleet[726]: 2021/08/22 01:54:32 http: TLS handshake error from 127.0.0.1:4760: write tcp 127.0.0.1:2498->127.0.0.1:4760: i/o timeout
Aug 22 05:37:47   fleet[726]: 2021/08/22 01:55:00 http: TLS handshake error from 127.0.0.1:4768: write tcp 127.0.0.1:2498->127.0.0.1:4768: i/o timeout
Aug 22 05:37:47   fleet[726]: 2021/08/22 01:56:02 http: TLS handshake error from 127.0.0.1:4792: read tcp 127.0.0.1:2498->127.0.0.1:4792: i/o timeout
Aug 22 05:37:47   fleet[726]: 2021/08/22 01:56:12 http: TLS handshake error from 127.0.0.1:4770: write tcp 127.0.0.1:2498->127.0.0.1:4770: i/o timeout
Aug 22 05:37:47   fleet[726]: 2021/08/22 01:56:17 http: TLS handshake error from 127.0.0.1:4786: write tcp 127.0.0.1:2498->127.0.0.1:4786: i/o timeout
Aug 22 05:37:47   fleet[726]: 2021/08/22 01:56:29 http: TLS handshake error from 127.0.0.1:4776: write tcp 127.0.0.1:2498->127.0.0.1:4776: i/o timeout
Aug 22 05:37:47   fleet[726]: 2021/08/22 01:58:14 http: TLS handshake error from 127.0.0.1:4808: read tcp 127.0.0.1:2498->127.0.0.1:4808: i/o timeout
Aug 22 05:37:47   fleet[726]: 2021/08/22 01:58:40 http: TLS handshake error from 127.0.0.1:4806: read tcp 127.0.0.1:2498->127.0.0.1:4806: i/o timeout
Aug 22 05:37:47   fleet[726]: 2021/08/22 01:59:33 http: TLS handshake error from 127.0.0.1:4790: write tcp 127.0.0.1:2498->127.0.0.1:4790: i/o timeout
Aug 22 05:37:47   fleet[726]: 2021/08/22 02:00:08 http: TLS handshake error from 127.0.0.1:4788: write tcp 127.0.0.1:2498->127.0.0.1:4788: i/o timeout
Aug 22 05:37:47   fleet[726]: 2021/08/22 02:00:14 http: TLS handshake error from 127.0.0.1:4810: read tcp 127.0.0.1:2498->127.0.0.1:4810: i/o timeout
Aug 22 05:37:47   fleet[726]: 2021/08/22 02:01:54 http: TLS handshake error from 127.0.0.1:4818: read tcp 127.0.0.1:2498->127.0.0.1:4818: i/o timeout
Aug 22 05:37:47   fleet[726]: 2021/08/22 02:02:05 http: TLS handshake error from 127.0.0.1:4822: read tcp 127.0.0.1:2498->127.0.0.1:4822: i/o timeout
Aug 22 05:37:47   fleet[726]: 2021/08/22 02:03:42 http: TLS handshake error from 127.0.0.1:4824: read tcp 127.0.0.1:2498->127.0.0.1:4824: i/o timeout
Aug 22 05:37:47   fleet[726]: 2021/08/22 02:03:41 http: TLS handshake error from 127.0.0.1:4794: write tcp 127.0.0.1:2498->127.0.0.1:4794: i/o timeout
Aug 22 05:37:47   fleet[726]: 2021/08/22 02:04:33 http: TLS handshake error from 127.0.0.1:4826: read tcp 127.0.0.1:2498->127.0.0.1:4826: i/o timeout
Aug 22 05:37:47   fleet[726]: 2021/08/22 02:05:37 http: TLS handshake error from 127.0.0.1:4828: read tcp 127.0.0.1:2498->127.0.0.1:4828: i/o timeout
Aug 22 05:37:47   fleet[726]: 2021/08/22 02:09:23 http: TLS handshake error from 127.0.0.1:4830: read tcp 127.0.0.1:2498->127.0.0.1:4830: i/o timeout
Aug 22 05:37:47   fleet[726]: 2021/08/22 02:13:08 http: TLS handshake error from 127.0.0.1:4838: read tcp 127.0.0.1:2498->127.0.0.1:4838: i/o timeout
Aug 22 05:37:47   fleet[726]: 2021/08/22 02:15:11 http: TLS handshake error from 127.0.0.1:4800: write tcp 127.0.0.1:2498->127.0.0.1:4800: i/o timeout
Aug 22 05:37:40   systemd[1]: fleet.service: Main process exited, code=killed, status=9/KILL
Aug 22 05:37:40   systemd[1]: fleet.service: Failed with result 'signal'.
1:01 AM
message has been deleted
1:09 AM
And when I restart fleet after it crashed it says: fleet[10461]: {"component":"crons","cron":"vulnerabilities","err":"getting cpes for: increase-memory-limit: fts5: syntax error near "."","level":"error","software-\u003ecpe":"error translating to CPE, skipping...","ts":"2021-08-22T01:07:17.030214237Z"}
1:09 AM
I feel like it ate all my RAM and that's what's killing my server
1:11 AM
I'd say that's exactly what's happened, it's consuming RAM to the point where everything dies even my server dies and turns off