https://github.com/osquery/osquery logo
Join Slack
Powered by
Hi Team, I’m getting this error when clicking on a...
# fleet
a
Hi Team, I’m getting this error when clicking on any host after the upgrade to latest version of fleet.
Copy code
500: Oh, something went wrong.
Please file an issue if you believe this is a bug.

Error 1305: FUNCTION fleetdb.JSON_ARRAYAGG does not exist
t
hi Ahmed, sorry to hear you're having trouble. I'm going to guess you have MariaDB 10.5.4 or lower?
a
Copy code
mysql --version
mysql  Ver 15.1 Distrib 10.4.11-MariaDB, for Linux (x86_64) using readline 5.1
t
I recommend you upgrade to 10.5.10 or later
👍 1
m
@Tomas Touceda this prompted me to click on a host in the UI and my UI hangs and then the server crashes
t
could you provide me with fleet logs and possibly the network requests in the site before it hangs?
m
I'll try, but it seems it's actually crashed my server entirely so I'm trying to bring it back up
@Tomas Touceda my mysql database won't even start anymore I think it killed it
t
if you share logs from the fleet server and maybe your database, we can dive deeper in the issue
m
Yep just trying to get the mysql server back up then I can start fleet and observe and collect logs.
@Tomas Touceda got mysql restored, got everything started and waited for the vuln thing to happen, it's totally killed fleet and the server is crippled I can barely type a character when ssh in, I'll see if I can find any logs
@Tomas Touceda
Copy code
Aug 22 01:27:18   fleet[726]: {"component":"crons","cron":"vulnerabilities","err":"getting cpes for: npm: fts5: syntax error near \".\"","level":"error","software-\u003ecpe":"error translating to CPE, skipping...","ts":"2021-08-21T15:27:18.112656123Z"}
Aug 22 01:27:18   fleet[726]: {"component":"crons","cron":"vulnerabilities","err":"getting cpes for: npm: fts5: syntax error near \".\"","level":"error","software-\u003ecpe":"error translating to CPE, skipping...","ts":"2021-08-21T15:27:18.113840108Z"}
Aug 22 01:28:50   fleet[726]: 2021/08/22 01:28:50 timeout waiting for SETTINGS frames from 127.0.0.1:3258
Aug 22 01:30:58   fleet[726]: 2021/08/22 01:30:58 timeout waiting for SETTINGS frames from 127.0.0.1:3554
Aug 22 01:37:26   fleet[726]: 2021/08/22 01:37:26 timeout waiting for SETTINGS frames from 127.0.0.1:4198
Aug 22 01:42:37   fleet[726]: 2021/08/22 01:42:37 http: TLS handshake error from 127.0.0.1:4654: write tcp 127.0.0.1:2498->127.0.0.1:4654: i/o timeout
Aug 22 01:42:40   fleet[726]: 2021/08/22 01:42:40 http2: server: error reading preface from client 127.0.0.1:4642: read tcp 127.0.0.1:2498->127.0.0.1:4642: i/o timeout
Aug 22 01:42:44   fleet[726]: 2021/08/22 01:42:44 http: TLS handshake error from 127.0.0.1:4668: write tcp 127.0.0.1:2498->127.0.0.1:4668: write: broken pipe
Aug 22 01:42:48   fleet[726]: 2021/08/22 01:42:48 http: TLS handshake error from 127.0.0.1:4672: write tcp 127.0.0.1:2498->127.0.0.1:4672: write: broken pipe
Aug 22 01:43:43   fleet[726]: 2021/08/22 01:43:40 http: TLS handshake error from 127.0.0.1:4680: write tcp 127.0.0.1:2498->127.0.0.1:4680: write: broken pipe
Aug 22 01:43:49   fleet[726]: 2021/08/22 01:43:49 http: TLS handshake error from 127.0.0.1:4700: write tcp 127.0.0.1:2498->127.0.0.1:4700: write: broken pipe
Aug 22 01:44:41   fleet[726]: 2021/08/22 01:44:41 http: TLS handshake error from 127.0.0.1:4708: write tcp 127.0.0.1:2498->127.0.0.1:4708: i/o timeout
Aug 22 01:44:47   fleet[726]: 2021/08/22 01:44:47 http: TLS handshake error from 127.0.0.1:4716: write tcp 127.0.0.1:2498->127.0.0.1:4716: write: broken pipe
Aug 22 01:44:49   fleet[726]: 2021/08/22 01:44:49 http: TLS handshake error from 127.0.0.1:4714: write tcp 127.0.0.1:2498->127.0.0.1:4714: write: broken pipe
Aug 22 01:44:50   fleet[726]: 2021/08/22 01:44:50 http: TLS handshake error from 127.0.0.1:4726: write tcp 127.0.0.1:2498->127.0.0.1:4726: write: broken pipe
Aug 22 01:46:43   fleet[726]: 2021/08/22 01:45:34 http: TLS handshake error from 127.0.0.1:4710: write tcp 127.0.0.1:2498->127.0.0.1:4710: write: broken pipe
Aug 22 01:46:43   fleet[726]: 2021/08/22 01:46:22 http: TLS handshake error from 127.0.0.1:4736: read tcp 127.0.0.1:2498->127.0.0.1:4736: i/o timeout
Aug 22 01:46:43   fleet[726]: 2021/08/22 01:46:34 http: TLS handshake error from 127.0.0.1:4728: write tcp 127.0.0.1:2498->127.0.0.1:4728: i/o timeout
Aug 22 05:37:47   fleet[726]: 2021/08/22 01:47:53 http: TLS handshake error from 127.0.0.1:4748: read tcp 127.0.0.1:2498->127.0.0.1:4748: i/o timeout
Aug 22 05:37:47   fleet[726]: 2021/08/22 01:49:00 http: TLS handshake error from 127.0.0.1:4758: read tcp 127.0.0.1:2498->127.0.0.1:4758: i/o timeout
Aug 22 05:37:47   fleet[726]: 2021/08/22 01:49:39 http: TLS handshake error from 127.0.0.1:4750: write tcp 127.0.0.1:2498->127.0.0.1:4750: i/o timeout
Aug 22 05:37:47   fleet[726]: 2021/08/22 01:50:21 http: TLS handshake error from 127.0.0.1:4756: write tcp 127.0.0.1:2498->127.0.0.1:4756: i/o timeout
Aug 22 05:37:47   fleet[726]: 2021/08/22 01:50:33 http: TLS handshake error from 127.0.0.1:4738: write tcp 127.0.0.1:2498->127.0.0.1:4738: i/o timeout
Aug 22 05:37:47   fleet[726]: 2021/08/22 01:51:27 http: TLS handshake error from 127.0.0.1:4774: read tcp 127.0.0.1:2498->127.0.0.1:4774: i/o timeout
Aug 22 05:37:47   fleet[726]: 2021/08/22 01:54:32 http: TLS handshake error from 127.0.0.1:4760: write tcp 127.0.0.1:2498->127.0.0.1:4760: i/o timeout
Aug 22 05:37:47   fleet[726]: 2021/08/22 01:55:00 http: TLS handshake error from 127.0.0.1:4768: write tcp 127.0.0.1:2498->127.0.0.1:4768: i/o timeout
Aug 22 05:37:47   fleet[726]: 2021/08/22 01:56:02 http: TLS handshake error from 127.0.0.1:4792: read tcp 127.0.0.1:2498->127.0.0.1:4792: i/o timeout
Aug 22 05:37:47   fleet[726]: 2021/08/22 01:56:12 http: TLS handshake error from 127.0.0.1:4770: write tcp 127.0.0.1:2498->127.0.0.1:4770: i/o timeout
Aug 22 05:37:47   fleet[726]: 2021/08/22 01:56:17 http: TLS handshake error from 127.0.0.1:4786: write tcp 127.0.0.1:2498->127.0.0.1:4786: i/o timeout
Aug 22 05:37:47   fleet[726]: 2021/08/22 01:56:29 http: TLS handshake error from 127.0.0.1:4776: write tcp 127.0.0.1:2498->127.0.0.1:4776: i/o timeout
Aug 22 05:37:47   fleet[726]: 2021/08/22 01:58:14 http: TLS handshake error from 127.0.0.1:4808: read tcp 127.0.0.1:2498->127.0.0.1:4808: i/o timeout
Aug 22 05:37:47   fleet[726]: 2021/08/22 01:58:40 http: TLS handshake error from 127.0.0.1:4806: read tcp 127.0.0.1:2498->127.0.0.1:4806: i/o timeout
Aug 22 05:37:47   fleet[726]: 2021/08/22 01:59:33 http: TLS handshake error from 127.0.0.1:4790: write tcp 127.0.0.1:2498->127.0.0.1:4790: i/o timeout
Aug 22 05:37:47   fleet[726]: 2021/08/22 02:00:08 http: TLS handshake error from 127.0.0.1:4788: write tcp 127.0.0.1:2498->127.0.0.1:4788: i/o timeout
Aug 22 05:37:47   fleet[726]: 2021/08/22 02:00:14 http: TLS handshake error from 127.0.0.1:4810: read tcp 127.0.0.1:2498->127.0.0.1:4810: i/o timeout
Aug 22 05:37:47   fleet[726]: 2021/08/22 02:01:54 http: TLS handshake error from 127.0.0.1:4818: read tcp 127.0.0.1:2498->127.0.0.1:4818: i/o timeout
Aug 22 05:37:47   fleet[726]: 2021/08/22 02:02:05 http: TLS handshake error from 127.0.0.1:4822: read tcp 127.0.0.1:2498->127.0.0.1:4822: i/o timeout
Aug 22 05:37:47   fleet[726]: 2021/08/22 02:03:42 http: TLS handshake error from 127.0.0.1:4824: read tcp 127.0.0.1:2498->127.0.0.1:4824: i/o timeout
Aug 22 05:37:47   fleet[726]: 2021/08/22 02:03:41 http: TLS handshake error from 127.0.0.1:4794: write tcp 127.0.0.1:2498->127.0.0.1:4794: i/o timeout
Aug 22 05:37:47   fleet[726]: 2021/08/22 02:04:33 http: TLS handshake error from 127.0.0.1:4826: read tcp 127.0.0.1:2498->127.0.0.1:4826: i/o timeout
Aug 22 05:37:47   fleet[726]: 2021/08/22 02:05:37 http: TLS handshake error from 127.0.0.1:4828: read tcp 127.0.0.1:2498->127.0.0.1:4828: i/o timeout
Aug 22 05:37:47   fleet[726]: 2021/08/22 02:09:23 http: TLS handshake error from 127.0.0.1:4830: read tcp 127.0.0.1:2498->127.0.0.1:4830: i/o timeout
Aug 22 05:37:47   fleet[726]: 2021/08/22 02:13:08 http: TLS handshake error from 127.0.0.1:4838: read tcp 127.0.0.1:2498->127.0.0.1:4838: i/o timeout
Aug 22 05:37:47   fleet[726]: 2021/08/22 02:15:11 http: TLS handshake error from 127.0.0.1:4800: write tcp 127.0.0.1:2498->127.0.0.1:4800: i/o timeout
Aug 22 05:37:40   systemd[1]: fleet.service: Main process exited, code=killed, status=9/KILL
Aug 22 05:37:40   systemd[1]: fleet.service: Failed with result 'signal'.
And when I restart fleet after it crashed it says: fleet[10461]: {"component":"crons","cron":"vulnerabilities","err":"getting cpes for: increase-memory-limit: fts5: syntax error near \".\"","level":"error","software-\u003ecpe":"error translating to CPE, skipping...","ts":"2021-08-22T010717.030214237Z"}
I feel like it ate all my RAM and that's what's killing my server
I'd say that's exactly what's happened, it's consuming RAM to the point where everything dies even my server dies and turns off
33 Views
Open in Slack
PreviousNext
Powered by