Terra
10/13/2022, 5:32 PMKathy Satterlee
10/13/2022, 5:45 PMTerra
10/13/2022, 5:50 PMsystemctl start fleet.service
, memory utilization continues to spike until the service kills itself. Checked the journalctl logs, there are only authentication logs {"component":"http","err":"authentication error: find host: context canceled","level":"info","path":"/api/v1/osquery/distributed/read","ts":"2022-10-13T13:50:30.147331381Z"}
. Non-Prod server is just fine.Kathy Satterlee
10/13/2022, 5:53 PMshow engine innodb status;
show processlist;
And what version of Fleet are you running?Terra
10/13/2022, 6:07 PM/etc/kolide.yaml
, but the kolide user is just a regular userKathy Satterlee
10/13/2022, 6:10 PMTerra
10/13/2022, 7:44 PM| InnoDB | |
=====================================
2022-10-13 12:43:05 0x7fac71af3700 INNODB MONITOR OUTPUT
=====================================
Per second averages calculated from the last 64 seconds
-----------------
BACKGROUND THREAD
-----------------
srv_master_thread loops: 0 srv_active, 0 srv_shutdown, 63 srv_idle
srv_master_thread log flush and writes: 63
----------
SEMAPHORES
----------
OS WAIT ARRAY INFO: reservation count 4
OS WAIT ARRAY INFO: signal count 4
RW-shared spins 0, rounds 6, OS waits 3
RW-excl spins 0, rounds 0, OS waits 0
RW-sx spins 0, rounds 0, OS waits 0
Spin rounds per wait: 6.00 RW-shared, 0.00 RW-excl, 0.00 RW-sx
------------
TRANSACTIONS
------------
Trx id counter 9477104131
Purge done for trx's n:o < 9477102680 undo n:o < 0 state: running but idle
History list length 2
LIST OF TRANSACTIONS FOR EACH SESSION:
---TRANSACTION 421853972091160, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
---TRANSACTION 421853972086936, not started
0 lock struct(s), heap size 1136, 0 row lock(s)
--------
FILE I/O
--------
I/O thread 0 state: waiting for completed aio requests (insert buffer thread)
I/O thread 1 state: waiting for completed aio requests (log thread)
I/O thread 2 state: waiting for completed aio requests (read thread)
I/O thread 3 state: waiting for completed aio requests (read thread)
I/O thread 4 state: waiting for completed aio requests (read thread)
I/O thread 5 state: waiting for completed aio requests (read thread)
I/O thread 6 state: waiting for completed aio requests (write thread)
I/O thread 7 state: waiting for completed aio requests (write thread)
I/O thread 8 state: waiting for completed aio requests (write thread)
I/O thread 9 state: waiting for completed aio requests (write thread)
Pending normal aio reads: [0, 0, 0, 0] , aio writes: [0, 0, 0, 0] ,
ibuf aio reads:, log i/o's:, sync i/o's:
Pending flushes (fsync) log: 0; buffer pool: 0
2290 OS file reads, 137 OS file writes, 6 OS fsyncs
35.78 reads/s, 17293 avg bytes/read, 2.14 writes/s, 0.09 fsyncs/s
-------------------------------------
INSERT BUFFER AND ADAPTIVE HASH INDEX
-------------------------------------
Ibuf: size 1, free list len 0, seg size 2, 0 merges
merged operations:
insert 0, delete mark 0, delete 0
discarded operations:
insert 0, delete mark 0, delete 0
Hash table size 34673, node heap has 0 buffer(s)
Hash table size 34673, node heap has 0 buffer(s)
Hash table size 34673, node heap has 0 buffer(s)
Hash table size 34673, node heap has 0 buffer(s)
Hash table size 34673, node heap has 0 buffer(s)
Hash table size 34673, node heap has 0 buffer(s)
Hash table size 34673, node heap has 0 buffer(s)
Hash table size 34673, node heap has 0 buffer(s)
0.00 hash searches/s, 10.69 non-hash searches/s
---
LOG
---
Log sequence number 3205314557869
Log flushed up to 3205314557869
Pages flushed up to 3205314557869
Last checkpoint at 3205314557860
0 pending log flushes, 0 pending chkp writes
10 log i/o's done, 0.16 log i/o's/second
----------------------
BUFFER POOL AND MEMORY
----------------------
Total large memory allocated 137494528
Dictionary memory allocated 168192
Buffer pool size 8191
Free buffers 5871
Database pages 2320
Old database pages 876
Modified db pages 0
Percent of dirty pages(LRU & free pages): 0.000
Max dirty pages percent: 75.000
Pending reads 0
Pending writes: LRU 0, flush list 0, single page 0
Pages made young 0, not young 0
0.00 youngs/s, 0.00 non-youngs/s
Pages read 2189, created 131, written 132
34.20 reads/s, 2.05 creates/s, 2.06 writes/s
Buffer pool hit rate 539 / 1000, young-making rate 0 / 1000 not 0 / 1000
Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s
LRU len: 2320, unzip_LRU len: 0
I/O sum[0]:cur[0], unzip sum[0]:cur[0]
--------------
ROW OPERATIONS
--------------
0 queries inside InnoDB, 0 queries in queue
0 read views open inside InnoDB
Process ID=9573, Main thread ID=140378519127808, state: sleeping
Number of rows inserted 0, updated 0, deleted 0, read 0
0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s
Number of system rows inserted 0, updated 0, deleted 0, read 0
0.00 inserts/s, 0.00 updates/s, 0.00 deletes/s, 0.00 reads/s
----------------------------
END OF INNODB MONITOR OUTPUT
============================
{
"mysql": "could not connect to db: dial tcp 127.0.0.1:3306: connect: connection refused, sleeping 0s",
"ts": "2022-10-13T19:55:01.604501336Z"
}
I do not know where the original admin credentials are stored or where to change this so Fleet can connect to the db again...Kathy Satterlee
10/13/2022, 8:03 PMTerra
10/13/2022, 8:36 PMKathy Satterlee
10/13/2022, 8:52 PMRSA
cert?Terra
10/13/2022, 8:56 PMhtop
is showing 85% usage on the Pro server whereas the Non-Prod server, fleet is only at 2%... We are using Digicert for both environmentsBenjamin Edwards
10/13/2022, 10:33 PMTerra
10/14/2022, 1:19 PM