The socket usage should not create diskio. Please ...
# coret
theopolis
12/03/2019, 1:37 PM
The socket usage should not create diskio. Please investigate and let us know what you find.
s
seph
12/03/2019, 1:47 PM
Oh? I'll need to keep digging. The Internet implies unix sockets, being filesystem based, can trigger disk activity stuff
t
theopolis
12/03/2019, 1:50 PM
Do you have a reference? I can see that being within the realm of possibility but it's not where I would start my investigation.
- How often is the configuration updated
- What is the situation with logs, is something trying to write and failing
- What is osquery/extensions doing exactly, can you restart and strace it
- What queries are running / not just in the schedule, but in decorators, watchdog, etc
s
seph
12/03/2019, 2:01 PM
Still researching.
So far, I’m mostly tracing the kolide product, so it’s not a totally clean setup. But, what I see with strace:
1. A bunch of initial diskio from sqlite as tables are created
2. After that (during quintessence) socket and network writes.
seph
12/03/2019, 2:01 PM
Reading the internet again this morning — socket io is fileio but not inherently diskio
t
theopolis
12/03/2019, 2:04 PM
I am going off of my understanding, but sockets are implemented in memory, so you can create pressure that could lead to diskio, e.g., swapping, but that should be exceptional. Another curiosity, how are you measuring diskio?
s
seph
12/03/2019, 2:08 PM
Yes, measurement is where I need to dig next. I don’t have good tooling there. Some observations in iostat, but I need to get iotop running some more.
seph
12/03/2019, 2:09 PM
Either we don’t understand how tools report socketio, there’s something I’m not seeing in strace; or the end user is making some other error
t
timb
12/03/2019, 8:30 PMJust to confirm, this is only an issue for MacOS, no?
s
seph
12/03/2019, 10:11 PM
It’s not clear there is any issue here at all. Vs some confusion around how metrics are recorded. Currently, I’m leaning towards confusion.
seph
12/03/2019, 10:11 PM
But, this is unrelated top platforms, and I am primarily debugging this on linux
seph
12/04/2019, 3:40 AM
Digging a lot more, I have a new theory.
seph
12/04/2019, 3:40 AM
Some tools (
iotopseph
12/04/2019, 3:41 AM
However, taskset isn’t looking at diskio, it’s actually summing all read/write calls. So this includes any kind of io. The man page for
procrchar: characters read
The number of bytes which this task has caused to be
read from storage. This is simply the sum of bytes
which this process passed to read(2) and similar system
calls. It includes things such as terminal I/O and is
unaffected by whether or not actual physical disk I/O
was required (the read might have been satisfied from
pagecache).
seph
12/04/2019, 3:42 AM
So I think the extension traffic is yes io, but not disk io. And it’s simply misreported by common too.s
seph
12/04/2019, 3:42 AM
strace remains king here.
seph
12/04/2019, 3:42 AM
This conclusion brought to you by:
• docker
• testing
• reading the kernel source in places
• Reading docs
I have not yet written a test program to hammer on sockets and confirm it. strace seems clear enough
👍 1
t
theopolis
12/04/2019, 4:02 AM
This is sort of what I expected, a monitoring tool is confusing fileio as diskio (I can empathize!!)
s
seph
12/04/2019, 4:08 AM
Yes, it is what strace hinted at, in the beginning.
seph
12/04/2019, 4:09 AM
But I did not expect the monitoring tool to be the linux kernel.
t
theopolis
12/04/2019, 5:23 AM
something... something... spanish inquisition
4 Views