Title
#core
theopolis

theopolis

12/03/2019, 1:37 PM
The socket usage should not create diskio. Please investigate and let us know what you find.
s

seph

12/03/2019, 1:47 PM
Oh? I'll need to keep digging. The Internet implies unix sockets, being filesystem based, can trigger disk activity stuff
theopolis

theopolis

12/03/2019, 1:50 PM
Do you have a reference? I can see that being within the realm of possibility but it's not where I would start my investigation. - How often is the configuration updated - What is the situation with logs, is something trying to write and failing - What is osquery/extensions doing exactly, can you restart and strace it - What queries are running / not just in the schedule, but in decorators, watchdog, etc
s

seph

12/03/2019, 2:01 PM
Still researching. So far, I’m mostly tracing the kolide product, so it’s not a totally clean setup. But, what I see with strace:1. A bunch of initial diskio from sqlite as tables are created 2. After that (during quintessence) socket and network writes.
2:01 PM
Reading the internet again this morning — socket io is fileio but not inherently diskio
theopolis

theopolis

12/03/2019, 2:04 PM
I am going off of my understanding, but sockets are implemented in memory, so you can create pressure that could lead to diskio, e.g., swapping, but that should be exceptional. Another curiosity, how are you measuring diskio?
s

seph

12/03/2019, 2:08 PM
Yes, measurement is where I need to dig next. I don’t have good tooling there. Some observations in iostat, but I need to get iotop running some more.
2:09 PM
Either we don’t understand how tools report socketio, there’s something I’m not seeing in strace; or the end user is making some other error
t

timb

12/03/2019, 8:30 PM
Just to confirm, this is only an issue for MacOS, no?
s

seph

12/03/2019, 10:11 PM
It’s not clear there is any issue here at all. Vs some confusion around how metrics are recorded. Currently, I’m leaning towards confusion.
10:11 PM
But, this is unrelated top platforms, and I am primarily debugging this on linux
3:40 AM
Digging a lot more, I have a new theory.
3:40 AM
Some tools (
iotop
I’m looking at you) use linux’s taskset system to report “disk” activity.
3:41 AM
However, taskset isn’t looking at diskio, it’s actually summing all read/write calls. So this includes any kind of io. The man page for
proc
calls this out:
rchar: characters read
The number of bytes which this task has caused to be
read from storage. This is simply the sum of bytes
which this process passed to read(2) and similar system
calls. It includes things such as terminal I/O and is
unaffected by whether or not actual physical disk I/O
was required (the read might have been satisfied from
pagecache).
3:42 AM
So I think the extension traffic is yes io, but not disk io. And it’s simply misreported by common too.s
3:42 AM
strace remains king here.
3:42 AM
This conclusion brought to you by: • docker • testing • reading the kernel source in places • Reading docs I have not yet written a test program to hammer on sockets and confirm it. strace seems clear enough
theopolis

theopolis

12/04/2019, 4:02 AM
This is sort of what I expected, a monitoring tool is confusing fileio as diskio (I can empathize!!)
s

seph

12/04/2019, 4:08 AM
Yes, it is what strace hinted at, in the beginning.
4:09 AM
But I did not expect the monitoring tool to be the linux kernel.
theopolis

theopolis

12/04/2019, 5:23 AM
something... something... spanish inquisition