Channels
doorman
infrastructure
random
zercurity
community-feeds
fleet-dev
code-review
queryhub
apple-silicon
carving
tls
fim
goquery
zentral
aws
querycon
golang
zeek
file-carving
fuzzing
auditing-warroom
linen-dev
fleetosquery
plugins
jobs
arm-architecture
darkbytes
process-auditing
uptycs
android_tests
selfgroup
vendor-feeds
fleet
eclecticiq-polylogyx-extension
ebpf
website
core
general
macos
kolide
osctrl
extensions
foundation
sql
officehours
linux
windows
Title
t
theopolis
12/03/2019, 1:37 PMThe socket usage should not create diskio. Please investigate and let us know what you find.
s
seph
12/03/2019, 1:47 PMOh? I'll need to keep digging. The Internet implies unix sockets, being filesystem based, can trigger disk activity stuff
t
theopolis
12/03/2019, 1:50 PMDo you have a reference? I can see that being within the realm of possibility but it's not where I would start my investigation.
- How often is the configuration updated
- What is the situation with logs, is something trying to write and failing
- What is osquery/extensions doing exactly, can you restart and strace it
- What queries are running / not just in the schedule, but in decorators, watchdog, etc
s
seph
12/03/2019, 2:01 PMStill researching.
So far, I’m mostly tracing the kolide product, so it’s not a totally clean setup. But, what I see with strace:
1. A bunch of initial diskio from sqlite as tables are created
2. After that (during quintessence) socket and network writes.
Reading the internet again this morning — socket io is fileio but not inherently diskio
t
theopolis
12/03/2019, 2:04 PMI am going off of my understanding, but sockets are implemented in memory, so you can create pressure that could lead to diskio, e.g., swapping, but that should be exceptional. Another curiosity, how are you measuring diskio?
s
seph
12/03/2019, 2:08 PMYes, measurement is where I need to dig next. I don’t have good tooling there. Some observations in iostat, but I need to get iotop running some more.
Either we don’t understand how tools report socketio, there’s something I’m not seeing in strace; or the end user is making some other error
t
timb
12/03/2019, 8:30 PMJust to confirm, this is only an issue for MacOS, no?
s
seph
12/03/2019, 10:11 PMIt’s not clear there is any issue here at all. Vs some confusion around how metrics are recorded. Currently, I’m leaning towards confusion.
But, this is unrelated top platforms, and I am primarily debugging this on linux
Digging a lot more, I have a new theory.
Some tools (
iotopHowever, taskset isn’t looking at diskio, it’s actually summing all read/write calls. So this includes any kind of io. The man page for
procrchar: characters read
The number of bytes which this task has caused to be
read from storage. This is simply the sum of bytes
which this process passed to read(2) and similar system
calls. It includes things such as terminal I/O and is
unaffected by whether or not actual physical disk I/O
was required (the read might have been satisfied from
pagecache).
So I think the extension traffic is yes io, but not disk io. And it’s simply misreported by common too.s
strace remains king here.
This conclusion brought to you by:
• docker
• testing
• reading the kernel source in places
• Reading docs
I have not yet written a test program to hammer on sockets and confirm it. strace seems clear enough
👍 1
t
theopolis
12/04/2019, 4:02 AMThis is sort of what I expected, a monitoring tool is confusing fileio as diskio (I can empathize!!)
s
seph
12/04/2019, 4:08 AMYes, it is what strace hinted at, in the beginning.
But I did not expect the monitoring tool to be the linux kernel.
t
theopolis
12/04/2019, 5:23 AMsomething... something... spanish inquisition