did you do an sst_dump on the database?

If you set disable_events=true , and restart osqueryd I would expect the IO stats to be smaller. I'm guessing it's windows_events writing lots to disk

I'll give it a go with the events disabled

I'll have to check that tomorrow, thanks for your help. I'll also try dumping the db and see what's actually going into it

Turns out its a combination of things (but mostly you were right about it being events). First the new process table causes some verbose error messages about trying to get cwd for system processes - which when verbose is on those messages ended up being written to the database. So when my join happened the process table ended being called 17 times (the number of ports open) leading to a bunch of entries in the db . Second the processes table now seems to cause a large number of calls to enumerate users local group membership - which in turn leads to lots of windows events - which get written to the database. I saw ~900 events appear after running the suspect query. For comparison when I run the same query with 3.3.2 (on the same machine) I only saw 8 entries...

Looks like in the changes to processes in experimental the constraint checking on pid was removed so now every time the processes table is called it runs for every process whereas previously it only collected data for specified pids (from the join)

You are right, the new processes table does not implement pid index, which is costly. It's not hard to add. https://github.com/osql/osql/pull/3

I was using aws_kinesis which the docs suggest would buffer and flush based on aws_kinesis_period. I didn't realise the buffering was done in the DB