grahamgilbert
10/01/2022, 1:18 AMunified_log
tabletimestamp
clause we use. How can we get all of the results out of this table? Or is the recommendation to run it every 60 or 30 seconds or something?osquery> select count(*) from unified_log where subsystem="com.apple.SoftwareUpdate" AND timestamp > (SELECT unix_time from time) - 3600;
count(*) = 100
seph
10/01/2022, 12:34 PMmax_rows
rows column is how many it’ll fetch from the underlying API-1
, and it will behave like it’s evented, and a suitable max_rows and query frequency.max_rows
can be depends a lot on your specific installation environmentgrahamgilbert
10/01/2022, 5:37 PMseph
10/03/2022, 12:00 PMunified_log
is unique as a table.
If you query it normally, it searches the log. The log is very large, thus there is a max_rows parameter. Otherwise an exploratory select * from unified_log
would overwhelm something.
If you include timestamp = -1
it behaves like a log follower. It tracks the last returned timestamp, and appends that. This feature was contributed to allow someone to pull the whole log. It’s a bit like it’s evented, but implemented very differently. (It’s somewhat beta, it may not work quite right, and it may change, etc)
As for what you should do… I don’t know. I imagine up the max_rows to as high as your pipeline can handle, and set timestamp to -1. Then fetch on a suitable interval.Brandon Kurtz
10/03/2022, 5:44 PMIf you includeWhat do you mean when you say "the last returned timestamp"? we have to run the query multiple times in a row in order to get more and more (hence appened) results?it behaves like a log follower. It tracks the last returned timestamp, and appends that.timestamp = -1
select * from unified_log WHERE process="kernel" AND timestamp=-1 AND max_rows=1000;
Console shows results from this as of 10s ago but not seeing it in the tablezwass
10/04/2022, 1:10 AMtimestamp > -1
. If you schedule that query with a sufficiently high max_rows
you should eventually get the entire log and then be essentially "streaming" new results as they come in.max_rows
is too small then the log would grow faster than you were collecting it)Daniel Bretón Suárez
10/04/2022, 7:02 AMtimestamp > -1
, the table uses a mode that saves the last log entry it has returned, so the next time you query the table, it will return logs starting from the entry after the last it has return.
Imagine we have entries like:
| ***last (0)*** | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 |
select * from unified_log where timestamp > -1 and max_rows = 10
It returns the entries ***1st to 10th***
| 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 ***last*** | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 |
And again, select * from unified_log where timestamp > -1 and max_rows = 10
It returns the entries ***11th to 20th***
| 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 ***last*** | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31 | 32 | 33 |
If the OS deletes entries (we can't control that), you will still be getting sequential logs, but you could lose some.
| ***last (20)*** | 45 | 46 | 47 | 48 | 49 | 50 | 51 | 52 | 53 | 54 | 55 | 56 | 57 | 58 | 59 | 60 | 61 | 62 | 63 | 64 |
My advice is to set max_rows so that eventually the number of rows returned is less than max_rows.
I mean, if max_rows = 100 and you're always getting 100 entries, it means there were more entries in the buffer. If max_rows = 500, and eventually you get i.e. 450 entries, you know you have "emptied" the buffer.Brandon Kurtz
10/04/2022, 6:15 PMzwass
10/04/2022, 10:27 PMtimestamp
column. If you want the entire log, you would have to record it in an external system.