https://github.com/osquery/osquery logo
Join Slack
Powered by
Is there a way to run all queries in a JSON pack f...
# general
t
Is there a way to run all queries in a JSON pack file via stdin or arbitrary path? I'm looking for the spiritual equivalent of 
echo query | osqueryi
today. I found https://github.com/osquery/osquery/pull/2093 which added 
--pack
, but it appears to only run ones that are already defined as part of your config. As a workaround I was considering dynamically generating a JSON config file and passing it in via 
--config_path
, but it seems pretty hacky & the sort of thing that someone probably found a better solution for.
I've been working on a packer program to generate JSON pack files from a series of arbitrary SQL statements, but would like to make sure that the resulting output is run through a validation stage.
As a temporary workaround, I was reading the JSON in myself and running the queries by hand, but I've discovered that osquery's JSON deserialization diverges a bit from others (Python, Go), particularly with how escaping is handled.
s
I’m not sure anything exists that you didn’t already find.
I think you could reasonable do the hacky thing and generate a config. But 🤷
t
Thanks for the confirmation! I was able to use it and conclude that Kolide's frontend ingester is the source of JSON deserialization divergence 😞 I'm going to work on open-sourcing this tool & generating a reduced test case to open a bug w/ support.
s
Which kolide front end ingester? (I’m heading off for the day, but that sounds odd)
t
Tools > Log Pipelines > Add Pack > Import JSON Pack It fails to parse JSON files which are acceptable to osquery & generated by Go's 
json.Marshal
function. For example, it trips up on queries which contain double quotes (escaped via \)
FWIW, I ended up reporting it via support, but the main problem appears to be how Kolide's web frontend handles importing packs that include queries with double quotes in them. Here's a reduced example pack:
Copy code
{
  "queries": {
    "waldo": {
      "query": "SELECT * FROM chrome_extensions WHERE name IN (\"Where's waldo?\", \"Google Docs Offline\");",
      "interval": "3600"
    }
  }
}
s
Reporting through support is 👍
Might be worth trying single quotes inside — SQL prefers single quotes, though osquery handles either. But it’s an easy way to handled nested quotes
Ah, I see you mentioned that
t
I tried, but wasn't able to figure out how to handle queries referencing fields with single quotes in them. I know there's got to be a way to escape them, but I failed. I'll look into it again 😞
s
Nesting quotes is such a hassle 😕
Reading the internet, and testing at an interactive prompt. Doubled quotes works. eg:
Copy code
osquery> select 'hello''s';
+------------+
| 'hello''s' |
+------------+
| hello's    |
+------------+
t
Hmm, that's true - I did do that elsewhere already for double quotes. Let me see if that passes the JSON importer. It does break the ability to cut and paste strings (like Chrome Extension extensions) into SQL, but there's only an apostrophe in about 1 out of 100 of those, so I can just deal with that.
s
I agree it’s imperfect, the double quote thing ought work. But I don’t know how quickly that will get tracked down and fixed.
Mitigation better than none.
t
Thanks so much @seph - this workaround appears to work fantastically! I'll work on adjusting our queries back to single quotes. Here's a sneak peak at what I've been up to: https://github.com/chainguard-dev/osquery-defense-kit
s
I’d say leave the support note open, since I think there’s a bug somewhere. But I’m glad you’re not blocked
Speaking of query packs… One of the uncomfortable truths about osquery, is that the packs that osquery ships are old enough to be at best pointless. We’ve talked about how to approach this over the years, but never really found a good answer. Our current plan is to stop shipping packs and move all of that over into the docs as examples of what one can do.
👍 1
It’s probably super out of date, but https://github.com/teoseller/osquery-attck is interesting if you haven’t seen it
t
I personally like that there is some sort of reference pool to pull from, but that could easily be moved into it's own repo (
osquery/packs
). Both as an idea of best practices, and as an idea of what data sources are relevant. didn't find the detection queries to be that useful, but the 
incident-response
gave me additional ideas I hadn't considerde.
s
Exactly — there’s a lot of value there as ideas, and possibility, and historic interest. But shipping in the official osquery packages is a stronger endorsement than they merit. (Though if you’re using Kolide packages, you don’t get them shipped anyhow)
t
I had seen that repo, and took some inspiration from it. The main difference in our approach is that the queries in 
detect/
are designed to be usable as a source of alerts. In fact, we use them that way. It does mean an insane amount of false-positive management though.
I don't think our current approach is the right one long-term, but it's at least proved useful for us over the last couple of months.
s
The thing I liked about the attck stuff, was that it used the common MITRE ATT&CK language. I always feel like there’s a couple axis of categorization. Or tags. Or something like that
👍 1
m
I read over on Stack Exchange that RFC7159 or maybe RFC8259 does not allow any control characters in JSON texts and requires they be escaped. Maybe these packs are not valid JSON? But they pass as valid in a JSON linter I tried...
19 Views
Open in Slack
PreviousNext
Powered by