https://github.com/osquery/osquery logo
Join Slack
Powered by
Hi! Looking at the managed_policies, it seems that...
# macos
n
Hi! Looking at the managed_policies, it seems that when the values are not integers, strings or booleans, they are reported as empty strings, probably because of this. It would be interesting to report all the values, but I understand it is important to maintain backward compatibilities. What about introducing a “key” column that is a flattened path to the value (dot separated key or array index) ? Has something similar already been discussed ?
s
Hi there! This came up in office hours, and we talked a little about it
Thank you for bringing it up, and thank you Henry for raising it at ofice hours.
If I understand correctly, the basic issue here is that the existing 
managed_policies
table is assuming data is a simple key:value form, but in reality the data may be more complex. So you’re proposing bringing in some data flattening to make something closer to an EAV table.
Broadly speaking, this makes sense to me. (But I work for Kolide, and have written a lot of EAV tables) But, I think there are some caveats….
First, I think any kind of change here has the potential to break someone’s usage. I think that’s okay, and it’s worth doing, but I want to call it out.
Second, discussion suggested using 
/
instead of 
.
as the separator. That makes sense to me.
n
yes, because it is already used somewhere else
i found it after making the first suggestion
and adding an extra column should not break existing usages, except if someone is testing for the empty value…
thanks for looking into it!
s
Third, I’m not sure what column names make sense. 
key
feels wrong to me. I’d probably use 
fullkey
. Looking at 
plist
I don’t see a great example. Hrm, If I look further, I think your example is amiss
n
“subpath” ? I am not good at naming things
s
Hrm. Give me a moment to write something. This data is weird. (Not your fault)
For:
Copy code
<dict>
  […]
  <key>PayloadType</key>
  <string>com.apple.system-extension-policy</string>
  <key>AllowUserOverrides</key>
  <true/>
  <key>AllowedSystemExtensions</key>
  <dict>
    <key>EQHXZ8M8AV</key>
    <array>
      <string>com.google.santa.daemon</string>
    </array>
  </dict>
  <key>AllowedSystemExtensionTypes</key>
  <dict>
    <key>EQHXZ8M8AV</key>
    <array>
      <string>EndpointSecurityExtension</string>
    </array>
  </dict>
</dict>
I’d expect raw data to be something like:
Copy code
[
  {fullkey: AllowUserOverrides, value: 1 },
  {fullkey: AllowedSystemExtensionTypes/EQHXZ8M8AV/0, value: EndpointSecurityExtension}
  {fullkey: AllowedSystemExtensions/EQHXZ8M8AV/0, value: com.google.santa.daemon}
]
And then there’s this mess because the existing 
name
is really “top key”, which always feels weird and backwards to me. So I’m not sure subkey is great — my experience with the plist table is that it just doesn’t flatten correctly.
n
yes, I was trying to maintain a kind of backward compatibility
s
You could keep the existing 
name
as the top key for compatibility, and then add 
fullkey
, 
parent
, and 
key
. And kinda expect that people will either use the new columns, or the old legacy one
If it’s interesting to see, I took your example xml, and parsed it with 
kolide_xml
. The resultant table is:
Copy code
+--------------------------+-------------------+--------+-----------------------------------+-------+------------+
| fullkey                  | parent            | key    | value                             | query | path       |
+--------------------------+-------------------+--------+-----------------------------------+-------+------------+
| dict/key/0               | dict/key          | 0      | PayloadType                       | *     | /tmp/e.xml |
| dict/key/1               | dict/key          | 1      | AllowUserOverrides                | *     | /tmp/e.xml |
| dict/key/2               | dict/key          | 2      | AllowedSystemExtensions           | *     | /tmp/e.xml |
| dict/key/3               | dict/key          | 3      | AllowedSystemExtensionTypes       | *     | /tmp/e.xml |
| dict/string              | dict              | string | com.apple.system-extension-policy | *     | /tmp/e.xml |
| dict/true                | dict              | true   |                                   | *     | /tmp/e.xml |
| dict/dict/0/key          | dict/dict/0       | key    | EQHXZ8M8AV                        | *     | /tmp/e.xml |
| dict/dict/0/array/string | dict/dict/0/array | string | com.google.santa.daemon           | *     | /tmp/e.xml |
| dict/dict/1/array/string | dict/dict/1/array | string | EndpointSecurityExtension         | *     | /tmp/e.xml |
| dict/dict/1/key          | dict/dict/1       | key    | EQHXZ8M8AV                        | *     | /tmp/e.xml |
+--------------------------+-------------------+--------+-----------------------------------+-------+------------+
(Because it’s xml, the dict shows up a bunch)
Dropping the username and manual columns, I might suggest:
Copy code
+-----------------------------------+--------------------------------------+-----------------------------+------------------------------------------+----------------------------------------+--------------------+---------------------------+
| domain                            | uuid                                 | name                        | fullkey                                  | parentkey                              |   key              |   value                   |
+-----------------------------------+--------------------------------------+-----------------------------+------------------------------------------+----------------------------------------+--------------------+---------------------------+
| com.apple.system-extension-policy | 00000000-0000-0000-0000-000000000000 | AllowUserOverrides          | AllowUserOverrides                       |                                        | AllowUserOverrides | 1                         |
| com.apple.system-extension-policy | 00000000-0000-0000-0000-000000000000 | AllowedSystemExtensions     | AllowedSystemExtensions/EQHXZ8M8AV/0     | AllowedSystemExtensions/EQHXZ8M8AV     | 0                  | com.google.santa.daemon   |
| com.apple.system-extension-policy | 00000000-0000-0000-0000-000000000000 | AllowedSystemExtensionTypes | AllowedSystemExtensionTypes/EQHXZ8M8AV/0 | AllowedSystemExtensionTypes/EQHXZ8M8AV | 0                  | EndpointSecurityExtension |
+-----------------------------------+--------------------------------------+-----------------------------+------------------------------------------+----------------------------------------+--------------------+---------------------------+
n
You beat me to it!
s
(and if your question was about whether we’d accept a PR that does ^^^ — yes, we probably would. We might squabble over the column names)
I think the notable change from your suggestion, is that I have 
com.google.santa.daemon
as a string value in an array, and not as a key name with a 
1
value
n
I do not see that in my original example, I had only AllowUserOverrides with 1 as a value i think
so, be explicit about the extra columns, even if only adding a “path” to the value could be enough to reconstruct fullkey, key, …
s
My mistake — you have them as value as well. I don’t know why I was confused.
You really only need the path, but I wouldn’t use 
path
since that often referes to path on disk.
n
I understand, in the osquery context
s
But yeah, 
fullkey
is the same 
path
in a sense. 
parent
and 
key
are useful as shortcuts. It’s nicer to say 
select 1 where value = 'com.google.santa.daemon' and parentkey = 'AllowedSystemExtensions/EQHXZ8M8AV'
than to split or string match on fullkey
n
Yes, can be useful. Thanks for your time. I will see what I can do, and when.
It would be really nice to have that for our compliance checks.
s
I don’t have a good sense of the code backing this, and cannot scope the difficulty. (The Kolide tables are all in go, and we do this a lot)
n
I will have to look at the plist implementation. Maybe factor this out to use it in both tables, although I would quickly reach my C++ limits…
there is already some code to limit the recursion
s
I know there are bugs in the plist one
2 Views
Open in Slack
PreviousNext
Powered by