Title
#general
r

Robin Powell

09/23/2021, 10:15 PM
I'm trying very hard to understand https://github.com/osquery/osquery/pull/6982 and what, exactly, it breaks, and I'm having some trouble.
10:16 PM
I think that in <5.0,
select * from augeas where path LIKE '/etc/hosts%';
returns a bunch of stuff but in >=5.0 it's going to return nothing.
10:17 PM
I can't see anything else that looks breaking-ish. Anyone have any clues to dole out? 🙂
10:21 PM
I am, in particular, deeply confused by this part:
10:21 PM
ASSERT_EQ(
       SQL("select * from augeas where path LIKE '/etc/hosts/%'").rows().size(),
       0U);
10:22 PM
AFAICT, that should get converted to
match /files/etc/hosts/*
, which totally works and does not return zero results.
s

seph

09/23/2021, 11:25 PM
It's always possible that I introduced a bug. I'll try to look harder sometime in the next couple days.
r

Robin Powell

09/23/2021, 11:43 PM
Oh hello PR author. 😄
11:45 PM
I don't actually even know if our (internal) customers use LIKEs with augeas; I was just trying to understand the change for producing a report on 5.0 in general.
s

seph

09/24/2021, 1:26 AM
Hi!
1:26 AM
Okay, I’m trying to remember all the things.
1:28 AM
The old version of this table basically ran a
match /files/*
and let sqlite filter the output. This presented two issues. The bigger issue was that there was no way to access things outside the files tree. (eg,
/augeas
). Second, and smaller, was that the “return everything” approach just feels wrong to me. I think practically speaking the performance was okay, but it feels like there are dragons about. I’d usually rather call underlying APIs narrowly.
1:31 AM
Wildcard handling between the two was problematic. And TBH, I’m not sure I got it right.
1:32 AM
In augeas, wildcards are tree based.
*
is a single level, and
//*
is recursive. But in sql, wildcards are simple strings
1:32 AM
And there’s some magic in how
/
is treated.
1:37 AM
The end results is, I think: •
/etc/hosts/%
is converted to `/files/etc/hosts/%’. So augeas returns data. But sql filters it. (because the augeas return is is missing that trailing slash) •
/etc/host%
is converted to
/files/etc/hosts*
which augeas has no matches for, because it’s a weird postfix search. •
/etc/host%%
is converted to
/files/etc/hosts/*
which is a full recursion return, and it will get passed back through the sql filter
1:38 AM
For
path
I don’t think it’s very meaningful to wildcard a file. Wildcarding a directory is more meaningful. Compare
select * from augeas where path LIKE '/etc/%';
and
select * from augeas where path LIKE '/etc/%%';
1:38 AM
Does that help any?
r

Robin Powell

09/24/2021, 3:49 PM
It does, thank you!
3:50 PM
Some questions still though. 😄
3:50 PM
/etc/hosts/%
 is converted to `/files/etc/hosts/%’.
^^ Why doesn't that get converted to
/filles/etc/hosts/*
? Like, not "why did you make that decision?" but "where in the code does that happen?".
3:50 PM
(because the augeas return is is missing that trailing slash)
^^ I didn't follow that part at alll.
3:54 PM
I still don't feel like I have a handle on what's breaking. I understand the changes and why you made them and I think they make sense, but I can't come up with any examples of queries that work in 4.9.0 that'll break in 5.0
3:55 PM
Oh, actually, in the PR thread there's:
This seems reasonable but we should mark it as an API change due to change with queries like 
select * from augeas where path LIKE '/etc/hosts%';
, where before this would full-scan and have SQL apply the 
LIKE
 filtering.
; do I correctly understand that that query currrently returns stuff (which I just checked) but it won't in 5.0 because it gets converted to
match /files/etc/hosts*
?
3:56 PM
If that's the only breaking example we have, that seems like it's no going to come up very much. 🙂 Which is yay.
s

seph

09/24/2021, 4:30 PM
/etc/hosts/% is converted to `/files/etc/hosts/%’.
^^ Why doesn’t that get converted to /filles/etc/hosts/*?  Like, not “why did you make that decision?” but “where in the code does that happen?”
Typo, I mean to
/files/etc/hosts/*
And all the conversion is in
patternsFromOsquery
https://github.com/osquery/osquery/blob/master/osquery/tables/system/posix/augeas.cpp#L156
4:32 PM
This is a specific case of the the only breaking example I know. Namely, I introduced
%
and
%%
akin to the existing file pattern as single wild card, vs recursive. Thus breaking a couple of places
4:33 PM
So if you did
path like '/etc/hosts/%
, it would map to
/files/etc/hosts/*'
might work. (don’t remember). But the data that the table implementation returns would be
path: /etc/hosts
which won’t match the sql expression.