Title
#general
Mystery Incorporated

Mystery Incorporated

09/10/2021, 9:07 AM
Hello please see here because the query for python packages is returning different to the python packages installed on the device. https://osquery.slack.com/archives/C01DXJL16D8/p1630781846091900
s

seph

09/10/2021, 1:10 PM
Debugging via screenshot can be hard. I have to squint much harder than cut and pasted text in code blocks.
1:10 PM
But, it’s not clear to me what any of that is showing.
1:13 PM
the python table enumerate the python packages at various directories. You can probably tell it what directory to enumerate to.
1:14 PM
Is your pip command using the same directory?
1:14 PM
Who knows. It’s not in your screen shots.
Mystery Incorporated

Mystery Incorporated

09/11/2021, 4:27 AM
I was just doing
pip list
4:27 AM
I was not aware of any directories or anything I just use
pip install x
4:28 AM
i think
pip list
is meant to list all packages installed by pip isn't it?
s

seph

09/12/2021, 4:41 PM
Which pip. You could have many.
Mystery Incorporated

Mystery Incorporated

09/17/2021, 3:12 PM
@seph well irrespective of which pip,
SELECT name AS name, version AS version, 'Package (Python)' AS type, 'python_packages' AS source FROM python_packages
Should list ALL python packages installed on the system right? Even if I had 1 million pips, it's not. I can see for sure that cryptography 3.4.8 is installed, yet that query only lists 2.8 so it is totally broken listing phantom packages that no longer exist and not listing the ones that updated them.
3:13 PM
This is causing the vulnerability checker in fleet to fail, and fleet says it is osquery being the problem due to the erroneous output from that query.
3:15 PM
Here is fleet
3:15 PM
Yet the query above only returns v2.8 (which it should not as it doesn't exist)
s

seph

09/17/2021, 3:17 PM
Answering the question about a million pips… I think this is a philosophical question. osquery examines common locations for python packages. These are enumerated by https://github.com/osquery/osquery/blob/482a751f9f55ad31333ecf75b645a2334d943774/osquery/tables/system/python_packages.cpp#L34-L48 It does not search your entire disk looking for likely python packages. It will never find venvs built into random places, or similar. I tend to think osquery’s approach is correct. But it is not always what’s expected.
3:17 PM
I have no idea what fleet is doing, what you’re doing, or any of the larger context needed to debug that. Fleet is, ultimately, just running queries. So if fleet can find it, you should be able to find it otherwise.
3:23 PM
That code snippet is probably incomplete — it looks like other parts also search additional places.
Mystery Incorporated

Mystery Incorporated

09/17/2021, 3:28 PM
Ok I see the issue, that cryptography v2.8 is in /usr/lib/python3/dist-packages. But cryptography v3.4.8 is in /usr/local/lib/python3.8/dist-packages/ But osquery is only reporting v2.8 but according to the code you just pointed me too it should be reporting from both paths?
Mystery Incorporated

Mystery Incorporated

09/17/2021, 3:31 PM
Yep v3.4.8 is in there, but the query is not returning it
s

seph

09/17/2021, 3:32 PM
Are you using osqueryi? Or distributing this via fleet? If the latter, can you repreoduce on the local machine?
Mystery Incorporated

Mystery Incorporated

09/17/2021, 3:32 PM
I am just running the query as an on demand query from fleet
s

seph

09/17/2021, 3:33 PM
How confident are you that fleet isn’t doing any weird caching here? Can you preproduce this on the local machine?
3:34 PM
(If you have osqueryd, you can invoke it interactively with the
-S
option)
Mystery Incorporated

Mystery Incorporated

09/17/2021, 3:35 PM
Output from osqueryi on device only lists v2.8
s

seph

09/17/2021, 3:35 PM
And that’s running as root, correct?
Mystery Incorporated

Mystery Incorporated

09/17/2021, 3:37 PM
I just ran osqueryi with a sudo and same result
3:38 PM
oh hang on it's not in alphabetical order
3:38 PM
there is another list below that list
3:38 PM
so the v3.4.8 is further down the list
s

seph

09/17/2021, 3:39 PM
In retrospect, that’s not surprising. osquery searches the directories and appends. If you want to sort it, add an ORDER BY to the sql.
Mystery Incorporated

Mystery Incorporated

09/17/2021, 3:39 PM
yea let me see if it's same thing from fleet
3:41 PM
yea same in fleet, ok cool thanks