Hey I updated my fleet and osquery recently but now I am not

Question

Hey I updated my fleet and osquery recently but now I am not able to find the osquery logs in var osquery not in C Program Files osquery

Ojas · Answer

there seems to be a new folder orbit Inside this i did find osquery db but the logs are not the sme way it used to be its very badly formatted and just in one single file Most of them just say what fleet tried to run on the machine and no results are logged

Ojas · Answer

mac logs

Ojas · Answer

ubuntu

Ojas · Answer

Windows

Ojas · Answer

All i need now is someone to show me how to configure the logs to be stored on the host itself This is my config rn and it dosent seem to do the trick

Luke Heath · Answer

Hi Ojas yes you can do this using the filesystem as the logger plugin <https fleetdm com docs using fleet osquery logs filesystem>

Lucas Rodriguez · Answer

Hi Ojas `logger plugin` is for osquery `result` scheduled queries results and osquery `status` logs If you are interested on osquery logs in general you can check the following paths <https github com fleetdm fleet tree main orbit logs> Both Orbit and osquery logs should be in those locations

Ojas · Answer

< Luke Heath> Thanks for you response So i have configured correctly as given in above screenshot Coz even after configuring this i dont see any proper logs on my hosts I do see a file which contains data of what agent is connecting to and what query ran on the machine and everything But not he proper logs like which contains the results of the query which ran on the machine

Ojas · Answer

< Lucas Rodriguez> thanks for you response I have configured the logger plugin but still no logs on the endpoint The orbit logs are not as proper as the older osquery logs which were is var log osquery In the orbit logs file we have everything logged like what agent is doing what query ran and not in a standard format any way i can configure it to goto old format

Ojas · Answer

I have this err on mac os E0304 05 47 53 505184 129021440 shutdown cpp 75 Cannot activate filesystem logger plugin Could not create file var log osquery osqueryd results log

Ojas · Answer

If i create the folder osquery there it starts working fine but when the folder is not there it throws err that cannot create it Any fix for this as i have to install it on alot of systems manually creating those folders would be painfull

Ojas · Answer

Same err on windows Cannot activate filesystem logger plugin Could not create file Program Files osquery

Lucas Rodriguez · Answer

Hi < Ojas> I believe you hit this bug Which will be fixed on our next release coming today early next week TBD

Lucas Rodriguez · Answer

You will have to remove `filesystem` from `logger plugin` and just leave `tls` or empty until the new release is out

Lucas Rodriguez · Answer

Though once we release the new Orbit version your instances should auto update

Ojas · Answer

alright Thanks Lucas I ll wait for the update

Ojas · Answer

So after the update do i need to again generate the installer agent or do i need to install osquery on machine again to get updated config or will that be auto updated too

Lucas Rodriguez · Answer

Hi Ojas once we release fleet osquery aka orbit sometime early this week it should auto update automatically If things work as expected you won t need to re generate the installers

Ojas · Answer

awesome thanks

Ojas · Answer

Hey < Lucas Rodriguez> I still dont see any logs created Any update on patch Do i need to generate a new installer

Ojas · Answer

Also now i see another older issue my fleet osquery service in windows keeps stopping

Lucas Rodriguez · Answer

Hi < Ojas> gt Do i need to generate a new installer No should auto update gt Also now i see another older issue my fleet osquery service in windows keeps stopping Could you check `C Windows system32 config systemprofile AppData Local FleetDM Orbit Logs orbit osquery log` You may be hitting a known issue we are trying to fix for next release

Ojas · Answer

it says Cannot activate filesystem logger plugin

Ojas · Answer

< U02J4JCKDL2> i still see the old error 2022 03 30T11 19 42Z INF start osqueryd cmd= C Program Files Orbit bin osqueryd windows stable osqueryd exe pidfile=C Program Files Orbit osquery pid database path=C Program Files Orbit osquery db extensions socket= pipe orbit osquery extension enroll secret env ENROLL SECRET host identifier=uuid tls hostname= enroll tls endpoint= api v1 osquery enroll config plugin=tls config tls endpoint= api v1 osquery config config refresh=60 disable distributed=false distributed plugin=tls distributed tls max attempts=10 distributed tls read endpoint= api v1 osquery distributed read distributed tls write endpoint= api v1 osquery distributed write logger plugin=tls logger tls endpoint= api v1 osquery log disable carver=false carver disable function=false carver start endpoint= api v1 osquery carve begin carver continue endpoint= api v1 osquery carve block carver block size=2000000 tls server certs C Program Files Orbit certs pem force flagfile C Program Files Orbit osquery flags E0330 11 22 40 311066 3380 shutdown cpp 79 Cannot activate filesystem logger plugin Could not create file Program Files osquery log osqueryd results log E0330 11 22 42 511026 5160 shutdown cpp 79 Worker returned exit status

Ojas · Answer

Also on manually creating folders it works fine It s still the issue of not able to create the folders

Lucas Rodriguez · Answer

OK could you run ` C Program Files Orbit bin orbit orbit exe version` gt Could not create file Program Files osquery log osqueryd results log On the latest version we changed the path that looks like the old default path

Ojas · Answer

orbit 0 0 6

Lucas Rodriguez · Answer

OK latest is 0 0 7 and soon 0 0 8 For some reason it s not auto updating

Lucas Rodriguez · Answer

Does the host have access to <https tuf fleetctl com>

Ojas · Answer

how do i check that

Lucas Rodriguez · Answer

Any other network error logs related to updating in `C Windows system32 config systemprofile AppData Local FleetDM Orbit Logs orbit osquery log`

Lucas Rodriguez · Answer

gt how do i check that Try visiting the URL from a browser in the host or using the `curl` or `wget` commands if available

Ojas · Answer

i can ping it from the host

Ojas · Answer

2022 03 30T11 19 42Z INF start osqueryd cmd= C Program Files Orbit bin osqueryd windows stable osqueryd exe pidfile=C Program Files Orbit osquery pid database path=C Program Files Orbit osquery db extensions socket= pipe orbit osquery extension enroll secret env ENROLL SECRET host identifier=uuid tls hostname= enroll tls endpoint= api v1 osquery enroll config plugin=tls config tls endpoint= api v1 osquery config config refresh=60 disable distributed=false distributed plugin=tls distributed tls max attempts=10 distributed tls read endpoint= api v1 osquery distributed read distributed tls write endpoint= api v1 osquery distributed write logger plugin=tls logger tls endpoint= api v1 osquery log disable carver=false carver disable function=false carver start endpoint= api v1 osquery carve begin carver continue endpoint= api v1 osquery carve block carver block size=2000000 tls server certs C Program Files Orbit certs pem force flagfile C Program Files Orbit osquery flags E0330 11 19 44 483722 4888 shutdown cpp 79 Cannot activate filesystem logger plugin Could not create file Program Files osquery log osqueryd results log E0330 11 19 45 709700 5776 shutdown cpp 79 Worker returned exit status 2022 03 30T11 19 45Z ERR unexpected exit error= osqueryd exited with error exit status 78 2022 03 30T11 19 47Z INF start osqueryd cmd= C Program Files Orbit bin osqueryd windows stable osqueryd exe pidfile=C Program Files Orbit osquery pid database path=C Program Files Orbit osquery db extensions socket= pipe orbit osquery extension enroll secret env ENROLL SECRET host identifier=uuid tls hostname= enroll tls endpoint= api v1 osquery enroll config plugin=tls config tls endpoint= api v1 osquery config config refresh=60 disable distributed=false distributed plugin=tls distributed tls max attempts=10 distributed tls read endpoint= api v1 osquery distributed read distributed tls write endpoint= api v1 osquery distributed write logger plugin=tls logger tls endpoint= api v1 osquery log disable carver=false carver disable function=false carver start endpoint= api v1 osquery carve begin carver continue endpoint= api v1 osquery carve block carver block size=2000000 tls server certs C Program Files Orbit certs pem force flagfile C Program Files Orbit osquery flags E0330 11 19 48 367669 5808 shutdown cpp 79 Cannot activate filesystem logger plugin Could not create file Program Files osquery log osqueryd results log E0330 11 19 51 128486 3348 shutdown cpp 79 Worker returned exit status 2022 03 30T11 21 33Z INF start osqueryd cmd= C Program Files Orbit bin osqueryd windows stable osqueryd exe pidfile=C Program Files Orbit osquery pid database path=C Program Files Orbit osquery db extensions socket= pipe orbit osquery extension enroll secret env ENROLL SECRET host identifier=uuid tls hostname= enroll tls endpoint= api v1 osquery enroll config plugin=tls config tls endpoint= api v1 osquery config config refresh=60 disable distributed=false distributed plugin=tls distributed tls max attempts=10 distributed tls read endpoint= api v1 osquery distributed read distributed tls write endpoint= api v1 osquery distributed write logger plugin=tls logger tls endpoint= api v1 osquery log disable carver=false carver disable function=false carver start endpoint= api v1 osquery carve begin carver continue endpoint= api v1 osquery carve block carver block size=2000000 tls server certs C Program Files Orbit certs pem force flagfile C Program Files Orbit osquery flags E0330 11 21 34 335160 3884 shutdown cpp 79 Cannot activate filesystem logger plugin Could not create file Program Files osquery log osqueryd results log E0330 11 21 37 178841 3876 shutdown cpp 79 Worker returned exit status 2022 03 30T11 22 38Z INF start osqueryd cmd= C Program Files Orbit bin osqueryd windows stable osqueryd exe pidfile=C Program Files Orbit osquery pid database path=C Program Files Orbit osquery db extensions socket= pipe orbit osquery extension enroll secret env ENROLL SECRET host identifier=uuid tls hostname= enroll tls endpoint= api v1 osquery enroll config plugin=tls config tls endpoint= api v1 osquery config config refresh=60 disable distributed=false distributed plugin=tls distributed tls max attempts=10 distributed tls read endpoint= api v1 osquery distributed read distributed tls write endpoint= api v1 osquery distributed write logger plugin=tls logger tls endpoint= api v1 osquery log disable carver=false carver disable function=false carver start endpoint= api v1 osquery carve begin carver continue endpoint= api v1 osquery carve block carver block size=2000000 tls server certs C Program Files Orbit certs pem force flagfile C Program Files Orbit osquery flags E0330 11 22 40 311066 3380 shutdown cpp 79 Cannot activate filesystem logger plugin Could not create file Program Files osquery log osqueryd results log E0330 11 22 42 511026 5160 shutdown cpp 79 Worker returned exit status 2022 03 30T11 25 42Z INF start osqueryd cmd= C Program Files Orbit bin osqueryd windows stable osqueryd exe pidfile=C Program Files Orbit osquery pid database path=C Program Files Orbit osquery db extensions socket= pipe orbit osquery extension enroll secret env ENROLL SECRET host identifier=uuid tls hostname= enroll tls endpoint= api v1 osquery enroll config plugin=tls config tls endpoint= api v1 osquery config config refresh=60 disable distributed=false distributed plugin=tls distributed tls max attempts=10 distributed tls read endpoint= api v1 osquery distributed read distributed tls write endpoint= api v1 osquery distributed write logger plugin=tls logger tls endpoint= api v1 osquery log disable carver=false carver disable function=false carver start endpoint= api v1 osquery carve begin carver continue endpoint= api v1 osquery carve block carver block size=2000000 tls server certs C Program Files Orbit certs pem force flagfile C Program Files Orbit osquery flags I0330 11 25 43 293851 3316 eventfactory cpp 156 Event publisher not enabled ntfs event publisher NTFS event publisher disabled via configuration I0330 11 25 46 206353 1712 interfaces cpp 102 Failed to retrieve network statistics for interface 4 I0330 11 25 46 266278 1712 interfaces cpp 102 Failed to retrieve network statistics for interface 1 I0330 11 25 46 269050 1712 interfaces cpp 130 Failed to retrieve physical state for interface 1 I0330 11 25 46 287117 1712 interfaces cpp 157 Failed to retrieve DHCP and DNS information for interface 1 I0330 11 25 46 377097 1712 registry cpp 555 Failed to expand globs Failed to open registry handle I0330 11 25 46 389892 1712 registry cpp 555 Failed to expand globs Failed to open registry handle I0330 11 25 46 390563 1712 registry cpp 555 Failed to expand globs Failed to open registry handle I0330 11 25 46 392493 1712 registry cpp 555 Failed to expand globs Failed to open registry handle I0330 11 25 46 395962 1712 registry cpp 555 Failed to expand globs Failed to open registry handle I0330 11 25 46 396636 1712 registry cpp 555 Failed to expand globs Failed to open registry handle I0330 11 25 46 398648 1712 registry cpp 555 Failed to expand globs Failed to open registry handle I0330 11 25 46 400411 1712 registry cpp 555 Failed to expand globs Failed to open registry handle I0330 11 25 46 401648 1712 registry cpp 555 Failed to expand globs Failed to open registry handle W0330 11 25 46 763278 1712 chocolatey packages cpp 65 Did not find chocolatey path environment variable E0330 11 25 48 131688 1712 distributed cpp 144 Error executing distributed query fleet policy query 1 no such table gatekeeper W0330 11 25 48 153442 1712 bitlocker info cpp 52 Error retreiving information from WMI E0330 11 25 48 159821 1712 distributed cpp 144 Error executing distributed query fleet policy query 3 no such table disk encryption E0330 11 25 48 164297 1712 distributed cpp 144 Error executing distributed query fleet policy query 4 no such table disk encryption E0330 11 25 48 168017 1712 distributed cpp 144 Error executing distributed query fleet policy query 5 no such table sip config E0330 11 25 48 170768 1712 distributed cpp 144 Error executing distributed query fleet policy query 6 no such table managed policies E0330 11 25 48 173740 1712 distributed cpp 144 Error executing distributed query fleet policy query 7 no such table managed policies E0330 11 25 48 176471 1712 distributed cpp 144 Error executing distributed query fleet policy query 8 no such table managed policies E0330 11 25 48 179143 1712 distributed cpp 144 Error executing distributed query fleet policy query 9 no such table plist I0330 11 26 12 553747 2764 registry cpp 555 Failed to expand globs Failed to open registry handle W0330 11 26 12 555688 2764 virtual table cpp 961 The chrome extensions table returns data based on the current user by default consider JOINing against the users table W0330 11 26 12 558739 2764 virtual table cpp 961 The firefox addons table returns data based on the current user by default consider JOINing against the users table W0330 11 26 12 572578 2764 chocolatey packages cpp 65 Did not find chocolatey path environment variable W0330 11 26 12 583083 2764 virtual table cpp 961 The atom packages table returns data based on the current user by default consider JOINing against the users table I0330 11 26 12 585950 2764 query cpp 102 Storing initial results for new scheduled query pack test Get installed Windows software 2022 03 30T11 29 09Z INF start osqueryd cmd= C Program Files Orbit bin osqueryd windows stable osqueryd exe pidfile=C Program Files Orbit osquery pid database path=C Program Files Orbit osquery db extensions socket= pipe orbit osquery extension enroll secret env ENROLL SECRET host identifier=uuid tls hostname= enroll tls endpoint= api v1 osquery enroll config plugin=tls config tls endpoint= api v1 osquery config config refresh=60 disable distributed=false distributed plugin=tls distributed tls max attempts=10 distributed tls read endpoint= api v1 osquery distributed read distributed tls write endpoint= api v1 osquery distributed write logger plugin=tls logger tls endpoint= api v1 osquery log disable carver=false carver disable function=false carver start endpoint= api v1 osquery carve begin carver continue endpoint= api v1 osquery carve block carver block size=2000000 tls server certs C Program Files Orbit certs pem force flagfile C Program Files Orbit osquery flags I0330 11 29 09 557931 3980 eventfactory cpp 156 Event publisher not enabled ntfs event publisher NTFS event publisher disabled via configuration I0330 11 32 44 828908 3940 registry cpp 555 Failed to expand globs Failed to open registry handle W0330 11 32 44 830849 3940 virtual table cpp 961 The chrome extensions table returns data based on the current user by default consider JOINing against the users table W0330 11 32 44 834173 3940 virtual table cpp 961 The firefox addons table returns data based on the current user by default consider JOINing against the users table W0330 11 32 44 848613 3940 chocolatey packages cpp 65 Did not find chocolatey path environment variable W0330 11 32 44 859524 3940 virtual table cpp 961 The atom packages table returns data based on the current user by default consider JOINing against the users table

Ojas · Answer

Able to access the site from browser as well

Lucas Rodriguez · Answer

OK a guess so due to the issue which causes orbit to crash orbit cannot auto update Could you disable the `filesystem` logger configuration from Fleet temporarily to give the process a chance to auto update

Ojas · Answer

Do i change it to tls or just remove it from there all together

Lucas Rodriguez · Answer

Yes just `tls`

Ojas · Answer

alright Done config options logger plugin tls

Lucas Rodriguez · Answer

What I fear is that osquery already stored `filesystem` in its internal rocksdb local storage but let s see if this helps

Ojas · Answer

should i restart the machine or just wait few minutes

Lucas Rodriguez · Answer

Let s wait a few minutes first

Lucas Rodriguez · Answer

Worst case scenario You will need to regenerate a `msi` installer and re install Orbit

Ojas · Answer

for that would orbit need to be cleaned from the machine first or as i run new installer it will auto install new one

Lucas Rodriguez · Answer

Yes Running a new installer should override current installation Let me know if it doesn t

Ojas · Answer

yea with new installer it got updated to orbit 0 0 7

Lucas Rodriguez · Answer

OK let me know if the `filesystem` change is still not working

Ojas · Answer

Tried it on a new machine from scratch Works nicely thanks So everytime there is a patch update do i need to create a new installer

Lucas Rodriguez · Answer

No orbit should automatically update The issue here is that the filesystem configuration was causing issues at startup so it didn t have the chance to run the auto updater routine