Hey I updated my fleet and osquery recently but no...
# fleet
o
Hey I updated my fleet and osquery recently but now: I am not able to find the osquery logs in /var/osquery/* not in C:\Program Files\osquery\*
đź‘€ 1
there seems to be a new folder orbit. Inside this i did find osquery.db but the logs are not the sme way it used to be. its very badly formatted and just in one single file. Most of them just say what fleet tried to run on the machine and no results are logged
mac logs
ubuntu
Windows
All i need now is someone to show me how to configure the logs to be stored on the host itself. This is my config rn and it dosent seem to do the trick
l
Hi Ojas, yes you can do this using the filesystem as the logger plugin: https://fleetdm.com/docs/using-fleet/osquery-logs#filesystem
l
Hi Ojas!
logger_plugin
is for osquery
result
(scheduled queries results) and osquery
status
logs. If you are interested on osquery logs in general, you can check the following paths: https://github.com/fleetdm/fleet/tree/main/orbit#logs (Both Orbit and osquery logs should be in those locations.)
o
@Luke Heath Thanks for you response. So i have configured correctly as given in above screenshot? Coz even after configuring this i dont see any proper logs on my hosts. I do see a file which contains data of what agent is connecting to and what query ran on the machine and everything. But not he proper logs like which contains the results of the query which ran on the machine.
@Lucas Rodriguez thanks for you response. I have configured the logger_plugin but still no logs on the endpoint. The orbit logs are not as proper as the older osquery logs which were is /var/log/osquery/* In the orbit logs file we have everything logged like what agent is doing what query ran and not in a standard format. any way i can configure it to goto old format
I have this err on mac os: E0304 054753.505184 129021440 shutdown.cpp:75] Cannot activate filesystem logger plugin: Could not create file: /var/log/osquery/osqueryd.results.log
If i create the folder osquery there it starts working fine but when the folder is not there it throws err that cannot create it. Any fix for this? as i have to install it on alot of systems, manually creating those folders would be painfull
Same err on windows: Cannot activate filesystem logger plugin: Could not create file: \Program Files\osquery\
l
Hi @Ojas! I believe you hit this bug: https://github.com/fleetdm/fleet/issues/4146 (Which will be fixed on our next release coming today/early-next-week, TBD)
You will have to remove
filesystem
from
logger_plugin
(and just leave
tls
or empty until the new release is out).
Though, once we release the new Orbit version, your instances should auto-update.
o
alright. Thanks Lucas. I’ll wait for the update.
So after the update do i need to again generate the installer agent or do i need to install osquery on machine again to get updated config? or will that be auto-updated too
l
Hi Ojas!, once we release fleet-osquery (aka orbit) - sometime early this week - it should auto-update automatically. (If things work as expected you won't need to re-generate the installers.)
o
awesome thanks
Hey @Lucas Rodriguez I still dont see any logs created. Any update on patch? Do i need to generate a new installer?
Also now i see another older issue, my fleet_osquery service in windows keeps stopping.
l
Hi @Ojas!
Do i need to generate a new installer?
No, should auto-update.
Also now i see another older issue, my fleet_osquery service in windows keeps stopping.
Could you check
C:\Windows\system32\config\systemprofile\AppData\Local\FleetDM\Orbit\Logs\orbit-osquery.log
? (You may be hitting a known issue we are trying to fix for next release.)
o
it says ” Cannot activate filesystem logger plugin” :?
@User i still see the old error: 2022-03-30T111942Z INF start osqueryd cmd=“C:\\Program Files\\Orbit\\bin\\osqueryd\\windows\\stable\\osqueryd.exe --pidfile=C:\\Program Files\\Orbit\\osquery.pid --database_path=C:\\Program Files\\Orbit\\osquery.db --extensions_socket=\\\\.\\pipe\\orbit-osquery-extension --enroll_secret_env ENROLL_SECRET --host_identifier=uuid --tls_hostname=fleet.tpsec.co --enroll_tls_endpoint=/api/v1/osquery/enroll --config_plugin=tls --config_tls_endpoint=/api/v1/osquery/config --config_refresh=60 --disable_distributed=false --distributed_plugin=tls --distributed_tls_max_attempts=10 --distributed_tls_read_endpoint=/api/v1/osquery/distributed/read --distributed_tls_write_endpoint=/api/v1/osquery/distributed/write --logger_plugin=tls --logger_tls_endpoint=/api/v1/osquery/log --disable_carver=false --carver_disable_function=false --carver_start_endpoint=/api/v1/osquery/carve/begin --carver_continue_endpoint=/api/v1/osquery/carve/block --carver_block_size=2000000 --tls_server_certs C:\\Program Files\\Orbit\\certs.pem --force --flagfile C:\\Program Files\\Orbit\\osquery.flags” E0330 112240.311066 3380 shutdown.cpp:79] Cannot activate filesystem logger plugin: Could not create file: \Program Files\osquery\log\osqueryd.results.log E0330 112242.511026 5160 shutdown.cpp:79] Worker returned exit status
Also on manually creating folders it works fine. It’s still the issue of not able to create the folders
l
OK, could you run
"C:\Program Files\Orbit\bin\orbit\orbit.exe" --version
?
Could not create file: \Program Files\osquery\log\osqueryd.results.log
On the latest version we changed the path, that looks like the old default path.
o
orbit 0.0.6
l
OK, latest is 0.0.7 (and soon 0.0.8). For some reason it's not auto-updating.
Does the host have access to https://tuf.fleetctl.com?
o
how do i check that?
l
Any other network error logs related to updating (in
C:\Windows\system32\config\systemprofile\AppData\Local\FleetDM\Orbit\Logs\orbit-osquery.log
)?
how do i check that?
Try visiting the URL from a browser in the host, or using the
curl
or
wget
commands (if available).
o
i can ping it from the host
2022-03-30T111942Z INF start osqueryd cmd=“C:\\Program Files\\Orbit\\bin\\osqueryd\\windows\\stable\\osqueryd.exe --pidfile=C:\\Program Files\\Orbit\\osquery.pid --database_path=C:\\Program Files\\Orbit\\osquery.db --extensions_socket=\\\\.\\pipe\\orbit-osquery-extension --enroll_secret_env ENROLL_SECRET --host_identifier=uuid --tls_hostname=fleet.tpsec.co --enroll_tls_endpoint=/api/v1/osquery/enroll --config_plugin=tls --config_tls_endpoint=/api/v1/osquery/config --config_refresh=60 --disable_distributed=false --distributed_plugin=tls --distributed_tls_max_attempts=10 --distributed_tls_read_endpoint=/api/v1/osquery/distributed/read --distributed_tls_write_endpoint=/api/v1/osquery/distributed/write --logger_plugin=tls --logger_tls_endpoint=/api/v1/osquery/log --disable_carver=false --carver_disable_function=false --carver_start_endpoint=/api/v1/osquery/carve/begin --carver_continue_endpoint=/api/v1/osquery/carve/block --carver_block_size=2000000 --tls_server_certs C:\\Program Files\\Orbit\\certs.pem --force --flagfile C:\\Program Files\\Orbit\\osquery.flags” E0330 111944.483722 4888 shutdown.cpp:79] Cannot activate filesystem logger plugin: Could not create file: \Program Files\osquery\log\osqueryd.results.log E0330 111945.709700 5776 shutdown.cpp:79] Worker returned exit status 2022-03-30T111945Z ERR unexpected exit error=“osqueryd exited with error: exit status 78" 2022-03-30T111947Z INF start osqueryd cmd=“C:\\Program Files\\Orbit\\bin\\osqueryd\\windows\\stable\\osqueryd.exe --pidfile=C:\\Program Files\\Orbit\\osquery.pid --database_path=C:\\Program Files\\Orbit\\osquery.db --extensions_socket=\\\\.\\pipe\\orbit-osquery-extension --enroll_secret_env ENROLL_SECRET --host_identifier=uuid --tls_hostname=fleet.tpsec.co --enroll_tls_endpoint=/api/v1/osquery/enroll --config_plugin=tls --config_tls_endpoint=/api/v1/osquery/config --config_refresh=60 --disable_distributed=false --distributed_plugin=tls --distributed_tls_max_attempts=10 --distributed_tls_read_endpoint=/api/v1/osquery/distributed/read --distributed_tls_write_endpoint=/api/v1/osquery/distributed/write --logger_plugin=tls --logger_tls_endpoint=/api/v1/osquery/log --disable_carver=false --carver_disable_function=false --carver_start_endpoint=/api/v1/osquery/carve/begin --carver_continue_endpoint=/api/v1/osquery/carve/block --carver_block_size=2000000 --tls_server_certs C:\\Program Files\\Orbit\\certs.pem --force --flagfile C:\\Program Files\\Orbit\\osquery.flags” E0330 111948.367669 5808 shutdown.cpp:79] Cannot activate filesystem logger plugin: Could not create file: \Program Files\osquery\log\osqueryd.results.log E0330 111951.128486 3348 shutdown.cpp:79] Worker returned exit status 2022-03-30T112133Z INF start osqueryd cmd=“C:\\Program Files\\Orbit\\bin\\osqueryd\\windows\\stable\\osqueryd.exe --pidfile=C:\\Program Files\\Orbit\\osquery.pid --database_path=C:\\Program Files\\Orbit\\osquery.db --extensions_socket=\\\\.\\pipe\\orbit-osquery-extension --enroll_secret_env ENROLL_SECRET --host_identifier=uuid --tls_hostname=fleet.tpsec.co --enroll_tls_endpoint=/api/v1/osquery/enroll --config_plugin=tls --config_tls_endpoint=/api/v1/osquery/config --config_refresh=60 --disable_distributed=false --distributed_plugin=tls --distributed_tls_max_attempts=10 --distributed_tls_read_endpoint=/api/v1/osquery/distributed/read --distributed_tls_write_endpoint=/api/v1/osquery/distributed/write --logger_plugin=tls --logger_tls_endpoint=/api/v1/osquery/log --disable_carver=false --carver_disable_function=false --carver_start_endpoint=/api/v1/osquery/carve/begin --carver_continue_endpoint=/api/v1/osquery/carve/block --carver_block_size=2000000 --tls_server_certs C:\\Program Files\\Orbit\\certs.pem --force --flagfile C:\\Program Files\\Orbit\\osquery.flags” E0330 112134.335160 3884 shutdown.cpp:79] Cannot activate filesystem logger plugin: Could not create file: \Program Files\osquery\log\osqueryd.results.log E0330 112137.178841 3876 shutdown.cpp:79] Worker returned exit status 2022-03-30T112238Z INF start osqueryd cmd=“C:\\Program Files\\Orbit\\bin\\osqueryd\\windows\\stable\\osqueryd.exe --pidfile=C:\\Program Files\\Orbit\\osquery.pid --database_path=C:\\Program Files\\Orbit\\osquery.db --extensions_socket=\\\\.\\pipe\\orbit-osquery-extension --enroll_secret_env ENROLL_SECRET --host_identifier=uuid --tls_hostname=fleet.tpsec.co --enroll_tls_endpoint=/api/v1/osquery/enroll --config_plugin=tls --config_tls_endpoint=/api/v1/osquery/config --config_refresh=60 --disable_distributed=false --distributed_plugin=tls --distributed_tls_max_attempts=10 --distributed_tls_read_endpoint=/api/v1/osquery/distributed/read --distributed_tls_write_endpoint=/api/v1/osquery/distributed/write --logger_plugin=tls --logger_tls_endpoint=/api/v1/osquery/log --disable_carver=false --carver_disable_function=false --carver_start_endpoint=/api/v1/osquery/carve/begin --carver_continue_endpoint=/api/v1/osquery/carve/block --carver_block_size=2000000 --tls_server_certs C:\\Program Files\\Orbit\\certs.pem --force --flagfile C:\\Program Files\\Orbit\\osquery.flags” E0330 112240.311066 3380 shutdown.cpp:79] Cannot activate filesystem logger plugin: Could not create file: \Program Files\osquery\log\osqueryd.results.log E0330 112242.511026 5160 shutdown.cpp:79] Worker returned exit status 2022-03-30T112542Z INF start osqueryd cmd=“C:\\Program Files\\Orbit\\bin\\osqueryd\\windows\\stable\\osqueryd.exe --pidfile=C:\\Program Files\\Orbit\\osquery.pid --database_path=C:\\Program Files\\Orbit\\osquery.db --extensions_socket=\\\\.\\pipe\\orbit-osquery-extension --enroll_secret_env ENROLL_SECRET --host_identifier=uuid --tls_hostname=fleet.tpsec.co --enroll_tls_endpoint=/api/v1/osquery/enroll --config_plugin=tls --config_tls_endpoint=/api/v1/osquery/config --config_refresh=60 --disable_distributed=false --distributed_plugin=tls --distributed_tls_max_attempts=10 --distributed_tls_read_endpoint=/api/v1/osquery/distributed/read --distributed_tls_write_endpoint=/api/v1/osquery/distributed/write --logger_plugin=tls --logger_tls_endpoint=/api/v1/osquery/log --disable_carver=false --carver_disable_function=false --carver_start_endpoint=/api/v1/osquery/carve/begin --carver_continue_endpoint=/api/v1/osquery/carve/block --carver_block_size=2000000 --tls_server_certs C:\\Program Files\\Orbit\\certs.pem --force --flagfile C:\\Program Files\\Orbit\\osquery.flags” I0330 112543.293851 3316 eventfactory.cpp:156] Event publisher not enabled: ntfs_event_publisher: NTFS event publisher disabled via configuration I0330 112546.206353 1712 interfaces.cpp:102] Failed to retrieve network statistics for interface 4 I0330 112546.266278 1712 interfaces.cpp:102] Failed to retrieve network statistics for interface 1 I0330 112546.269050 1712 interfaces.cpp:130] Failed to retrieve physical state for interface 1 I0330 112546.287117 1712 interfaces.cpp:157] Failed to retrieve DHCP and DNS information for interface 1 I0330 112546.377097 1712 registry.cpp:555] Failed to expand globs: Failed to open registry handle I0330 112546.389892 1712 registry.cpp:555] Failed to expand globs: Failed to open registry handle I0330 112546.390563 1712 registry.cpp:555] Failed to expand globs: Failed to open registry handle I0330 112546.392493 1712 registry.cpp:555] Failed to expand globs: Failed to open registry handle I0330 112546.395962 1712 registry.cpp:555] Failed to expand globs: Failed to open registry handle I0330 112546.396636 1712 registry.cpp:555] Failed to expand globs: Failed to open registry handle I0330 112546.398648 1712 registry.cpp:555] Failed to expand globs: Failed to open registry handle I0330 112546.400411 1712 registry.cpp:555] Failed to expand globs: Failed to open registry handle I0330 112546.401648 1712 registry.cpp:555] Failed to expand globs: Failed to open registry handle W0330 112546.763278 1712 chocolatey_packages.cpp:65] Did not find chocolatey path environment variable E0330 112548.131688 1712 distributed.cpp:144] Error executing distributed query: fleet_policy_query_1: no such table: gatekeeper W0330 112548.153442 1712 bitlocker_info.cpp:52] Error retreiving information from WMI. E0330 112548.159821 1712 distributed.cpp:144] Error executing distributed query: fleet_policy_query_3: no such table: disk_encryption E0330 112548.164297 1712 distributed.cpp:144] Error executing distributed query: fleet_policy_query_4: no such table: disk_encryption E0330 112548.168017 1712 distributed.cpp:144] Error executing distributed query: fleet_policy_query_5: no such table: sip_config E0330 112548.170768 1712 distributed.cpp:144] Error executing distributed query: fleet_policy_query_6: no such table: managed_policies E0330 112548.173740 1712 distributed.cpp:144] Error executing distributed query: fleet_policy_query_7: no such table: managed_policies E0330 112548.176471 1712 distributed.cpp:144] Error executing distributed query: fleet_policy_query_8: no such table: managed_policies E0330 112548.179143 1712 distributed.cpp:144] Error executing distributed query: fleet_policy_query_9: no such table: plist I0330 112612.553747 2764 registry.cpp:555] Failed to expand globs: Failed to open registry handle W0330 112612.555688 2764 virtual_table.cpp:961] The chrome_extensions table returns data based on the current user by default, consider JOINing against the users table W0330 112612.558739 2764 virtual_table.cpp:961] The firefox_addons table returns data based on the current user by default, consider JOINing against the users table W0330 112612.572578 2764 chocolatey_packages.cpp:65] Did not find chocolatey path environment variable W0330 112612.583083 2764 virtual_table.cpp:961] The atom_packages table returns data based on the current user by default, consider JOINing against the users table I0330 112612.585950 2764 query.cpp:102] Storing initial results for new scheduled query: pack_test_Get installed Windows software 2022-03-30T112909Z INF start osqueryd cmd=“C:\\Program Files\\Orbit\\bin\\osqueryd\\windows\\stable\\osqueryd.exe --pidfile=C:\\Program Files\\Orbit\\osquery.pid --database_path=C:\\Program Files\\Orbit\\osquery.db --extensions_socket=\\\\.\\pipe\\orbit-osquery-extension --enroll_secret_env ENROLL_SECRET --host_identifier=uuid --tls_hostname=fleet.tpsec.co --enroll_tls_endpoint=/api/v1/osquery/enroll --config_plugin=tls --config_tls_endpoint=/api/v1/osquery/config --config_refresh=60 --disable_distributed=false --distributed_plugin=tls --distributed_tls_max_attempts=10 --distributed_tls_read_endpoint=/api/v1/osquery/distributed/read --distributed_tls_write_endpoint=/api/v1/osquery/distributed/write --logger_plugin=tls --logger_tls_endpoint=/api/v1/osquery/log --disable_carver=false --carver_disable_function=false --carver_start_endpoint=/api/v1/osquery/carve/begin --carver_continue_endpoint=/api/v1/osquery/carve/block --carver_block_size=2000000 --tls_server_certs C:\\Program Files\\Orbit\\certs.pem --force --flagfile C:\\Program Files\\Orbit\\osquery.flags” I0330 112909.557931 3980 eventfactory.cpp:156] Event publisher not enabled: ntfs_event_publisher: NTFS event publisher disabled via configuration I0330 113244.828908 3940 registry.cpp:555] Failed to expand globs: Failed to open registry handle W0330 113244.830849 3940 virtual_table.cpp:961] The chrome_extensions table returns data based on the current user by default, consider JOINing against the users table W0330 113244.834173 3940 virtual_table.cpp:961] The firefox_addons table returns data based on the current user by default, consider JOINing against the users table W0330 113244.848613 3940 chocolatey_packages.cpp:65] Did not find chocolatey path environment variable W0330 113244.859524 3940 virtual_table.cpp:961] The atom_packages table returns data based on the current user by default, consider JOINing against the users table
Able to access the site from browser as well.
l
OK, a guess: so due to the issue (which causes orbit to crash), orbit cannot auto-update. Could you disable the
filesystem
logger configuration from Fleet temporarily? (to give the process a chance to auto-update)
o
Do i change it to tls? or just remove it from there all together?
l
Yes, just
tls
.
o
alright. Done config: options: logger_plugin: tls
l
What I fear is that osquery already stored
filesystem
in its internal rocksdb local storage..., but let's see if this helps.
o
should i restart the machine? or just wait few minutes?
l
Let's wait a few minutes first.
âś… 1
Worst case scenario: You will need to regenerate a
msi
installer and re-install Orbit.
o
for that would orbit need to be cleaned from the machine first? or as i run new installer it will auto install new one
l
Yes. Running a new installer should override current installation. Let me know if it doesn't.
o
yea with new installer it got updated to orbit 0.0.7
l
OK, let me know if the
filesystem
change is still not working.
o
Tried it on a new machine from scratch. Works nicely. thanks :) So everytime there is a patch/update do i need to create a new installer?
l
No, orbit should automatically update. The issue here is that the filesystem configuration was causing issues at startup, so it didn't have the chance to run the auto-updater routine...
🙏 1