Title
#general
s

sanjaykcse

10/08/2020, 8:55 AM
Getting SIGSEGV  for osquery daemon (osqueryd). It cores even there is no execution of pack/query from fleet. I have downloaded the debug symbols  also , but still not getting the address mapping to symbol file . Any pointers how to debug this Osquery  daemon crash ?

gdb -iex "set auto-load safe-path /lib" /tmp/osqueryd-4.4.0-1.x86_64.debug -c /var/cores/core.osqueryd

GNU gdb (GDB) 7.10 Copyright (C) 2015 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-openwrt-linux". Type "show configuration" for configuration details. For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/
. Find the GDB manual and other documentation resources online at: <
http://www.gnu.org/software/gdb/documentation/
. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from /tmp/osqueryd-4.4.0-1.x86_64.debug...done.
warning: core file may not match specified executable file. [New LWP 4862] [New LWP 4871] [New LWP 4870] [New LWP 4869] [New LWP 4868] [New LWP 4867] warning: Unexpected size of section `.reg-xstate/4862' in core file. warning: Could not load shared library symbols for linux-vdso.so.1. Do you need "set solib-search-path" or "set sysroot"? warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available. warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available. Core was generated by
osquery                         '.
Program terminated with signal SIGSEGV, Segmentation fault.

warning: Unexpected size of section
.reg-xstate/4862' in core file. #0 0x00007f43fac971db in ?? () [Current thread is 1 (LWP 4862)] (gdb) bt  #0 0x00007f43fac971db in ?? () #1 0x0e24f4832af29500 in ?? () #2 0x0e24f4832af29500 in ?? () #3 0x00007f43fd415a80 in ?? () #4 0x00007f43fd408318 in ?? () #5 0x00007f43fd43aa48 in ?? () #6 0x00007fff53d99700 in ?? () #7 0x00007fff53d997a0 in ?? () #8 0x00007f43fb44425a in ?? () #9 0x00007f4300000000 in ?? () #10 0x00007fff53d99730 in ?? () #11 0xffffffffffffffff in ?? () #12 0x00007f43fba1ff4d in ?? () #13 0x0000000000000000 in ?? () (gdb) info r rax      0xe24f4832af29500 1019208259891008768 rbx      0x7f43fd415a80 139929988455040 rcx      0x7f43f92c2b98 139929919957912 rdx      0x7f43fa28a7b7 139929936504759 rsi      0x0 0 rdi      0x7fff53d995a0 140734600156576 rbp      0x7fff53d99c90 0x7fff53d99c90 rsp      0x7fff53d99540 0x7fff53d99540 r8       0x7fff53d99560 140734600156512 r9       0x7f43f905dda0 139929917447584 r10      0x4e585f3531314758 5645366814972135256 r11      0x7f43f9097f40 139929917685568 r12      0x0 0 r13      0x7f43fd43aa48 139929988606536 r14      0x7f43fa28a7b7 139929936504759 r15      0x7fff53d995a0 140734600156576 rip      0x7f43fac971db 0x7f43fac971db eflags     0x10246 [ PF ZF IF RF ] cs       0x33 51 ss       0x2b 43 ds       0x0 0 es       0x0 0 fs       0x0 0 gs       0x0 0 (gdb) x/i 0x7f43fac971db => 0x7f43fac971db: Cannot access memory at address 0x7f43fac971db (gdb) x/i 0x7fff53d99540   0x7fff53d99540: add  %dl,-0xb7cd50e(%rbp) (gdb) x/i $rsp         0x7fff53d99540: add  %dl,-0xb7cd50e(%rbp) (gdb) x/10x $rsp 0x7fff53d99540: 0x2af29500 0x0e24f483 0x2af29500 0x0e24f483 0x7fff53d99550: 0xfd415a80 0x00007f43 0xfd408318 0x00007f43 0x7fff53d99560: 0xfd43aa48 0x00007f43 (gdb) x/10i $pc => 0x7f43fac971db: Cannot access memory at address 0x7f43fac971db (gdb) x/10x $bsp Value can't be converted to integer. (gdb) x/10x $rbp 0x7fff53d99c90: 0xfd43aa10 0x00007f43 0xfd43aa10 0x00007f43 0x7fff53d99ca0: 0x00000001 0x00000000 0x6261740a 0x0000656c 0x7fff53d99cb0: 0x00000000 0x00000000
s

seph

10/08/2020, 1:09 PM
A bit of a sidestep, but does osquery log anything? Running in verbose mode may get you an error. I’d also recommend trying the new 4.5.1, it added an exception catcher around all virtual table calls
s

sanjaykcse

10/08/2020, 4:47 PM
Thanks Seph . Will enable verbose mode and check the log.
2:46 PM
warning: Unexpected size of section `.reg-xstate/25287' in core file. #0 std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::__is_long (this=0x98) at /usr/local/osquery-toolchain/usr/bin/../include/c++/v1/string:1426 1426 /usr/local/osquery-toolchain/usr/bin/../include/c++/v1/string: No such file or directory. [Current thread is 1 (LWP 25287)] (gdb) bt #0 std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::__is_long (this=0x98) at /usr/local/osquery-toolchain/usr/bin/../include/c++/v1/string:1426 #1 std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >:😒ize (this=0x98) at /usr/local/osquery-toolchain/usr/bin/../include/c++/v1/string:953 #2 std::__1:😮perator+<char, std::__1::char_traits<char>, std::__1::allocator<char> > (__lhs=..., _rhs=95 '') at /usr/local/osquery-toolchain/usr/bin/../include/c++/v1/string:4097 #3 0x0000000001cdbd00 in std::__1::messages<wchar_t>& std::__1:😦anonymous namespace)::make&lt;std::__1::messages<wchar_t>, unsigned int>(unsigned int)::buf () #4 0xba0461c30f1cd400 in ?? () #5 0x00007fffd2a2c9b8 in ?? () #6 0x00007fffd2a2ca48 in ?? () #7 0x0000000000000000 in ?? () (gdb) info reg rax      0xba0461c30f1cd400 -5042798192348048384 rbx      0x7fffd2a2c9b8 140736727271864 rcx      0x0 0 rdx      0x5f 95 rsi      0x98 152 rdi      0x7fffd2a2c9b8 140736727271864 rbp      0x98 0x98 rsp      0x7fffd2a2c980 0x7fffd2a2c980 r8       0x7fffd2a2cc28 140736727272488 r9       0x7f0f1dd08da0 139702901444000 r10      0x6974696e69207265 7598814394211922533 r11      0x7f0f1dd42f40 139702901681984 r12      0x3393650 54081104 r13      0x0 0 r14      0x5f 95 r15      0x0 0 rip      0xba9c8d 0xba9c8d <std::__1:😮perator+<char, std::__1::char_traits<char>, std::__1::allocator<char> >(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, char)+47> eflags     0x10246 [ PF ZF IF RF ] cs       0x33 51 ss       0x2b 43 ds       0x0 0 es       0x0 0 fs       0x0 0 gs       0x0 0
2:47 PM
got better stack trace . Analysing it . Should I log a bug ??
s

seph

10/11/2020, 3:00 PM
@theopolis thoughts?
theopolis

theopolis

10/11/2020, 3:02 PM
@sanjaykcse is that with 4.5.1?
3:03 PM
did you try to run with
--verbose
? what logs were printed?
s

sanjaykcse

10/12/2020, 1:28 PM
Hi @theopolis this crash is from 4.4.0 . Today , I have updated osquery to 4.5.1. will update if get see any crash . From the logs it appears the fleet was down. {"enroll_secret":"OXZMOA9Wr6AZy7N95UJBxxxxxds7Z2CN","host_identifier":"191534d5-a9d0-4c69-bk85-fb3557c5edba","platform_type":"9","host_details":{"os_version":{"arch":"x86_64","major":"0","minor":"0","name":"Unknown","patch":"0","pid_with_namespace":"0","platform":"posix"},"osquery_info":{"build_distro":"centos7","build_platform":"1","config_hash":"","config_valid":"0","extensions":"inactive","instance_id":"b9d0f315-be38-4f29-9ce6-d394e7813951","pid":"2546","platform_mask":"9","start_time":"1602223587","uuid":"191534d5-a9d0-4c69-bf85-fb3557c5edba","version":"4.4.0","watcher":"2518"},"platform_info":{"address":"0xf000","date":"10/08/2018","extra":"","revision":"5.12","size":"5242880","vendor":"American Megatrends Inc.","version":"5.12 (Z131E1009 11:38:14.526799 2546 init.cpp:714] Cannot activate tls logger plugin: No node key, TLS logging disabled. I1009 11:38:15.003304 2538 watcher.cpp:585] osqueryd watcher (2518) executing worker (5519) I1009 11:38:15.024971 5519 init.cpp:340] osquery worker initialized [watcher=2518] I1009 11:38:15.025318 5519 dispatcher.cpp:77] Adding new service: WatcherWatcherRunner (0x2cdcac8) to thread: 139827282855680 (0x2cdcb70) in process 5519 I1009 11:38:15.025480 5519 rocksdb.cpp:131] Opening RocksDB handle: /tmp/osquery.db E1009 11:38:15.028797 5519 init.cpp:568] An error occured during extension manager startup: Extension socket directory missing: /var/osquery/osquery.em I1009 11:38:15.029001 5519 tls_enroll.cpp:69] TLSEnrollPlugin requesting a node enroll key from: https://ec2-52-25-115-238.us-west-2.compute.amazonaws.com:8080/api/v1/osquery/enroll I1009 11:38:15.029183 5519 system.cpp:294] Using host identifier: 191534d5-a9d0-4c69-bk85-fb3557c5edba
7:15 AM
Hi @theopolis I am seeing this crash 0n 4.5.2 also
7:15 AM
Core was generated by
osquery                         '.
Program terminated with signal SIGSEGV, Segmentation fault.

warning: Unexpected size of section
.reg-xstate/10711' in core file. #0 std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::__is_long (this=0x98)   at /opt/osquery-toolchain/usr/bin/../include/c++/v1/string:1426 1426 /opt/osquery-toolchain/usr/bin/../include/c++/v1/string: No such file or directory. [Current thread is 1 (LWP 10711)] (gdb) bt #0 std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >::__is_long (this=0x98)   at /opt/osquery-toolchain/usr/bin/../include/c++/v1/string:1426 #1 std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >:😒ize (this=0x98)   at /opt/osquery-toolchain/usr/bin/../include/c++/v1/string:953 #2 std::__1:😮perator+<char, std::__1::char_traits<char>, std::__1::allocator<char> > (__lhs=..., _rhs=95 '')   at /opt/osquery-toolchain/usr/bin/../include/c++/v1/string:4097 #3 0x00007f81304bbd90 in std::__1::messages<wchar_t>& std::__1:😦anonymous namespace)::make&lt;std::__1::messages<wchar_t>, unsigned int>(unsigned int)::buf () #4 0xe527c8e13c6f9600 in ?? () #5 0x00007fff87502998 in ?? () #6 0x00007fff87502a28 in ?? () #7 0x0000000000000000 in ?? () (gdb) info reg rax      0xe527c8e13c6f9600 -1934356645225196032 rbx      0x7fff87502998 140735463565720 rcx      0x0 0 rdx      0x5f 95 rsi      0x98 152 rdi      0x7fff87502998 140735463565720 rbp      0x98 0x98 rsp      0x7fff87502960 0x7fff87502960 r8       0x7fff87502c08 140735463566344 r9       0x7f812d6ccda0 140192789613984 r10      0x6974696e69207265 7598814394211922533 r11      0x7f812d706f40 140192789851968 r12      0x7f81319f6530 140192860038448 r13      0x0 0 r14      0x5f 95 r15      0x0 0 rip      0x7f812f31452d 0x7f812f31452d <std::__1:😮perator+<char, std::__1::char_traits<char>, std::__1::allocator<char> >(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, char)+47> eflags     0x10246 [ PF ZF IF RF ] cs       0x33 51 ss       0x2b 43 ds       0x0 0 es       0x0 0 fs       0x0 0 gs       0x0 0
7:15 AM
Logs:
7:20 AM
@theopolis As I mentioned earlier , when Kolide fleet is offline, I am seeing this crash . Ideally , what should be external conditions , osquery daemon should not crash . It should stop after few retries .
2:36 PM
1013 19:54:49.313827 31762 init.cpp:340] osquery initialized [version=4.5.1] I1013 19:54:49.366040 31762 system.cpp:342] Found stale process for osqueryd (30509) I1013 19:54:49.366170 31762 system.cpp:374] Writing osqueryd pid (31762) to /tmp/osquery.pid I1013 19:54:49.366307 31762 extensions.cpp:453] Could not autoload extensions: Cannot open file for reading: /etc/osquery/extensions.load I1013 19:54:49.366431 31762 dispatcher.cpp:78] Adding new service: WatcherRunner (0x7f002f7fc5b8) to thread: 139638718797568 (0x7f002f7ac890) in process 31762 I1013 19:54:49.367628 31763 watcher.cpp:613] osqueryd watcher (31762) executing worker (31764) I1013 19:54:49.393806 31764 init.cpp:337] osquery worker initialized [watcher=31762] I1013 19:54:49.394104 31764 dispatcher.cpp:78] Adding new service: WatcherWatcherRunner (0x7f11a74fd918) to thread: 139713725265664 (0x7f11a73c4850) in process 31764 I1013 19:54:49.394260 31764 rocksdb.cpp:132] Opening RocksDB handle: /tmp/osquery.db E1013 19:54:49.414405 31764 init.cpp:553] An error occured during extension manager startup: Extension socket directory missing: /var/osquery/osquery.em I1013 19:54:49.414578 31764 auto_constructed_tables.cpp:97] Removing stale ATC entries I1013 19:54:49.415323 31764 tls.cpp:254] TLS/HTTPS POST request to URI: https://kolide-fleet.com:443/api/v1/osquery/config I1013 19:54:50.425562 31764 tls.cpp:254] TLS/HTTPS POST request to URI: https://kolide-fleet.com:443/api/v1/osquery/config I1013 19:54:54.437562 31764 tls.cpp:254] TLS/HTTPS POST request to URI: https://kolide-fleet.com:443/api/v1/osquery/config W1013 19:54:54.439086 31764 init.cpp:587] Error reading config: Request error: Failed to connect to kolide-fleet.com:443: Host not found (authoritative) I1013 19:54:54.442153 31764 dispatcher.cpp:78] Adding new service: TLSLogForwarder (0x7f11a752e558) to thread: 139713670649600 (0x7f11a75333c0) in process 31764 I1013 19:54:54.442387 31764 system.cpp:301] Using host identifier: aa0c5a1e-6f89-4144-9156-aaf9564268be I1013 19:54:54.443763 31764 events.cpp:867] Event publisher not enabled: auditeventpublisher: Publisher disabled via configuration I1013 19:54:54.443861 31764 events.cpp:867] Event publisher not enabled: syslog: Publisher disabled via configuration I1013 19:54:54.444042 31764 events.cpp:1126] Error registering subscriber: apparmor_events: Subscriber disabled via configuration I1013 19:54:54.444229 31764 events.cpp:1126] Error registering subscriber: process_file_events: Subscriber disabled via configuration I1013 19:54:54.444324 31764 events.cpp:1126] Error registering subscriber: selinux_events: Subscriber disabled via configuration I1013 19:54:54.444412 31764 events.cpp:1126] Error registering subscriber: socket_events: Subscriber disabled via configuration I1013 19:54:54.444758 31784 events.cpp:786] Starting event publisher run loop: inotify I1013 19:54:54.445164 31786 tls.cpp:254] TLS/HTTPS POST request to URI: https://kolide-fleet.com:443/api/v1/osquery/distributed/read I1013 19:54:54.444795 31785 events.cpp:786] Starting event publisher run loop: udev I1013 19:54:54.444773 31764 dispatcher.cpp:78] Adding new service: DistributedRunner (0x7f11a752cf58) to thread: 139713645471488 (0x7f11a752cea0) in process 31764 I1013 19:54:54.445492 31764 dispatcher.cpp:78] Adding new service: SchedulerRunner (0x7f11a7531e68) to thread: 139713637078784 (0x7f11a7548e90) in process 31764 I1013 19:54:54.999884 31783 tls.cpp:254] TLS/HTTPS POST request to URI: https://kolide-fleet.com:443/api/v1/osquery/log {"node_key":"t2uvm1nT3eSx2k6lrgXq9OcUI3ldM24W"} {"node_key":"t2uvm1nT3eSx2k6lrgXq9OcUI3ldM24W"} {"node_key":"t2uvm1nT3eSx2k6lrgXq9OcUI3ldM24W"} {"node_key":"t2uvm1nT3eSx2k6lrgXq9OcUI3ldM24W"}
theopolis

theopolis

10/15/2020, 9:19 PM
Ok I’ll take a look later today or tomorrow
👍 1
s

sanjaykcse

10/20/2020, 12:10 PM
@theopolis osquery crashes almost after a day . Even the kolide fleet is down , osqueryd keeps on executing the packs and try to send logs/results without checking the connectivity with fleet.
12:23 PM
My config : --config_plugin=tls --config_tls_endpoint=/api/v1/osquery/config --config_tls_refresh=10 --disable_distributed=false --disable_watchdog=false --distributed_interval=3 --distributed_plugin=tls --distributed_tls_max_attempts=3 --distributed_tls_read_endpoint=/api/v1/osquery/distributed/read --distributed_tls_write_endpoint=/api/v1/osquery/distributed/write --force=true --hash_delay=20 --host_identifier=uuid --logger_plugin=filesystem --pack_refresh_interval=60 --schedule_splay_percent=20 --table_delay=20 --tls_hostname=myserver.com:8080 --enroll_secret_path=/var/osquery/cert/secret --tls_server_certs=/var/osquery/cert/kolide.cert --watchdog_delay=120 --watchdog_level=1 --database_path=/tmp/osquery.db --pidfile=/tmp/osquery.pid --enroll_tls_endpoint=/api/v1/osquery/enroll --watchdog_level=1 --verbose --tls_dump --logger_path=/tmp