Getting SIGSEGV for osquery daemon (osqueryd). I...
# generals
sanjaykcse
10/08/2020, 8:55 AMGetting SIGSEGV for osquery daemon (osqueryd). It cores even there is no execution of pack/query from fleet.
I have downloaded the debug symbols also , but still not getting the address mapping to symbol file . Any pointers how to debug this Osquery daemon crash ?
# gdb -iex "set auto-load safe-path /lib" /tmp/osqueryd-4.4.0-1.x86_64.debug -c /var/cores/core.osqueryd
GNU gdb (GDB) 7.10
Copyright (C) 2015 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-openwrt-linux".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /tmp/osqueryd-4.4.0-1.x86_64.debug...done.
warning: core file may not match specified executable file.
[New LWP 4862]
[New LWP 4871]
[New LWP 4870]
[New LWP 4869]
[New LWP 4868]
[New LWP 4867]
warning: Unexpected size of section `.reg-xstate/4862' in core file.
warning: Could not load shared library symbols for linux-vdso.so.1.
Do you need "set solib-search-path" or "set sysroot"?
warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available.
Core was generated by
Copy code
osquery '.
Program terminated with signal SIGSEGV, Segmentation fault.
warning: Unexpected size of sections
seph
10/08/2020, 1:09 PM
A bit of a sidestep, but does osquery log anything? Running in verbose mode may get you an error.
I’d also recommend trying the new 4.5.1, it added an exception catcher around all virtual table calls
s
sanjaykcse
10/08/2020, 4:47 PMThanks Seph . Will enable verbose mode and check the log.
sanjaykcse
10/11/2020, 2:46 PMwarning: Unexpected size of section `.reg-xstate/25287' in core file.
#0 std: 1:basic_string<char, std: 1:char_traits<char>, std: 1:allocator<char> >::__is_long (this=0x98) at /usr/local/osquery-toolchain/usr/bin/../include/c++/v1/string:1426
1426 /usr/local/osquery-toolchain/usr/bin/../include/c++/v1/string: No such file or directory.
[Current thread is 1 (LWP 25287)]
(gdb) bt
#0 std: 1:basic_string<char, std: 1:char_traits<char>, std: 1:allocator<char> >::__is_long (this=0x98) at /usr/local/osquery-toolchain/usr/bin/../include/c++/v1/string:1426
#1 std: 1:basic_string<char, std: 1:char_traits<char>, std: 1:allocator<char> >::size (this=0x98) at /usr/local/osquery-toolchain/usr/bin/../include/c++/v1/string:953
#2 std: 1:operator+<char, std: 1:char_traits<char>, std: 1:allocator<char> > (__lhs=..., __rhs=95 '_') at /usr/local/osquery-toolchain/usr/bin/../include/c++/v1/string:4097
#3 0x0000000001cdbd00 in std: 1:messages<wchar_t>& std: 1:(anonymous namespace):make<std 1:messages<wchar_t>, unsigned int>(unsigned int)::buf ()
#4 0xba0461c30f1cd400 in ?? ()
#5 0x00007fffd2a2c9b8 in ?? ()
#6 0x00007fffd2a2ca48 in ?? ()
#7 0x0000000000000000 in ?? ()
(gdb) info reg
rax 0xba0461c30f1cd400 -5042798192348048384
rbx 0x7fffd2a2c9b8 140736727271864
rcx 0x0 0
rdx 0x5f 95
rsi 0x98 152
rdi 0x7fffd2a2c9b8 140736727271864
rbp 0x98 0x98
rsp 0x7fffd2a2c980 0x7fffd2a2c980
r8 0x7fffd2a2cc28 140736727272488
r9 0x7f0f1dd08da0 139702901444000
r10 0x6974696e69207265 7598814394211922533
r11 0x7f0f1dd42f40 139702901681984
r12 0x3393650 54081104
r13 0x0 0
r14 0x5f 95
r15 0x0 0
rip 0xba9c8d 0xba9c8d <std: 1:operator+<char, std: 1:char_traits<char>, std: 1:allocator<char> >(std: 1:basic_string<char, std: 1:char_traits<char>, std: 1:allocator<char> > const&, char)+47>
eflags 0x10246 [ PF ZF IF RF ]
cs 0x33 51
ss 0x2b 43
ds 0x0 0
es 0x0 0
fs 0x0 0
gs 0x0 0
sanjaykcse
10/11/2020, 2:47 PMgot better stack trace . Analysing it . Should I log a bug ??
s
seph
10/11/2020, 3:00 PM
@theopolis thoughts?
t
theopolis
10/11/2020, 3:02 PM
@sanjaykcse is that with 4.5.1?
theopolis
10/11/2020, 3:03 PM
did you try to run with
--verboses
sanjaykcse
10/12/2020, 1:28 PMHi @theopolis this crash is from 4.4.0 . Today , I have updated osquery to 4.5.1. will update if get see any crash .
From the logs it appears the fleet was down.
{"enroll_secret":"OXZMOA9Wr6AZy7N95UJBxxxxxds7Z2CN","host_identifier":"191534d5-a9d0-4c69-bk85-fb3557c5edba","platform_type":"9","host_details":{"os_version":{"arch":"x86_64","major":"0","minor":"0","name":"Unknown","patch":"0","pid_with_namespace":"0","platform":"posix"},"osquery_info":{"build_distro":"centos7","build_platform":"1","config_hash":"","config_valid":"0","extensions":"inactive","instance_id":"b9d0f315-be38-4f29-9ce6-d394e7813951","pid":"2546","platform_mask":"9","start_time":"1602223587","uuid":"191534d5-a9d0-4c69-bf85-fb3557c5edba","version":"4.4.0","watcher":"2518"},"platform_info":{"address":"0xf000","date":"10/08/2018","extra":"","revision":"5.12","size":"5242880","vendor":"American Megatrends Inc.","version":"5.12 (Z131E1009 113814.526799 2546 init.cpp:714] Cannot activate tls logger plugin: No node key, TLS logging disabled.
I1009 113815.003304 2538 watcher.cpp:585] osqueryd watcher (2518) executing worker (5519)
I1009 113815.024971 5519 init.cpp:340] osquery worker initialized [watcher=2518]
I1009 113815.025318 5519 dispatcher.cpp:77] Adding new service: WatcherWatcherRunner (0x2cdcac8) to thread: 139827282855680 (0x2cdcb70) in process 5519
I1009 113815.025480 5519 rocksdb.cpp:131] Opening RocksDB handle: /tmp/osquery.db
E1009 113815.028797 5519 init.cpp:568] An error occured during extension manager startup: Extension socket directory missing: /var/osquery/osquery.em
I1009 113815.029001 5519 tls_enroll.cpp:69] TLSEnrollPlugin requesting a node enroll key from: https://ec2-52-25-115-238.us-west-2.compute.amazonaws.com:8080/api/v1/osquery/enroll
I1009 113815.029183 5519 system.cpp:294] Using host identifier: 191534d5-a9d0-4c69-bk85-fb3557c5edba
sanjaykcse
10/15/2020, 7:15 AMHi @theopolis I am seeing this crash 0n 4.5.2 also
sanjaykcse
10/15/2020, 7:15 AMCore was generated by
Copy code
osquery '.
Program terminated with signal SIGSEGV, Segmentation fault.
warning: Unexpected size of sectionsanjaykcse
10/15/2020, 7:15 AMLogs:
sanjaykcse
10/15/2020, 7:20 AM@theopolis As I mentioned earlier , when Kolide fleet is offline, I am seeing this crash . Ideally , what should be external conditions , osquery daemon should not crash . It should stop after few retries .
sanjaykcse
10/15/2020, 2:36 PM1013 195449.313827 31762 init.cpp:340] osquery initialized [version=4.5.1]
I1013 195449.366040 31762 system.cpp:342] Found stale process for osqueryd (30509)
I1013 195449.366170 31762 system.cpp:374] Writing osqueryd pid (31762) to /tmp/osquery.pid
I1013 195449.366307 31762 extensions.cpp:453] Could not autoload extensions: Cannot open file for reading: /etc/osquery/extensions.load
I1013 195449.366431 31762 dispatcher.cpp:78] Adding new service: WatcherRunner (0x7f002f7fc5b8) to thread: 139638718797568 (0x7f002f7ac890) in process 31762
I1013 195449.367628 31763 watcher.cpp:613] osqueryd watcher (31762) executing worker (31764)
I1013 195449.393806 31764 init.cpp:337] osquery worker initialized [watcher=31762]
I1013 195449.394104 31764 dispatcher.cpp:78] Adding new service: WatcherWatcherRunner (0x7f11a74fd918) to thread: 139713725265664 (0x7f11a73c4850) in process 31764
I1013 195449.394260 31764 rocksdb.cpp:132] Opening RocksDB handle: /tmp/osquery.db
E1013 195449.414405 31764 init.cpp:553] An error occured during extension manager startup: Extension socket directory missing: /var/osquery/osquery.em
I1013 195449.414578 31764 auto_constructed_tables.cpp:97] Removing stale ATC entries
I1013 195449.415323 31764 tls.cpp:254] TLS/HTTPS POST request to URI: https://kolide-fleet.com:443/api/v1/osquery/config
I1013 195450.425562 31764 tls.cpp:254] TLS/HTTPS POST request to URI: https://kolide-fleet.com:443/api/v1/osquery/config
I1013 195454.437562 31764 tls.cpp:254] TLS/HTTPS POST request to URI: https://kolide-fleet.com:443/api/v1/osquery/config
W1013 195454.439086 31764 init.cpp:587] Error reading config: Request error: Failed to connect to kolide-fleet.com:443: Host not found (authoritative)
I1013 195454.442153 31764 dispatcher.cpp:78] Adding new service: TLSLogForwarder (0x7f11a752e558) to thread: 139713670649600 (0x7f11a75333c0) in process 31764
I1013 195454.442387 31764 system.cpp:301] Using host identifier: aa0c5a1e-6f89-4144-9156-aaf9564268be
I1013 195454.443763 31764 events.cpp:867] Event publisher not enabled: auditeventpublisher: Publisher disabled via configuration
I1013 195454.443861 31764 events.cpp:867] Event publisher not enabled: syslog: Publisher disabled via configuration
I1013 195454.444042 31764 events.cpp:1126] Error registering subscriber: apparmor_events: Subscriber disabled via configuration
I1013 195454.444229 31764 events.cpp:1126] Error registering subscriber: process_file_events: Subscriber disabled via configuration
I1013 195454.444324 31764 events.cpp:1126] Error registering subscriber: selinux_events: Subscriber disabled via configuration
I1013 195454.444412 31764 events.cpp:1126] Error registering subscriber: socket_events: Subscriber disabled via configuration
I1013 195454.444758 31784 events.cpp:786] Starting event publisher run loop: inotify
I1013 195454.445164 31786 tls.cpp:254] TLS/HTTPS POST request to URI: https://kolide-fleet.com:443/api/v1/osquery/distributed/read
I1013 195454.444795 31785 events.cpp:786] Starting event publisher run loop: udev
I1013 195454.444773 31764 dispatcher.cpp:78] Adding new service: DistributedRunner (0x7f11a752cf58) to thread: 139713645471488 (0x7f11a752cea0) in process 31764
I1013 195454.445492 31764 dispatcher.cpp:78] Adding new service: SchedulerRunner (0x7f11a7531e68) to thread: 139713637078784 (0x7f11a7548e90) in process 31764
I1013 195454.999884 31783 tls.cpp:254] TLS/HTTPS POST request to URI: https://kolide-fleet.com:443/api/v1/osquery/log
{"node_key":"t2uvm1nT3eSx2k6lrgXq9OcUI3ldM24W"}
{"node_key":"t2uvm1nT3eSx2k6lrgXq9OcUI3ldM24W"}
{"node_key":"t2uvm1nT3eSx2k6lrgXq9OcUI3ldM24W"}
{"node_key":"t2uvm1nT3eSx2k6lrgXq9OcUI3ldM24W"}
t
theopolis
10/15/2020, 9:19 PM
Ok I’ll take a look later today or tomorrow
👍 1
s
sanjaykcse
10/20/2020, 12:10 PM@theopolis osquery crashes almost after a day . Even the kolide fleet is down , osqueryd keeps on executing the packs and try to send logs/results without checking the connectivity with fleet.
sanjaykcse
10/20/2020, 12:23 PMMy config :
--config_plugin=tls
--config_tls_endpoint=/api/v1/osquery/config
--config_tls_refresh=10
--disable_distributed=false
--disable_watchdog=false
--distributed_interval=3
--distributed_plugin=tls
--distributed_tls_max_attempts=3
--distributed_tls_read_endpoint=/api/v1/osquery/distributed/read
--distributed_tls_write_endpoint=/api/v1/osquery/distributed/write
--force=true
--hash_delay=20
--host_identifier=uuid
--logger_plugin=filesystem
--pack_refresh_interval=60
--schedule_splay_percent=20
--table_delay=20
--tls_hostname=myserver.com:8080
--enroll_secret_path=/var/osquery/cert/secret
--tls_server_certs=/var/osquery/cert/kolide.cert
--watchdog_delay=120
--watchdog_level=1
--database_path=/tmp/osquery.db
--pidfile=/tmp/osquery.pid
--enroll_tls_endpoint=/api/v1/osquery/enroll
--watchdog_level=1
--verbose
--tls_dump
--logger_path=/tmp
3 Views