Getting SIGSEGV for osquery daemon osqueryd It cores even th

Question

Getting SIGSEGV for osquery daemon osqueryd It cores even there is no execution of pack query from fleet I have downloaded the debug symbols also but still not getting the address mapping to symbol file Any pointers how to debug this Osquery daemon crash gdb iex set auto load safe path lib tmp osqueryd 4 4 0 1 x86 64 debug c var cores core osqueryd GNU gdb GDB 7 10 Copyright C 2015 Free Software Foundation Inc License GPLv3+ GNU GPL version 3 or later lt gt This is free software you are free to change and redistribute it There is NO WARRANTY to the extent permitted by law Type show copying and show warranty for details This GDB was configured as x86 64 openwrt linux Type show configuration for configuration details For bug reporting instructions please see lt gt Find the GDB manual and other documentation resources online at lt gt For help type help Type apropos word to search for commands related to word Reading symbols from tmp osqueryd 4 4 0 1 x86 64 debug done warning core file may not match specified executable file New LWP 4862 New LWP 4871 New LWP 4870 New LWP 4869 New LWP 4868 New LWP 4867 warning Unexpected size of section ` reg xstate 4862 in core file warning Could not load shared library symbols for linux vdso so 1 Do you need set solib search path or set sysroot warning Unable to find libthread db matching inferior s thread library thread debugging will not be available warning Unable to find libthread db matching inferior s thread library thread debugging will not be available Core was generated by `osquery Program terminated with signal SIGSEGV Segmentation fault warning Unexpected size of section ` reg xstate 4862 in core file 0 0x00007f43fac971db in Current thread is 1 LWP 4862 gdb bt 0 0x00007f43fac971db in 1 0x0e24f4832af29500 in 2 0x0e24f4832af29500 in 3 0x00007f43fd415a80 in 4 0x00007f43fd408318 in 5 0x00007f43fd43aa48 in 6 0x00007fff53d99700 in 7 0x00007fff53d997a0 in 8 0x00007f43fb44425a in 9 0x00007f4300000000 in 10 0x00007fff53d99730 in 11 0xffffffffffffffff in 12 0x00007f43fba1ff4d in 13 0x0000000000000000 in gdb info r rax 0xe24f4832af29500 1019208259891008768 rbx 0x7f43fd415a80 139929988455040 rcx 0x7f43f92c2b98 139929919957912 rdx 0x7f43fa28a7b7 139929936504759 rsi 0x0 0 rdi 0x7fff53d995a0 140734600156576 rbp 0x7fff53d99c90 0x7fff53d99c90 rsp 0x7fff53d99540 0x7fff53d99540 r8 0x7fff53d99560 140734600156512 r9 0x7f43f905dda0 139929917447584 r10 0x4e585f3531314758 5645366814972135256 r11 0x7f43f9097f40 139929917685568 r12 0x0 0 r13 0x7f43fd43aa48 139929988606536 r14 0x7f43fa28a7b7 139929936504759 r15 0x7fff53d995a0 140734600156576 rip 0x7f43fac971db 0x7f43fac971db eflags 0x10246 PF ZF IF RF cs 0x33 51 ss 0x2b 43 ds 0x0 0 es 0x0 0 fs 0x0 0 gs 0x0 0 gdb x i 0x7f43fac971db = gt 0x7f43fac971db Cannot access memory at address 0x7f43fac971db gdb x i 0x7fff53d99540 0x7fff53d99540 add dl 0xb7cd50e rbp gdb x i $rsp 0x7fff53d99540 add dl 0xb7cd50e rbp gdb x 10x $rsp 0x7fff53d99540 0x2af29500 0x0e24f483 0x2af29500 0x0e24f483 0x7fff53d99550 0xfd415a80 0x00007f43 0xfd408318 0x00007f43 0x7fff53d99560 0xfd43aa48 0x00007f43 gdb x 10i $pc = gt 0x7f43fac971db Cannot access memory at address 0x7f43fac971db gdb x 10x $bsp Value can t be converted to integer gdb x 10x $rbp 0x7fff53d99c90 0xfd43aa10 0x00007f43 0xfd43aa10 0x00007f43 0x7fff53d99ca0 0x00000001 0x00000000 0x6261740a 0x0000656c 0x7fff53d99cb0 0x00000000 0x00000000

seph · Answer

A bit of a sidestep but does osquery log anything Running in verbose mode may get you an error I d also recommend trying the new 4 5 1 it added an exception catcher around all virtual table calls

sanjaykcse · Answer

Thanks Seph Will enable verbose mode and check the log

sanjaykcse · Answer

warning Unexpected size of section ` reg xstate 25287 in core file 0 std 1 basic string lt char std 1 char traits lt char gt std 1 allocator lt char gt gt is long this=0x98 at usr local osquery toolchain usr bin include c++ v1 string 1426 1426 usr local osquery toolchain usr bin include c++ v1 string No such file or directory Current thread is 1 LWP 25287 gdb bt 0 std 1 basic string lt char std 1 char traits lt char gt std 1 allocator lt char gt gt is long this=0x98 at usr local osquery toolchain usr bin include c++ v1 string 1426 1 std 1 basic string lt char std 1 char traits lt char gt std 1 allocator lt char gt gt size this=0x98 at usr local osquery toolchain usr bin include c++ v1 string 953 2 std 1 operator+ lt char std 1 char traits lt char gt std 1 allocator lt char gt gt lhs= rhs=95 at usr local osquery toolchain usr bin include c++ v1 string 4097 3 0x0000000001cdbd00 in std 1 messages lt wchar t gt amp std 1 anonymous namespace make lt std 1 messages lt wchar t gt unsigned int gt unsigned int buf 4 0xba0461c30f1cd400 in 5 0x00007fffd2a2c9b8 in 6 0x00007fffd2a2ca48 in 7 0x0000000000000000 in gdb info reg rax 0xba0461c30f1cd400 5042798192348048384 rbx 0x7fffd2a2c9b8 140736727271864 rcx 0x0 0 rdx 0x5f 95 rsi 0x98 152 rdi 0x7fffd2a2c9b8 140736727271864 rbp 0x98 0x98 rsp 0x7fffd2a2c980 0x7fffd2a2c980 r8 0x7fffd2a2cc28 140736727272488 r9 0x7f0f1dd08da0 139702901444000 r10 0x6974696e69207265 7598814394211922533 r11 0x7f0f1dd42f40 139702901681984 r12 0x3393650 54081104 r13 0x0 0 r14 0x5f 95 r15 0x0 0 rip 0xba9c8d 0xba9c8d lt std 1 operator+ lt char std 1 char traits lt char gt std 1 allocator lt char gt gt std 1 basic string lt char std 1 char traits lt char gt std 1 allocator lt char gt gt const amp char +47 gt eflags 0x10246 PF ZF IF RF cs 0x33 51 ss 0x2b 43 ds 0x0 0 es 0x0 0 fs 0x0 0 gs 0x0 0

sanjaykcse · Answer

got better stack trace Analysing it Should I log a bug

seph · Answer

< theopolis> thoughts

theopolis · Answer

< sanjaykcse> is that with 4 5 1

theopolis · Answer

did you try to run with ` verbose` what logs were printed

sanjaykcse · Answer

Hi < theopolis> this crash is from 4 4 0 Today I have updated osquery to 4 5 1 will update if get see any crash From the logs it appears the fleet was down enroll secret OXZMOA9Wr6AZy7N95UJBxxxxxds7Z2CN host identifier 191534d5 a9d0 4c69 bk85 fb3557c5edba platform type 9 host details os version arch x86 64 major 0 minor 0 name Unknown patch 0 pid with namespace 0 platform posix osquery info build distro centos7 build platform 1 config hash config valid 0 extensions inactive instance id b9d0f315 be38 4f29 9ce6 d394e7813951 pid 2546 platform mask 9 start time 1602223587 uuid 191534d5 a9d0 4c69 bf85 fb3557c5edba version 4 4 0 watcher 2518 platform info address 0xf000 date 10 08 2018 extra revision 5 12 size 5242880 vendor American Megatrends Inc version 5 12 Z131E1009 11 38 14 526799 2546 init cpp 714 Cannot activate tls logger plugin No node key TLS logging disabled I1009 11 38 15 003304 2538 watcher cpp 585 osqueryd watcher 2518 executing worker 5519 I1009 11 38 15 024971 5519 init cpp 340 osquery worker initialized watcher=2518 I1009 11 38 15 025318 5519 dispatcher cpp 77 Adding new service WatcherWatcherRunner 0x2cdcac8 to thread 139827282855680 0x2cdcb70 in process 5519 I1009 11 38 15 025480 5519 rocksdb cpp 131 Opening RocksDB handle tmp osquery db E1009 11 38 15 028797 5519 init cpp 568 An error occured during extension manager startup Extension socket directory missing var osquery osquery em I1009 11 38 15 029001 5519 tls enroll cpp 69 TLSEnrollPlugin requesting a node enroll key from I1009 11 38 15 029183 5519 system cpp 294 Using host identifier 191534d5 a9d0 4c69 bk85 fb3557c5edba

sanjaykcse · Answer

Hi < theopolis> I am seeing this crash 0n 4 5 2 also

sanjaykcse · Answer

Core was generated by `osquery Program terminated with signal SIGSEGV Segmentation fault warning Unexpected size of section ` reg xstate 10711 in core file 0 std 1 basic string lt char std 1 char traits lt char gt std 1 allocator lt char gt gt is long this=0x98 at opt osquery toolchain usr bin include c++ v1 string 1426 1426 opt osquery toolchain usr bin include c++ v1 string No such file or directory Current thread is 1 LWP 10711 gdb bt 0 std 1 basic string lt char std 1 char traits lt char gt std 1 allocator lt char gt gt is long this=0x98 at opt osquery toolchain usr bin include c++ v1 string 1426 1 std 1 basic string lt char std 1 char traits lt char gt std 1 allocator lt char gt gt size this=0x98 at opt osquery toolchain usr bin include c++ v1 string 953 2 std 1 operator+ lt char std 1 char traits lt char gt std 1 allocator lt char gt gt lhs= rhs=95 at opt osquery toolchain usr bin include c++ v1 string 4097 3 0x00007f81304bbd90 in std 1 messages lt wchar t gt amp std 1 anonymous namespace make lt std 1 messages lt wchar t gt unsigned int gt unsigned int buf 4 0xe527c8e13c6f9600 in 5 0x00007fff87502998 in 6 0x00007fff87502a28 in 7 0x0000000000000000 in gdb info reg rax 0xe527c8e13c6f9600 1934356645225196032 rbx 0x7fff87502998 140735463565720 rcx 0x0 0 rdx 0x5f 95 rsi 0x98 152 rdi 0x7fff87502998 140735463565720 rbp 0x98 0x98 rsp 0x7fff87502960 0x7fff87502960 r8 0x7fff87502c08 140735463566344 r9 0x7f812d6ccda0 140192789613984 r10 0x6974696e69207265 7598814394211922533 r11 0x7f812d706f40 140192789851968 r12 0x7f81319f6530 140192860038448 r13 0x0 0 r14 0x5f 95 r15 0x0 0 rip 0x7f812f31452d 0x7f812f31452d lt std 1 operator+ lt char std 1 char traits lt char gt std 1 allocator lt char gt gt std 1 basic string lt char std 1 char traits lt char gt std 1 allocator lt char gt gt const amp char +47 gt eflags 0x10246 PF ZF IF RF cs 0x33 51 ss 0x2b 43 ds 0x0 0 es 0x0 0 fs 0x0 0 gs 0x0 0

sanjaykcse · Answer

Logs

sanjaykcse · Answer

< theopolis> As I mentioned earlier when Kolide fleet is offline I am seeing this crash Ideally what should be external conditions osquery daemon should not crash It should stop after few retries

sanjaykcse · Answer

1013 19 54 49 313827 31762 init cpp 340 osquery initialized version=4 5 1 I1013 19 54 49 366040 31762 system cpp 342 Found stale process for osqueryd 30509 I1013 19 54 49 366170 31762 system cpp 374 Writing osqueryd pid 31762 to tmp osquery pid I1013 19 54 49 366307 31762 extensions cpp 453 Could not autoload extensions Cannot open file for reading etc osquery extensions load I1013 19 54 49 366431 31762 dispatcher cpp 78 Adding new service WatcherRunner 0x7f002f7fc5b8 to thread 139638718797568 0x7f002f7ac890 in process 31762 I1013 19 54 49 367628 31763 watcher cpp 613 osqueryd watcher 31762 executing worker 31764 I1013 19 54 49 393806 31764 init cpp 337 osquery worker initialized watcher=31762 I1013 19 54 49 394104 31764 dispatcher cpp 78 Adding new service WatcherWatcherRunner 0x7f11a74fd918 to thread 139713725265664 0x7f11a73c4850 in process 31764 I1013 19 54 49 394260 31764 rocksdb cpp 132 Opening RocksDB handle tmp osquery db E1013 19 54 49 414405 31764 init cpp 553 An error occured during extension manager startup Extension socket directory missing var osquery osquery em I1013 19 54 49 414578 31764 auto constructed tables cpp 97 Removing stale ATC entries I1013 19 54 49 415323 31764 tls cpp 254 TLS HTTPS POST request to URI I1013 19 54 50 425562 31764 tls cpp 254 TLS HTTPS POST request to URI I1013 19 54 54 437562 31764 tls cpp 254 TLS HTTPS POST request to URI W1013 19 54 54 439086 31764 init cpp 587 Error reading config Request error Failed to connect to Host not found authoritative I1013 19 54 54 442153 31764 dispatcher cpp 78 Adding new service TLSLogForwarder 0x7f11a752e558 to thread 139713670649600 0x7f11a75333c0 in process 31764 I1013 19 54 54 442387 31764 system cpp 301 Using host identifier aa0c5a1e 6f89 4144 9156 aaf9564268be I1013 19 54 54 443763 31764 events cpp 867 Event publisher not enabled auditeventpublisher Publisher disabled via configuration I1013 19 54 54 443861 31764 events cpp 867 Event publisher not enabled syslog Publisher disabled via configuration I1013 19 54 54 444042 31764 events cpp 1126 Error registering subscriber apparmor events Subscriber disabled via configuration I1013 19 54 54 444229 31764 events cpp 1126 Error registering subscriber process file events Subscriber disabled via configuration I1013 19 54 54 444324 31764 events cpp 1126 Error registering subscriber selinux events Subscriber disabled via configuration I1013 19 54 54 444412 31764 events cpp 1126 Error registering subscriber socket events Subscriber disabled via configuration I1013 19 54 54 444758 31784 events cpp 786 Starting event publisher run loop inotify I1013 19 54 54 445164 31786 tls cpp 254 TLS HTTPS POST request to URI I1013 19 54 54 444795 31785 events cpp 786 Starting event publisher run loop udev I1013 19 54 54 444773 31764 dispatcher cpp 78 Adding new service DistributedRunner 0x7f11a752cf58 to thread 139713645471488 0x7f11a752cea0 in process 31764 I1013 19 54 54 445492 31764 dispatcher cpp 78 Adding new service SchedulerRunner 0x7f11a7531e68 to thread 139713637078784 0x7f11a7548e90 in process 31764 I1013 19 54 54 999884 31783 tls cpp 254 TLS HTTPS POST request to URI node key t2uvm1nT3eSx2k6lrgXq9OcUI3ldM24W node key t2uvm1nT3eSx2k6lrgXq9OcUI3ldM24W node key t2uvm1nT3eSx2k6lrgXq9OcUI3ldM24W node key t2uvm1nT3eSx2k6lrgXq9OcUI3ldM24W

theopolis · Answer

Ok I ll take a look later today or tomorrow

sanjaykcse · Answer

< theopolis> osquery crashes almost after a day Even the kolide fleet is down osqueryd keeps on executing the packs and try to send logs results without checking the connectivity with fleet

sanjaykcse · Answer

My config config plugin=tls config tls endpoint= api v1 osquery config config tls refresh=10 disable distributed=false disable watchdog=false distributed interval=3 distributed plugin=tls distributed tls max attempts=3 distributed tls read endpoint= api v1 osquery distributed read distributed tls write endpoint= api v1 osquery distributed write force=true hash delay=20 host identifier=uuid logger plugin=filesystem pack refresh interval=60 schedule splay percent=20 table delay=20 tls hostname=<http myserver com|myserver com> 8080 enroll secret path= var osquery cert secret tls server certs= var osquery cert kolide cert watchdog delay=120 watchdog level=1 database path= tmp osquery db pidfile= tmp osquery pid enroll tls endpoint= api v1 osquery enroll watchdog level=1 verbose tls dump logger path= tmp