https://github.com/osquery/osquery logo
#fleet
Title
# fleet
g

Gudina

02/09/2023, 6:28 PM
Has anybody seen this error in osquery logs before? The osuqery and fleet config looks ok, but I am seeing this error coming up periodically
I0208 17:44:39.302111 254910464 buffered.cpp:75] Error sending results to logger: Cannot parse JSON: The document root must not be followed by other values. Offset: 4
k

Kathy Satterlee

02/09/2023, 6:36 PM
HI @Gudina! It sounds like there may be an issue with the Fleet agent options. Can you share your configuration here?
Sorry, I see that's related to logging. can you share your osquery flags as well if you are using plain osquery?
g

Gudina

02/09/2023, 6:46 PM
Here is the content of my osquery flag file
Copy code
--verbose=true
--disable_events=false
--disable_audit =false
--enable_monitor=true
--audit_allow_config=true
--audit_allow_sockets=true
--disable_endpointsecurity=false
--worker_threads=2
--tls_hostname=<server>:443
--tls_server_certs=/private/var/osquery/cert_bundle.crt
--enroll_secret_path=/private/var/osquery/fleet_secret
--enroll_tls_endpoint=/api/v1/osquery/enroll
--config_plugin=tls
--config_tls_endpoint=/api/v1/osquery/config
--config_refresh=10
--disable_distributed=false
--distributed_tls_read_endpoint=/api/v1/osquery/distributed/read
--distributed_tls_write_endpoint=/api/v1/osquery/distributed/write
--logger_plugin=filesystem
--logger_tls_endpoint=/api/v1/osquery/log
--logger_path=/var/log/osquery
And I am using plan osquery
k

Kathy Satterlee

02/09/2023, 6:51 PM
I see that you're logging your results locally. Is there any chance that the log file might have been modified? We could check with a fresh log by renaming the existing log file to something like
osqueryd.results.archive.log
g

Gudina

02/09/2023, 6:58 PM
My bad, I am actually using
logger_plugin=tls
. I changed it to
logger_plugin=filesystem
for testing.
k

Kathy Satterlee

02/09/2023, 7:00 PM
Gotcha, same error with both options?
g

Gudina

02/09/2023, 7:00 PM
And yes I am seeing result logs written to
/var/log/osquery/osqueryd.results.log
I am not seeing the error when using
logger_plugin=filesystem
k

Kathy Satterlee

02/09/2023, 7:01 PM
What logger plugin are you using in fleet?
g

Gudina

02/09/2023, 7:02 PM
I am using filesystem in fleet
k

Kathy Satterlee

02/09/2023, 7:02 PM
What versions of Fleet/osquery?
Are you seeing the error for all hosts, or just this host?
g

Gudina

02/09/2023, 7:03 PM
osquery version 5.5.1 Feet Version 4.22.1
All hosts.
z

zwass

02/09/2023, 7:14 PM
Can you try adding
--verbose --tls_dump
to your osquery flagfile and then seeing what Fleet returns for that log request?
g

Gudina

02/09/2023, 7:15 PM
I found a conflicting config on the Feet server for logger_tls_endpoint that could have caused the
can not parse JSON
issue. On the agent, I have
--logger_tls_endpoint=/api/v1/osquery/log
and on Fleet Osquery options
logger_tls_endpoint: /api/osquery/log
z

zwass

02/09/2023, 7:16 PM
IIRC both should work, but we are moving towards
/api/osquery/*
for all the osquery endpoints -- you could try removing the
v1
from your flagfile.
g

Gudina

02/09/2023, 7:39 PM
I am currently running a test on my device to write the result logs to the filesystem to analyze the scheduled queues' size. I will be running the test for the next 45 minutes. I already have the --verbose=true in the flag file. Once I am done with my testing, I will update the flag file with the --tls_dump=true and get back to you. Do I need to be looking for anything in particular?
z

zwass

02/09/2023, 8:28 PM
Look for the request/response to
api/osquery/log
g

Gudina

02/10/2023, 1:32 AM
Seeing the below:
Copy code
I0209 17:31:18.297050 1850322944 http_client.cpp:420] HTTP(S) request re-directed to: /api/v1/osquery/api/osquery/log
I0209 17:31:18.370936 1850322944 buffered.cpp:90] Error sending status to logger: Cannot parse JSON: The document root must not be followed by other values. Offset: 4
z

zwass

02/10/2023, 1:33 AM
That path doesn't look right
I wonder if somehow osquery has a bug where it appended the two paths (from your flagfile and from the TLS config)?
Maybe try configuring it in just one place?
g

Gudina

02/10/2023, 1:39 AM
Yes, I just removed the client-side config and restarted osqueryd. And I am still seeing the same error. Yes, it could be a bug that osquery tries to append its default.
z

zwass

02/10/2023, 1:40 AM
Can you look at what osquery gets when it makes the config request to Fleet?
g

Gudina

02/10/2023, 1:41 AM
Logs shows for config it does a post to /api/v1/osquery/config
z

zwass

02/10/2023, 1:41 AM
What's in the response?
g

Gudina

02/10/2023, 1:42 AM
For config.cpp, i see I0209 174036.144910 1803808768 config.cpp:1238] Refreshing configuration state
z

zwass

02/10/2023, 1:44 AM
Is
--tls_dump
turned on? You should be able to see the JSON response
g

Gudina

02/10/2023, 1:44 AM
Yes, I have it in the flag file. Does that need to be passed a CLI argument?
z

zwass

02/10/2023, 1:45 AM
No it should work in the flagfile
g

Gudina

02/10/2023, 1:46 AM
trying it again
I ended up running as below, and I now I see the 404 response. Seam like some firewall issue?
osqueryd --flagfile=/private/var/osquery/osquery.flags
I0209 174834.201555 1871458304 http_client.cpp:420] HTTP(S) request re-directed to: /api/v1/osquery/api/osquery/log 404 page not found
z

zwass

02/10/2023, 1:52 AM
It looks like the wrong path. Somehow it's getting configured to that path that's got the extra api and osquery
g

Gudina

02/10/2023, 1:55 AM
Got you, it makes sense. I need to check on some more external configs and see if I can figure it out. I will get back to you with my finding. Thank you for your assistance.
It was some external configs that caused the issue. I am good now. Thanks again!
z

zwass

02/10/2023, 7:07 PM
Glad to hear it!
3 Views