Has anybody seen this error in osquery logs before? The osuqery and fleet config looks ok, but I am seeing this error coming up periodically

I0208 17:44:39.302111 254910464 buffered.cpp:75] Error sending results to logger: Cannot parse JSON: The document root must not be followed by other values. Offset: 4

HI @Gudina! It sounds like there may be an issue with the Fleet agent options. Can you share your configuration here?

Here is the content of my osquery flag file

--verbose=true
--disable_events=false
--disable_audit =false
--enable_monitor=true
--audit_allow_config=true
--audit_allow_sockets=true
--disable_endpointsecurity=false
--worker_threads=2
--tls_hostname=<server>:443
--tls_server_certs=/private/var/osquery/cert_bundle.crt
--enroll_secret_path=/private/var/osquery/fleet_secret
--enroll_tls_endpoint=/api/v1/osquery/enroll
--config_plugin=tls
--config_tls_endpoint=/api/v1/osquery/config
--config_refresh=10
--disable_distributed=false
--distributed_tls_read_endpoint=/api/v1/osquery/distributed/read
--distributed_tls_write_endpoint=/api/v1/osquery/distributed/write
--logger_plugin=filesystem
--logger_tls_endpoint=/api/v1/osquery/log
--logger_path=/var/log/osquery

I see that you're logging your results locally. Is there any chance that the log file might have been modified? We could check with a fresh log by renaming the existing log file to something like

osqueryd.results.archive.log

My bad, I am actually using

logger_plugin=tls

. I changed it to

logger_plugin=filesystem

for testing.

Gotcha, same error with both options?

And yes I am seeing result logs written to

/var/log/osquery/osqueryd.results.log

What logger plugin are you using in fleet?

I am using filesystem in fleet

What versions of Fleet/osquery?

osquery version 5.5.1 Feet Version 4.22.1

Can you try adding

--verbose --tls_dump

to your osquery flagfile and then seeing what Fleet returns for that log request?

I found a conflicting config on the Feet server for logger_tls_endpoint that could have caused the

can not parse JSON

issue. On the agent, I have

--logger_tls_endpoint=/api/v1/osquery/log

and on Fleet Osquery options

logger_tls_endpoint: /api/osquery/log

IIRC both should work, but we are moving towards

/api/osquery/*

for all the osquery endpoints -- you could try removing the

v1

from your flagfile.

I am currently running a test on my device to write the result logs to the filesystem to analyze the scheduled queues' size. I will be running the test for the next 45 minutes. I already have the --verbose=true in the flag file. Once I am done with my testing, I will update the flag file with the --tls_dump=true and get back to you. Do I need to be looking for anything in particular?

Look for the request/response to

api/osquery/log

Seeing the below:

I0209 17:31:18.297050 1850322944 http_client.cpp:420] HTTP(S) request re-directed to: /api/v1/osquery/api/osquery/log
I0209 17:31:18.370936 1850322944 buffered.cpp:90] Error sending status to logger: Cannot parse JSON: The document root must not be followed by other values. Offset: 4

That path doesn't look right

Yes, I just removed the client-side config and restarted osqueryd. And I am still seeing the same error. Yes, it could be a bug that osquery tries to append its default.

Can you look at what osquery gets when it makes the config request to Fleet?

Logs shows for config it does a post to /api/v1/osquery/config

What's in the response?

For config.cpp, i see I0209 17:40:36.144910 1803808768 config.cpp:1238] Refreshing configuration state

Is

--tls_dump

turned on? You should be able to see the JSON response

Yes, I have it in the flag file. Does that need to be passed a CLI argument?

No it should work in the flagfile

trying it again

It looks like the wrong path. Somehow it's getting configured to that path that's got the extra api and osquery

Got you, it makes sense. I need to check on some more external configs and see if I can figure it out. I will get back to you with my finding. Thank you for your assistance.

Glad to hear it!