conversation osquery #fleet

Title

Ojas

04/25/2022, 10:34 AM

message-2.txt

I have 600+ windows hosts and all are offline

Lucas Rodriguez

04/25/2022, 11:54 AM

Hi Ojas, could you check if any of your Windows hosts can connect to tuf.fleetctl.com? (

ping <http://tuf.fleetctl.com|tuf.fleetctl.com>

and maybe

curl <https://tuf.fleetctl.com>

)

Also, if you have access to any of such hosts, please run:

C:\Program Files\Orbit\bin\orbit.exe --version

Ojas

04/25/2022, 11:57 AM

i am able to ping and version is orbit 0.0.9

Lucas Rodriguez

04/25/2022, 11:58 AM

How about the curl or wget command? Asking this due to the

lookup <http://tuf.fleetctl.com|tuf.fleetctl.com>: no such host

logs.

Ojas

04/25/2022, 11:59 AM

curl https://tuf.fleetctl.om does give me result

message-3.txt

@Lucas Rodriguez seems like it got screwed up after new updates maybe?

Lucas Rodriguez

04/25/2022, 12:53 PM

Mhm... something to check: any AV or firewall configuration that could be preventing Orbit to connect to

<http://tuf.fleetctl.com|tuf.fleetctl.com>

Ojas

04/25/2022, 12:54 PM

but then how i am able to curl and ping to it?

Lucas Rodriguez

04/25/2022, 12:55 PM

Not sure, but maybe something is preventing the Orbit process itself to making such connection or making the DNS request? (the error seems to suggest Orbit cannot even resolve DNS for tuf.fleetctl.com)

Ojas

04/25/2022, 1:02 PM

how can i debug it?

Lucas Rodriguez

04/25/2022, 1:03 PM

You could try running Orbit manually in

cmd

and see if it provides more logs/information there.

To do so, you can do the following:

Click start, open "Services" as administrator, right click

fleet-osquery

and select

Properties

. And you will be able to copy/paste the command+arguments used to start Orbit.

Ojas

04/25/2022, 1:12 PM

this is what i get

message-4.txt

Lucas Rodriguez

04/25/2022, 1:13 PM

Error reading config: Cannot parse JSON: The document root must not be followed by other values. Offset: 4

Did you change the agents options in Fleet recently?

Ojas

04/25/2022, 1:14 PM

config: options: logger_plugin: filesystem filesystem: enable_log_rotation: true

i added filesystem: enable_log_rotation: true

Lucas Rodriguez

04/25/2022, 1:16 PM

Mhm... is that the

agent options

or the

fleet config

? By agent options I mean "Global agent options" from the settings:

Ojas

04/25/2022, 1:17 PM

Lucas Rodriguez

04/25/2022, 1:17 PM

If it's the global agent options, then:

filesystem:
    enable_log_rotation: true

is not valid, as it's a fleet config, not an osquery config.

Removing that should fix the issue.

osquery flags are documented here https://osquery.readthedocs.io/en/stable/installation/cli-flags/

Ojas

04/25/2022, 1:18 PM

where do i put the osquery flasgs then. I though it goes in global agent options only

Lucas Rodriguez

04/25/2022, 1:18 PM

filesystem.enable_log_rotation

that configures log rotation for fleet logs, not osquery logs.

By fleet logs I mean Fleet log files.

Such config should be added via

fleetctl

, it doesn't affect agents, it affects fleet log files.

Ojas

04/25/2022, 1:20 PM

oh alright

i have removed the config but still nothing.

Lucas Rodriguez

04/25/2022, 2:05 PM

OK, let me check, osquery agents may be trying to use the old config...

By executing Orbit manually in

cmd

as before do you still get the same error?

Any reason to not use some of the default values (other than

logger_plugin

) for

Global agent settings

configuration that comes with fleet?

config:
  options:
    logger_plugin: tls
    pack_delimiter: /
    logger_tls_period: 10
    distributed_plugin: tls
    disable_distributed: false
    logger_tls_endpoint: /api/osquery/log
    distributed_interval: 10
    distributed_tls_max_attempts: 3
  decorators:
    load:
      - SELECT uuid AS host_uuid FROM system_info;
      - SELECT hostname AS hostname FROM system_info;
overrides: {}

Ojas

04/25/2022, 7:03 PM

It was working well without all these, so i removed them. Didnt have any issues till now.

also yes i am getting the same error on executing orbit manually

Lucas Rodriguez

04/25/2022, 7:16 PM

OK, so currently you have the following agent config?

config:
  options:
    logger_plugin: filesystem

Ojas

04/25/2022, 7:16 PM

nope i changed to the default one and set logger plugin to filesystem

config: options: pack_delimiter: / logger_tls_period: 10 distributed_plugin: tls disable_distributed: false logger_tls_endpoint: /api/osquery/log distributed_interval: 10 distributed_tls_max_attempts: 3 decorators: load: - SELECT uuid AS host_uuid FROM system_info; - SELECT hostname AS hostname FROM system_info; overrides: {}

Lucas Rodriguez

04/25/2022, 7:17 PM

OK, and you still get the

Error reading config: Cannot parse JSON: The document root must not be followed by other values. Offset: 4

when running Orbit manually in

cmd

Ojas

04/25/2022, 7:17 PM

yep yep

any fix for it? Also how do i stop auto update in future?

Lucas Rodriguez

04/25/2022, 8:38 PM

Mhm... that's odd, I'll check with the team, but the issue doesn't seem related to the Orbit upgrade.

Also, Orbit now has a

--disable-updates

option (can be set in Orbit manually or you can define it when generating the packages (MSI, PKG, DEB, RPM) with

fleetctl package

Dawei Zhang

04/26/2022, 2:54 AM

hi, I got the exactly same problem

W0426 02:47:01.457355 247643 init.cpp:616] Error reading config: Cannot parse JSON: The document root must not be followed by other values. Offset: 4 is this caused by orbit automatic upgrade?

according last date, it seems this was working fine until April 22nd

Ojas

04/26/2022, 6:06 AM

@Lucas Rodriguez Is there an option in which we can revet to the older version where things were stable for us?

Lucas Rodriguez

04/26/2022, 11:25 AM

Hi folks, I'll check with the team and get back to you.

🙌 1

OK, I was able to reproduce the issue. We will be working on a fix today and release an auto-update as soon as we can. Issue: #5367.

👍 1

Ojas

04/26/2022, 12:02 PM

Amazing 🔥 Thanks Lucas

7 Views

#fleet Join Slack