How could one collect process events and socket events from osquery #fleet

How could one collect process_events and socket_ev...

Sunil

01/30/2023, 3:07 PM

How could one collect process_events and socket_events from OSQuery to fleetdm (fleet) and then put it in kafkarest ? I had already gone thro' fleetdm documentation for kafkarest but not really understood the whole setup.

Rachel Perkins

01/30/2023, 3:14 PM

Hi Sunil! Were you able to create and run queries on those tables successfully, but just need help with configuring to Kafka rest?

Rachel Perkins

01/30/2023, 3:19 PM

Is this the documentation you were referring to needing clarification on? https://github.com/fleetdm/fleet/blob/main/docs/Deploying/Configuration.md#kafka-rest-proxy-logging

Sunil

01/30/2023, 3:44 PM

Yes, you are right. I am able to run the live queries (process_events and socket_events) thro' fleetdm web console. Looking for help how could I configure the kafkarest to get these audit events posted on kafkarest topic. I had gone thro' the link you have provided, but however I need configuration to create the docker for kafkarest and create the topic etc...

Rachel Perkins

01/30/2023, 4:53 PM

I'm not quite sure what you have and have not done. Do you have a Kafkarest proxy setup (sans Fleet)? What does your Fleet configuration file look like so far?

Sunil

01/30/2023, 5:40 PM

I have installed the OSQuery on endpoint and fleet/mysql/redis running inside the docker and I am able to do live and schedule queries to fetch process_events and socket_events.

Sunil

01/30/2023, 5:41 PM

I am using docker-compose.yml file to run the fleet server.

Rachel Perkins

01/30/2023, 6:18 PM

But you have a kafkarest proxyhost setup?

Rachel Perkins

01/30/2023, 6:19 PM

You can run the fleet server with flags or environment variables pointing to

kafkarest

but you need a kafkarest proxyhost setup first to point to as well

Sunil

01/30/2023, 6:41 PM

Yes, I am using this yml configuration to run kafka and kafkarest. https://gist.github.com/ayubmalik/9300bbd8f62ce401c4d1f69eae0cb5d7

Sunil

01/30/2023, 6:44 PM

environment: &fleet_environment - FLEET_MYSQL_ADDRESS=mysql:3306 - FLEET_MYSQL_DATABASE=fleet - FLEET_MYSQL_USERNAME=root - FLEET_MYSQL_PASSWORD=xxxxx - FLEET_REDIS_ADDRESS=redis:6379 - FLEET_SERVER_CERT=/fleet/server.cert - FLEET_SERVER_KEY=/fleet/server.pem - FLEET_OSQUERY_POLICY_UPDATE_INTERVAL=30s - FLEET_ACTIVITY_AUDIT_LOG_PLUGIN=kafkarest - FLEET_KAFKAREST_PROXYHOST=http://localhost:38082 - FLEET_KAFKAREST_AUDIT_TOPIC=fleet_audit - FLEET_KAFKAREST_TIMEOUT=5 - FLEET_KAFKAREST_CONTENT_TYPE_VALUE=application/vnd.kafka.json.v2+json - FLEET_ACTIVITY_ENABLE_AUDIT_LOG=true - FLEET_LOGGING_JSON=true

Sunil

01/30/2023, 6:44 PM

Above is my fleet environment config.

Rachel Perkins

01/30/2023, 8:21 PM

Try https://?

Kathy Satterlee

01/30/2023, 10:28 PM

Hey @Sunil! Have you added the services from https://gist.github.com/ayubmalik/9300bbd8f62ce401c4d1f69eae0cb5d7 to your existing docker-compose file, or is that running separately from the Fleet setup?

Kathy Satterlee

01/30/2023, 11:04 PM

In addition to networking concerns (depending on the answer to the previous question), you'll also need to set the osquery result log plugin: https://fleetdm.com/docs/deploying/configuration#osquery-result-log-plugin Then, you can start running scheduled queries on those tables and your results should start flowing in to Kafka

Kathy Satterlee

01/30/2023, 11:11 PM

One more setting for Fleet: https://fleetdm.com/docs/deploying/configuration#kafkarest-result-topic

Kathy Satterlee

01/30/2023, 11:17 PM

One note I wanted to add about the configuration:

Copy code

- FLEET_KAFKAREST_AUDIT_TOPIC=fleet_audit
- FLEET_ACTIVITY_AUDIT_LOG_PLUGIN=kafkarest

Those two values are related to admin activities in Fleet. Wanted to make sure that was clear given that we're also talking about process auditing using the process_events and socket_events tables.

Sunil

01/31/2023, 10:35 AM

@Kathy Satterlee, it was in 2 different yml files but trying to combine them in one yml file.

Kathy Satterlee

01/31/2023, 2:45 PM

Once you get them combined, update the Fleet environment with the additional variables (and update the url for Kafka) and you should be golden. I’ll keep an eye out!

Rachel Perkins

01/31/2023, 3:23 PM

Thanks for hopping in Kathy! 🙌🏽

Sunil

01/31/2023, 7:04 PM

Just wondering if this 'FLEET_OSQUERY_RESULT_LOG_PLUGIN=filesystem' right ?

Kathy Satterlee

01/31/2023, 7:30 PM

Here are all of the environmental variables you need to send your scheduled query results to Kafka:

Copy code

## Connection settings for Kafka Rest Proxy 
##This assumes that your services are running on the same network in Docker

- FLEET_KAFKAREST_PROXYHOST=kafka-rest:38082 
- FLEET_KAFKAREST_TIMEOUT=5
- FLEET_KAFKAREST_CONTENT_TYPE_VALUE=application/vnd.kafka.json.v2+json

## Settings to sent osquery result logs for scheduled queries to Kafka:

- FLEET_OSQUERY_RESULT_LOG_PLUGIN=kafkarest
- FLEET_KAFKAREST_RESULT_TOPIC=fleet_result

## Settings to also send the activity stream to Kafka

- FLEET_ACTIVITY_ENABLE_AUDIT_LOG=true
- FLEET_ACTIVITY_AUDIT_LOG_PLUGIN=kafkarest
- FLEET_KAFKAREST_AUDIT_TOPIC=fleet_audit

Kathy Satterlee

01/31/2023, 7:33 PM

Your hosts also need to be configured to send results logs to Fleet:

Copy code

--logger_plugin=tls 
 --logger_tls_endpoint=/api/v1/osquery/log 
 --logger_tls_period=10

If you're using Fleet's installer packages, those are the default settings.

Sunil

02/01/2023, 6:19 AM

Yes, my settings are almost same, just added this new one you mentioned above , FLEET_KAFKAREST_RESULT_TOPIC=fleet_result

Sunil

02/01/2023, 6:20 AM

I have one docker-compose.yml in which defined all the services so I am assuming they are in same network. I am not sure if I have to explicitly define network as below ? (just as an example)

Sunil

02/01/2023, 6:21 AM

version: '3.5' networks: default: name: kafka-net

Sunil

02/01/2023, 6:25 AM

BTW I have installed OSQuery 5.7 package on endpoint machine and I have almost same settings you mentioned above.

Sunil

02/01/2023, 6:25 AM

Copy code

--logger_plugin=tls,filesystem 
 --logger_tls_endpoint=/api/v1/osquery/log 
 --logger_tls_period=10

Sunil

02/01/2023, 7:46 AM

I am getting the following error:

Sunil

02/01/2023, 7:46 AM

fleetdm-fleet-1 | Failed to start: initializing osqueryd result logging: create kafka rest result logger: kafka rest topic check: Get "kafka-rest:38082/topics/?topic=fleet_result": unsupported protocol scheme "kafka-rest"

Sunil

02/01/2023, 7:47 AM

Not sure what version of kafka, kafka-rest, zookeeper and schema-registry is suitable for fleet ?

Kathy Satterlee

02/01/2023, 3:26 PM

Sorry about that! Networking in Docker is weird and I’m still figuring it out :). Try

<http://kafka-rest:38082>

Sunil

02/01/2023, 3:44 PM

this is part of cmd output 'docker-compose logs fleet'

Sunil

02/01/2023, 3:45 PM

On a host machine where I am running these docker containers, http://localhost:38082/topics works.

Kathy Satterlee

02/01/2023, 3:47 PM

What have you got set for the proxyhost address in the Fleet env currently?

Sunil

02/01/2023, 3:47 PM

On a host machine even http://localhost:38082/topics/fleet_result also works.

Sunil

02/01/2023, 3:48 PM

- FLEET_KAFKAREST_PROXYHOST=kafka-rest:38082

Kathy Satterlee

02/01/2023, 4:28 PM

That’s where the change needs to happen. Try adding the

http://

there.

Sunil

02/01/2023, 4:59 PM

I tried http:// but now I have slightly different error, 'fleetdm-fleet-1 | Failed to start: initializing osqueryd result logging: create kafka rest result logger: kafka rest topic check: Get "http://kafka-rest:38082/topics/?topic=fleet_result": dial tcp 172.18.0.838082 connect: connection refused'

Kathy Satterlee

02/01/2023, 5:09 PM

Just for giggles, try

<https://localhost:38082>

instead. If that doesn’t do the trick, I’ll reach out for some backup :)

Sunil

02/01/2023, 5:12 PM

fleetdm-fleet-1 | Failed to start: initializing osqueryd result logging: create kafka rest result logger: kafka rest topic check: Get "https://localhost:38082/topics/?topic=fleet_result": dial tcp 127.0.0.138082 connect: connection refused

Kathy Satterlee

02/01/2023, 5:25 PM

And what happens if you try to

curl

that url?

<https://localhost:38082/topics/?topic=fleet_result>

*Just noticed that I instinctively added the

. Try with just

http

as well.

Kathy Satterlee

02/01/2023, 5:26 PM

This is likely a configuration issue on the Kafka side, but I want to rule out any issues with the Fleet configuration

Sunil

02/01/2023, 6:10 PM

'curl http://localhost:38082/topics' this works as '["__confluent.support.metrics","_schemas","fleet_audit","fleet_result","jsontest"]'

Sunil

02/01/2023, 6:11 PM

Where as 'curl http://localhost:38082/topics/?topic=fleet_result' this gives me an error, 'zsh: no matches found: http://localhost:38082/topics/?topic=fleet_result'

Sunil

02/01/2023, 6:15 PM

'curl https://localhost:38082/topics/?topic=fleet_result' gives me same error 'zsh: no matches found: https://localhost:38082/topics/?topic=fleet_result'

Sunil

02/01/2023, 6:16 PM

http or https I am getting same results.

Kathy Satterlee

02/01/2023, 6:26 PM

It definitely looks like there is some additional configuration that may need to happen on the Kafka side. I'll reach out to the team to see if anyone has experience with that, but no promises.

Sunil

02/03/2023, 6:21 AM

My configuration looks good to some extent, there seems to be some race condition since if I launch fleet container later (with some delay) where kafka/zookeeper/kafka-rest all are initialised and ready then things are working better than before. I am able to see schedule queries data on kafka topic 'fleet_result' however still struggling to get the audit data on 'fleet_audit' topic although it is populating process_events and socket_events tables at endpoint (osquery agent side).

Kathy Satterlee

02/03/2023, 6:31 PM

Just to clarify there,

fleet_audit

is for admin activity in Fleet (logins, running/saving/modifying queries, etc). Next step will be tot set up scheduled queries using those tables, then the results will come through with the rest of your scheduled queries.

Kathy Satterlee

02/03/2023, 6:37 PM

https://fleetdm.com/docs/using-fleet/audit-activities#audit-activities

Open in Slack

Previous Next