https://github.com/osquery/osquery logo
Powered by
Title
s

Sunil

01/30/2023, 3:07 PM
How could one collect process_events and socket_events from OSQuery to fleetdm (fleet) and then put it in kafkarest ? I had already gone thro' fleetdm documentation for kafkarest but not really understood the whole setup.
r

Rachel Perkins

01/30/2023, 3:14 PM
Hi Sunil! Were you able to create and run queries on those tables successfully, but just need help with configuring to Kafka rest?
Is this the documentation you were referring to needing clarification on? https://github.com/fleetdm/fleet/blob/main/docs/Deploying/Configuration.md#kafka-rest-proxy-logging
s

Sunil

01/30/2023, 3:44 PM
Yes, you are right. I am able to run the live queries (process_events and socket_events) thro' fleetdm web console. Looking for help how could I configure the kafkarest to get these audit events posted on kafkarest topic. I had gone thro' the link you have provided, but however I need configuration to create the docker for kafkarest and create the topic etc...
r

Rachel Perkins

01/30/2023, 4:53 PM
I'm not quite sure what you have and have not done. Do you have a Kafkarest proxy setup (sans Fleet)? What does your Fleet configuration file look like so far?
s

Sunil

01/30/2023, 5:40 PM
I have installed the OSQuery on endpoint and fleet/mysql/redis running inside the docker and I am able to do live and schedule queries to fetch process_events and socket_events.
I am using docker-compose.yml file to run the fleet server.
r

Rachel Perkins

01/30/2023, 6:18 PM
But you have a kafkarest proxyhost setup?
You can run the fleet server with flags or environment variables pointing to 
kafkarest
but you need a kafkarest proxyhost setup first to point to as well
s

Sunil

01/30/2023, 6:41 PM
Yes, I am using this yml configuration to run kafka and kafkarest. https://gist.github.com/ayubmalik/9300bbd8f62ce401c4d1f69eae0cb5d7
environment: &fleet_environment - FLEET_MYSQL_ADDRESS=mysql:3306 - FLEET_MYSQL_DATABASE=fleet - FLEET_MYSQL_USERNAME=root - FLEET_MYSQL_PASSWORD=xxxxx - FLEET_REDIS_ADDRESS=redis:6379 - FLEET_SERVER_CERT=/fleet/server.cert - FLEET_SERVER_KEY=/fleet/server.pem - FLEET_OSQUERY_POLICY_UPDATE_INTERVAL=30s - FLEET_ACTIVITY_AUDIT_LOG_PLUGIN=kafkarest - FLEET_KAFKAREST_PROXYHOST=http://localhost:38082 - FLEET_KAFKAREST_AUDIT_TOPIC=fleet_audit - FLEET_KAFKAREST_TIMEOUT=5 - FLEET_KAFKAREST_CONTENT_TYPE_VALUE=application/vnd.kafka.json.v2+json - FLEET_ACTIVITY_ENABLE_AUDIT_LOG=true - FLEET_LOGGING_JSON=true
Above is my fleet environment config.
r

Rachel Perkins

01/30/2023, 8:21 PM
Try https://?
k

Kathy Satterlee

01/30/2023, 10:28 PM
Hey @Sunil! Have you added the services from https://gist.github.com/ayubmalik/9300bbd8f62ce401c4d1f69eae0cb5d7 to your existing docker-compose file, or is that running separately from the Fleet setup?
In addition to networking concerns (depending on the answer to the previous question), you'll also need to set the osquery result log plugin: https://fleetdm.com/docs/deploying/configuration#osquery-result-log-plugin Then, you can start running scheduled queries on those tables and your results should start flowing in to Kafka
One more setting for Fleet: https://fleetdm.com/docs/deploying/configuration#kafkarest-result-topic
One note I wanted to add about the configuration: 
- FLEET_KAFKAREST_AUDIT_TOPIC=fleet_audit
- FLEET_ACTIVITY_AUDIT_LOG_PLUGIN=kafkarest
Those two values are related to admin activities in Fleet. Wanted to make sure that was clear given that we're also talking about process auditing using the process_events and socket_events tables.
s

Sunil

01/31/2023, 10:35 AM
@Kathy Satterlee, it was in 2 different yml files but trying to combine them in one yml file.
k

Kathy Satterlee

01/31/2023, 2:45 PM
Once you get them combined, update the Fleet environment with the additional variables (and update the url for Kafka) and you should be golden. I’ll keep an eye out!
r

Rachel Perkins

01/31/2023, 3:23 PM
Thanks for hopping in Kathy! 🙌🏽
s

Sunil

01/31/2023, 7:04 PM
Just wondering if this 'FLEET_OSQUERY_RESULT_LOG_PLUGIN=filesystem' right ?
k

Kathy Satterlee

01/31/2023, 7:30 PM
Here are all of the environmental variables you need to send your scheduled query results to Kafka: 
## Connection settings for Kafka Rest Proxy 
##This assumes that your services are running on the same network in Docker

- FLEET_KAFKAREST_PROXYHOST=kafka-rest:38082 
- FLEET_KAFKAREST_TIMEOUT=5
- FLEET_KAFKAREST_CONTENT_TYPE_VALUE=application/vnd.kafka.json.v2+json

## Settings to sent osquery result logs for scheduled queries to Kafka:

- FLEET_OSQUERY_RESULT_LOG_PLUGIN=kafkarest
- FLEET_KAFKAREST_RESULT_TOPIC=fleet_result

## Settings to also send the activity stream to Kafka

- FLEET_ACTIVITY_ENABLE_AUDIT_LOG=true
- FLEET_ACTIVITY_AUDIT_LOG_PLUGIN=kafkarest
- FLEET_KAFKAREST_AUDIT_TOPIC=fleet_audit
Your hosts also need to be configured to send results logs to Fleet: 
--logger_plugin=tls 
 --logger_tls_endpoint=/api/v1/osquery/log 
 --logger_tls_period=10
If you're using Fleet's installer packages, those are the default settings.
s

Sunil

02/01/2023, 6:19 AM
Yes, my settings are almost same, just added this new one you mentioned above , FLEET_KAFKAREST_RESULT_TOPIC=fleet_result
I have one docker-compose.yml in which defined all the services so I am assuming they are in same network. I am not sure if I have to explicitly define network as below ? (just as an example)
version: '3.5' networks: default: name: kafka-net
BTW I have installed OSQuery 5.7 package on endpoint machine and I have almost same settings you mentioned above.
--logger_plugin=tls,filesystem 
 --logger_tls_endpoint=/api/v1/osquery/log 
 --logger_tls_period=10
I am getting the following error:
fleetdm-fleet-1 | Failed to start: initializing osqueryd result logging: create kafka rest result logger: kafka rest topic check: Get "kafka-rest:38082/topics/?topic=fleet_result": unsupported protocol scheme "kafka-rest"
Not sure what version of kafka, kafka-rest, zookeeper and schema-registry is suitable for fleet ?
k

Kathy Satterlee

02/01/2023, 3:26 PM
Sorry about that! Networking in Docker is weird and I’m still figuring it out :). Try 
<http://kafka-rest:38082>
s

Sunil

02/01/2023, 3:44 PM
this is part of cmd output 'docker-compose logs fleet'
On a host machine where I am running these docker containers, http://localhost:38082/topics works.
k

Kathy Satterlee

02/01/2023, 3:47 PM
What have you got set for the proxyhost address in the Fleet env currently?
s

Sunil

02/01/2023, 3:47 PM
On a host machine even http://localhost:38082/topics/fleet_result also works.
- FLEET_KAFKAREST_PROXYHOST=kafka-rest:38082
k

Kathy Satterlee

02/01/2023, 4:28 PM
That’s where the change needs to happen. Try adding the 
http://
there.
s

Sunil

02/01/2023, 4:59 PM
I tried http:// but now I have slightly different error, 'fleetdm-fleet-1 | Failed to start: initializing osqueryd result logging: create kafka rest result logger: kafka rest topic check: Get "http://kafka-rest:38082/topics/?topic=fleet_result": dial tcp 172.18.0.8:38082: connect: connection refused'
k

Kathy Satterlee

02/01/2023, 5:09 PM
Just for giggles, try 
<https://localhost:38082>
instead. If that doesn’t do the trick, I’ll reach out for some backup :)
s

Sunil

02/01/2023, 5:12 PM
fleetdm-fleet-1 | Failed to start: initializing osqueryd result logging: create kafka rest result logger: kafka rest topic check: Get "https://localhost:38082/topics/?topic=fleet_result": dial tcp 127.0.0.1:38082: connect: connection refused
k

Kathy Satterlee

02/01/2023, 5:25 PM
And what happens if you try to 
curl
that url? 
<https://localhost:38082/topics/?topic=fleet_result>
*Just noticed that I instinctively added the 
s
. Try with just 
http
as well.
This is likely a configuration issue on the Kafka side, but I want to rule out any issues with the Fleet configuration
s

Sunil

02/01/2023, 6:10 PM
'curl http://localhost:38082/topics' this works as '["__confluent.support.metrics","_schemas","fleet_audit","fleet_result","jsontest"]'
Where as 'curl http://localhost:38082/topics/?topic=fleet_result' this gives me an error, 'zsh: no matches found: http://localhost:38082/topics/?topic=fleet_result'
'curl https://localhost:38082/topics/?topic=fleet_result' gives me same error 'zsh: no matches found: https://localhost:38082/topics/?topic=fleet_result'
http or https I am getting same results.
k

Kathy Satterlee

02/01/2023, 6:26 PM
It definitely looks like there is some additional configuration that may need to happen on the Kafka side. I'll reach out to the team to see if anyone has experience with that, but no promises.
s

Sunil

02/03/2023, 6:21 AM
My configuration looks good to some extent, there seems to be some race condition since if I launch fleet container later (with some delay) where kafka/zookeeper/kafka-rest all are initialised and ready then things are working better than before. I am able to see schedule queries data on kafka topic 'fleet_result' however still struggling to get the audit data on 'fleet_audit' topic although it is populating process_events and socket_events tables at endpoint (osquery agent side).
k

Kathy Satterlee

02/03/2023, 6:31 PM
Just to clarify there, 
fleet_audit
is for admin activity in Fleet (logins, running/saving/modifying queries, etc). Next step will be tot set up scheduled queries using those tables, then the results will come through with the rest of your scheduled queries.
https://fleetdm.com/docs/using-fleet/audit-activities#audit-activities
#fleetJoin Slack
Powered by