Title
#general
j

Jason NG

08/04/2021, 3:20 AM
Does anybody have experience with connecting osquery to amazon MSK via the kafka producer logger plugin? I am facing an error, [2021-08-04 01:40:35,823] WARN [Consumer clientId=consumer-console-consumer-41657-1, groupId=console-consumer-41657] Connection to node -1 (<name>.kafka.ap-southeast-1.amazonaws.com/<ip>:9092) terminated during authentication. This may happen due to any of the following reasons: (1) Authentication failed due to invalid credentials with brokers older than 1.0.0, (2) Firewall blocking Kafka TLS traffic (eg it may only allow HTTPS traffic), (3) Transient network issue. (org.apache.kafka.clients.NetworkClient) [2021-08-04 01:40:35,823] WARN [Consumer clientId=consumer-console-consumer-41657-1, groupId=console-consumer-41657] Bootstrap broker <name>.kafka.ap-southeast-1.amazonaws.com:9092 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient) this happens when i try to connect using the non tls broker. When i use the tls broker which listens on port 9094, nothing happens. I dont think its a firewall issue since i opened all ports. my flagfile looks like this:
# Server
--tls_hostname=<http://osquery-demo.com:8080|osquery-demo.com:8080>
--tls_server_certs=/etc/osquery/fleet.pem

# Enrollment
--host_identifier=instance
--enroll_secret_path=/etc/osquery/secret.txt
--enroll_tls_endpoint=/api/v1/osquery/enroll

# Configuration
--config_plugin=tls
--config_tls_endpoint=/api/v1/osquery/config
--config_refresh=10

# Live query
--disable_distributed=false
--distributed_plugin=tls
--distributed_interval=10
--distributed_tls_max_attempts=3
--distributed_tls_read_endpoint=/api/v1/osquery/distributed/read
--distributed_tls_write_endpoint=/api/v1/osquery/distributed/write

# Logging
--logger_plugin=kafka_producer
--logger_kafka_brokers=<name>.<http://kafka.ap-southeast-1.amazonaws.com:9094,<name>.kafka.ap-southeast-1.amazonaws.com:9094|kafka.ap-southeast-1.amazonaws.com:9094,<name>.kafka.ap-southeast-1.amazonaws.com:9094>
--logger_kafka_topic=AWSKafkaTutorialTopic
--logger_kafka_acks=0
--logger_tls_endpoint=/api/v1/osquery/log
--logger_tls_period=10

# File carving
--disable_carver=false
--carver_start_endpoint=/api/v1/osquery/carve/begin
--carver_continue_endpoint=/api/v1/osquery/carve/block
--carver_block_size=2000000
theopolis

theopolis

08/07/2021, 2:26 AM
I wish I could help out but I've never used Kafka before 😢