Hey folks, I’m trying to setup fleet’s kafka plugin. We are using the strimzi operator for Kafka und their implementation of Kafka REST API (Kafka Bridge). Fleet’s docs mention the confluentic rest api for Kafka, but AFAIK (I’m not well versed here at all), they have the same basic endpoints. To the specific problem: Having configured fleet to send osquery status logs to Bridge (Kafka rest), I get the following error:

Failed to start: initializing osqueryd status logging: create kafka rest status logger: kafka rest topic check: Get "bridge-main-bridge-service.observability-main.svc:8080/topics/?topic=osquery-status-logs": unsupported protocol scheme "bridge-main-bridge-service.observability-main.svc"

What bothers me here is this url:

bridge-main-bridge-service.observability-main.svc:8080/topics/?topic=osquery-status-logs

To get the details of this tops, fleet needs to make a get on the following url:

bridge-main-bridge-service.observability-main.svc:8080/topics/osquery-status-logs

This is also the case with the confluentic rest api both v2 and v3: https://docs.confluent.io/platform/current/kafka-rest/api.html#get--topics-(string-topic_name) https://docs.confluent.io/platform/current/kafka-rest/api.html#get--clusters-cluster_id-topics-topic_name My kafka rest api works fine if I use the v2 scheme:

curl -i bridge-main-bridge-service.observability-main.svc:8080/topics/osquery-status-logs
HTTP/1.1 200 OK
content-type: application/vnd.kafka.v2+json
content-length: 1003

{"name":"osquery-status-logs","configs":{"compression.type":"gzip","leader.replication.throttled.replicas":"","message.downconversion.enable":"true","min.insync.replicas":"1","<http://segment.jitter.ms|segment.jitter.ms>":"0","cleanup.policy":"delete","<http://flush.ms|flush.ms>":"9223372036854775807","follower.replication.throttled.replicas":"","segment.bytes":"1073741824","<http://retention.ms|retention.ms>":"7200000","flush.messages":"9223372036854775807","message.format.version":"3.0-IV1","<http://max.compaction.lag.ms|max.compaction.lag.ms>":"9223372036854775807","<http://file.delete.delay.ms|file.delete.delay.ms>":"60000","max.message.bytes":"1048588","<http://min.compaction.lag.ms|min.compaction.lag.ms>":"0","message.timestamp.type":"CreateTime","preallocate":"false","min.cleanable.dirty.ratio":"0.5","index.interval.bytes":"4096","unclean.leader.election.enable":"false","retention.bytes":"-1","<http://delete.retention.ms|delete.retention.ms>":"86400000","<http://segment.ms|segment.ms>":"604800000","<http://message.timestamp.difference.max.ms|message.timestamp.difference.max.ms>":"9223372036854775807","segment.index.bytes":"10485760"},"partitions":[{"partition":0,"leader":1,"replicas":[{"broker":1,"leader":true,"in_sync":true}]}]}

Of course, if I do a curl with the url fleet is using (using old style key value pairs), kafka rest doesn’t recognize the parameters and just gives me back the topic names:

curl bridge-main-bridge-service.observability-main.svc:8080/topics/?topic=osquery-status-logs
["topic-1","topic-2","topic-3","osquery-status-logs","some-other-topic"]

And throws the unsupported protocol scheme error. I’m not sure if this is intended or a bug on fleet’s side. Really appreciate any input.

Hi @Ali! It sounds like that version of the Kafka Proxy is set up a little differently than the Confluent proxy, which is what the

kafka-rest

plugin is designed for. You may need to set up something to transform the request to match the expected format, or look in to any settings in the Proxy that would allow the request to come through as formatted. You’re always welcome to submit a feature request or PR to add support for additional plugins or expand the existing ones!

I have already opened an issue here: https://github.com/fleetdm/fleet/issues/17266 What bothers me is this url scheme

topics/?topic=

that’s currently used by fleet’s rest client. Nowhere in confluentic rest proxy docs (v2 and v3) is any mention of this: https://docs.confluent.io/platform/current/kafka-rest/api.html#get--topics-(string-topic_name) https://docs.confluent.io/platform/current/kafka-rest/api.html#get--clusters-cluster_id-topics-topic_name I imagine that this is a deprecated scheme that is kept for backwards compatibility but isn’t documented? Anyway, I guess that changing this line https://github.com/fleetdm/fleet/blob/f248701e0f83627e66e7f3902e8365ab5fc8637e/server/logging/kafkarest.go#L20 would solve the problem and AFAIK wouldn’t have any negative effects for the confluentic rest proxy. What I mean is this: As far as I can tell, the strimzi rest api and confluentic one aren’t different in this specific case (getting details about a topic). Problem is that Fleet is using a url that is probably deprecated and is also undocumented and bumping it would probably help everyone.