How are folks who use GKE as a deployment option shipping osquery_result logs? Since GKE doesn’t support readWriteMany with GCP Persistent disk you can’t share a PV with a log shipper pod. I’m investigating using a sidecar or using the Azurefile storageclass (on GKE yeah it’s messy) but wondered if there are other ways to do it?

Could you use the Pubsub logger plugin and handle logs from there?

yep I’ll try that, thanks!

Here is a pretty vanilla sidecar example used on GKE for a couple K hosts

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: fleet-webserver
  labels:
    app: fleet-webserver
spec:
  replicas: 3
  selector:
    matchLabels:
      app: fleet-webserver
  template:
    metadata:
      labels:
        app: fleet-webserver
    spec:
      volumes:
        - name: fleet-tls
          secret:
            secretName: fleet-tls
        - name: log-storage
          emptyDir: {}
        - name: config
          configMap:
            name: fleet-filebeat-config
      containers:
        - name: fleet-webserver
          image: fleetdm/fleet:3.6.0
          command: [fleet, serve]
          ports:
            - containerPort: 8080
          volumeMounts:
            - name: fleet-tls
              mountPath: /secrets/fleet-tls
              readOnly: true
            - name: log-storage
              mountPath: /var/log/
          env:
            - name: KOLIDE_MYSQL_ADDRESS
              valueFrom:
                secretKeyRef:
                  name: fleet-mysql
                  key: address
            - name: KOLIDE_MYSQL_DATABASE
              valueFrom:
                secretKeyRef:
                  name: fleet-mysql
                  key: database
            - name: KOLIDE_MYSQL_USERNAME
              valueFrom:
                secretKeyRef:
                  name: fleet-mysql
                  key: username
            - name: KOLIDE_MYSQL_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: fleet-mysql
                  key: password
            - name: KOLIDE_REDIS_ADDRESS
              value: 10.0.0.1:6379
            - name: KOLIDE_AUTH_JWT_KEY
              valueFrom:
                secretKeyRef:
                  name: fleet-server-auth-key
                  key: fleet-server-auth-key
            - name: KOLIDE_SERVER_ADDRESS
              value: 0.0.0.0:8080
            - name: KOLIDE_SERVER_CERT
              value: /secrets/fleet-tls/tls.crt
            - name: KOLIDE_SERVER_KEY
              value: /secrets/fleet-tls/tls.key
            - name: KOLIDE_LOGGING_JSON
              value: 'true'
            - name: KOLIDE_OSQUERY_STATUS_LOG_PLUGIN
              value: filesystem
            - name: KOLIDE_FILESYSTEM_STATUS_LOG_FILE
              value: /var/log/osqueryd.status.log
            - name: KOLIDE_FILESYSTEM_RESULT_LOG_FILE
              value: /var/log/osqueryd.results.log
          readinessProbe:
            httpGet:
              scheme: HTTPS
              path: /healthz
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 10
        - name: filebeat
          volumeMounts:
            - name: config
              mountPath: /usr/share/filebeat/filebeat.yml
              subPath: filebeat.yml
            - name: log-storage
              mountPath: /var/log/
          image: elastic/filebeat:7.9.0
          securityContext:
            runAsUser: 0
          resources:
            limits:
              memory: 200Mi
            requests:
              cpu: 100m
              memory: 100Mi

yeah I was running filebeat in it’s own pod and ran into the readWriteMany limitation in GKE PV. I guess if is a sidecar / container then I avoid that limitation

Yeah this approach uses a tempFS so you may lose logs which have not been read by the sidecar and shipped if the pod is killed etc but that’s really something you need to weigh up. I don’t like the approach of using a on node FS for persistence. Note if you’re running a recent version of Fleet which has Stdout logging enabled you can avoid a sidecar and do a filebeat daemonset which is nice if you’re also collecting other logs.

I got it working via pubsub and it’s working nicely except I can’t use the built-in osquery module from filebeat anymore. This is easily fixed by using an ingest processor like :

[
  {
    "json": {
      "field": "message",
      "target_field": "osquery",
      "ignore_failure": true
    }
  }
]