Title
#fleet
hilt

hilt

01/19/2021, 1:08 PM
How are folks who use GKE as a deployment option shipping osquery_result logs? Since GKE doesn’t support readWriteMany with GCP Persistent disk you can’t share a PV with a log shipper pod. I’m investigating using a sidecar or using the Azurefile storageclass (on GKE yeah it’s messy) but wondered if there are other ways to do it?
zwass

zwass

01/19/2021, 3:42 PM
Could you use the Pubsub logger plugin and handle logs from there?
hilt

hilt

01/21/2021, 1:05 AM
yep I’ll try that, thanks!
Gavin

Gavin

01/21/2021, 1:08 AM
Here is a pretty vanilla sidecar example used on GKE for a couple K hosts
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: fleet-webserver
  labels:
    app: fleet-webserver
spec:
  replicas: 3
  selector:
    matchLabels:
      app: fleet-webserver
  template:
    metadata:
      labels:
        app: fleet-webserver
    spec:
      volumes:
        - name: fleet-tls
          secret:
            secretName: fleet-tls
        - name: log-storage
          emptyDir: {}
        - name: config
          configMap:
            name: fleet-filebeat-config
      containers:
        - name: fleet-webserver
          image: fleetdm/fleet:3.6.0
          command: [fleet, serve]
          ports:
            - containerPort: 8080
          volumeMounts:
            - name: fleet-tls
              mountPath: /secrets/fleet-tls
              readOnly: true
            - name: log-storage
              mountPath: /var/log/
          env:
            - name: KOLIDE_MYSQL_ADDRESS
              valueFrom:
                secretKeyRef:
                  name: fleet-mysql
                  key: address
            - name: KOLIDE_MYSQL_DATABASE
              valueFrom:
                secretKeyRef:
                  name: fleet-mysql
                  key: database
            - name: KOLIDE_MYSQL_USERNAME
              valueFrom:
                secretKeyRef:
                  name: fleet-mysql
                  key: username
            - name: KOLIDE_MYSQL_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: fleet-mysql
                  key: password
            - name: KOLIDE_REDIS_ADDRESS
              value: 10.0.0.1:6379
            - name: KOLIDE_AUTH_JWT_KEY
              valueFrom:
                secretKeyRef:
                  name: fleet-server-auth-key
                  key: fleet-server-auth-key
            - name: KOLIDE_SERVER_ADDRESS
              value: 0.0.0.0:8080
            - name: KOLIDE_SERVER_CERT
              value: /secrets/fleet-tls/tls.crt
            - name: KOLIDE_SERVER_KEY
              value: /secrets/fleet-tls/tls.key
            - name: KOLIDE_LOGGING_JSON
              value: 'true'
            - name: KOLIDE_OSQUERY_STATUS_LOG_PLUGIN
              value: filesystem
            - name: KOLIDE_FILESYSTEM_STATUS_LOG_FILE
              value: /var/log/osqueryd.status.log
            - name: KOLIDE_FILESYSTEM_RESULT_LOG_FILE
              value: /var/log/osqueryd.results.log
          readinessProbe:
            httpGet:
              scheme: HTTPS
              path: /healthz
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 10
        - name: filebeat
          volumeMounts:
            - name: config
              mountPath: /usr/share/filebeat/filebeat.yml
              subPath: filebeat.yml
            - name: log-storage
              mountPath: /var/log/
          image: elastic/filebeat:7.9.0
          securityContext:
            runAsUser: 0
          resources:
            limits:
              memory: 200Mi
            requests:
              cpu: 100m
              memory: 100Mi
1:09 AM
---
filebeat.inputs:
  - type: log
    enabled: true
    paths:
      - /var/log/osqueryd.results.log
    json.keys_under_root: true
    fields_under_root: true
    publisher_pipeline.disable_host: true

filebeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml
  # Set to true to enable config reloading
  reload.enabled: false
  # Period on which files under path should be checked for changes
  # reload.period: 10s

output.logstash:
  enabled: true
  hosts:
    - logstash-ilb.elastic-system.svc:10105
  ssl.enabled: false
hilt

hilt

01/21/2021, 1:13 AM
yeah I was running filebeat in it’s own pod and ran into the readWriteMany limitation in GKE PV. I guess if is a sidecar / container then I avoid that limitation
Gavin

Gavin

01/21/2021, 1:16 AM
Yeah this approach uses a tempFS so you may lose logs which have not been read by the sidecar and shipped if the pod is killed etc but that’s really something you need to weigh up. I don’t like the approach of using a on node FS for persistence. Note if you’re running a recent version of Fleet which has Stdout logging enabled you can avoid a sidecar and do a filebeat daemonset which is nice if you’re also collecting other logs.
hilt

hilt

01/21/2021, 6:21 AM
I got it working via pubsub and it’s working nicely except I can’t use the built-in osquery module from filebeat anymore. This is easily fixed by using an ingest processor like :
[
  {
    "json": {
      "field": "message",
      "target_field": "osquery",
      "ignore_failure": true
    }
  }
]