Hi I am trying to send log files to an external file at flee osquery #fleet

Hi, I am trying to send log files to an external f...

Ibra

08/08/2023, 11:29 AM

Hi, I am trying to send log files to an external file at fleet, but when I enter the lines below, I get the following error: FLEET_LOGGING_JSON: "true" FLEET_OSQUERY_STATUS_LOG_PLUGIN: filesystem FLEET_FILESYSTEM_STATUS_LOG_FILE: /var/log/osqueryd.status.log FLEET_OSQUERY_RESULT_LOG_PLUGIN: filesystem FLEET_FILESYSTEM_RESULT_LOG_FILE: /var/log/osqueryd.results.log FLEET_OSQUERY_LABEL_UPDATE_INTERVAL: 1m error: {"component":"redis","level":"info","mode":"standalone","ts":"2023-08-08T112221.115897319Z"} Failed to start: initializing osqueryd status logging: create filesystem status logger: perm check: open /var/log/osqueryd.status.log: permission denied right now fleet logs can only be seen by doing docker logs Fleet, how do I send the above and audit logs to an external file, via my docker-compose.yml? also, is it possible to manage the rotation of the log files or do I have to use an external agent that works on the files slaved on the server? thanks

Rachel Perkins

08/08/2023, 2:11 PM

Hi Ibra, are you configuring your logs using this documentation?

Ibra

08/08/2023, 2:19 PM

yes @Rachel Perkins but i have the same error

Ibra

08/08/2023, 3:14 PM

$ docker logs -f Fleet Migrations already completed. Nothing to do. {"component":"redis","level":"info","mode":"standalone","ts":"2023-08-08T151319.162251913Z"} Failed to start: initializing osqueryd status logging: create filesystem status logger: perm check: open /var/log/fleet/osqueryd.status.log: permission denied Migrations already completed. Nothing to do. {"component":"redis","level":"info","mode":"standalone","ts":"2023-08-08T151319.976867671Z"} Failed to start: initializing osqueryd status logging: create filesystem status logger: perm check: open /var/log/fleet/osqueryd.status.log: permission denied Migrations already completed. Nothing to do. {"component":"redis","level":"info","mode":"standalone","ts":"2023-08-08T151320.890729979Z"} Failed to start: initializing osqueryd status logging: create filesystem status logger: perm check: open /var/log/fleet/osqueryd.status.log: permission denied Migrations already completed. Nothing to do. {"component":"redis","level":"info","mode":"standalone","ts":"2023-08-08T151322.014817415Z"} Failed to start: initializing osqueryd status logging: create filesystem status logger: perm check: open /var/log/fleet/osqueryd.status.log: permission denied Migrations already completed. Nothing to do. {"component":"redis","level":"info","mode":"standalone","ts":"2023-08-08T151323.526966511Z"} Failed to start: initializing osqueryd status logging: create filesystem status logger: perm check: open /var/log/fleet/osqueryd.status.log: permission denied

Ibra

08/08/2023, 3:16 PM

image.png

Kathy Satterlee

08/08/2023, 3:33 PM

Hey @Ibra. You're seeing that error because the user running Fleet doesn't have access to the local location you're writing to. If you create the log files and give them the needed permissions, you should be good.

Copy code

touch /data/docker/logs/fleet/osqueryd.results.log && chmod 666 /data/docker/logs/fleet/osqueryd.results.log
touch /data/docker/logs/fleet/osqueryd.status.log && chmod 666 /data/docker/logs/fleet/osqueryd.status.log

Kathy Satterlee

08/08/2023, 3:38 PM

also, is it possible to manage the rotation of the log files or do I have to use an external agent that works on the files slaved on the server?

For the osquery status and results logs, you can use filesystem.enable_log_rotation

Ibra

08/08/2023, 3:42 PM

thanks @Kathy Satterlee, files created and service started, but it keeps logging into the docker logs and the files remain empty, even trying to delete a host, I see that the audit.log file remains empty

Kathy Satterlee

08/08/2023, 3:49 PM

right now fleet logs can only be seen by doing docker logs Fleet, how do I send the above and audit logs to an external file, via my docker-compose.yml?

There are a couple of ways to accomplish this for the Fleet server logs. You could change the logging driver that the container is using, or you can modify the command you're using to start Fleet and pipe the logs to a file:

Copy code

command: {{current command}} &> /var/log/fleet/fleet_server.log

Note: This log would not be rotated for you and you would need to handle that manually The location for audit logs is set in the Fleet configuration. That is a Fleet Premium feature. If you're using the Community edition, you can use the REST API to grab that data.

Ibra

08/09/2023, 3:14 PM

hi @Kathy Satterlee trying to put that command doesn't work, currently my commands line is as follows:

command: sh -c "echo '\n' | /usr/bin/fleet prepare db && /usr/bin/fleet serve" &> /var/log/fleet/audit.log

also I saw this morning that the files are audit.log and osquery.results.log are stationary instead osquery.status.log contains data such as:

{"hostIdentifier":"4C4C4544-0036-4A10-805A-B6C04F4B4733","calendarTime":"Wed Aug 9 10:18:56 2023 UTC","unixTime":"1691576336","severity":"0","filename":"rocksdb. cpp","line":"67","message":"RocksDB: [WARN] [db\\db_impl\\<http://db_impl_open.cc:1846|db_impl_open.cc:1846>] Persisting Option File error: OK","version":"5.8.2","decorations":{"host_uuid":"4C4C4544-0036-4A10-805A-B6C04F4B4733","hostname":"IGHETTI"}}

Kathy Satterlee

08/09/2023, 3:18 PM

Just to clarify, have you purchased a Fleet Premium subscription?

Ibra

08/09/2023, 3:19 PM

Doing docker logs Fleet instead, I see the following:

{"component":"http","err":"Requires Fleet Premium license","ip_addr":"172.21.0.4","level":"error","method":"GET","took":"1.851051ms","ts":"2023-08-09T15:14:13.076793332Z","uri":"/api/latest/fleet/device/17ced84e-3fd1-4674-b6d7-ff34725c2a5d/desktop","uuid":"61776b8a-74de-498f-810e-b24497dceb96","x_for_ip_addr":""}

{"component":"http","err":"Requires Fleet Premium license","ip_addr":"172.21.0.4","level":"error","method":"GET","took":"1.776859ms","ts":"2023-08-09T15:14:41.146840255Z","uri":"/api/latest/fleet/device/8ba5c372-bf4a-4c71-ae71-2728dbb345c4/desktop","uuid":"61776b8a-74de-498f-810e-b24497dceb96","x_for_ip_addr":""}

{"component":"http","err":"Requires Fleet Premium license","ip_addr":"172.21.0.4","level":"error","method":"GET","took":"2.947334ms","ts":"2023-08-09T15:14:51.675895678Z","uri":"/api/latest/fleet/device/889ce4bf-9faf-464d-b830-82fe55bd35ca/desktop","uuid":"61776b8a-74de-498f-810e-b24497dceb96","x_for_ip_addr":""}

{"component":"http","err":"Requires Fleet Premium license","ip_addr":"172.21.0.4","level":"error","method":"GET","took":"2.718546ms","ts":"2023-08-09T15:15:00.249902751Z","uri":"/api/latest/fleet/device/2b382d69-155d-4000-a565-8e60d04cebb0/desktop","uuid":"61776b8a-74de-498f-810e-b24497dceb96","x_for_ip_addr":""}

{"cron":"integrations","instanceID":"HmCNouAUGAbhw7LWyBH2+RtP5CPmHsykxrfrFnt0ufREX3Kwr3CPSgmUUNpeaIXFdFFYspwX1SnkvYqXyTrfwg==","level":"info","schedule":"integrations","status":"pending","ts":"2023-08-09T15:15:03.016305894Z"}

{"cron":"integrations","instanceID":"HmCNouAUGAbhw7LWyBH2+RtP5CPmHsykxrfrFnt0ufREX3Kwr3CPSgmUUNpeaIXFdFFYspwX1SnkvYqXyTrfwg==","level":"info","schedule":"integrations","status":"completed","ts":"2023-08-09T15:15:03.024433072Z"}

{"level":"warn","op":"directIngestWindowsUpdateHistory","skipped":"KB id not found in Dell, Inc. - Firmware - 0.1.29.0","ts":"2023-08-09T15:15:17.844533349Z"}

{"level":"warn","op":"directIngestWindowsUpdateHistory","skipped":"KB id not found in Dell, Inc. - Firmware - 0.1.28.1","ts":"2023-08-09T15:15:17.844784387Z"}

{"level":"warn","op":"directIngestWindowsUpdateHistory","skipped":"KB id not found in Realtek Semiconductor Corp. - MEDIA - 6.0.9511.1","ts":"2023-08-09T15:15:17.844845459Z"}

{"level":"warn","op":"directIngestWindowsUpdateHistory","skipped":"KB id not found in Intel(R) Corporation - Extension - 1.0.2896.0","ts":"2023-08-09T15:15:17.844897622Z"}

{"level":"warn","op":"directIngestWindowsUpdateHistory","skipped":"KB id not found in Dell - Extension - 1.7.2.565","ts":"2023-08-09T15:15:17.844950301Z"}

{"level":"warn","op":"directIngestWindowsUpdateHistory","skipped":"KB id not found in Realtek Semiconductor Corp. - Extension - 6.0.9511.1","ts":"2023-08-09T15:15:17.845013496Z"}

{"level":"warn","op":"directIngestWindowsUpdateHistory","skipped":"KB id not found in Intel Corporation - Display - 31.0.101.4255","ts":"2023-08-09T15:15:17.845222924Z"}

{"level":"warn","op":"directIngestWindowsUpdateHistory","skipped":"KB id not found in Intel Corporation - Extension - 31.0.101.4255","ts":"2023-08-09T15:15:17.845295602Z"}

{"level":"warn","op":"directIngestWindowsUpdateHistory","skipped":"KB id not found in Dell, Inc. - Firmware - 0.1.27.0","ts":"2023-08-09T15:15:17.845489799Z"}

{"level":"warn","op":"directIngestWindowsUpdateHistory","skipped":"KB id not found in Realtek - SoftwareComponent - 1.0.596.0","ts":"2023-08-09T15:15:17.845566101Z"}

{"level":"warn","op":"directIngestWindowsUpdateHistory","skipped":"KB id not found in Realtek - SoftwareComponent - 11.0.6000.1096","ts":"2023-08-09T15:15:17.845627562Z"}

{"level":"warn","op":"directIngestWindowsUpdateHistory","skipped":"KB id not found in Dell - Extension - 1.0.0.61","ts":"2023-08-09T15:15:17.845685376Z"}

{"level":"warn","op":"directIngestWindowsUpdateHistory","skipped":"KB id not found in Dell - Extension - 1.0.0.61","ts":"2023-08-09T15:15:17.845777324Z"}

{"level":"warn","op":"directIngestWindowsUpdateHistory","skipped":"KB id not found in Realtek Semiconductor Corp. - Extension - 6.0.9481.1","ts":"2023-08-09T15:15:17.845845518Z"}

{"level":"warn","op":"directIngestWindowsUpdateHistory","skipped":"KB id not found in Dell - Extension - 1.7.2.563","ts":"2023-08-09T15:15:17.845926392Z"}

{"level":"warn","op":"directIngestWindowsUpdateHistory","skipped":"KB id not found in Intel(R) Corporation - Extension - 1.0.2391.0","ts":"2023-08-09T15:15:17.8459684Z"}

{"level":"warn","op":"directIngestWindowsUpdateHistory","skipped":"KB id not found in Dell, Inc. - Firmware - 0.1.26.2","ts":"2023-08-09T15:15:17.846078889Z"}

{"level":"warn","op":"directIngestWindowsUpdateHistory","skipped":"KB id not found in DisplayLink - Display - 9.3.3324.0","ts":"2023-08-09T15:15:17.84613441Z"}

{"level":"warn","op":"directIngestWindowsUpdateHistory","skipped":"KB id not found in Dell - Extension - 1.0.0.61","ts":"2023-08-09T15:15:17.846192662Z"}

{"level":"warn","op":"directIngestWindowsUpdateHistory","skipped":"KB id not found in Realtek Semiconductor Corp. - MEDIA - 6.3.9600.2277","ts":"2023-08-09T15:15:17.846258436Z"}

{"level":"warn","op":"directIngestWindowsUpdateHistory","skipped":"KB id not found in Dell, Inc. - Firmware - 0.1.25.0","ts":"2023-08-09T15:15:17.84631594Z"}

{"level":"warn","op":"directIngestWindowsUpdateHistory","skipped":"KB id not found in Dell, Inc. - Firmware - 10.41.1.6","ts":"2023-08-09T15:15:17.846387237Z"}

{"level":"warn","op":"directIngestWindowsUpdateHistory","skipped":"KB id not found in Intel Corporation - Extension - 18.7.6.1010","ts":"2023-08-09T15:15:17.846610357Z"}

{"level":"warn","op":"directIngestWindowsUpdateHistory","skipped":"KB id not found in Intel - net - 22.190.0.4","ts":"2023-08-09T15:15:17.846678728Z"}

{"level":"warn","op":"directIngestWindowsUpdateHistory","skipped":"KB id not found in Realtek Semiconductor Corp. - Extension - 10.0.212.29","ts":"2023-08-09T15:15:17.846714989Z"}

{"level":"warn","op":"directIngestWindowsUpdateHistory","skipped":"KB id not found in Intel(R) Corporation - Extension - 1.0.2256.0","ts":"2023-08-09T15:15:17.846771516Z"}

{"level":"warn","op":"directIngestWindowsUpdateHistory","skipped":"KB id not found in Intel - System - 8.7.10802.26924","ts":"2023-08-09T15:15:17.846838399Z"}

{"level":"warn","op":"directIngestWindowsUpdateHistory","skipped":"KB id not found in Realtek Semiconductor Corp. - MTD - 10.0.22000.21350","ts":"2023-08-09T15:15:17.846903561Z"}

{"level":"warn","op":"directIngestWindowsUpdateHistory","skipped":"KB id not found in Intel Corporation - SoftwareComponent - 18.7.1.1003","ts":"2023-08-09T15:15:17.846972744Z"}

{"level":"warn","op":"directIngestWindowsUpdateHistory","skipped":"KB id not found in Intel Corporation - SoftwareComponent - 18.7.1.1003","ts":"2023-08-09T15:15:17.847015334Z"}

{"level":"warn","op":"directIngestWindowsUpdateHistory","skipped":"KB id not found in Realtek Semiconductor Corp. - Extension - 10.0.19041.1","ts":"2023-08-09T15:15:17.847143327Z"}

could you please tell me how to fix the docker-compose.yml file so I can better manage the logs and saving everything in one location also managing the rotation without ever having to delete the logs, unless I do it manually? Below is my docker-compose.yml, how can I modify it?

fleetdm:

restart: always

container_name: Fleet

image: fleetdm/fleet:v4.34.1

volumes:

- /data/docker/appdata/fleet:/fleet

- /data/docker/logs/fleet:/var/log/fleet

command: sh -c "echo '\n' | /usr/bin/fleet prepare db && /usr/bin/fleet serve" &> /var/log/fleet/audit.log

environment:

FLEET_MYSQL_ADDRESS: 172.21.x.x:3306

FLEET_MYSQL_DATABASE: fleet

FLEET_MYSQL_USERNAME: app

FLEET_MYSQL_PASSWORD: *******

FLEET_REDIS_ADDRESS: redis:6379

FLEET_SERVER_CERT: /fleet/itl-cslapp-54local.crt

FLEET_SERVER_KEY: /fleet/itl-cslapp-54local.key

FLEET_LOGGING_JSON: "true"

FLEET_AUTH_JWT_KEY:

FLEET_ACTIVITY_ENABLE_AUDIT_LOG: "true"

#FLEET_FILESYSTEM_ENABLE_LOG_ROTATION: "true"

FLEET_ACTIVITY_AUDIT_LOG_PLUGIN: filesystem

#FLEET_FILESYSTEM_ENABLE_LOG_COMPRESSION: "true"

FLEET_FILESYSTEM_AUDIT_LOG_FILE: /var/log/fleet/audit.log

FLEET_OSQUERY_STATUS_LOG_PLUGIN: filesystem

FLEET_FILESYSTEM_STATUS_LOG_FILE: /var/log/fleet/osqueryd.status.log

FLEET_OSQUERY_RESULT_LOG_PLUGIN: filesystem

FLEET_FILESYSTEM_RESULT_LOG_FILE: /var/log/fleet/osqueryd.results.log

labels:

- "traefik.enable=true"

`- "traefik.tcp.routers.fleetdm.rule=HostSNI(

)"`

- "traefik.tcp.routers.fleetdm.entrypoints=websecure"

- "traefik.tcp.services.fleetdm.loadbalancer.server.port=8080"

networks:

- Proxy

thanks

Ibra

08/09/2023, 3:19 PM

no, I'm keeping the free version but I want to configure it as best I can

Kathy Satterlee

08/09/2023, 3:21 PM

Gotcha. In that case, I would expect you not to have audit logs. External logging for the activity feed is a Fleet Premium feature For the osquery results logs, do you have any scheduled queries set up? how many hosts do you have enrolled?

Ibra

08/09/2023, 3:25 PM

I did enroll 90 devices, but I was waiting to schedule the queries until I was sure I had set up all the configuration correctly. by audit files, I meant to record the accesses and changes made by users such as host removal, query execution, and so on, so that I have complete control of the system.

Kathy Satterlee

08/09/2023, 3:26 PM

Yes. That audit information is available without a Premium license through the REST API, but not through external logging.

Kathy Satterlee

08/09/2023, 3:27 PM

The osquery results logs are for scheduled queries, so you'll see data there once you schedule queries.

Ibra

08/09/2023, 3:28 PM

ok, I tried a query now and saw that the osquery result file filled in with the following rows:

{"snapshot":[],"action":"snapshot","name":"pack/Global/Verifica Bitlocker","hostIdentifier":"4C4C4544-0043-4A10-805A-B9C04F4B4733","calendarTime":"Wed Aug  9 15:24:12 2023 UTC","unixTime":1691594652,"epoch":0,"counter":0,"numerics":false,"decorations":{"host_uuid":"4C4C4544-0043-4A10-805A-B9C04F4B4733","hostname":"EXP-MSANCILLO"}}

{"snapshot":[{"1":"1"}],"action":"snapshot","name":"pack/Global/Verifica Bitlocker","hostIdentifier":"4C4C4544-0039-4A10-805A-C7C04F4B4733","calendarTime":"Wed Aug  9 15:24:45 2023 UTC","unixTime":1691594685,"epoch":0,"counter":0,"numerics":false,"decorations":{"host_uuid":"4C4C4544-0039-4A10-805A-C7C04F4B4733","hostname":"EXP-MBAVIERA"}}

{"snapshot":[],"action":"snapshot","name":"pack/Global/Verifica Bitlocker","hostIdentifier":"4C4C4544-0056-5110-8036-B1C04F57344A","calendarTime":"Wed Aug  9 15:25:18 2023 UTC","unixTime":1691594718,"epoch":0,"counter":0,"numerics":false,"decorations":{"host_uuid":"4C4C4544-0056-5110-8036-B1C04F57344A","hostname":"<http://EXP-WS19.experim.it|EXP-WS19.experim.it>"}}

{"snapshot":[{"1":"1"}],"action":"snapshot","name":"pack/Global/Verifica Bitlocker","hostIdentifier":"4C4C4544-0051-4C10-8058-B3C04F354A33","calendarTime":"Wed Aug  9 15:25:33 2023 UTC","unixTime":1691594733,"epoch":0,"counter":0,"numerics":false,"decorations":{"host_uuid":"4C4C4544-0051-4C10-8058-B3C04F354A33","hostname":"EXP-IALI"}}

{"snapshot":[{"1":"1"}],"action":"snapshot","name":"pack/Global/Verifica Bitlocker","hostIdentifier":"4C4C4544-0032-5010-8031-C7C04F364A33","calendarTime":"Wed Aug  9 15:25:57 2023 UTC","unixTime":1691594757,"epoch":0,"counter":0,"numerics":false,"decorations":{"host_uuid":"4C4C4544-0032-5010-8031-C7C04F364A33","hostname":"EXP-AROMANO"}}

What is the difference between snapshot and differential?

Kathy Satterlee

08/09/2023, 3:29 PM

Snapshot returns the full results every time the query runs. Differential returns any changes since the last run.

Ibra

08/09/2023, 3:31 PM

ok, one more thing, in this case the logs are written to both osquery files, but the loggates related to device enrollment do I have a way to log them to external file instead of docker logs like the example I posted earlier? also having plans to develop an ELK Stack how do I get it to interact with fleet to collect all the logs via the REST API? in a nutshell I want to log everything in debug mode of this instance

Ibra

08/09/2023, 3:33 PM

managing log rotation, it was written that files older than 30 days were deleted, but in this case how do I make them stay? should I stop the rotation via docker compose and manage it with external software such as logrotate on the server itself?

Kathy Satterlee

08/09/2023, 3:40 PM

also having plans to develop an ELK Stack how do I get it to interact with fleet to collect all the logs via the REST API?

You'd likely need to use a tool to gather the data from the API and forward to your ultimate logging destination

managing log rotation, it was written that files older than 30 days were deleted, but in this case how do I make them stay?

You can manage those settings in your Fleet config: https://fleetdm.com/docs/configuration/fleet-server-configuration#filesystem-max-age https://fleetdm.com/docs/configuration/fleet-server-configuration#filesystem-max-backups

loggates related to device enrollment do I have a way to log them to external file instead of docker logs like the example I posted earlier

There is not a way to separate those from the logs within Fleet. If you get those logs saving to file, you could use a file listener like filebeat to look for those entries and parse them out, For getting those logs to file, try putting the output portion inside your quotes:

Copy code

command: sh -c "echo '\n' | /usr/bin/fleet prepare db && /usr/bin/fleet serve &> /var/log/fleet/audit.log"

Ibra

08/09/2023, 3:42 PM

I will try now to modify the command and see if it works

Ibra

08/09/2023, 3:47 PM

ok, by editing the command I saw that it moves the logs and by trying to delete my pc I saw that it also logs the enroll that made the pc

{"component":"http","hardware_serial":"","hardware_uuid":"4C4C4544-0051-4C10-8058-B3C04F354A33","hostname":"EXP-IALI","level":"info","method":"POST","platform":"windows","took":"8. 059256ms","ts":"2023-08-09T15:45:21.324309345Z","uri":"/api/fleet/orbit/enroll","user":"unauthenticated"}

Ibra

08/09/2023, 3:53 PM

okay so to summarize and get your confirmation that everything is okay: 1) log management: - the container logs have been moved to the audit.log file and the same for the osquery files. - I will remove the lines #FLEET_FILESYSTEM_ENABLE_LOG_ROTATION: "true" and #FLEET_FILESYSTEM_ENABLE_LOG_COMPRESSION: "true" so that the system continues to write to the same files and handle rotation and compression via external logging system. - I will find a way to handle RESTs to make them communicate with ELK Stack 2) personalization: - I saw that it allows me to add the company logo in the header but not in the login screen. 3) outgoing emails in case of problems with queries/policies. - will not accept connection to the company smtp server on port 465, despite being reachable via telnet from within the container. is it possible to use mailhog to send mails externally via dummy mail?

Kathy Satterlee

08/09/2023, 4:02 PM

1 and 2 looking good. For 3, What error are you getting back?

Ibra

08/09/2023, 4:04 PM

I have mailhog active, but when I enter our mail server data, on the browser console I see: PATCH https://it-asset.company.it/api/latest/fleet/config 422 (Unprocessable entity) and from the logs I see: {"component": "http", "err": "validation failed: SMTP Options a mail error occurred: sending mail: could not dial smtp host: SMTP connection error: read tcp 172.21.0.5:49700->35.152.65. 1465 i/o timeout", "level": "error", "method": "PATCH", "takes": "30. 002517912s", "ts": "2023-08-09T160217.340423034Z", "uri":"/api/latest/fleet/config", "user": "iali@company.it", "uuid": "b91bace2-ff3a-46cf-8f91-2924a81099ce"}

Ibra

08/09/2023, 4:24 PM

any suggestions?

Kathy Satterlee

08/09/2023, 4:29 PM

It looks like there might be a space in the IP address for the mail server.

Kathy Satterlee

08/09/2023, 4:29 PM

Not sure if that's just in the logs, or in the setup

Ibra

08/09/2023, 4:32 PM

the space is my fault for a typo, but in the configuration I entered smtps.company.it also i can't telnet again as it tells me it can't access the apk db to install it and it doesn't recognize sudo.

Edival Gonçalves Gomes

03/01/2024, 8:18 PM

Hello, I'm adding my hosts to Fleet and it appears on the Web, but when it brings in the data it goes offline. Can anyone help me please?

50 Views

Open in Slack

Previous Next