In looking at the fleet service status, I see that the address should be serving over IPv4: {"address":"0.0.0.0:443","msg":"listening","transport":"https","ts":"2023-03-20T153417.857040269Z"} But when looking at netstat, I only see Fleet serving on IPv6: Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN 463/systemd-resolve tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 858/sshd: /usr/sbin tcp 0 0 127.0.0.1:33060 0.0.0.0:* LISTEN 675/mysqld tcp 0 0 127.0.0.1:6379 0.0.0.0:* LISTEN 535/redis-server 12 tcp 0 0 127.0.0.1:3306 0.0.0.0:* LISTEN 675/mysqld tcp6 0 0 :::443 :::* LISTEN 15860/fleet

To clarify, since after rereading my initial post I didn't really describe the issue well: Fleet is not serving over IPv4, it is only serving over IPv6. IPv4 is what I need to work.

I set the fleet.yml file to include debug logging - after a service restart, I didn't see any errors.

I validated that 443 is working for other applications on the server, so this appears to be Fleet related as opposed to an IPv4 or OS issue.

I rolled back to 4.26.0 (what I used successfully in my lab environment) to see if this would make a difference - no change.

YML config file contents: mysql: address: '127.0.0.1:3306' database: 'fleet' username: 'fleet' password: '<REDACTED>' redis: address: '127.0.0.1:6379' server: address: '0.0.0.0:443' cert: '/usr/bin/fleet/fleet.cert' key: '/usr/bin/fleet/fleet.key' logging: json: 'true' debug: 'true' activity: enable_audit_log: 'true' audit_log_plugin: 'filesystem'

for the record my listen looks like:

tcp        0      0 :::8080                 :::*                    LISTEN      514/fleet

has the :::8080 but obviously

tcp

and not

tcp6

as yours

I'm thinking I might just start over to eliminate any rogue issues. What would be the best way to go about that? - Disregard, I think I have that sorted out.

So I removed/reinstalled Redis, mysql and Fleet - looks like the same issue is occurring.

root@fleetdm:/usr/bin/fleetdm# netstat -tulpn Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN 4522/systemd-resolv tcp 0 0 127.0.0.1:3306 0.0.0.0:* LISTEN 16405/mysqld tcp 0 0 127.0.0.1:33060 0.0.0.0:* LISTEN 16405/mysqld tcp 0 0 127.0.0.1:6379 0.0.0.0:* LISTEN 14925/redis-server tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 10752/sshd: /usr/sb tcp6 0 0 :16379 :::* LISTEN 14925/redis-server tcp6 0 0 :::4421 :::* LISTEN 9503/sftd tcp6 0 0 :::443 :::* LISTEN 16537/./fleet tcp6 0 0 :::22 :::* LISTEN 10752/sshd: /usr/sb

Here is my Fleet config file: Using config file: fleet.yml mysql: protocol: tcp address: 127.0.0.1:3306 username: fleet password: <REDACTED> password_path: "" database: fleet tls_cert: "" tls_key: "" tls_ca: "" tls_server_name: "" tls_config: "" max_open_conns: 50 max_idle_conns: 50 conn_max_lifetime: 0 sql_mode: "" mysql_read_replica: protocol: tcp address: "" username: fleet password: "" password_path: "" database: fleet tls_cert: "" tls_key: "" tls_ca: "" tls_server_name: "" tls_config: "" max_open_conns: 50 max_idle_conns: 50 conn_max_lifetime: 0 sql_mode: "" redis: address: 127.0.0.1:6379 username: "" password: "" database: 0 use_tls: false duplicate_results: false connect_timeout: 5s keep_alive: 10s connect_retry_attempts: 0 cluster_follow_redirections: false cluster_read_from_replica: false tls_cert: "" tls_key: "" tls_ca: "" tls_server_name: "" tls_handshake_timeout: 10s max_idle_conns: 3 max_open_conns: 0 conn_max_lifetime: 0s idle_timeout: 4m0s conn_wait_timeout: 0s write_timeout: 10s read_timeout: 10s server: address: 0.0.0.0:443 cert: /usr/bin/fleetdm/fleet.cert key: /usr/bin/fleetdm/fleet.key tls: true tls_compatibility: intermediate url_prefix: "" keepalive: true sandbox_enabled: false auth: bcrypt_cost: 12 salt_key_size: 24 app: token_key_size: 24 invite_token_validity_period: 120h0m0s enable_scheduled_query_stats: true session: key_size: 64 duration: 120h0m0s osquery: node_key_size: 24 host_identifier: provided enroll_cooldown: 0s status_log_plugin: filesystem result_log_plugin: filesystem label_update_interval: 1h0m0s policy_update_interval: 1h0m0s detail_update_interval: 1h0m0s status_log_file: "" result_log_file: "" enable_log_rotation: false max_jitter_percent: 10 enable_async_host_processing: "false" async_host_collect_interval: 30s async_host_collect_max_jitter_percent: 10 async_host_collect_lock_timeout: 1m0s async_host_collect_log_stats_interval: 1m0s async_host_insert_batch: 2000 async_host_delete_batch: 2000 async_host_update_batch: 1000 async_host_redis_pop_count: 1000 async_host_redis_scan_keys_count: 1000 min_software_last_opened_at_diff: 1h0m0s activity: enable_audit_log: true audit_log_plugin: filesystem logging: debug: true json: true disable_banner: false error_retention_period: 24h0m0s tracing_enabled: false tracing_type: opentelemetry firehose: region: "" endpoint_url: "" access_key_id: "" secret_access_key: "" sts_assume_role_arn: "" status_stream: "" result_stream: "" audit_stream: "" kinesis: region: "" endpoint_url: "" access_key_id: "" secret_access_key: "" sts_assume_role_arn: "" status_stream: "" result_stream: "" audit_stream: "" lambda: region: "" access_key_id: "" secret_access_key: "" sts_assume_role_arn: "" status_function: "" result_function: "" audit_function: "" s3: bucket: "" prefix: "" region: "" endpoint_url: "" access_key_id: "" secret_access_key: "" sts_assume_role_arn: "" disable_ssl: false force_s3_path_style: false pubsub: project: "" status_topic: "" result_topic: "" audit_topic: "" add_attributes: false filesystem: status_log_file: /tmp/osquery_status result_log_file: /tmp/osquery_result audit_log_file: /tmp/audit enable_log_rotation: false enable_log_compression: false max_size: 500 max_age: 28 max_backups: 3 kafkarest: status_topic: "" result_topic: "" audit_topic: "" proxyhost: "" content_type_value: application/vnd.kafka.json.v1+json timeout: 5 license: key: "" enforce_host_limit: false vulnerabilities: databases_path: /tmp/vulndbs periodicity: 1h0m0s cpe_database_url: "" cpe_translations_url: "" cve_feed_prefix_url: "" current_instance_checks: auto disable_schedule: false disable_data_sync: false recent_vulnerability_max_age: 720h0m0s disable_win_os_vulnerabilities: false upgrades: allow_missing_migrations: false sentry: dsn: "" geoip: database_path: "" prometheus: basic_auth: username: "" password: "" packaging: global_enroll_secret: "" s3: bucket: "" prefix: "" region: "" endpoint_url: "" access_key_id: "" secret_access_key: "" sts_assume_role_arn: "" disable_ssl: false force_s3_path_style: false mdm: apple_apns_cert: "" apple_apns_cert_bytes: "" apple_apns_key: "" apple_apns_key_bytes: "" apple_scep_cert: "" apple_scep_cert_bytes: "" apple_scep_key: "" apple_scep_key_bytes: "" apple_bm_server_token: "" apple_bm_server_token_bytes: "" apple_bm_cert: "" apple_bm_cert_bytes: "" apple_bm_key: "" apple_bm_key_bytes: "" mdm_apple: enable: false scep: signer: validity_days: 365 allow_renewal_days: 14 challenge: "" dep: sync_periodicity: 1m0s

Is the web server contained within the Fleet binary?

Also - switching from 443 to 8080 provides the same error - nothing served on IPv4.

I even tried port 7890 (arbitrary port) just to see if the behavior changed. Same issue.

Ok - so I have no idea what changed but it's working.

Apologies for the spam comments!

Interesting note - I still don't see IPv4 serving the port up even when it's working which is behavior I would expect to see, but I may have gotten too fixated on that factoid. 🙂

Just to close this out - the issue was a red herring. When TCP6 is showing as listening on 443 using fleet, it is in fact listening on IPv4 and 6. https://www.rfc-editor.org/rfc/rfc3493#section-3.7 - Thank you for having me dig deeper into the system itself!