Hi again team When serving Fleet I am seeing in netstat that

Question

Hi again team When serving Fleet I am seeing in netstat that fleet is only serving 443 over IPv6 despite my disabling IPv6 in the OS Is there a setting I inadvertently tweaked that would have caused that

Mike S. · Answer

In looking at the fleet service status I see that the address should be serving over IPv4 address 0 0 0 0 443 msg listening transport https ts 2023 03 20T15 34 17 857040269Z But when looking at netstat I only see Fleet serving on IPv6 Proto Recv Q Send Q Local Address Foreign Address State PID Program name tcp 0 0 127 0 0 53 53 0 0 0 0 LISTEN 463 systemd resolve tcp 0 0 0 0 0 0 22 0 0 0 0 LISTEN 858 sshd usr sbin tcp 0 0 127 0 0 1 33060 0 0 0 0 LISTEN 675 mysqld tcp 0 0 127 0 0 1 6379 0 0 0 0 LISTEN 535 redis server 12 tcp 0 0 127 0 0 1 3306 0 0 0 0 LISTEN 675 mysqld tcp6 0 0 443 LISTEN 15860 fleet

Mike S. · Answer

To clarify since after rereading my initial post I didn t really describe the issue well Fleet is not serving over IPv4 it is only serving over IPv6 IPv4 is what I need to work

Mike S. · Answer

I set the fleet yml file to include debug logging after a service restart I didn t see any errors

Mike S. · Answer

I validated that 443 is working for other applications on the server so this appears to be Fleet related as opposed to an IPv4 or OS issue

Mike S. · Answer

I rolled back to 4 26 0 what I used successfully in my lab environment to see if this would make a difference no change

Mike S. · Answer

YML config file contents mysql address 127 0 0 1 3306 database fleet username fleet password lt REDACTED gt redis address 127 0 0 1 6379 server address 0 0 0 0 443 cert usr bin fleet fleet cert key usr bin fleet fleet key logging json true debug true activity enable audit log true audit log plugin filesystem

chrismsnz · Answer

are you running fleet as root it will need to be root to bind to 443 which is why it defaults to httpalt

Mike S. · Answer

Hi < chrismsnz> yes I am I am running it under the root account and tested it using fleet serve config fleet yml same issue

chrismsnz · Answer

my only thought is that there is something binding to another ipv4 on 443 somewhere e g localhost 443 which is causing 0 0 0 0 443 to fail but not 443

chrismsnz · Answer

for the record my listen looks like `tcp 0 0 8080 LISTEN 514 fleet` has the 8080 but obviously `tcp` and not `tcp6` as yours

Mike S. · Answer

I was thinking that too when I stop the fleet service and run netcat nc 4 l 443 it is able to listen on the port When I start the fleet service netcat is not able to work stating the address is already in use

Mike S. · Answer

I m thinking I might just start over to eliminate any rogue issues What would be the best way to go about that Disregard I think I have that sorted out

Mike S. · Answer

So I removed reinstalled Redis mysql and Fleet looks like the same issue is occurring

Mike S. · Answer

root fleetdm usr bin fleetdm netstat tulpn Active Internet connections only servers Proto Recv Q Send Q Local Address Foreign Address State PID Program name tcp 0 0 127 0 0 53 53 0 0 0 0 LISTEN 4522 systemd resolv tcp 0 0 127 0 0 1 3306 0 0 0 0 LISTEN 16405 mysqld tcp 0 0 127 0 0 1 33060 0 0 0 0 LISTEN 16405 mysqld tcp 0 0 127 0 0 1 6379 0 0 0 0 LISTEN 14925 redis server tcp 0 0 0 0 0 0 22 0 0 0 0 LISTEN 10752 sshd usr sb tcp6 0 0 1 6379 LISTEN 14925 redis server tcp6 0 0 4421 LISTEN 9503 sftd tcp6 0 0 443 LISTEN 16537 fleet tcp6 0 0 22 LISTEN 10752 sshd usr sb

Mike S. · Answer

Here is my Fleet config file Using config file fleet yml mysql protocol tcp address 127 0 0 1 3306 username fleet password lt REDACTED gt password path database fleet tls cert tls key tls ca tls server name tls config max open conns 50 max idle conns 50 conn max lifetime 0 sql mode mysql read replica protocol tcp address username fleet password password path database fleet tls cert tls key tls ca tls server name tls config max open conns 50 max idle conns 50 conn max lifetime 0 sql mode redis address 127 0 0 1 6379 username password database 0 use tls false duplicate results false connect timeout 5s keep alive 10s connect retry attempts 0 cluster follow redirections false cluster read from replica false tls cert tls key tls ca tls server name tls handshake timeout 10s max idle conns 3 max open conns 0 conn max lifetime 0s idle timeout 4m0s conn wait timeout 0s write timeout 10s read timeout 10s server address 0 0 0 0 443 cert usr bin fleetdm fleet cert key usr bin fleetdm fleet key tls true tls compatibility intermediate url prefix keepalive true sandbox enabled false auth bcrypt cost 12 salt key size 24 app token key size 24 invite token validity period 120h0m0s enable scheduled query stats true session key size 64 duration 120h0m0s osquery node key size 24 host identifier provided enroll cooldown 0s status log plugin filesystem result log plugin filesystem label update interval 1h0m0s policy update interval 1h0m0s detail update interval 1h0m0s status log file result log file enable log rotation false max jitter percent 10 enable async host processing false async host collect interval 30s async host collect max jitter percent 10 async host collect lock timeout 1m0s async host collect log stats interval 1m0s async host insert batch 2000 async host delete batch 2000 async host update batch 1000 async host redis pop count 1000 async host redis scan keys count 1000 min software last opened at diff 1h0m0s activity enable audit log true audit log plugin filesystem logging debug true json true disable banner false error retention period 24h0m0s tracing enabled false tracing type opentelemetry firehose region endpoint url access key id secret access key sts assume role arn status stream result stream audit stream kinesis region endpoint url access key id secret access key sts assume role arn status stream result stream audit stream lambda region access key id secret access key sts assume role arn status function result function audit function s3 bucket prefix region endpoint url access key id secret access key sts assume role arn disable ssl false force s3 path style false pubsub project status topic result topic audit topic add attributes false filesystem status log file tmp osquery status result log file tmp osquery result audit log file tmp audit enable log rotation false enable log compression false max size 500 max age 28 max backups 3 kafkarest status topic result topic audit topic proxyhost content type value application vnd kafka json v1+json timeout 5 license key enforce host limit false vulnerabilities databases path tmp vulndbs periodicity 1h0m0s cpe database url cpe translations url cve feed prefix url current instance checks auto disable schedule false disable data sync false recent vulnerability max age 720h0m0s disable win os vulnerabilities false upgrades allow missing migrations false sentry dsn geoip database path prometheus basic auth username password packaging global enroll secret s3 bucket prefix region endpoint url access key id secret access key sts assume role arn disable ssl false force s3 path style false mdm apple apns cert apple apns cert bytes apple apns key apple apns key bytes apple scep cert apple scep cert bytes apple scep key apple scep key bytes apple bm server token apple bm server token bytes apple bm cert apple bm cert bytes apple bm key apple bm key bytes mdm apple enable false scep signer validity days 365 allow renewal days 14 challenge dep sync periodicity 1m0s

chrismsnz · Answer

I don t think its fleet or your config tbh something on your system is preventing fleet from binding to the ipv4 443 maybe specify the listen address rather than 0 0 0 0

Mike S. · Answer

Unfortunately I ve never been able to input the specific listening address and get it to work in lab or in prod There doesn t appear to be anything binding to that IP or port I rebuilt the EC2 this morning with no IPS AV only the software that comes with Ubuntu minimal install and MySQL Redis Fleet Same issue

Mike S. · Answer

Is the web server contained within the Fleet binary

Mike S. · Answer

Also switching from 443 to 8080 provides the same error nothing served on IPv4

Mike S. · Answer

I even tried port 7890 arbitrary port just to see if the behavior changed Same issue

Mike S. · Answer

Ok so I have no idea what changed but it s working

Mike S. · Answer

Apologies for the spam comments

Mike S. · Answer

Interesting note I still don t see IPv4 serving the port up even when it s working which is behavior I would expect to see but I may have gotten too fixated on that factoid slightly smiling face

Mike S. · Answer

Just to close this out the issue was a red herring When TCP6 is showing as listening on 443 using fleet it is in fact listening on IPv4 and 6 <https www rfc editor org rfc rfc3493 section 3 7> Thank you for having me dig deeper into the system itself