Title
#fleet
pvirani

pvirani

02/01/2022, 12:32 AM
Hey friends! Is there no way to get device enrolment to work with OAuth on at the moment? (assuming that's why my enrolment is failing going by the following error message)
Could not read SMBIOS memory
I0201 00:31:42.133729  1860 tls.cpp:255] TLS/HTTPS POST request to URI: <https://fleetdm.segment.build/api/v1/osquery/enroll>
I0201 00:31:42.152062  1860 http_client.cpp:420] HTTP(S) request re-directed to: <https://segment.okta.com/oauth2/v1/authorize?client_id=**********************************>
^CW0201 00:31:42.745014  1860 tls_enroll.cpp:101] Failed enrollment request to <https://fleetdm.segment.build/api/v1/osquery/enroll> (Cannot parse JSON: Invalid value. Offset: 0) retrying...
zwass

zwass

02/01/2022, 12:38 AM
Hmmm, sounds like you have an oauth proxy? You probably need to add exceptions for the
/api/v1/osquery/*
endpoints as osquery agents are not going to be able to complete an oauth flow.
12:39 AM
If that's not possible, you can expose only those API endpoints on a different hostname (filtering routes via nginx or your load balancer).
pvirani

pvirani

02/01/2022, 12:40 AM
I think the exceptions route should work. I didn't even know that was an option with Oauth setups. TIL!
7:56 PM
I removed SAML and OIDC from my Stage deployment and the enrolment seemed to have worked briefly but now I have this very weird issue where the hosts tab is completely blank and nothing loads in the activity section. Screenshots below. Queries weren't loading for a while either but they've started loading now somehow. (have already tried logging out, back in etc basic troubleshooting methods) I'm trying to debug it on my own but somehow my App logs aren't showing up via kubectl either so there's that I'm trying to solve first but also posting here just in case you or anyone else in the community has seen a similar behaviour before.
7:57 PM
Queries page has stopped loading again too! Screenshot below
zwass

zwass

02/01/2022, 7:57 PM
Can you open up the network inspector and see what's happening with the requests?
pvirani

pvirani

02/01/2022, 10:12 PM
ts=2022-02-01T21:44:22.46995579Z mysql="could not connect to db: dial tcp 10.80.113.69:3306: connect: connection timed out, sleeping 0s"
ts=2022-02-01T21:46:33.541954613Z mysql="could not connect to db: dial tcp 10.80.113.69:3306: connect: connection timed out, sleeping 1s"
ts=2022-02-01T21:48:44.613944779Z mysql="could not connect to db: dial tcp 10.80.113.69:3306: connect: connection timed out, sleeping 2s"
ts=2022-02-01T21:50:57.733936566Z mysql="could not connect to db: dial tcp 10.80.113.69:3306: connect: connection timed out, sleeping 3s"
ts=2022-02-01T21:53:10.853938337Z mysql="could not connect to db: dial tcp 10.80.113.69:3306: connect: connection timed out, sleeping 4s"
ts=2022-02-01T21:55:26.021983497Z mysql="could not connect to db: dial tcp 10.80.113.69:3306: connect: connection timed out, sleeping 5s"
ts=2022-02-01T21:57:41.193944455Z mysql="could not connect to db: dial tcp 10.80.113.69:3306: connect: connection timed out, sleeping 6s"
ts=2022-02-01T21:59:58.405983358Z mysql="could not connect to db: dial tcp 10.80.113.69:3306: connect: connection timed out, sleeping 7s"
ts=2022-02-01T22:02:15.621947566Z mysql="could not connect to db: dial tcp 10.80.113.69:3306: connect: connection timed out, sleeping 8s"
ts=2022-02-01T22:04:32.837935262Z mysql="could not connect to db: dial tcp 10.80.113.69:3306: connect: connection timed out, sleeping 9s"
ts=2022-02-01T22:06:52.102010455Z mysql="could not connect to db: dial tcp 10.80.113.69:3306: connect: connection timed out, sleeping 10s"
ts=2022-02-01T22:09:11.366299187Z mysql="could not connect to db: dial tcp 10.80.113.69:3306: connect: connection timed out, sleeping 11s"
ts=2022-02-01T22:11:32.677947186Z mysql="could not connect to db: dial tcp 10.80.113.69:3306: connect: connection timed out, sleeping 12s"
Suddenly the mysql connection times out. weird
10:12 PM
coz this was ofcourse never a problem when the app was deployed. I can try redeploying it if needed but that wouldn't be ideal
10:13 PM
why would an attempt to enroll a host break mysql 🤔
zwass

zwass

02/01/2022, 10:32 PM
Seems unlikely that it would have caused that...? Never know though
pvirani

pvirani

02/01/2022, 10:34 PM
humm if it was broken from the get to, Fleet wouldn't even have deployed right? I remember not being able to start the app due to failing SQL connection and the app didn't start till I fixed that
10:40 PM
anyways trying to upgrade to the latest version now and seeing if that deploys ok somehow or if it complains about connecting with the mysql DB 🤞🏽
zwass

zwass

02/01/2022, 10:42 PM
If you were able to get past the login screen in the past then you definitely had MySQL working. The server won't start up without connection to MySQL, but it will stay up if it was previously connected and keep trying to connect.
pvirani

pvirani

02/01/2022, 11:52 PM
All upgraded, found the root cause and fixed. TL;DR it wasn't fleet it was Segment's Terraform modules. womp womp! When I removed the OIDC settings it overwrote my RDS security group (SG) settings. Even though there was a policy defined my TF file to allow 3306 connections between my EKS pool SG and RDS SG it somehow didn't apply them. That broke the App's RDS connection with MySQL and the rest of it as we know it. A good exercise in trying to upgrade the App version though! This was my first upgrade after setting it up 🙂
zwass

zwass

02/01/2022, 11:54 PM
Oh glad you worked it out! Thank you for sharing 🙂
pvirani

pvirani

02/02/2022, 12:00 AM
As always, thanks for patiently replying to my debugging rants and being an awesome collaborator 🙏🏽