Hey friends I have a RDS MySQL DB up and running with a cust

Question

Hey friends I have a RDS MySQL DB up and running with a custom username but whenever I run fleet serve command with the username env variable it keeps saying `Unknown database fleet ` Works fine otherwise Any ideas why

pvirani · Answer

```apiVersion kind SegmentApplicationExperimental metadata name fleetdm webserver namespace fleetdm webserver labels app fleetdm webserver spec targets name fleetdm stage usw2 cluster sec tooling stage us west 2 fleetdm targetGroupBinding autoDiscover true replicatedService amp replicatedService iamRoleName fleetdm webserver autoScale amp autoScale minReplicas 1 maxReplicas 6 resources resource cpu utilization 15 resource memory utilization 15 maincontainers name fleetdm webserver ctlstore disabled true imageRegistry imageName fleetdm fleet command chamber exec fleetdm fleet serve ports containerPort 443 env name FLEET MYSQL ADDRESS value name FLEET MYSQL USERNAME value fleet rds name FLEET REDIS ADDRESS value name FLEET SERVER ADDRESS value 0 0 0 0 443 ``` This is my Kubernetes App config file

Martavis Parker · Answer

Try using the `mysql database` flag <https github com fleetdm fleet blob main docs 02 Deploying 02 Configuration md mysql database>

pvirani · Answer

oh so you mean instead of FLEET MYSQL ADDRESS I should use FLEET MYSQL DATABASE

pvirani · Answer

``` env name FLEET MYSQL ADDRESS value name FLEET MYSQL DATABASE value name FLEET MYSQL USER value fleet rds name FLEET REDIS ADDRESS value name FLEET SERVER ADDRESS value 0 0 0 0 443 ``` Added it like so It doesn t seem to be picking up the username and keeps saying ```ts=2021 10 07T19 30 24 561132517Z mysql= could not connect to db Error 1045 Access denied for user fleet 10 80 94 227 using password YES sleeping 14s ```

pvirani · Answer

``` kc exec fleetdm webserver 58c85b9bf7 kqfdr n fleetdm webserver fleet serve ts=2021 10 07T20 03 13 427339705Z mysql= could not connect to db Error 1045 Access denied for user fleet 10 80 79 187 using password YES ``` however just after adding the mysql username flag it starts complaining about the database being unknown ``` kc exec fleetdm webserver 58c85b9bf7 kqfdr n fleetdm webserver fleet serve mysql username=fleet rds ts=2021 10 07T20 04 42 621780497Z mysql= could not connect to db Error 1049 Unknown database <http fleetdatabase crqbc9r8uf32 us west 2 rds amazonaws com 3306|fleetdatabase crqbc9r8uf32 us west 2 rds amazonaws com 3306> sleeping 0s ```

pvirani · Answer

is the only way to solve this is to not use a custom username

zwass · Answer

No both are supported Perhaps you need to actually `CREATE DATABASE fleet` within your MySQL instance

zwass · Answer

`FLEET MYSQL ADDRESS` refers to the address the MySQL server is listening on while `FLEET MYSQL DATABASE` refers to the logical database within MySQL eg one created in MySQL with `CREATE DATABASE` and accessed with `USE DATABASE`

pvirani · Answer

Ah got it

pvirani · Answer

It s up and running tada Thanks folks

pvirani · Answer

suuuper silly question how do I find out which URL it s up and running at

Benjamin Edwards · Answer

How are you running fleet If you have access to the host that is running fleet you should be able to access it at localhost 8080

pvirani · Answer

Im running it on an EKS cluster and I think I know what s broken smile I need to make the app aware of the ALB that it should be serving off

pvirani · Answer

hey guys I was on this basically all of last night still debugging Now at a point where I need to rule out the possibility that the App isn t up and running before going any further What s the best way to do that Once I can verify that I ll continue to debug AWS networking issues

zwass · Answer

Can you connect into a pod in the cluster and `curl` the internal address

zwass · Answer

In my experience this kind of issue on AWS is often associated with an improperly configured security group Very painful to debug

pvirani · Answer

yeahhhh that s literally what I was doing last night and the first thing I made sure was that My Load Balancer was allowing inbound traffic from all IPs on port 80 and 443

pvirani · Answer

pod info incoming

pvirani · Answer

``` kc logs n fleetdm webserver f fleetdm webserver 85484d6c79 v4h6h warning service fleetdm overwriting environment variable FLEET MYSQL PASSWORD warning service fleetdm overwriting environment variable FLEET REDIS PASSWORD``` ``` kc exec fleetdm webserver 85484d6c79 v4h6h n fleetdm webserver fleet config dump mysql protocol tcp address username fleet password password path database fleetdatabase tls cert tls key tls ca tls server name tls config max open conns 50 max idle conns 50 conn max lifetime 0 mysql read replica protocol tcp address username fleet password password path database fleet tls cert tls key tls ca tls server name tls config max open conns 50 max idle conns 50 conn max lifetime 0 redis address password database 0 use tls false duplicate results false connect timeout 5s keep alive 10s connect retry attempts 0 cluster follow redirections false server address 0 0 0 0 443 cert tools osquery fleet crt key tools osquery fleet key tls true tls compatibility intermediate url prefix keepalive true auth bcrypt cost 12 salt key size 24 app token key size 24 invite token validity period 120h0m0s session key size 64 duration 4h0m0s osquery node key size 24 host identifier provided enroll cooldown 0s status log plugin filesystem result log plugin filesystem label update interval 1h0m0s policy update interval 1h0m0s detail update interval 1h0m0s status log file result log file enable log rotation false max jitter percent 10 logging debug false json false disable banner false firehose region endpoint url access key id secret access key sts assume role arn status stream result stream kinesis region endpoint url access key id secret access key sts assume role arn status stream result stream lambda region access key id secret access key sts assume role arn status function result function s3 bucket prefix access key id secret access key sts assume role arn pubsub project status topic result topic add attributes false filesystem status log file tmp osquery status result log file tmp osquery result enable log rotation false enable log compression false license key vulnerabilities databases path periodicity 1h0m0s cpe database url cve feed prefix url current instance checks auto disable data sync false```

zwass · Answer

Maybe something like `kc exec fleetdm webserver 85484d6c79 v4h6h n fleetdm webserver curl <http localhost 8080>`

pvirani · Answer

yeah can t curl lol coz no perms updating the dockerfile with run add apk url just now and pushing the image then will spin up a new pod gimme a few

pvirani · Answer

``` kc exec fleetdm webserver 7fd4b65998 5njdc n fleetdm webserver curl <https 0 0 0 0 443> Total Received Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 0 curl 7 Failed to connect to 0 0 0 0 port 443 after 0 ms Connection refused command terminated with exit code 7```

pvirani · Answer

omfg of all the things I checked last night one that I didn t was aws controller lb logs Just did that and tada tada now figuring out how to make it find port https on the server and not http ``` level error ts 1633716912 7189045 logger controller msg Reconciler error reconcilerGroup elbv2 k8s aws reconcilerKind TargetGroupBinding controller targetGroupBinding name fleetdm webserver namespace fleetdm webserver error unable to find port http on service fleetdm webserver fleetdm webserver ```

zwass · Answer

Were you able to verify that the server is listening

pvirani · Answer

yeah so I tried curling to permutation combinations of http https localhost 0 0 0 0 443 80 8080 and ain t able to curl into nothing

pvirani · Answer

so yeah looks like the server is not listening either

zwass · Answer

Do you have logs from the Fleet server pod

pvirani · Answer

quick question Is Fleet set up such that that webserver won t get up and running until the mysql DB connection is healthy

pvirani · Answer

yes I do

zwass · Answer

Can you paste some of those logs

zwass · Answer

gt webserver won t get up and running until the mysql DB connection is healthy Yes it should not start up if it can t connect to MySQL

zwass · Answer

Logs would look like ```ts=2021 10 08T20 45 24 715901Z mysql= could not connect to db dial tcp 1 3306 connect connection refused sleeping 0s ts=2021 10 08T20 45 24 716481Z mysql= could not connect to db dial tcp 1 3306 connect connection refused sleeping 1s ```

pvirani · Answer

so fyi I got the logs only after I se ` logging debug=true` this flag Without it it was showing nothing

pvirani · Answer

``` kc exec fleetdm webserver 7cdc49c6d6 z7h54 n fleetdm webserver chamber exec fleetdm fleet serve logging debug=true server tls=false warning service fleetdm overwriting environment variable FLEET REDIS PASSWORD warning service fleetdm overwriting environment variable FLEET MYSQL PASSWORD ts=2021 10 08T20 32 26 026929776Z mysql= could not connect to db Error 1045 Access denied for user fleet 10 80 93 203 using password YES sleeping 0s ts=2021 10 08T20 32 26 030594949Z mysql= could not connect to db Error 1045 Access denied for user fleet 10 80 93 203 using password YES sleeping 1s ```

zwass · Answer

Looks like this exec command is probably not working as expected with your env vars

pvirani · Answer

ensured the credentials are loading correct

pvirani · Answer

it is

pvirani · Answer

wait i ll show you da proof

zwass · Answer

Wasn t your db username something other than `fleet`

pvirani · Answer

it was but I changed it to `fleet` and redeployed to keep things simple

zwass · Answer

Ah I see

pvirani · Answer

but I guess things still aren t simple enough coz my db password is still a long complex string with special characters like security engineers recommend it to be smile and I am thinking of making that super simple now as well for debugging purposes

zwass · Answer

Yeah making it simple for debugging could help

zwass · Answer

Maybe try `kc exec fleetdm webserver 7cdc49c6d6 z7h54 n fleetdm webserver chamber exec fleetdm fleet serve logging debug=true server tls=false mysql password= lt PASTE HERE gt `

zwass · Answer

you ll want to change the password after doing that once you get it working as it s going to leak the password into shell history

pvirani · Answer

yeah for sure

pvirani · Answer

Ran this gt `kc exec fleetdm webserver 7cdc49c6d6 z7h54 n fleetdm webserver chamber exec fleetdm fleet serve logging debug=true server tls=false mysql password= lt new pass gt ` there are no errors now but curl still doesn t work on 0 0 0 0 8080

pvirani · Answer

The earlier DB connection error has disappeared and logs also show only these two lines now ``` kc logs fleetdm webserver 7cdc49c6d6 z7h54 n fleetdm webserver warning service fleetdm overwriting environment variable FLEET MYSQL PASSWORD warning service fleetdm overwriting environment variable FLEET REDIS PASSWORD sec tooling stage write sec tooling stage us west 2 fleetdm prima virani Primas MacBook Pro terracode ops ``` but curling still doesn t work ``` kc exec fleetdm webserver 7cdc49c6d6 z7h54 n fleetdm webserver curl <http 0 0 0 0 8080> Total Received Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 0 curl 7 Failed to connect to 0 0 0 0 port 8080 after 0 ms Connection refused command terminated with exit code 7```

zwass · Answer

After you run `kc exec fleetdm webserver 7cdc49c6d6 z7h54 n fleetdm webserver chamber exec fleetdm fleet serve logging debug=true server tls=false mysql password= lt new pass gt ` do you get any logs You should see something like ```level=info ts=2021 10 08T21 34 57 476844Z component=crons cron=vulnerabilities vulnerabilityscanning= not configured ts=2021 10 08T21 34 57 501393Z transport=https address=0 0 0 0 8080 msg=listening```

pvirani · Answer

no logs

pvirani · Answer

at all

zwass · Answer

Does the process exit

pvirani · Answer

I believe so but how do I confirm

zwass · Answer

Do you end up back at a shell prompt

pvirani · Answer

yes I do

zwass · Answer

Can you just exec `sh` on that pod

pvirani · Answer

let me try

zwass · Answer

And then try the `fleet serve` command from that shell

pvirani · Answer

btw I changed the redis password too and made it like as simple redis would let me make it let it finish deploying and I will try redeploying the pod and restarting everything

pvirani · Answer

would it hurt if I remove this flag from my yaml file no right coz it defaults to 0 0 0 0 8080 anyways ``` name FLEET SERVER ADDRESS value 0 0 0 0 8080 ```

zwass · Answer

Yeah that should be the default

pvirani · Answer

now the logs aren t populating at all

pvirani · Answer

new config ``` kc exec fleetdm webserver 57bc6745 kj6h4 n fleetdm webserver chamber exec fleetdm printenv PATH= usr local sbin usr local bin usr sbin usr bin sbin bin HOSTNAME=fleetdm webserver 57bc6745 kj6h4 FLEET MYSQL ADDRESS= FLEET MYSQL DATABASE=fleetdatabase FLEET SERVER TLS=false NODE IP=10 80 65 190 AWS DEFAULT REGION=us west 2 AWS WEB IDENTITY TOKEN FILE= var run secrets eks amazonaws com serviceaccount token FLEET REDIS ADDRESS= FLEET LOGGING DEBUG=true SEGMENT CELL=core AWS REGION=us west 2 AWS ROLE ARN=arn aws iam 169172804835 role fleetdm usw2 eks fleetdm webserver KUBERNETES PORT 443 TCP= KUBERNETES PORT 443 TCP PROTO=tcp KUBERNETES PORT 443 TCP PORT=443 KUBERNETES PORT 443 TCP ADDR=172 20 0 1 KUBERNETES SERVICE HOST=172 20 0 1 KUBERNETES SERVICE PORT=443 KUBERNETES SERVICE PORT HTTPS=443 KUBERNETES PORT= HOME= home fleet FLEET MYSQL PASSWORD= FLEET REDIS PASSWORD= ```

pvirani · Answer

``` kc logs n fleetdm webserver f fleetdm webserver 57bc6745 kj6h4 ```

pvirani · Answer

``` kc exec fleetdm webserver 57bc6745 kj6h4 n fleetdm webserver sh ``` no output nothing

zwass · Answer

Stretching my k8s knowledge here but I think you need ` i t` to connect to an interactive shell

pvirani · Answer

``` kc exec ti fleetdm webserver 57bc6745 tw4kh n fleetdm webserver bin sh $ ```

zwass · Answer

Can you then try your full fleet serve command

pvirani · Answer

so the service seems running when I do top ```Load average 0 29 0 18 0 18 2 995 106 PID PPID USER STAT VSZ VSZ CPU CPU COMMAND 1 0 fleet S 1084m 14 1 0 fleet serve 94 0 fleet S 1668 0 0 0 bin sh 106 94 fleet R 1600 0 0 0 top```

pvirani · Answer

on doing netstat I do see the connections with mysql redis working ``` $ netstat Active Internet connections w o servers Proto Recv Q Send Q Local Address Foreign Address State tcp 0 0 fleetdm webserver 57bc6745 tw4kh 38904 ip 10 80 140 145 us west 2 compute internal mysql ESTABLISHED tcp 0 0 fleetdm webserver 57bc6745 tw4kh 39298 ip 10 80 50 194 us west 2 compute internal redis ESTABLISHED Active UNIX domain sockets w o servers Proto RefCnt Flags Type State I Node Path```

zwass · Answer

`netstat tulpn` do you see anything in state `LISTEN` on 8080

pvirani · Answer

nope

pvirani · Answer

and apparently status S above means the service is in interruptible sleep right now

pvirani · Answer

in which directory do I find the service logs

zwass · Answer

Fleet logs to stderr I don t know what k8s would do with that

zwass · Answer

Can you try running the `fleet serve` command in this shell and see if there s any output

pvirani · Answer

yeah doing it now

pvirani · Answer

``` $ fleet serve ts=2021 10 08T23 57 38 521866342Z mysql= could not connect to db Error 1045 Access denied for user fleet 10 80 94 227 using password NO sleeping 0s ts=2021 10 08T23 57 38 525573097Z mysql= could not connect to db Error 1045 Access denied for user fleet 10 80 94 227 using password NO sleeping 1s ts=2021 10 08T23 57 39 529444031Z mysql= could not connect to db Error 1045 Access denied for user fleet 10 80 94 227 using password NO sleeping 2s ts=2021 10 08T23 57 41 533580427Z mysql= could not connect to db Error 1045 Access denied for user fleet 10 80 94 227 using password NO sleeping 3s ts=2021 10 08T23 57 44 537292093Z mysql= could not connect to db Error 1045 Access denied for user fleet 10 80 94 227 using password NO sleeping 4s ts=2021 10 08T23 57 48 541150984Z mysql= could not connect to db Error 1045 Access denied for user fleet 10 80 94 227 using password NO sleeping 5s ```

zwass · Answer

Now try adding the flags for the db connection

pvirani · Answer

yeah constructing the whole flags rn

pvirani · Answer

```~ $ fleet serve mysql database=fleetdatabase redis address= mysql address=fleetdatabase crq mysql password= redis password= logging debug=true serve r tls=false ```

pvirani · Answer

nothing no results doesn t fail either it s just sitting there

zwass · Answer

Can you `ping <http master fleet cache l5ryax usw2 cache amazonaws com|master fleet cache l5ryax usw2 cache amazonaws com>`

zwass · Answer

and also `ping <http fleetdatabase crqbc9r8uf32 us west 2 rds amazonaws com|fleetdatabase crqbc9r8uf32 us west 2 rds amazonaws com>`

pvirani · Answer

``` ping PING 10 80 50 194 56 data bytes Request timeout for icmp seq 0 Request timeout for icmp seq 1 Request timeout for icmp seq 2 Request timeout for icmp seq 3 Request timeout for icmp seq 4 ^C ping statistics 6 packets transmitted 0 packets received 100 0 packet loss sec tooling stage write sec tooling stage us west 2 fleetdm prima virani Primas MacBook Pro terracode security ping PING 10 80 140 145 56 data bytes Request timeout for icmp seq 0 Request timeout for icmp seq 1 Request timeout for icmp seq 2 Request timeout for icmp seq 3 Request timeout for icmp seq 4 ^C ping statistics 6 packets transmitted 0 packets received 100 0 packet loss```

zwass · Answer

Oh well that probably explains some things Looks like there s no connectivity to the MySQL or Redis instances Fleet should definitely be helping you out more here though I ll file an issue for that

pvirani · Answer

oh wait wait

pvirani · Answer

that was run from my localhost let me run that fromt the pod

zwass · Answer

So do the security groups allow outbound connections from the cluster to the db redis And do db redis allow inbound connections from the cluster

zwass · Answer

oh

zwass · Answer

lmk how it goes then

pvirani · Answer

yes I ve set the redis and DB inbound rules to allow any any for the meanwhile

pvirani · Answer

and all the security groups allow outbound any any

pvirani · Answer

ugh no perms to ping fleet user has limited perms

pvirani · Answer

deploying a new image with sudo permissions for the fleet user at least for debugging purposes because it s limiting during debugging otherwise

zwass · Answer

Sounds good Alternatively could deploy an `alpine` image or similar

pvirani · Answer

it s already running on Alpine

pvirani · Answer

```FROM segment chamber 2 10 6 as chamber FROM fleetdm fleet v4 3 2 as fleet FROM fleetdm fleetctl v4 3 2 as fleetctl FROM alpine 3 14 2 RUN apk update add ca certificates RUN apk add curl RUN apk add no cache su exec Create FleetDM group and user RUN addgroup S fleet amp amp adduser S fleet G fleet Add Chamber Binary COPY from=chamber chamber usr local bin chamber Add Fleet Binary COPY from=fleet usr bin usr bin COPY from=fleetctl usr bin usr bin USER fleet CMD fleet serve ```

pvirani · Answer

I ll stop pinging if Im the only thing between you and the weekend right now smile Thanks for all the help thus far

zwass · Answer

Ha thank you I m still wrapping up some other things so feel free to keep it up

zwass · Answer

I ll let you know when I gotta go

zwass · Answer

I was thinking an `alpine` base image where you d be `root` But your idea of removing the `USER fleet` line should work fine

pvirani · Answer

actually I don t need to ping to check the connection

pvirani · Answer

```$ netstat Active Internet connections w o servers Proto Recv Q Send Q Local Address Foreign Address State tcp 0 0 fleetdm webserver 57bc6745 tw4kh 38904 ip 10 80 140 145 us west 2 compute internal mysql ESTABLISHED tcp 0 0 fleetdm webserver 57bc6745 tw4kh 39298 ip 10 80 50 194 us west 2 compute internal redis ESTABLISHED ```

pvirani · Answer

we already know the connection is established

pvirani · Answer

and it stays established

zwass · Answer

I think those are different addresses is that intentional

pvirani · Answer

which addresses

pvirani · Answer

the addresses are of the redis and mysql hosted in aws

zwass · Answer

I see Foreign Address `ip 10 80 140 145 us west 2 compute internal mysql` and `ip 10 80 50 194 us west 2 compute internal redis` those don t seem to match the addresses used in <https osquery slack com archives C01DXJL16D8 p1633738072026500 thread ts=1633632344 474100 amp cid=C01DXJL16D8>

zwass · Answer

Looks like they are probably some DNS internal to the cluster or something I don t know if that could make a difference

pvirani · Answer

yeah so I turned on logging on our database and here s a warning I saw May or may not mean the database can t connect back to the pod ```2021 10 09T00 12 50 627 07 00 2021 10 09T07 12 50 627260Z 231 Warning MY 010055 Server IP address 10 80 94 227 could not be resolved Name or service not known``` but also confused by the fact that netstat shows connection established when `fleet serve` is running we hardcoded all the flags in env for now If it isn t able to connect successfully how would the connection be in established state thinking face ``` $ netstat Active Internet connections w o servers Proto Recv Q Send Q Local Address Foreign Address State tcp 0 0 fleetdm webserver 589fc79fdf 5ltl9 38136 ip 10 80 50 194 us west 2 compute internal redis ESTABLISHED tcp 0 0 fleetdm webserver 589fc79fdf 5ltl9 37742 ip 10 80 140 145 us west 2 compute internal mysql ESTABLISHED tcp 0 0 fleetdm webserver 589fc79fdf 5ltl9 47692 ip 10 80 50 194 us west 2 compute internal redis ESTABLISHED ``` then `fleet serve` runs and runs and runs without any errors or any info about where it s getting stuck and we ctrl+c out of it and then I run netstat again and look what happens ``` $ netstat Active Internet connections w o servers Proto Recv Q Send Q Local Address Foreign Address State tcp 0 0 fleetdm webserver 589fc79fdf 5ltl9 47692 ip 10 80 50 194 us west 2 compute internal redis ESTABLISHED Active UNIX domain sockets w o servers Proto RefCnt Flags Type State I Node Path``` the connection with redis remains but the one with mysql goes away I don t know if any of this even related or points to anything Just sharing it here for my own better understanding for most parts I ve just been RTFMing as hard as possible

pvirani · Answer

Hey team I just confirmed the database connection is working fine I tried connecting using the following commands and here s what worked and what didn t work ``` $ mysql host port 3306 database fleetdatabase password lt my password gt ``` ^^ This did not work and gave an error saying `ERROR 1049 42000 Unknown database lt my password gt ` ```$ mysql host port 3306 database fleetdatabase password Enter password Reading table information for completion of table and column names You can turn off this feature to get a quicker startup with A Welcome to the MariaDB monitor Commands end with or g Your MySQL connection id is 1002 Server version 8 0 25 Source distribution Copyright c 2000 2018 Oracle MariaDB Corporation Ab and others Type help or h for help Type c to clear the current input statement MySQL fleetdatabase gt ``` ^^^ This worked

pvirani · Answer

but fleet serve still does not work disappointed

Martavis Parker · Answer

What is the error for `fleet serve`

zwass · Answer

Still just hangs with no output

pvirani · Answer

``` kc logs n fleetdm webserver fleetdm webserver 5576655d6f 9vsd7 level=info ts=2021 10 13T17 33 43 324015438Z component=crons cron=vulnerabilities vulnerabilityscanning= not configured ts=2021 10 13T17 33 43 725825861Z transport=http address=0 0 0 0 80 msg=listening``` Finally tada but ``` $ curl <http 0 0 0 0 80> lt a href= setup gt Temporary Redirect lt a gt ```

pvirani · Answer

The problem was Redis In AWS Redis lets you set auth token only if TLS is enabled and I luckily noticed a question about redis TLS support in Fleet that someone had posted here yesterday which kind of gave me a lead to start looking in that direction although netstat output saying the redis connection is ESTABLISHED was a bit misleading initially But yeah I tried connecting to the redis cache cluster with `redis cli` and it showed the same behaviour as the app No logs and the connection was simply hanging which made me even surer that it has to be redis related Today I deployed a new cluster without auth and TLS disabled and it worked slightly smiling face Now figuring out what s up with that temporary redirect

zwass · Answer

tada

zwass · Answer

You re being redirected to ` setup` to create the initial user

zwass · Answer

I did not realize that AWS Redis had that limitation with auth We will work on getting TLS support into 4 5 0

pvirani · Answer

ohhhh cool can I setup via cli as well

zwass · Answer

Yep `fleetctl setup`

pvirani · Answer

``` $ fleetctl setup email name Prima password org name Twilio error creating Fleet API client handler address must start with https for remote connections``` I don t want to set up TLS on fleet server for now The traffic from the internet will hit the load balancer on https but from load balancer gt fleetdm webserver I want it to be on http only so I ve set up my webserver on but I get this error at the time of setup what do I do

zwass · Answer

Try this ```fleetctl config set tls skip verify=true address=<http localhost 80>``` and then try it again

pvirani · Answer

tada ``` $ curl < DOCTYPE html> lt html gt lt head gt lt meta charset= UTF 8 gt lt meta name= viewport content= width=device width initial scale=1 gt lt meta name= robots content= noindex gt lt link rel= stylesheet type= text css href= assets bundle 10a9feb55abb44f6de13 css gt lt link rel= shortcut icon href= assets favicon ico gt lt title gt Fleet for osquery lt title gt lt script type= text javascript gt var urlPrefix = lt script gt lt head gt lt body gt lt div id= app gt lt div gt lt script async defer src= assets bundle f79228ceeede741c7007 js onload= this parentElement removeChild this gt lt script gt lt script gt document addEventListener touchstart function false lt script gt lt body gt lt html gt ```

zwass · Answer

Nice

pvirani · Answer

last piece of the puzzle I think html ain t rendering on the webpage thinking face

zwass · Answer

Can you open the browser devtools and see what the HTTP responses look like

pvirani · Answer

yeah was literally doing that this is what the page source looks like looking further now

zwass · Answer

Ah sorry the network tab

zwass · Answer

Also any errors in the JS console

pvirani · Answer

yeah was just fetching you the screenshot lol I see a bunch of 400s

pvirani · Answer

first thing I did was comparing the curl response on the webserver to the one on my local machine which is running successfully rn with fleetctl preview

pvirani · Answer

that one shows no 400s the webserver one shows a bunch of 400s

zwass · Answer

Is there any response body in one of those 400s

zwass · Answer

Actually can you paste me that URL I can just look for myself

zwass · Answer

and typing it out manually from your screenshot seems painful

pvirani · Answer

```Request URL Request Method GET Status Code 400 Remote Address 52 13 90 185 443 Referrer Policy strict origin when cross origin``` Looks like it s got to do with traffic between my loadbalancer and the app ```$ nslookup Server 169 254 20 10 Address 169 254 20 10 53 Non authoritative answer Can t find No answer Non authoritative answer Name Address 52 40 25 210 Name Address 54 203 88 105 Name Address 52 13 90 185 Name Address 44 236 167 10```

pvirani · Answer

and yes sorry my bad <http fleet20210924001411691500000003 825359854 us west 2 elb amazonaws com|fleet20210924001411691500000003 825359854 us west 2 elb amazonaws com> this is the URL

zwass · Answer

Ah yeah thank you

zwass · Answer

I can see the 400s

pvirani · Answer

yeah Imm looking at the security groups again

zwass · Answer

Are there any errors in the Fleet server logs

zwass · Answer

I suspect this 400 is from the load balancer

pvirani · Answer

yeah exactly

zwass · Answer

Seems like any request to a path other than ` ` results in a 400

zwass · Answer

I think it s probably not security group because the ` ` request is definitely making it to the Fleet server

zwass · Answer

Is there something in the ingress for k8s that needs to be configured to allow certain paths Again at the limit of my k8s knowledge

pvirani · Answer

yeah you know when we were on the setup stage the LB url was also successfully getting redirected to setup even so Im also a little confused there

pvirani · Answer

looking at k8s stuff now

zwass · Answer

I just took a double check and there s nowhere that Fleet explicitly returns a 400 It s possible something in the underlying library code could be doing so

pvirani · Answer

I have a suspicion of what might be causing it Imma flipping it now and then will report back if when it works

pvirani · Answer

<https fleetdm segment build > tada

zwass · Answer

What was the missing piece

pvirani · Answer

TL DR The Load Balancer was set up with a custom internal module and it was set to forward only ` ` path to my webserver and for all the paths inside that path it would return a 400 Fixing that fixed everything else long bumpy road but finally we here confetti ball tada rocket Thanks for all the help Zach and Martavis Definitely could not have done this without both of you can t thank you enough Blogpost incoming soon to help the community set up a similar structure

zwass · Answer

Thank you for your patience and persistence Blog post will be very much appreciated smile let us know when it s up and we will help promote

pvirani · Answer

Im so excited honestly I was so flustered there in between for a while that I was literally forgetting my debugging basics even but now we here