pvirani
10/07/2021, 6:45 PMUnknown database 'fleet'
. Works fine otherwise. Any ideas why?apiVersion: <http://platform.segment.com/v1alpha1|platform.segment.com/v1alpha1>
kind: SegmentApplicationExperimental
metadata:
name: fleetdm-webserver
namespace: fleetdm-webserver
labels:
app: fleetdm-webserver
spec:
targets:
- name: fleetdm-stage-usw2
cluster: sec-tooling-stage:us-west-2:fleetdm
targetGroupBinding:
autoDiscover: true
replicatedService: &replicatedService
iamRoleName: fleetdm-webserver
autoScale: &autoScale
minReplicas: 1
maxReplicas: 6
resources:
- resource: cpu
utilization: 15
- resource: memory
utilization: 15
maincontainers:
- name: fleetdm-webserver
ctlstore:
disabled: true
imageRegistry: <http://528451384384.dkr.ecr.us-west-2.amazonaws.com|528451384384.dkr.ecr.us-west-2.amazonaws.com>
imageName: fleetdm/fleet
command:
- chamber
- exec
- fleetdm
- --
- fleet
- serve
ports:
- containerPort: 443
env:
- name: FLEET_MYSQL_ADDRESS
value: <http://fleetdatabase.crqbc9r8uf32.us-west-2.rds.amazonaws.com:3306|fleetdatabase.crqbc9r8uf32.us-west-2.rds.amazonaws.com:3306>
- name: FLEET_MYSQL_USERNAME
value: fleet_rds
- name: FLEET_REDIS_ADDRESS
value: <http://master.fleet-cache.l5ryax.usw2.cache.amazonaws.com|master.fleet-cache.l5ryax.usw2.cache.amazonaws.com>
- name: FLEET_SERVER_ADDRESS
value: "0.0.0.0:443"
This is my Kubernetes App config fileMartavis Parker
10/07/2021, 7:20 PMmysql_database
flag: https://github.com/fleetdm/fleet/blob/main/docs/02-Deploying/02-Configuration.md#mysql_databasepvirani
10/07/2021, 7:26 PMenv:
- name: FLEET_MYSQL_ADDRESS
value: <http://fleetdatabase.crqbc9r8uf32.us-west-2.rds.amazonaws.com:3306|fleetdatabase.crqbc9r8uf32.us-west-2.rds.amazonaws.com:3306>
- name: FLEET_MYSQL_DATABASE
value: <http://fleetdatabase.crqbc9r8uf32.us-west-2.rds.amazonaws.com:3306|fleetdatabase.crqbc9r8uf32.us-west-2.rds.amazonaws.com:3306>
- name: FLEET_MYSQL_USER
value: "fleet_rds"
- name: FLEET_REDIS_ADDRESS
value: <http://master.fleet-cache.l5ryax.usw2.cache.amazonaws.com|master.fleet-cache.l5ryax.usw2.cache.amazonaws.com>
- name: FLEET_SERVER_ADDRESS
value: "0.0.0.0:443"
Added it like so.
It doesn't seem to be picking up the username and keeps saying
ts=2021-10-07T19:30:24.561132517Z mysql="could not connect to db: Error 1045: Access denied for user 'fleet'@'10.80.94.227' (using password: YES), sleeping 14s"
% kc exec fleetdm-webserver-58c85b9bf7-kqfdr -n fleetdm-webserver -- fleet serve
ts=2021-10-07T20:03:13.427339705Z mysql="could not connect to db: Error 1045: Access denied for user 'fleet'@'10.80.79.187' (using password: YES)
however just after adding the mysql_username flag it starts complaining about the database being unknown
% kc exec fleetdm-webserver-58c85b9bf7-kqfdr -n fleetdm-webserver -- fleet serve --mysql_username=fleet_rds
ts=2021-10-07T20:04:42.621780497Z mysql="could not connect to db: Error 1049: Unknown database '<http://fleetdatabase.crqbc9r8uf32.us-west-2.rds.amazonaws.com:3306|fleetdatabase.crqbc9r8uf32.us-west-2.rds.amazonaws.com:3306>', sleeping 0s"
zwass
CREATE DATABASE fleet
within your MySQL instance?FLEET_MYSQL_ADDRESS
refers to the address the MySQL server is listening on, while FLEET_MYSQL_DATABASE
refers to the logical database within MySQL (eg. one created in MySQL with CREATE DATABASE
and accessed with USE DATABASE
)pvirani
10/07/2021, 10:00 PMBenjamin Edwards
10/07/2021, 11:19 PMpvirani
10/07/2021, 11:25 PMzwass
curl
the internal address?pvirani
10/08/2021, 4:53 PM% kc logs -n fleetdm-webserver -f fleetdm-webserver-85484d6c79-v4h6h
warning: service fleetdm overwriting environment variable FLEET_MYSQL_PASSWORD
warning: service fleetdm overwriting environment variable FLEET_REDIS_PASSWORD
% kc exec fleetdm-webserver-85484d6c79-v4h6h -n fleetdm-webserver -- fleet config_dump
mysql:
protocol: tcp
address: <http://fleetdatabase.crqbc9r8uf32.us-west-2.rds.amazonaws.com:3306|fleetdatabase.crqbc9r8uf32.us-west-2.rds.amazonaws.com:3306>
username: fleet
password: ""
password_path: ""
database: fleetdatabase
tls_cert: ""
tls_key: ""
tls_ca: ""
tls_server_name: ""
tls_config: ""
max_open_conns: 50
max_idle_conns: 50
conn_max_lifetime: 0
mysql_read_replica:
protocol: tcp
address: ""
username: fleet
password: ""
password_path: ""
database: fleet
tls_cert: ""
tls_key: ""
tls_ca: ""
tls_server_name: ""
tls_config: ""
max_open_conns: 50
max_idle_conns: 50
conn_max_lifetime: 0
redis:
address: <http://master.fleet-cache.l5ryax.usw2.cache.amazonaws.com:6379|master.fleet-cache.l5ryax.usw2.cache.amazonaws.com:6379>
password: ""
database: 0
use_tls: false
duplicate_results: false
connect_timeout: 5s
keep_alive: 10s
connect_retry_attempts: 0
cluster_follow_redirections: false
server:
address: 0.0.0.0:443
cert: ./tools/osquery/fleet.crt
key: ./tools/osquery/fleet.key
tls: true
tls_compatibility: intermediate
url_prefix: ""
keepalive: true
auth:
bcrypt_cost: 12
salt_key_size: 24
app:
token_key_size: 24
invite_token_validity_period: 120h0m0s
session:
key_size: 64
duration: 4h0m0s
osquery:
node_key_size: 24
host_identifier: provided
enroll_cooldown: 0s
status_log_plugin: filesystem
result_log_plugin: filesystem
label_update_interval: 1h0m0s
policy_update_interval: 1h0m0s
detail_update_interval: 1h0m0s
status_log_file: ""
result_log_file: ""
enable_log_rotation: false
max_jitter_percent: 10
logging:
debug: false
json: false
disable_banner: false
firehose:
region: ""
endpoint_url: ""
access_key_id: ""
secret_access_key: ""
sts_assume_role_arn: ""
status_stream: ""
result_stream: ""
kinesis:
region: ""
endpoint_url: ""
access_key_id: ""
secret_access_key: ""
sts_assume_role_arn: ""
status_stream: ""
result_stream: ""
lambda:
region: ""
access_key_id: ""
secret_access_key: ""
sts_assume_role_arn: ""
status_function: ""
result_function: ""
s3:
bucket: ""
prefix: ""
access_key_id: ""
secret_access_key: ""
sts_assume_role_arn: ""
pubsub:
project: ""
status_topic: ""
result_topic: ""
add_attributes: false
filesystem:
status_log_file: /tmp/osquery_status
result_log_file: /tmp/osquery_result
enable_log_rotation: false
enable_log_compression: false
license:
key: ""
vulnerabilities:
databases_path: ""
periodicity: 1h0m0s
cpe_database_url: ""
cve_feed_prefix_url: ""
current_instance_checks: auto
disable_data_sync: false
zwass
kc exec fleetdm-webserver-85484d6c79-v4h6h -n fleetdm-webserver -- curl <http://localhost:8080>
?pvirani
10/08/2021, 5:02 PM% kc exec fleetdm-webserver-7fd4b65998-5njdc -n fleetdm-webserver -- curl <https://0.0.0.0:443>
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
curl: (7) Failed to connect to 0.0.0.0 port 443 after 0 ms: Connection refused
command terminated with exit code 7
{
"level": "error",
"ts": 1633716912.7189045,
"logger": "controller",
"msg": "Reconciler error",
"reconcilerGroup": "elbv2.k8s.aws",
"reconcilerKind": "TargetGroupBinding",
"controller": "targetGroupBinding",
"name": "fleetdm-webserver",
"namespace": "fleetdm-webserver",
"error": "unable to find port http on service fleetdm-webserver/fleetdm-webserver"
}
zwass
pvirani
10/08/2021, 7:26 PMzwass
pvirani
10/08/2021, 8:52 PMzwass
webserver won't get up and running until the mysql DB connection is healthy?Yes it should not start up if it can't connect to MySQL
ts=2021-10-08T20:45:24.715901Z mysql="could not connect to db: dial tcp [::1]:3306: connect: connection refused, sleeping 0s"
ts=2021-10-08T20:45:24.716481Z mysql="could not connect to db: dial tcp [::1]:3306: connect: connection refused, sleeping 1s"
pvirani
10/08/2021, 8:53 PM--logging_debug=true
this flag. Without it it was showing nothing% kc exec fleetdm-webserver-7cdc49c6d6-z7h54 -n fleetdm-webserver -- chamber exec fleetdm -- fleet serve --logging_debug=true --server_tls=false
warning: service fleetdm overwriting environment variable FLEET_REDIS_PASSWORD
warning: service fleetdm overwriting environment variable FLEET_MYSQL_PASSWORD
ts=2021-10-08T20:32:26.026929776Z mysql="could not connect to db: Error 1045: Access denied for user 'fleet'@'10.80.93.203' (using password: YES), sleeping 0s"
ts=2021-10-08T20:32:26.030594949Z mysql="could not connect to db: Error 1045: Access denied for user 'fleet'@'10.80.93.203' (using password: YES), sleeping 1s"
zwass
pvirani
10/08/2021, 8:54 PMzwass
fleet
?pvirani
10/08/2021, 8:55 PMfleet
and redeployed to keep things simplezwass
pvirani
10/08/2021, 8:55 PMzwass
kc exec fleetdm-webserver-7cdc49c6d6-z7h54 -n fleetdm-webserver -- chamber exec fleetdm -- fleet serve --logging_debug=true --server_tls=false --mysql_password='<PASTE HERE>'
pvirani
10/08/2021, 8:57 PMkc exec fleetdm-webserver-7cdc49c6d6-z7h54 -n fleetdm-webserver -- chamber exec fleetdm -- fleet serve --logging_debug=true --server_tls=false --mysql_password='<new_pass>'
... there are no errors now but curl still doesn't work on 0.0.0.0:8080% kc logs fleetdm-webserver-7cdc49c6d6-z7h54 -n fleetdm-webserver
warning: service fleetdm overwriting environment variable FLEET_MYSQL_PASSWORD
warning: service fleetdm overwriting environment variable FLEET_REDIS_PASSWORD
(sec-tooling-stage-write/sec-tooling-stage:us-west-2:fleetdm) prima.virani@Primas-MacBook-Pro terracode-ops %
but curling still doesn't work
% kc exec fleetdm-webserver-7cdc49c6d6-z7h54 -n fleetdm-webserver -- curl <http://0.0.0.0:8080>
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
curl: (7) Failed to connect to 0.0.0.0 port 8080 after 0 ms: Connection refused
command terminated with exit code 7
zwass
kc exec fleetdm-webserver-7cdc49c6d6-z7h54 -n fleetdm-webserver -- chamber exec fleetdm -- fleet serve --logging_debug=true --server_tls=false --mysql_password='<new_pass>'
do you get any logs? You should see something like
level=info ts=2021-10-08T21:34:57.476844Z component=crons cron=vulnerabilities vulnerabilityscanning="not configured"
ts=2021-10-08T21:34:57.501393Z transport=https address=0.0.0.0:8080 msg=listening
pvirani
10/08/2021, 9:47 PMzwass
pvirani
10/08/2021, 9:57 PMzwass
pvirani
10/08/2021, 9:58 PMzwass
sh
on that pod?pvirani
10/08/2021, 9:58 PMzwass
fleet serve
command from that shellpvirani
10/08/2021, 9:59 PM- name: FLEET_SERVER_ADDRESS
value: "0.0.0.0:8080"
zwass
pvirani
10/08/2021, 10:23 PM% kc exec fleetdm-webserver-57bc6745-kj6h4 -n fleetdm-webserver -- chamber exec fleetdm -- printenv
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=fleetdm-webserver-57bc6745-kj6h4
FLEET_MYSQL_ADDRESS=<http://fleetdatabase.crqbc9r8uf32.us-west-2.rds.amazonaws.com:3306|fleetdatabase.crqbc9r8uf32.us-west-2.rds.amazonaws.com:3306>
FLEET_MYSQL_DATABASE=fleetdatabase
FLEET_SERVER_TLS=false
NODE_IP=10.80.65.190
AWS_DEFAULT_REGION=us-west-2
AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token
FLEET_REDIS_ADDRESS=<http://master.fleet-cache.l5ryax.usw2.cache.amazonaws.com:6379|master.fleet-cache.l5ryax.usw2.cache.amazonaws.com:6379>
FLEET_LOGGING_DEBUG=true
SEGMENT_CELL=core
AWS_REGION=us-west-2
AWS_ROLE_ARN=arn:aws:iam::169172804835:role/fleetdm.usw2.eks.fleetdm-webserver
KUBERNETES_PORT_443_TCP=<tcp://172.20.0.1:443>
KUBERNETES_PORT_443_TCP_PROTO=tcp
KUBERNETES_PORT_443_TCP_PORT=443
KUBERNETES_PORT_443_TCP_ADDR=172.20.0.1
KUBERNETES_SERVICE_HOST=172.20.0.1
KUBERNETES_SERVICE_PORT=443
KUBERNETES_SERVICE_PORT_HTTPS=443
KUBERNETES_PORT=<tcp://172.20.0.1:443>
HOME=/home/fleet
FLEET_MYSQL_PASSWORD=***
FLEET_REDIS_PASSWORD=***
% kc logs -n fleetdm-webserver -f fleetdm-webserver-57bc6745-kj6h4
% kc exec fleetdm-webserver-57bc6745-kj6h4 -n fleetdm-webserver -- sh
no output. nothingzwass
-i -t
to connect to an interactive shellpvirani
10/08/2021, 11:44 PM% kc exec -ti fleetdm-webserver-57bc6745-tw4kh -n fleetdm-webserver -- /bin/sh
/ $
zwass
pvirani
10/08/2021, 11:47 PMLoad average: 0.29 0.18 0.18 2/995 106
PID PPID USER STAT VSZ %VSZ CPU %CPU COMMAND
1 0 fleet S 1084m 14% 1 0% fleet serve
94 0 fleet S 1668 0% 0 0% /bin/sh
106 94 fleet R 1600 0% 0 0% top
/ $ netstat
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 fleetdm-webserver-57bc6745-tw4kh:38904 ip-10-80-140-145.us-west-2.compute.internal:mysql ESTABLISHED
tcp 0 0 fleetdm-webserver-57bc6745-tw4kh:39298 ip-10-80-50-194.us-west-2.compute.internal:redis ESTABLISHED
Active UNIX domain sockets (w/o servers)
Proto RefCnt Flags Type State I-Node Path
zwass
netstat -tulpn
do you see anything in state LISTEN
on 8080?pvirani
10/08/2021, 11:55 PMzwass
fleet serve
command in this shell and see if there's any output?pvirani
10/08/2021, 11:57 PM$ fleet serve
ts=2021-10-08T23:57:38.521866342Z mysql="could not connect to db: Error 1045: Access denied for user 'fleet'@'10.80.94.227' (using password: NO), sleeping 0s"
ts=2021-10-08T23:57:38.525573097Z mysql="could not connect to db: Error 1045: Access denied for user 'fleet'@'10.80.94.227' (using password: NO), sleeping 1s"
ts=2021-10-08T23:57:39.529444031Z mysql="could not connect to db: Error 1045: Access denied for user 'fleet'@'10.80.94.227' (using password: NO), sleeping 2s"
ts=2021-10-08T23:57:41.533580427Z mysql="could not connect to db: Error 1045: Access denied for user 'fleet'@'10.80.94.227' (using password: NO), sleeping 3s"
ts=2021-10-08T23:57:44.537292093Z mysql="could not connect to db: Error 1045: Access denied for user 'fleet'@'10.80.94.227' (using password: NO), sleeping 4s"
ts=2021-10-08T23:57:48.541150984Z mysql="could not connect to db: Error 1045: Access denied for user 'fleet'@'10.80.94.227' (using password: NO), sleeping 5s"
zwass
pvirani
10/08/2021, 11:59 PM~ $ fleet serve --mysql_database=fleetdatabase --redis_address=<http://master.fleet-cache.l5ryax.usw2.cache.amazonaws.com:6379|master.fleet-cache.l5ryax.usw2.cache.amazonaws.com:6379> --mysql_address=fleetdatabase.crq
<http://bc9r8uf32.us-west-2.rds.amazonaws.com:3306|bc9r8uf32.us-west-2.rds.amazonaws.com:3306> --mysql_password=*** --redis_password=*** --logging_debug=true --serve
r_tls=false
zwass
ping <http://master.fleet-cache.l5ryax.usw2.cache.amazonaws.com|master.fleet-cache.l5ryax.usw2.cache.amazonaws.com>
?ping <http://fleetdatabase.crqbc9r8uf32.us-west-2.rds.amazonaws.com|fleetdatabase.crqbc9r8uf32.us-west-2.rds.amazonaws.com>
pvirani
10/09/2021, 12:10 AM% ping <http://master.fleet-cache.l5ryax.usw2.cache.amazonaws.com|master.fleet-cache.l5ryax.usw2.cache.amazonaws.com>
PING <http://fleet-cache-001.fleet-cache.l5ryax.usw2.cache.amazonaws.com|fleet-cache-001.fleet-cache.l5ryax.usw2.cache.amazonaws.com> (10.80.50.194): 56 data bytes
Request timeout for icmp_seq 0
Request timeout for icmp_seq 1
Request timeout for icmp_seq 2
Request timeout for icmp_seq 3
Request timeout for icmp_seq 4
^C
--- <http://fleet-cache-001.fleet-cache.l5ryax.usw2.cache.amazonaws.com|fleet-cache-001.fleet-cache.l5ryax.usw2.cache.amazonaws.com> ping statistics ---
6 packets transmitted, 0 packets received, 100.0% packet loss
(sec-tooling-stage-write/sec-tooling-stage:us-west-2:fleetdm) prima.virani@Primas-MacBook-Pro terracode-security % ping <http://fleetdatabase.crqbc9r8uf32.us-west-2.rds.amazonaws.com|fleetdatabase.crqbc9r8uf32.us-west-2.rds.amazonaws.com>
PING <http://fleetdatabase.crqbc9r8uf32.us-west-2.rds.amazonaws.com|fleetdatabase.crqbc9r8uf32.us-west-2.rds.amazonaws.com> (10.80.140.145): 56 data bytes
Request timeout for icmp_seq 0
Request timeout for icmp_seq 1
Request timeout for icmp_seq 2
Request timeout for icmp_seq 3
Request timeout for icmp_seq 4
^C
--- <http://fleetdatabase.crqbc9r8uf32.us-west-2.rds.amazonaws.com|fleetdatabase.crqbc9r8uf32.us-west-2.rds.amazonaws.com> ping statistics ---
6 packets transmitted, 0 packets received, 100.0% packet loss
zwass
pvirani
10/09/2021, 12:11 AMzwass
pvirani
10/09/2021, 12:12 AMzwass
alpine
image or similarpvirani
10/09/2021, 12:21 AMFROM segment/chamber:2.10.6 as chamber
FROM fleetdm/fleet:v4.3.2 as fleet
FROM fleetdm/fleetctl:v4.3.2 as fleetctl
FROM alpine:3.14.2
RUN apk --update add ca-certificates
RUN apk add curl
RUN apk add --no-cache su-exec
# Create FleetDM group and user
RUN addgroup -S fleet && adduser -S fleet -G fleet
# Add Chamber Binary
COPY --from=chamber /chamber /usr/local/bin/chamber
# Add Fleet Binary
COPY --from=fleet /usr/bin/ /usr/bin/
COPY --from=fleetctl /usr/bin /usr/bin/
USER fleet
CMD ["fleet", "serve"]
zwass
alpine
base image where you'd be root
. But your idea of removing the USER fleet
line should work fine.pvirani
10/09/2021, 12:27 AM$ netstat
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 fleetdm-webserver-57bc6745-tw4kh:38904 ip-10-80-140-145.us-west-2.compute.internal:mysql ESTABLISHED
tcp 0 0 fleetdm-webserver-57bc6745-tw4kh:39298 ip-10-80-50-194.us-west-2.compute.internal:redis ESTABLISHED
zwass
pvirani
10/09/2021, 12:35 AMzwass
ip-10-80-140-145.us-west-2.compute.internal:mysql
and ip-10-80-50-194.us-west-2.compute.internal:redis
-- those don't seem to match the addresses used in https://osquery.slack.com/archives/C01DXJL16D8/p1633738072026500?thread_ts=1633632344.474100&cid=C01DXJL16D8pvirani
10/09/2021, 6:24 PM2021-10-09T00:12:50.627-07:00 2021-10-09T07:12:50.627260Z 231 [Warning] [MY-010055] [Server] IP address '10.80.94.227' could not be resolved: Name or service not known
but also confused by the fact that netstat shows connection established when fleet serve
is running (we hardcoded all the flags in env for now). If it isn't able to connect successfully how would the connection be in 'established' state 🤔
/ $ netstat
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 fleetdm-webserver-589fc79fdf-5ltl9:38136 ip-10-80-50-194.us-west-2.compute.internal:redis ESTABLISHED
tcp 0 0 fleetdm-webserver-589fc79fdf-5ltl9:37742 ip-10-80-140-145.us-west-2.compute.internal:mysql ESTABLISHED
tcp 0 0 fleetdm-webserver-589fc79fdf-5ltl9:47692 ip-10-80-50-194.us-west-2.compute.internal:redis ESTABLISHED
then fleet serve
runs and runs and runs without any errors or any info about where it's getting stuck and we ctrl+c out of it. and then I run netstat again and look what happens!
/ $ netstat
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 fleetdm-webserver-589fc79fdf-5ltl9:47692 ip-10-80-50-194.us-west-2.compute.internal:redis ESTABLISHED
Active UNIX domain sockets (w/o servers)
Proto RefCnt Flags Type State I-Node Path
the connection with redis remains but the one with mysql goes away!
I don't know if any of this even related or points to anything. Just sharing it here for my own better understanding for most parts. I've just been RTFMing as hard as possible/ $ mysql --host <http://fleet.crqbc9r8uf32.us-west-2.rds.amazonaws.com|fleet.crqbc9r8uf32.us-west-2.rds.amazonaws.com> --port 3306 --database fleetdatabase --password <my_password>
^^ This did not work and gave an error saying ERROR 1049 (42000): Unknown database <my_password>
$ mysql --host <http://fleet.crqbc9r8uf32.us-west-2.rds.amazonaws.com|fleet.crqbc9r8uf32.us-west-2.rds.amazonaws.com> --port 3306 --database fleetdatabase --password
Enter password:
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MySQL connection id is 1002
Server version: 8.0.25 Source distribution
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MySQL [fleetdatabase]>
^^^ This workedMartavis Parker
10/12/2021, 3:16 PMfleet serve
?zwass
pvirani
10/13/2021, 5:36 PM% kc logs -n fleetdm-webserver fleetdm-webserver-5576655d6f-9vsd7
level=info ts=2021-10-13T17:33:43.324015438Z component=crons cron=vulnerabilities vulnerabilityscanning="not configured"
ts=2021-10-13T17:33:43.725825861Z transport=http address=0.0.0.0:80 msg=listening
Finally!!! 🎉
but
/ $ curl <http://0.0.0.0:80>
<a href="/setup">Temporary Redirect</a>.
redis-cli
and it showed the same behaviour as the app. No logs and the connection was simply hanging which made me even surer that it has to be redis-related. Today I deployed a new cluster without auth and TLS disabled and it worked 🙂 Now figuring out what's up with that temporary redirectzwass
/setup
to create the initial userpvirani
10/13/2021, 6:11 PMzwass
fleetctl setup
pvirani
10/13/2021, 9:31 PM/ $ fleetctl setup --email <mailto:pvirani@twilio.com|pvirani@twilio.com> --name Prima --password *** --org-name Twilio
error creating Fleet API client handler: address must start with https:// for remote connections
I don't want to set up TLS on fleet server for now. The traffic from the internet will hit the load balancer on https but from load balancer -> fleetdm-webserver I want it to be on http only so I've set up my webserver on http://0.0.0.0:80 ... but I get this error at the time of setup ... what do I do?zwass
fleetctl config set --tls-skip-verify=true --address=<http://localhost:80>
and then try it again.pvirani
10/13/2021, 9:40 PM/ $ curl <http://localhost:80>
<!DOCTYPE html>
<html>
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="robots" content="noindex">
<link rel="stylesheet" type="text/css" href="/assets/bundle-10a9feb55abb44f6de13.css">
<link rel="shortcut icon" href="/assets/favicon.ico">
<title>Fleet for osquery</title>
<script type="text/javascript">
var urlPrefix = "";
</script>
</head>
<body>
<div id="app"></div>
<script async defer src="/assets/bundle-f79228ceeede741c7007.js" onload="this.parentElement.removeChild(this)"></script>
<script>document.addEventListener("touchstart", function() {},false);</script>
</body>
</html>
zwass
pvirani
10/13/2021, 9:53 PMzwass
pvirani
10/13/2021, 9:56 PMzwass
pvirani
10/13/2021, 10:02 PMzwass
pvirani
10/13/2021, 10:08 PMRequest URL: <https://fleet20210924001411691500000003-825359854.us-west-2.elb.amazonaws.com/assets/bundle-10a9feb55abb44f6de13.css>
Request Method: GET
Status Code: 400
Remote Address: 52.13.90.185:443
Referrer Policy: strict-origin-when-cross-origin
Looks like it's got to do with traffic between my loadbalancer and the app
$ nslookup <http://fleet20210924001411691500000003-825359854.us-west-2.elb.amazonaws.com|fleet20210924001411691500000003-825359854.us-west-2.elb.amazonaws.com>
Server: 169.254.20.10
Address: 169.254.20.10:53
Non-authoritative answer:
*** Can't find <http://fleet20210924001411691500000003-825359854.us-west-2.elb.amazonaws.com|fleet20210924001411691500000003-825359854.us-west-2.elb.amazonaws.com>: No answer
Non-authoritative answer:
Name: <http://fleet20210924001411691500000003-825359854.us-west-2.elb.amazonaws.com|fleet20210924001411691500000003-825359854.us-west-2.elb.amazonaws.com>
Address: 52.40.25.210
Name: <http://fleet20210924001411691500000003-825359854.us-west-2.elb.amazonaws.com|fleet20210924001411691500000003-825359854.us-west-2.elb.amazonaws.com>
Address: 54.203.88.105
Name: <http://fleet20210924001411691500000003-825359854.us-west-2.elb.amazonaws.com|fleet20210924001411691500000003-825359854.us-west-2.elb.amazonaws.com>
Address: 52.13.90.185
Name: <http://fleet20210924001411691500000003-825359854.us-west-2.elb.amazonaws.com|fleet20210924001411691500000003-825359854.us-west-2.elb.amazonaws.com>
Address: 44.236.167.10
zwass
pvirani
10/13/2021, 10:10 PMzwass
pvirani
10/13/2021, 10:11 PMzwass
/
results in a 400/
request is definitely making it to the Fleet serverpvirani
10/13/2021, 10:15 PMzwass
pvirani
10/13/2021, 10:27 PMzwass
pvirani
10/14/2021, 12:12 AM/
path to my webserver and for all the paths inside that path it would return a 400. Fixing that fixed everything else
long bumpy road but finally we here 🎊🎉🚀 Thanks for all the help Zach and Martavis! Definitely could not have done this without both of you ... can't thank you enough. Blogpost incoming soon to help the community set up a similar structurezwass
pvirani
10/14/2021, 12:21 AM