https://github.com/osquery/osquery logo
Join Slack
Powered by
Hi folks, we upgrade to v4.10.0 (from v4.9.1) this...
# fleet
r
Hi folks, we upgrade to v4.10.0 (from v4.9.1) this afternoon, and it seems to have turned off the software inventory. The 
vulnerabilities
key is present in the 
fleet.yml
and it was enabled previously. Has anyone else encountered this? Thanks in advance 🙂
g
I have similar but didn’t bother investigating , so if this is a bug.
s
Hi Ryan and Gavin! If you have a chance would you please check if the 
enable_software_inventory
flag is set to true in your config? You can check this in the network tab of your broswer’s dev console. Please let me know what you find and we can keep troubleshooting from there. If you aren’t familiar with that tool or have any trouble tracking down the config network response, please let me know and I can provide more detailed instructions.
g
It does not look like it.
Copy code
"host_settings": {
    "enable_host_users": true,
    "enable_software_inventory": false
  },
s
Can you apply this spec via fleetctl?
Copy code
---
apiVersion: v1
kind: config
spec:
  host_settings:
    enable_software_inventory: true
fleetctl apply -f <file-path>
g
Yup on it.
Let me just restart fleet is this evaluated dynamically or at runtime ?
s
You’ll need to restart fleet I believe
g
Yaml applied + restarted same reponse.
s
Hmm, the config still shows it set to false?
You should see a change reflected on the software page of the UI after restart. I think your prior inventory should be available right away, but you may need to wait for the inventory to be collected (it runs on an hourly chron by default) before you are able to see the software details.
g
Config response still shows 
false
, host software inventory from prior runs I assume is still present.
s
Thanks! Let us dig into this over here and I’ll follow up in this thread
g
Additional info 
select json_value from app_config_json;
shows as false , setting the Environment flag + Config both seem to not have any impact.
I am not going to yolo edit the DB but slightly tempted.
I have also tried setting the config as false first and removing the env flag.
s
@Gavin Would you please check what version of fleetctl you are running?
g
Copy code
{
  "version": "4.10.0",
  "branch": "HEAD",
  "revision": "67827474c22b61a08f112e682b994aa3e9356133",
  "go_version": "go1.17.7",
  "build_date": "2022-02-14",
  "build_user": "runner"
}
Ahh version of fleetctl
Copy code
fleetctl --version
fleetctl - version 4.10.0
  branch: 	HEAD
  revision: 	67827474c22b61a08f112e682b994aa3e9356133
  build date: 	2022-02-14
  build user: 	runner
  go version: 	go1.17.7
s
Thank you! And you would also please check the output of 
fleetctl get config --include-server-config
g
Copy code
host_expiry_settings:
    host_expiry_enabled: true
    host_expiry_window: 30
  host_settings:
    enable_host_users: true
    enable_software_inventory: false
s
Ok, we’ll keep digging!
Does the network tab show anything from the software endpoint?
g
Update, I now have a working return value.
I don’t know how however.
Only difference Is when I originally applied the fleetctl manifest I was running 3 replica’s and I scaled it down to 1
s
Glad it is working for you now! And thanks for letting us know about the replicas. We’ll keep looking into it to see what might have caused the issue.
@Ryan please let us know how things are going for you after you’ve had a chance to catch up on the thread
t
I use - FLEET_BETA_SOFTWARE_INVENTORY=1 in my docker-compose.yml file; at some point I guess this won't be a BETA variable? :)
s
@Tor Houghton as of version 4.7 you should no longer need to use an environmental variable (beta or otherwise) for software inventory. Please let us know if you run into any issues making the switch over. https://fleetdm.com/docs/using-fleet/vulnerability-processing#configuration
👍 1
t
Many thanks; does this mean that I do not need the FLEET_SOFTWARE_INVENTORY=1 variable but only the FLEET_VULNERABILITIES_DATABASES_PATH var (just making sure, since the docs only mention the latter)?
b
I think software inventory is enabled by default as if 4.7 or higher. If you were running fleet prior to 4.7 I think you'd need to enable explicitly via config or env var as you've shown above. Lastly yes database path is required because by default it's empty string which means it won't be downloaded.
👍 1
r
Hi, just checking this thread.
Output from the 
fleetctl get
command:
Copy code
vulnerabilities:
    cpe_database_url: ""
    current_instance_checks: auto
    cve_feed_prefix_url: ""
    databases_path: /var/lib/fleet
    disable_data_sync: false
    periodicity: 3600000000000
  vulnerability_settings:
    databases_path: ""
Note that the software inventory was already enabled - we enabled it in the first version that contained it, and it’s been enabled ever since, up to v4.9.1, but upgrading from v4.9.1 to v4.10.0 seems to have caused this issue.
I applied this https://osquery.slack.com/archives/C01DXJL16D8/p1645034311367239?thread_ts=1645031148.445939&amp;cid=C01DXJL16D8 and now the UI shows this, so I guess it somehow got disabled as part of the upgrade?
Is it possible to make the 
enable_software_inventory
a proper config flag we can just set in 
/etc/fleet/fleet.yml
though?
it’s a bit tricky to automate otherwise - we use Ansible to template the config file, so having to upload a YAML file and apply it via fleetctl doesn’t really fit the pattern there
g
Once you apply you need to reboot.
s
@Ryan @Gavin, I was incorrect earlier regarding the need to restart after applying the 
enable_software_inventory
spec. You should not need to restart the server after applying that one in order for it to take effect.
g
I left it over night and had no results show until reboot.
t
do you have access to a host to run osqueryd with verbose logs so that we can see if the hosts are getting the queries we expect?
g
I don’t believe that would be of value now, post restart of the TLS server.
t
so all hosts are reporting software as expected?
g
As far as I can tell everything is good now.
t
ok, well, glad to hear! it's sad your upgrade experience was this bumpy
r
Ah, yes I will also restart the service, one sec …
g
It happens one item I was thinking was I enabled software by default with the CLI flag, however I don’t use that flag when running migrations.
r
This is the same as earlier you see.
and it’s been about 3 hours
ok, restarted
Copy code
Feb 17 17:19:19 de-kolide-fleet-01 fleet[23116]: level=info ts=2022-02-17T17:19:19.448414969Z component=crons cron=vulnerabilities databases-path=/var/lib/fleet
Feb 17 17:19:19 de-kolide-fleet-01 fleet[23116]: level=info ts=2022-02-17T17:19:19.448459145Z component=crons cron=vulnerabilities periodicity=1h0m0s
looking good
I’ll see if the software starts to be collected now, and check back later (probably tomorrow morning actually)
g
I got info in < 5 minutes
Depends on your config however.
t
one thing that is important to differentiate here is that the overall view of software doesn't depend on collection only, for verifying whether collection is happening you can spot check some individual host to see if they are reporting it
later on we process that list to count the hosts that have that software, and that's what might vary in periodicity because it's done at the same periodicity as vulnerability check (in fact, it's done right after)
r
Yeah it’s working now actually 🙂
Though I’ve spotted one super weird thing - if I right-click on the menu buttons at the top and choose “open in new tab” when I go to the second tab, it’s logged me out
this then seems to invalidate my session in the previous tab
s
Hmmm, I’m not seeing the “open in new tab” issue in my instance. Which browser version are you using? We’ll take a closer look and see if we can reproduce.
r
Safari browser here 🙂
“Open Link in New Tab” is from the browser’s context menu.
Also weirdly I got an email from Fleet saying SMTP setup was confirmed, but we did that ages ago with the initial setup.
I think my server is haunted 👻 😅
😆 1
s
Thanks! I’m using Chrome so I’ll check that out on Safari
r
Thanks Sarah 🙂
I’m calling it a night now, but I’ll catch up on this thread tomorrow.
👍 1
Just before I go, adding this from the server logs:
Copy code
fleet[23116]: level=info ts=2022-02-17T18:43:55.847040829Z component=http path=/api/v1/fleet/me internal="timestamp: 2022-02-17T18:43:55Z: Session <key redacted> was not found in the datastore" err="Authentication required"
s
All, we’ve opened a tracking ticket in case the enable_software_inventory issue is recurring. Feel free to add any details or reproduction steps to the ticket if you think they could be helpful.
Ryan, we weren’t able to reproduce the issue with the session tabs. Please let us know if you are still seeing it in the morning. Thanks again bringing all this to the slack!
r
Thanks, I added a comment to that ticket with our findings, and that feature request 🙂 I can still reproduce the issue in Safari this morning, even when I disabled my content blockers. I’ll try restarting Safari itself and then the whole Mac, in case $something has happened 😄
No dice 🤔
It’s super weird - it also means deeplinking to things is broken too, for example we have a Grafana dashboard which links back to the specific query pack, and following that link shows the same issue, invalidating your session.
6 Views
Open in Slack
PreviousNext
Powered by