https://github.com/osquery/osquery logo
Powered by
#fleet
Title
# fleet
r

Ryan

02/16/2022, 5:05 PM
Hi folks, we upgrade to v4.10.0 (from v4.9.1) this afternoon, and it seems to have turned off the software inventory. The 
vulnerabilities
key is present in the 
fleet.yml
and it was enabled previously. Has anyone else encountered this? Thanks in advance 🙂
g

Gavin

02/16/2022, 5:47 PM
I have similar but didn’t bother investigating , so if this is a bug.
s

Sarah Gillespie

02/16/2022, 5:53 PM
Hi Ryan and Gavin! If you have a chance would you please check if the 
enable_software_inventory
flag is set to true in your config? You can check this in the network tab of your broswer’s dev console. Please let me know what you find and we can keep troubleshooting from there. If you aren’t familiar with that tool or have any trouble tracking down the config network response, please let me know and I can provide more detailed instructions.
g

Gavin

02/16/2022, 5:57 PM
It does not look like it.
Copy code
"host_settings": {
    "enable_host_users": true,
    "enable_software_inventory": false
  },
s

Sarah Gillespie

02/16/2022, 5:58 PM
Can you apply this spec via fleetctl?
Copy code
---
apiVersion: v1
kind: config
spec:
  host_settings:
    enable_software_inventory: true
fleetctl apply -f <file-path>
g

Gavin

02/16/2022, 5:59 PM
Yup on it.
Let me just restart fleet is this evaluated dynamically or at runtime ?
s

Sarah Gillespie

02/16/2022, 6:05 PM
You’ll need to restart fleet I believe
g

Gavin

02/16/2022, 6:09 PM
Yaml applied + restarted same reponse.
s

Sarah Gillespie

02/16/2022, 6:09 PM
Hmm, the config still shows it set to false?
You should see a change reflected on the software page of the UI after restart. I think your prior inventory should be available right away, but you may need to wait for the inventory to be collected (it runs on an hourly chron by default) before you are able to see the software details.
g

Gavin

02/16/2022, 6:10 PM
Config response still shows 
false
, host software inventory from prior runs I assume is still present.
s

Sarah Gillespie

02/16/2022, 6:26 PM
Thanks! Let us dig into this over here and I’ll follow up in this thread
g

Gavin

02/16/2022, 6:29 PM
Additional info 
select json_value from app_config_json;
shows as false , setting the Environment flag + Config both seem to not have any impact.
I am not going to yolo edit the DB but slightly tempted.
I have also tried setting the config as false first and removing the env flag.
s

Sarah Gillespie

02/16/2022, 6:53 PM
@Gavin Would you please check what version of fleetctl you are running?
g

Gavin

02/16/2022, 6:54 PM
Copy code
{
  "version": "4.10.0",
  "branch": "HEAD",
  "revision": "67827474c22b61a08f112e682b994aa3e9356133",
  "go_version": "go1.17.7",
  "build_date": "2022-02-14",
  "build_user": "runner"
}
Ahh version of fleetctl
Copy code
fleetctl --version
fleetctl - version 4.10.0
  branch: 	HEAD
  revision: 	67827474c22b61a08f112e682b994aa3e9356133
  build date: 	2022-02-14
  build user: 	runner
  go version: 	go1.17.7
s

Sarah Gillespie

02/16/2022, 6:58 PM
Thank you! And you would also please check the output of 
fleetctl get config --include-server-config
g

Gavin

02/16/2022, 6:59 PM
Copy code
host_expiry_settings:
    host_expiry_enabled: true
    host_expiry_window: 30
  host_settings:
    enable_host_users: true
    enable_software_inventory: false
s

Sarah Gillespie

02/16/2022, 6:59 PM
Ok, we’ll keep digging!
Does the network tab show anything from the software endpoint?
g

Gavin

02/16/2022, 7:13 PM
Update, I now have a working return value.
I don’t know how however.
Only difference Is when I originally applied the fleetctl manifest I was running 3 replica’s and I scaled it down to 1
s

Sarah Gillespie

02/16/2022, 7:16 PM
Glad it is working for you now! And thanks for letting us know about the replicas. We’ll keep looking into it to see what might have caused the issue.
@Ryan please let us know how things are going for you after you’ve had a chance to catch up on the thread
t

Tor Houghton

02/16/2022, 7:27 PM
I use - FLEET_BETA_SOFTWARE_INVENTORY=1 in my docker-compose.yml file; at some point I guess this won't be a BETA variable? :)
s

Sarah Gillespie

02/16/2022, 7:36 PM
@Tor Houghton as of version 4.7 you should no longer need to use an environmental variable (beta or otherwise) for software inventory. Please let us know if you run into any issues making the switch over. https://fleetdm.com/docs/using-fleet/vulnerability-processing#configuration
👍 1
t

Tor Houghton

02/16/2022, 7:45 PM
Many thanks; does this mean that I do not need the FLEET_SOFTWARE_INVENTORY=1 variable but only the FLEET_VULNERABILITIES_DATABASES_PATH var (just making sure, since the docs only mention the latter)?
b

Benjamin Edwards

02/16/2022, 9:43 PM
I think software inventory is enabled by default as if 4.7 or higher. If you were running fleet prior to 4.7 I think you'd need to enable explicitly via config or env var as you've shown above. Lastly yes database path is required because by default it's empty string which means it won't be downloaded.
👍 1
r

Ryan

02/17/2022, 2:25 PM
Hi, just checking this thread.
Output from the 
fleetctl get
command:
Copy code
vulnerabilities:
    cpe_database_url: ""
    current_instance_checks: auto
    cve_feed_prefix_url: ""
    databases_path: /var/lib/fleet
    disable_data_sync: false
    periodicity: 3600000000000
  vulnerability_settings:
    databases_path: ""
Note that the software inventory was already enabled - we enabled it in the first version that contained it, and it’s been enabled ever since, up to v4.9.1, but upgrading from v4.9.1 to v4.10.0 seems to have caused this issue.
I applied this https://osquery.slack.com/archives/C01DXJL16D8/p1645034311367239?thread_ts=1645031148.445939&amp;cid=C01DXJL16D8 and now the UI shows this, so I guess it somehow got disabled as part of the upgrade?
Is it possible to make the 
enable_software_inventory
a proper config flag we can just set in 
/etc/fleet/fleet.yml
though?
it’s a bit tricky to automate otherwise - we use Ansible to template the config file, so having to upload a YAML file and apply it via fleetctl doesn’t really fit the pattern there
g

Gavin

02/17/2022, 2:44 PM
Once you apply you need to reboot.
s

Sarah Gillespie

02/17/2022, 5:07 PM
@Ryan @Gavin, I was incorrect earlier regarding the need to restart after applying the 
enable_software_inventory
spec. You should not need to restart the server after applying that one in order for it to take effect.
g

Gavin

02/17/2022, 5:09 PM
I left it over night and had no results show until reboot.
t

Tomas Touceda

02/17/2022, 5:12 PM
do you have access to a host to run osqueryd with verbose logs so that we can see if the hosts are getting the queries we expect?
g

Gavin

02/17/2022, 5:12 PM
I don’t believe that would be of value now, post restart of the TLS server.
t

Tomas Touceda

02/17/2022, 5:14 PM
so all hosts are reporting software as expected?
g

Gavin

02/17/2022, 5:15 PM
As far as I can tell everything is good now.
t

Tomas Touceda

02/17/2022, 5:17 PM
ok, well, glad to hear! it's sad your upgrade experience was this bumpy
r

Ryan

02/17/2022, 5:18 PM
Ah, yes I will also restart the service, one sec …
g

Gavin

02/17/2022, 5:18 PM
It happens one item I was thinking was I enabled software by default with the CLI flag, however I don’t use that flag when running migrations.
r

Ryan

02/17/2022, 5:18 PM
This is the same as earlier you see.
and it’s been about 3 hours
ok, restarted
Copy code
Feb 17 17:19:19 de-kolide-fleet-01 fleet[23116]: level=info ts=2022-02-17T17:19:19.448414969Z component=crons cron=vulnerabilities databases-path=/var/lib/fleet
Feb 17 17:19:19 de-kolide-fleet-01 fleet[23116]: level=info ts=2022-02-17T17:19:19.448459145Z component=crons cron=vulnerabilities periodicity=1h0m0s
looking good
I’ll see if the software starts to be collected now, and check back later (probably tomorrow morning actually)
g

Gavin

02/17/2022, 5:21 PM
I got info in < 5 minutes
Depends on your config however.
t

Tomas Touceda

02/17/2022, 5:24 PM
one thing that is important to differentiate here is that the overall view of software doesn't depend on collection only, for verifying whether collection is happening you can spot check some individual host to see if they are reporting it
later on we process that list to count the hosts that have that software, and that's what might vary in periodicity because it's done at the same periodicity as vulnerability check (in fact, it's done right after)
r

Ryan

02/17/2022, 6:22 PM
Yeah it’s working now actually 🙂
Though I’ve spotted one super weird thing - if I right-click on the menu buttons at the top and choose “open in new tab” when I go to the second tab, it’s logged me out
this then seems to invalidate my session in the previous tab
s

Sarah Gillespie

02/17/2022, 6:29 PM
Hmmm, I’m not seeing the “open in new tab” issue in my instance. Which browser version are you using? We’ll take a closer look and see if we can reproduce.
r

Ryan

02/17/2022, 6:32 PM
Safari browser here 🙂
“Open Link in New Tab” is from the browser’s context menu.
Also weirdly I got an email from Fleet saying SMTP setup was confirmed, but we did that ages ago with the initial setup.
I think my server is haunted 👻 😅
😆 1
s

Sarah Gillespie

02/17/2022, 6:34 PM
Thanks! I’m using Chrome so I’ll check that out on Safari
r

Ryan

02/17/2022, 6:34 PM
Thanks Sarah 🙂
I’m calling it a night now, but I’ll catch up on this thread tomorrow.
👍 1
Just before I go, adding this from the server logs:
Copy code
fleet[23116]: level=info ts=2022-02-17T18:43:55.847040829Z component=http path=/api/v1/fleet/me internal="timestamp: 2022-02-17T18:43:55Z: Session <key redacted> was not found in the datastore" err="Authentication required"
s

Sarah Gillespie

02/17/2022, 8:31 PM
All, we’ve opened a tracking ticket in case the enable_software_inventory issue is recurring. Feel free to add any details or reproduction steps to the ticket if you think they could be helpful.
Ryan, we weren’t able to reproduce the issue with the session tabs. Please let us know if you are still seeing it in the morning. Thanks again bringing all this to the slack!
r

Ryan

02/18/2022, 10:28 AM
Thanks, I added a comment to that ticket with our findings, and that feature request 🙂 I can still reproduce the issue in Safari this morning, even when I disabled my content blockers. I’ll try restarting Safari itself and then the whole Mac, in case $something has happened 😄
No dice 🤔
It’s super weird - it also means deeplinking to things is broken too, for example we have a Grafana dashboard which links back to the specific query pack, and following that link shows the same issue, invalidating your session.
4 Views
Powered by