https://github.com/osquery/osquery logo
#fleet
Title
# fleet
m

Mike S.

04/14/2023, 7:51 PM
Hi Team - have you seen any previous issues where a Windows Fleet Agent is being blocked/flagged by Cloudflare as "Command injection - Common Attack Commands"? This does not appear to be a rule we have written, but an automatic CF detection. Agent package creation was done only with --fleet-url and --enroll-secret. The issue is not being seen on Mac or Linux agents.
z

Zay Hanlon

04/14/2023, 8:27 PM
Hi @Mike S. - Are you deploying fleetd (orbit) or plain osquery?
Is it actually fully blocked, or just flagged and then the deployment completes?
m

Mike S.

04/14/2023, 8:35 PM
Hi Zay - I am generating the package in Fleet (community version) - I believe that's orbit. Apologies, I'm still very much a novice. It's not fully blocked. The deployment completes, and Fleet can sometimes pull the vitals, but no queries can be run. It ends up looking like this:
z

Zay Hanlon

04/14/2023, 8:41 PM
@Kathy Satterlee Can you take a look at this?
k

Kathy Satterlee

04/14/2023, 8:45 PM
While it’s definitely odd that it’s only blocking those Windows devices, this seems like something that would need to be managed on the CloudFlare side. That being said, I’m going to reach out to the team to see if there’s a substantial difference in the way osquery returns data from Windows.
m

Mike S.

04/14/2023, 8:47 PM
Hi @Kathy Satterlee agreed, we are looking at how to properly manage that on the CF side. It was just odd that Windows was the only agent doing this.
I am looking at upgrading to the latest version to see if the issue is still there.
k

Kathy Satterlee

04/14/2023, 8:48 PM
100% agree that it’s odd!
m

Mike S.

04/14/2023, 8:48 PM
I do have debug logs if those would help.
k

Kathy Satterlee

04/14/2023, 8:49 PM
I’m wondering if it’s a Window’s specific detail query that’s triggering CF.
You could try running the detail queries as live queries and see what (if any) of them make it through.
All of the Windows only queries end in
_windows
m

Mike S.

04/14/2023, 9:01 PM
Looks like the agent doesn't respond to the queries.
z

Zay Hanlon

04/14/2023, 9:08 PM
@Mike S. We are having the team look into this, but we may need to get back to you on Monday. I hope this doesn't stall you on any progress with Fleet! We have a reminder to get back to you next week
Please keep us posted in this thread if you discover anything else
m

Mike S.

04/14/2023, 9:10 PM
I will, thanks all! No worries on getting back on Monday.
l

Lucas Rodriguez

04/14/2023, 9:43 PM
Windows Fleet Agent is being blocked/flagged by Cloudflare as "Command injection - Common Attack Commands"?
Hey, quick question: Is a cloudflare component running in the host blocking Orbit? Or is Cloudflare blocking network connections to/from the Windows agent?
m

Mike S.

04/14/2023, 9:44 PM
Hi Lucas, it is blocking network connections to/from the agent.
j

Jason

04/14/2023, 9:53 PM
this is at the WAF, right ?
I've seen this on other WAFs because the agent and fleetd are literally passing SQL back and forth. It's a very common thing for WAFs to block any strings that look like SQL as an injection attack.
m

Mike S.

04/14/2023, 10:15 PM
Yes, this is at the WAF
d

defensivedepth

04/15/2023, 3:53 PM
So the WAF is decrypting the Fleet comms and then re-encrypting and sending it along to the Fleet server? Is there a possibility of allow-listing the osqueryd traffic? There is potentially a bunch of sensitive data that could be queried.
k

Kathy Satterlee

04/17/2023, 2:55 PM
Is there any chance that your non-Windows hosts share something in common that might explain why they’re not getting caught up? For example, location (they’re all connected to an internal network that has different rules), VPN, etc?
m

Mike S.

04/17/2023, 4:06 PM
Hi @defensivedepth - That's something we are looking into. I noticed that there's an osquery user-agent we may be able to allow-list. That could be a quick win to get that traffic through. Hi @Kathy Satterlee - sorry, I'm not sure what you mean. Our non-Windows hosts appear to be working properly at the moment. Sorry if I'm misunderstanding here. 🙂
k

Kathy Satterlee

04/17/2023, 5:40 PM
I was wondering if there might something different between the two groups other than just OS that could explain why some requests are making it though and others won’t.
m

Mike S.

04/17/2023, 5:54 PM
Ok, I was misunderstanding! So the the Mac systems are all laptops, on-site and remote. The Linux system is an EC2 instance. The Windows systems are also laptops that can be on-site/remote - Win10/11. They all use the same security stack, and have the same agents installed.
Correction - They all SHOULD use the same security stack. Let me check that assumption. 🙂
So the Mac systems have the same security stack as the Windows systems.
Does the Mac agent communicate with Fleet in a different way than a Windows agent?
k

Kathy Satterlee

04/18/2023, 8:02 PM
No, they're both using the same API endpoints and the same general request body.
m

Mike S.

04/18/2023, 8:15 PM
So far I'm not seeing any differences on the Mac side. I'll keep looking.
I finally got to check out the Cloudflare alert - I'm trying to see what commands CF actually flagged on. Hopefully more to come on that.
l

Lucas Rodriguez

04/18/2023, 9:15 PM
Interesting, it flagged the
distributed/write
request, which means it flagged some part of the query results as command injection. I wonder what exactly caused the flagging, but I can see "machine learning" there so it's kind of a black box?...
a

Ando

04/26/2023, 1:22 PM
Similar issue for my usecase. I was working on Windows enrollment today (orbit+osquery on Windows, packaged by fleetctl). Mac agents already running with no issues. Going from endpoint -> cloudflare (WAF) -> fleet server. Cloudflare let through windows host enrollment, but queries are getting blocked on path
/api/v1/osquery/distributed/write
UA
osquery/5.8.2
CF WAF rule
Command Injection - Common Attack Commands
. Very odd that from CF WAF logic Mac queries are safe while Windows is not. I'll be doing some researching, and probably also look for a way to specifically exclude the osquery user-agent/path from cloudflare WAF on cloudflare side.
j

Jason

04/26/2023, 1:24 PM
Yeah that is really odd. We are using the FortiWeb Cloud product as a WAF and noticed SQL injection being flagged -- but by all platforms, not just Windows. I had to remove those checks from certain endpoints.
(Web endpoints, not device endpoints)
a

Ando

04/27/2023, 12:33 PM
Plot twist. I was using the "old" CloudFlare WAF rules, they started rolling out the next generation of the rules engine last year. After upgrading to the new WAF model today, and without any additional changes/exceptions, the Windows agent is now able to pass CF WAF and send back telemetry. TL;DR possible "fix" to this: upgrade to new CF WAF rules?
z

Zay Hanlon

04/27/2023, 1:54 PM
Thats really interesting @Ando! Please keep us posted if this resolves the issue.
m

Mike S.

04/27/2023, 3:29 PM
Thanks for posting this @Ando! I will bring this up with our Cloud team and see if this is an option for us. If not, we'll probably just look for a bypass rule for the user-agent or something to that effect.
a

Ando

05/05/2023, 1:22 PM
I'd say updating to the new WAF rule engine has resolved the issue. Been running with this for a week with no problems. Also, cloudflare recently sent an e-mail announcing the old rules engine will stop working on May 2024 (so migration is mandatory within a year).