I would love to second @sanjaykcse’s question - I believe the issue is around getting the feature sponsored, is that correct @zwass ?

Yes. Essentially, I work on Fleet when I can. This usually means larger features are only addressed when there is a corporate sponsor.

Have you ball-parked what the cost of the carving feature would be?

I think it would be in the 20-30k range but depends on exactly how it is scoped out.

thanks - that helps!

If this happens, I would greatly appreciate adding an argument to the Fleet binary config so we can ensure this is disabled if we don’t want it. (edit: typos)

Note that file carving is disabled by default by the osqueryd agent itself

Thanks, I had forgotten about that.

We try to be careful to keep any agent features that may have significant effect on privacy, security, or performance behind flags.

It’s a bit fuzzy though, fleet may be able to control whether it’s enabled on the client. Etc.

@zwass what would be performance penalty of adding "file carving" feature. Suppose in certain config setup ,kolide fleet scales for 300K hosts. Would adding this feature ( file carving) have adverse impact on scaling/performance ?

@sanjaykcse It would depend on how many files you are carving. It would certainly add a fair amount of network and CPU usage if, for example, you were sending every file saved in each user’s downloads folder for 300k endpoints

Conversely, not using it, but having the code path, shouldn’t have any performance penalty.

file carving is required only for forensics . Intent is not to pull files at regular interval form all the nodes ,but as and when some attack signature is observed, few set of files need to pull in for dipper analysis .

In that case, the number of endpoints wouldn’t really matter. The number of files pulled would be what causes the resource usage to go up.