👋 Question related to ec2, specifically aws_utils and the ec2_instance_metadata table. The

getInstanceIDAndRegion

function (here) is called by this table (here) when it's generated but there are no checks to determine if it's running on an ec2 instance. As a result, it fails and logs a warning related to being unable to find the imds service on non-ec2 systems. The docs for

getInstanceIDAndRegion

mention this: > One has to ensure that the instance is an EC2 one before calling this method, using isEc2Instance. But I can't seem to find a

isEc2Instance

function, and the ec2_instance_metadata table doesn't appear to have any similar checking. My question: Is there, or should there be, logic already in osquery for checking whether it's running on an ec2 instance?

I’m not sure I can go digging, but I’m not sure how one would check if something was an EC2 node. Other than by trying to get metadata

I noticed this issue also. I didn't find a way to make it stop with any osquery flags.

Out of curiosity, what’s the goal here?

I also wonder what would be the case where a machine starts with being on a non EC2 instance, and then it does, and then maybe it also goes back to not being an EC2 instance. Even if you didn't know before hand, you can test it and then I would expect the ability to mark those instances as such (and have a different query pack that doesn't include queries for EC2, if the issues are the retries). The original code, as described in the PR, was sometimes failing in determining automatically if osquery was on EC2 or not, so it was making an incorrect decision, blocking functionality until restarted. I think human knowledge, or custom ways to determine if an instance is or is not on EC2 are needed. It's still possible to have a single query pack and use discovery queries to decide if to run the EC2 query or not too.

Thank you all for your replies/insights. The goal would be to prevent ec2 related queries from running on non-ec2 systems. Or at the very least, prevent checks to determine if the system is ec2 from continuously attempting to connect to the metadata service. The issue, for our use case, are the logs that get generated during failed attempts, and the ongoing connection retries. The discovery query doesn't help much because they're refreshed every

pack_refresh_interval

. This can't be applied on a per-pack basis, which results in further attempts at the metadata service, and more logged warnings/errors. The original check that was removed would cache the result, preventing continued attempts. I agree that this is a difficult issue to solve. I think something that checks the metadata service and caches the result would work well. But I also recognize that it may not always be that simple, and even if it was added back, accounting for everything in the identifying ec2 docs for linux / windows, it may not work for all use cases / deployments. That being said, it's completely reasonable to say that this isn't a problem that OSQuery needs to be solving. Instead, system owners should make that determination through other means, as you mention. This is what we're doing now, we just happen to be exploring solutions because we recently upgraded from 5.4.0, which had the checks, to 5.9.1, and have had system owners begin to ask about the logs.

When you say “logs” what logs are you referring to? Are they from osquery, or some other system?

failed connection attempt logs for imds, one moment, I can link the log entries

they are probably

LOG(WARNING)

or

LOG(ERROR)

; not unreasonable to move those to a verbose log I think to quiet it down a bit

this one - and others related to that code path

Really, I just just want to know if osquery is generating them, or if it’s some other network monitoring system

changing the log level would certainly cover up the issue, but part of it is the continued connection attempts

gotcha

ideally, we have something similar to before that does a check and caches the result so there aren't continued errors in retries

But it sounds like osquery… So if I understand this correctly: 1. the old behavior pinged the ec2 metadata, and if it couldn’t get it, it would cache that stop. 2. The new version has retry logic, but does not cache failure 3. Leading to repeated logging about failures Ya? On the face of it, it seems reasonable to cache a failure here, and assume it’s not going to resolve during osquery’s uptime. Though I wonder if there are contrary situations, like firewalls

yeah, effectively that's correct

@seph the contrary situation and the need for the retry mechanism on IMDS was exactly because sometimes the request fails, even on EC2 instances

I don’t know what correct is, I’m mostly trying to summarize… So on one extreme we don’t want to cache a failure forever, because sometimes we need to retry. And on the other extreme, it might be nice to cache a little, so we don’t spew endless inapplicable failures into the logs

And deciding that the osquery instance is not on EC2 and losing results seemed to be worse than getting error logs of retry attempts on a non-EC2 instance, where one can not do the query.

I have no strong opinions here.

restoring it behind a flag would be great, and maybe updating it to follow all the checks in the aws docs

github issue + PR sounds great!

It’s not wholly clear to me what the right end state is. I imagine another flag, and maybe tweaking some log levels. But I think the best way to get consensus is to iterate in a PR.

sounds good, we'll start there then. thank you all again for your input. It's much appreciated!

(which is basically saying that I think there’s enough consensus to do something. And I don’t think you strictly need an issue. PRs serve just as well)

Going back to this, I want to avoid querying the

ec2_instance_metadata

table on non-EC2 nodes. This query works, but am I missing a more obvious way to do something like this?

WITH checks AS (
    SELECT 1 ec2 FROM platform_info WHERE vendor = 'Amazon EC2' or (vendor = 'Xen' and version like '%.amazon')
)
SELECT  osquery_info.version,
        osquery_info.uuid,
        osquery_info.build_distro,
        CASE when (
        SELECT ec2 FROM checks
        ) THEN (SELECT instance_id FROM ec2_instance_metadata) ELSE NULL END instance_id,
        CASE when (
        SELECT ec2 FROM checks
        ) THEN (SELECT local_hostname FROM ec2_instance_metadata) ELSE NULL END local_hostname,
        CASE when (
        SELECT ec2 FROM checks
        ) THEN (SELECT local_ipv4 FROM ec2_instance_metadata) ELSE NULL END local_ipv4,
        CASE when (
        SELECT ec2 FROM checks
        ) THEN (SELECT account_id FROM ec2_instance_metadata) ELSE NULL END account_id,
        CASE when (
        SELECT ec2 FROM checks
        ) THEN (SELECT instance_type FROM ec2_instance_metadata) ELSE NULL END instance_type,
        CASE when (
        SELECT ec2 FROM checks
        ) THEN (SELECT architecture FROM ec2_instance_metadata) ELSE NULL END architecture,
        CASE when (
        SELECT ec2 FROM checks
        ) THEN (SELECT region FROM ec2_instance_metadata) ELSE NULL END region,
        CASE when (
        SELECT ec2 FROM checks
        ) THEN (SELECT mac FROM ec2_instance_metadata) ELSE NULL END mac,
        CASE when (
        SELECT ec2 FROM checks
        ) THEN (SELECT ami_id FROM ec2_instance_metadata) ELSE NULL END ami_id
FROM osquery_info;

This query runs very fast, e.g.

Run Time: real 0.003 user 0.000878 sys 0.002050

compared to the naive query that doesn't check if it is running on an EC2 node, which takes much longer, e.g.

Run Time: real 273.072 user 0.024563 sys 0.041598