Aws Client slow in initialization (authentication with metadata service)

[emdotem]

Describe the bug

Hello,

We are running a very annoying bug that is making the whole application slow. Basically the first time we interact with AWS sdk, whatever is the command, it takes around 1 second, the subsequent commands are 100x faster (2 order of magnitude).

My best guess is that it takes 1 second to contact the metadata service for the authentication.

We run on EC2, using Docker and we use Apache as webserver. Every new request performs a new call to the metadata service since the authentication is not shared across different processes.

Finally, I tag an old issue that it doesn't seem to be fixed:

• Average response time exceeds 1 second #1908
• SQS first send message command 20x/30x slower than following ones #2427 (this is mine, for SQS, eventaully we migrated to Redis because was unusable)

Expected Behavior

Authentication should be fast, in the order of milliseconds, not seconds.

Current Behavior

Authentication using metadata service (my guess) takes 1 second. Every request that need to contact AWS take at least 1 second.

Reproduction Steps

Initialize the AWS client without specifying access secret and key, call the same command one after the other:

$t0 = time();
$client->getObjectUrl("bucket", "file");
$t1 = time();
$client->getObjectUrl("bucket", "file");
$t2 = time();

You'l notice that t1-t0 is greater than 1s while t2-t1 is in the order of milliseconds

Possible Solution

No response

Additional Information/Context

No response

SDK version used

3.179.2

Environment details (Version of PHP (php -v)? OS name and version, etc.)

PHP 7.4

[yenfryherrerafeliz]

Hi @emdotem, thanks for opening this issue. We will investigate the reported behavior and I will get back to you as soon as possible.

Thanks!

[emdotem]

Thank you for replying this fast. If you want me to provide more info, I can help you, even with a live call together.

[yenfryherrerafeliz]

Hi @emdotem, sorry for the delayed response. After looking deep into this I found that after implementing the solution from this comment the latency is gone, which is basically increasing the hop limit for my instance from 1 to >1.

Please see below the command to do this from the aws cli:

• Increase the hop limit

aws ec2 modify-instance-metadata-options
        --instance-id YOUR-INSTANCE-ID \
    --http-put-response-hop-limit 3 

So, could you please try this and let me know if work for you?

Another alternatives would be to use another credential provider like shared credentials file. You can find more information here.

Thanks!

[emdotem]

Another alternatives would be to use another credential provider like shared credentials file. You can find more information here

This is not really a good approach, AWS suggests to not generate key and rather use the metadata server to retrieve the credential.

Can you explain what's the hop limit? What are the consequences on doing so?

[emdotem]

It solves the issue, but I'd like to know more about this limit if possible.

Before:

createPresignedRequest n°1 time: 1.0097961425781
createPresignedRequest n°2 time: 0.00055503845214844

After:

createPresignedRequest n°1 time: 0.027683019638062
createPresignedRequest n°2 time: 0.00051784515380859

[yenfryherrerafeliz]

@emdotem You can find further information about this here, most likely search for the title "Protecting against open layer 3 firewalls and NATs".

Last, there is a final layer of defense in IMDSv2 that is designed to protect EC2 instances that have been misconfigured as open routers, layer 3 firewalls, VPNs, tunnels, or NAT devices. With IMDSv2, the PUT response containing the secret token will, by default, not be able to travel outside the instance. This is accomplished by having the default Time To Live (TTL) on the low-level IP packets containing the secret token set to “1,” much lower than a typical value, such as “64.” Hardware and software that handle packets, including EC2 instances, subtract 1 from each packet’s TTL field whenever they pass it on. If the TTL gets to 0, the packet is discarded, and an error message is sent back to the sender. A packet with a TTL of “64” can therefore make sixty-four “hops” in a network before giving up, while a packet with a TTL of “1” can exist in just one. This feature allows legitimate traffic to get to an intended destination, but is designed to stop packets from endlessly running around in circles if there’s a loop in a network.

With IMDSv2, setting the TTL value to “1” means that requests from the EC2 instance itself will work because they’re returned to the caller (on the instance) before the subtraction occurs. But if the EC2 instance has been misconfigured as an open router, layer 3 firewall, VPN, tunnel, or NAT device, the response containing the token will have its TTL reduced to zero before leaving the instance, and the packet containing the response will be discarded on its way out of the instance, preventing transport to the attacker. The information simply won’t make it further than the EC2 instance itself, which means that an attacker won’t get the response back with the token, and with it the ability to access instance metadata, even if they’ve been successful at getting past all other defenses.

Thanks!

[yenfryherrerafeliz]

@emdotem I will close this issue now, but please if you have any questions just let me know.

Thanks!

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.