RFC: make `--ecs-server` the default `--server` behaviour #572

mtibben · 2020-04-30T15:08:57Z

#556 introduced the ECS server, and it has many advantages. I think we should just make it the default --server behaviour for the improved UX and major security improvements

The text was updated successfully, but these errors were encountered:

frezbo · 2020-04-30T15:18:57Z

Does that mean the the default --server functionality is going away? We have a lot of people over here using the --server option. The ECS server option won't work with any tools.

mtibben · 2020-04-30T15:32:52Z

Any tool using the SDK should be supported. It would be good to understand what tools won't support it, do you have examples?

I was thinking the --ec2-server flag would stay

frezbo · 2020-04-30T15:43:05Z

@mtibben It seems the functionality is not working as expected:

fedora31 (aws:none)(kc:none)$ aws-vault exec  --ecs-server dev-ro

fedora31 (aws:dev-ro)(kc:dev-ro)$ aws sts get-caller-identity
Enter token for arn:aws:iam::************:mfa/noel.georgi: 615244

Error when retrieving credentials from container-role: Error retrieving metadata: Received error when attempting to retrieve ECS metadata: Read timeout on endpoint URL: "http://127.0.0.1:64434"
fedora31 (aws:hcap-dev-ro)(kc:hcap-dev-ro)$ 615244
bash: 615244: command not found

frezbo · 2020-04-30T19:42:00Z

@mtibben I have seen some issues with my testing of ecs-server which otherwise does not exist when using normal server mode:

If a profile has mfa_serial set, none of the commands work as is my previous error message. If i remove the mfa_serial it seems to be working
Second issue (with mfa_serial not set). Running tools like terraform works, but the aws ruby sdk does not work:

# frozen_string_literal: true

require 'aws-sdk-core'

p Aws::STS::Client.new().get_caller_identity

Traceback (most recent call last):
	20: from test.rb:5:in `<main>'
	19: from /Users/ngeorgi/.rbenv/versions/2.5.7/lib/ruby/gems/2.5.0/gems/aws-sdk-core-3.94.0/lib/aws-sdk-sts/client.rb:1694:in `get_caller_identity'
	18: from /Users/ngeorgi/.rbenv/versions/2.5.7/lib/ruby/gems/2.5.0/gems/aws-sdk-core-3.94.0/lib/seahorse/client/request.rb:70:in `send_request'
	17: from /Users/ngeorgi/.rbenv/versions/2.5.7/lib/ruby/gems/2.5.0/gems/aws-sdk-core-3.94.0/lib/seahorse/client/plugins/response_target.rb:23:in `call'
	16: from /Users/ngeorgi/.rbenv/versions/2.5.7/lib/ruby/gems/2.5.0/gems/aws-sdk-core-3.94.0/lib/aws-sdk-core/plugins/response_paging.rb:10:in `call'
	15: from /Users/ngeorgi/.rbenv/versions/2.5.7/lib/ruby/gems/2.5.0/gems/aws-sdk-core-3.94.0/lib/aws-sdk-core/plugins/param_converter.rb:24:in `call'
	14: from /Users/ngeorgi/.rbenv/versions/2.5.7/lib/ruby/gems/2.5.0/gems/aws-sdk-core-3.94.0/lib/aws-sdk-core/plugins/idempotency_token.rb:17:in `call'
	13: from /Users/ngeorgi/.rbenv/versions/2.5.7/lib/ruby/gems/2.5.0/gems/aws-sdk-core-3.94.0/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:20:in `call'
	12: from /Users/ngeorgi/.rbenv/versions/2.5.7/lib/ruby/gems/2.5.0/gems/aws-sdk-core-3.94.0/lib/seahorse/client/plugins/raise_response_errors.rb:14:in `call'
	11: from /Users/ngeorgi/.rbenv/versions/2.5.7/lib/ruby/gems/2.5.0/gems/aws-sdk-core-3.94.0/lib/aws-sdk-core/plugins/param_validator.rb:24:in `call'
	10: from /Users/ngeorgi/.rbenv/versions/2.5.7/lib/ruby/gems/2.5.0/gems/aws-sdk-core-3.94.0/lib/seahorse/client/plugins/endpoint.rb:45:in `call'
	 9: from /Users/ngeorgi/.rbenv/versions/2.5.7/lib/ruby/gems/2.5.0/gems/aws-sdk-core-3.94.0/lib/aws-sdk-core/plugins/endpoint_discovery.rb:78:in `call'
	 8: from /Users/ngeorgi/.rbenv/versions/2.5.7/lib/ruby/gems/2.5.0/gems/aws-sdk-core-3.94.0/lib/aws-sdk-core/plugins/endpoint_pattern.rb:28:in `call'
	 7: from /Users/ngeorgi/.rbenv/versions/2.5.7/lib/ruby/gems/2.5.0/gems/aws-sdk-core-3.94.0/lib/aws-sdk-core/plugins/user_agent.rb:11:in `call'
	 6: from /Users/ngeorgi/.rbenv/versions/2.5.7/lib/ruby/gems/2.5.0/gems/aws-sdk-core-3.94.0/lib/aws-sdk-core/query/handler.rb:28:in `call'
	 5: from /Users/ngeorgi/.rbenv/versions/2.5.7/lib/ruby/gems/2.5.0/gems/aws-sdk-core-3.94.0/lib/aws-sdk-core/plugins/retry_errors.rb:350:in `call'
	 4: from /Users/ngeorgi/.rbenv/versions/2.5.7/lib/ruby/gems/2.5.0/gems/aws-sdk-core-3.94.0/lib/aws-sdk-core/plugins/helpful_socket_errors.rb:10:in `call'
	 3: from /Users/ngeorgi/.rbenv/versions/2.5.7/lib/ruby/gems/2.5.0/gems/aws-sdk-core-3.94.0/lib/aws-sdk-core/plugins/transfer_encoding.rb:24:in `call'
	 2: from /Users/ngeorgi/.rbenv/versions/2.5.7/lib/ruby/gems/2.5.0/gems/aws-sdk-core-3.94.0/lib/aws-sdk-core/plugins/signature_v4.rb:65:in `call'
	 1: from /Users/ngeorgi/.rbenv/versions/2.5.7/lib/ruby/gems/2.5.0/gems/aws-sdk-core-3.94.0/lib/aws-sdk-core/plugins/signature_v4.rb:123:in `apply_signature'
/Users/ngeorgi/.rbenv/versions/2.5.7/lib/ruby/gems/2.5.0/gems/aws-sdk-core-3.94.0/lib/aws-sdk-core/plugins/signature_v4.rb:72:in `sign_request': unable to sign request without credentials set (Aws::Errors::MissingCredentialsError)

This code would otherwise work with --server mode.

So I think it may break many applications/sdk where it's not set to use ECS credentials by default

jstewmon · 2020-05-01T03:12:17Z

Looks like the following AWS SDKs do not currently support AWS_CONTAINER_CREDENTIALS_FULL_URI:

ruby
php
.net

@frezbo are you sure terraform worked with the --ecs-server option? 🤔
see: hashicorp/aws-sdk-go-base#6

mtibben · 2020-05-01T03:14:53Z

Dang that's a shame. It's been working great for me so far with go

mtibben · 2020-05-01T03:19:50Z

@jstewmon does that mean AWS_CONTAINER_CREDENTIALS_RELATIVE_URI does work for those SDKs?

jstewmon · 2020-05-01T03:23:23Z

In my experience, such discrepancies in the credential providers are unintentional, and AWS accept PRs to bring continuity.

Yes, they all support AWS_CONTAINER_CREDENTIALS_RELATIVE_URI, but it requires the IP 169.254.170.2, and may require listening on port 80 if any of the libraries are not simply using string concatenation of the env var to construct the full address - I suppose if they all use string concatenation, you could just set the relative URI to :<port>

frezbo · 2020-05-01T03:31:53Z

@jstewmon Terraform works fine. Version: v0.12.24. I haven't tested all use cases.

mtibben · 2020-05-01T03:35:00Z

Hmm should we support AWS_CONTAINER_CREDENTIALS_RELATIVE_URI as well? Or should we dive in with PRs to the SDKs to get the much preferable AWS_CONTAINER_CREDENTIALS_FULL_URI support?

frezbo · 2020-05-01T03:42:31Z

@mtibben We should first fix the issue with --ecs-server with mfa_serial set. Seems the prompy is happening after the server is started.

i guess we can add AWS_CONTAINER_CREDENTIALS_RELATIVE_URI for backwards compatibility. Suggestions?

mtibben · 2020-05-01T03:44:22Z

i guess we can add AWS_CONTAINER_CREDENTIALS_RELATIVE_URI for backwards compatibility.

I think the EC2 server covers that use-case pretty well though? I'm more excited to get the root-less server, so maybe we stick with AWS_CONTAINER_CREDENTIALS_FULL_URI and try get it merged in the SDKs

frezbo · 2020-05-01T03:46:11Z

Yeh, I can raise a PR with the ruby sdk

jstewmon · 2020-05-01T03:51:23Z

@frezbo thanks for confirming terraform is working 🙏

It looks like hashicorp/aws-sdk-go-base#5 made this possible in terraform 🎉

mtibben · 2020-05-01T04:01:14Z

@frezbo for the mfa_serial issue, you'll want to set a gui prompt

e.g. on macOS aws-vault exec --ecs-server --prompt=osascript dev-ro

We need to document this better or display a warning when server is used without a prompt

frezbo · 2020-05-01T06:00:32Z

@mtibben That seeems a usabiliy issue, the -s mode promps for the MFA token properly

frezbo · 2020-05-01T06:05:20Z

(aws:none)(kc:none)(git:RD-2485)$ aws-vault exec --ecs-server --prompt=osascript dev-ro

(aws:dev-ro)(kc:dev-ro)(git:RD-2485)$ aws sts get-caller-identity

Error when retrieving credentials from container-role: Error retrieving metadata: Received error when attempting to retrieve ECS metadata: Read timeout on endpoint URL: "http://127.0.0.1:57018"

So if I take time to type in the MFA it throws this error and the first command fails, the error occurs after the command execution. I'm not sure why I need to set the prompt, why can't it do the sam thing as the -s mode where it asks for MFA on the command prompt itself.

Sorry, just pointing out usability issues.

server mode

(aws:none)(kc:none)(git:RD-2485)$ ave -s dev-ro
Enter token for arn:aws:iam::<redacted>:mfa/noel.georgi: 675037
(aws:dev-ro)(kc:dev-ro)(git:RD-2485)$

it seems the UX for the plain -s mode is clean, it seems there's some issue with the code regarding usability.

mtibben · 2020-05-01T06:08:53Z

the -s mode promps for the MFA token properly

Yeah that's only because SDK timeouts for the ec2 server are very aggressive, so we do the first credential load up-front. ECS is currently doing it on-demand

Remember the creds getting generated are only temporary. The MFA session needs to be re-authed, so needs a non-terminal prompt, as the terminal likely has another process running in the foreground.

Agree this needs better UX, feel free to PR a proposal, the code there should be pretty understandable

philsnow · 2020-05-04T23:22:11Z

@frezbo 's log here #572 (comment) shows another case where the current behavior of aws-vault exec --server (starting a process in the background invisibly and creating a sub-shell) is confusing.

This has come up in the past, see #222 (comment) . Suggesting --prompt=osascript to get the MFA code out of band papers over the UI issue (and there might even be a race if there's some other process on the system that is creating osascript prompts).

I think it makes more sense to remove --server from exec and instead use the top-level command aws-vault server (maybe with --ecs or --ec2 flags for the flavor(s) of metadata service that get started). It would be clearer and less implicit.

(I'm not suggesting removing the ability to do --prompt=osascript, as I don't have a better workaround for needing to periodically enter an MFA code. I would personally prefer the osascript prompt just saying "you need to go back to the console and enter a new MFA" because I know exactly where that terminal is, but that's probably not the tradeoff that a lot of people would prefer.)

mtibben · 2020-05-05T03:04:42Z

So we're on the same page, currently aws-vault exec --server will start

A metadata server (in-process), with access to credentials. Prompts also come from this process
A proxy server running on a privileged, well-known port (an external process is started via aws-vault server). This process needs root and is purely a proxy, it has no direct access to credentials.

When you aws-vault exec -s NEWCMD, the metadata server will shutdown as soon as the child process NEWCMD exits. The proxy process is not managed and sticks around after NEWCMD and the credential aws-vault process exits, which is not ideal, but harmless because it's just a proxy with no access to credentials

Creating a stand-alone credential "server" would be expected to be non-interactive, so not sure we could use it for interactive prompts. Plus it would be a second stand-alone process to manage, but this time with access to credentials which would be a security concern

stale · 2021-02-14T19:55:08Z

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

mtibben added this to the v6 milestone Apr 30, 2020

mtibben added the server label Jun 5, 2020

mtibben removed this from the v6 milestone Aug 17, 2020

stale bot added the stale label Feb 14, 2021

stale bot closed this as completed Mar 19, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

RFC: make `--ecs-server` the default `--server` behaviour #572

RFC: make `--ecs-server` the default `--server` behaviour #572

mtibben commented Apr 30, 2020 •

edited

Loading

frezbo commented Apr 30, 2020

mtibben commented Apr 30, 2020

frezbo commented Apr 30, 2020

frezbo commented Apr 30, 2020

jstewmon commented May 1, 2020

mtibben commented May 1, 2020

mtibben commented May 1, 2020

jstewmon commented May 1, 2020

frezbo commented May 1, 2020 •

edited

Loading

mtibben commented May 1, 2020

frezbo commented May 1, 2020

mtibben commented May 1, 2020

frezbo commented May 1, 2020

jstewmon commented May 1, 2020

mtibben commented May 1, 2020

frezbo commented May 1, 2020

frezbo commented May 1, 2020 •

edited

Loading

mtibben commented May 1, 2020 •

edited

Loading

philsnow commented May 4, 2020

mtibben commented May 5, 2020 •

edited

Loading

stale bot commented Feb 14, 2021

RFC: make --ecs-server the default --server behaviour #572

RFC: make --ecs-server the default --server behaviour #572

Comments

mtibben commented Apr 30, 2020 • edited Loading

frezbo commented Apr 30, 2020

mtibben commented Apr 30, 2020

frezbo commented Apr 30, 2020

frezbo commented Apr 30, 2020

jstewmon commented May 1, 2020

mtibben commented May 1, 2020

mtibben commented May 1, 2020

jstewmon commented May 1, 2020

frezbo commented May 1, 2020 • edited Loading

mtibben commented May 1, 2020

frezbo commented May 1, 2020

mtibben commented May 1, 2020

frezbo commented May 1, 2020

jstewmon commented May 1, 2020

mtibben commented May 1, 2020

frezbo commented May 1, 2020

frezbo commented May 1, 2020 • edited Loading

mtibben commented May 1, 2020 • edited Loading

philsnow commented May 4, 2020

mtibben commented May 5, 2020 • edited Loading

stale bot commented Feb 14, 2021

RFC: make `--ecs-server` the default `--server` behaviour #572

RFC: make `--ecs-server` the default `--server` behaviour #572

mtibben commented Apr 30, 2020 •

edited

Loading

frezbo commented May 1, 2020 •

edited

Loading

frezbo commented May 1, 2020 •

edited

Loading

mtibben commented May 1, 2020 •

edited

Loading

mtibben commented May 5, 2020 •

edited

Loading