Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

Support capacity provider strategy in task run request #312

Conversation

kaaloo
Copy link

@kaaloo kaaloo commented Sep 13, 2023

This PR allows setting a capacityProviderStrategy in the task_run_request part of an ECS Task template. It works by checking for the capacityProviderStrategy key when preparing the run request and dropping the launchType key since these are mutually exclusive. See:

https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_RunTask.html#ECS-RunTask-request-capacityProviderStrategy

Closes PrefectHQ/prefect#13030

Example

The following Gist illustrates an AWS Push Work Pool template that has been modified to include a new capacityProvider variable, which is then used in the capacityProviderStrategy section of the task_run_template.

Screenshots

Checklist

  • References any related issue by including "Closes #" or "Closes ".
    • If no issue exists and your change is not a small fix, please create an issue first.
  • Includes tests or only affects documentation.
  • Passes pre-commit checks.
    • Run pre-commit install && pre-commit run --all locally for formatting and linting.
  • Includes screenshots of documentation updates.
    • Run mkdocs serve view documentation locally.
  • Summarizes PR's changes in CHANGELOG.md

@kaaloo kaaloo requested a review from a team as a code owner September 13, 2023 13:36
@jakekaplan
Copy link
Contributor

Apologies for the delay on this, I have not had time to test it. Will try to do so ASAP

@jggatter
Copy link

Looking forward to this, thank you @kaaloo!

Copy link
Contributor

@jakekaplan jakekaplan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the contribution! Apologies for the delay in the review. Approved barring any necessary changes to make to pass CI.

@kaaloo kaaloo requested a review from jakekaplan September 26, 2023 14:06
@kaaloo
Copy link
Author

kaaloo commented Sep 26, 2023

Hello @jakekaplan , I updated my PR. Could you please approve running the workflows?

@jakekaplan
Copy link
Contributor

Hey it looks like the tests are still failing unfortunately. If you're able to fix that would be great! I can try and take a look myself if I have sometime.

Copy link
Contributor

@EmilRex EmilRex left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kaaloo Thank you again for putting in this PR! Just suggested a couple of changes that should fix the linting issues and a question about the test you've implemented.

prefect_aws/workers/ecs_worker.py Outdated Show resolved Hide resolved
tests/workers/test_ecs_worker.py Show resolved Hide resolved
tests/workers/test_ecs_worker.py Show resolved Hide resolved
Comment on lines +1876 to +1877
task = describe_task(ecs_client, task_arn)
assert task.get("capacityProviderName") == "FOO"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems like the right approach but the tests are failing saying that capacityProviderName is None. Maybe it's not a value that's mocked correctly? The value is supposed to be there according to the spec. Perhaps it's not coming through because it's not one of the accepted values? @kaaloo was the rest passing for you locally?

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd like to chime in here and see that mock ecs cluster is defined here without capacity provider created. That might be the issue that it's None here in this test case.

kaaloo and others added 3 commits October 9, 2023 09:16
Formatting fix

Co-authored-by: Emil Christensen <EmilRex@users.noreply.github.com>
Formatting fix

Co-authored-by: Emil Christensen <EmilRex@users.noreply.github.com>
Formatting fix

Co-authored-by: Emil Christensen <EmilRex@users.noreply.github.com>
@desertaxle
Copy link
Member

Thank you for your willingness to contribute to prefect-aws! The code for this library has been moved to https://github.com/PrefectHQ/prefect/tree/main/src/integrations/prefect-aws. Please reopen this PR there if this functionality is still needed!

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Setting capacityProviderStrategy not working in Push Work Pool
6 participants