How to manage auto scaling using an ECS work pool?

[rmnvncnt]

I'm currently running flows on ECS by leveraging an EC2 capacity provider. The initial intent was to save costs by using EC2 Spot instances rather than Fargate for infrequent and demanding workloads (think 16 vCPU+). In this particular use case, I can't use Fargate Spot because my architecture is not available for this launch type. My idea was to attach my EC2 capacity provider to an auto scaling group that could scale in and out from zero so that no instances are running when no work has to be done.

However, my understanding of ECS is that tasks cannot be submitted to an empty capacity provider : you need to have at least one instance running otherwise the request is not picked up. Of course this would mean having at least one (large) instance running at all times, which is really not practical.

I was wondering if there is a Prefect way of going around this issue. I was thinking of two solutions, but none seem flawless :

• Have a light LocalProcess flow that pokes my auto scaling group prior to task submission to see if any instance is running, then spins one if needed, wait, then runs the ECS flow. It might work, but feels hacky and needs for some code to be stored server-side.
• Ditch the auto scaling group and use AWS Batch instead. This might be more appropriate, but its more of a launch-and-forget solution where I would lose the task level logs.
• Refactor my demanding flows to fit on Dask and let Dask manage the scale-in. This requires a bit of work.

Maybe I'm missing something, but if anyone has been there, any input is appreciated!

[kevingrismore]

I'm not sure there's a simple answer in ECS, but my experience with it is limited. Stepping back and looking at the nature of what you're trying to accomplish however, an effective alternative might be EKS with:
Node group 1:

• a single small node to run core services and a Prefect Kubernetes worker

Node group 2:

• an autoscaling node group with a minimum node count of 0 that uses EC2 spot instances for running your actual jobs on

There's the persistent cost of k8s with the single node, but this gets past the empty capacity provider problem and feels in the spirit of the cost optimization you're looking for.

[rmnvncnt]

I'm not very familiar with Kubernetes, but I will look into it! Thanks @kevingrismore