Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement graceful deployments for Fly.io #119

Closed

Conversation

vishwamartur
Copy link

@vishwamartur vishwamartur commented Nov 17, 2024

Related to #78

Implement graceful shutdown of old machines during Fly.io deployments to ensure livestreams are not interrupted.

  • Pipeline Changes:

    • Modify lib/algora/pipeline.ex to list pipelines using Membrane.Pipeline.list_pipelines and check for running livestreams.
    • Add logic to destroy old machines if no livestreams are running in the :end_of_stream callback.
  • Termination Logic:

    • Update lib/algora/terminate.ex to include a function terminate_interrupted_streams that lists pipelines and destroys old machines if no livestreams are running.
  • Deployment Controller:

    • Add lib/algora_web/controllers/deployment_controller.ex to handle deployment-related actions such as starting/stopping livestreams, triggering deployments, confirming livestream continuity, and destroying old machines.
  • Router Updates:

    • Modify lib/algora_web/router.ex to add routes for the new deployment-related actions in the DeploymentController.
  • Tests:

    • Add test/algora_web/controllers/deployment_controller_test.exs to test the new deployment-related actions in the DeploymentController.

    /claim Graceful deployments #78

Related to algora-io#78

Implement graceful shutdown of old machines during Fly.io deployments to ensure livestreams are not interrupted.

* **Pipeline Changes**:
  - Modify `lib/algora/pipeline.ex` to list pipelines using `Membrane.Pipeline.list_pipelines` and check for running livestreams.
  - Add logic to destroy old machines if no livestreams are running in the `:end_of_stream` callback.

* **Termination Logic**:
  - Update `lib/algora/terminate.ex` to include a function `terminate_interrupted_streams` that lists pipelines and destroys old machines if no livestreams are running.

* **Deployment Controller**:
  - Add `lib/algora_web/controllers/deployment_controller.ex` to handle deployment-related actions such as starting/stopping livestreams, triggering deployments, confirming livestream continuity, and destroying old machines.

* **Router Updates**:
  - Modify `lib/algora_web/router.ex` to add routes for the new deployment-related actions in the `DeploymentController`.

* **Tests**:
  - Add `test/algora_web/controllers/deployment_controller_test.exs` to test the new deployment-related actions in the `DeploymentController`.
@CLAassistant
Copy link

CLAassistant commented Nov 17, 2024

CLA assistant check
All committers have signed the CLA.

@algora-pbc algora-pbc bot mentioned this pull request Nov 17, 2024
Copy link

algora-pbc bot commented Nov 17, 2024

💵 To receive payouts, sign up on Algora, link your Github account and connect with Stripe.

@lastcanal
Copy link
Collaborator

lastcanal commented Dec 11, 2024

Hi @vishwamartur. I think you might have missed the mark with this one. I see five HTTP routes but only one route is required: /health. You are also missing the AppHealth GenServer which serves as a backend for the /health route, which should be setup to control Fly's proxy.

This deployment script is the crux of the graceful deployment implementation and you are missing the RPC call to AppHealth.down and the associated fly.yaml config changes.

I am going to close this PR because it is far from implementing graceful deployments. Feel free to re-submit when you are closer to having more a complete solution and can test real graceful deployments on fly.

Thanks.

@lastcanal lastcanal closed this Dec 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants