Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Graceful deployments #78

Open
zcesur opened this issue Sep 3, 2024 · 7 comments
Open

Graceful deployments #78

zcesur opened this issue Sep 3, 2024 · 7 comments
Labels
💎 Bounty

Comments

@zcesur
Copy link
Member

zcesur commented Sep 3, 2024

Let's set up a Fly.io deployment pipeline such that machines running with an old version are shut down gracefully. This is important to ensure livestreams are not interrupted during deployments

This tweet along with this complementary repo explain how we can implement this

The only missing part is to destroy old machines after all running livestreams are terminated, which we can implement inside the end_of_stream callback using Membrane.Pipeline.list_pipelines

If anyone wants to tackle this, please include a short demo of the entire process in the PR, which should include:

  1. Starting a livestream
  2. Triggering a deployment
  3. Confirming that the livestream is still running once the new version is deployed
  4. Stopping the livestream
  5. Confirming that the old machine is destroyed automatically
@zcesur
Copy link
Member Author

zcesur commented Sep 3, 2024

/bounty $50

@algora-pbc Algora PBC
Copy link

algora-pbc bot commented Sep 3, 2024 •
edited
Loading

💎 $50 bounty • Algora

Steps to solve:

  1. Start working: Comment /attempt #78 with your implementation plan
  2. Submit work: Create a pull request including /claim #78 in the PR body to claim the bounty
  3. Receive payment: 100% of the bounty is received 2-5 days post-reward. Make sure you are eligible for payouts

❗ Important guidelines:

  • To claim a bounty, you need to provide a short demo video of your changes in your pull request
  • If anything is unclear, ask for clarification before starting as this will help avoid potential rework
  • For assistance or questions, join our Discord

Thank you for contributing to algora-io/tv!

Add a bounty • Share on socials

Attempt Started (GMT+0) Solution
🔴 @aybanda Sep 5, 2024, 6:04:09 AM WIP
🟢 @DestinedCodes Sep 12, 2024, 4:36:40 PM WIP
🟢 @vishwamartur Nov 17, 2024, 5:19:09 PM #119

@aybanda
Copy link

aybanda commented Sep 5, 2024 •
edited by algora-pbc bot
Loading

@zcesur /attempt #78
Here is my implementation plan
1.Set up environment
2.Implement livestream status check
3.Implement old machine destruction check
4.Create livestream control endpoints
5.Test livestream start
6.Set up Fly.io deployment
7.Implement deployment monitoring
8.Run full demo deployment
9.Error handling

Algora profile Completed bounties Tech Active attempts Options
@aybanda 1 bounty from 1 project
Cancel attempt

@aybanda aybanda mentioned this issue Sep 5, 2024
Closed
4 tasks
@ThePrimeJnr
Copy link

ThePrimeJnr commented Sep 12, 2024 •
edited by algora-pbc bot
Loading

/attempt #78

Options

@raphaelbernhart
Copy link

Just checking in, is this still needed?

@zcesur
Copy link
Member Author

zcesur commented Oct 23, 2024

Yep, still open @raphaelbernhart

@vishwamartur
Copy link

vishwamartur commented Nov 17, 2024 •
edited by algora-pbc bot
Loading

/attempt #78

Options

vishwamartur added a commit to vishwamartur/tv that referenced this issue Nov 17, 2024
@vishwamartur
Implement graceful deployments for Fly.io
934a34c 
Related to algora-io#78

Implement graceful shutdown of old machines during Fly.io deployments to ensure livestreams are not interrupted.

* **Pipeline Changes**:
  - Modify `lib/algora/pipeline.ex` to list pipelines using `Membrane.Pipeline.list_pipelines` and check for running livestreams.
  - Add logic to destroy old machines if no livestreams are running in the `:end_of_stream` callback.

* **Termination Logic**:
  - Update `lib/algora/terminate.ex` to include a function `terminate_interrupted_streams` that lists pipelines and destroys old machines if no livestreams are running.

* **Deployment Controller**:
  - Add `lib/algora_web/controllers/deployment_controller.ex` to handle deployment-related actions such as starting/stopping livestreams, triggering deployments, confirming livestream continuity, and destroying old machines.

* **Router Updates**:
  - Modify `lib/algora_web/router.ex` to add routes for the new deployment-related actions in the `DeploymentController`.

* **Tests**:
  - Add `test/algora_web/controllers/deployment_controller_test.exs` to test the new deployment-related actions in the `DeploymentController`.
@vishwamartur vishwamartur mentioned this issue Nov 17, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
💎 Bounty
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@zcesur @raphaelbernhart @vishwamartur @ThePrimeJnr @aybanda