feat(engine): improve stability of workers with terminations and disconnects #229

abelanger5 · 2024-03-03T20:27:59Z

Description

Adds handling of worker connection edge cases where our worker heartbeat goroutine could have leaked.
Adds a new feature which reassigns a step run to a different worker if the worker has not heartbeat for 60 seconds, which will almost always mean the worker is dead.
Note that this takes into account retries, because we view this as a failure from an execution perspective.

vercel · 2024-03-03T20:28:04Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
hatchet	⬜️ Ignored (Inspect)	Visit Preview		Mar 4, 2024 3:59am

frontend/app/src/components/ui/json-form.tsx

Co-authored-by: Gabe Ruttner <gabriel.ruttner@gmail.com>

abelanger5 added 2 commits March 2, 2024 20:41

fix: support json array types

718c7ee

feat(engine): worker reassignment

62c0144

grutt approved these changes Mar 3, 2024

View reviewed changes

frontend/app/src/components/ui/json-form.tsx Outdated Show resolved Hide resolved

fix: shutdown gocron scheduler

950cceb

vercel bot deployed to Preview March 3, 2024 22:49 View deployment

fix: non-blocking send

63766d7

vercel bot deployed to Preview March 4, 2024 02:01 View deployment

abelanger5 and others added 2 commits March 3, 2024 22:57

Update frontend/app/src/components/ui/json-form.tsx

9888b31

Co-authored-by: Gabe Ruttner <gabriel.ruttner@gmail.com>

merge

820dbb7

vercel bot deployed to Preview March 4, 2024 03:59 View deployment

abelanger5 merged commit 6a6038b into main Mar 4, 2024
17 checks passed

abelanger5 deleted the belanger/worker-stability branch March 4, 2024 03:59