This repo provides an implemented concept of task scheduling using azure service bus delayed messages. this project includes also a client simulator that initiated new Job Definitions Upserts that trigger the flows.
using Publish Subscribe, the console application when starting, registers subscribers (handlers) to service bus topic subscriptions.
Each handler, processes the message and publishes to next topic if needed.
To initiate the process, the Tester Client simulates JobDefinition Upserts.
see TopicsFlow.vsdx visio file for logical flow
- enables scheduling by time interval moreinfo
- enables scheduling by cron expressions moreinfo
- immediate retries on errors
- exponential backoff retries by config
- dead letter queueing for zero data loss + transparency
- enables azure service bus partitioning for high scale by config
- enables ad-hoc job runs for manual/recovery scenarios moreinfo
- flexabilty to run as single process, or process that regsieters one or more handlers
- simuate errors (for the POC)
- local database for JobDefinition (for the POC)
- local database for JobOutputs (for the POC)
- dotnet 5 sdk
- optional: azure account
- optional: azure service bus namespace
- optional: docker on windows
git clone https://github.com/AmirSasson/Servicebus-JobScheduler.git
cd Servicebus-JobScheduler
code .
- You can choose 2 modes for the service bus:
-
Local - used for development, when you less care about persistency and stabilty. see run option
--local
. -
Cloud - production use, using azure ServiceBus, to use this option add
appsettings.overrides.json
undersrc/Servicebus.JobScheduler.ExampleApp
next to existingappsettings.json
with content:{ "ServiceBus": { "ConnectionString": "<<YOUR AZURE SERVICE BUS CONNECTION STRING>>" } }
-
Where to find the ConnectionString in the portal
- from within the root folder
dotnet build
cd ./src/Servicebus.JobScheduler.ExampleApp
- from within the ExampleApp folder
dotnet run
- first run, by default, will configure Azure ServiceBus Entities for you. - (Non-Cloud) - you can also run a local In Memory version of ServiceBus
dotnet run -l true
- you can also use
--modes <<mode[]>>
argument to run a specific/multiple modes of process i.e.
dotnet run --run-setup true --all-modes false --tester-simulator false
or
dotnet run --modes JobDefinitionChange_ImmediateScheduleRule --tester-simulator false
or
dotnet run --modes JobWindowValid_ScheduleNextRun JobWindowValid_RuleTimeWindowExecution --tester-simulator false
seedotnet run -- --help
to view more running modes
- from within the root folder
dotnet test
(tests tbd)
currently this app is running as a linux container on AKS.
jobs can be scheduled in 3 different methods:
- Time Interval sliding window - occurs every X seconds from the actual schduling time.
in the below example, lets assume it was called on time10:30:10
so the first job window will start immediately(10:30:10)
and its time range would be10:27:10 <-> 10:30:10
(range of 180 seconds)
second job would start on10:32:10
and its time range would be10:29:10 <-> 10:32:10
(after 120 from previouse window)
new JobDefinition
{
....
Schedule = new JobSchedule { PeriodicJob = true, RunIntervalSeconds = 120 },
....
}
- Cron Job scheduling .in the below example, lets assume it was called on time
10:30:10
, the cron represents At every 2nd minute so the first job window will start immediately(10:30:10)
and its time range would be10:28:10 <-> 10:30:00
(up to range of 120 seconds)
second job would start on10:32:00
and its time range would be10:30:00 <-> 10:32:00
(exatly 120 seconds and At every 2nd minute)
new JobDefinition
{
....
Schedule = new JobSchedule { PeriodicJob = true, CronSchedulingExpression = "*/2 * * * *" },
....
}
OR you can schedule a cron job that will look on a specific lookback time window, in this example it would run every 2 minutes, and look back on a 1 hour (3600sec) time range, (overlapped time ranges)
new JobDefinition
{
....
Schedule = new JobSchedule { PeriodicJob = true, CronSchedulingExpression = "*/2 * * * *" ,RunIntervalSeconds = 3600},
....
}
- adhoc runs of a predfeined time window
in this example lets assume you would like to re-run jobs from12/28/2020 1:00:00 PM
till12/28/2020 4:00:00 PM
, we cancel the window validation phase in this case.
new JobDefinition
{
....
Schedule = new JobSchedule { PeriodicJob = true, ScheduleEndTime = DateTime.Parse("12/28/2020 4:00:00 PM"), ForceSuppressWindowValidation = true, RunIntervalSeconds = 120 },
LastRunWindowUpperBound = DateTime.Parse("12/28/2020 1:00:00 PM")
....
from root folder run:
create an azure container registry, with admin credentials you can provide to the login command
docker login <<registryname>>.azurecr.io
docker build -t <<registryname>>.azurecr.io/<<my-repository-name>>:latest .
docker push --all-tags <<registryname>>.azurecr.io/<<my-repository-name>>
to run the container locally:
docker run <<registryname>>.azurecr.io/<<my-repository-name>>:latest
- currently as docker container on AKS (linux), aks deployment yml included.
- make sure docker deamon is runnning linux containers
- restart docker
- Azure/azure-service-bus#182