feat(router): support for isolation modes using limiters #3379

atzoum · 2023-05-23T15:35:33Z

Description

Introducing isolation strategies in router, following the same pattern that we are already using in processor & batchrouter.

Router is one of a kind, since it now uses not 1 but 2 different types of workers:

partition worker: multiple partition workers are being created by the worker pool according to the configured isolation strategy.
(internal) worker: each partition worker has a pool of (internal) workers which ultimately process the jobs and deliver them to their destination. By default, 64 internal workers are spawn for every single partition worker.

Isolation modes

none - a single partition worker picks up and processes jobs for all workspaces & destinations.
workspace - using a separate partition worker per active workspace (default mode in multi-tenant deployments).
destination - using a separate partition worker per active destination (default mode in non-multi-tenant deployments).

Limiters

pickup - for picking up jobs from jobsdb and assigning them to workers (limit: 50 goroutines/destType)
transform - for controlling concurrency while transforming jobs (limit: 100 goroutines/destType)
batch - for controlling concurrency while batching jobs (limit: 100 goroutines/destType)
process - for controlling concurrency while processing jobs within workers (limit: 100 goroutines/destType)

Dynamic limiter priorities

Router uses dynamic priorities for executing tasks inside its limiters. Each partition keeps track of it's throughput and error rate for every limiter-enabled operation, by using partition.Stats. These statistics are used to calculate a performance score between 0-100, which in turn gets mapped to a limiter priority between 1-4. This priority is especially important when there is congestion in the limiter (goroutines requesting more slots than the limiter's total cap). The limiter itself has a mechanism to avoid starvation of lower-priority jobs, by periodically incrementing their priority (by default every 1 sec, see WithLimiterDynamicPeriod).

Future cleanup work

The existing fair pickup algorithm has been retained and will be used only when JobsDB.fairPickup is enabled and isolation mode is set to none. This is to be able to revert in case an unexpected issue happens during the rollout. However, the final plan is to remove MultiTenantJobsDB - unionQuery.go and MultiTenantI - tenantstats.go in the next release, after router isolation has been released, tested and proven with production workloads.

Performance

It comes as no surprise to say that performance is again on par (and even better) with the previous solution up to 200 partitions. After that threshold, throughput starts deteriorating.

The relevant benchmark can be found inside router_isolation_test.go and a visualisation of one execution of this benchmark is available here.

Other notable changes

Reorganised package's code in separate, logically-grouped files, trying to make it easier to read and follow.
Removed global variables and Init function from the package.
AdaptivePayloadLimiter.enabled is now true by default.
Setting a soft memory limit (aka GOMEMLIMIT) to 80% of the container's available memory.
Using bytesize, queue, mem, profiler and sync packages from rudder-go-kit.
Capturing pipeline delay statistics through pipeline_delay_min & pipeline_delay_max.

Notion Ticket

Link

Security

The code changed/added as part of this pull request won't create any security issues with how the software is being used.

codecov · 2023-05-23T16:18:57Z

Codecov Report

Patch coverage: 81.59% and project coverage change: -0.03 ⚠️

Comparison is base (9e32802) 68.47% compared to head (8c0f9d7) 68.44%.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #3379      +/-   ##
==========================================
- Coverage   68.47%   68.44%   -0.03%     
==========================================
  Files         330      327       -3     
  Lines       53042    52801     -241     
==========================================
- Hits        36319    36140     -179     
+ Misses      14358    14312      -46     
+ Partials     2365     2349      -16

Impacted Files	Coverage Δ
app/apphandlers/processorAppHandler.go	`9.20% <0.00%> (-0.04%)`	⬇️
jobsdb/unionQuery.go	`84.94% <ø> (-3.09%)`	⬇️
processor/stash/stash.go	`52.88% <ø> (ø)`
...hema-forwarder/internal/forwarder/baseforwarder.go	`81.57% <ø> (ø)`
...hema-forwarder/internal/forwarder/jobsforwarder.go	`67.32% <ø> (ø)`
services/alert/pagerduty.go	`0.00% <0.00%> (ø)`
services/alert/victorops.go	`0.00% <0.00%> (ø)`
services/alerta/client.go	`82.97% <ø> (ø)`
services/multitenant/tenantstats.go	`92.83% <ø> (ø)`
warehouse/archive/archiver.go	`65.86% <0.00%> (ø)`
... and 38 more

... and 5 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

atzoum · 2023-05-26T08:21:45Z

jobsdb/unionQuery.go

 }

+// TODO: delete this once we remove the old fair pickup algorithm and move MultiTenantLegacy#GetAllJobs inside JobsDB


[note] added some todos for cleaning up in the next release

atzoum · 2023-05-26T08:24:04Z

main.go

+	if memStat, err := mem.Get(); err == nil {
+		memoryLimit := int64(80 * memStat.Total / 100)
+		logger.NewLogger().Infow("Setting memory limit to", "limit", memoryLimit)
+		debug.SetMemoryLimit(memoryLimit)
+	}


[note] setting a soft memory limit (aka GOMEMLIMIT) to 80% of the container's available memory.

atzoum · 2023-05-26T08:28:32Z

utils/workerpool/worker_pool.go

+		go func() {
+			wstart := time.Now()
+			w.Stop()
+			wg.Done()
+			wp.logger.Debugf("worker %s stopped in %s", w.partition, time.Since(wstart))
+		}()


[note] stopping workers in parallel

utils/payload/limiter_setup.go

atzoum · 2023-05-26T08:30:00Z

utils/misc/misc.go

@@ -1289,7 +1268,7 @@ func Unique(stringSlice []string) []string {
 }

 func UseFairPickup() bool {
-	return config.GetBool("JobsDB.fairPickup", false) || config.GetBool("EnableMultitenancy", false)
+	return config.GetBool("JobsDB.fairPickup", false) && config.GetString("Router.isolationMode", "default") == "none"


[note] fair pickup will be used only if it is enabled and isolation mode is none

atzoum · 2023-05-26T08:34:45Z

router/types.go

+	status                 *jobsdb.JobStatusT
+}
+
+type reloadableConfig struct {


[note] since we will be starting new routers after switching from degraded to normal mode, moving all reloadable configuration to a separate struct, so that there is minimal leakage!
And promise to prepare a solution for non-leaking reloadable configuration in rudder-go-kit in the future!

cisse21 · 2023-06-07T06:21:19Z

gateway/integration_test.go

@@ -22,7 +22,7 @@ import (
 	"github.com/ory/dockertest/v3"
 	"github.com/stretchr/testify/require"

-	kitHelper "github.com/rudderlabs/rudder-go-kit/testhelper"
+	kit_helper "github.com/rudderlabs/rudder-go-kit/testhelper"


Aren't we adhering to the outcome of the poll :P

We voted for the same thing but lost :'(

cisse21 · 2023-06-07T07:15:17Z

router/worker.go

@@ -289,7 +249,7 @@ func (w *worker) WorkerProcess() {
 			}

 		case <-timeout:
-			timeout = time.After(jobsBatchTimeout)
+			timeout = time.After(w.rt.reloadableConfig.jobsBatchTimeout)


Why do have this both here and at line 60?

further refactoring is required indeed, but will refrain from doing it now as changes are already enough :)

router/worker.go

router/batchrouter/handle_observability.go

github-actions bot added server-team with tests labels May 23, 2023

atzoum force-pushed the feat.routerIsolation branch from b0c180a to e94c239 Compare May 23, 2023 15:44

atzoum changed the title ~~[WIP] feat(router): isolation modes with limiters~~ [WIP] feat(router): support for isolation modes using limiters May 24, 2023

atzoum force-pushed the feat.routerIsolation branch from e94c239 to f58d64c Compare May 24, 2023 06:11

github-actions bot added warehouse-team and removed server-team labels May 24, 2023

atzoum force-pushed the feat.routerIsolation branch 11 times, most recently from 0bc1a2c to b552588 Compare May 26, 2023 08:05

atzoum commented May 26, 2023

View reviewed changes

utils/payload/limiter_setup.go Show resolved Hide resolved

atzoum commented May 26, 2023

View reviewed changes

atzoum force-pushed the feat.routerIsolation branch 3 times, most recently from b6f698d to 5131cd0 Compare May 29, 2023 07:40

atzoum changed the title ~~[WIP] feat(router): support for isolation modes using limiters~~ feat(router): support for isolation modes using limiters May 29, 2023

atzoum requested a review from cisse21 May 29, 2023 15:13

atzoum force-pushed the feat.routerIsolation branch 13 times, most recently from db7358c to 9a87e48 Compare June 7, 2023 05:59

cisse21 reviewed Jun 7, 2023

View reviewed changes

atzoum force-pushed the feat.routerIsolation branch from 9a87e48 to b53907e Compare June 7, 2023 06:31

cisse21 reviewed Jun 7, 2023

View reviewed changes

router/worker.go Outdated Show resolved Hide resolved

cisse21 approved these changes Jun 7, 2023

View reviewed changes

lvrach approved these changes Jun 7, 2023

View reviewed changes

router/batchrouter/handle_observability.go Outdated Show resolved Hide resolved

atzoum force-pushed the feat.routerIsolation branch from b53907e to 03e1197 Compare June 8, 2023 09:18

atzoum added 3 commits June 8, 2023 12:27

feat(router): support for isolation modes using limiters

51a77cc

chore: add stats for pipeline delay (min, max)

114ef6c

chore: naming conventions

8c0f9d7

atzoum force-pushed the feat.routerIsolation branch from 2b97b12 to 8c0f9d7 Compare June 8, 2023 09:27

atzoum merged commit fbe109f into master Jun 8, 2023

atzoum deleted the feat.routerIsolation branch June 8, 2023 10:07

rudder-server-bot mentioned this pull request Jun 16, 2023

chore: prerelease 1.10.0-rc.1 #3506

Merged

devops-github-rudderstack mentioned this pull request Jun 16, 2023

chore: release 1.10.0 #3513

Merged

This was referenced Jun 21, 2023

chore: prerelease 1.10.0-rc.2 #3533

Merged

chore: prerelease 1.10.0-rc.3 #3544

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(router): support for isolation modes using limiters #3379

feat(router): support for isolation modes using limiters #3379

atzoum commented May 23, 2023 •

edited

Loading

codecov bot commented May 23, 2023 •

edited

Loading

atzoum May 26, 2023

atzoum May 26, 2023

atzoum May 26, 2023

atzoum May 26, 2023 •

edited

Loading

atzoum May 26, 2023

cisse21 Jun 7, 2023

atzoum Jun 7, 2023

cisse21 Jun 7, 2023

cisse21 Jun 7, 2023

atzoum Jun 8, 2023

		}

		// TODO: delete this once we remove the old fair pickup algorithm and move MultiTenantLegacy#GetAllJobs inside JobsDB

feat(router): support for isolation modes using limiters #3379

feat(router): support for isolation modes using limiters #3379

Conversation

atzoum commented May 23, 2023 • edited Loading

Description

Isolation modes

Limiters

Dynamic limiter priorities

Future cleanup work

Performance

Other notable changes

Notion Ticket

Security

codecov bot commented May 23, 2023 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

atzoum May 26, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

atzoum commented May 23, 2023 •

edited

Loading

codecov bot commented May 23, 2023 •

edited

Loading

atzoum May 26, 2023 •

edited

Loading