Lower data-layer TTLs (#1962)

We currently have two TTLs on many services: the minTTL that is enforced via the database lock, and a minTTL that is enforced by the service being invoked by Cloud Scheduler. There were some mismatches over which was the "authorative" schedule. This commit lowers the minTTLs on most services to 5min. This will prevent concurrent runs and crazy DOsing. However, the _intended_ execution schedule's source of truth is Cloud Scheduler.

I believe this is the root cause of the recent alert, because the "4 hour" windows between the backup job's minTTL and Cloud Scheduler's 4 hours misaligned.

pkg/config/backup_config.go

@@ -36,10 +36,9 @@ type BackupConfig struct {
 	Port string `env:"PORT, default=8080"`
 
 	// MinTTL is the minimum amount of time that must elapse between attempting
-	// backups. This is used to control whether the pull is actually attempted at
-	// the controller layer, independent of the data layer. In effect, it rate
-	// limits the number of requests.
-	MinTTL time.Duration `env:"BACKUP_MIN_PERIOD, default=4h"`
+	// backups. This is used to control whether the backup is actually executed
+	// and to prevent concurrent backups.
+	MinTTL time.Duration `env:"BACKUP_MIN_PERIOD, default=5m"`
 
 	// Timeout is the maximum amount of time to wait for a backup operation to
 	// complete.

pkg/config/cleanup_server_config.go

@@ -46,7 +46,7 @@ type CleanupConfig struct {
 	// Cleanup config
 	AuditEntryMaxAge    time.Duration `env:"AUDIT_ENTRY_MAX_AGE, default=720h"`
 	AuthorizedAppMaxAge time.Duration `env:"AUTHORIZED_APP_MAX_AGE, default=336h"`
-	CleanupMinPeriod    time.Duration `env:"CLEANUP_MIN_PERIOD, default=15m"`
+	CleanupMinPeriod    time.Duration `env:"CLEANUP_MIN_PERIOD, default=5m"`
 	// KeyServerStatsMaxAge is the maximum amount of time to retain key-server stats.
 	KeyServerStatsMaxAge time.Duration `env:"KEY_SERVER_STATS_MAX_AGE, default=720h"`
 	MobileAppMaxAge      time.Duration `env:"MOBILE_APP_MAX_AGE, default=168h"`

pkg/config/rotation_config.go

@@ -43,7 +43,7 @@ type RotationConfig struct {
 	// rotation events. This is used to control whether rotation is actually
 	// attempted at the controller layer, independent of the data layer. In
 	// effect, it rate limits the number of rotation requests.
-	MinTTL time.Duration `env:"MIN_TTL, default=15m"`
+	MinTTL time.Duration `env:"MIN_TTL, default=5m"`
 
 	// TokenSigning is the token signing configuration. This defines the parent
 	// key and common data like issuer, but the individual versions are controlled

pkg/config/stats_puller_config.go

@@ -54,7 +54,7 @@ type StatsPullerConfig struct {
 	// stats-pull events. This is used to control whether the pull is actually
 	// attempted at the controller layer, independent of the data layer. In
 	// effect, it rate limits the number of rotation requests.
-	MinTTL time.Duration `env:"MIN_TTL, default=15m"`
+	MinTTL time.Duration `env:"MIN_TTL, default=5m"`
 
 	// StatsPullerMinPeriod defines the period for which the stats puller will hold a lock
 	// which prevents other calls from entering.