Configurable resource quotas (#27)

* Add configurable resource quotas for the service and spawned jobs
keptn-contrib · Jun 23, 2021 · cb8ba14 · cb8ba14
1 parent 9c5ae4b
commit cb8ba14
Show file tree

Hide file tree

Showing 15 changed files with 350 additions and 80 deletions.
diff --git a/README.md b/README.md
@@ -58,17 +58,17 @@ actions:
 The tasks of an action are executed if the event name matches. Wildcards can also be used, e.g.
 
 ```yaml
-    - name: "sh.keptn.event.*.triggered"
+- name: "sh.keptn.event.*.triggered"
 ```
 
 Would match events `sh.keptn.event.test.triggered`, `sh.keptn.event.deployment.triggered` and so on.
 
 Optionally the following section can be added to an event:
 
 ```yaml
-      jsonpath:
-        property: "$.data.test.teststrategy"
-        match: "locust"
+jsonpath:
+  property: "$.data.test.teststrategy"
+  match: "locust"
 ```
 
 If the service receives an event which matches the name, and the jsonpath match expression, the specified tasks are
@@ -106,14 +106,14 @@ executed. E.g. the following cloud event would match the jsonpath above:
 The configuration contains the following section:
 
 ```yaml
-    tasks:
-      - name: "Run locust smoke tests"
-        files:
-          - locust/basic.py
-          - locust/import.py
-          - locust/locust.conf
-        image: "locustio/locust"
-        cmd: "locust --config /keptn/locust/locust.conf -f /keptn/locust/basic.py --host $(HOST)"
+tasks:
+  - name: "Run locust smoke tests"
+    files:
+      - locust/basic.py
+      - locust/import.py
+      - locust/locust.conf
+    image: "locustio/locust"
+    cmd: "locust --config /keptn/locust/locust.conf -f /keptn/locust/basic.py --host $(HOST)"
 ```
 
 It contains the tasks which should be executed as Kubernetes job. The service schedules a different job for each of
@@ -135,11 +135,11 @@ The following environment variable has the name `HOST`, and the value is whateve
 jsonpath `$.data.deployment.deploymentURIsLocal[0]` resolves to.
 
 ```yaml
-        cmd: "locust --config /keptn/locust/locust.conf -f /keptn/locust/basic.py --host $(HOST)"
-        env:
-          - name: HOST
-            value: "$.data.deployment.deploymentURIsLocal[0]"
-            valueFrom: event
+cmd: "locust --config /keptn/locust/locust.conf -f /keptn/locust/basic.py --host $(HOST)"
+env:
+  - name: HOST
+    value: "$.data.deployment.deploymentURIsLocal[0]"
+    valueFrom: event
 ```
 
 In the above example the json path for `HOST` would resolve into `https://keptn.sh` for the below event
@@ -184,10 +184,10 @@ The following configuration looks up a kubernetes secret with the name `locust-s
 secret will be available as separate environment variables in the job.
 
 ```yaml
-        cmd: "locust --config /keptn/locust/locust.conf -f /keptn/locust/$(FILE) --host $(HOST)"
-        env:
-          - name: locust-secret
-            valueFrom: secret
+cmd: "locust --config /keptn/locust/locust.conf -f /keptn/locust/$(FILE) --host $(HOST)"
+env:
+  - name: locust-secret
+    valueFrom: secret
 ```
 
 With the secret below, there will be two environment variables available in the job. `HOST` with the
@@ -214,10 +214,10 @@ metadata:
 Files can be added to your running tasks by specifying them in the `files` section of your tasks:
 
 ```yaml
-        files:
-          - locust/basic.py
-          - locust/import.py
-          - locust/locust.conf
+files:
+  - locust/basic.py
+  - locust/import.py
+  - locust/locust.conf
 ```
 
 This is done by using an `initcontainer` for the scheduled Kubernetes Job which prepares the `èmptyDir` volume mounted
@@ -228,7 +228,7 @@ When using these files in your container command, please make sure to reference
 E.g.:
 
 ```yaml
-        cmd: "locust --config /keptn/locust/locust.conf -f /keptn/locust/basic.py"
+cmd: "locust --config /keptn/locust/locust.conf -f /keptn/locust/basic.py"
 ```
 
 ### Silent mode
@@ -244,6 +244,64 @@ actions:
     silent: true
 ```
 
+### Resource quotas for jobs
+
+The `initcontainer` and the `job` container will use the default resource quotas defined as environment variables. They
+can be set in [`deploy/service.yaml`](deploy/service.yaml):
+
+```yaml
+- name: DEFAULT_RESOURCE_LIMITS_CPU
+  value: "1"
+- name: DEFAULT_RESOURCE_LIMITS_MEMORY
+  value: "512Mi"
+- name: DEFAULT_RESOURCE_REQUESTS_CPU
+  value: "50m"
+- name: DEFAULT_RESOURCE_REQUESTS_MEMORY
+  value: "128Mi"
+```
+
+or for helm in [`helm/templates/configmap.yaml`](helm/templates/configmap.yaml):
+
+```yaml
+default_resource_limits_cpu: "1"
+default_resource_limits_memory: "512Mi"
+default_resource_requests_cpu: "50m"
+default_resource_requests_memory: "128Mi"
+```
+
+The default resource quotas can be easily overwritten for each task. Add the following block to the configuration on
+task level:
+
+```yaml
+tasks:
+  - name: "Run locust smoke tests"
+    ...
+    resources:
+      limits:
+        cpu: 1
+        memory: 512Mi
+      requests:
+        cpu: 50m
+        memory: 128Mi
+```
+
+Now each job that gets spawned for the task will have the configured resource quotas. There is no need to specify all
+values, as long as the configuration makes sense for kubernetes. E.g. the following configuration
+
+```yaml
+tasks:
+  - name: "Run locust smoke tests"
+    ...
+    resources:
+      limits:
+        cpu: 1
+      requests:
+        cpu: 50m
+```
+
+would result in resource quotas for `cpu`, but in none for `memory`. If the `resources` block is present
+(even if empty), all default resource quotas are ignored for this task.
+
 ### Remote Control Plane
 
 If you are using the service in a remote control plane setup make sure the distributor is configured to forward all
@@ -384,8 +442,8 @@ To make use of the built-in automation using GH Actions for releasing a new vers
 * check the output of GH Actions builds for the release branch,
 * verify that your image was built and pushed to DockerHub with the right tags,
 * update the image tags for `job-executor-service` and `job-executor-service-initcontainer`
-  in [deploy/service.yaml](deploy/service.yaml), [helm/Chart.yaml](helm/Chart.yaml),
-  [helm/values.yaml](helm/values.yaml) and [helm/templates/configmap.yaml](helm/templates/configmap.yaml),
+  in [`deploy/service.yaml`](deploy/service.yaml), [`helm/Chart.yaml`](helm/Chart.yaml),
+  [`helm/values.yaml`](helm/values.yaml) and [`helm/templates/configmap.yaml`](helm/templates/configmap.yaml),
 * test your service against a working Keptn installation.
 
 If any problems occur, fix them in the release branch and test them again.

diff --git a/cmd/job-executor-service/main.go b/cmd/job-executor-service/main.go
@@ -3,7 +3,9 @@ package main
 import (
 	"context"
 	"errors"
+	v1 "k8s.io/api/core/v1"
 	"keptn-sandbox/job-executor-service/pkg/eventhandler"
+	"keptn-sandbox/job-executor-service/pkg/k8sutils"
 	"log"
 	"os"
 	"strings"
@@ -32,13 +34,25 @@ type envConfig struct {
 	JobNamespace string `envconfig:"JOB_NAMESPACE" required:"true"`
 	// The token of the keptn API
 	KeptnAPIToken string `envconfig:"KEPTN_API_TOKEN"`
-	// The token of the keptn API
+	// The init container image to use
 	InitContainerImage string `envconfig:"INIT_CONTAINER_IMAGE"`
+	// Default resource limits cpu for job and init container
+	DefaultResourceLimitsCPU string `envconfig:"DEFAULT_RESOURCE_LIMITS_CPU"`
+	// Default resource limits memory for job and init container
+	DefaultResourceLimitsMemory string `envconfig:"DEFAULT_RESOURCE_LIMITS_MEMORY"`
+	// Default resource requests cpu for job and init container
+	DefaultResourceRequestsCPU string `envconfig:"DEFAULT_RESOURCE_REQUESTS_CPU"`
+	// Default resource requests memory for job and init container
+	DefaultResourceRequestsMemory string `envconfig:"DEFAULT_RESOURCE_REQUESTS_MEMORY"`
 }
 
 // ServiceName specifies the current services name (e.g., used as source when sending CloudEvents)
 const ServiceName = "job-executor-service"
 
+// DefaultResourceRequirements contains the default k8s resource requirements for the job and initcontainer, parsed on
+// startup from env (treat as const)
+var /* const */ DefaultResourceRequirements *v1.ResourceRequirements
+
 /**
  * Parses a Keptn Cloud Event payload (data attribute)
  */
@@ -77,14 +91,17 @@ func processKeptnCloudEvent(ctx context.Context, event cloudevents.Event) error
 	}
 
 	eventHandler := &eventhandler.EventHandler{
-		Keptn:        myKeptn,
-		Event:        event,
-		EventData:    eventData,
-		ServiceName:  ServiceName,
-		JobNamespace: env.JobNamespace,
-		InitContainerConfigurationServiceAPIEndpoint: env.InitContainerConfigurationServiceAPIEndpoint,
-		KeptnAPIToken:      env.KeptnAPIToken,
-		InitContainerImage: env.InitContainerImage,
+		Keptn:       myKeptn,
+		Event:       event,
+		EventData:   eventData,
+		ServiceName: ServiceName,
+		JobSettings: k8sutils.JobSettings{
+			JobNamespace: env.JobNamespace,
+			InitContainerConfigurationServiceAPIEndpoint: env.InitContainerConfigurationServiceAPIEndpoint,
+			KeptnAPIToken:               env.KeptnAPIToken,
+			InitContainerImage:          env.InitContainerImage,
+			DefaultResourceRequirements: DefaultResourceRequirements,
+		},
 	}
 
 	// prevent duplicate events - https://github.com/keptn/keptn/issues/3888
@@ -105,6 +122,17 @@ func main() {
 		log.Fatalf("Failed to process env var: %s", err)
 	}
 
+	var err error
+	DefaultResourceRequirements, err = k8sutils.CreateResourceRequirements(
+		env.DefaultResourceLimitsCPU,
+		env.DefaultResourceLimitsMemory,
+		env.DefaultResourceRequestsCPU,
+		env.DefaultResourceRequestsMemory,
+	)
+	if err != nil {
+		log.Fatalf("unable to create default resource requirements: %v", err.Error())
+	}
+
 	os.Exit(_main(os.Args[1:], env))
 }
 

diff --git a/deploy/service.yaml b/deploy/service.yaml
@@ -20,6 +20,13 @@ spec:
           image: keptnsandbox/job-executor-service:0.1.1
           ports:
             - containerPort: 8080
+          resources:
+            limits:
+              cpu: 1
+              memory: 512Mi
+            requests:
+              cpu: 50m
+              memory: 128Mi
           env:
             - name: INIT_CONTAINER_CONFIGURATION_SERVICE_API_ENDPOINT
               value: "http://configuration-service:8080"
@@ -29,6 +36,14 @@ spec:
               value: 'keptn'
             - name: INIT_CONTAINER_IMAGE
               value: 'keptnsandbox/job-executor-service-initcontainer:0.1.1'
+            - name: DEFAULT_RESOURCE_LIMITS_CPU
+              value: "1"
+            - name: DEFAULT_RESOURCE_LIMITS_MEMORY
+              value: "512Mi"
+            - name: DEFAULT_RESOURCE_REQUESTS_CPU
+              value: "50m"
+            - name: DEFAULT_RESOURCE_REQUESTS_MEMORY
+              value: "128Mi"
         - name: distributor
           image: keptn/distributor:0.8.3
           livenessProbe:
@@ -42,11 +57,11 @@ spec:
             - containerPort: 8080
           resources:
             requests:
-              memory: "16Mi"
-              cpu: "25m"
+              memory: "32Mi"
+              cpu: "50m"
             limits:
               memory: "128Mi"
-              cpu: "250m"
+              cpu: "500m"
           env:
             - name: PUBSUB_URL
               value: 'nats://keptn-nats-cluster'

diff --git a/helm/templates/configmap.yaml b/helm/templates/configmap.yaml
@@ -5,3 +5,7 @@ metadata:
 data:
   job_namespace: "{{ .Release.Namespace }}"
   init_container_image: "{{ .Values.jobexecutorserviceinitcontainer.image.repository }}:{{ .Values.jobexecutorserviceinitcontainer.image.tag | default .Chart.AppVersion }}"
+  default_resource_limits_cpu: "1"
+  default_resource_limits_memory: "512Mi"
+  default_resource_requests_cpu: "50m"
+  default_resource_requests_memory: "128Mi"
diff --git a/helm/templates/deployment.yaml b/helm/templates/deployment.yaml
@@ -62,6 +62,26 @@ spec:
               configMapKeyRef:
                 name: job-service-config
                 key: init_container_image
+          - name: DEFAULT_RESOURCE_LIMITS_CPU
+            valueFrom:
+              configMapKeyRef:
+                name: job-service-config
+                key: default_resource_limits_cpu
+          - name: DEFAULT_RESOURCE_LIMITS_MEMORY
+            valueFrom:
+              configMapKeyRef:
+                name: job-service-config
+                key: default_resource_limits_memory
+          - name: DEFAULT_RESOURCE_REQUESTS_CPU
+            valueFrom:
+              configMapKeyRef:
+                name: job-service-config
+                key: default_resource_requests_cpu
+          - name: DEFAULT_RESOURCE_REQUESTS_MEMORY
+            valueFrom:
+              configMapKeyRef:
+                name: job-service-config
+                key: default_resource_requests_memory
           livenessProbe:
             httpGet:
               path: /health

diff --git a/helm/values.yaml b/helm/values.yaml
@@ -58,15 +58,11 @@ securityContext: { }                          # Set the security context (e.g. r
 #  runAsUser: 1000
 
 resources: # Resource limits and requests
-  # We usually recommend not to specify default resources and to leave this as a conscious
-  # choice for the user. This also increases chances charts run on environments with little
-  # resources, such as Minikube. If you do want to specify resources, uncomment the following
-  # lines, adjust them as necessary, and remove the curly braces after 'resources:'.
-  # limits:
-  #   cpu: 100m
-  #   memory: 128Mi
+  limits:
+    cpu: 1
+    memory: 512Mi
   requests:
-    cpu: 100m
+    cpu: 50m
     memory: 128Mi
 
 nodeSelector: { }                                # Node selector configuration

diff --git a/pkg/config/config.go b/pkg/config/config.go
@@ -38,11 +38,12 @@ type JSONPath struct {
 
 // Task this is the actual task which can be triggered within an Action
 type Task struct {
-	Name  string   `yaml:"name"`
-	Files []string `yaml:"files"`
-	Image string   `yaml:"image"`
-	Cmd   string   `yaml:"cmd"`
-	Env   []Env    `yaml:"env"`
+	Name      string     `yaml:"name"`
+	Files     []string   `yaml:"files"`
+	Image     string     `yaml:"image"`
+	Cmd       string     `yaml:"cmd"`
+	Env       []Env      `yaml:"env"`
+	Resources *Resources `yaml:"resources"`
 }
 
 // Env value from the event which will be added as env to the job
@@ -52,6 +53,18 @@ type Env struct {
 	ValueFrom string `yaml:"valueFrom"`
 }
 
+// Resources defines the resource requirements of a task
+type Resources struct {
+	Limits   ResourceList `yaml:"limits"`
+	Requests ResourceList `yaml:"requests"`
+}
+
+// ResourceList contains resource requirement keys
+type ResourceList struct {
+	CPU    string `yaml:"cpu"`
+	Memory string `yaml:"memory"`
+}
+
 // NewConfig creates a new configuration from the provided config file content
 func NewConfig(yamlContent []byte) (*Config, error) {