CRABServer helm: deploy all crab services in single helm chart

.gitignore

@@ -15,7 +15,7 @@ hmac
 # 3rd party packages
 kubernetes-prometheus
 
-# docker area 
+# docker area
 docker/sqoop/awg-ETL-crons/
 
 # skip configs/secrets
@@ -81,3 +81,5 @@ kubernetes/rucio/futureprod-db.yaml
 
 
 kubernetes/rucio/dev_grafana_password.txt
+
+!helm/crabserver/config

helm/crabserver/Chart.yaml

@@ -15,9 +15,9 @@ type: application
 # This is the chart version. This version number should be incremented each time you make changes
 # to the chart and its templates, including the app version.
 # Versions are expected to follow Semantic Versioning (https://semver.org/)
-version: 0.1.11
+version: 0.2.0
 
 # This is the version number of the application being deployed. This version number should be
 # incremented each time you make changes to the application. Versions are not expected to
 # follow Semantic Versioning. They should reflect the version the application is using.
-appVersion: "v3.240731"
+appVersion: "v3.240809"

helm/crabserver/README.md

@@ -1,19 +1,63 @@
-## Deploy CRAB
+# Deploy CRAB
 
-At the moment CRAB team is transitioning to using helm. However, we do noy like
-to use helm for deployment. See this 
-[comment](https://github.com/dmwm/CRABServer/issues/7843#issuecomment-2025085120)
-for some context.
+In this helm chart, we separate k8s resources into 3 resource groups:
 
-We like to use helm for templating, using the helm charts to generate the 
-manifest file, then applying it with `kubectl apply`.
+1. `crabserver` manifest for CRAB REST, including `crabserver-canary`.
+2. `logPipline`, the log pipeline which have logstash, filebeat and it config.
+3. `ciServiceAccount` with deployment permission, for deploying new image from CI. Note that only cluster admin could not `rolebinding` in testbed and production cluster.
 
-We therefore ask cmsweb operators to deploy crab following the procedure at
-[1].
+CRAB team will only handle the first resources group, `crabserver`, to deploy a new service version or modify how we start our service. The rest we kindly ask cluster admin to take care of.
 
-Changes to `../../kubernetes/cmsweb/services/crabserver.yaml` should not be committed.
+## Configuration (helm's value)
 
----
+There are helm value, `enabled`, that is used to control which resource group will be generated.
 
-[1] https://cmscrab.docs.cern.ch/technical/crab-rest/deploy.html#deploy-on-kubernetes-use-helm-template-to-generate-manifest-preferred
+#### enabled
 
+Default: `true`
+
+Generate `crabserver` resources group.
+
+#### canary.enabled
+
+Default: `false`
+
+Enable canary deployment, a.k.a. the `crabserver-canary` Deployment manifest.
+
+#### logPipeline.enabled
+
+Default: `false`
+
+Generate `logPipeline` resources group.
+
+#### ciServiceAccount.enabled
+
+Generate `ciServiceAccount` resources group. Need cluster admin for applying `role` and `rolebinding` of k8s service account.
+
+## Deploy
+
+### For cluster admins
+
+To deploy everything on new testbed and production cluster:
+
+1. Deploy `crabserver-secrets` credential.
+
+2. Deploy all services:
+    ```
+    helm install crabserver . -f values.yaml -f values-clusteradmin.yaml --set environment=preprod
+    ```
+    Please change `environment=preprod` to the cluster you are deploying (`preprod` for testbed and preprod cluster, `prod` for production cluster).
+
+Please leave test cluster to CRAB operators.
+
+### For CRAB operators
+
+Please consult [Deploying CRAB REST](https://cmscrab.docs.cern.ch/crab-components/crab-rest/deploy.html).
+
+## Regarding deploy with `helm install` command
+
+CRAB team does not use helm for deployment. We like to use helm for templating, using the helm charts to generate the manifest file, then applying it with `kubectl apply`. See this [comment](https://github.com/dmwm/CRABServer/issues/7843#issuecomment-2025085120) for some context.
+
+However, it should not have any conflict if cluster admins run `helm install` **before** CRAB operators apply new manifests with `kubectl apply`.
+
+In case of conflict, feel free to purge all resources and reinstall with helm again.

helm/crabserver/apply_logstash.sh

@@ -0,0 +1,6 @@
+#!/bin/bash
+set -x
+set -euo pipefail
+ENV=test
+kubectl create configmap logstash-crab --from-file=config/${ENV}/logstash/logstash.conf --from-file config/${ENV}/logstash/logstash.yml --dry-run=client -oyaml | kubectl apply -f -
+kubectl delete pod $(kubectl get pod --no-headers -o custom-columns=":metadata.name" | grep logstash)

helm/crabserver/config/preprod/config.py

@@ -0,0 +1,49 @@
+from WMCore.Configuration import Configuration
+import re
+import socket
+import time
+import os
+
+myhost = socket.getfqdn().lower()
+
+conf = Configuration()
+main = conf.section_('main')
+srv = main.section_('server')
+srv.thread_pool = int(os.getenv('CRABSERVER_THREAD_POOL', 15))
+main.application = 'crabserver'
+main.port = 8270
+main.index = 'data'
+
+main.authz_defaults = { 'role': None, 'group': None, 'site': None }
+main.section_('tools').section_('cms_auth').key_file = "%s/auth/crabserver/header-auth-key" % __file__.rsplit('/', 3)[0]
+
+app = conf.section_('crabserver')
+app.admin = 'cms-service-webtools@cern.ch'
+app.description = 'CRABServer RESTFull API'
+app.title = 'CRABRESTFull'
+
+views = conf.section_('views')
+ui = views.section_('ui')
+ui.object = 'CRABInterface.Pages.FrontPage.FrontPage'
+
+data = views.section_('data')
+data.object = 'CRABInterface.RESTBaseAPI.RESTBaseAPI'
+data.phedexurl = 'https://cmsweb.cern.ch/phedex/datasvc/xml/prod/'
+data.dbsurl = 'http://cmsdbsprod.cern.ch/cms_dbs_prod_global/servlet/DBSServlet'
+data.defaultBlacklist = ['T0_CH_CERN']
+data.serverhostcert = "%s/auth/crabserver/dmwm-service-cert.pem" % __file__.rsplit('/', 3)[0]
+data.serverhostkey = "%s/auth/crabserver/dmwm-service-key.pem" % __file__.rsplit('/', 3)[0]
+data.credpath = '%s/state/crabserver/proxy/' % __file__.rsplit('/', 4)[0]
+data.backend = 'oracle'
+data.db = 'CRABServerAuth.dbconfig'
+data.s3 = 'CRABServerAuth.s3'
+data.workflowManager = 'HTCondorDataWorkflow'
+
+data.extconfigurl = 'http://gitlab.cern.ch/crab3/CRAB3ServerConfig/raw/master/cmsweb-rest-config.json'
+
+data.loggingLevel = 10
+data.loggingFile = '%s/logs/crabserver/CRAB-%s.log' % (__file__.rsplit('/', 4)[0], myhost)
+data.keptLogDays = 7
+data.mode = "cmsweb-preprod"
+
+data.delegateDN = "/DC=ch/DC=cern/OU=computers/CN=crab-(preprod|prod)-tw(01|02).cern.ch|/DC=ch/DC=cern/OU=computers/CN=crab-dev-tw(01|02|03|04).cern.ch|/DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=cmscrab/CN=(817881|373708)/CN=Robot: cms crab|/DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=crabint1/CN=373708/CN=Robot: CMS CRAB Integration 1"

kubernetes/cmsweb/monitoring/crab/logstash.conf → helm/crabserver/config/preprod/logstash/logstash.conf

@@ -176,8 +176,8 @@ filter {
       }
 
       grok {
-          # [07/Dec/2022:20:06:47] crabserver-56c69685b6-477jr 188.184.91.103:41924 "GET /crabserver/devthree/info HTTP/1.1" 200 OK [data: 9354 in 103 out 4870 us ] [auth: ok "/DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=tseethon/CN=856006/CN=Thanayut Seethongchuen" "" ] [ref: "https://cmsweb-test12.cern.ch" "Mozilla/5.0 (X11; Linux x86_64; rv:107.0) Gecko/20100101 Firefox/107.0" ] Type=cherrypylog
-+         match => { "message" => '\[%{NOTSPACE:timestamp_temp}\] %{DATA:backend} %{IPORHOST:clientip}(|:%{NUMBER}) "%{WORD:method} %{NOTSPACE:request} %{DATA:httpversion}" %{NUMBER:code:int} %{DATA} \[data: (-|%{NUMBER:bytes_sent:int}) in (-|%{NUMBER:bytes_received:int}) out (-|%{NUMBER:time_spent_ms:int}) us \] \[auth: %{DATA} "%{DATA:dn}".*\] \[ref: "%{DATA}.*" "%{DATA:client}" \]' }
+          # [07/Dec/2022:20:06:47] crabserver-56c69685b6-477jr 188.184.91.103:41924 "GET /crabserver/devthree/info HTTP/1.1" 200 OK [data: 9354 in 103 out 4870 us ] [auth: ok "/DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=tseethon/CN=856006/CN=Thanayut Seethongchuen" "" ] [ref: "https://cmsweb-test12.cern.ch" "Mozilla/5.0 (X11; Linux x86_64; rv:107.0) Gecko/20100101 Firefox/107.0" ] - Podname=crabserver-6cf49bcfff-bxvb6 Type=cherrypylog
+          match => { "message" => '\[%{NOTSPACE:timestamp_temp}\] %{DATA:backend} %{IPORHOST:clientip}(|:%{NUMBER}) "%{WORD:method} %{NOTSPACE:request} %{DATA:httpversion}" %{NUMBER:code:int} %{DATA} \[data: (-|%{NUMBER:bytes_sent:int}) in (-|%{NUMBER:bytes_received:int}) out (-|%{NUMBER:time_spent_ms:int}) us \] \[auth: %{DATA} "%{DATA:dn}".*\] \[ref: "%{DATA}.*" "%{DATA:client}" \]' }
       }
       grok {
         match => { "request" => '/%{WORD:system}%{UNIXPATH:uri_path}%{URIPARAM:uri_params}?' }

helm/crabserver/config/prod/config.py

@@ -0,0 +1,47 @@
+from WMCore.Configuration import Configuration
+import re
+import socket
+import time
+import os
+
+myhost = socket.getfqdn().lower()
+
+conf = Configuration()
+main = conf.section_('main')
+srv = main.section_('server')
+srv.thread_pool = int(os.getenv('CRABSERVER_THREAD_POOL', 15))
+main.application = 'crabserver'
+main.port = 8270
+main.index = 'data'
+
+main.authz_defaults = { 'role': None, 'group': None, 'site': None }
+main.section_('tools').section_('cms_auth').key_file = "%s/auth/crabserver/header-auth-key" % __file__.rsplit('/', 3)[0]
+
+app = conf.section_('crabserver')
+app.admin = 'cms-service-webtools@cern.ch'
+app.description = 'CRABServer RESTFull API'
+app.title = 'CRABRESTFull'
+
+views = conf.section_('views')
+ui = views.section_('ui')
+ui.object = 'CRABInterface.Pages.FrontPage.FrontPage'
+
+data = views.section_('data')
+data.object = 'CRABInterface.RESTBaseAPI.RESTBaseAPI'
+data.phedexurl = 'https://cmsweb.cern.ch/phedex/datasvc/xml/prod/'
+data.dbsurl = 'http://cmsdbsprod.cern.ch/cms_dbs_prod_global/servlet/DBSServlet'
+data.defaultBlacklist = ['T0_CH_CERN']
+data.serverhostcert = "%s/auth/crabserver/dmwm-service-cert.pem" % __file__.rsplit('/', 3)[0]
+data.serverhostkey = "%s/auth/crabserver/dmwm-service-key.pem" % __file__.rsplit('/', 3)[0]
+data.credpath = '%s/state/crabserver/proxy/' % __file__.rsplit('/', 4)[0]
+data.backend = 'oracle'
+data.db = 'CRABServerAuth.dbconfig'
+data.s3 = 'CRABServerAuth.s3'
+data.workflowManager = 'HTCondorDataWorkflow'
+
+data.extconfigurl = 'http://gitlab.cern.ch/crab3/CRAB3ServerConfig/raw/master/cmsweb-rest-config.json'
+
+data.loggingLevel = 10
+data.loggingFile = '%s/logs/crabserver/CRAB-%s.log' % (__file__.rsplit('/', 4)[0], myhost)
+data.keptLogDays = 7
+data.mode = "cmsweb-prod"