-
Notifications
You must be signed in to change notification settings - Fork 107
Injecting, assigning and validating test requests
This is a quick tutorial aimed for the creation and assignment of test requests into cmsweb(-testbed) or in your own private machine. In order to create these requests, you need to:
- have a valid proxy in your environment (X509_USER_PROXY)
- download this script: https://github.com/dmwm/WMCore/blob/master/test/data/ReqMgr/inject-test-wfs.py
In order to use WMAgent setup please follow these steps:
-
obtain access to one of the WMAgent nodes, e.g. in this example we will use vocms0262
-
login to the node
-
switch to
cmst1
account
amaltaro@vocms0262:~ $ sudo -u cmst1 /bin/bash --init-file ~cmst1/.bashrc
cmst1@vocms0262:/afs/cern.ch/user/a/amaltaro $
If you don't have access to cmst1
account please request it from dmwm/WMCore
admins, your name should appear in the following
puppet config
- setup WMAgent environment
## setup the agent environment
$ agentenv
## optionally you may also set your PYTHON path using wmagent init script
source /data/srv/wmagent/current/apps/wmagentpy3/etc/profile.d/init.sh
- perform some action with WMAgent
# start services (CouchDB + MariaDB)
$ $manage start-services (or to stop: $manage stop-services)
# start all the components (WorkQueueManager, DBS3Upload, and a bunch others)
$ $manage start-agent (or to stop: $manage stop-agent)
# restart a single component
$manage execute-agent wmcoreD --restart --components=JobSubmitter
# shutdown a single component
$manage execute-agent wmcoreD --shutdown --components=JobSubmitter
Here is a simple example on how to quickly inject a test workflow
If you use key and cert. (need to set the correct location not exact copy of following
> export X509_USER_CERT=/data/srv/current/auth/reqmgr2/dmwm-service-cert.pem
> export X509_USER_KEY=/data/srv/current/auth/reqmgr2/dmwm-service-key.pem
If you use proxy
> export X509_USER_PROXY=/data/cert/myproxy.pem
in the test directory you want to install test code
> mkdir test_injection
> cd test_injection
> curl https://raw.githubusercontent.com/dmwm/WMCore/master/test/data/ReqMgr/inject-test-wfs.py > inject-test-wfs.py
> python inject-test-wfs.py -u "https://reqmgr2-dev.cern.ch" -m Integration -f TaskChainPyquenZeemumuJets_PU.json -c Agent105_Validation -r Agent105_CDB16_forceComplete -t testbed-dev -a Integ_Test -p forceComplete_Alan_TEST_v13
In summary, here is what this script can do for you:
- it will clone WMCore repository (or it can use a local WMCore copy)
- it will use the json templates available under "WMCore/test/data/ReqMgr/requests/", either from DMWM or Integration directories.
- in general, DMWM templates should create less and smaller jobs, so you get faster results
- the Integration templates are meant for full WMAgent or CMSWEB validations, usually longer jobs.
- it will create AND assign requests for you (against ReqMgr2, by default), according to the parameters given in the command line
- you can also only inject workflows, skipping the assignment part.
Some useful examples (don't forget to adapt parameters according to your needs):
1. Injecting and assigning ALL templates under DMWM directory, against cmsweb-testbed
python inject-test-wfs.py -m DMWM -c DMWM_Test -r Test_bugFix_4scriptInject -t testbed-cmssrv95 -a DMWM_Test -p bugFix_TEST_injectScript_v4
where:
- -m: mode, either DMWM or Integration directories
- -c: campaign name (so you can look for this campaign in WMStats)
- -r: request string (how your workflow will be named, excluding the usual username and timestamp)
-
-t: team name (the agent you want to assign this workflow), the team name
can be found by visiting
wmstats->agent info
tab in wmstats - -a: acquisition era (for TaskChain workflows, it will use the CMSSWVersion from the template - due to DQM harvesting rules...)
- -p: processing string (for TaskChain workflows, it will append your string provided to the TaskName)
1. Injecting and assigning a specific template against my private services:
python inject-test-wfs.py -u "https://alan-cloud1.cern.ch" -m Integration -f TaskChainPyquenZeemumuJets_PU.json -c Agent105_Validation -r Agent105_CDB16_forceComplete -t alan-devvm -a Integ_Test -p forceComplete_Alan_TEST_v13
1. Only injecting a specific workflow but do not assigning it:
python inject-test-wfs.py -m DMWM -f MonteCarlo_eff.json -c Agent105_Validation -r Agent105_CDB16_forceComplete --injectOnly
When these workflows move to 'completed' status, then we can use a secondary script to do some request validation against couch/dbs/phedex. It would check the input x output lumi section, events, number of files and etc.
Here is how you can use this script to validate a workflow (you need to have the env variable set):
> curl https://raw.githubusercontent.com/dmwm/WMCore/master/test/data/ReqMgr/validate-test-wfs.py > validate-test-wfs.py
> python validate-test-wfs.py -r alan-cloud1.cern.ch -w WORKFLOW_NAME -v
or, you can pass a text file with the workflow names you want to validate, like
> python validate-test-wfs.py -i FILE_NAME -v
where:
- -w: a workflow name
- -i: a input text file
- -c: Url for DBS/PhEDEx (it defaults to cmsweb-testbed)
- -r: Url for the reqmgr instance (it defaults to cmsweb-testbed)
- -v: enables verbose mode