refactor: Refactor load test (#1163)

* fix(KONFLUX-2465): Refresh framework before access token expires on Stage

* fix: Fix panic because NewFramework do not provide options at all

* refactor: Rewrite loattest

* cleanup: Remove unneeded option

* chore: Use correct integration-service repo

* refactor: Simplify handling JSON with Stage user secrets

* feat: Make it possible to run user journey multiple times

* refactor: Merge two related packages and log FAILures to separate CSV

* feat: Make it possible to skip test and just run the purge

* refactor: Move user init fron user journey function so we do not have to handle PurgeOnly there

* fix: Remove gomega from Github.CreateRef method

Changes in wait for branch to appear discussed with Pavel Sturc in https://redhat-internal.slack.com/archives/C02FANRBZQD/p1713856264091809

* feat: Delete more resources when purging in Stage as these are staying behind otherwise

* feat(KONFLUX-2495): Also measure signing speed

* feat: Measure more relevant functions

* feat: Script to summarize timings

* feat(KONFLUX-2086): Make it possible to create multiple apps per user

* fix: Make these in-progress status messages just a trace level

* style: gofmt

* fix(KFLUXBUGS-1296): Template repo by creating fork, not just branch as that is not working, see KFLUXBUGS-1296

* fix: Format these debug messages properly

* fix: Skip purge if disabled and imply purge on purge-only

* feat: Dump options we got to file for future refference

* docs: Added/improved code comments

* cleanup: This function is not needed and import stuff in grepable way

* style: "io/ioutil" has been deprecated since Go 1.19

* lint: unreachable: unreachable code (govet)

* lint: S1021: should merge variable declaration with assignment on next line (gosimple)

* lint: printf: fmt.Errorf call has arguments but no formatting directives (govet)

* refactor: Setup app and comp name in journey as that will allow additional repo templating

* fix: We need more templating to configure app and comp name in the PaC yaml

* fix: gosec: Expect file permissions to be 0600 or less

* feat: Commit to PaC to trigger build

* Revert "feat: Commit to PaC to trigger build"

This reverts commit 27d2798884b5851e9e868ea6a5efc16e3766a85d.

As per Stuart D., to get initial build, I need to merge the PR instead

* feat: When creating PaC components, also wait for 'build.appstudio.openshift.io/status' annotation

* fix: Remove --component-repo-template as we have --pipeline-request-configure-pac that implies repo templating

* fix: 1 of 2: Handle multi-arch PaC usecase

* feat: Add function GetComponentPipelineRunsWithType to list all matching PRs, not just first one

* feat: Create function DeletePipelineRunIgnoreFinalizers to delete just one PR

* fix: 2 of 2: Handle multi-arch PaC usecase

* style: Make function calls readable

* feat(KONFLUX-746): Abstract YAMLs gathering, add script we can use in Stage

* feat: Tool to visualize when PipelineRuns and TaskRuns were running

* fix: Average of error rates value does not make sense, use just count of errors

* fix: Collect info from all users

* fix: I changed significant fields in the JSON report, changing reporting tool as well

* feat: Record function parameters in log message for easier debugging

* feat: Print parameters sorted by key (i.e. type)

* fix: Capture all params even if more than one have same types

* feat: Also add pod logs

* feat: Labels in the middle of the TaskRun and ability to print INFO+ log level

* feat: Also store pod logs

* feat: Also collect JSONs of PRs and TRs and Pods

* feat: Load everithing, not just 'kind: List'

* fix: Collect data even when something failed

* fix: Pass expected number of PRs to list (and delete)

* docs: Typo and add repo URL to the log message

* feat: Add script to breakdown PipelineRuns by TaskRuns by duration, used for multi-arch workflow

* docs: Simplify the log message

* feat: Disable collecting pod/trs/prs as it is now collected by loadtest itself

* feat: Fixing Stage GH action

* docs: Fix loadtest description

* chore: Repos moved

* feat: More removal CDQ and BPS

* fix: Do not wait in collect

* fix: Simplify users.json splitting script

* feat: Show also failed TRs

* feat: Load failing PRs and TRs as well and rename one var to be more clear what is it for

* feat: Make failed PRs/TRs appear on the graph, but hatched

* feat: Make x axis to start from 0

* fix: Put csv files to right directory

* fix: No conditions now in Applications and Components: #1207

* style: If function do not have to be public, make it private

* fix: Fix function names, make output nicer

* fix: Collect right stuff in Stage job

* fix: Correctly detect when PipelineRun fails

* fix: Make date parsing to work on older Python, add two more places where we call `fromisoformat()` directly

* fix: We need more Python modules for utils

* refactor: Rewrite run.sh for new code

* feat: These defaults should work for CI job

* fix: Use default repo that actually have Dockerfile

* style: Fix issues raised by linters

* style: Fix issues reported by gosec

* fix: Make collect script on CI work

* fix: Make max-concurrency work

* feat: More log messages, we already collect monitoring for whole run

* fix: Test run did not ended when expected, also stop on big error rate

* fix: Also make branch name available when templating

* fix: Rename forgotten occurance, thank you Pavel

* fix: Not needed, thank you Pavel

* fix: Fail if test iteration results processing fails

* fix: Collect all iterations

* debug: Just to see what is where

* fix: Another round of fixes to get collection script in max-concurrency job to work

* fix: Rename verbose log level option to info to match what it does

* feat: Change default lvl as discussd with Pavel

.github/workflows/loadtest.yaml

@@ -3,12 +3,12 @@ name: CI Stage Load Test
 on:
   workflow_dispatch:
     inputs:
-      users:
+      concurrency:
         type: string
-        description: Number of Users
-      threads:
+        description: Number of concurrent user journey threads
+      journey_repeats:
         type: string
-        description: Number of threads
+        description: Number of sequential user journey repetitions
 
 jobs:
   load_test:
@@ -37,19 +37,19 @@ jobs:
     - name: Run Load Test
       working-directory: ./tests/load-tests
       env:
-        USERS_PER_THREAD: ${{ github.event.inputs.users }}
-        THREADS: ${{ github.event.inputs.threads }}
-        PUSHGATEWAY_URL: ${{ secrets.PUSHGATEWAY_URL }}
-        DISABLE_METRICS: ${{ secrets.DISABLE_METRICS }}
-      run: ./run-stage.sh
+        CONCURRENCY: ${{ github.event.inputs.concurrency }}
+        JOURNEY_REPEATS: ${{ github.event.inputs.journey_repeats }}
+      run: |
+        ./run-stage.sh
 
     - name: Collect results
       working-directory: ./tests/load-tests
       env:
+        CONCURRENCY: ${{ github.event.inputs.concurrency }}
         STAGE_MEMBER_CLUSTER: ${{ secrets.STAGE_MEMBER_CLUSTER }}
         OCP_PROMETHEUS_TOKEN: ${{ secrets.OCP_PROMETHEUS_TOKEN }}
       run: |
-        ./ci-scripts/stage/collect-results.sh .
+        ./ci-scripts/stage/collect-results.sh $CONCURRENCY .
 
     - name: Archive artifacts
       uses: actions/upload-artifact@v3