chore(ci): bump riot version for 3.7 compatibility issues [backport 2…

….15] (#11088)

Backport a201fc0 from #11085 to 2.15.

This PR bumps the riot version in dd-trace-py from 0.19.1 to 0.20.0 to
fix virtualenv versioning incompatibilities with python3.7.

With the new release of `virtualenv==20.27.0`, we were seeing new
failures with `build_base_venv[3.7]` (GitLab) and `build_base_venv`
(CircleCI) jobs when running:
```
riot -P -v generate --python=$PYTHON_VERSION
...
  File "/go/src/github.com/DataDog/apm-reliability/dd-trace-py/.riot/venv_py3717/lib/python3.7/site-packages/pip/_vendor/typing_extensions.py", line 1039
    def TypedDict(typename, fields=_marker, /, *, total=True, closed=False, **kwargs):
                                            ^
SyntaxError: invalid syntax
ERROR:riot.riot:Dev install failed, aborting!
```

### What changed in `riot`

We needed to pin to `virtualenv==20.26.6` to continue supporting
python3.7

The `upload-artifact` and `download-artifact` bumps were required to
unblock the riot CI. It had been a while since the last update to riot,
and since then, v2 has been deprecated and v3 will be deprecated next
month (Nov 2024), so we are bumping to v4 via tag.

The above changes were implemented in riot
(DataDog/riot#232).

## Checklist
- [x] PR author has checked that all the criteria below are met
- The PR description includes an overview of the change
- The PR description articulates the motivation for the change
- The change includes tests OR the PR description describes a testing
strategy
- The PR description notes risks associated with the change, if any
- Newly-added code is easy to change
- The change follows the [library release note
guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html)
- The change includes or references documentation updates if necessary
- Backport labels are set (if
[applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting))

## Reviewer Checklist
- [x] Reviewer has checked that all the criteria below are met 
- Title is accurate
- All changes are related to the pull request's stated goal
- Avoids breaking
[API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces)
changes
- Testing strategy adequately addresses listed risks
- Newly-added code is easy to change
- Release note makes sense to a user of the library
- If necessary, author has acknowledged and discussed the performance
implications of this PR as reported in the benchmarks PR comment
- Backport labels are set in a manner that is consistent with the
[release branch maintenance
policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)

Co-authored-by: Taegyun Kim <taegyun.kim@datadoghq.com>

.circleci/config.templ.yml

@@ -80,7 +80,7 @@ commands:
     description: "Install riot"
     steps:
       # Make sure we install and run riot on Python 3
-      - run: pip3 install riot==0.19.1
+      - run: pip3 install riot==0.20.0
 
   setup_rust:
     description: "Install rust toolchain"

.circleci/config.yml

@@ -31,7 +31,7 @@ jobs:
           name: Generate config
           command: |
             export GIT_COMMIT_DESC=$(git log -n 1 $CIRCLE_SHA1)
-            pip3 install riot==0.19.1
+            pip3 install riot==0.20.0
             riot -P -v run --pass-env -s circleci-gen-config -- -v
       - continuation/continue:
           configuration_path: .circleci/config.gen.yml

.github/workflows/generate-package-versions.yml

@@ -13,6 +13,7 @@ jobs:
       actions: read
       contents: write
       pull-requests: write
+
     steps:
       - uses: actions/checkout@v4
 
@@ -45,7 +46,7 @@ jobs:
         uses: actions/setup-python@v5
         with:
           python-version: "3.12"
-      
+
       - name: Set up QEMU
         uses: docker/setup-qemu-action@v2
 
@@ -76,7 +77,7 @@ jobs:
           python -m pip install --upgrade pip
           pip install packaging
           pip install requests
-          pip install riot==0.19.1
+          pip install riot==0.20.0
 
       - name: Run regenerate-riot-latest
         run: scripts/regenerate-riot-latest.sh
@@ -98,4 +99,4 @@ jobs:
           base: main
           title: "chore: update ${{ env.VENV_NAME }} latest version to ${{ env.NEW_LATEST }}"
           labels: changelog/no-changelog
-          body-path: .github/PULL_REQUEST_TEMPLATE.md
+          body-path: .github/PULL_REQUEST_TEMPLATE.md

.github/workflows/requirements-locks.yml

@@ -25,7 +25,7 @@ jobs:
         run: pyenv global 3.10 3.7 3.8 3.9 3.11 3.12
 
       - name: Install Dependencies
-        run: pip install --upgrade pip && pip install riot
+        run: pip install --upgrade pip && pip install riot==0.20.0
 
       - name: Generate riot locks
         run: scripts/compile-and-prune-test-requirements

.gitlab/package.yml

@@ -8,7 +8,7 @@ build_base_venvs:
     CMAKE_BUILD_PARALLEL_LEVEL: 12
     PIP_VERBOSE: 1
   script:
-    - pip install riot~=0.19.1
+    - pip install riot==0.20.0
     - riot -P -v generate --python=$PYTHON_VERSION
   artifacts:
     name: venv_$PYTHON_VERSION

.gitlab/tests.yml

@@ -40,7 +40,7 @@ variables:
   services:
     - !reference [.services, ddagent]
   script:
-    - pip install riot~=0.19.1
+    - pip install riot==0.20.0
     - unset DD_SERVICE
     - unset DD_ENV
     - unset DD_TAGS

hatch.toml

@@ -17,7 +17,7 @@ dependencies = [
     "ddapm-test-agent>=1.2.0",
     "packaging==23.1",
     "pygments==2.16.1",
-    "riot==0.19.1",
+    "riot==0.20.0",
     "ruff==0.1.3",
     "clang-format==18.1.5",
     "cmake-format==0.6.13",

scripts/ddtest

@@ -19,8 +19,7 @@ fi
 for i in {1..3}; do $compose_cmd pull -q testrunner && break || sleep 3; done
 
 # TODO(DEV): Install riot in the docker image
-FULL_CMD="pip install -q --disable-pip-version-check riot==0.19.1 && $CMD"
-
+FULL_CMD="pip install -q --disable-pip-version-check riot==0.20.0 && $CMD"
 
 # install and upgrade riot in case testrunner image has not been updated
 # DEV: Use `--no-TTY` and `--quiet-pull` when running in CircleCI