Update the list of rejected jobs if match function is used (bugfix)

[pieqq]

Description

The jobs that are being pruned by the match function should be added to the list of rejected jobs, otherwise the metadata generated at the end of a test run is incorrect.

Because the list of rejected jobs is initially empty but can be altered in several places (run_alternate_selection() and finish_bootstrap()), it is updated in these functions instead of being replaced by another list.

The metadata that states if a test plan has been modified (custom_joblist) should also be set to True in these cases.

Resolved issues

Fix #1568

Documentation

Tests

Unit tests have been updated.

The following end to end tests have been run:

Normal call without use of match

1. Run checkbox-cli
2. Select camera-cert-automated test plan
3. In the Camera category, de-select everything except camera/detect and run the tests

The generated submission.json shows:

{
    "title": "session title",
    "testplan_id": "com.canonical.certification::camera-cert-automated",
    "custom_joblist": true,
    (...)
    "rejected-jobs": [
        {
            <the de-selected tests are here>
        }

Normal call with use of match

The following launcher is used:

[test plan]
unit = com.canonical.certification::camera-cert-automated
forced = yes

[test selection]
match = com.canonical.certification::camera/detect

1. Run checkbox-cli
2. Notice that in the test selection screen, in the Camera cateogory, only camera/detect is presented
3. De-select it, and run the tests

The submission.json generated looks like this:

{
    "title": "session title",
    "testplan_id": "com.canonical.certification::camera-cert-automated",
    "custom_joblist": true,
    "results": [
    ],
    (...)
    "rejected-jobs": [
        {
        <all the camera tests are listed here, including camera/detect>
        }

Automated run with the use of match

The following launcher is used:

[test plan]
unit = com.canonical.certification::camera-cert-automated
forced = yes

[test selection]
forced = yes
match = com.canonical.certification::camera/detect

The generated submission.json looks like this:

{
    "title": "session title",
    "testplan_id": "com.canonical.certification::camera-cert-automated",
    "custom_joblist": true,
    "results": [
        {
            "id": "camera/detect",
            "full_id": "com.canonical.certification::camera/detect",
            "name": "This Automated test attempts to detect a camera.",
            "certification_status": "blocker",
            "category": "Camera tests",
            "category_id": "com.canonical.plainbox::camera",
            "status": "pass",
            "outcome": "pass",
            "comments": null,
            "io_log": "/dev/video0: OK\n    name   : Integrated_Webcam_FHD: Integrat\n    driver : uvcvideo\n    version: 6.8.12\n    flags  : 0x84a00001 [ CAPTURE STREAMING ]\n    Format: MJPG (Motion-JPEG)\n    Resolutions: 1920x1080,1280x960,1280x720,640x480,640x360\n    Format: YUYV (YUYV 4:2:2)\n    Resolutions: 1920x1080,1280x720,640x480,640x360,320x240,320x180,160x120\n\n/dev/video1: OK\n    name   : Integrated_Webcam_FHD: Integrat\n    driver : uvcvideo\n    version: 6.8.12\n    flags  : 0x84a00001 [ CAPTURE STREAMING ]\n    Format: YUYV (YUYV)\n    Resolutions: 640x480\n\n/dev/video2: OK\n    name   : Integrated_Webcam_FHD: Integrat\n    driver : uvcvideo\n    version: 6.8.12\n    flags  : 0x84a00001 [ CAPTURE STREAMING ]\n    Format: GREY (8-bit Greyscale)\n    Resolutions: 640x360\n\n/dev/video3: OK\n    name   : Integrated_Webcam_FHD: Integrat\n    driver : uvcvideo\n    version: 6.8.12\n    flags  : 0x84a00001 [ CAPTURE STREAMING ]\n    Format: YUYV (YUYV)\n    Resolutions: 640x480\n\n",
            "type": "test",
            "project": "certification",
            "duration": 2.1785783767700195,
            "plugin": "shell",
            "template_id": null
        }
    ],
    (...)
    "rejected-jobs": [
        <every camera-related test is listed here, EXCEPT camera/detect>
        ]

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 48.00%. Comparing base (9c994f1) to head (c087555).
Report is 2 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #1575   +/-   ##
=======================================
  Coverage   48.00%   48.00%           
=======================================
  Files         371      371           
  Lines       39833    39837    +4     
  Branches     6730     6733    +3     
=======================================
+ Hits        19121    19125    +4     
  Misses      19994    19994           
  Partials      718      718           

Flag	Coverage Δ
checkbox-ng	68.60% <100.00%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.