Add cert-blocker status based on coverage guide for IoT test plans (BugFix) #1206
Conversation
pieqq
changed the title
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #1206 +/- ##
=======================================
Coverage 45.72% 45.72%
=======================================
Files 367 367
Lines 39134 39134
Branches 6618 6618
=======================================
Hits 17895 17895
Misses 20565 20565
Partials 674 674
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
|
I've checked the cert-blocker status on the Desktop 22.04 test plan and it looks appropriate, so no changes are required. This PR is ready for review. |
There was a problem hiding this comment.
Hi @pieqq,
A few years earlier, there was a discussion between Cert/CE/OEMQA and the decision made in that meeting was[1]
Any job included in the project provider is a blocker
And the "CE IoT QA - Enablement Cycle QA Process" document has been updated according to that conclusion[2]
If we're now only adding blocker certification status according to the jammy test coverage guide then the certification scope will be changed for commercial enablement projects and ODM program projects as well.
Therefore I'd like to suggest applying blocker certification status to all the test jobs included in the client-cert-iot and client-cert-odm test plans.
[1] https://docs.google.com/document/d/1j0sOvJSODp_qlsTvjQcJA6umHSv-B18h8e506fz1sSE/edit#heading=h.6llmkwta1g0g
[2] https://docs.google.com/document/d/1QIaNbzu0bQXLMNTKPHJ21Tvb0upgHKb3UVvn0KQV6h8/edit#heading=h.pvqm21uto89l
fernando79513
added
the
waiting-for-changes
The "Internet of Things Certified Hardware Coverage for Ubuntu Core 22 / Ubuntu 22.04" document specifies what jobs should be considered as blockers. For IoT devices, the after-suspend jobs are not considered blockers, so they have been left out. Fix CER-2593
The Should I only add the certification status overrides to the |
Use the certification_status_overrides section of test plan units to force every jobs to have the `certification-status` key set to `blocker`.⚠️ Warning⚠️ Currently, Checkbox will apply certification-status=blocker to every job of every test plan that use any of these as a nested part. This is a bug that has to be addressed eventually, but it was deemed OK for the time being.
pieqq
force-pushed
the
CER-2593-cert-status-consistency
branch
from
dad92ea to
3291867
Compare
|
I'd say we should add the certification status to the following test plans: As for the certification criteria of ODM program, I agree with you this should go to someone more familiar with certifying ODM program devices. |
pieqq
dismissed
zongminl’s stale review
Vic's comments have been taked into account in later commits
Description
The "Internet of Things Certified Hardware Coverage for Ubuntu Core 22 / Ubuntu 22.04" document specifies what jobs should be considered as blockers.
For IoT devices, the after-suspend jobs are not considered blockers, so they have been left out.
Considerations
The document I used as a "source of truth" specifies a few things that look suspicious. Feedback from other teams would be appreciated:
Resolved issues
CER-2593
Documentation
Tests
Running a
list-bootstrappedof theclient-cert-iot-ubuntucore-22test plan on my laptop yields the following output:Notes:
non-blockerlist-bootstrapped, that's why they don't appear belowWARNING: This modifies com.canonical.certification::sru-server