[tests-only] Lint expected-failures files

[phil-davis]

Description

It is easy to get errors in an expected-failures file, specially with the link to the line number of the file in GitHub:

-   [apiWebdavMove1/moveFolder.feature:27](https://github.com/owncloud/core/blob/master/tests/acceptance/features/apiWebdavMove1/moveFolder.feature#L27)

If there is any typo in the link, that is the (https://github.com/...) string then it does not get really noticed, but ends up being an invalid link and is annoying for humans who click on it, or humans who wonder why the link points to somewhere different.

The common errors are that the suite and/or feature file name is wrong (a cut-paste error from another entry...), or that the line number is wrong (the :27 got updated but the old #L26 gets forgotten when line numbers change).

This PR adds the script lint-expected-failures.sh. The existing checks of the expected-failures file have been moved there, and additional checks added to make sure that the link to GitHub is the correct link that matches the suite/feature/line number in [apiWebdavMove1/moveFolder.feature:27]

The referenced repo where the feature files live is usually owncloud/core so that is the default for the check. But sometimes the feature files are in somewhere like owncloud/ocis and the core run.sh script and the core test code is being run but to execute "local" feature files in oCIS (or maybe even reva, or whatever. In the general case, the core run.sh and lint-expected-failures.sh does not know where the feature files are that are about to be executed. (Maybe the script could somehow try really hard to deduce it, but it is not simple) Sometimes tests are executed from release QA tarballs and so the test scripts and test feature files are not in a local clone of the various git repos involved, so we can't rely on grabbing a git "repo slug" to help us guess.

If the linter sees a sentence like:
The expected failures in this file are from features in the owncloud/ocis repo.
then it learns that the expected-failures file is supposed to be for feature files that are stored in owncloud/ocis and checks that the links do go to that GitHub repo.

How Has This Been Tested?

CI - see owncloud/ocis#3147

I purposely put errors in an expected-failures file and the linter reported them and failed the pipelines where they were used.

I added the line The expected failures in this file are from features in the owncloud/ocis repo. to the expected-failures file for localAPI tests, and they get linted correctly.

Having this linter code in a separate bash script means we can call it locally/manually to check an expected-failures fail if we want. In future we could add a Makefile target to repos that would help people lint-check expected-failures files before pushing to GitHub.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Database schema changes (next release will require increase of minor version instead of patch)
• Breaking change (fix or feature that would cause existing functionality to change)
• Technical debt
• Tests only (no source changes)

Checklist:

• Code changes
• Unit tests added
• Acceptance tests added
• Documentation ticket raised:
• Changelog item, see TEMPLATE

[sonarqubecloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
No Duplication information

[kiranparajuli589]

lgtm 👑

[kiranparajuli589]

@phil-davis should we document this new feature? Information like how to check the updated failures locally will be helpful. Also, the docs will show us the general format of how the expected failures are needed to be added on different repos.

[phil-davis]

@phil-davis should we document this new feature? Information like how to check the updated failures locally will be helpful. Also, the docs will show us the general format of how the expected failures are needed to be added on different repos.

Thanks for the reminder. I had already opened a general docs issue to document expected-failures. I have transferred it to owncloud/docs-server#57 and will write a comment about the lint checks in the issue. Hopefully I do the documentation "soon".