Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add tests for analysis using the Clang and GCC analyzers #95

Merged
merged 3 commits into from
Oct 4, 2023
Merged

Add tests for analysis using the Clang and GCC analyzers #95

merged 3 commits into from
Oct 4, 2023

Conversation

lzaoral
Copy link
Member

@lzaoral lzaoral commented Feb 2, 2023

No description provided.

@lzaoral lzaoral added the ci CI related changes label Feb 2, 2023
@lzaoral lzaoral self-assigned this Feb 2, 2023
github-advanced-security[bot]
github-advanced-security bot found potential problems Feb 2, 2023
View reviewed changes
@lzaoral
Copy link
Member Author

lzaoral commented Feb 6, 2023 •
edited
Loading

This PR is blocked since it requires a newer release of csdiff and Testing Farm tries to install all packages from the given COPR build:

Error: 
 Problem: conflicting requests
  - nothing provides csdiff > 2.8.0 needed by csmock-plugin-snyk-3.3.5.20230206.145923.g5d0b606.pr_95-1.fc38.noarch

@kdudka
Copy link
Member

kdudka commented Feb 6, 2023

@lzaoral Could we please configure csmock CI to use csdiff from the Packit-managed COPR?

1 similar comment
@kdudka
Copy link
Member

kdudka commented Feb 6, 2023

@lzaoral Could we please configure csmock CI to use csdiff from the Packit-managed COPR?

@kdudka
Copy link
Member

kdudka commented Feb 6, 2023

Another option would be not to install csmock-plugin-snyk when we only need csmock-plugin-clang.

csmock-plugin-clang depends on csmock-common, which installs everything what is needed. There is no need to explicitly install the csmock metapackage, which pulls in all the plug-ins.

@lzaoral
Copy link
Member Author

lzaoral commented Feb 6, 2023

Hmm, it looks like Testing Farm does that regardless of the contents in the given fmf files.

@lzaoral
Copy link
Member Author

lzaoral commented Feb 22, 2023

Looks like this is not possible at the moment: https://packit.dev/docs/testing-farm/#running-tests-with-builds-from-another-pull-request

@kdudka
Copy link
Member

kdudka commented Feb 22, 2023

@lzaoral We do not need to use builds from another pull request. I meant to use csdiff from https://copr.fedorainfracloud.org/coprs/g/codescan/csutils/ which corresponds to the main branch of csdiff upstream. Anyway, in case it helped, I can roll out a new release of csdiff today.

@lzaoral
Copy link
Member Author

lzaoral commented Feb 22, 2023

Yes, I know. But this seems to be the only way to install the contents of the csdiff and csmock COPR repositories at the same time. Other methods like 3330b2a install them too late...

@kdudka
Copy link
Member

kdudka commented Feb 22, 2023

I see. csdiff-2.9.0 is being built...

@lzaoral
Copy link
Member Author

lzaoral commented Feb 23, 2023

/packit retest-failed

@thrix
Copy link

thrix commented Feb 23, 2023

hmm, we should somehow disable those unsupported combos on Testing Farm ... it is hitting our error rate :)

@TomasTomecek
Copy link
Contributor

Folks, could you please drop the testing targets for architectures that are not available in the Testing Farm? https://docs.testing-farm.io/general/0.1/test-environment.html#_supported_architectures

So for tests, please keep only x86_64 and aarch64

We are working in Packit and Testing Farm to account for this, but right now all requests are sent to TF as Miro points out.

github-advanced-security[bot]
github-advanced-security bot found potential problems Mar 17, 2023
View reviewed changes
tests/simple_build/test.sh
@@ -0,0 +1,38 @@
#!/bin/bash
# vim: dict+=/usr/share/beakerlib/dictionary.vim cpt=.,w,b,u,t,i,k
. /usr/share/beakerlib/beakerlib.sh || exit 1

Check notice

Code scanning / shellcheck

Not following: /usr/share/beakerlib/beakerlib.sh: openBinaryFile: does not exist (No such file or directory)

Not following: /usr/share/beakerlib/beakerlib.sh: openBinaryFile: does not exist (No such file or directory)
@lzaoral
Copy link
Member Author

lzaoral commented Mar 17, 2023

Sorry @thrix and @TomasTomecek! I've deleted everything except x86_64 as that's what we are mainly interested in anyway.

@lzaoral
Copy link
Member Author

lzaoral commented Jun 13, 2023

The latest release of Testing Farm should support fix the unjustifiable dependence on a newer version of csdiff.

@lzaoral
Copy link
Member Author

lzaoral commented Sep 13, 2023

I've dropped the integration with TF completely for now. It can be added in the future and the tests themselves can be merged even in the current state.

@lzaoral lzaoral requested a review from kdudka September 13, 2023 15:27
@lzaoral lzaoral marked this pull request as ready for review September 13, 2023 15:27
@kdudka
Copy link
Member

kdudka commented Sep 13, 2023

So the tests we are about to merge will not run automatically at all?

Then we should at least mention in README how to run the tests manually. If nobody knows about the tests, they will break sooner or later.

@lzaoral
Copy link
Member Author

lzaoral commented Sep 14, 2023

Hmm, everything except CS8 passed but the table below still shows all Fedora tests as running. 🤔

@lzaoral lzaoral force-pushed the clang-test branch 2 times, most recently from a8a846d to a140582 Compare September 14, 2023 09:43
@lzaoral lzaoral force-pushed the clang-test branch 4 times, most recently from d94cddf to fd3c086 Compare September 25, 2023 14:27
@lzaoral lzaoral marked this pull request as draft September 25, 2023 16:07
@lzaoral
Copy link
Member Author

lzaoral commented Sep 25, 2023

Current state: The @csutils copr repository is properly installed, however, the packages are not properly updated on stable Fedora releases for an unknown reason.

@lzaoral lzaoral changed the title Add a test for analysis using the Clang analyzer Add tests for analysis using the Clang and Cppcheck analyzers Sep 26, 2023
@lzaoral lzaoral changed the title Add tests for analysis using the Clang and Cppcheck analyzers Add tests for analysis using the Clang and GCC analyzers Sep 26, 2023
@lzaoral lzaoral force-pushed the clang-test branch 2 times, most recently from a5d0f95 to 9209ee8 Compare October 2, 2023 08:52
@lzaoral lzaoral marked this pull request as ready for review October 2, 2023 08:55
@lzaoral
Copy link
Member Author

lzaoral commented Oct 2, 2023 •
edited
Loading

@kdudka, the PR should be ready to be reviewed. The test is modular, therefore, if you want to test a new tool, you only need to create corresponding .fmf file and set proper environment variables. Also, the tar.xz archive with results should be provided by the Testing Farm for each test (open the log dir link for given test and them select the data directory).

edit: add instructions to get the logs

kdudka
kdudka approved these changes Oct 3, 2023
View reviewed changes
Copy link
Member

@kdudka kdudka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work. Thank you!

@lzaoral lzaoral merged commit b6d8562 into csutils:main Oct 4, 2023
@lzaoral lzaoral deleted the clang-test branch October 4, 2023 07:44
@lzaoral lzaoral mentioned this pull request Oct 19, 2023
Merged
@jamacku jamacku mentioned this pull request Oct 23, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kdudka kdudka kdudka approved these changes

Assignees

@lzaoral lzaoral

Labels
ci CI related changes
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@lzaoral @kdudka @thrix @TomasTomecek