Add support for xUnit.net v3 #2718

0xced · 2024-08-06T14:30:22Z

Fixes #2709

Notes

The XUnit3Core.Specs project ensures that the catch (FileNotFoundException) branch is covered in the FindExceptionAssembly method. XUnit3Core.Specs takes a dependency on xunit.v3.core instead of xunit.v3. And the difference between xunit.v3 and xunit.v3.core is that the former has a dependency on xunit.v3.assert (the exception assembly) and the later does not.
The new LateBoundTestFramework covers all case without duplicating code.
- Try first if the exception assembly is already loaded into the AppDomain (all test frameworks) ⇒ available
- The exception assembly is not loaded and we don't try to load it (mspec, nunit and mstestv2) ⇒ not available
- The exception assembly is not loaded and we try to load it (tunit, xunit2 and xunit3) ⇒ available if it can be loaded
Since FluentAssertions don't take hard dependencies on test frameworks it can't know if an assembly is already loaded. So verifying first if the assembly is already loaded avoids going through the try/catch approach (which was done in the LoadableTestFramework implementation) in case the assembly is already loaded.

IMPORTANT

If the PR touches the public API, the changes have been approved in a separate issue with the "api-approved" label.
The code complies with the Coding Guidelines for C#.
The changes are covered by unit tests which follow the Arrange-Act-Assert syntax and the naming conventions such as is used in these tests.
If the PR adds a feature or fixes a bug, please update the release notes with a functional description that explains what the change means to consumers of this library, which are published on the website.
If the PR changes the public API the changes needs to be included by running AcceptApiChanges.ps1 or AcceptApiChanges.sh.
If the PR affects the documentation, please include your changes in this pull request so the documentation will appear on the website.
- Please also run ./build.sh --target spellcheck or .\build.ps1 --target spellcheck before pushing and check the good outcome

github-actions · 2024-08-06T14:33:39Z

Qodana for .NET

It seems all right 👌

No new problems were found according to the checks applied

💡 Qodana analysis was run in the pull request mode: only the changed files were checked
☁️ View the detailed Qodana report

Contact Qodana team

Contact us at qodana-support@jetbrains.com

Or via our issue tracker: https://jb.gg/qodana-issue
Or share your feedback: https://jb.gg/qodana-discussions

coveralls · 2024-08-07T08:03:11Z

Pull Request Test Coverage Report for Build 11978451397

Details

26 of 26 (100.0%) changed or added relevant lines in 5 files are covered.
24 unchanged lines in 6 files lost coverage.
Overall coverage decreased (-0.4%) to 97.409%

Files with Coverage Reduction	New Missed Lines	%
Src/FluentAssertions/Common/MemberPathSegmentEqualityComparer.cs	1	86.67%
Src/FluentAssertions/Types/AssemblyAssertions.cs	1	98.91%
Src/FluentAssertions/CallerIdentifier.cs	1	91.19%
Src/FluentAssertions/Common/ExpressionExtensions.cs	2	98.72%
Src/FluentAssertions/Streams/BufferedStreamAssertions.cs	2	91.67%
Src/FluentAssertions/Common/TypeExtensions.cs	17	80.21%

Totals
Change from base Build 11909568876:	-0.4%
Covered Lines:	12370
Relevant Lines:	12557

💛 - Coveralls

0xced · 2024-08-09T08:06:56Z

Oops, Qodana Scan is failing because of a license error. 🙁

License verification failed. Please ensure that the token provided through the QODANA_TOKEN environment variable is correct and that you have a valid license.

dennisdoomen · 2024-08-09T11:12:38Z

Oops, Qodana Scan is failing because of a license error

Yes, and I already have a nice license, but somehow it's not working yet. Working with JetBrains to resolve this ASAP.

jnyrup · 2024-08-18T09:33:48Z

Src/FluentAssertions/Execution/XUnit2TestFramework.cs

-                // For netfx the assembly is not in AppDomain by default, so we can't just scan AppDomain.CurrentDomain
-                assembly = Assembly.Load(new AssemblyName("xunit.assert"));


We added this in #384 to correctly throw XUnitException despite xunit.assert.dll not yet being loaded into the App Domain for netfx.

The reason this PR works is due to how https://github.com/fluentassertions/fluentassertions/pull/2712/files#diff-69d48f2874ce90a847fb038d9eb42b104dba380c129f7babfde78e9a3d73e65aL16-L17 changed the test.
By explicitly referencing XunitException, we now trigger loading the xunit.assert dll.

We should revert those simplifications.
At least for xunit as that's the only framework that is spread across multiple dll's, but I'd say we do it for all test frameworks.

We should revert those simplifications.

Indeed, I figured out why XunitException was not referenced while working on this pull request. I'll revert #2712 as part of this pull request and add a comment why act.Should().Throw<XunitException>(); should not be used.

ITaluone · 2024-08-22T05:30:17Z

Src/FluentAssertions/Execution/LateBoundTestFramework.cs

+            catch (FileNotFoundException)
+            {
+                return null;
+            }


Should we add explicit tests that challenge these two catches? This one and the catch below...

When_xunit3_is_used_it_should_throw_xunit_exceptions_for_assertion_failures covers this catch (FileNotFoundException) since it tries to load xunit.assert first and fails. But adding a None.Specs that references none of the supported testing frameworks and asserting that the fallback AssertionFailedException is thrown is probably a good idea anyway.

For the second catch (around expression.Compile()) it's going to be very hard to test. I added it as a safety thinking that maybe Compile() might fail on some platforms and falling back on Activator.CreateInstance which was the original implementation. But maybe this whole expression thing is over-engineering since we're throwing an exception anyway.

adding a None.Specs that references none of the supported testing frameworks

That was a stupid idea! How are the tests going to be run without any testing framework? 😂

When_xunit3_is_used_it_should_throw_xunit_exceptions_for_assertion_failures covers this catch (FileNotFoundException) since it tries to load xunit.assert first and fails.

Coveralls mentioned this as "no coverage": see https://coveralls.io/builds/69359638/source?filename=Src%2FFluentAssertions%2FExecution%2FLateBoundTestFramework.cs#L45-47

That was a stupid idea! How are the tests going to be run without any testing framework?

Using an unsupported test framework like NSpec3 🤔

Coveralls mentioned this as "no coverage"

That's weird, I ran coverage in Rider and it was displayed as covered. Does Coveralls merge all different test runs? Also, I can't see anything in the link you posted, it says Source Not Available and doesn't display anything beside the file path: /Src/FluentAssertions/Execution/LateBoundTestFramework.cs.

Using an unsupported test framework like NSpec3 🤔

I just tried with Fuchu but it did not end well. 🙁

Test run for ~/fluentassertions/Tests/TestFrameworks/Fuchu.Specs/bin/Debug/net472/Fuchu.Specs.exe (.NETFramework,Version=v4.7.2) Microsoft (R) Test Execution Command Line Tool Version 17.10.0 (x64) Copyright (c) Microsoft Corporation. All rights reserved. Starting test execution, please wait... A total of 1 test files matched the specified pattern. The active test run was aborted. Reason: Test host process crashed : ================================================================= External Debugger Dump: ================================================================= Test Run Aborted. Test run for ~/fluentassertions/Tests/TestFrameworks/Fuchu.Specs/bin/Debug/net6.0/Fuchu.Specs.dll (.NETCoreApp,Version=v6.0) Microsoft (R) Test Execution Command Line Tool Version 17.10.0 (x64) Copyright (c) Microsoft Corporation. All rights reserved. Starting test execution, please wait... A total of 1 test files matched the specified pattern. No test is available in ~/fluentassertions/Tests/TestFrameworks/Fuchu.Specs/bin/Debug/net6.0/Fuchu.Specs.dll. Make sure that test discoverer & executors are registered and platform & framework version settings are appropriate and try again. Additionally, path to test adapters can be specified using /TestAdapterPath command. Example /TestAdapterPath:<pathToCustomAdapters>.

it says Source Not Available

You have to sign-in first (e.g. using Github OAuth)

Does Coveralls merge all different test runs?

I think so. If you run build.(sh|ps1) you run this like in the pipeline. When finished, you get a reports directory beneath TestResults. And there you can run the index.html and browse the coverage files

ITaluone · 2024-08-22T12:45:51Z

Something went wrong with the coverage, since a lot of unrelated files are mentioned...

0xced · 2024-08-22T20:31:48Z

We're out of luck today!

🚀 Posting coverage data to https://coveralls.io/api/v1/jobs
62
⚠️ Internal server error. Please contact Coveralls team.

And indeed, the Coveralls status page acknowledges 500 errors. 🙁

Investigating - We have received reports of 500 errors received as responses from the Coveralls API upon coverage report uploads. We are investigating.

But I found a solution to make 100% sure that the catch (FileNotFoundException) part is covered. I added a new project (XUnit3Core.Specs) which uses xunit.v3.core instead of xunit.v3 so that xunit.v3.assert is not referenced. This will eventually throw the fallback AssertionFailedException since none of the supported testing frameworks will report being available.

I'll force-push once the Coveralls outage is over to trigger a new build and see what's going on with the coverage.

0xced · 2024-09-04T12:32:53Z

Not sure what's happening with the coverage. Build 10700844516 says that AssemblyAssertions line 216 was uncovered. But looking at the base build, AssemblyAssertions line 216 is NOT covered!

dennisdoomen · 2024-10-23T17:52:30Z

Time to rebase on develop

0xced · 2024-10-27T08:48:50Z

I juste rebased on develop. Qodana says Not a valid commit name 659dc90, I don't know why!

I also changed the status of this pull request from draft to ready for review.

dennisdoomen · 2024-10-27T19:20:03Z

We know about the Qodana issue and are working with JetBrains to fix it.

This reverts commit feb3147 and add a comment explaining why the exception types must not be referenced explicitly.

Fixes fluentassertions#2709

It's not relevant actually

github-actions · 2024-10-28T21:28:30Z

Qodana for .NET

It seems all right 👌

No new problems were found according to the checks applied

💡 Qodana analysis was run in the pull request mode: only the changed files were checked
☁️ View the detailed Qodana report

Contact Qodana team

Contact us at qodana-support@jetbrains.com

Or via our issue tracker: https://jb.gg/qodana-issue
Or share your feedback: https://jb.gg/qodana-discussions

dennisdoomen · 2024-10-29T07:45:32Z

Src/FluentAssertions/Execution/LateBoundTestFramework.cs

❓ Why did this class have to change?

This new implementations covers all case without duplicating code.

Try first if the exception assembly is already loaded into the AppDomain (all test frameworks) ⇒ available

The exception assembly is not loaded and we don't try to load it (mspec, nunit and mstestv2) ⇒ not available

The exception assembly is not loaded and we try to load it (tunit, xunit2 and xunit3) ⇒ available if it can be loaded

Since FluentAssertions don't take hard dependencies on test frameworks it can't know if an assembly is already loaded. So verifying first if the assembly is already loaded avoids going through the try/catch approach (which was done in the LoadableTestFramework implementation) in case the assembly is already loaded.

Src/FluentAssertions/Execution/LateBoundTestFramework.cs

dennisdoomen · 2024-10-29T08:32:51Z

Tests/TestFrameworks/XUnit3Core.Specs/XUnit3Core.Specs.csproj

🤔Why do we need two new projects?

The XUnit3Core.Specs project ensures that the catch (FileNotFoundException) branch is covered in the FindExceptionAssembly method. XUnit3Core.Specs takes a dependency on xunit.v3.core instead of xunit.v3. And the difference between xunit.v3 and xunit.v3.core is that the former has a dependency on xunit.v3.assert (the exception assembly) and the later does not.

That's why I tried to explain in #2718 (comment)

dennisdoomen · 2024-11-22T17:01:30Z

@0xced the ball is yours for a while now ;-)

0xced · 2024-11-22T18:44:00Z

I tried to address everything, let me know if something needs more clarification.

dennisdoomen · 2024-11-23T08:22:23Z

@jnyrup do you feel the need to do an explicit review?

jnyrup

Haven't looked deeply into LateBoundTestFramework (on mobile right now) but everything looks good.
If there is something it can wait...

dennisdoomen · 2024-11-23T13:50:59Z

Awesome work @0xced

(cherry picked from commit 3e00296)

0xced force-pushed the xunit-v3 branch from f751aa5 to c6b825b Compare August 7, 2024 07:50

dennisdoomen self-requested a review August 10, 2024 09:26

jnyrup reviewed Aug 18, 2024

View reviewed changes

0xced force-pushed the xunit-v3 branch 4 times, most recently from 1aedaa7 to 62e78f3 Compare August 21, 2024 21:45

ITaluone reviewed Aug 22, 2024

View reviewed changes

0xced force-pushed the xunit-v3 branch from 9c092c7 to 127073e Compare August 22, 2024 20:02

0xced force-pushed the xunit-v3 branch 2 times, most recently from 074483e to dbc451f Compare August 24, 2024 12:13

0xced force-pushed the xunit-v3 branch 2 times, most recently from afe1754 to fc2d540 Compare September 4, 2024 11:35

0xced force-pushed the xunit-v3 branch 2 times, most recently from 565ec55 to bf96c6e Compare September 13, 2024 09:50

jnyrup mentioned this pull request Sep 24, 2024

TUnit Framework Support #2758

Merged

0xced force-pushed the xunit-v3 branch from bf96c6e to 659dc90 Compare October 27, 2024 08:17

0xced marked this pull request as ready for review October 27, 2024 08:48

0xced added 2 commits October 28, 2024 22:10

Revert "Simplify FrameworkSpecs (fluentassertions#2712)"

c643bed

This reverts commit feb3147 and add a comment explaining why the exception types must not be referenced explicitly.

Add support for xUnit.net v3

c1cdd10

Fixes fluentassertions#2709

Remove tests for xUnit v3 with Microsoft.Testing.Platform

114fd41

It's not relevant actually

0xced force-pushed the xunit-v3 branch from 659dc90 to 114fd41 Compare October 28, 2024 21:16

dennisdoomen reviewed Oct 29, 2024

View reviewed changes

0xced added 2 commits November 22, 2024 19:15

Rename GetAssembly into FindExceptionAssembly

615ef6a

Nitpick: perform "Safe Cleanup" with Rider

a68da50

Merge branch 'develop' into xunit-v3

ca934de

dennisdoomen approved these changes Nov 23, 2024

View reviewed changes

jnyrup approved these changes Nov 23, 2024

View reviewed changes

dennisdoomen merged commit 3e00296 into fluentassertions:develop Nov 23, 2024
8 checks passed

0xced deleted the xunit-v3 branch November 23, 2024 14:19

jon-armen mentioned this pull request Jan 7, 2025

Fluent Assertions (v7) & xunit.v3 framework detection #2935

Closed

0xced mentioned this pull request Jan 16, 2025

Update ownership and license #2943

Merged

dennisdoomen mentioned this pull request Jan 16, 2025

Backport xUnit 3 support #2970

Merged

dennisdoomen pushed a commit to dennisdoomen/fluentassertions that referenced this pull request Jan 16, 2025

Add support for xUnit.net v3 (fluentassertions#2718)

ff4fb39

(cherry picked from commit 3e00296)

dennisdoomen pushed a commit to dennisdoomen/fluentassertions that referenced this pull request Jan 16, 2025

Add support for xUnit.net v3 (fluentassertions#2718)

7b094d2

(cherry picked from commit 3e00296)

dennisdoomen pushed a commit that referenced this pull request Jan 16, 2025

Add support for xUnit.net v3 (#2718)

0273ed7

(cherry picked from commit 3e00296)

ScarletKuro referenced this pull request in AwesomeAssertions/AwesomeAssertions Jan 16, 2025

Add support for xUnit.net v3 (#2718)

d07f8d1

(cherry picked from commit 3e00296)

		// For netfx the assembly is not in AppDomain by default, so we can't just scan AppDomain.CurrentDomain
		assembly = Assembly.Load(new AssemblyName("xunit.assert"));

Add support for xUnit.net v3 #2718

Add support for xUnit.net v3 #2718

Uh oh!

Conversation

0xced commented Aug 6, 2024 • edited by dennisdoomen Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

IMPORTANT

Uh oh!

github-actions bot commented Aug 6, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Qodana for .NET

Uh oh!

coveralls commented Aug 7, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Test Coverage Report for Build 11978451397

Details

💛 - Coveralls

Uh oh!

0xced commented Aug 9, 2024

Uh oh!

dennisdoomen commented Aug 9, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ITaluone Aug 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ITaluone commented Aug 22, 2024

Uh oh!

0xced commented Aug 22, 2024

Uh oh!

0xced commented Sep 4, 2024

Uh oh!

dennisdoomen commented Oct 23, 2024

Uh oh!

0xced commented Oct 27, 2024

Uh oh!

dennisdoomen commented Oct 27, 2024

Uh oh!

github-actions bot commented Oct 28, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Qodana for .NET

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dennisdoomen commented Nov 22, 2024

Uh oh!

0xced commented Nov 22, 2024

Uh oh!

dennisdoomen commented Nov 23, 2024

Uh oh!

jnyrup left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

dennisdoomen commented Nov 23, 2024

Uh oh!

0xced commented Aug 6, 2024 •

edited by dennisdoomen

Loading

github-actions bot commented Aug 6, 2024 •

edited

Loading

coveralls commented Aug 7, 2024 •

edited

Loading

ITaluone Aug 22, 2024 •

edited

Loading

github-actions bot commented Oct 28, 2024 •

edited

Loading