Use attribute instead of base class for MSTest #1193

MattKotsenas · 2024-04-13T01:01:02Z

Overview

Introduce an attribute [UsesVerify] for MSTest so that tests can set up Verify without needing to use the VerifyBase class. Relaxing the base class requirement makes it easier to use Verify in more cases, where other test scenarios may also require using a base class.

The attribute triggers a source generator that adds the required TestContext property. The setter then stashes the context on an async local for use by the Verifier.

Highlights

Created a static Verifier class like the other test frameworks do and added it to the implict usings
Removed logic from VerifyBase and forward calls to Verifier
- This means we can only rely on the information in TestContext to get the Type and MethodInfo used for deriving path info
- For better performance the Verifier first tries reflection to get the assembly of the test class, and then falls back to assembly scanning
Created source generator that adds the TestContext property that sets the AsyncLocal on Verifier
- Unit tests intentionally do not use the Verify.SourceGenerators to avoid a circular dependency between repos
Moved all existing unit tests from VerifyBase to [UsesVerify]
- Added a simple set of tests to verify the base class doesn't regress
Added a dependency on ViHo.PackAsAnalyzer
- This is the same logic the runtime repo uses to package analyzers / source generators into the main package

Open Issues

This PR is big and (in my opinion) difficult to review; do you want me to split it up into preparatory refactoring changes?
- Calculate name and path info from TestContext and stop relying on VerifyBase inheritance
- Create the Verifier class and forward calls from VerifyBase
- Create the actual source generator & update tests
Agree on reflection / strategy for finding test Type and MethodInfo
Agree on if we want to allow hooking the TestContext setter
Agree on using ForAttributeWithMetadataName, which limits users to .NET 7 SDK+ (but can still use any supported TFM)
- Where should this be documented?
Are there other scenarios that should have VerifyBase tests?

src/Verify.MSTest.Tests/Tests.cs

src/Verify.MSTest/Verifier.cs

src/Verify.MSTest.Tests/Tests.cs

MattKotsenas · 2024-05-01T05:39:15Z

I haven't forgotten about this. I got pulled into since stuff at work and have to put this on hold for another week or two.

- Code generation attribute missing a quote - Fix reuse of generic parameter name in test - Fix incorrect nesting depth braces

…names

scenarios

This reverts commit 902a9c6.

src/Verify.MSTest.SourceGenerator/ClassToGenerate.cs

src/Verify.MSTest.SourceGenerator.Tests/Tests.HasAttributeInNamespace.verified.txt

src/Verify.MSTest.SourceGenerator.Tests/Tests.cs

src/Verify.MSTest.SourceGenerator/Parser.cs

src/Verify.MSTest.SourceGenerator/TrackingNames.cs

SimonCropp · 2024-05-16T11:28:14Z

src/Verify.MSTest/Verifier.cs

+    /// <param name="testContext">The <see cref="TestContext"/> of the current test.</param>
+    /// <param name="type">The <see cref="Type"/> of the currently running test.</param>
+    /// <returns><c>true</c> if the reflection succeeded; <c>false</c> otherwise.</returns>
+    static bool TryGetTypeFromTestContext(string typeName, TestContext testContext, [NotNullWhen(true)] out Type? type)


why so defensive in this method. IMO it should assume the correct shape and throw if that shape is not correct

I'm hesitant to assert on the specific shape here because I'm reaching into the internals of MSTest and not relying on any public API contracts.

More specifically, I'm concerned that this code would break and become an adoption blocker for early adopters of TestAnywhere (I'm not a user of it myself, but I did verify this code works with the current version).

In general I'm not super thrilled with the amount of reflection going on here, but I think reducing the amount would require digging more into Verify's internals, which feels to me like it should be a separate change.

Thus, I'd prefer leaving it as-is, but I'll do whatever you'd like. I'll volunteer to do follow up work in this space to reduce the amount of reflection overall, but I'd need some direction or a brain dump from you on how to proceed.

Let me know if this makes sense. Thanks!

An idea that just hit me would be to augment what we stash in the AsyncLocal.

Rather than try to get back to the test class after-the-fact, grab everything we need in the TestContext property setter (which means we're already in the test class). Then Verifier or VerifyBase doesn't need to do any work.

If possible I'd like to keep this PR source generator focused, so if this sounds good to you I can do that as a follow up change, or even as a standalone change and then merge that into this branch.

Thoughts?

An idea that just hit me would be to augment what we stash in the AsyncLocal.

sounds good. i will merge this now. can u have a go at an AsyncLocal PR next?

src/Verify.MSTest/Verifier.cs

MattKotsenas · 2024-05-16T17:14:53Z

src/Verify.MSTest.SourceGenerator/Emitter.cs

+    {
+        if (classToGenerate.Namespace is not null)
+        {
+            builder.Append("namespace ").AppendLine(classToGenerate.Namespace)


I did a partial revert of the transformation of

builder.Append("namespace ").AppendLine(classToGenerate.Namespace)

to

builder.AppendLine($"namespace {classToGenerate.Namespace}")

(and related changes) for performance.

Because source generators must target netstandard2.0, the underlying StringBuilder doesn't have any interpolated string handling. That means interpolated strings will result in an intermediate allocation.

We could write handling for interpolated strings ourselves, but the code to do so looks nontrivial: https://github.com/dotnet/runtime/pull/51653/files#diff-072e8df6b3d249d58da2b688cbd9369ae5d1e9fb4c1c1da77a19384f3f7711a0.

I also added a comment to the top of IndentedStringBuilder explaining the same decision.

Let me know if you strongly disagree and I can try to find another solution.

We can't overload on return type anyways, so no point in checking the type name of the property.

MattKotsenas · 2024-05-16T22:07:49Z

OK, all comments addressed. The two items to specifically take a look at are:

Take a look at the note around IndentedStringBuilder allocations
Let me know if you agree / disagree on the TestContext reflection choice

In all other cases I think I took your recommendations / suggestions. If you have any other feedback or thoughts, please let me know. Thanks!

MattKotsenas commented Apr 13, 2024

View reviewed changes

src/Verify.MSTest.Tests/Tests.cs Outdated Show resolved Hide resolved

src/Verify.MSTest/Verifier.cs Outdated Show resolved Hide resolved

src/Verify.MSTest.Tests/Tests.cs Outdated Show resolved Hide resolved

MattKotsenas and others added 28 commits May 14, 2024 16:17

Clone VerifyBase -> Verifier

3f952bf

Forward VerifyBase calls to Verifier

0b41bfb

Move to-be-generated code to region

0114f84

Clean up deprecation comments

5180715

Initial implementation of source generator

e9b3aab

Add basic tests -- namespaces incorrect

e7c6e89

Fix incorrect namespacing

833b7bc

Handle nested partial classes

3e0d747

Add autogenerted header to top of generated source file

86b0445

Refactor IndentedStringBuilder for chaining and reduced allocations

0649dc6

Fix invalid generated code

4126d4e

- Code generation attribute missing a quote - Fix reuse of generic parameter name in test - Fix incorrect nesting depth braces

Fix nullable namespace

f34fb5d

Partial classes don't need to specify type constraints

b122ef0

Simplify ParentClass to use a stack instead of a LinkedList

941331c

Write all generated classes to a single file to avoid ambiguous hint …

a275e12

…names

Extract parsing to its own class

7b5a9c3

Simplify how type parameters are handled

d64246b

Simplify StringBuilder usage

b981557

Simplify StringBuilder usage

fd8652e

Use EquatableArray

9e29380

Clean up namespaces and usings

c8cff10

Add source generator to main package

3cd57c8

Existing tests pass (still need to write / convert new ones)

79d5b15

Use reflection + lookup cache to speed up getting types

212b18e

Add comments and clean up exceptions

7878ef3

Fix Source Generator to not double-emit property in inheritence

5abb182

scenarios

Switch tests over to attribute

4527a66

Clean up tests

74eeef5

SimonCropp added 4 commits May 16, 2024 20:04

use file scoped namespace

902a9c6

Revert "use file scoped namespace"

dd8fc71

This reverts commit 902a9c6.

Merge branch 'main' into pr/1193

b15a2bd

fix merge

4665eda

SimonCropp reviewed May 16, 2024

View reviewed changes

src/Verify.MSTest.SourceGenerator/ClassToGenerate.cs Outdated Show resolved Hide resolved

SimonCropp added 4 commits May 16, 2024 21:11

redundant namespace

c07eaaf

redundant namespaces

be8864d

cleanup

c24e5b0

cleanup

64f72fc

SimonCropp reviewed May 16, 2024

View reviewed changes

src/Verify.MSTest.SourceGenerator.Tests/Tests.HasAttributeInNamespace.verified.txt Show resolved Hide resolved

SimonCropp reviewed May 16, 2024

View reviewed changes

src/Verify.MSTest.SourceGenerator.Tests/Tests.cs Show resolved Hide resolved

cleanup

115f0c9

SimonCropp reviewed May 16, 2024

View reviewed changes

src/Verify.MSTest.SourceGenerator/Parser.cs Outdated Show resolved Hide resolved

SimonCropp reviewed May 16, 2024

View reviewed changes

src/Verify.MSTest.SourceGenerator/TrackingNames.cs Outdated Show resolved Hide resolved

SimonCropp reviewed May 16, 2024

View reviewed changes

src/Verify.MSTest/Verifier.cs Outdated Show resolved Hide resolved

SimonCropp reviewed May 16, 2024

View reviewed changes

src/Verify.MSTest/Verifier.cs Outdated Show resolved Hide resolved

SimonCropp and others added 4 commits May 16, 2024 21:41

simplify string building

36684cf

simplify string building

ae27278

Replace TrackingNames reflection with simple array

e421f5f

Partially revert "simplify string building" to reduce allocations

bc274b1

MattKotsenas commented May 16, 2024

View reviewed changes

MattKotsenas added 3 commits May 16, 2024 10:38

Remove namespace check when serarching for TestContext property

155dc19

We can't overload on return type anyways, so no point in checking the type name of the property.

Move TestContext reflection to helper class

225f1e8

Remove HashCode and EquatableArray classes and implement equals directly

27bca2c

MattKotsenas requested a review from SimonCropp May 16, 2024 22:06

SimonCropp merged commit 4463576 into VerifyTests:main May 17, 2024
3 checks passed

SimonCropp added this to the 24.3.0 milestone May 17, 2024

MattKotsenas deleted the feature/mstest-no-base branch May 19, 2024 04:13

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use attribute instead of base class for MSTest #1193

Use attribute instead of base class for MSTest #1193

MattKotsenas commented Apr 13, 2024 •

edited

Loading

MattKotsenas commented May 1, 2024 •

edited

Loading

SimonCropp May 16, 2024

MattKotsenas May 16, 2024

MattKotsenas May 17, 2024

SimonCropp May 17, 2024

MattKotsenas May 16, 2024

MattKotsenas commented May 16, 2024

Use attribute instead of base class for MSTest #1193

Use attribute instead of base class for MSTest #1193

Conversation

MattKotsenas commented Apr 13, 2024 • edited Loading

Overview

Highlights

Open Issues

MattKotsenas commented May 1, 2024 • edited Loading

SimonCropp May 16, 2024

Choose a reason for hiding this comment

MattKotsenas May 16, 2024

Choose a reason for hiding this comment

MattKotsenas May 17, 2024

Choose a reason for hiding this comment

SimonCropp May 17, 2024

Choose a reason for hiding this comment

MattKotsenas May 16, 2024

Choose a reason for hiding this comment

MattKotsenas commented May 16, 2024

MattKotsenas commented Apr 13, 2024 •

edited

Loading

MattKotsenas commented May 1, 2024 •

edited

Loading