Mention containers and tests that were ignored because of their tags

[nipafx]

Tagging and filtering works great, but developers could be irritated when their test is not executed (because of a tag) and they don't know why (maybe being new to the code base).

Jupiter could print a message that tests were ignored because their tag was not activated. If these information are easy to gather without performance impact, it could even print some numbers, something like:

12 classes and 132 methods are tagged with "integration" and not executed

[debugduck]

I'm new, but I'd like to try this out. Do you mind if I give it a try?

[nipafx]

Hi @debugduck, I can't make that call. While I occasionally contribute, I'm not a member of @junit-team and can hence not decide what gets implemented and what doesn't. The team is currently working on the 5.3.0 release - let's hope they take a look at this issue over the next days.

[sbrannen]

@debugduck, I cannot promise that we will include this feature, but I do think it stands a good chance for inclusion since "diagnostics" is an area in which we would like to make improvements in general.

In light of that, I have added the "up for grabs" label.

So feel free to submit a PR.

Cheers,

Sam

[sbrannen]

Assigned to the General Backlog for the time being.

[antimpatel]

@sbrannen I came across this issue looking for up-for-grabs on junit5.
I have spent a few days to understand the issue and have been able to reproduce it and perform some tweaks on it. Here is my understanding :

• The exclusion of tests based on tags is done in discovery phase. Exclusions are removed from ExecutionTree itself.
• Reporting or providing a summary is done via listeners that are invoked on execution events such as test execution start or end etc.

There are two ways that I could think of approaching this.

1. Receive another listener / reporter, for discovery phase events / reports at the start of execution, similar to TestExecutionListener. Pass such exclusion information to that during the discovery phase. Engines can then consume that and get a summary of what happened during the discovery phase. This could be along the lines of Introduce error handling mechanism for validation and discovery errors #242 and can serve general purpose reporting of events (maybe validation errors etc as well) of discovery phase.
2. Do not remove the excluded tests during discovery phase from the execution tree, pass some information indicating that these are excluded methods in the TestDescriptor and consume that during the execution. (Flow similar to skipped tests.) During the execution we can report the events of exclusion to the test execution listener (will need to add some new methods to the listener interface for such events).
3. Extend the TestExecutionListener, add some methods to the interface for discovery phase events. Pass the listener to discoveryRequest and eventually to Root.java. The filters can then send events / errors to this listener.

I along with @logic-1 want to take a shot at this issue and similar issues in junit diagnostics (Mostly the ones summarised in #242 ). Please guide us here, we are new to open source but we would like to give it a try.

[marcphilipp]

Thanks for the detailed summary. I think (2) is not an option because it would require all test engines to be adapted and change the current behavior.

As I currently don't have any idea how build tools or IDEs would display such information, how about we collect this information in DefaultLauncher and log it (if tests have been excluded)? If we later find a use case for reporting it to tools using some form of a listener, we can always change it.

[antimpatel]

Since junit already provide interface to use for build tools/ IDEs to gather result of each test method /containers , so they can create final test summary in whatever way they want to display.
Similar to this can we provide one more Listener to discovery phase, which will listen events that occurred during discovery phase.
instead of collecting all exclusion reasons at default launcher we can pass that information directly to listener so corresponding build tools/IDEs can consume that in same way they do for ,TestExecutionListener.

@sbrannen can you suggest something around this.?

[RoryBlevins]

@sbrannen Is this issue still being worked on? Looks like it's gone silent and I'd like to have a go at it as it's up-for-grabs

[sbrannen]

@antimpatel, sorry for the belated reply. This somehow slipped under the radar.

I tend to agree with @marcphilipp. At this point in time, we are not aware of considerable interest in having this information reported in builds or IDEs.

Rather, we believe that logging filtered out containers and tests would be sufficient for an initial attempt at providing such diagnostic information to users.

If we later determine that there is significant community interest in a formal reporting mechanism for this, we then might consider options # 1 and # 3 that you suggested.

[sbrannen]

@RoryBlevins, I'm not sure if @antimpatel is still working on this or interested in continuing to work on this.

So let's see if we receive a response from @antimpatel.

[RoryBlevins]

👍 Feel free to point me in the direction of something else sensible for someone new to the project to pick up

[sbrannen]

@RoryBlevins, see if there's anything in the following list that piques your interest: up-for-grabs

If not, you can always look at open issues in one of the backlogs and ask on the issue if contributions would be welcome.

[ptlanu22]

@sbrannen, @marcphilipp, Yes would like to continue here.

@marcphilipp As mentioned by you I need to collect data in Default Launcher, After browsing code here is what I have figured out, let me know if a better way exists,

Since Root object is responsible for removing the methods with tags(considering only methods should be printed which are excluded not the tags),

Approach: I can pass reason from TagFilter for both inclusion and exclusion, which will be available in filterResult object, and in Root, store(maybe a map with TestEngine) filter results of all excluded methods. which will be accessible in DefaultLauncher. Here We can not club filterResult only for TagFilter since we don't have any type in postDiscoveryFilter.

[marcphilipp]

Can we group the excluded TestDescriptors by their exclusion reason and take into account all PostDiscoveryFilters?

[ptlanu22]

@marcphilipp will do it and raise the PR for the same.