nunit console can't parse test case with special characters

[WindingWinter]

I'm using nunit console 3.18.1, and here's a very interesting case concerning a .net 8.0 windows test project. Here's the project configuration

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <TargetFramework>net8.0-windows</TargetFramework>
	<UseWPF>True</UseWPF>
	<OutputType>Library</OutputType>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="NUnit" Version="3.14.0" />
	<PackageReference Include="Microsoft.Net.Test.Sdk" Version="17.10.0" />
	<PackageReference Include="NUnit.ConsoleRunner" Version="3.18.1" />
	    <PackageReference Include="NUnit3TestAdapter" Version="4.6.0" />
	<PackageReference Include="NUnit.Extension.VSProjectLoader" Version="3.9.0" />
  </ItemGroup>

</Project>

And inside the project it contains one single test class, with one single test case:

  public class SpecialCharTestClass
   
  {
      public static IEnumerable<char> CharTestCases()
      {
          for (char c = char.MinValue; c < 0xDEEE; c += (char)0x0FED)
              yield return c;
          yield return char.MaxValue;
      }
      [TestCaseSource(nameof(CharTestCases))]
      public void CharTest(char c)
      {


      }


  }

I can run it in NUnit3TestAdapter without problem. But if I run it from nunit console, then I will get this XML Exception

NUnit Console 3.18.1+5e16ca2ef85f57f9f09bb41ff08feb8f3168a384 (Release)
Copyright (c) 2022 Charlie Poole, Rob Prouse
Tuesday, September 3, 2024 2:11:23 PM

Runtime Environment
   OS Version: Microsoft Windows NT 6.2.9200.0
   Runtime: .NET Framework CLR v4.0.30319.42000

Test Files
    XXX.sln

System.Xml.XmlException : '?', hexadecimal value 0xFFFF, is an invalid character. Line 1, position 1856.

--XmlException
'?', hexadecimal value 0xFFFF, is an invalid character. Line 1, position 1856.
   at System.Xml.XmlTextReaderImpl.Throw(Exception e)
   at System.Xml.XmlTextReaderImpl.ThrowInvalidChar(Char[] data, Int32 length, Int32 invCharPos)
   at System.Xml.XmlTextReaderImpl.ParseCDataOrComment(XmlNodeType type, Int32& outStartPos, Int32& outEndPos)
   at System.Xml.XmlTextReaderImpl.ParseCDataOrComment(XmlNodeType type)
   at System.Xml.XmlTextReaderImpl.ParseElementContent()
   at System.Xml.XmlLoader.LoadNode(Boolean skipOverWhitespace)
   at System.Xml.XmlLoader.LoadDocSequence(XmlDocument parentDoc)
   at System.Xml.XmlDocument.Load(XmlReader reader)
   at System.Xml.XmlDocument.LoadXml(String xml)
   at NUnit.ConsoleRunner.TestEventHandler.OnTestEvent(String report)
   at NUnit.Engine.Runners.TestEventDispatcher.OnTestEvent(String report)
   at NUnit.Engine.Runners.MasterTestRunner.RunTests(ITestEventListener listener, TestFilter filter)
   at NUnit.Engine.Runners.MasterTestRunner.Run(ITestEventListener listener, TestFilter filter)
   at NUnit.ConsoleRunner.ConsoleRunner.RunTests(TestPackage package, TestFilter filter)
   at NUnit.ConsoleRunner.Program.Main(String[] args)

I don't think it appears if I compile the test project as .net framework 4.8

[CharliePoole]

@WindingWinter Please include the exact command-line used to start the console runner.

[WindingWinter]

@CharliePoole , here's the command line

call  "%NunitConsole%\nunit3-console.exe" "%slnFile%" /config:Debug --skipnontestassemblies  --result="%testresultxml%"  --timeout=2000 > "%testresulttxt%"

I'm running it with a solution file, but if I run it with a DLL file the problem is there all the same.

[WindingWinter]

Also, if I run it with dotnet test "%slnFile%" then there is no such problem.

[CharliePoole]

The framework generates an XML fragment for each test event (like starting the test, finishing the test, etc.), which contains the name NUnit gives to your test. In the case of a parameterized test, the default test name contains the arguments provided. In your case, the name contains an invalid character.

When you run under nunit3-console, these events are all handled. The first step is to process the XML fragment received by creating an XmlDocument. That's what it's failing here. I guess dotnet test works because it doesn't process those events.

Ways to fix this:

1. NUnit framework could stop generating invalid XML
2. The console runner could examine the string for invalid characters before trying to create an XML document from it.
3. The console runner could stop using XmlDocument entirely and parse the string itself.
4. We could simply document the fact that the default naming doesn't work, requiring the user to specify an alternate name for any test with invalid characters in the provided arguments.

@nunit Any thoughts on how to handle this?

[WindingWinter]

@CharliePoole , the interesting thing to note is that if I run the code under .net framework 4.8 ( and use 4.8 agent), then it works.
Only under .net 8.0 it doesn't.

I'm not too sure how to square your explanation with my observation

[OsirisTerje]

The adapter based test will get 15 tests, one is not executed, and nothing crashes.

Here is the dump file, showing the XML content coming from the NUnit.Engine
E_Issue1469.dll.zip

Btw: The one character that crashes doesnt work in the adapter either, it comes out as not run:

The adapter does process the events, and reads and parses the events, using XmlNode (XmlDocument based), so not sure why it would crash in the console and not in the adapter.

The \uFFFF char is the End_Of_Everything character, or a non-character, and should not really be a part of something you would test. If you filter that off from your test, the rest should work

[CharliePoole]

@WindingWinter @OsirisTerje That's all very odd. It seems like it should either work or not in all environments. I'll run some more tests.

[CharliePoole]

Looks somewhat related to #1349 although it occurs in the context of a user-supplied name with special characters. Probably should fix both at the same time.

[CharliePoole]

I have replicated this in a package test and in the unit tests. The unit tests run under NUnitLite and pass. The package tests run in an agent and fail for both .NET 6.0 and .NET 8.0.

I also added package tests to the NetCore console runner, which doesn't use agents, and they pass as well. So... problem is probably in the agent code.

[CharliePoole]

One thing for sure: the exception is caused by /uffff coming from the framework as part of the test name. But this is still a console/engine issue because the runner supports multiple frameworks and needs to protect itself from bad data.

[CharliePoole]

I filed issue nunit/nunit#4826 to deal with the framework part of this issue.

I initially patched the engine and console separately to deal with two different places where an exception was being thrown but in the end I found one central location to fix it in the engine's TestEventDispatcher.

[CharliePoole]

Fix available on MyGet as version 3.18.2-dev00027

[CharliePoole]

This issue has been resolved in version 3.18.2

The release is available on:
GitHub.
NuGet packages are also available NuGet.org and
Chocolatey Packages may be found at Chocolatey.org