Solved multiple issues/PRs:

Issue tcunit#15 - "The verifier wastes quite a bit of time"
Issue tcunit#17 - "There's no need to add any attributes to test suites"
PR tcunit#16 - "Feature/speed up"
Issue tcunit#18 - "Discrepancy between 4022 and 4024 in TIME_TO_STRING makes tests fail"

.gitignore

@@ -9,9 +9,11 @@
 *.compiled-library
 *.compileinfo
 *.tmc
+*.tmcRefac
 *.library
 *.project.~u
 *.tsproj.bak
+*.xti.bak
 _Boot/
 _CompileInfo/
 _Libraries/
@@ -47,6 +49,7 @@ bld/
 [Bb]in/
 [Oo]bj/
 [Ll]og/
+[Ll]ogs/
 
 # Visual Studio 2015/2017 cache/options directory
 .vs/
@@ -144,9 +147,6 @@ _ReSharper*/
 *.[Rr]e[Ss]harper
 *.DotSettings.user
 
-# JustCode is a .NET coding add-in
-.JustCode
-
 # TeamCity is a build add-in
 _TeamCity*
 

README.md

@@ -34,8 +34,8 @@ running, another program is necessary that verifies that the output of the TcVT
 is as expected.
 
 ## TcUnit-Verifier_DotNet
-The TcUnit-Verifier_DotNet (TcVD) is a C# program that opens and runs the
-TcUnit-Verifier_TwinCAT project by the usage of the TwinCAT automation
+The TcUnit-Verifier_DotNet (TcVD) is a (Visual Studio 2013) C# program that opens
+and runs the TcUnit-Verifier_TwinCAT project by the usage of the TwinCAT automation
 interface. It basically does the following:
 - Starts Visual Studio (using the same version that was used developing TcVT)
 - Opens TcVT
@@ -59,23 +59,20 @@ All test classes are instantiated in the class `Program.cs` starting from the
 lines:
 ```
 /* Insert the test classes here */
-new FB_PrimitiveTypes(errorItems);
-new FB_AssertTrueFalse(errorItems);
-new FB_AssertEveryFailedTestTwice(errorItems);
+new FB_PrimitiveTypes(errors);
+new FB_AssertTrueFalse(errors);
+new FB_AssertEveryFailedTestTwice(errors);
 ...
 ...
 ...
 ```
 
 To create a new test class and make sure that it will be running all that is
-necessary is to make sure to instantiate it with two arguments:
-1. A reference to errorItems (just as above)
-2. A string with the name of what the function block is called in TcVT in the
-   PRG_TEST-program
-3. If you have added a test in TcVT that is supposed to fail, and thus adding an
-   additional failed test to the output, you need to increment the variable
-   `expectedNumberOfFailedTests` in TcVD by one for every failed test that you
-   have added. 
+necessary is to make sure to instantiate it with the argument `errors`
+(just as above). If you have added a test in TcVT that is supposed to fail, and
+thus adding an additional failed test to the output, you need to increment the
+variable `expectedNumberOfFailedTests` in TcVD by one for every failed test
+that you have added. 
 
 For example, if we in the PRG_TEST-program of TcVT have a function block
 instantiated in this way:
@@ -85,47 +82,39 @@ VAR
     AssertEveryFailedTestTwiceArrayVersion : FB_AssertEveryFailedTestTwiceArrayVersion;
 END_VAR
 ```
-The equivalent test class in TcVD needs to be instantiated with the second
-argument using the same name as in PRG_TEST. If not provided, the argument's default
-value is the C# class name with the `FB_` prefix removed (if any).
-
-In this example, the class name is `FB_AssertEveryFailedTestTwiceArrayVersion` and thus
-the test class's default argument value is `AssertEveryFailedTestTwiceArrayVersion`. The
-two lines below are equivalent, and the shorter form is preferred whenever possible to
-keep the code DRY (Don't Repeat Yourself).
-
+The equivalent test class in TcVD needs to be instantiated in the following way:
 ```
-new FB_AssertEveryFailedTestTwiceArrayVersion(errorItems);
-// equivalent to
-new FB_AssertEveryFailedTestTwiceArrayVersion(errorItems, "AssertEveryFailedTestTwiceArrayVersion");
+new FB_AssertEveryFailedTestTwiceArrayVersion(errors);
 ```
 
 This is an example of how it can look running the TcUnit-Verifier_DotNet:
 
 ```
 C:\Code\TcUnit\TcUnit-Verifier\TcUnit-Verifier_DotNet\TcUnit-Verifier\bin\Debug>TcUnit-Verifier.exe -v "C:\Code\TcUnit\TcUnit-Verifier\TcUnit-Verifier_TwinCAT\TcUnit-Verifier_TwinCAT.sln"
-2019-12-05 16:23:23 - Starting TcUnit-Verifier...
-2019-12-05 16:23:23 - Loading the Visual Studio Development Tools Environment (DTE)...
-2019-12-05 16:23:49 - Cleaning and building TcUnit-Verifier_TwinCAT solution...
-2019-12-05 16:23:49 - Generating TcUnit-Verifier_TwinCAT boot project...
-2019-12-05 16:24:03 - Activating TcUnit-Verifier_TwinCAT configuration...
-2019-12-05 16:24:07 - Restarting TwinCAT...
-2019-12-05 16:24:07 - Waiting for TcUnit-Verifier_TwinCAT to finish running tests...
-2019-12-05 16:24:08 - ... got 5 report lines so far.
-2019-12-05 16:24:10 - ... got 5 report lines so far.
-2019-12-05 16:24:11 - ... got 26 report lines so far.
-2019-12-05 16:24:12 - ... got 51 report lines so far.
-2019-12-05 16:24:14 - ... got 76 report lines so far.
-2019-12-05 16:24:15 - ... got 93 report lines so far.
-2019-12-05 16:24:16 - ... got 118 report lines so far.
-2019-12-05 16:24:17 - ... got 134 report lines so far.
-2019-12-05 16:24:18 - ... got 151 report lines so far.
-2019-12-05 16:24:20 - ... got 176 report lines so far.
-2019-12-05 16:24:21 - ... got 184 report lines so far.
-2019-12-05 16:24:21 - Asserting results...
-2019-12-05 16:24:21 - Done.
-2019-12-05 16:24:21 - Closing the Visual Studio Development Tools Environment (DTE), please wait...
-2019-12-05 16:24:24 - Exiting application...
+2019-12-08 14:42:40 - Starting TcUnit-Verifier...
+2019-12-08 14:42:40 - Loading the Visual Studio Development Tools Environment (DTE)...
+2019-12-08 14:42:50 - Cleaning and building TcUnit-Verifier_TwinCAT solution...
+2019-12-08 14:42:50 - Generating TcUnit-Verifier_TwinCAT boot project...
+2019-12-08 14:42:55 - Activating TcUnit-Verifier_TwinCAT configuration...
+2019-12-08 14:42:57 - Restarting TwinCAT...
+2019-12-08 14:42:57 - Waiting for TcUnit-Verifier_TwinCAT to finish running tests...
+2019-12-08 14:42:58 - ... got 0 report lines so far.
+2019-12-08 14:42:59 - ... got 0 report lines so far.
+2019-12-08 14:43:00 - ... got 0 report lines so far.
+2019-12-08 14:43:01 - ... got 14 report lines so far.
+2019-12-08 14:43:02 - ... got 31 report lines so far.
+2019-12-08 14:43:03 - ... got 47 report lines so far.
+2019-12-08 14:43:04 - ... got 64 report lines so far.
+2019-12-08 14:43:05 - ... got 81 report lines so far.
+2019-12-08 14:43:06 - ... got 97 report lines so far.
+2019-12-08 14:43:07 - ... got 114 report lines so far.
+2019-12-08 14:43:08 - ... got 131 report lines so far.
+2019-12-08 14:43:09 - ... got 147 report lines so far.
+2019-12-08 14:43:10 - ... got 168 report lines so far.
+2019-12-08 14:43:10 - Asserting results...
+2019-12-08 14:43:11 - Done.
+2019-12-08 14:43:11 - Closing the Visual Studio Development Tools Environment (DTE), please wait...
+2019-12-08 14:43:32 - Exiting application...
 ```
 If there was an error in the TcUnit framework this would be shown between the
 lines `Asserting results...` and `Done.`. If nothing is shown between these

TcUnit-Verifier_DotNet/TcUnit-Verifier/FB_AnyPrimitiveTypes.cs

@@ -226,14 +226,21 @@ private void Test_ANY_STRING_Differ()
 
         private void Test_ANY_TIME_Equals()
         {
+            // TwinCAT 3.1.4020 & 3.1.4022
             string testMessage = CreateFailedTestMessage("Test_ANY_TIME_Equals", "T#694m13s244ms", "T#694m13s244ms", "Values differ");
             AssertDoesNotContainMessage(testMessage);
+            // TwinCAT 3.1.4024 and newer
+            testMessage = CreateFailedTestMessage("Test_ANY_TIME_Equals", "T#12h34m15s10ms", "T#11h34m13s244ms", "Values differ");
+            AssertDoesNotContainMessage(testMessage);
         }
 
         private void Test_ANY_TIME_Differ()
         {
-            string testMessage = CreateFailedTestMessage("Test_ANY_TIME_Differ", "T#754m15s10ms", "T#694m13s244ms", "Values differ");
-            AssertContainsMessage(testMessage);
+            // Differ between TwinCAT 3.1.4020/4022 and 4024 (or newer)
+            string testMessage4022 = CreateFailedTestMessage("Test_ANY_TIME_Differ", "T#754m15s10ms", "T#694m13s244ms", "Values differ");
+            string testMessage4024AndNewer = CreateFailedTestMessage("Test_ANY_TIME_Differ", "T#12h34m15s10ms", "T#11h34m13s244ms", "Values differ");
+            string[] messages = new String[] { testMessage4022, testMessage4024AndNewer };
+            AssertContainsAtLeastOneMessage(messages);
         }
 
         private void Test_ANY_TIME_OF_DAY_Equals()

TcUnit-Verifier_DotNet/TcUnit-Verifier/FB_AssertEveryFailedTestTwice.cs

@@ -72,8 +72,11 @@ private void TwiceAssertCall()
             testMessage = CreateFailedTestMessage("TwiceAssertCall", "34", "36", "Not equal USINT");
             AssertMessageCount(testMessage, 1);
 
-            testMessage = CreateFailedTestMessage("TwiceAssertCall", "T#754m15s10ms", "T#694m13s244ms", "Not equal TIME");
-            AssertMessageCount(testMessage, 1);
+            // Differ between TwinCAT 3.1.4020/4022 and 4024 (or newer)
+            string testMessage4022 = CreateFailedTestMessage("TwiceAssertCall", "T#754m15s10ms", "T#694m13s244ms", "Not equal TIME");
+            string testMessage4024AndNewer = CreateFailedTestMessage("TwiceAssertCall", "T#12h34m15s10ms", "T#11h34m13s244ms", "Not equal TIME");
+            string[] messages = new String[] { testMessage4022, testMessage4024AndNewer };
+            AssertAtLeastOneMessageCount(messages, 1);
 
             testMessage = CreateFailedTestMessage("TwiceAssertCall", "TOD#15:36:30.123", "TOD#06:21:11.492", "Not equal TIME_OF_DAY");
             AssertMessageCount(testMessage, 1);

TcUnit-Verifier_DotNet/TcUnit-Verifier/FB_PrimitiveTypes.cs

@@ -238,14 +238,21 @@ private void Test_STRING_Differ()
 
         private void Test_TIME_Equals()
         {
+            // TwinCAT 3.1.4020 & 3.1.4022
             string testMessage = CreateFailedTestMessage("Test_TIME_Equals", "T#694m13s244ms", "T#694m13s244ms", "Values differ");
             AssertDoesNotContainMessage(testMessage);
+            // TwinCAT 3.1.4024 and newer
+            testMessage = CreateFailedTestMessage("Test_TIME_Equals", "T#12h34m15s10ms", "T#11h34m13s244ms", "Values differ");
+            AssertDoesNotContainMessage(testMessage);
         }
 
         private void Test_TIME_Differ()
         {
-            string testMessage = CreateFailedTestMessage("Test_TIME_Differ", "T#754m15s10ms", "T#694m13s244ms", "Values differ");
-            AssertContainsMessage(testMessage);
+            // Differ between TwinCAT 3.1.4020/4022 and 4024 (or newer)
+            string testMessage4022 = CreateFailedTestMessage("Test_TIME_Differ", "T#754m15s10ms", "T#694m13s244ms", "Values differ");
+            string testMessage4024AndNewer = CreateFailedTestMessage("Test_TIME_Differ", "T#12h34m15s10ms", "T#11h34m13s244ms", "Values differ");
+            string[] messages = new String[] { testMessage4022, testMessage4024AndNewer };
+            AssertContainsAtLeastOneMessage(messages);
         }
 
         private void Test_TIME_OF_DAY_Equals()

TcUnit-Verifier_DotNet/TcUnit-Verifier/TestFunctionBlockAssert.cs

@@ -55,6 +55,31 @@ protected void AssertMessageCount(string message, int messageCount)
             }
         }
 
+        /// <summary>
+        /// Asserts that at least one of the messages in the array exists the messageCount amount of times. Note that if the messageCount will
+        /// increase also if both messages exist.
+        /// </summary>
+        protected void AssertAtLeastOneMessageCount(string[] messages, int messageCount)
+        {
+            int actualCount = 0;
+            foreach (string s in messages)
+            {
+                int count = CountErrorItemsContainingTestMessage(s);
+                actualCount = actualCount + count;
+            }
+
+            if (actualCount != messageCount)
+            {
+                Console.Write("Test suite " + _testFunctionBlockInstance + " reports the messages [");
+                foreach (string s in messages)
+                {
+                    Console.Write(s + ",");
+                }
+                Console.Write("] ");
+                Console.Write(actualCount + " times" + Environment.NewLine);
+            }
+        }
+
         protected void AssertContainsMessage(string message)
         {
             if (!AreErrorItemsContainingTestMessage(message))
@@ -63,6 +88,30 @@ protected void AssertContainsMessage(string message)
             }
         }
 
+        /// <summary>
+        /// Asserts that at least one message in the array exists
+        /// </summary>
+        protected void AssertContainsAtLeastOneMessage(string[] messages)
+        {
+            bool foundMessage = false;
+            foreach (string s in messages)
+            {
+                if (AreErrorItemsContainingTestMessage(s)) { 
+                    foundMessage = true;
+                    break;
+                }
+            }
+            if (!foundMessage)
+            {
+                Console.Write("Test suite " + _testFunctionBlockInstance + " does not report any of the messages: [");
+                foreach (string s in messages)
+                {
+                    Console.Write(s + ",");
+                }
+                Console.Write("]" + Environment.NewLine);
+            }
+        }
+
         protected void AssertDoesNotContainMessage(string message)
         {
             if (AreErrorItemsContainingTestMessage(message))

TcUnit-Verifier_DotNet/TcUnit-Verifier/VisualStudioInstance.cs

@@ -85,6 +85,7 @@ public void Close()
             if (loaded)
             {
                 log.Info("Closing the Visual Studio Development Tools Environment (DTE), please wait...");
+                Thread.Sleep(20000); // Makes sure that there are no visual studio processes left in the system if the user interrupts this program (for example by CTRL+C)
                 dte.Quit();
             }
             loaded = false;

TcUnit-Verifier_TwinCAT/TcUnit-Verifier_TwinCAT/TcUnitVerifier/Test/FB_AdjustAssertFailureMessageToMax252CharLengthTest.TcPOU

@@ -4,7 +4,6 @@
     <Declaration><![CDATA[(* This testsuite tests the function block FB_AdjustAssertFailureMessageToMax255CharLength_Test
    The total printed message can not be more than 252 characters long.
 *)
-{attribute 'call_after_init'}
 FUNCTION_BLOCK FB_AdjustAssertFailureMessageToMax252CharLengthTest EXTENDS TcUnit.FB_TestSuite]]></Declaration>
     <Implementation>
       <ST><![CDATA[TestInstancePath252CharsExpectTooLongTestInstancePath();

TcUnit-Verifier_TwinCAT/TcUnit-Verifier_TwinCAT/TcUnitVerifier/Test/FB_AnyPrimitiveTypes.TcPOU

@@ -10,7 +10,6 @@
     For every failing test the printed value should be the same as if we were using the primitive data type assert
     directly. 
 *)
-{attribute 'call_after_init'}
 FUNCTION_BLOCK FB_AnyPrimitiveTypes EXTENDS TcUnit.FB_TestSuite]]></Declaration>
     <Implementation>
       <ST><![CDATA[Test_ANY_BYTE_Equals();

TcUnit-Verifier_TwinCAT/TcUnit-Verifier_TwinCAT/TcUnitVerifier/Test/FB_AnyToUnionValue.TcPOU

@@ -4,7 +4,6 @@
     <Declaration><![CDATA[(*
     Tests the free function F_AnyToUnionValue.
 *)
-{attribute 'call_after_init'}
 FUNCTION_BLOCK FB_AnyToUnionValue EXTENDS TcUnit.FB_TestSuite
 VAR
     aBit : BIT := 1; // Only structures and function blocks can have BIT defined

TcUnit-Verifier_TwinCAT/TcUnit-Verifier_TwinCAT/TcUnitVerifier/Test/FB_ArrayPrimitiveTypes.TcPOU

@@ -9,7 +9,6 @@
         2. One test that fails, where the size of the arrays differs
         3. One test that fails, where the content of the array differs
 *)
-{attribute 'call_after_init'}
 FUNCTION_BLOCK FB_ArrayPrimitiveTypes EXTENDS TcUnit.FB_TestSuite]]></Declaration>
     <Implementation>
       <ST><![CDATA[Test_BOOL_Array_Equals();

TcUnit-Verifier_TwinCAT/TcUnit-Verifier_TwinCAT/TcUnitVerifier/Test/FB_AssertEveryFailedTestTwice.TcPOU

@@ -5,7 +5,6 @@
     In this testsuite we are running the same testcase over two cycles, and thus calling every assert method twice.
     This is in order to verify that we don't print every FAILED assert twice.
 *)
-{attribute 'call_after_init'}
 FUNCTION_BLOCK FB_AssertEveryFailedTestTwice EXTENDS TcUnit.FB_TestSuite
 VAR
     AssertCount : USINT := 0;

TcUnit-Verifier_TwinCAT/TcUnit-Verifier_TwinCAT/TcUnitVerifier/Test/FB_AssertEveryFailedTestTwiceArrayVersion.TcPOU

@@ -6,7 +6,6 @@
     every assert method twice. This is in order to verify that we don't print every FAILED assert twice.
     With this version we are doing it with arrays.
 *)
-{attribute 'call_after_init'}
 FUNCTION_BLOCK FB_AssertEveryFailedTestTwiceArrayVersion EXTENDS TcUnit.FB_TestSuite
 VAR
     AssertCount : USINT := 0;

TcUnit-Verifier_TwinCAT/TcUnit-Verifier_TwinCAT/TcUnitVerifier/Test/FB_AssertTrueFalse.TcPOU

@@ -6,7 +6,6 @@
     by making sure that every assert-method (assert_true & assert_false) does both one failing and one successful
     assert.
 *)
-{attribute 'call_after_init'}
 FUNCTION_BLOCK FB_AssertTrueFalse EXTENDS TcUnit.FB_TestSuite]]></Declaration>
     <Implementation>
       <ST><![CDATA[AssertThatINTsAreEqual();

TcUnit-Verifier_TwinCAT/TcUnit-Verifier_TwinCAT/TcUnitVerifier/Test/FB_CheckIfSpecificTestIsFinished.TcPOU

@@ -4,7 +4,6 @@
     <Declaration><![CDATA[(*
     Verifies that the function IS_TEST_FINISHED('Name') works as intended.
 *)
-{attribute 'call_after_init'}
 FUNCTION_BLOCK FB_CheckIfSpecificTestIsFinished EXTENDS TcUnit.FB_TestSuite]]></Declaration>
     <Implementation>
       <ST><![CDATA[CheckBeforeAndAfterFinishing();]]></ST>

TcUnit-Verifier_TwinCAT/TcUnit-Verifier_TwinCAT/TcUnitVerifier/Test/FB_CreateDisabledTest.TcPOU

@@ -5,7 +5,6 @@
     This testsuite runs two tests. One that is enabled (and that is supposed to fail), and one that is disabled (and
     even though it also is failing is not supposed to run).
 *)
-{attribute 'call_after_init'}
 FUNCTION_BLOCK FB_CreateDisabledTest EXTENDS TcUnit.FB_TestSuite]]></Declaration>
     <Implementation>
       <ST><![CDATA[TestEnabled();

TcUnit-Verifier_TwinCAT/TcUnit-Verifier_TwinCAT/TcUnitVerifier/Test/FB_CreateFourTestsWithSameName.TcPOU

@@ -5,7 +5,6 @@
     This testsuite creates four tests in the same testsuite with the same name. Every test in a testsuite should have
     an unique name. Every assert should be ignored if a duplicate test is being run.
 *)
-{attribute 'call_after_init'}
 FUNCTION_BLOCK FB_CreateFourTestsWithSameName EXTENDS TcUnit.FB_TestSuite]]></Declaration>
     <Implementation>
       <ST><![CDATA[TestOne();

TcUnit-Verifier_TwinCAT/TcUnit-Verifier_TwinCAT/TcUnitVerifier/Test/FB_EmptyTestSuite.TcPOU

@@ -3,7 +3,6 @@
   <POU Name="FB_EmptyTestSuite" Id="{1ac34d34-5767-43bd-8deb-b4e9cf217e56}" SpecialFunc="None">
     <Declaration><![CDATA[(* This is an empty testsuite with no test cases, so that we can verify that the system still prints the
    test results. *)
-{attribute 'call_after_init'}
 FUNCTION_BLOCK FB_EmptyTestSuite EXTENDS TcUnit.FB_TestSuite
 VAR
 END_VAR]]></Declaration>

TcUnit-Verifier_TwinCAT/TcUnit-Verifier_TwinCAT/TcUnitVerifier/Test/FB_MultipleAssertWithSameParametersInDifferentCyclesAndInSameTest.TcPOU

@@ -9,7 +9,6 @@
     same message, using the same test with the same data values for both expected and actual, and splitting the asserts
     on different cycles.
 *)
-{attribute 'call_after_init'}
 FUNCTION_BLOCK FB_MultipleAssertWithSameParametersInDifferentCyclesAndInSameTest EXTENDS TcUnit.FB_TestSuite
 VAR
     TimerAgain : Tc2_Standard.TON := (PT := T#1S); // Timer to call "AssertSeveralTimesAgain"

TcUnit-Verifier_TwinCAT/TcUnit-Verifier_TwinCAT/TcUnitVerifier/Test/FB_MultipleAssertWithSameParametersInDifferentCyclesButWithDifferentTests.TcPOU

@@ -9,7 +9,6 @@
     same message, in different tests, with the same data values for both expected and actual, although the data that the
     user does the assert on might be from different data sets.
 *)
-{attribute 'call_after_init'}
 FUNCTION_BLOCK FB_MultipleAssertWithSameParametersInDifferentCyclesButWithDifferentTests EXTENDS TcUnit.FB_TestSuite
 VAR
     TimerAgain : Tc2_Standard.TON := (PT := T#1S); // Timer to call "AssertSeveralTimesAgain"

TcUnit-Verifier_TwinCAT/TcUnit-Verifier_TwinCAT/TcUnitVerifier/Test/FB_MultipleAssertWithSameParametersInSameCycleWithSameTest.TcPOU

@@ -8,7 +8,6 @@
     same message, in the same test, with the same data values for both expected and actual, although the data that the
     user does the assert on might be from different data sets.
 *)
-{attribute 'call_after_init'}
 FUNCTION_BLOCK FB_MultipleAssertWithSameParametersInSameCycleWithSameTest EXTENDS TcUnit.FB_TestSuite
 ]]></Declaration>
     <Implementation>

TcUnit-Verifier_TwinCAT/TcUnit-Verifier_TwinCAT/TcUnitVerifier/Test/FB_PrimitiveTypes.TcPOU

@@ -7,7 +7,6 @@
         1. One test that succeeds (Expected and actual are equal)
         2. One test that fails (Expected and actual are false).
 *)
-{attribute 'call_after_init'}
 FUNCTION_BLOCK FB_PrimitiveTypes EXTENDS TcUnit.FB_TestSuite]]></Declaration>
     <Implementation>
       <ST><![CDATA[Test_ANY_Equals();