OSOE-770: Add support for structured html-validate output #354
Conversation
Added ability to get parsed html-validate output
| [JsonPropertyName("ruleUrl")] | ||
| public string RuleUrl { get; set; } | ||
| [JsonPropertyName("context")] | ||
| public JsonElement Context { get; set; } |
There was a problem hiding this comment.
The reason this is JsonElement instead of something else is because this doesn't seem to have the same properties every time see sample output below:
| /// Gets the parsed errors from the HTML validation result. | ||
| /// Can only be used if the output formatter is set to JSON. | ||
| /// </summary> | ||
| public static async Task<IEnumerable<HtmlValidationError>> GetParsedErrorsAsync(this HtmlValidationResult result) |
There was a problem hiding this comment.
It might be a good idea to rename it to something more specific like GetJsonParsedErrorsAsync, if we would add more formatters.
|
|
||
| namespace Lombiq.Tests.UI.Models; | ||
|
|
||
| public class HtmlValidationError |
There was a problem hiding this comment.
I think this should be more specific too (like JsonHtmlValidationError or something like that.
| @@ -28,6 +30,8 @@ public class HtmlValidationConfiguration | |||
| /// </summary> | |||
| public HtmlValidationOptions HtmlValidationOptions { get; set; } = new() | |||
| { | |||
| ResultFileFormatter = HtmlValidateFormatter.Names.Json, | |||
There was a problem hiding this comment.
Add a note about the formatter in: https://github.com/Lombiq/UI-Testing-Toolbox/blob/dev/Lombiq.Tests.UI/Docs/Configuration.md and a note on how to use the original one instead (if someone wants to).
There was a problem hiding this comment.
It's good that now we can filter in a simpler way, but the output message should be adjusted too, because now it's not human readable:
Lombiq.Tests.UI.Exceptions.PageChangeAssertionException : An assertion during the page change event has failed on page https://localhost:9024/ (Lombiq's OSOCE - UI Testing - Blog).
---- Lombiq.Tests.UI.Exceptions.HtmlValidationAssertionException : await validationResult.GetParsedErrorsAsync()
should be empty but had
1
item and was
[Lombiq.Tests.UI.Models.HtmlValidationError (2681320)] Check the HTML validation report in the failure dump for details.
-------- Shouldly.ShouldAssertException : await validationResult.GetParsedErrorsAsync()
should be empty but had
1
item and was
[Lombiq.Tests.UI.Models.HtmlValidationError (2681320)]
Previously the errors names were listed here.
Like this:
Lombiq.Tests.UI.Exceptions.PageChangeAssertionException : An assertion during the page change event has failed on page https://localhost:9022/ (Lombiq's OSOCE - UI Testing - Blog).
---- Lombiq.Tests.UI.Exceptions.HtmlValidationAssertionException : validationResult.Output
should be empty but was
"G:\Work\Open-Source-Orchard-Core-Extensions\test\Lombiq.OSOCE.Tests.UI\bin\Debug\net8.0\HtmlValidationTemp\5bc0f7c6-8eb3-4ec7-acf5-6744e3373ea6.html
44:8 error Prefer to use the native <button> element prefer-native-element
✖ 1 problem (1 error, 0 warnings)
More information:
https://html-validate.org/rules/prefer-native-element.html
" Check the HTML validation report in the failure dump for details.
-------- Shouldly.ShouldAssertException : validationResult.Output
should be empty but was
"G:\Work\Open-Source-Orchard-Core-Extensions\test\Lombiq.OSOCE.Tests.UI\bin\Debug\net8.0\HtmlValidationTemp\5bc0f7c6-8eb3-4ec7-acf5-6744e3373ea6.html
44:8 error Prefer to use the native <button> element prefer-native-element
✖ 1 problem (1 error, 0 warnings)
More information:
https://html-validate.org/rules/prefer-native-element.html
"
(To repro this, just remove the custom configuration.HtmlValidationConfiguration.AssertHtmlValidationResultAsync from a test that uses it. Also to get what should the output look like, you can check out the dev branch and do the same thing.)
And I think the same thing applies to the HtmlValidationReport.txt (that is generated in the UI test failure dump (locally \Open-Source-Orchard-Core-Extensions\test\Lombiq.OSOCE.Tests.UI\bin\Debug\net8.0\FailureDumps). I think the format of that changed too:
Previously:
HTMLVA~1.TXT
There was a problem hiding this comment.
@DemeSzabolcs I have set the output back to the text for the output file so that should be fixed. For the message on the actual test run I can not fully put that back the way it was because it's just the way that Shouldly reports on the errors based on the new format.
What I did try was pass a custom message to shouldly and then the test output will look like: here
But this would mean that we'd have to pass in the custom string formatter into everywhere we use shouldly. If you have another idea on how to solve this I'd be happy to hear.
There was a problem hiding this comment.
Well if it's this way because of how Shouldly summarizes the error and it's not our code, then I think using that custom message is a fine solution.
OSOE-770
Added ability to get parsed html-validate output
Fixes #337