Reports

Reports

<<<<<<< HEAD Automated tests need good reporting so that the test results are meaningful. For this purpose, we use the well-known Allure framework. We have a short guide on how to set up Allure.

Automated tests need good reporting so that the test results are meaningful. For this purpose, we use the well-known Allure framework.

We have a short guide on how to set up Allure.

#302-support-lighthouse-reports

To get even more insights into your test results we enriched the reports by additional data. This article is about the additional features we have added and how to use them.

Add Links To Report

Sometimes it is necessary to know the exact link a step in the automation opened and sometimes it's just convenient to jump into a flow at a certain point. For this we have created a function which adds the currently displayed link as a step to the Allure report.

As soon as it is activated, every time a new page is called, a link to exactly this page is stored in the report. This way you can see if the link is correct and even open the page to have a look at it.

This is activated per default but can be deactivated via the neodymium.report.enableStepLinks property in the Neodymium configuration.

Add Test Data To Report

When looking into a failed testcase, it is necessary to get the corresponding test data to be able to replicate the issue. To not be forced to look through a bunch of code just to find the used test data, we have implemented the display of test data in JSON format for each test in the Allure report.

This is activated by default but can be deactivated if the neodymium.report.enableTestDataInReport property in the Neodymium configuration is set to false.

<<<<<<< HEAD While activated, this feature adds the test data as an attachment with the name always starting with "Testdata" to the Allure report. It is important to mention, that it only applies to test data that is initialized with DataUtils or DataItem, both belonging to Neodymium.

While activated, this feature adds the test data as an attachment with the name always starting with "Testdata" to the Allure report. It is important to mention, that it only applies to test data that is initialized with DataUtils or DataItem, both belonging to Neodymium.

#302-support-lighthouse-reports

If the test data changes during the test run, it is also possible to add the changes as an additional attachment to the Allure report. To accomplish that, we implemented a helper function addDataAsJsonToReport(String name, Object data) in AllureAddons, belonging to Neodymium. The parameter name is the name of the attachment and the parameter data is the changed testdata.

Add Json Compare To Report

If you compare different json data and the comparison fails, you either get a huge and confusing error message or a short meaningless one, where you don't see what exactly went wrong. In order to prevent wasting time on looking through those kind of error messages, we have implemented the class JsonAssert, for what we utilized the already existing JSONAssert.

If assertEquals fails, an attachment named "Json Compare" containing the differences of the json data is added to the Allure report. If assertNotEquals fails, an attachment named "Json View" containing the json data once is added to the Allure report.

Add custom data to Environment Section of the Report

The Allure report offers a "Environment" section to store information about the testenvironment. This is especially helpful for keeping track of all the changes made to the testenvironment and their impact on the testreults. If you want to store your own information in this section, Neodymium provides a quick and easy way to do so. To activate the usage of custom data, you have to activate the neodymium.report.environment.enableCustomData property in the Neodymium configuration.

Now to add a line to the environment section of the report simply add a line to a property file starting with neodymium.report.environment.custom. and a custom key value pair. This key value pair will than be added to your report. An example could look like this:

neodymium.report.environment.custom.yourKey = yourValue

The following spaces are eligible for storing such custom data pairs:

1. config/dev-neodymium.propeties in the neodymium testsuite
2. System environment variables
3. System properties
4. config/credentials.properties in the neodymium testsuite
5. config/neodymium.properties in the neodymium testsuite

This is also the order in which the custom data is read.

Note: If there are duplicate keys, the value of the higher ranked file is taken.

Add used Browser Configurations to report

Neodymium provides the ability to define different Browser configurations for tests. It also displays all actually used configurations displayed in the enrionment section of the Allure report. This can be turned of by setting the neodymium.report.environment.enableBrowserData to false in the Neodymium configuration.

Selenide Error Details Configuration

Selenide prints a lot of information to their error messages by default. Also Allure categorizes errors in each run by simmilar error messages. Unfortunately, this does not work together well. Due to specific information like the path to the Selenide screenshot, each Selenide error message is unique.

To make sure both features can be used as indet we introduced a toggle to reduce the information of each Selenide error message, to enable Allures category feature.

This can be configured using the neodymium.report.showSelenideErrorDetails property in neodymium.properties file.

If you run you run tests locally without Allure you might want to set this value to true in the dev-neodymium.properties file to see the screenshot path and other information.

Lighthouse

Accessibility reports are crucial when testing web pages because they help to ensure that a site is usable for people of all abilities. That is why we introduced Google Lighthouse to Neodymium, an open-source tool to improve the quality of web pages.

Install Lighthouse CLI

First of all we recommend installing a package manager like npm, which we are also going to use in order to install Lighthouse CLI. After you went through the installation process of npm, open a terminal and enter the command below.

npm install -g lighthouse

To make sure the Lighthouse installation was successful, you can run the following command.

lighthouse --version

Use Lighthouse Inside Neodymium

With the objective of creating Lighthouse reports inside Neodymium we implemented the class LighthouseUtils, containing the function createLightHouseReport(String reportName). By calling this function, a Lighthouse report of the current web page is generated and automatically added to the Allure report with the name specified in the reportName parameter. Keep in mind that creating a Lighthouse report only works while using Chrome or Chromium-based browsers.

This method call can be added at every point of your test and will create a report of the currently opened page. This also works for pages, which rely on session data like a login state or products that should have been added to a shopping cart. Keep in mind that it may not work on certain pages that, if refreshed, load a different page, like checkout pages for example.

Example:

    @NeodymiumTest
    public void testLoginAsRegisteredUser() throws Exception
    {
        // go to homepage
        var homePage = OpenHomePageFlow.flow();
        LighthouseUtils.createLightHouseReport("Homepage");

        // go to register page
        var registerPage = homePage.header.userMenu.openRegisterPage();

        // send register form
        var loginPage = registerPage.sendRegisterForm(registeredOrderTestData.getUser());

        // send login form
        var accountOverviewPage = loginPage.sendLoginForm(registeredOrderTestData.getUser());
        accountOverviewPage.validateSuccessfulLogin(registeredOrderTestData.getUser().getFirstName());

        LighthouseUtils.createLightHouseReport("Account Overview Page");
    }

Please note that this example uses page object model classes which are not shipped with neodymium.

Lighthouse Report

The generated Lighthouse reports can be found in the target directory of the user's repository and in the Allure report as an Attachment, visualized in the following image.

A Lighthouse report consists of the following four categories.

1. Performance
2. Accessibility
3. Best Practices
4. Search Engine Optimisation (SEO)

Each of those categories get a score between 1 and 100, which reveals how well the web page performed in every category. Therefore Google defines the following ranges.

• 0 to 49: Poor
• 50 to 89: Needs Improvement
• 90 to 100: Good

Lighthouse Report Validation

Lighthouse Category Score Validation

To enable validating the scores of all four categories, we implemented the following score thresholds in the neodymium.properties file.

• neodymium.lighthouse.assert.thresholdScore.performance
• neodymium.lighthouse.assert.thresholdScore.accessiblity
• neodymium.lighthouse.assert.thresholdScore.bestPractices
• neodymium.lighthouse.assert.thresholdScore.seo

All of those configuration properties are set to 0.5 per default, which sets the score threshold of all categories to 50. That means each and every category needs to match or exceed a score of 50 for the test not to fail. All of the score thresholds can be changed depending on the user's wishes.

Please be aware that the Lighthouse performance score is not the most stable value and is affected by many factors like networkload on th the test mashine. This can lead to random outliers in the measurement making your tests flaky. A check against the 75th percentile is recomended to give the test some stability (see for example here). Therefore Lighthouse reports during test automation are not recommended to be used for performance testing, but can be a helpful tool to do add a lower border for your performance to get an alert if something is wrong.

Lighthouse Audit Validation

We also implemented the property neodymium.lighthouse.assert.audits in the neodymium.properties file. This property makes it possible to validate Lighthouse audits. In order to do that the user has to specify the id of all audits that should be validated as the property itself. For example: neodymium.lighthouse.assert.audits = aria-roles aria-text validates that no error occurs in the Lighthouse audits aria-roles and aria-text. All existing audit ID's and their correponsing titles are visualized in the table below.

id	title
is-on-https	Uses HTTPS
redirects-http	Redirects HTTP traffic to HTTPS
viewport	Has a <meta name="viewport"> tag with width or initial-scale
first-contentful-paint	First Contentful Paint
largest-contentful-paint	Largest Contentful Paint
first-meaningful-paint	First Meaningful Paint
speed-index	Speed Index
screenshot-thumbnails	Screenshot Thumbnails
final-screenshot	Final Screenshot
total-blocking-time	Total Blocking Time
max-potential-fid	Max Potential First Input Delay
cumulative-layout-shift	Cumulative Layout Shift
errors-in-console	No browser errors logged to the console
server-response-time	Initial server response time was short
interactive	Time to Interactive
user-timings	User Timing marks and measures
critical-request-chains	Avoid chaining critical requests
redirects	Avoid multiple page redirects
image-aspect-ratio	Displays images with correct aspect ratio
image-size-responsive	Serves images with appropriate resolution
deprecations	Avoids deprecated APIs
third-party-cookies	Avoids third-party cookies
mainthread-work-breakdown	Minimizes main-thread work
bootup-time	JavaScript execution time
uses-rel-preconnect	Preconnect to required origins
font-display	Ensure text remains visible during webfont load
diagnostics	Diagnostics
network-requests	Network Requests
network-rtt	Network Round Trip Times
network-server-latency	Server Backend Latencies
main-thread-tasks	Tasks
metrics	Metrics
resource-summary	Resources Summary
third-party-summary	Minimize third-party usage
third-party-facades	Lazy load third-party resources with facades
largest-contentful-paint-element	Largest Contentful Paint element
lcp-lazy-loaded	Largest Contentful Paint image was not lazily loaded
layout-shifts	Avoid large layout shifts
long-tasks	Avoid long main-thread tasks
non-composited-animations	Avoid non-composited animations
unsized-images	Image elements do not have explicit width and height
valid-source-maps	Page has valid source maps
prioritize-lcp-image	Preload Largest Contentful Paint image
csp-xss	Ensure CSP is effective against XSS attacks
script-treemap-data	Script Treemap Data
accesskeys	[accesskey] values are unique
aria-allowed-attr	[aria-*] attributes match their roles
aria-allowed-role	Uses ARIA roles only on compatible elements
aria-command-name	button, link, and menuitem elements have accessible names
aria-conditional-attr	ARIA attributes are used as specified for the element's role
aria-deprecated-role	Deprecated ARIA roles were not used
aria-dialog-name	Elements with role="dialog" or role="alertdialog" have accessible names.
aria-hidden-body	[aria-hidden="true"] is not present on the document <body>
aria-hidden-focus	[aria-hidden="true"] elements do not contain focusable descendents
aria-input-field-name	ARIA input fields have accessible names
aria-meter-name	ARIA meter elements have accessible names
aria-progressbar-name	ARIA progressbar elements have accessible names
aria-prohibited-attr	Elements use only permitted ARIA attributes
aria-required-attr	[role]s have all required [aria-*] attributes
aria-required-children	Elements with an ARIA [role] that require children to contain a specific [role] have all required children.
aria-required-parent	[role]s are contained by their required parent element
aria-roles	[role] values are valid
aria-text	Elements with the role=text attribute do not have focusable descendents.
aria-toggle-field-name	ARIA toggle fields have accessible names
aria-tooltip-name	ARIA tooltip elements have accessible names
aria-treeitem-name	ARIA treeitem elements have accessible names
aria-valid-attr-value	[aria-*] attributes have valid values
aria-valid-attr	[aria-*] attributes are valid and not misspelled
button-name	Buttons do not have an accessible name
bypass	The page contains a heading, skip link, or landmark region
color-contrast	Background and foreground colors have a sufficient contrast ratio
definition-list	<dl>'s contain only properly-ordered <dt> and <dd> groups, <script>, <template> or <div> elements.
dlitem	Definition list items are wrapped in <dl> elements
document-title	Document has a <title> element
duplicate-id-aria	ARIA IDs are unique
empty-heading	All heading elements contain content.
form-field-multiple-labels	No form fields have multiple labels
frame-title	<frame> or <iframe> elements have a title
heading-order	Heading elements are not in a sequentially-descending order
html-has-lang	<html> element has a [lang] attribute
html-lang-valid	<html> element has a valid value for its [lang] attribute
html-xml-lang-mismatch	<html> element has an [xml:lang] attribute with the same base language as the [lang] attribute.
identical-links-same-purpose	Identical links have the same purpose.
image-alt	Image elements have [alt] attributes
image-redundant-alt	Image elements do not have [alt] attributes that are redundant text.
input-button-name	Input buttons have discernible text.
input-image-alt	<input type="image"> elements have [alt] text
label-content-name-mismatch	Elements with visible text labels have matching accessible names.
label	Form elements have associated labels
landmark-one-main	Document has a main landmark.
link-name	Links do not have a discernible name
link-in-text-block	Links are distinguishable without relying on color.
list	Lists contain only <li> elements and script supporting elements (<script> and <template>).
listitem	List items (<li>) are contained within <ul>, <ol> or <menu> parent elements
meta-refresh	The document does not use <meta http-equiv="refresh">
meta-viewport	[user-scalable="no"] is not used in the <meta name="viewport"> element and the [maximum-scale] attribute is not less than 5.
object-alt	<object> elements have alternate text
select-name	Select elements have associated label elements.
skip-link	Skip links are focusable.
tabindex	No element has a [tabindex] value greater than 0
table-duplicate-name	Tables have different content in the summary attribute and <caption>.
table-fake-caption	Tables use <caption> instead of cells with the [colspan] attribute to indicate a caption.
target-size	Touch targets do not have sufficient size or spacing.
td-has-header	<td> elements in a large <table> have one or more table headers.
td-headers-attr	Cells in a <table> element that use the [headers] attribute refer to table cells within the same table.
th-has-data-cells	<th> elements and elements with [role="columnheader"/"rowheader"] have data cells they describe.
valid-lang	[lang] attributes have a valid value
video-caption	<video> elements contain a <track> element with [kind="captions"]
custom-controls-labels	Custom controls have associated labels
custom-controls-roles	Custom controls have ARIA roles
focus-traps	User focus is not accidentally trapped in a region
focusable-controls	Interactive controls are keyboard focusable
interactive-element-affordance	Interactive elements indicate their purpose and state
logical-tab-order	The page has a logical tab order
managed-focus	The user's focus is directed to new content added to the page
offscreen-content-hidden	Offscreen content is hidden from assistive technology
use-landmarks	HTML5 landmark elements are used to improve navigation
visual-order-follows-dom	Visual order on the page follows DOM order
uses-long-cache-ttl	Serve static assets with an efficient cache policy
total-byte-weight	Avoids enormous network payloads
offscreen-images	Defer offscreen images
render-blocking-resources	Eliminate render-blocking resources
unminified-css	Minify CSS
unminified-javascript	Minify JavaScript
unused-css-rules	Reduce unused CSS
unused-javascript	Reduce unused JavaScript
modern-image-formats	Serve images in next-gen formats
uses-optimized-images	Efficiently encode images
uses-text-compression	Enable text compression
uses-responsive-images	Properly size images
efficient-animated-content	Use video formats for animated content
duplicated-javascript	Remove duplicate modules in JavaScript bundles
legacy-javascript	Avoid serving legacy JavaScript to modern browsers
doctype	Page has the HTML doctype
charset	Properly defines charset
dom-size	Avoids an excessive DOM size
geolocation-on-start	Avoids requesting the geolocation permission on page load
inspector-issues	No issues in the Issues panel in Chrome Devtools
no-document-write	Avoids document.write()
js-libraries	Detected JavaScript libraries
notification-on-start	Avoids requesting the notification permission on page load
paste-preventing-inputs	Allows users to paste into input fields
uses-http2	Use HTTP/2
uses-passive-event-listeners	Uses passive listeners to improve scrolling performance
meta-description	Document does not have a meta description
http-status-code	Page has successful HTTP status code
font-size	Document uses legible font sizes
link-text	Links have descriptive text
crawlable-anchors	Links are crawlable
is-crawlable	Page isn’t blocked from indexing
robots-txt	robots.txt is not valid
hreflang	Document has a valid hreflang
canonical	Document has a valid rel=canonical
structured-data	Structured data is valid
bf-cache	Page prevented back/forward cache restoration