Support lighthouse reports #310

kqmpetenz · 2024-11-18T08:23:25Z

added the possibility to create lighthouse reports

wurzelkuchen

nice work, now for the polishing ;)

src/main/java/com/xceptance/neodymium/util/LighthouseUtils.java

config/neodymium.properties

src/main/java/com/xceptance/neodymium/util/LighthouseUtils.java

wurzelkuchen · 2024-11-21T11:21:40Z

src/main/java/com/xceptance/neodymium/util/LighthouseUtils.java

+        validateAudits(jsonFile);
+
+        // add report html to allure
+        Allure.addAttachment(reportName, "text/html", FileUtils.openInputStream(new File("target/" + reportName + ".report.html")), "html");


if any assertion is thrown, we won't add the report as an attachment. we need to first add it as an attachment, and afterwards, validate it.

wurzelkuchen · 2024-11-21T12:59:36Z

src/main/java/com/xceptance/neodymium/util/LighthouseUtils.java

+        String assertAuditsString = Neodymium.configuration().lighthouseAssertAudits();
+        List<String> errorAudits = new ArrayList<>();
+
+        if (!assertAuditsString.isEmpty()) 


This value is null if not set, so calling isEmpty() on a null object will crash. Use StringUtils.isNotBlank(assertAuditString) for this.

wurzelkuchen · 2024-11-21T13:07:36Z

src/main/java/com/xceptance/neodymium/util/LighthouseUtils.java

+     *            The name of the Lighthouse report attachment in the Allure report
+     * @throws Exception
+     */
+    public static void createLightHouseReport(WebDriver driver, String reportName) throws Exception 


We should make the interface as simple as possible. There is no need to have the webdriver given as a parameter. Just use Neodymium.getWebdriver() inside this method.

wurzelkuchen · 2024-11-21T13:20:20Z

src/test/java/com/xceptance/neodymium/util/LighthouseUtilsTest.java

+        else if (System.getProperty("os.name").toLowerCase().contains("linux") || System.getProperty("os.name").toLowerCase().contains("mac"))
+        {
+            Neodymium.configuration().setProperty("neodymium.lighthouse.binaryPath", "echo {\"categories\": {\"performance\": {\"score\": 0.5}, \"accessibility\": {\"score\": 0.5}, \"best-practices\": {\"score\": 0.5}, \"seo\": {\"score\": 0.5}}} > target/lighthouseUtilsReport.report.json");
+            builder = new ProcessBuilder("fabricatedHtml > target/lighthouseUtilsReport.report.html");


this needs to be changed to new ProcessBuilder("sh", "-c", "echo fabricatedHtml > target/lighthouseUtilsReport.report.html");

wurzelkuchen · 2024-11-21T13:21:56Z

src/main/java/com/xceptance/neodymium/util/LighthouseUtils.java

+        }
+        else if (System.getProperty("os.name").toLowerCase().contains("linux") || System.getProperty("os.name").toLowerCase().contains("mac"))
+        {
+            builder = new ProcessBuilder(Neodymium.configuration().lighthouseBinaryPath(), "--chrome-flags=\"--ignore-certificate-errors\"", URL, "--port=" + Neodymium.getRemoteDebuggingPort(), "--preset=desktop", "--output=json", "--output=html", "--output-path=target/" + reportName + ".json");


for linux please change this to call the shell by using new ProcessBuilder("sh", "-c", Neodymium.config....

wurzelkuchen · 2024-11-21T13:22:17Z

src/main/java/com/xceptance/neodymium/util/LighthouseUtils.java

+            }
+            else if (System.getProperty("os.name").toLowerCase().contains("linux") || System.getProperty("os.name").toLowerCase().contains("mac"))
+            {
+                new ProcessBuilder(Neodymium.configuration().lighthouseBinaryPath(), "--version").start();


for linux please change this to call the shell by using new ProcessBuilder("sh", "-c", Neodymium.config....

kqmpetenz

only the documentation of the audits property and the link to the documentation in the neo config (when finished)

config/neodymium.properties

src/main/java/com/xceptance/neodymium/util/LighthouseUtils.java

kqmpetenz · 2024-11-22T09:26:13Z

src/main/java/com/xceptance/neodymium/util/LighthouseUtils.java

+     *            The name of the Lighthouse report attachment in the Allure report
+     * @throws Exception
+     */
+    public static void createLightHouseReport(WebDriver driver, String reportName) throws Exception 


kqmpetenz · 2024-11-22T09:33:20Z

src/main/java/com/xceptance/neodymium/util/LighthouseUtils.java

+        }
+        else if (System.getProperty("os.name").toLowerCase().contains("linux") || System.getProperty("os.name").toLowerCase().contains("mac"))
+        {
+            builder = new ProcessBuilder(Neodymium.configuration().lighthouseBinaryPath(), "--chrome-flags=\"--ignore-certificate-errors\"", URL, "--port=" + Neodymium.getRemoteDebuggingPort(), "--preset=desktop", "--output=json", "--output=html", "--output-path=target/" + reportName + ".json");


kqmpetenz · 2024-11-22T09:35:26Z

src/main/java/com/xceptance/neodymium/util/LighthouseUtils.java

+        String assertAuditsString = Neodymium.configuration().lighthouseAssertAudits();
+        List<String> errorAudits = new ArrayList<>();
+
+        if (!assertAuditsString.isEmpty()) 


src/main/java/com/xceptance/neodymium/util/LighthouseUtils.java

kqmpetenz · 2024-11-22T09:49:00Z

src/main/java/com/xceptance/neodymium/util/LighthouseUtils.java

+            }
+            if (errorAudits.size() > 0) 
+            {
+                throw new Exception("the following audits " + errorAudits + " contain errors that need to be fixed, please look into the corresponding html for further information");


kqmpetenz · 2024-11-22T09:50:30Z

src/test/java/com/xceptance/neodymium/util/LighthouseUtilsTest.java

+        else if (System.getProperty("os.name").toLowerCase().contains("linux") || System.getProperty("os.name").toLowerCase().contains("mac"))
+        {
+            Neodymium.configuration().setProperty("neodymium.lighthouse.binaryPath", "echo {\"categories\": {\"performance\": {\"score\": 0.5}, \"accessibility\": {\"score\": 0.5}, \"best-practices\": {\"score\": 0.5}, \"seo\": {\"score\": 0.5}}} > target/lighthouseUtilsReport.report.json");
+            builder = new ProcessBuilder("fabricatedHtml > target/lighthouseUtilsReport.report.html");


wurzelkuchen · 2024-11-29T05:18:47Z

config/neodymium.properties

+# The actual value for the seo score varies alot, so consider using a lower threshold to avoid a lot of false alerts
+neodymium.lighthouse.assert.thresholdScore.seo = 0.5
+
+# To be able to validate Lighthouse report audits, wo use internal json id's from the report itself


wurzelkuchen · 2024-11-29T07:50:21Z

src/main/java/com/xceptance/neodymium/util/LighthouseUtils.java

+            }
+            else if (System.getProperty("os.name").toLowerCase().contains("linux") || System.getProperty("os.name").toLowerCase().contains("mac"))
+            {
+                new ProcessBuilder("sh", "-c", Neodymium.configuration().lighthouseBinaryPath(), "--version").start();


Several things here:

We need to check for the error code as we did for windows

the "sh" "c" parameter need to be removed to make it work (on my system) and the mac I tested on, recheck on another linux mashine is WIP.

since this seems to be the main cause of pain and other users will have other setups, we should improve our error handling here. We should add the proccess output to the error message in cases something goes wrong.

since we have for cases (these two and the two where we actually create the report) where we need to start a process with a given set of parameters, check for the error code and collect the logs, we should create a seperate method for this, and just pass all the parameters we would give to the ProcessBuilder

Inside this method we could also add the PATH variable of the process to the error message in a failure case.

Once we have a seperate method We can do some double safety and try each process with "sh" "-c" and if it fails we can do it without. that way we have an additional safety net for strange setups and will make sure our unit test mokup works as intendet ;)

src/main/java/com/xceptance/neodymium/util/LighthouseUtils.java

Jonas Hoyer added 9 commits October 16, 2024 10:47

added LighthouseUtils

8ade00f

added lighthouse report handling

be17fa3

added lighthouse binary validation and report creation for linux

c7f6f4c

added lighthouse binary validation and report creation for mac

35b2d6a

#302 added javadoc

aa4460a

#302 added usage of free port for chrome

9765b9f

added unit tests

5d0daac

corrected grammar mistake

def4f49

changed back version in pom.xml

85358f2

kqmpetenz requested review from wurzelkuchen and oomelianchuk November 18, 2024 08:23

kqmpetenz linked an issue Nov 18, 2024 that may be closed by this pull request

Integrate Google Lighthouse into Neodyimum #302

Open

7 tasks

Jonas Hoyer added 4 commits November 20, 2024 14:53

#302 - added lighthouse audits validation

4a81633

#302 - added javadoc for validate audits method

8802418

Merge branch 'develop' into support-lighthouse-reports

e3a85c9

#302 - updated audits condition

549219d

wurzelkuchen requested changes Nov 21, 2024

View reviewed changes

kqmpetenz commented Nov 22, 2024

View reviewed changes

Jonas Hoyer added 3 commits November 22, 2024 10:54

#302 - code review changes

1d0f013

#302 - added missing parameter to javadoc

30e10d1

#302 - added link to docu in new audit property

ef9429b

wurzelkuchen requested changes Nov 29, 2024

View reviewed changes

Jonas Hoyer and others added 2 commits December 2, 2024 12:40

#302 - added requested changes in review

fe414b8

[#302] some refactoring + improved error message

5f5228e

wurzelkuchen approved these changes Dec 3, 2024

View reviewed changes

Merge branch 'develop' into support-lighthouse-reports

55167d4

wurzelkuchen merged commit ff72b1c into develop Dec 3, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support lighthouse reports #310

Support lighthouse reports #310

kqmpetenz commented Nov 18, 2024

wurzelkuchen left a comment

wurzelkuchen Nov 21, 2024

kqmpetenz Nov 22, 2024

wurzelkuchen Nov 21, 2024

kqmpetenz Nov 22, 2024

wurzelkuchen Nov 21, 2024

kqmpetenz Nov 22, 2024

wurzelkuchen Nov 21, 2024

kqmpetenz Nov 22, 2024

wurzelkuchen Nov 21, 2024

kqmpetenz Nov 22, 2024

wurzelkuchen Nov 21, 2024

kqmpetenz Nov 22, 2024

kqmpetenz left a comment

kqmpetenz Nov 22, 2024

kqmpetenz Nov 22, 2024

kqmpetenz Nov 22, 2024

kqmpetenz Nov 22, 2024

kqmpetenz Nov 22, 2024

wurzelkuchen Nov 29, 2024

wurzelkuchen Nov 29, 2024

wurzelkuchen Nov 29, 2024

wurzelkuchen Nov 29, 2024

Support lighthouse reports #310

Support lighthouse reports #310

Conversation

kqmpetenz commented Nov 18, 2024

wurzelkuchen left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kqmpetenz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment