[BUG] Regression: missing snapshot attachment

[edumserrano]

Hi @cenfun,

Something has happened and the attachments aren't displaying properly, at least for snapshots. From version 1.6.31 until latest what I get when a test fails is the following images:

When I downgrade to version 1.6.30 the attachments start showing:

I've re-used a demo playwright project I had to show the bug. You can go to edumserrano/playwright-bug-vrt-diff/tree/monocart-reporter and checkout the monocart-reporter branch.

Then from the root of the repo just do:

• npm install
• npm run test

This will give you the result above where no attachments are shown. Any ideas what might be going wrong? I don't understand how this was working and suddenly stopped.

[edumserrano]

This definitely seems like a regression since version 1.6.31 forward.

I've been able to narrow the problem down a bit further. It seems that the snapshot attachments aren't being displayed unless there are other attachments added after them. Most of my tests are running with an auto fixture to attach code coverage on tests and that's why I didn't notice this sooner.

I've updated the tests in the edumserrano/playwright-bug-vrt-diff/tree/monocart-reporter on the monocart-reporter branch to show the problem.

The test this shows attachments well works well and produces the snapshot and code coverage attachments but only because the code coverage attachment is added after the snapshot attachment.

The test this only shows code coverage only shows code coverage. Note that here the code coverage is attached to the test before the snapshots.

The test this does not show any attachment doesn't show any attachment even though it has a await expect(page).toHaveScreenshot(); which fails and therefore the snapshots comparison should be visible on the attachments section.

This was actually hard for me to figure out but hopefully the above info helps you quickly narrow down the problem and fix it. Let me know if you need more details @cenfun

[cenfun]

ok, it should be a bug since image diff feature adding. I will check it.

[edumserrano]

I also found what I believe is a related bug. See image below, note that the test failed, there are only 3 images generated from the test failure in the test results folder but the tabs for snapshot attachments are duplicated.

I believe (not sure) the way to reproduce this is to have the test fail and have a retries option in your playwright config:

export default defineConfig({
   ...
    retries: 2,
...
});

In this scenario I had 2 retries and I think what is happening is that we would get the tabs 3x, as in 3xDiff, 3xActual and 3xExpected, but we got the tabs 2x because one of the original bug reported here which is when it fails with no retries there isn't even any attachment showing.

So overall this bug report is probably:

1. attachments are missing when there are no retries
2. when there are retries the image tabs are duplicated per retry.

[zhukovsv]

Hi, everyone,

Possibly I have the same issue.

Config info:

1. "nodeJS": v18.16.0
2. "@playwright/test": "1.35.0"
3. "monocart-reporter": "^1.6.33"
4. monocart config:
   

Bug Info:

• Use attached generated monocart report for more bug info:
  

• Use attached "Open link in another tab." screen:
  

Thanks.

[cenfun]

@edumserrano it should be fixed in monocart-reporter@1.6.34

[edumserrano]

@zhukovsv I think your issue is different. It looks like the problem described here #34.

If you read #34, you probably have the same issue in that you can't configure the path for your reporter like you did. Try to change the outputFile to ./test-results/index.html.

In short, the index.html for your monocart-reporter must be at the same directory as your test results directory.

[edumserrano]

@cenfun it's almost fixed. I've pushed a small commit in in the edumserrano/playwright-bug-vrt-diff/tree/monocart-reporter on the monocart-reporter branch to show the problem that still occurs with retries.

I've updated to version 1.6.4 and set retries to 2 on playwright's config:

export default defineConfig({
   ...
    retries: 2,
...
});

and when you run npm run test you will see that the test this does not show any attachment shows multiple tabs:

Note
The test names now don't make sense, ignore the names. After updating to 1.6.34 the attachments for snapshots always show. The only problem that remains is that with retries the tabs seem to sometimes be repeated.

[cenfun]

it should be fixed, see version 1.6.35

[zhukovsv]

Hi @cenfun,

Config info:

• "nodeJS": v18.16.0
• "@playwright/test": "1.35.0"
• "monocart-reporter": "^1.6.35"
• monocart config:
  

If test file contains test suites:

1. Diff tab is OK.
   

2. Actual tab does not show exists snapshot.

3. Expected tab does not show exists snapshot.
   

BUT if test file does not contains test suites:

1. Diff tab is OK.
   

2. Actual tab is OK.

3. Expected tab is OK.
   

Bug: If test file contains test suite or parent + child test suites Actual and Expected snapshots are not shown.

Could you look into this issue?

[cenfun]

@zhukovsv Are there any special characters in the suite title?
can you share the full image path?

[zhukovsv]

@cenfun

test.describe('DT BASE: ', () => {

const testData: Array<[number, string]> = [
[1, 'DC (EU)'],
[2, 'DC (ASIA)'],
[0, 'DC (US)']
];

for (const [testCaseId, toolTitle] of testData) {
test(${testCaseId}@${toolTitle} page should be opened., async () => {

page = app.setPageByName(toolTitle, userCredential);

await page.get();

const pageTitle = await page.getTitle();
await expect(page.page).toHaveScreenshot();

});
}
}); // describe

Hope it will help.

[cenfun]

it works well with a mock page.

const { test, expect } = require('@playwright/test');

test.describe('DT BASE: ', () => {

    const testData = [
        [1, 'DC (EU)'],
        [2, 'DC (ASIA)'],
        [0, 'DC (US)']
    ];

    for (const [testCaseId, toolTitle] of testData) {
        test(`${testCaseId}@${toolTitle} page should be opened.`, async ({ page }) => {
            await page.setContent('<h3>Title</h3>');
            await expect(page).toHaveScreenshot();
        });
    }
});

[zhukovsv]

@cenfun
Just run your example:

1. with await page.setContent('

   Title

   '); to get snapshots
2. with await page.setContent('

   Title1

   '); to get failed snapshots.

Everything is fine.

Is it possible that the problem in path length?

file:///D:/_AG/VSTS_Tests/pw-dxxxx-txxxxxxx-tests-dev/e2e/test-results/p-w-report/a-g-project-tests-dxxx-txxxxxxx-smoke-01-base--ec2be-BASE-2-Dxxx-Cxxxxxx-ASIA-page-should-be-opened--Chrome-Stable/DxxxTxxxxxxx-BASE-2-Dxxx-Cxxxxxx-ASIA-page-should-be-opened-1-actual.png

I will short names in my scrips and check

[cenfun]

I don't think is path length problem.
Can you open the brower DevTools? (press F12)
There would be some errors in the console if any images failed to load.

[zhukovsv]

@cenfun

I have code like this and it raise the issue:
test.describe('Dxxx Txxxxxxx BASE: ', () => {
const testData: Array<[number, string]> = [
[1, 'Dxxx Cxxxxxx (EU)'],
[2, 'Dxxx Cxxxxxx (ASIA)'],
[0, 'Dxxx Cxxxxxx (US)']
];

for (const [testCaseId, toolTitle] of testData) {
test(${testCaseId}@${toolTitle} page should be opened., async () => {

  page = app.setPageByName(toolTitle, userCredential);

  await page.get();

  const pageTitle = await page.getTitle();
  await expect(page.page).toHaveScreenshot();   
});

}
}); // Main Test Suite

Then I shorten names

[zhukovsv]

@cenfun
I shorten names in tests and received code like this and there is no issue now:

test.describe('DT BASE: ', () => {
const testData: Array<[number, string, string]> = [
[1, 'Dxxx Cxxxxxx (EU)','DC2'],
[2, 'Dxxx Cxxxxxx (ASIA)','DC2'],
[0, 'Dxxx Cxxxxxx (US)','DC2']
];

for (const [testCaseId, toolTitle, shortName] of testData) {
test(${testCaseId}@${shortName} page should be opened., async () => {

page = app.setPageByName(toolTitle, userCredential);

await page.get();

const pageTitle = await page.getTitle();
await expect(page.page).toHaveScreenshot();
});
}
}); // Main Test Suite

Path Length?

Do you still need console logs?

[cenfun]

yes, the error log will tell us why image failed to load.

[zhukovsv]

@cenfun

console-log-error.log

[cenfun]

the error message is net::ERR_FILE_NOT_FOUND
I note that the protocol is file:///D:/_AG/
Could you please try open the report with CLI

npx monocart show-report <your-outputFile-path>

the image url will be over the http protocol, and see if works

[zhukovsv]

@cenfun

npx monocart show-report test-results\multiple-html\index.html

[zhukovsv]

[zhukovsv]

I run tests again with shorten names in tests and open report by test-results\multiple-html\index.html double click:

test.describe('DT BASE: ', () => {
const testData: Array<[number, string, string]> = [
[1, 'Dxxx Cxxxxxx (EU)','DC2'],
[2, 'Dxxx Cxxxxxx (ASIA)','DC2'],
[0, 'Dxxx Cxxxxxx (US)','DC2']
];

for (const [testCaseId, toolTitle, shortName] of testData) {
test(${testCaseId}@${shortName} page should be opened., async () => {

page = app.setPageByName(toolTitle, userCredential);

await page.get();

const pageTitle = await page.getTitle();
await expect(page.page).toHaveScreenshot();
});
}
}); // Main Test Suite

Everything is fine. Report is correct:

But if I run it by using CLI: npx monocart show-report test-results\multiple-html\index.html

• All snapshots are not shown ((((
  

[cenfun]

I note that the output dir should be test-results/multiple-html in your config

but the image url shows test-results/p-w-report/ in logs you provided
seems the image url is wrong.

[zhukovsv]

I made this because of conflict PW and HTML report. I will made changes as you say and remove html report. 2 minutes please

[zhukovsv]

Short Names is correct in both cases:

1. CLI
2. Double click

I am runing long names. Please wait 2-3 minutes

[zhukovsv]

Long names:

1. CLI - ALL is OK.
   

2. Double click is not OK!
   

[cenfun]

Good to know short names will works, but it's unreasonable, because your url should not great than 2k, chrome should support 32k length url, see infomation about max url length here
https://stackoverflow.com/questions/417142/what-is-the-maximum-length-of-a-url-in-different-browsers
and here
https://www.sistrix.com/ask-sistrix/technical-seo/site-structure/url-length-how-long-can-a-url-be

[zhukovsv]

@cenfun

My reports are automatically saved to a network share at night. I usually just launch them in the morning and review them.

Do you have any idea how to fix a report launched by double-clicking?

Sorry I need to go. It not urgent. So I can help you with testing tomorrow.

PS. Thanks a lot for your report and your time.

[cenfun]

ok im going to sleep. one more option, can you share the folder test-results with all files in (you can zip it)

[zhukovsv]

I will do it 2-3 hours later.

[zhukovsv]

@cenfun

Please find prepared test-results folder:

• test-results.zip

[edumserrano]

@cenfun I can confirm the issues I raised are now fixed. Didn't test on version 1.6.35 but on 1.6.36 all is ok, no more duplicated tabs. 🥳

[cenfun]

@zhukovsv
Test and it works on my PC (both Chrome v112 and MS Edge v113), there is no error on image diff, actual and exprected.
I have to close this issue now since it's about another question, feel free to create a new issue if you need more discussion. Thanks