Proposal: Playwright-based Framework for End-to-end Testing

[planger]

Alongside unit and integration tests, it is often very useful to have end-to-end tests, which interact with the full application just like a user would do, e.g. for smoke testing the most relevant user flows. This entails that the end-to-end tests typically interact with the user interface and also only observe the results via the user interface (and not necessarily the internal state), just like a user.

Proposal

To simplify writing end-to-end tests for Theia-based applications, we have developed a Playwright-based page object framework, which introduces an abstraction layer over Theia's user interfaces, encapsulating the details of the user interface interactions, wait conditions, etc., to help keeping end-to-end tests more concise, maintainable, and stable. This framework covers the generic user interface interactions of Theia applications, such as dealing with Theia views or editors (opening, closing, querying dirty state), file explorer, text editor, main menus, context menus, status bar elements, etc. Moreover, the framework is extensible so you can add dedicated page objects for custom Theia components, such as custom views, editors, status bar items, etc.

We'd be happy to contribute the page object framework to Theia!

Example

Below, we show an example of how the framework's page objects TheiaWorkspace, TheiaApp, TheiaTextEditor etc. are used in a test.

Screencast of debugging the example

Code of the example test

it("should undo and redo text changes and correctly update the dirty state", async () => {
  // 1. set up workspace contents and open Theia app
  const ws = new TheiaWorkspace(["tests/resources/sample-files1"]);
  const app = await TheiaApp.load(page, "http://localhost:3000", ws);

  // 2. open Theia text editor
  const sampleTextEditor = await app.openEditor("sample.txt", TheiaTextEditor);

  // 3. make a change and verify contents and dirty
  await sampleTextEditor.replaceLineWithLineNumber("change", 1);
  expect(await sampleTextEditor.textContentOfLineByLineNumber(1)).toBe(
    "change"
  );
  expect(await sampleTextEditor.isDirty()).toBe(true);

  // 4. undo and verify contents and dirty state
  await sampleTextEditor.undo(2);
  expect(await sampleTextEditor.textContentOfLineByLineNumber(1)).toBe(
    "this is just a sample file"
  );
  expect(await sampleTextEditor.isDirty()).toBe(false);

  // 5. undo and verify contents and dirty state
  await sampleTextEditor.redo(2);
  expect(await sampleTextEditor.textContentOfLineByLineNumber(1)).toBe(
    "change"
  );
  expect(await sampleTextEditor.isDirty()).toBe(true);

  // 6. save verify dirty state
  await sampleTextEditor.save();
  expect(await sampleTextEditor.isDirty()).toBe(false);
  await sampleTextEditor.close();

  // 7. reopen editor and verify dirty state
  const reopenedEditor = await app.openEditor("sample.txt", TheiaTextEditor);
  expect(await reopenedEditor.textContentOfLineByLineNumber(1)).toBe("change");

  await reopenedEditor.close();
});

Related issues and links

• consider to use playwright instead of puppeteer #6950
• https://github.com/microsoft/playwright
• https://martinfowler.com/bliki/PageObject.html