Add e2e tests to the build pipeline.

[amass01]

We would like to add the e2e tests to our build pipeline to make sure PRs aren't breaking them.

Note: Currently the e2e tests need running BE instance in order to run.

[lukebp]

We've had some brief internal discussions about this and have decided that mocking the APIs is the best path foward.

Are we already doing this at all in the politeiagui repo?
Do we need to make any decisions about what mocking framework should be used?
Is there anything that is blocking us from creating actionable issues for this work?

When we convert this work into issues, we should not mock the user API routes fow now. The user routes are being replaced on the backend as part of the user layer rewrite and we don't want to do work that will be invalidated in a few months. We can, however, start mocking all of the new API routes: records, comments, and ticketvote.

[victorgcramos]

The Mocked API data is already working on records list, comments and comments votes testing. We can use the cy.intercept command from Cypress to mock the responses, so I don't think we need a new framework for that.

[amass01]

yep, agreed with @victorgcramos.

We will need to adjust the existing e2e test to work with 100% mocked API, currently we mock the
API in specific e2e tests to cover specific cases.

[victorgcramos]

Hey guys, wrote a small tutorial to guide contributors to follow our e2e testing patters. I'm thinking about opening an issue with this description and then open other small issues for the specific e2e tests for each container. This can be used as a coordination issue so we can have a better view of the test coverage.

Feel free to add more cases too 😁

---

After the discussion on #2576, we figured out that the best way to add E2E tests into the build pipeline was to mock API requests so we could run the cypress testing environment detached from Politeia backend.

How to mock API requests

Luckily, Cypress has a good tool for tests: the intercept command. Which we can use to mock the api data and manipulate the responses according to its request data. See the Cypress Documentation to have more information regarding interceptors and aliases.

Let's create a mocked API response for the /ticketvote/v1/inventory call. We can manage the request object req to get the request params, headers, and also manage the response (req.reply). Check the docs to see how to manage intercepted requests and intercepted responses

// ============
// your-test.js
// ============
describe("Ticketvote inventory", () => {
  it("can fetch the ticketvote inventory", () => {
    cy.intercept("/api/ticketvote/v1/inventory", (req) => {
      req.reply({
        statusCode: 200, // set custom response status
        body: {
          vetted: {
            unauthorized: ["abcdefg"], // return some fake token
          },
        },
      });
    });
    // ...
    // assertions regarding the ticketvote inventory
    // ...
  });
});

This will intercept the /records/v1/inventory request and return the mocked inventory in a 200 status response. Even if our server returned an errored request, the cypress environment would ignore the errors and return the response the way you specify.

However, intercept commands can be used on many different tests, hence, they should be more generic. This is why we came up with middlewares, which are nothing but an interceptor + alias. Let's see how can we use the /api/ticketvote/v1/inventory interceptor as a middleware:

1. Create the middleware into our support mock file for the plugin you want.
   • In this case, we call the ticketvote api, so we will add it into support/mock/ticketvote.js:

export const middlewares = {
  // ...
  inventory: () =>
    cy.intercept("/api/ticketvote/v1/inventory", (req) => {
      req.reply({
        statusCode: 200, // set custom response status
        body: {
          vetted: {
            unauthorized: ["abcdefg"], // return some fake token
          },
        },
      });
    }),
  // ...
};

2. Add the bundle into the cypress custom cy.middleware command on support/commands.js:

import { middlewares as ticketVoteMiddlewares } from "./mock/ticketvote";

// ...

Cypress.Commands.add("middleware", (path, ...args) => {
  const mw = get(path)({
    ticketvote: ticketVoteMiddlewares,
    // ... other middlewares
  });
  return mw(...args).as(path); // alias automatically generated
});

// ...

3. Now, when you want to use the middleware on your tests, you can just pass the query selector for the middleware you need:

// ============
// your-test.js
// ============
describe("Ticketvote Inventory", () => {
  it("can fetch the ticketvote inventory", () => {
    cy.middleware("ticketvote.inventory"); // accessing the middleware path using the string pattern.
    // ...
    cy.wait("@ticketvote.inventory").its("status").should("eq", 200); // Success
    // ...
  });

  // ...
});

4. Now you can test pages that call the /ticketvote/v1/inventory without being connected to the ticketvote api.

Pi test cases

In order to write better e2e tests, we should give a priority to the user actions among the platform. Let's see the expected behavior for the following containers:

All checked items are using mocked api data

Proposal

• Details:

  • can render proposal by full or short token on URL (fix: Use short token on record details page. #2536)
  • proposal's full token is copyable (fix: Use short token on record details page. #2536)
  • proposal metadata loaded correctly (fix: Use short token on record details page. #2536)
  • proposal's index.md data (name and description) is displayed correctly (fix: Use short token on record details page. #2536)
  • If some proposal does not exist, display the correct error message
  • check proposal's voting progress and status
  • proposal's author is displayed as a link to author's profile
  • display readable timestamps for published, edited, abandoned or censored proposals
  • list proposal's versions
  • download proposal's timestamps and bundle
  • download proposal's voting timestamps and bundle
  • authors can authorize voting

• List:

  • can render records list (e2e: tests for proposals list #2473 and fix: Under review tab pagination bug. #2552)
  • can fetch "under review" proposals pagination without missing any tokens or status (fix: Under review tab pagination bug. #2552)
  • can switch tabs (e2e: tests for proposals list #2473)
  • can fetch next inventory page when 20 proposals are fetched (e2e: tests for proposals list #2473 and fix: Under review tab pagination bug. #2552)
  • can display the loading indicator while loading proposals (e2e: tests for proposals list #2473)
  • can list legacy proposals (e2e: tests for proposals list #2473)
  • can list unvetted proposals (fix: Make admin proposals page work. #2532)
  • display the empty proposals message when there are no proposals listed.
  • list user proposals

• Create:

  • can create proposals
  • can't create proposals if user hasn't paid the paywall
  • should display warnings for invalid field values and block its submission
  • can't submit proposals without credits

• Edit:

  • only authors can edit their proposals
  • can't edit an authorized voting proposal
  • can't edit an approved proposal
  • can't edit without making any changes

• Diff:

  • display the text differences between two proposal's versions
  • display the attachment differences between two proposal's versions
  • detect changes on proposal's metadata, title and description
  • change comparing versions
  • download timestamp for specific versions
  • can switch between "attachments" and "text" tabs

• RFP:

  • RFP Proposals:

    • create RFP Proposals without regular metadata
    • can't submit RFP proposals with a past deadline
    • display listed submissions for an RFP proposal
    • RFP Proposal author can start the runoff vote
    • must be tagged with the RFP tag

  • RFP Submissions:

    • can only submit RFP submissions to existing and approved RFP proposals
    • display the linked RFP proposal link for some submission
    • new RFP Submissions can't be published when the runoff vote has already started
    • only one submission can be approved

Comments

• Comments:

  • can't comment if hasn't paid the paywall
  • can comment, vote and reply on other's comments if paid the paywall
  • sort comments by top, old and new
  • display comments on flat and cascade mode
  • display author and submitted timestamp
  • comment's permalink URL only renders the full comment's thread
  • can scroll to comments using the scrollToComments URL param
  • comments counter matches comments amount
  • download comments timestamps and bundle
  • regular users can only comment on authorized and unauthorized proposals
  • can see if some coment is censored
  • authors can comment on their unreviewed (unvetted) proposal

• Votes:

  • can upvote/downvote some comment (e2e: tests for proposals list #2473)
  • votes are optimistically updated (e2e: tests for proposals list #2473)
  • prevent multi-clicking on vote button (e2e: tests for proposals list #2473)
  • display error message for failed votes (e2e: tests for proposals list #2473)
  • reset votes count when upvote/downvote action fails (e2e: tests for proposals list #2473)
  • voting on a comment shouldn't change list order (e2e: tests for proposals list #2473)

• Author Updates:

  • can submit author updates
  • only the author is able to publish updates
  • updates are only available for approved proposals
  • can reply author updates
  • can see the progress update title

Admin

• Proposals:

  • start a proposal's voting
  • can't start vote for unauthorized proposals
  • abandon a public proposal
  • can't abandon unvetted proposals
  • mark vetted or unvetted proposals as spam
  • can approve an unreviewed proposal
  • can see unvetted proposals

• Comments:

  • can censor comments
  • can comment on unreviewed proposals

---

[lukebp]

Very nice. This is looking good.

Maybe the tutorial should go in the politeiagui/docs/?

I'd like you to be in charge of managing the completion of this work. The way I would do it would be to open up issues for the uncompleted list items, grouped however you think is best. I wouldn't open more than 2-3 issues at a time, where each issue is a few days worth of work.

[xaur]

A reusable list of test cases is a great asset not only for writing e2e tests, but also for manual testing. More automation is good but some amout of manual testing will always be around and having a checklist to run through saves time. Consider adding the test cases to docs, GitHub Discussions are not as visible.

[victorgcramos]

Update:

we are now able to execute e2e tests on github ci using the mock structure. Here's a draft that illustrates the e2e execution on the build pipeline for the politeia app:

https://github.com/victorgcramos/politeiagui/actions/runs/2792384449

However, e2e tests are only available for the new plugin-architecture, due to the lack of user layer implementation. The legacy app (which is currently on production) does not support this service on its build pipeline.