Skip to content

Commit

Permalink
Add server-side pagination to cluster explorer lists (#11672)
Browse files Browse the repository at this point in the history 
* cluster list now uses resource table

* New PaginatedResourceTable

* WIP

* add context to pag setting

* tidying up

* Add server-side pagination to cluster explorer events and general events list

* Servier-side pagination for catalog apps, cluster repo and operation

* Servier-side pagination for Service --> HPA, inmgress, services

* Server-side pagination for home page clusters list and side bar clusters

- Functional Changes
  - SSP now works after vue3 bump
  - Home Page Clusters list now uses server-side pagination
  - Side Bar clusters list now uses server-side pagination
  - Wire in now supported sorting / filtering by id and name used for table columns
  - Allow pagination to be enabled given a specific context
  - Call findPage without persisting to store

- New Pagination Tools
  - PaginatedResourceTable - Convenience Component, wraps ResourceTable with pagination specific props
  - PaginationWrapper - Convenience class to handle requests for resources and updates to them (avoiding store)

- Regressions
  - Side Nav menu ready state was `mgmtCluster.isReady && !pCluster?.hasError`, now ???

* fixes after merge. apply PagResTable to node list, vue3 fixes

* - Fix issue where list components containing PaginatedResourceTables would incorrectly fetch all resources anyway
- convert reminaing storage lists

* policy section

* fix async button in manual refresh mode

* fgh

* aaaaa

* Server-side pagination for home page clusters list and side bar clusters

- Functional Changes
  - SSP now works after vue3 bump
  - Home Page Clusters list now uses server-side pagination
  - Side Bar clusters list now uses server-side pagination
  - Wire in now supported sorting / filtering by id and name used for table columns
  - Allow pagination to be enabled given a specific context
  - Call findPage without persisting to store

- New Pagination Tools
  - PaginatedResourceTable - Convenience Component, wraps ResourceTable with pagination specific props
  - PaginationWrapper - Convenience class to handle requests for resources and updates to them (avoiding store)

- Regressions
  - Side Nav menu ready state was `mgmtCluster.isReady && !pCluster?.hasError`, now ???

* Iteration

Note - prov clusters is broken (only fetches local) due to blocking pr. breals
- notPinned list

* Fix dupe inStore
- remove from resource list, put in resource-fetch (used also by pag res table)

* Fix dupe inStore
- remove from resource list, put in resource-fetch (used also by pag res table)

* Two fixes
- changes namespaces kicked of side nav cluster requests (thought pinnedIds changed)
- fix generic lists re-fetching given ns filter changes (they don't have namespaced arg)

* fixes / updates

* testing/tidying

* WIP - Rest of workload lists

* finish off workloads

* tidying up

* fix non-generic list filtering by namespace

* remove comment, backport fix

* test fixes

* E2E: Ensure we wait for cluster entries to exist before clicking on them

* backport fix for local/api filtering

* Remove debug code

* Changes after review

* e2e fixes / debugging

* More e2e fixes

* More e2e fixes

* More e2e fixes

* Fix generic pages that filter on pagination

* Attempt to fix flaky vai test

* Fix after merge from master

* Updates following new indexed files

* Fix lint and test

* Changes given real cluster tests
- general fixes
- correct issue were sorting prov clusters on mgmt cluster props (issue in master as well...)
- bit the bullet, we now don't fetch all mgmt clusters on dashboard visit.
  - there could be knock on affects, but we'd need to remove it sometime in 2.11....

* Fix issues with diplaying rke1 data in home page
- includes #12881

* Fix unit tests

* remove invalid sort/filter type

* Running through new indexed fields

* fix unit tests

* lint fixes

* Fixes after small review

* lint

* Test for check-plugin-gates pkg build from shell

* e2e fixes

* Testing new indexed fields with dev image

* tidyuing up imports in hope to fix ts error in check-plugins gate

* Fix failing check-plugin-gates
- caused by shell/scripts/test-plugins-build.sh importing list/catalog.cattle.io.clusterrepo.vue
- the component had been updated to a TS component
- check-plugin build outputs TS errors for a component file imports
- vs code shows no errors for imported file

* tidying up a smidge

* Many tweaks
- Remove final todo's
  - includes fix for service type clusterip/headless overlap
- Removed ununused ENDPOINT column (note ENDPOINT formatter used in other columns)
- Testing freshly added index fields

* Improve location of new place where componentWillFetch is set

* fix lint, comment updated

* Fix failing e2e tests
- fix cluster dashboard events test
- fix and greatly improve flaky events test
- fix hpa test
- functional fixes
  - fix sorting/filtering events by object type
  - fix hpa columns
  • Loading branch information
@richard-cox
richard-cox authored Jan 7, 2025
1 parent 0e51cd0 commit f980283
Show file tree
Hide file tree
Showing 45 changed files with 996 additions and 479 deletions.
2 changes: 1 addition & 1 deletion cypress/e2e/blueprints/nav/fake-cluster.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2474,7 +2474,7 @@ function generateFakeNavClusterData(provClusterId = 'some-prov-cluster-id', mgmt
}

export function generateFakeClusterDataAndIntercepts(fakeProvClusterId = 'some-prov-cluster-id', fakeMgmtClusterId = 'some-mgmt-cluster-id', addEditClusterCapabilities = false): {} {
const longClusterDescription = 'this-is-some-really-really-really-really-really-really-long-decription';
const longClusterDescription = 'this-is-some-really-really-really-really-really-really-long-description';
const fakeNavClusterData = generateFakeNavClusterData(fakeProvClusterId, fakeMgmtClusterId, addEditClusterCapabilities);

// add cluster to fleet clusters for testing https://github.com/rancher/dashboard/issues/9984
Expand Down
2 changes: 1 addition & 1 deletion cypress/e2e/po/pages/explorer/cluster-dashboard.po.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ export default class ClusterDashboardPagePo extends PagePo {
}

fullEventsLink() {
return cy.get('.events-table-link').contains('Full events list');
return cy.get('[data-testid="events-link"]').contains('Full events list');
}

fullSecretsList() {
Expand Down
2 changes: 1 addition & 1 deletion cypress/e2e/tests/navigation/side-nav/main-side-menu.spec.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ import PagePo from '@/cypress/e2e/po/pages/page.po';
import ProductNavPo from '@/cypress/e2e/po/side-bars/product-side-nav.po';
import { generateFakeClusterDataAndIntercepts } from '@/cypress/e2e/blueprints/nav/fake-cluster';

const longClusterDescription = 'this-is-some-really-really-really-really-really-really-long-decription';
const longClusterDescription = 'this-is-some-really-really-really-really-really-really-long-description';
const fakeProvClusterId = 'some-fake-cluster-id';
const fakeMgmtClusterId = 'some-fake-mgmt-id';

Expand Down
57 changes: 28 additions & 29 deletions cypress/e2e/tests/pages/explorer/dashboard/cluster-dashboard.spec.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ describe('Cluster Dashboard', { testIsolation: 'off', tags: ['@explorer', '@admi

clusterDashboard.waitForPage(undefined, 'cluster-events');

// check if burguer menu nav is highlighted correctly for local cluster
// check if burger menu nav is highlighted correctly for local cluster
BurgerMenuPo.checkIfClusterMenuLinkIsHighlighted('local');
});

Expand Down Expand Up @@ -257,48 +257,47 @@ describe('Cluster Dashboard', { testIsolation: 'off', tags: ['@explorer', '@admi
});

it('can view events table empty if no events', { tags: ['@vai', '@adminUser'] }, () => {
cy.visit(clusterDashboard.urlPath(), {
onBeforeLoad(win) {
cy.stub(win.console, 'error').as('consoleError');
cy.stub(win.console, 'warn').as('consoleWarn');
},
});

eventsNoDataset();
clusterDashboard.goTo();

cy.get('@consoleError').should('not.be.called'); // See error lot
cy.get('@consoleWarn').should('not.be.called'); // See warning log (there will be some....)

cy.wait('@eventsNoData');
clusterDashboard.waitForPage(undefined, 'cluster-events');

clusterDashboard.eventsList().resourceTable().sortableTable().checkRowCount(true, 1);

const expectedHeaders = ['Reason', 'Object', 'Message', 'Name', 'Date'];
let expectedHeaders = ['Reason', 'Object', 'Message', 'Name', 'Date'];

clusterDashboard.eventsList().resourceTable().sortableTable().tableHeaderRow()
.within('.table-header-container .content')
.each((el, i) => {
expect(el.text().trim()).to.eq(expectedHeaders[i]);
});
cy.isVaiCacheEnabled().then((isVaiCacheEnabled) => {
if (isVaiCacheEnabled) {
expectedHeaders = ['Reason', 'Object', 'Message', 'Name', 'First Seen', 'Last Seen', 'Count'];
}

clusterDashboard.fullEventsLink().click();
cy.wait('@eventsNoData');
const events = new EventsPagePo('local');
clusterDashboard.eventsList().resourceTable().sortableTable().tableHeaderRow()
.self()
.scrollIntoView();
clusterDashboard.eventsList().resourceTable().sortableTable().tableHeaderRow()
.within('.table-header-container .content')
.each((el, i) => {
expect(el.text().trim()).to.eq(expectedHeaders[i]);
});

events.waitForPage();
clusterDashboard.fullEventsLink().click();
cy.wait('@eventsNoData');
const events = new EventsPagePo('local');

events.eventslist().resourceTable().sortableTable().checkRowCount(true, 1);
events.waitForPage();

const expectedFullHeaders = ['State', 'Last Seen', 'Type', 'Reason', 'Object',
'Subobject', 'Source', 'Message', 'First Seen', 'Count', 'Name', 'Namespace'];
events.eventslist().resourceTable().sortableTable().checkRowCount(true, 1);

events.eventslist().resourceTable().sortableTable().tableHeaderRow()
.within('.table-header-container .content')
.each((el, i) => {
expect(el.text().trim()).to.eq(expectedFullHeaders[i]);
});
const expectedFullHeaders = ['State', 'Last Seen', 'Type', 'Reason', 'Object',
'Subobject', 'Source', 'Message', 'First Seen', 'Count', 'Name', 'Namespace'];

events.eventslist().resourceTable().sortableTable().tableHeaderRow()
.within('.table-header-container .content')
.each((el, i) => {
expect(el.text().trim()).to.eq(expectedFullHeaders[i]);
});
});
});

describe('Cluster dashboard with limited permissions', () => {
Expand Down
188 changes: 131 additions & 57 deletions cypress/e2e/tests/pages/explorer/dashboard/events.spec.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,27 @@ import SortableTablePo from '@/cypress/e2e/po/components/sortable-table.po';
const cluster = 'local';
const clusterDashboard = new ClusterDashboardPagePo(cluster);
const events = new EventsPagePo(cluster);
const pageSize = 10;
// Should be enough to create at least 3 pages of events
const podCount = 15;

const countHelper = {
setupCount: (vaiCacheEnabled: boolean, initialCount: number) => {
if (vaiCacheEnabled) {
cy.intercept('GET', '/v1/events?*').as('getCount');
} else {
cy.wrap(initialCount).as('count');
}
},
handleCount: (vaiCacheEnabled) => {
if (vaiCacheEnabled) {
cy.wait('@getCount').then((interception) => {
cy.wrap(interception.response.body.count).as('count');
});
}
},
getCount: () => cy.get('@count').then((count) => count as any as number),
};

describe('Events', { testIsolation: 'off', tags: ['@explorer', '@adminUser'] }, () => {
before(() => {
Expand All @@ -20,7 +41,12 @@ describe('Events', { testIsolation: 'off', tags: ['@explorer', '@adminUser'] },
let nsName2: string;

before('set up', () => {
cy.updateNamespaceFilter(cluster, 'none', '{\"local\":[]}');
cy.tableRowsPerPageAndPreferences(pageSize, {
clusterName: cluster,
groupBy: 'none',
namespaceFilter: '{\"local\":[]}',
allNamespaces: 'true',
});

cy.createE2EResourceName('ns1').then((ns1) => {
nsName1 = ns1;
Expand All @@ -30,7 +56,7 @@ describe('Events', { testIsolation: 'off', tags: ['@explorer', '@adminUser'] },
// create pods
let i = 0;

while (i < 125) {
while (i < podCount) {
const podName = Cypress._.uniqueId(Date.now().toString());

cy.createPod(nsName1, podName, 'nginx:latest', false, { createNameOptions: { prefixContext: true } }).then((resp) => {
Expand All @@ -52,6 +78,9 @@ describe('Events', { testIsolation: 'off', tags: ['@explorer', '@adminUser'] },
uniquePod = resp.body.metadata.name;
});
});

// I'm loathed to do this, but the events created from the pods need to settle before we start
cy.wait(20000); // eslint-disable-line cypress/no-unnecessary-waiting
});

it('pagination is visible and user is able to navigate through events data', () => {
Expand All @@ -61,73 +90,113 @@ describe('Events', { testIsolation: 'off', tags: ['@explorer', '@adminUser'] },
EventsPagePo.navTo();
events.waitForPage();

cy.getRancherResource('v1', 'events').then((resp: Cypress.Response<any>) => {
// Why 500? there's a hardcoded figure to stops ui from storing more than 500 events ...
const count = resp.body.count < 500 ? resp.body.count : 500;

// Test break down if less than 400...
expect(count).to.be.greaterThan(400);
let vaiCacheEnabled = false;

// pagination is visible
events.sortableTable().pagination().checkVisible();
cy.isVaiCacheEnabled()
.then((isVaiCacheEnabled) => {
vaiCacheEnabled = isVaiCacheEnabled;

const loadingPo = new LoadingPo('.title .resource-loading-indicator');
return cy.getRancherResource('v1', 'events');
})
.then((resp: Cypress.Response<any>) => {
let initialCount = resp.body.count;

loadingPo.checkNotExists();
if (!vaiCacheEnabled && resp.body.count > 500) {
// Why 500? there's a hardcoded figure to stops ui from storing more than 500 events ...
initialCount = 500;
}

// basic checks on navigation buttons
events.sortableTable().pagination().beginningButton().isDisabled();
events.sortableTable().pagination().leftButton().isDisabled();
events.sortableTable().pagination().rightButton().isEnabled();
events.sortableTable().pagination().endButton().isEnabled();
// Test break down if less than 3 pages...
expect(initialCount).to.be.greaterThan(3 * pageSize);

// check text before navigation
events.sortableTable().pagination().paginationText().then((el) => {
expect(el.trim()).to.eq(`1 - 100 of ${ count } Events`);
});

// navigate to next page - right button
events.sortableTable().pagination().rightButton().click();
// pagination is visible
events.sortableTable().pagination().checkVisible();

// check text and buttons after navigation
events.sortableTable().pagination().paginationText().then((el) => {
expect(el.trim()).to.eq(`101 - 200 of ${ count } Events`);
});
events.sortableTable().pagination().beginningButton().isEnabled();
events.sortableTable().pagination().leftButton().isEnabled();
const loadingPo = new LoadingPo('.title .resource-loading-indicator');

// navigate to first page - left button
events.sortableTable().pagination().leftButton().click();
loadingPo.checkNotExists();

// check text and buttons after navigation
events.sortableTable().pagination().paginationText().then((el) => {
expect(el.trim()).to.eq(`1 - 100 of ${ count } Events`);
});
events.sortableTable().pagination().beginningButton().isDisabled();
events.sortableTable().pagination().leftButton().isDisabled();
// basic checks on navigation buttons
events.sortableTable().pagination().beginningButton().isDisabled();
events.sortableTable().pagination().leftButton().isDisabled();
events.sortableTable().pagination().rightButton().isEnabled();
events.sortableTable().pagination().endButton().isEnabled();

// navigate to last page - end button
events.sortableTable().pagination().endButton().scrollIntoView()
.click();
// check text before navigation
events.sortableTable().pagination().self().scrollIntoView();
events.sortableTable().pagination().paginationText().then((el) => {
expect(el.trim()).to.eq(`1 - ${ pageSize } of ${ initialCount } Events`);
});

// check row count on last page
events.sortableTable().checkRowCount(false, 100);
// navigate to next page - right button
countHelper.setupCount(vaiCacheEnabled, initialCount);
events.sortableTable().pagination().rightButton().click();
countHelper.handleCount(vaiCacheEnabled);

// check text and buttons after navigation
events.sortableTable().pagination().self().scrollIntoView();
countHelper.getCount().then((count) => {
return events.sortableTable().pagination().paginationText().then((el) => {
expect(el.trim()).to.eq(`${ pageSize + 1 } - ${ 2 * pageSize } of ${ count } Events`);
});
});
events.sortableTable().pagination().beginningButton().isEnabled();
events.sortableTable().pagination().leftButton().isEnabled();

// navigate to first page - left button
countHelper.setupCount(vaiCacheEnabled, initialCount);
events.sortableTable().pagination().leftButton().click();
countHelper.handleCount(vaiCacheEnabled);

// check text and buttons after navigation
events.sortableTable().pagination().self().scrollIntoView();
countHelper.getCount().then((count) => {
return events.sortableTable().pagination().paginationText().then((el) => {
expect(el.trim()).to.eq(`1 - ${ pageSize } of ${ count } Events`);
});
});

// check text after navigation
events.sortableTable().pagination().paginationText().then((el) => {
expect(el.trim()).to.eq(`401 - ${ count } of ${ count } Events`);
});
events.sortableTable().pagination().beginningButton().isDisabled();
events.sortableTable().pagination().leftButton().isDisabled();

// navigate to last page - end button
countHelper.setupCount(vaiCacheEnabled, initialCount);
events.sortableTable().pagination().endButton().scrollIntoView()
.click();
countHelper.handleCount(vaiCacheEnabled);

// check text after navigation
events.sortableTable().pagination().self().scrollIntoView();
countHelper.getCount().then((count) => {
return events.sortableTable().pagination().paginationText().then((el) => {
let pages = Math.floor(count / pageSize);

if (count % pageSize === 0) {
pages--;
}
const from = (pages * pageSize) + 1;
const to = count;

expect(el.trim()).to.eq(`${ from } - ${ to } of ${ to } Events`);
});
});

// navigate to first page - beginning button
events.sortableTable().pagination().beginningButton().click();
// navigate to first page - beginning button
countHelper.setupCount(vaiCacheEnabled, initialCount);
events.sortableTable().pagination().beginningButton().click();
countHelper.handleCount(vaiCacheEnabled);

// check text and buttons after navigation
events.sortableTable().pagination().self().scrollIntoView();
countHelper.getCount().then((count) => {
events.sortableTable().pagination().paginationText().then((el) => {
expect(el.trim()).to.eq(`1 - ${ pageSize } of ${ count } Events`);
});
});

// check text and buttons after navigation
events.sortableTable().pagination().paginationText().then((el) => {
expect(el.trim()).to.eq(`1 - 100 of ${ count } Events`);
events.sortableTable().pagination().beginningButton().isDisabled();
events.sortableTable().pagination().leftButton().isDisabled();
});
events.sortableTable().pagination().beginningButton().isDisabled();
events.sortableTable().pagination().leftButton().isDisabled();
});
});

it('filter events', () => {
Expand All @@ -138,7 +207,7 @@ describe('Events', { testIsolation: 'off', tags: ['@explorer', '@adminUser'] },

events.sortableTable().checkVisible();
events.sortableTable().checkLoadingIndicatorNotVisible();
events.sortableTable().checkRowCount(false, 100);
events.sortableTable().checkRowCount(false, pageSize);

// filter by namespace
events.sortableTable().filter(nsName2);
Expand Down Expand Up @@ -201,7 +270,12 @@ describe('Events', { testIsolation: 'off', tags: ['@explorer', '@adminUser'] },
});

after('clean up', () => {
cy.updateNamespaceFilter(cluster, 'none', '{"local":["all://user"]}');
cy.tableRowsPerPageAndPreferences(100, {
clusterName: cluster,
groupBy: 'none',
namespaceFilter: '{"local":["all://user"]}',
allNamespaces: 'false',
});

// delete namespace (this will also delete all pods in it)
cy.deleteRancherResource('v1', 'namespaces', nsName1);
Expand Down
9 changes: 6 additions & 3 deletions cypress/e2e/tests/pages/explorer/service-discovery/horizontal-pod-autoscalers.spec.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -19,8 +19,11 @@ describe('HorizontalPodAutoscalers', { testIsolation: 'off', tags: ['@explorer',
horizontalPodAutoscalersPage.waitForPage();
cy.wait('@horizontalpodautoscalerNoData');

const expectedHeaders = ['State', 'Name', 'Workload', 'Minimum Replicas', 'Maximum Replicas', 'Current Replicas', 'Age'];
const expectedHeaders = ['State', 'Name', 'Namespace', 'Workload', 'Minimum Replicas', 'Maximum Replicas', 'Current Replicas', 'Age'];

horizontalPodAutoscalersPage.list().resourceTable().sortableTable().tableHeaderRow()
.self()
.scrollIntoView();
horizontalPodAutoscalersPage.list().resourceTable().sortableTable().tableHeaderRow()
.get('.table-header-container .content')
.each((el, i) => {
Expand All @@ -39,7 +42,7 @@ describe('HorizontalPodAutoscalers', { testIsolation: 'off', tags: ['@explorer',
horizontalPodAutoscalersPage.header().selectNamespaceFilterOption('All Namespaces');

// check table headers are visible
const expectedHeaders = ['State', 'Name', 'Workload', 'Minimum Replicas', 'Maximum Replicas', 'Current Replicas', 'Age'];
const expectedHeaders = ['State', 'Name', 'Namespace', 'Workload', 'Minimum Replicas', 'Maximum Replicas', 'Current Replicas', 'Age'];

horizontalPodAutoscalersPage.list().resourceTable().sortableTable().tableHeaderRow()
.get('.table-header-container .content')
Expand All @@ -65,7 +68,7 @@ describe('HorizontalPodAutoscalers', { testIsolation: 'off', tags: ['@explorer',
horizontalPodAutoscalersPage.list().resourceTable().sortableTable().groupByButtons(1)
.click();

// check table headers are visible
// check table headers are visible (minus namespace given we're now grouped by it)
const expectedHeaders = ['State', 'Name', 'Workload', 'Minimum Replicas', 'Maximum Replicas', 'Current Replicas', 'Age'];

horizontalPodAutoscalersPage.list().resourceTable().sortableTable().tableHeaderRow()
Expand Down
Loading

0 comments on commit f980283

Please sign in to comment.