New PKI UI may saturate memory

[sereinity]

Describe the bug
Having a lot of certificates in the PKI (~95600 certificates), using hold UI it works and we can navigate through pages.
But on the newer UI listing certificates takes many Go of RAM (until I kill it), doing the same with less certificates shows

To Reproduce
Steps to reproduce the behavior:

1. Create a PKI and setup it
2. Create huge quantity of certificates
3. Use new UI to show certificates
4. Use RAM usage + swapping + host became un-usable

Expected behavior
The page load, and has it has infinite scrolling, it should load pages of certificates as we scroll.

Environment:

• Vault Server Version (retrieve with vault status): 1.13.1
• Vault CLI Version (retrieve with vault version): Vault v1.13.3 (cgo)
• Server Operating System/Architecture: kubernetes

Additional context
I "only" have 16Go of RAM + 4Go of zram

[cipherboy]

This is sadly unavoidable due to present design limitations of Vault.

The underlying storage backends do not expose any paginated listing capabilities, meaning that the only (storage) list option is a one-shot, all-or-nothing approach. Because of this, no plugins expose any paginated list, count, or otherwise interruptible list operations.

This results in the same behavior, here exposed via the UI, but also present via API or CLI, just with different sizes of records causing problems. If you were to replace this with KV instead (with 100k+ entries in a single folder), the same behavior would be replicated there. (Perhaps yes, the new UI's LIST endpoint has a lower number than some other mounts or the older version, fundamentally its the same limitation, regardless of RAM or CPU constraints, even 5k isn't really a usable number of certs to have on a single page).

Thus fundamentally, this is a storage shortcoming and until that can be addressed, it is perhaps best to avoid large LIST endpoints in the UI. But for users with fewer entries, the LIST interface is generally useful, so it remains.

[Monkeychip]

I wanted to link this ticket as it may help with any slow loading due to DOM rendering. This does not fix the issues mentioned by @cipherboy.

The UI team has added pagination to the certificate's view and others in the PKI secret engine. Because this is a front-end implementation of pagination (e.g. we still can't limit the number of records the API sends us), this will only help ease the burden of the DOM rendering all the records. The fix was backported to 1.14.x

[outscale-fba]

I'm facing a similar issue on the transit key management API (transit API).

I understand the storage limitation and the need to a one-shot approach in its current implementation.
There might be a way to mitigate this keeping the one-shode mode approach by introducing a filter parameter in the list request. This would allow limiting the amount of results in the backend.
Using this you can do a kind of "poor" pagination by requesting all objects starting with "a*", then all starting with "b*", etc. You don't get a strong warranty on the amount of returned results, but this is effective in reducing the amount a results and make the list operation useable on huge dataset.
Of course this only works if you have an id or text field on which you can apply the filter (which is the case for keys on the transit API)

[hellobontempo]

@outscale-fba Thank you for this comment! Could you open a new issue with the aforementioned details in the feature request? We're collecting all issues that relate to pagination and the larger number of issues we have, including this, the easier it will be to get the work considered and prioritized.

[outscale-fba]

Thanks for your reply @hellobontempo.
I created #24046 to track this request. Hope it will get some echo soon.