rgw/sfs: linear performance degradation with the number of objects in buckets

[giubacc]

Bucket::_refresh_objects() is a very frequent called function in sfs and it is iterating all the objects held by a bucket.
This exposes sfs to an inherent linear complexity directly proportional with the number of objects held by buckets.
I made a test filling a bucket with 10k objects and the latency increases significantly:

Every time a list_buckets is called there is a magnitude latency of seconds:

2022-11-08T15:17:58.941+0100 7fffed2aa6c0 10 req 0 0.004000061s s3:create_bucket list_buckets: marker (, ), max=1000
2022-11-08T15:18:02.745+0100 7fffed2aa6c0 10 req 0 3.808057070s s3:create_bucket list_buckets: buckets={foo3=:foo3[foo3.1667916356826389689]),foo4=:foo4[foo4.1667916475603620198]),foo5=:foo5[foo5.1667916497598257137])}

We should discuss how to avoid the _refresh_objects call in the sfs::Bucket class.
More in general, _refresh_objects is an inherently slow, linear operation and we should see if we can improve this design.
This also applies to _refresh_buckets function (it is less exposed, but the problem is the same).

[jecluis]

This is likely one of the culprits for the performance numbers being so crappy right now.

One of the paths forward may simply be caching objects in memory instead of refreshing all the time, invalidating solely when objects are changed (or deleted, depending on versioning being enabled or not). This can also limit the number of objects in memory, and do a basic LRU in case there's a missing object in cache that needs to be promoted should the cache be full.

We may not even have to refresh if objects are added or changed; we may just need to add a new object or update an existing one. There's a lot of room to be smarter here.

[giubacc]

I think the approach to define a maximum amount of memory loaded items per bucket is preferable (maybe something configurable).
Having all in memory definitely does not scale in many ways.
Absolutely agree to update memory objects only when needed.

[jhmarina]

Before we start working on implementing performance improvements, we want to set up a call and plan this properly.

[irq0]

https://aquarist-labs.github.io/s3gw-perf-reports/reports/release_comprehensive_v0.14.0.html
supports the importance. The LIST benchmarks are either slow (> 10 seconds latency) or time out.

[irq0]

Discussion moved on since December (last comment). Summary: aquarist-labs/ceph#124 refactored and removed the object cache. Plan is to rewrite list bucket as SQL queries. Right now it is basically a get all from database and filter the results. I'm working on it.

[giubacc]

@irq0 is this issue still valid?
Originally I was concerned that sfs code was linearly iterating all the objects inside a bucket, is that still the case?
Can we think to close this issue?

[irq0]

It's not solved, but enough changed to warrant a new issue. Closing in favor of #509