[leo_gateway] Allow user to choose strong consistency or performance of inner cache

[windkit]

Description

When using the inner cache in leo_gateway, ETag of cache is compared with the one in backend leo_storage for every cache hit.

The round trip is not favorable for small read intensive workload, users may want to relax the consistency for better performance.

Proposal

Reference to the expire mechanism in http cache, allow the user to choose from "Strictly checked" to "check once 5 minutes" by setting the cache.cache_expire

Setting to 0 means checking every time and other values mean check once x seconds

Related PRs

leo-project/leo_gateway#36

[windkit]

For 4KB Objects, the difference is significant. It would increase from ~30000 ops to ~45000 ops

[yosukehara]

Thanks for your proposal.

I've understood your request but there are some issues in it.

• All objects under buckets are affected to this feature and its configuration, and LeoFS is used by multi users, not one. If its configuration is able to set each bucket, it's complicated.
• It seems this feature's purpose is LeoFS' benchmark.

Actually, I've been considering the metadata-cache feature from last week. If we reached the consensus, I'll share that.

[windkit]

Sorry I don't understand the concern, this suggestion does not aim to remove the check completely but rather allow user to choose.

1. What bothers me at first is the inconsistency between http cache and inner cache, one is done async (check after Expire), one is "semi-sync" (check checksum, serve data from cache).
2. Yes, every performance tuning would affect benchmark. No, it would affect lots of workload with "write once read many" characteristics.

To me, one of the strengths of LeoFS would be low latency and so I would like to improve it in this aspect.

Once again, I just want to give the freedom to user given that the code/flow change is not much.

[yosukehara]

At the beginning of implementation of the cache feature, we considered consistency between gateway-node(s) and a storage cluster. Our viewpoint is users expect strong consistency, we chose the strong consistency solution. If it's inconsistency between http-cache and inner-cache, we need to fix http cache features which is similar to the inner cache feature to avoid confusion.

[ZhaoX]

@yosukehara Just for your consideration.
In my situation, I store data in LeoFS and never update them. So, If the consistency between leo_gateway cache and leo_storage data is configurable, I would prefer it.

[yosukehara]

@ZhaoX In our company, LeoFS has been used by lots of services. We cannot handle their needs both a strong consistency and an weak consistency with the LeoGateway's configuration.

In order to improve LeoFS performance, we've implemented concurrency of read-operation in LeoStorage.

• [leo_object_storage] Improve concurrency #450
• [leo_object_storage] Improve concurrency of read operations #467

By the feature, LeoFS performance is dramatically improved:

• https://github.com/leo-project/notes/tree/master/leofs/benchmark/leofs/1.2/1.2.20/front/20160316_image_f2m_r95w5_30min_1

[ZhaoX]

@yosukehara Thank you for your reply.
What's the LeoFS version being benchmarked? The title is "Benchmark LeoFS v1.2.20-dev" and the environment part said "LeoFS: v1.4.0-pre.3-dev".

[windkit]

@ZhaoX Sorry I made the mistakes in the environment session, it is done with 1.2.20-dev

[windkit]

@ZhaoX Thank you for you concern about this issue. But after a second thought, the benefit of the design not as huge as I first suggested.
It is really a rare case that application keep getting the same objects within a small period of time, if there is such kind of need, it would be more common to be solved by proxy in front of LeoFS.

[yosukehara]

@windkit @ZhaoX Thank you for your comment and suggestion. I've closed this because we reached the consensus.