[Filestore] Make Filestore read throughput scalable with respect to the number of clients #324

qkrorlqr · 2024-02-01T19:03:56Z

We want a single FS to be able to provide more read throughput when more clients (VMs) connect to the FS. The idea is simple: right now the bottleneck for large reads is the IndexTablet, because all data is proxied via it. But we don't really need to transfer all data via the tablet. The tablet can return <BSGroupId, BlobId, ByteRange> tupes to the client (filestore-vhost) instead of returning the requested data. The client can read the data from the specified BSGroups by itself. A fallback should be implemented for the case when the client cannot read the data by itself - e.g. if the specified BlobId has already been deleted or if the client has no direct access to the storage nodes.

…client executeaction, removed broken (and unneeded) json validation; fixed comments here and there, etc

…client executeaction, removed broken (and unneeded) json validation; fixed comments here and there, etc (#351)

qkrorlqr · 2024-02-06T19:48:21Z

Включать фичу per fs будем как-то так:
filestore-client executeaction --action changestorageconfig --input-json '{"FileSystemId": "cahh6fjsng36rp186gjq", "StorageConfig": {"TwoStageReadEnabled": true}}'

Проверять так:
filestore-client executeaction --action getstorageconfigfields --input-json '{FileSystemId: "cahh6fjsng36rp186gjq", StorageConfigFields: ["TwoStageReadEnabled"]}'

…to filestore-vhost via TCreateSessionResponse::FileStore::Features; outputting StorageConfig overrides on monpage; minor cleanup

…to filestore-vhost via TCreateSessionResponse::FileStore::Features; outputting StorageConfig overrides on monpage; minor cleanup (#352) * issue #324: cleanup: filesystem-id field is not needed for filestore-client executeaction, removed broken (and unneeded) json validation; fixed comments here and there, etc * issue #324: introduced TwoStageReadEnabled flag, returning this flag to filestore-vhost via TCreateSessionResponse::FileStore::Features; outputting StorageConfig overrides on monpage; minor cleanup

…client executeaction, removed broken (and unneeded) json validation; fixed comments here and there, etc (#351)

…to filestore-vhost via TCreateSessionResponse::FileStore::Features; outputting StorageConfig overrides on monpage; minor cleanup (#352) * issue #324: cleanup: filesystem-id field is not needed for filestore-client executeaction, removed broken (and unneeded) json validation; fixed comments here and there, etc * issue #324: introduced TwoStageReadEnabled flag, returning this flag to filestore-vhost via TCreateSessionResponse::FileStore::Features; outputting StorageConfig overrides on monpage; minor cleanup

…client executeaction, removed broken (and unneeded) json validation; fixed comments here and there, etc (#351)

…to filestore-vhost via TCreateSessionResponse::FileStore::Features; outputting StorageConfig overrides on monpage; minor cleanup (#352) * issue #324: cleanup: filesystem-id field is not needed for filestore-client executeaction, removed broken (and unneeded) json validation; fixed comments here and there, etc * issue #324: introduced TwoStageReadEnabled flag, returning this flag to filestore-vhost via TCreateSessionResponse::FileStore::Features; outputting StorageConfig overrides on monpage; minor cleanup

* issue #324: DescribeData API and implementation stub (#329) * issue #324: DescribeData implementation and ut (#343) * issue #324: cleanup: filesystem-id field is not needed for filestore-client executeaction, removed broken (and unneeded) json validation; fixed comments here and there, etc (#351) * issue #324: introduced TwoStageReadEnabled flag, returning this flag to filestore-vhost via TCreateSessionResponse::FileStore::Features; outputting StorageConfig overrides on monpage; minor cleanup (#352) * issue #324: cleanup: filesystem-id field is not needed for filestore-client executeaction, removed broken (and unneeded) json validation; fixed comments here and there, etc * issue #324: introduced TwoStageReadEnabled flag, returning this flag to filestore-vhost via TCreateSessionResponse::FileStore::Features; outputting StorageConfig overrides on monpage; minor cleanup * issue #95: 1. deduplicating out-of-order compaction map chunk load requests in queue 2. limiting the number of out-of-order compaction map chunk load requests in queue (#382) * fixed build after cherry-pick: contrib/ydb -> ydb + CMakeLists

…the two stage read implementation in TServiceActor

…the two stage read implementation in TServiceActor (#400)

…ockIndex since this block might be not initialized

…ockIndex since this block might be not initialized (#402)

…the two stage read implementation in TServiceActor (#400)

…ockIndex since this block might be not initialized (#402)

* issue-324: DescribeData: 1. BlobOffsets should be in bytes, not blocks 2. returning FileSize 3. added a ut that checks that we don't return data outside of the aligned superrange that contains the requested range (#399) * issue #324: storing NProto::TFileStore in session state to use it in the two stage read implementation in TServiceActor (#400) * issue #324: DescribeData: offset calculation shouldn't use TBlock::BlockIndex since this block might be not initialized (#402) * NBSNEBIUS-101: use vhost-side reads WIP. Issue: #95 (#394) * [Draft] NBSNEBIUS-101: use vhost-side reads * WIP: service_actor_readdata * add describe data test * add describe data test * add ut + fix readblob implementation * enable TwoStageReadEnabled feature-flag * fix ut + trigger large tests --------- Co-authored-by: Maxim Deb Natkh <debnatkh@yandex.ru> * issue-324: proper EvGet error handling, not outputting user data to logs, dependencies cleanup, code cleanup (#419) * issue-324: proper EvGet error handling, not outputting user data to logs, dependencies cleanup, code cleanup * issue-324: proper EvGet error handling, not outputting user data to logs, dependencies cleanup, code cleanup - forgot libs/storage/model/ut * issue-324: proper EvGet error handling, not outputting user data to logs, dependencies cleanup, code cleanup - forgot libs/storage/model/public.h * issue-324: proper EvGet error handling, not outputting user data to logs, dependencies cleanup, code cleanup - fixed event-log lib * issue-324: TServiceActor: ReadData fallback in case DescribeData or EvGet fail (#428) * issue-324: TServiceActor: ReadData fallback in case DescribeData or EvGet fail * issue-324: TServiceActor: ReadData fallback in case DescribeData or EvGet fail * issue-324: TServiceActor: ReadData fallback in case DescribeData or EvGet fail - discarding EvGetResults after switching to the ReadData fallback * issue-324: TServiceActor: ReadData fallback in case DescribeData or EvGet fail - cleanup * issue-324: added ut and loadtest for the TwoStageRead feature, fixed a crash in TReadDataActor::HandleReadBlobResponse (#441) * issue-324: added ut and loadtest for the TwoStageRead feature, fixed a crash in TReadDataActor::HandleReadBlobResponse * issue-324: added ut and loadtest for the TwoStageRead feature, fixed a crash in TReadDataActor::HandleReadBlobResponse - forgot nfs-storage.txt * updated CMakeLists after cherry-pick * updated CMakeLists after cherry-pick - forgot to add new CMakeLists * contrib/ydb -> ydb * contrib/ydb -> ydb --------- Co-authored-by: Maxim Deb Natkh <debnatkh@gmail.com> Co-authored-by: Maxim Deb Natkh <debnatkh@yandex.ru>

qkrorlqr · 2024-02-19T19:04:34Z

managed to scale read throughput to 10 GB/s total for 10 VMs

debnatkh · 2024-02-22T17:14:42Z

After optimizations in IndexTablet managed to get 13-15 GB/s for 10 vms.

qkrorlqr · 2024-04-02T17:27:05Z

Implemented, deployed, works well, closing the issue

debnatkh · 2024-04-02T17:45:31Z

20 clients, fio, num_jobs = 32, bs = 1M

~25-30 GB/s

qkrorlqr self-assigned this Feb 1, 2024

qkrorlqr added the 2024Q1 label Feb 2, 2024

qkrorlqr added a commit that referenced this issue Feb 2, 2024

issue #324: DescribeData API and implementation stub

e2fc8ed

qkrorlqr mentioned this issue Feb 2, 2024

issue #324: DescribeData API and implementation stub #329

Merged

qkrorlqr added a commit that referenced this issue Feb 2, 2024

issue #324: DescribeData API and implementation stub (#329)

1c5856c

qkrorlqr added a commit that referenced this issue Feb 5, 2024

issue #324: DescribeData implementation and ut

48b5245

qkrorlqr added a commit that referenced this issue Feb 6, 2024

issue #324: DescribeData implementation and ut (#343)

6105a56

qkrorlqr added a commit that referenced this issue Feb 6, 2024

issue #324: cleanup: filesystem-id field is not needed for filestore-…

e5789b8

…client executeaction, removed broken (and unneeded) json validation; fixed comments here and there, etc

qkrorlqr added a commit that referenced this issue Feb 6, 2024

issue #324: cleanup: filesystem-id field is not needed for filestore-…

885e799

…client executeaction, removed broken (and unneeded) json validation; fixed comments here and there, etc (#351)

qkrorlqr added a commit that referenced this issue Feb 6, 2024

issue #324: introduced TwoStageReadEnabled flag, returning this flag …

c4d7dd7

…to filestore-vhost via TCreateSessionResponse::FileStore::Features; outputting StorageConfig overrides on monpage; minor cleanup

qkrorlqr mentioned this issue Feb 6, 2024

issue #324: introduced TwoStageReadEnabled flag, returning this flag to filestore-vhost via TCreateSessionResponse::FileStore::Features; outputting StorageConfig overrides on monpage; minor cleanup #352

Merged

qkrorlqr added a commit that referenced this issue Feb 9, 2024

issue #324: DescribeData API and implementation stub (#329)

feee281

qkrorlqr added a commit that referenced this issue Feb 9, 2024

issue #324: DescribeData implementation and ut (#343)

99755c5

qkrorlqr added a commit that referenced this issue Feb 9, 2024

issue #324: cleanup: filesystem-id field is not needed for filestore-…

1cb7185

…client executeaction, removed broken (and unneeded) json validation; fixed comments here and there, etc (#351)

qkrorlqr added a commit that referenced this issue Feb 9, 2024

issue #324: DescribeData API and implementation stub (#329)

17c7bea

qkrorlqr added a commit that referenced this issue Feb 9, 2024

issue #324: DescribeData implementation and ut (#343)

3dde745

qkrorlqr added a commit that referenced this issue Feb 9, 2024

issue #324: cleanup: filesystem-id field is not needed for filestore-…

3e35ab5

…client executeaction, removed broken (and unneeded) json validation; fixed comments here and there, etc (#351)

qkrorlqr added a commit that referenced this issue Feb 9, 2024

issue #324: storing NProto::TFileStore in session state to use it in …

b134e4c

…the two stage read implementation in TServiceActor

qkrorlqr added a commit that referenced this issue Feb 10, 2024

issue #324: storing NProto::TFileStore in session state to use it in …

8615cdc

…the two stage read implementation in TServiceActor (#400)

qkrorlqr added a commit that referenced this issue Feb 11, 2024

issue #324: DescribeData: offset calculation shouldn't use TBlock::Bl…

90ce8fe

…ockIndex since this block might be not initialized

qkrorlqr added a commit that referenced this issue Feb 11, 2024

issue #324: DescribeData: offset calculation shouldn't use TBlock::Bl…

bdbe36c

…ockIndex since this block might be not initialized (#402)

qkrorlqr added a commit that referenced this issue Feb 15, 2024

issue #324: storing NProto::TFileStore in session state to use it in …

6a27364

…the two stage read implementation in TServiceActor (#400)

qkrorlqr added a commit that referenced this issue Feb 15, 2024

issue #324: DescribeData: offset calculation shouldn't use TBlock::Bl…

723aefc

…ockIndex since this block might be not initialized (#402)

qkrorlqr mentioned this issue Feb 17, 2024

issue-324: TServiceActor: fixing TReadDataActor leak #471

Merged

qkrorlqr mentioned this issue Feb 20, 2024

issue-324: added tablet-level metrics for WriteData/ReadData/DescribeData requests #496

Merged

debnatkh mentioned this issue Feb 21, 2024

[Filestore] NBSNEBIUS-121: TwoStageRead: add logging for ReadData fallback reason + fix ServiceActor not reporting ReadData counters #521

Merged

debnatkh mentioned this issue Feb 22, 2024

[Filestore] Make Filestore write throughput scalable with respect to the number of clients #539

Closed

budevg mentioned this issue Feb 28, 2024

users/evgenybud/vhost rdma server migration v2 #583

Closed

qkrorlqr added the filestore Add this label to run only cloud/filestore build and tests on PR label Mar 7, 2024

debnatkh mentioned this issue Mar 28, 2024

[Filestore] Support read-ahead for sequential reads #852

Closed

qkrorlqr closed this as completed Apr 2, 2024

debnatkh mentioned this issue Apr 12, 2024

issue-539: add stats for BS requests for clients #963

Merged

debnatkh mentioned this issue Apr 26, 2024

[Filestore] add missing Die(ctx) in two-stage-read fallback data path #1075

Merged

debnatkh mentioned this issue May 6, 2024

[Filestore] add mutilple-clients fio tests to eternal-test-runner #1137

Merged

qkrorlqr mentioned this issue May 9, 2024

[Filestore] Optimization epic #1160

Open

debnatkh mentioned this issue May 28, 2024

[Filestore] Actualize filestore sensors #1285

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Filestore] Make Filestore read throughput scalable with respect to the number of clients #324

[Filestore] Make Filestore read throughput scalable with respect to the number of clients #324

qkrorlqr commented Feb 1, 2024 •

edited

Loading

qkrorlqr commented Feb 6, 2024 •

edited

Loading

qkrorlqr commented Feb 19, 2024

debnatkh commented Feb 22, 2024

qkrorlqr commented Apr 2, 2024

debnatkh commented Apr 2, 2024

[Filestore] Make Filestore read throughput scalable with respect to the number of clients #324

[Filestore] Make Filestore read throughput scalable with respect to the number of clients #324

Comments

qkrorlqr commented Feb 1, 2024 • edited Loading

qkrorlqr commented Feb 6, 2024 • edited Loading

qkrorlqr commented Feb 19, 2024

debnatkh commented Feb 22, 2024

qkrorlqr commented Apr 2, 2024

debnatkh commented Apr 2, 2024

qkrorlqr commented Feb 1, 2024 •

edited

Loading

qkrorlqr commented Feb 6, 2024 •

edited

Loading