-
Notifications
You must be signed in to change notification settings - Fork 312
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Feature:Online Query and Dynamic Modification of Table-level RocksDB Options #1488
Labels
type/enhancement
Indicates new feature requests
Comments
ruojieranyishen
added a commit
to ruojieranyishen/incubator-pegasus
that referenced
this issue
May 30, 2023
… RocksDB Options (apache#1488) apache#1488 Complete the dynamic setting function of num_levels and write_buffer_size option.I use rocksdb.write_buffer_size as a dynamically modifiable parameter and rocksdb.num_levels as a non-dynamically modifiable parameter to test my idea.
ruojieranyishen
added a commit
to ruojieranyishen/incubator-pegasus
that referenced
this issue
May 30, 2023
… RocksDB Options (apache#1488) apache#1488 Complete the dynamic setting function of num_levels and write_buffer_size option.I use rocksdb.write_buffer_size as a dynamically modifiable parameter and rocksdb.num_levels as a non-dynamically modifiable parameter to test my idea.
ruojieranyishen
added a commit
to ruojieranyishen/incubator-pegasus
that referenced
this issue
May 30, 2023
… RocksDB Options (apache#1488) apache#1488 Complete the dynamic setting function of num_levels and write_buffer_size option.I use rocksdb.write_buffer_size as a dynamically modifiable parameter and rocksdb.num_levels as a non-dynamically modifiable parameter to test my idea.
ruojieranyishen
added a commit
to ruojieranyishen/incubator-pegasus
that referenced
this issue
May 30, 2023
… RocksDB Options (apache#1488) apache#1488 Complete the dynamic setting function of num_levels and write_buffer_size option.I use rocksdb.write_buffer_size as a dynamically modifiable parameter and rocksdb.num_levels as a non-dynamically modifiable parameter to test my idea.
ruojieranyishen
added a commit
to ruojieranyishen/incubator-pegasus
that referenced
this issue
Jun 2, 2023
… RocksDB Options (apache#1488) apache#1488 Complete the dynamic setting function of num_levels and write_buffer_size option.I use rocksdb.write_buffer_size as a dynamically modifiable parameter and rocksdb.num_levels as a non-dynamically modifiable parameter to test my idea.
ruojieranyishen
added a commit
to ruojieranyishen/incubator-pegasus
that referenced
this issue
Jun 3, 2023
… RocksDB Options (apache#1488) apache#1488 Complete the dynamic setting function of num_levels and write_buffer_size option.I use rocksdb.write_buffer_size as a dynamically modifiable parameter and rocksdb.num_levels as a non-dynamically modifiable parameter to test my idea. Pegasus shell case: create lpf -e rocksdb.num_levels=12,rocksdb.write_buffer_size=100 create lpf -e rocksdb.num_levels=5,rocksdb.write_buffer_size=100 create lpf -e rocksdb.num_levels=5,rocksdb.write_buffer_size=33554432 set_app_envs rocksdb.write_buffer_size 67108864 set_app_envs rocksdb.num_levels 4 get_app_envs
ruojieranyishen
added a commit
to ruojieranyishen/incubator-pegasus
that referenced
this issue
Jun 5, 2023
… RocksDB Options (apache#1488) apache#1488 Complete the dynamic setting function of num_levels and write_buffer_size option.I use rocksdb.write_buffer_size as a dynamically modifiable parameter and rocksdb.num_levels as a non-dynamically modifiable parameter to test my idea. Pegasus shell case: create lpf -e rocksdb.num_levels=12,rocksdb.write_buffer_size=100 create lpf -e rocksdb.num_levels=5,rocksdb.write_buffer_size=100 create lpf -e rocksdb.num_levels=5,rocksdb.write_buffer_size=33554432 set_app_envs rocksdb.write_buffer_size 67108864 set_app_envs rocksdb.num_levels 4 get_app_envs
ruojieranyishen
added a commit
to ruojieranyishen/incubator-pegasus
that referenced
this issue
Jun 5, 2023
… RocksDB Options (apache#1488) apache#1488 Complete the dynamic setting function of num_levels and write_buffer_size option.I use rocksdb.write_buffer_size as a dynamically modifiable parameter and rocksdb.num_levels as a non-dynamically modifiable parameter to test my idea. Pegasus shell case: create lpf -e rocksdb.num_levels=12,rocksdb.write_buffer_size=100 create lpf -e rocksdb.num_levels=5,rocksdb.write_buffer_size=100 create lpf -e rocksdb.num_levels=5,rocksdb.write_buffer_size=33554432 set_app_envs rocksdb.write_buffer_size 67108864 set_app_envs rocksdb.num_levels 4 get_app_envs
ruojieranyishen
added a commit
to ruojieranyishen/incubator-pegasus
that referenced
this issue
Jun 9, 2023
…Options (apache#1488) apache#1488 Complete the dynamic setting function of num_levels and write_buffer_size option.I use rocksdb.write_buffer_size as a dynamically modifiable parameter and rocksdb.num_levels as a non-dynamically modifiable parameter to test my idea. 1. Providing online modification functionality for Pegasus table-level RocksDB configuration: 1. The common RocksDB options can still be configured through the `config.ini` file. 2. For the RocksDB options that support dynamic modification, they can be modified using the `create` or `set_app_envs` command. 3. For the RocksDB options that do not support dynamic modification, they can be modified using the `create` command. 2. Providing online query functionality for Pegasus table-level RocksDB configuration: 1. Following the design principle of app environment (`app env`), only the RocksDB options set through the `create` or `set_app_envs` command will be available for online querying. 2. The remaining RocksDB option information is located in the `[pegasus.server]` section of the `config.ini` file. - Unit test Unit test is add in meta_app_envs_test.update_app_envs_test - Manual test (add detailed scripts or steps below) Pegasus shell case: create lpf -e rocksdb.num_levels=12,rocksdb.write_buffer_size=100 create lpf -e rocksdb.num_levels=5,rocksdb.write_buffer_size=100 create lpf -e rocksdb.num_levels=5,rocksdb.write_buffer_size=33554432 set_app_envs rocksdb.write_buffer_size 67108864 set_app_envs rocksdb.write_buffer_size 100 get_app_envs
ruojieranyishen
added a commit
to ruojieranyishen/incubator-pegasus
that referenced
this issue
Jun 9, 2023
…Options (apache#1488) apache#1488 Currently, I only perform strict validation of app envs during the create_app command process. However, in the set_app_envs command, there is no validation for invalid envs, and they are simply not processed. - Manual test (add detailed scripts or steps below) Pegasus shell case: create lpf -e rocksdb.num_levels=12,rocksdb.write_buffer_size=100 create lpf -e rocksdb.num_levels=5,rocksdb.write_buffer_size=100 create lpf -e rocksdb.num_levels=5,rocksdb.write_buffer_size=33554432 set_app_envs rocksdb.write_buffer_size 67108864 set_app_envs rocksdb.write_buffer_size 100 get_app_envs
ruojieranyishen
added a commit
to ruojieranyishen/incubator-pegasus
that referenced
this issue
Jun 9, 2023
…Options (apache#1488) apache#1488 Currently, I only perform strict validation of app envs during the create_app command process. However, in the set_app_envs command, there is no validation for invalid envs, and they are simply not processed. - Manual test (add detailed scripts or steps below) Pegasus shell case: create lpf -e rocksdb.num_levels=12,rocksdb.write_buffer_size=100 create lpf -e rocksdb.num_levels=5,rocksdb.write_buffer_size=100 create lpf -e rocksdb.num_levels=5,rocksdb.write_buffer_size=33554432 set_app_envs rocksdb.write_buffer_size 67108864 set_app_envs rocksdb.write_buffer_size 100 get_app_envs
This is the specific creation process of replica, including my two modifications. Respectively deal with creating a new rocksdb and restarting rocksdb. Create a replica key process:
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Background
Currently, in the Pegasus 2.4.1 version, dynamic querying and modification of RocksDB options at the table level are not supported. While the Pegasus shell provides commands like
set_app_envs
andcreate
for configuring certain table-level options, it does not include RocksDB options. Consequently, adding the configure table-level RocksDB options ability would enhance the app configuration flexibility in Pegasus.Goals
This issue aims to achieve the following goals:
Solution
Here are the solutions for goals 1 and 2.
The storage location of rocksdb option is as follows:
Three scenarios to set specific rocksdb option, and the solution as follows:
[Create a replica with specific rocksdb option]
Set specify option by
create appname -e rocksdb.options
commandIf a replica is not created, use the app envs rocksdb option from metaserver to create. For a already created replica, because the app envs is not transmitted from the metaserver, use options from the RocksDB OPTION file as the reference.
[Modify a specific rocksdb option of an online replica]
Set specify option by
set_app_envs rocksdb.options
command[Query the modified option online]
Get specify option by
get_app_envs
commandTo validate the feasibility of the proposed solution, I will use rocksdb.write_buffer_size as a dynamically modifiable parameter and rocksdb.num_levels as a non-dynamically modifiable parameter to test my idea.
The text was updated successfully, but these errors were encountered: