Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature:Online Query and Dynamic Modification of Table-level RocksDB Options #1488

Open
ruojieranyishen opened this issue May 25, 2023 · 3 comments
Labels
type/enhancement Indicates new feature requests

Comments

@ruojieranyishen
Copy link
Collaborator

ruojieranyishen commented May 25, 2023

Background

Currently, in the Pegasus 2.4.1 version, dynamic querying and modification of RocksDB options at the table level are not supported. While the Pegasus shell provides commands like set_app_envs and create for configuring certain table-level options, it does not include RocksDB options. Consequently, adding the configure table-level RocksDB options ability would enhance the app configuration flexibility in Pegasus.

Goals

This issue aims to achieve the following goals:

  1. Provide online modification functionality for table-level RocksDB configuration.
  2. Provide online query functionality for table-level RocksDB configuration.
  3. Research on the principles, interdependencies, and modification risks of the rocksdb options used in Pegasus with rocksdb-6.6.4 version, and compile a comprehensive documentation on the usage of table-level rocksdb options.
  4. By modifying the logic of the Usage Scenario feature, unify the approach for setting table-level RocksDB options.

Solution

Here are the solutions for goals 1 and 2.
The storage location of rocksdb option is as follows:

  1. Utilize the config.ini file to provide default global RocksDB options information.
  2. Store table-specific RocksDB options in the envs section of the app_info in ZooKeeper (zk).

Three scenarios to set specific rocksdb option, and the solution as follows:

  1. [Create a replica with specific rocksdb option]
    Set specify option by create appname -e rocksdb.options command
    If a replica is not created, use the app envs rocksdb option from metaserver to create. For a already created replica, because the app envs is not transmitted from the metaserver, use options from the RocksDB OPTION file as the reference.

  2. [Modify a specific rocksdb option of an online replica]
    Set specify option by set_app_envs rocksdb.options command

  3. [Query the modified option online]
    Get specify option by get_app_envs command

To validate the feasibility of the proposed solution, I will use rocksdb.write_buffer_size as a dynamically modifiable parameter and rocksdb.num_levels as a non-dynamically modifiable parameter to test my idea.

@ruojieranyishen ruojieranyishen added the type/enhancement Indicates new feature requests label May 25, 2023
ruojieranyishen added a commit to ruojieranyishen/incubator-pegasus that referenced this issue May 30, 2023
… RocksDB Options (apache#1488)

apache#1488

Complete the dynamic setting function of num_levels and write_buffer_size option.I use rocksdb.write_buffer_size as a dynamically modifiable parameter and rocksdb.num_levels as a non-dynamically modifiable parameter to test my idea.
ruojieranyishen added a commit to ruojieranyishen/incubator-pegasus that referenced this issue May 30, 2023
… RocksDB Options (apache#1488)

apache#1488

Complete the dynamic setting function of num_levels and write_buffer_size option.I use rocksdb.write_buffer_size as a dynamically modifiable parameter and rocksdb.num_levels as a non-dynamically modifiable parameter to test my idea.
ruojieranyishen added a commit to ruojieranyishen/incubator-pegasus that referenced this issue May 30, 2023
… RocksDB Options (apache#1488)

apache#1488

Complete the dynamic setting function of num_levels and write_buffer_size option.I use rocksdb.write_buffer_size as a dynamically modifiable parameter and rocksdb.num_levels as a non-dynamically modifiable parameter to test my idea.
ruojieranyishen added a commit to ruojieranyishen/incubator-pegasus that referenced this issue May 30, 2023
… RocksDB Options (apache#1488)

apache#1488

Complete the dynamic setting function of num_levels and write_buffer_size option.I use rocksdb.write_buffer_size as a dynamically modifiable parameter and rocksdb.num_levels as a non-dynamically modifiable parameter to test my idea.
ruojieranyishen added a commit to ruojieranyishen/incubator-pegasus that referenced this issue Jun 2, 2023
… RocksDB Options (apache#1488)

apache#1488

Complete the dynamic setting function of num_levels and write_buffer_size option.I use rocksdb.write_buffer_size as a dynamically modifiable parameter and rocksdb.num_levels as a non-dynamically modifiable parameter to test my idea.
ruojieranyishen added a commit to ruojieranyishen/incubator-pegasus that referenced this issue Jun 3, 2023
… RocksDB Options (apache#1488)

apache#1488

Complete the dynamic setting function of num_levels and write_buffer_size option.I use rocksdb.write_buffer_size as a dynamically modifiable parameter and rocksdb.num_levels as a non-dynamically modifiable parameter to test my idea.

Pegasus shell case:
create lpf -e rocksdb.num_levels=12,rocksdb.write_buffer_size=100
create lpf -e rocksdb.num_levels=5,rocksdb.write_buffer_size=100
create lpf -e rocksdb.num_levels=5,rocksdb.write_buffer_size=33554432
set_app_envs rocksdb.write_buffer_size  67108864
set_app_envs rocksdb.num_levels 4
get_app_envs
ruojieranyishen added a commit to ruojieranyishen/incubator-pegasus that referenced this issue Jun 5, 2023
… RocksDB Options (apache#1488)

apache#1488

Complete the dynamic setting function of num_levels and write_buffer_size option.I use rocksdb.write_buffer_size as a dynamically modifiable parameter and rocksdb.num_levels as a non-dynamically modifiable parameter to test my idea.

Pegasus shell case:
create lpf -e rocksdb.num_levels=12,rocksdb.write_buffer_size=100
create lpf -e rocksdb.num_levels=5,rocksdb.write_buffer_size=100
create lpf -e rocksdb.num_levels=5,rocksdb.write_buffer_size=33554432
set_app_envs rocksdb.write_buffer_size  67108864
set_app_envs rocksdb.num_levels 4
get_app_envs
ruojieranyishen added a commit to ruojieranyishen/incubator-pegasus that referenced this issue Jun 5, 2023
… RocksDB Options (apache#1488)

apache#1488

Complete the dynamic setting function of num_levels and write_buffer_size option.I use rocksdb.write_buffer_size as a dynamically modifiable parameter and rocksdb.num_levels as a non-dynamically modifiable parameter to test my idea.

Pegasus shell case:
create lpf -e rocksdb.num_levels=12,rocksdb.write_buffer_size=100
create lpf -e rocksdb.num_levels=5,rocksdb.write_buffer_size=100
create lpf -e rocksdb.num_levels=5,rocksdb.write_buffer_size=33554432
set_app_envs rocksdb.write_buffer_size  67108864
set_app_envs rocksdb.num_levels 4
get_app_envs
ruojieranyishen added a commit to ruojieranyishen/incubator-pegasus that referenced this issue Jun 9, 2023
…Options (apache#1488)

apache#1488
Complete the dynamic setting function of num_levels and write_buffer_size option.I use rocksdb.write_buffer_size as a dynamically modifiable parameter and rocksdb.num_levels as a non-dynamically modifiable parameter to test my idea.
1. Providing online modification functionality for Pegasus table-level RocksDB configuration:
   1. The common RocksDB options can still be configured through the `config.ini` file.
   2. For the RocksDB options that support dynamic modification, they can be modified using the `create` or `set_app_envs` command.
   3. For the RocksDB options that do not support dynamic modification, they can be modified using the `create` command.
2. Providing online query functionality for Pegasus table-level RocksDB configuration:
   1. Following the design principle of app environment (`app env`), only the RocksDB options set through the `create` or `set_app_envs` command will be available for online querying.
   2. The remaining RocksDB option information is located in the `[pegasus.server]` section of the `config.ini` file.
- Unit test
Unit test is add in meta_app_envs_test.update_app_envs_test
- Manual test (add detailed scripts or steps below)
Pegasus shell case:
create lpf -e rocksdb.num_levels=12,rocksdb.write_buffer_size=100
create lpf -e rocksdb.num_levels=5,rocksdb.write_buffer_size=100
create lpf -e rocksdb.num_levels=5,rocksdb.write_buffer_size=33554432
set_app_envs rocksdb.write_buffer_size  67108864
set_app_envs rocksdb.write_buffer_size  100
get_app_envs
ruojieranyishen added a commit to ruojieranyishen/incubator-pegasus that referenced this issue Jun 9, 2023
…Options (apache#1488)

apache#1488
Currently, I only perform strict validation of app envs during the create_app command process. However, in the set_app_envs command, there is no validation for invalid envs, and they are simply not processed.

- Manual test (add detailed scripts or steps below)
Pegasus shell case:
create lpf -e rocksdb.num_levels=12,rocksdb.write_buffer_size=100
create lpf -e rocksdb.num_levels=5,rocksdb.write_buffer_size=100
create lpf -e rocksdb.num_levels=5,rocksdb.write_buffer_size=33554432
set_app_envs rocksdb.write_buffer_size  67108864
set_app_envs rocksdb.write_buffer_size  100
get_app_envs
ruojieranyishen added a commit to ruojieranyishen/incubator-pegasus that referenced this issue Jun 9, 2023
…Options (apache#1488)

apache#1488
Currently, I only perform strict validation of app envs during the create_app command process. However, in the set_app_envs command, there is no validation for invalid envs, and they are simply not processed.

- Manual test (add detailed scripts or steps below)
Pegasus shell case:
create lpf -e rocksdb.num_levels=12,rocksdb.write_buffer_size=100
create lpf -e rocksdb.num_levels=5,rocksdb.write_buffer_size=100
create lpf -e rocksdb.num_levels=5,rocksdb.write_buffer_size=33554432
set_app_envs rocksdb.write_buffer_size  67108864
set_app_envs rocksdb.write_buffer_size  100
get_app_envs
@ruojieranyishen
Copy link
Collaborator Author

This is the app creation process, I show the app envs passing through the num_levels option.

App创建流程 (2)

@ruojieranyishen
Copy link
Collaborator Author

replica具体创建过程 (2)

This is the specific creation process of replica, including my two modifications. Respectively deal with creating a new rocksdb and restarting rocksdb.

Create a replica key process:

  1. new replica : Create a replica, which contains env information num_levels=5, and the actual rocksdb has not been created
  2. store_app_info(_app_info) : Persist app_info information to .app_info file
  3. _app.reset(replication_app_base::new_storage_instance(_app_info.app_type, this)); :Construct pegasus_server_impl. Parameter this pointer is replica*, which contains env information num_levels=5.
  4. pegasus_server_impl::pegasus_server_impl :Construct pegasus_server_impl. Initialize rocksdb option via FLAGS_rocksdb_*
  5. update_app_envs_before_open_db(envs); :Make the env information effective before creating the rocksdb instance.
  6. reset_usage_scenario_options : Load rocksdb option from OPTION-XXXXX file.

@ruojieranyishen
Copy link
Collaborator Author

I show the set_app_envs procedure using the write_buffer_size option. My third modification solves the online modification of dynamic rocksdb options.
app envupdate (2)

acelyc111 pushed a commit that referenced this issue Aug 11, 2023
…Options (#1511)

#1488

Complete the dynamic setting function of num_levels and write_buffer_size option.
I use rocksdb.write_buffer_size as a dynamically modifiable parameter and rocksdb.num_levels
as a non-dynamically modifiable parameter to test my idea.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type/enhancement Indicates new feature requests
Projects
None yet
Development

No branches or pull requests

1 participant