Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix:Add a flag "rocksdb_write_global_seqno" and set it to false by default. #1560

Closed
ruojieranyishen opened this issue Jul 6, 2023 · 0 comments
Labels
type/enhancement Indicates new feature requests

Comments

@ruojieranyishen
Copy link
Collaborator

Is your feature request related to a problem? Please describe:

In the production environment, Pegasus performs bulkload involving rocksdb ingestion. Since write_global_seqno is true, rocksdb will modify the rocksdb.external_sst_file.global_seqno field in the external sstable file during ingestion. The modified is sometimes inaccurate, which will cause rocksdb fail to read the external sstable file. This will cause rocksdb coredump.

Describe the feature you'd like:

After research, I verified that write_global_seqno can be set to false.

The following are conclusions:

  • write_global_seqno=false will give up modify the external sst file. rocksdb.external_sst_file.global_seqno will always be zero. The global seqno information is retained by MANIFEST (smallest_seqno and largest_seqno field), and no additional performance overhead will be generated. In rocksdb, the order of external files and internal files is still identified through global seqno.
  • write_global_seqno=false does not conflict with the ingest_behind function of bulkload. Read, write, and delete operations are performed normally. The sst file is no longer modified during the ingest process. Speed up the ingest speed and check the sst file through checksum.
  • Disadvantages: Not compatible with versions prior to rocksdb 5.16.
@ruojieranyishen ruojieranyishen added the type/enhancement Indicates new feature requests label Jul 6, 2023
ruojieranyishen added a commit to ruojieranyishen/incubator-pegasus that referenced this issue Jul 6, 2023
…o false by default. apache#1560

apache#1560

Add rocksdb_write_global_seqno, and set its default value to false.
ruojieranyishen added a commit to ruojieranyishen/incubator-pegasus that referenced this issue Jul 6, 2023
…o false by default. apache#1560

apache#1560

Add rocksdb_write_global_seqno, and set its default value to false.
@ruojieranyishen ruojieranyishen changed the title Feature enhancement:Add flag: rocksdb_write_global_seqno and set it to false by default. fix:Add a flag "rocksdb_write_global_seqno" and set it to false by default. Jul 7, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type/enhancement Indicates new feature requests
Projects
None yet
Development

No branches or pull requests

1 participant