You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe:
In the production environment, Pegasus performs bulkload involving rocksdb ingestion. Since write_global_seqno is true, rocksdb will modify the rocksdb.external_sst_file.global_seqno field in the external sstable file during ingestion. The modified is sometimes inaccurate, which will cause rocksdb fail to read the external sstable file. This will cause rocksdb coredump.
Describe the feature you'd like:
After research, I verified that write_global_seqno can be set to false.
The following are conclusions:
write_global_seqno=false will give up modify the external sst file. rocksdb.external_sst_file.global_seqno will always be zero. The global seqno information is retained by MANIFEST (smallest_seqno and largest_seqno field), and no additional performance overhead will be generated. In rocksdb, the order of external files and internal files is still identified through global seqno.
write_global_seqno=false does not conflict with the ingest_behind function of bulkload. Read, write, and delete operations are performed normally. The sst file is no longer modified during the ingest process. Speed up the ingest speed and check the sst file through checksum.
Disadvantages: Not compatible with versions prior to rocksdb 5.16.
The text was updated successfully, but these errors were encountered:
ruojieranyishen
changed the title
Feature enhancement:Add flag: rocksdb_write_global_seqno and set it to false by default.
fix:Add a flag "rocksdb_write_global_seqno" and set it to false by default.
Jul 7, 2023
Is your feature request related to a problem? Please describe:
In the production environment, Pegasus performs bulkload involving rocksdb ingestion. Since write_global_seqno is true, rocksdb will modify the
rocksdb.external_sst_file.global_seqno
field in the external sstable file during ingestion. The modified is sometimes inaccurate, which will cause rocksdb fail to read the external sstable file. This will cause rocksdb coredump.Describe the feature you'd like:
After research, I verified that write_global_seqno can be set to false.
The following are conclusions:
write_global_seqno=false
will give up modify the external sst file.rocksdb.external_sst_file.global_seqno
will always be zero. The global seqno information is retained by MANIFEST (smallest_seqno and largest_seqno field), and no additional performance overhead will be generated. In rocksdb, the order of external files and internal files is still identified through global seqno.write_global_seqno=false
does not conflict with the ingest_behind function of bulkload. Read, write, and delete operations are performed normally. The sst file is no longer modified during the ingest process. Speed up the ingest speed and check the sst file through checksum.The text was updated successfully, but these errors were encountered: