Add snapshot for prefix iterate. #3717

SuperYoko · 2022-01-13T10:06:29Z

What type of PR is this?

bug
feature
enhancement

What problem(s) does this PR solve?

Issue(s) number:

Description:
Use prefix to iter data on the same snapshot of rocksdb.

How do you solve it?

Special notes for your reviewer, ex. impact of this fix, design document, etc:

Checklist:

Tests:

Unit test(positive and negative cases)
Function test
Performance test
N/A

Affects:

Documentation affected (Please add the label if documentation needs to be modified.)
Incompatibility (If it breaks the compatibility, please describe it and add the label.）
If it's needed to cherry-pick (If cherry-pick to some branches is required, please label the destination version(s).)
Performance impacted: Consumes more CPU/Memory

Release notes:

Please confirm whether to be reflected in release notes and how to describe:

ex. Fixed the bug .....

critical27 · 2022-01-13T10:57:09Z

Close #3710

critical27 · 2022-01-13T10:57:22Z

Very quick fix 👍

src/kvstore/RocksEngine.h

darionyaphet · 2022-01-18T02:30:43Z

src/kvstore/RocksEngine.h

+  }
+
+  void ReleaseSnapshot(const void* snapshot) override {
+    db_->ReleaseSnapshot(reinterpret_cast<const rocksdb::Snapshot*>(snapshot));


ReleaseSnapshot would remove files from file system?

No, It's only a memory snapshot. But it do prevent files to be deleted before snapshot is released.

AFAIK snapshot will create some hard link about the previous SST files why don't remove them?

AFAIK snapshot will create some hard link about the previous SST files why don't remove them?

https://github.com/facebook/rocksdb/wiki/Snapshot

A snapshot captures a point-in-time view of the DB at the time it's created. Snapshots do not persist across DB restarts.

Maybe you mean Checkpoint.

https://github.com/facebook/rocksdb/wiki/Checkpoints

liwenhui-soul · 2022-01-18T03:01:46Z

what does void* mean? it doesn't have a type?

SuperYoko · 2022-01-18T03:34:50Z

what does void* mean? it doesn't have a type?

I wan't to create a wrapper like iterator first, but it didn't help much understanding.
Anyway, Rocksdb Snapshot pointer should pass to prefix function, and there should be a container to save pointer.

Maybe a warpper of vector boost::any is okey, but it still pass a pointer. I don't have another clear idea. WDYT?

* Add snapshot for prefix iterate. * Add header comment * Add happy test * Fix lint * Modify engine snap argument position * Fix other subclass * Fix code Co-authored-by: yaphet <4414314+darionyaphet@users.noreply.github.com> Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com> Co-authored-by: Doodle <13706157+critical27@users.noreply.github.com>

HarrisChu · 2022-01-22T06:29:14Z

#3782

seems db_upgrader crashed by this

* Add snapshot for prefix iterate. * Add header comment * Add happy test * Fix lint * Modify engine snap argument position * Fix other subclass * Fix code Co-authored-by: yaphet <4414314+darionyaphet@users.noreply.github.com> Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com> Co-authored-by: Doodle <13706157+critical27@users.noreply.github.com>

* drop host only when it registe heartbeat (#3739) * Fix race of create tag and edge (#3735) * fix rece * polish * fix review * Add version for `show hosts` (#3702) * version * tck * pytest * pytest Co-authored-by: cpw <13495049+CPWstatic@users.noreply.github.com> * Update client version to v3.0.0 (#3741) * Update python package name to v3 * Update client version white list * Update go mod to v3 * Change the python package name used in tests * Update client version white list * Fix create fulltext index failed (#3747) * fix bug: the same tagId/edgetype under different spaces, failed to create fulltext indexes * fix bug: the same tagId/edgetype under different spaces, failed to create fulltext indexes * fix writternBy delete by optimizer (#3746) * fix bugs * add tck features * style Co-authored-by: cpw <13495049+CPWstatic@users.noreply.github.com> * fix with & optimizer rule error (#3736) * fix error * fix collapseProjectRule Co-authored-by: cpw <13495049+CPWstatic@users.noreply.github.com> * fix upgrade bug (#3772) Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com> * fix 3727 (#3737) Co-authored-by: Doodle <13706157+critical27@users.noreply.github.com> * Balance fix (#3668) * fix core balance in zone removing the only host in zone * fix balance plan that make redundant part * fix deadlock when runInternal and finish callback called in the same thread * fix transleader return TERM_OUT_OF_DATE * continue running balance task from last status * fix deadlock when removing part Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com> * Add snapshot for prefix iterate. (#3717) * Add snapshot for prefix iterate. * Add header comment * Add happy test * Fix lint * Modify engine snap argument position * Fix other subclass * Fix code Co-authored-by: yaphet <4414314+darionyaphet@users.noreply.github.com> Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com> Co-authored-by: Doodle <13706157+critical27@users.noreply.github.com> * suppress undesired ASAN error (#3705) * suppress undesired error * fix unstable test case * fix upgrade (#3787) Co-authored-by: Doodle <13706157+critical27@users.noreply.github.com> * Create CODEOWNERS (#3770) Confirm with sophie.xie * disable data balance (#3756) Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com> * Fix iterator with snapshot(parameter position) (#3785) Co-authored-by: Harris.Chu <1726587+HarrisChu@users.noreply.github.com> Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com> Co-authored-by: yaphet <4414314+darionyaphet@users.noreply.github.com> Co-authored-by: jakevin <30525741+jackwener@users.noreply.github.com> Co-authored-by: cpw <13495049+CPWstatic@users.noreply.github.com> Co-authored-by: Yichen Wang <18348405+Aiee@users.noreply.github.com> Co-authored-by: panda-sheep <59197347+panda-sheep@users.noreply.github.com> Co-authored-by: endy.li <25311962+heroicNeZha@users.noreply.github.com> Co-authored-by: jimingquan <mingquan.ji@vesoft.com> Co-authored-by: hs.zhang <22708345+cangfengzhs@users.noreply.github.com> Co-authored-by: Doodle <13706157+critical27@users.noreply.github.com> Co-authored-by: liwenhui-soul <38217397+liwenhui-soul@users.noreply.github.com> Co-authored-by: Alex Xing <90179377+SuperYoko@users.noreply.github.com> Co-authored-by: Harris.Chu <1726587+HarrisChu@users.noreply.github.com>

panda-sheep · 2022-01-28T04:18:16Z

Good job

@panda-sheep

* add cmake git hooks options * githooks option * include what you use * make name follow convention * linker support lld, which is the fastest linker. If ci install lld when pull request , will set lld as default linker. * log rotate * delete werror * pragma once => ifndef * pragma once => ifndef * Werror bison * format code * fix issue 3601 (#3666) * fix meta upgrade for multi instance (#3734) * drop host only when it registe heartbeat (#3739) * Fix race of create tag and edge (#3735) * fix rece * polish * fix review * Add version for `show hosts` (#3702) * version * tck * pytest * pytest Co-authored-by: cpw <13495049+CPWstatic@users.noreply.github.com> * Enhancement/set expression depth by flag (#3722) * Set expression depth limit by flag. * Format. Co-authored-by: Yichen Wang <18348405+Aiee@users.noreply.github.com> * Update client version to v3.0.0 (#3741) * Update python package name to v3 * Update client version white list * Update go mod to v3 * Change the python package name used in tests * Update client version white list * Fix create fulltext index failed (#3747) * fix bug: the same tagId/edgetype under different spaces, failed to create fulltext indexes * fix bug: the same tagId/edgetype under different spaces, failed to create fulltext indexes * add comments and adjust log level for meta/processors/admin (#3662) * add comments and adjust log level for meta admin processors * change agent hb log level Co-authored-by: Doodle <13706157+critical27@users.noreply.github.com> * fix writternBy delete by optimizer (#3746) * fix bugs * add tck features * style Co-authored-by: cpw <13495049+CPWstatic@users.noreply.github.com> * fix with & optimizer rule error (#3736) * fix error * fix collapseProjectRule Co-authored-by: cpw <13495049+CPWstatic@users.noreply.github.com> * fix upgrade bug (#3772) Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com> * fix 3727 (#3737) Co-authored-by: Doodle <13706157+critical27@users.noreply.github.com> * Balance fix (#3668) * fix core balance in zone removing the only host in zone * fix balance plan that make redundant part * fix deadlock when runInternal and finish callback called in the same thread * fix transleader return TERM_OUT_OF_DATE * continue running balance task from last status * fix deadlock when removing part Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com> * Eliminte the useless row collect. (#3363) * Eliminte the useless row collect. * Remove comment. * Format. * Rebase. Co-authored-by: cpw <13495049+CPWstatic@users.noreply.github.com> Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com> * Add snapshot for prefix iterate. (#3717) * Add snapshot for prefix iterate. * Add header comment * Add happy test * Fix lint * Modify engine snap argument position * Fix other subclass * Fix code Co-authored-by: yaphet <4414314+darionyaphet@users.noreply.github.com> Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com> Co-authored-by: Doodle <13706157+critical27@users.noreply.github.com> * Read lock zone when create space. (#3693) Co-authored-by: yaphet <4414314+darionyaphet@users.noreply.github.com> Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com> * suppress undesired ASAN error (#3705) * suppress undesired error * fix unstable test case * 1. add comments and change log level for meta (#3709) 2. remove some unused meta kv files 3. remove unused tests 4. remove unused kv relative tests in metaclient Co-authored-by: Doodle <13706157+critical27@users.noreply.github.com> * fix upgrade (#3787) Co-authored-by: Doodle <13706157+critical27@users.noreply.github.com> * Create CODEOWNERS (#3770) Confirm with sophie.xie * disable data balance (#3756) Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com> * Fix iterator with snapshot(parameter position) (#3785) Co-authored-by: Harris.Chu <1726587+HarrisChu@users.noreply.github.com> Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com> * disable meta upgrade from v1 (#3798) * Add gflag validator for client_idle_timeout_secs (#3807) Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com> * disable zone commands (#3776) Co-authored-by: cpw <13495049+CPWstatic@users.noreply.github.com> * Add ci for standalone compile and ut. (#3637) * Add standalone workflow * remove tmp note * Add meta v2 thrift obj for sa * Add tck test * Fix port and dist tck * skip some test on sa * Fix merge * Fix add host Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com> * Fix the promise/future. (#3814) Co-authored-by: jimingquan <mingquan.ji@vesoft.com> * add comments & accumulate bug fix for TOSS (#3643) * 1. add comments & adjust some log level 2. accumulate bug fix for TOSS 2.1 one space may be affected by another space. when recover prime edges. 2.2 when call kvstore->get(), if leaderLeaseInvalid, it will report leader_change, change it to leader_lease_failed. 2.3 print all execution trace in ChainProcessor::finish(), make much more grep friendly. 2.4 some function of memory lock is hard to understand (forceLock, forceUnlock) , change to "setAutoUnlock(bool)" modify update fix compile error adjust finish() adjust finish() add rcPrepare_ = succeeded adjust delete local processor adjust log use rcPrepare, rcRemote rcCommit fix resume add edges adjust UT disable ioThreadPoolForMeta_ make int64 vid printable adjust some log to vlog 2 only print prepare succeeded result only print readable in finish() of resume processor adjust some log some log not print term desc class name in uuid debug add some more log looks like factory may fail? add leader lease error code add check when get lock report leader lease do some clean change table prefix to NebulaKeyUtil Style change some print disable UT for short clean compile error disable pool monitor * unify add/delete finish() as update * not stable will add it back later * fix a clang warning * fix a clang10 warning 2 Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com> * disable add hosts into zone (#3825) Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com> * Check active zone before create space (#3822) Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com> * abandon upgrader for old version (#3818) Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com> * put dataVersionKey on create space for storage (#3817) * put dataVersionKey on create space for storage * address some comments Co-authored-by: Doodle <13706157+critical27@users.noreply.github.com> Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com> * Update PR template (#3809) Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com> * fix meta http replace host handler (#3794) * fix meta http replace host handler * fix http replace test and add zone test Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com> * Fix `Match cannot be followed by use space` (#3748) * fix bug * add tck * remove set * fix * rm header * fix * remove changes in match * fix * enhance annotation * fix review * fix * fix review * remove redundant Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com> Co-authored-by: cpw <13495049+CPWstatic@users.noreply.github.com> * support toSet function (#3594) add test Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com> * Enhance date parser (#3797) * parse date * enhance datetime * add test * fix review * rename Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com> * raft: fix always loop when no log need to send (#3819) Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com> Co-authored-by: Doodle <13706157+critical27@users.noreply.github.com> * Update CODEOWNERS (#3835) * Update CODEOWNERS * Update CODEOWNERS Confirm with sophie.xie * refactor AdminClient (#3781) * refactor AdminClient * fix test case, unify all getAdminAddrFromStoreAddr into AdminClient * add more test cases * address @panda-sheep's comment * Feature/optimizer trait match (#3769) * Add trait for pattern match. * Implement match plan by trait and reduce some reduncant rules. * Remove unused header. * Fix warning. * Let optimize rule decide collection of matched nodes. Co-authored-by: kyle.cao <kyle.cao@vesoft.com> Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com> * Simplify datetime tostring. (#3712) * Simplify datetime tostring. * Remove unused code. * Fix error. Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com> * pragma once not allowd (#3842) Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com> * refactor LOG and add comment for meta, meta/processors/job (#3686) * refactor LOG and add comment for meta, meta/processors/job * refactor LOG and add comment for meta/processors/zone Co-authored-by: Doodle <13706157+critical27@users.noreply.github.com> * replace STRING to name_label for zone's name (#3780) Co-authored-by: Doodle <13706157+critical27@users.noreply.github.com> * Add test for rc. (#3847) * Add test for rc. * Fix cxx. Confirm with sophie.xie * cpp lint Co-authored-by: Yee <2520865+yixinglu@users.noreply.github.com> Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com> Co-authored-by: cpw <13495049+CPWstatic@users.noreply.github.com> Co-authored-by: hs.zhang <22708345+cangfengzhs@users.noreply.github.com> Co-authored-by: yaphet <4414314+darionyaphet@users.noreply.github.com> Co-authored-by: jakevin <30525741+jackwener@users.noreply.github.com> Co-authored-by: shylock <33566796+Shylock-Hg@users.noreply.github.com> Co-authored-by: Yichen Wang <18348405+Aiee@users.noreply.github.com> Co-authored-by: panda-sheep <59197347+panda-sheep@users.noreply.github.com> Co-authored-by: pengwei.song <90180021+pengweisong@users.noreply.github.com> Co-authored-by: Doodle <13706157+critical27@users.noreply.github.com> Co-authored-by: endy.li <25311962+heroicNeZha@users.noreply.github.com> Co-authored-by: jimingquan <mingquan.ji@vesoft.com> Co-authored-by: liwenhui-soul <38217397+liwenhui-soul@users.noreply.github.com> Co-authored-by: Alex Xing <90179377+SuperYoko@users.noreply.github.com> Co-authored-by: Harris.Chu <1726587+HarrisChu@users.noreply.github.com> Co-authored-by: lionel.liu@vesoft.com <52276794+liuyu85cn@users.noreply.github.com> Co-authored-by: kyle.cao <kyle.cao@vesoft.com>

@panda-sheep

* add cmake git hooks options * githooks option * include what you use * make name follow convention * linker support lld, which is the fastest linker. If ci install lld when pull request , will set lld as default linker. * log rotate * delete werror * pragma once => ifndef * pragma once => ifndef * Werror bison * format code * fix issue 3601 (vesoft-inc#3666) * fix meta upgrade for multi instance (vesoft-inc#3734) * drop host only when it registe heartbeat (vesoft-inc#3739) * Fix race of create tag and edge (vesoft-inc#3735) * fix rece * polish * fix review * Add version for `show hosts` (vesoft-inc#3702) * version * tck * pytest * pytest Co-authored-by: cpw <13495049+CPWstatic@users.noreply.github.com> * Enhancement/set expression depth by flag (vesoft-inc#3722) * Set expression depth limit by flag. * Format. Co-authored-by: Yichen Wang <18348405+Aiee@users.noreply.github.com> * Update client version to v3.0.0 (vesoft-inc#3741) * Update python package name to v3 * Update client version white list * Update go mod to v3 * Change the python package name used in tests * Update client version white list * Fix create fulltext index failed (vesoft-inc#3747) * fix bug: the same tagId/edgetype under different spaces, failed to create fulltext indexes * fix bug: the same tagId/edgetype under different spaces, failed to create fulltext indexes * add comments and adjust log level for meta/processors/admin (vesoft-inc#3662) * add comments and adjust log level for meta admin processors * change agent hb log level Co-authored-by: Doodle <13706157+critical27@users.noreply.github.com> * fix writternBy delete by optimizer (vesoft-inc#3746) * fix bugs * add tck features * style Co-authored-by: cpw <13495049+CPWstatic@users.noreply.github.com> * fix with & optimizer rule error (vesoft-inc#3736) * fix error * fix collapseProjectRule Co-authored-by: cpw <13495049+CPWstatic@users.noreply.github.com> * fix upgrade bug (vesoft-inc#3772) Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com> * fix 3727 (vesoft-inc#3737) Co-authored-by: Doodle <13706157+critical27@users.noreply.github.com> * Balance fix (vesoft-inc#3668) * fix core balance in zone removing the only host in zone * fix balance plan that make redundant part * fix deadlock when runInternal and finish callback called in the same thread * fix transleader return TERM_OUT_OF_DATE * continue running balance task from last status * fix deadlock when removing part Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com> * Eliminte the useless row collect. (vesoft-inc#3363) * Eliminte the useless row collect. * Remove comment. * Format. * Rebase. Co-authored-by: cpw <13495049+CPWstatic@users.noreply.github.com> Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com> * Add snapshot for prefix iterate. (vesoft-inc#3717) * Add snapshot for prefix iterate. * Add header comment * Add happy test * Fix lint * Modify engine snap argument position * Fix other subclass * Fix code Co-authored-by: yaphet <4414314+darionyaphet@users.noreply.github.com> Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com> Co-authored-by: Doodle <13706157+critical27@users.noreply.github.com> * Read lock zone when create space. (vesoft-inc#3693) Co-authored-by: yaphet <4414314+darionyaphet@users.noreply.github.com> Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com> * suppress undesired ASAN error (vesoft-inc#3705) * suppress undesired error * fix unstable test case * 1. add comments and change log level for meta (vesoft-inc#3709) 2. remove some unused meta kv files 3. remove unused tests 4. remove unused kv relative tests in metaclient Co-authored-by: Doodle <13706157+critical27@users.noreply.github.com> * fix upgrade (vesoft-inc#3787) Co-authored-by: Doodle <13706157+critical27@users.noreply.github.com> * Create CODEOWNERS (vesoft-inc#3770) Confirm with sophie.xie * disable data balance (vesoft-inc#3756) Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com> * Fix iterator with snapshot(parameter position) (vesoft-inc#3785) Co-authored-by: Harris.Chu <1726587+HarrisChu@users.noreply.github.com> Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com> * disable meta upgrade from v1 (vesoft-inc#3798) * Add gflag validator for client_idle_timeout_secs (vesoft-inc#3807) Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com> * disable zone commands (vesoft-inc#3776) Co-authored-by: cpw <13495049+CPWstatic@users.noreply.github.com> * Add ci for standalone compile and ut. (vesoft-inc#3637) * Add standalone workflow * remove tmp note * Add meta v2 thrift obj for sa * Add tck test * Fix port and dist tck * skip some test on sa * Fix merge * Fix add host Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com> * Fix the promise/future. (vesoft-inc#3814) Co-authored-by: jimingquan <mingquan.ji@vesoft.com> * add comments & accumulate bug fix for TOSS (vesoft-inc#3643) * 1. add comments & adjust some log level 2. accumulate bug fix for TOSS 2.1 one space may be affected by another space. when recover prime edges. 2.2 when call kvstore->get(), if leaderLeaseInvalid, it will report leader_change, change it to leader_lease_failed. 2.3 print all execution trace in ChainProcessor::finish(), make much more grep friendly. 2.4 some function of memory lock is hard to understand (forceLock, forceUnlock) , change to "setAutoUnlock(bool)" modify update fix compile error adjust finish() adjust finish() add rcPrepare_ = succeeded adjust delete local processor adjust log use rcPrepare, rcRemote rcCommit fix resume add edges adjust UT disable ioThreadPoolForMeta_ make int64 vid printable adjust some log to vlog 2 only print prepare succeeded result only print readable in finish() of resume processor adjust some log some log not print term desc class name in uuid debug add some more log looks like factory may fail? add leader lease error code add check when get lock report leader lease do some clean change table prefix to NebulaKeyUtil Style change some print disable UT for short clean compile error disable pool monitor * unify add/delete finish() as update * not stable will add it back later * fix a clang warning * fix a clang10 warning 2 Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com> * disable add hosts into zone (vesoft-inc#3825) Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com> * Check active zone before create space (vesoft-inc#3822) Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com> * abandon upgrader for old version (vesoft-inc#3818) Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com> * put dataVersionKey on create space for storage (vesoft-inc#3817) * put dataVersionKey on create space for storage * address some comments Co-authored-by: Doodle <13706157+critical27@users.noreply.github.com> Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com> * Update PR template (vesoft-inc#3809) Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com> * fix meta http replace host handler (vesoft-inc#3794) * fix meta http replace host handler * fix http replace test and add zone test Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com> * Fix `Match cannot be followed by use space` (vesoft-inc#3748) * fix bug * add tck * remove set * fix * rm header * fix * remove changes in match * fix * enhance annotation * fix review * fix * fix review * remove redundant Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com> Co-authored-by: cpw <13495049+CPWstatic@users.noreply.github.com> * support toSet function (vesoft-inc#3594) add test Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com> * Enhance date parser (vesoft-inc#3797) * parse date * enhance datetime * add test * fix review * rename Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com> * raft: fix always loop when no log need to send (vesoft-inc#3819) Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com> Co-authored-by: Doodle <13706157+critical27@users.noreply.github.com> * Update CODEOWNERS (vesoft-inc#3835) * Update CODEOWNERS * Update CODEOWNERS Confirm with sophie.xie * refactor AdminClient (vesoft-inc#3781) * refactor AdminClient * fix test case, unify all getAdminAddrFromStoreAddr into AdminClient * add more test cases * address @panda-sheep's comment * Feature/optimizer trait match (vesoft-inc#3769) * Add trait for pattern match. * Implement match plan by trait and reduce some reduncant rules. * Remove unused header. * Fix warning. * Let optimize rule decide collection of matched nodes. Co-authored-by: kyle.cao <kyle.cao@vesoft.com> Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com> * Simplify datetime tostring. (vesoft-inc#3712) * Simplify datetime tostring. * Remove unused code. * Fix error. Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com> * pragma once not allowd (vesoft-inc#3842) Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com> * refactor LOG and add comment for meta, meta/processors/job (vesoft-inc#3686) * refactor LOG and add comment for meta, meta/processors/job * refactor LOG and add comment for meta/processors/zone Co-authored-by: Doodle <13706157+critical27@users.noreply.github.com> * replace STRING to name_label for zone's name (vesoft-inc#3780) Co-authored-by: Doodle <13706157+critical27@users.noreply.github.com> * Add test for rc. (vesoft-inc#3847) * Add test for rc. * Fix cxx. Confirm with sophie.xie * cpp lint Co-authored-by: Yee <2520865+yixinglu@users.noreply.github.com> Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com> Co-authored-by: cpw <13495049+CPWstatic@users.noreply.github.com> Co-authored-by: hs.zhang <22708345+cangfengzhs@users.noreply.github.com> Co-authored-by: yaphet <4414314+darionyaphet@users.noreply.github.com> Co-authored-by: jakevin <30525741+jackwener@users.noreply.github.com> Co-authored-by: shylock <33566796+Shylock-Hg@users.noreply.github.com> Co-authored-by: Yichen Wang <18348405+Aiee@users.noreply.github.com> Co-authored-by: panda-sheep <59197347+panda-sheep@users.noreply.github.com> Co-authored-by: pengwei.song <90180021+pengweisong@users.noreply.github.com> Co-authored-by: Doodle <13706157+critical27@users.noreply.github.com> Co-authored-by: endy.li <25311962+heroicNeZha@users.noreply.github.com> Co-authored-by: jimingquan <mingquan.ji@vesoft.com> Co-authored-by: liwenhui-soul <38217397+liwenhui-soul@users.noreply.github.com> Co-authored-by: Alex Xing <90179377+SuperYoko@users.noreply.github.com> Co-authored-by: Harris.Chu <1726587+HarrisChu@users.noreply.github.com> Co-authored-by: lionel.liu@vesoft.com <52276794+liuyu85cn@users.noreply.github.com> Co-authored-by: kyle.cao <kyle.cao@vesoft.com>

@panda-sheep

* add cmake git hooks options * githooks option * include what you use * make name follow convention * linker support lld, which is the fastest linker. If ci install lld when pull request , will set lld as default linker. * log rotate * delete werror * pragma once => ifndef * pragma once => ifndef * Werror bison * format code * fix issue 3601 (vesoft-inc#3666) * fix meta upgrade for multi instance (vesoft-inc#3734) * drop host only when it registe heartbeat (vesoft-inc#3739) * Fix race of create tag and edge (vesoft-inc#3735) * fix rece * polish * fix review * Add version for `show hosts` (vesoft-inc#3702) * version * tck * pytest * pytest Co-authored-by: cpw <13495049+CPWstatic@users.noreply.github.com> * Enhancement/set expression depth by flag (vesoft-inc#3722) * Set expression depth limit by flag. * Format. Co-authored-by: Yichen Wang <18348405+Aiee@users.noreply.github.com> * Update client version to v3.0.0 (vesoft-inc#3741) * Update python package name to v3 * Update client version white list * Update go mod to v3 * Change the python package name used in tests * Update client version white list * Fix create fulltext index failed (vesoft-inc#3747) * fix bug: the same tagId/edgetype under different spaces, failed to create fulltext indexes * fix bug: the same tagId/edgetype under different spaces, failed to create fulltext indexes * add comments and adjust log level for meta/processors/admin (vesoft-inc#3662) * add comments and adjust log level for meta admin processors * change agent hb log level Co-authored-by: Doodle <13706157+critical27@users.noreply.github.com> * fix writternBy delete by optimizer (vesoft-inc#3746) * fix bugs * add tck features * style Co-authored-by: cpw <13495049+CPWstatic@users.noreply.github.com> * fix with & optimizer rule error (vesoft-inc#3736) * fix error * fix collapseProjectRule Co-authored-by: cpw <13495049+CPWstatic@users.noreply.github.com> * fix upgrade bug (vesoft-inc#3772) Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com> * fix 3727 (vesoft-inc#3737) Co-authored-by: Doodle <13706157+critical27@users.noreply.github.com> * Balance fix (vesoft-inc#3668) * fix core balance in zone removing the only host in zone * fix balance plan that make redundant part * fix deadlock when runInternal and finish callback called in the same thread * fix transleader return TERM_OUT_OF_DATE * continue running balance task from last status * fix deadlock when removing part Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com> * Eliminte the useless row collect. (vesoft-inc#3363) * Eliminte the useless row collect. * Remove comment. * Format. * Rebase. Co-authored-by: cpw <13495049+CPWstatic@users.noreply.github.com> Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com> * Add snapshot for prefix iterate. (vesoft-inc#3717) * Add snapshot for prefix iterate. * Add header comment * Add happy test * Fix lint * Modify engine snap argument position * Fix other subclass * Fix code Co-authored-by: yaphet <4414314+darionyaphet@users.noreply.github.com> Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com> Co-authored-by: Doodle <13706157+critical27@users.noreply.github.com> * Read lock zone when create space. (vesoft-inc#3693) Co-authored-by: yaphet <4414314+darionyaphet@users.noreply.github.com> Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com> * suppress undesired ASAN error (vesoft-inc#3705) * suppress undesired error * fix unstable test case * 1. add comments and change log level for meta (vesoft-inc#3709) 2. remove some unused meta kv files 3. remove unused tests 4. remove unused kv relative tests in metaclient Co-authored-by: Doodle <13706157+critical27@users.noreply.github.com> * fix upgrade (vesoft-inc#3787) Co-authored-by: Doodle <13706157+critical27@users.noreply.github.com> * Create CODEOWNERS (vesoft-inc#3770) Confirm with sophie.xie * disable data balance (vesoft-inc#3756) Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com> * Fix iterator with snapshot(parameter position) (vesoft-inc#3785) Co-authored-by: Harris.Chu <1726587+HarrisChu@users.noreply.github.com> Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com> * disable meta upgrade from v1 (vesoft-inc#3798) * Add gflag validator for client_idle_timeout_secs (vesoft-inc#3807) Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com> * disable zone commands (vesoft-inc#3776) Co-authored-by: cpw <13495049+CPWstatic@users.noreply.github.com> * Add ci for standalone compile and ut. (vesoft-inc#3637) * Add standalone workflow * remove tmp note * Add meta v2 thrift obj for sa * Add tck test * Fix port and dist tck * skip some test on sa * Fix merge * Fix add host Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com> * Fix the promise/future. (vesoft-inc#3814) Co-authored-by: jimingquan <mingquan.ji@vesoft.com> * add comments & accumulate bug fix for TOSS (vesoft-inc#3643) * 1. add comments & adjust some log level 2. accumulate bug fix for TOSS 2.1 one space may be affected by another space. when recover prime edges. 2.2 when call kvstore->get(), if leaderLeaseInvalid, it will report leader_change, change it to leader_lease_failed. 2.3 print all execution trace in ChainProcessor::finish(), make much more grep friendly. 2.4 some function of memory lock is hard to understand (forceLock, forceUnlock) , change to "setAutoUnlock(bool)" modify update fix compile error adjust finish() adjust finish() add rcPrepare_ = succeeded adjust delete local processor adjust log use rcPrepare, rcRemote rcCommit fix resume add edges adjust UT disable ioThreadPoolForMeta_ make int64 vid printable adjust some log to vlog 2 only print prepare succeeded result only print readable in finish() of resume processor adjust some log some log not print term desc class name in uuid debug add some more log looks like factory may fail? add leader lease error code add check when get lock report leader lease do some clean change table prefix to NebulaKeyUtil Style change some print disable UT for short clean compile error disable pool monitor * unify add/delete finish() as update * not stable will add it back later * fix a clang warning * fix a clang10 warning 2 Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com> * disable add hosts into zone (vesoft-inc#3825) Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com> * Check active zone before create space (vesoft-inc#3822) Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com> * abandon upgrader for old version (vesoft-inc#3818) Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com> * put dataVersionKey on create space for storage (vesoft-inc#3817) * put dataVersionKey on create space for storage * address some comments Co-authored-by: Doodle <13706157+critical27@users.noreply.github.com> Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com> * Update PR template (vesoft-inc#3809) Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com> * fix meta http replace host handler (vesoft-inc#3794) * fix meta http replace host handler * fix http replace test and add zone test Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com> * Fix `Match cannot be followed by use space` (vesoft-inc#3748) * fix bug * add tck * remove set * fix * rm header * fix * remove changes in match * fix * enhance annotation * fix review * fix * fix review * remove redundant Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com> Co-authored-by: cpw <13495049+CPWstatic@users.noreply.github.com> * support toSet function (vesoft-inc#3594) add test Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com> * Enhance date parser (vesoft-inc#3797) * parse date * enhance datetime * add test * fix review * rename Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com> * raft: fix always loop when no log need to send (vesoft-inc#3819) Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com> Co-authored-by: Doodle <13706157+critical27@users.noreply.github.com> * Update CODEOWNERS (vesoft-inc#3835) * Update CODEOWNERS * Update CODEOWNERS Confirm with sophie.xie * refactor AdminClient (vesoft-inc#3781) * refactor AdminClient * fix test case, unify all getAdminAddrFromStoreAddr into AdminClient * add more test cases * address @panda-sheep's comment * Feature/optimizer trait match (vesoft-inc#3769) * Add trait for pattern match. * Implement match plan by trait and reduce some reduncant rules. * Remove unused header. * Fix warning. * Let optimize rule decide collection of matched nodes. Co-authored-by: kyle.cao <kyle.cao@vesoft.com> Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com> * Simplify datetime tostring. (vesoft-inc#3712) * Simplify datetime tostring. * Remove unused code. * Fix error. Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com> * pragma once not allowd (vesoft-inc#3842) Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com> * refactor LOG and add comment for meta, meta/processors/job (vesoft-inc#3686) * refactor LOG and add comment for meta, meta/processors/job * refactor LOG and add comment for meta/processors/zone Co-authored-by: Doodle <13706157+critical27@users.noreply.github.com> * replace STRING to name_label for zone's name (vesoft-inc#3780) Co-authored-by: Doodle <13706157+critical27@users.noreply.github.com> * Add test for rc. (vesoft-inc#3847) * Add test for rc. * Fix cxx. Confirm with sophie.xie * cpp lint Co-authored-by: Yee <2520865+yixinglu@users.noreply.github.com> Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com> Co-authored-by: cpw <13495049+CPWstatic@users.noreply.github.com> Co-authored-by: hs.zhang <22708345+cangfengzhs@users.noreply.github.com> Co-authored-by: yaphet <4414314+darionyaphet@users.noreply.github.com> Co-authored-by: jakevin <30525741+jackwener@users.noreply.github.com> Co-authored-by: shylock <33566796+Shylock-Hg@users.noreply.github.com> Co-authored-by: Yichen Wang <18348405+Aiee@users.noreply.github.com> Co-authored-by: panda-sheep <59197347+panda-sheep@users.noreply.github.com> Co-authored-by: pengwei.song <90180021+pengweisong@users.noreply.github.com> Co-authored-by: Doodle <13706157+critical27@users.noreply.github.com> Co-authored-by: endy.li <25311962+heroicNeZha@users.noreply.github.com> Co-authored-by: jimingquan <mingquan.ji@vesoft.com> Co-authored-by: liwenhui-soul <38217397+liwenhui-soul@users.noreply.github.com> Co-authored-by: Alex Xing <90179377+SuperYoko@users.noreply.github.com> Co-authored-by: Harris.Chu <1726587+HarrisChu@users.noreply.github.com> Co-authored-by: lionel.liu@vesoft.com <52276794+liuyu85cn@users.noreply.github.com> Co-authored-by: kyle.cao <kyle.cao@vesoft.com> Co-authored-by: yuehua.jia <3423893+jiayuehua@users.noreply.github.com> Co-authored-by: Yee <2520865+yixinglu@users.noreply.github.com> Co-authored-by: Sophie <84560950+Sophie-Xie@users.noreply.github.com> Co-authored-by: cpw <13495049+CPWstatic@users.noreply.github.com> Co-authored-by: hs.zhang <22708345+cangfengzhs@users.noreply.github.com> Co-authored-by: yaphet <4414314+darionyaphet@users.noreply.github.com> Co-authored-by: jakevin <30525741+jackwener@users.noreply.github.com> Co-authored-by: shylock <33566796+Shylock-Hg@users.noreply.github.com> Co-authored-by: Yichen Wang <18348405+Aiee@users.noreply.github.com> Co-authored-by: panda-sheep <59197347+panda-sheep@users.noreply.github.com> Co-authored-by: pengwei.song <90180021+pengweisong@users.noreply.github.com> Co-authored-by: Doodle <13706157+critical27@users.noreply.github.com> Co-authored-by: endy.li <25311962+heroicNeZha@users.noreply.github.com> Co-authored-by: jimingquan <mingquan.ji@vesoft.com> Co-authored-by: liwenhui-soul <38217397+liwenhui-soul@users.noreply.github.com> Co-authored-by: Alex Xing <90179377+SuperYoko@users.noreply.github.com> Co-authored-by: Harris.Chu <1726587+HarrisChu@users.noreply.github.com> Co-authored-by: lionel.liu@vesoft.com <52276794+liuyu85cn@users.noreply.github.com> Co-authored-by: kyle.cao <kyle.cao@vesoft.com>

## What type of PR is this? - [ ] bug - [ ] feature - [X] enhancement ## What problem(s) does this PR solve? #### Issue(s) number: #### Description: This is some prepare work for later data balance test, mainly two changes: 1. Follow up of vesoft-inc#3717(In that pr we use snapshot when doing data scan, but the commit log id/term is not taken care). So in this PR, we create a rocksdb snapshot at first, read commit log id/term, and use the same snapshot to scan data. 2. We can't simply skip the special raft command, because if we skip the special log after rebooting, the state of raft peer will be wrong. 3. Add some metrics in raft, remove the `SlowOpTracker`. ## How do you solve it? ## Special notes for your reviewer, ex. impact of this fix, design document, etc: ## Checklist: Tests: - [X] Unit test(positive and negative cases) - [ ] Function test - [ ] Performance test - [ ] N/A Affects: - [ ] Documentation affected (Please add the label if documentation needs to be modified.) - [ ] Incompatibility (If it breaks the compatibility, please describe it and add the label.） - [ ] If it's needed to cherry-pick (If cherry-pick to some branches is required, please label the destination version(s).) - [ ] Performance impacted: Consumes more CPU/Memory ## Release notes: not related Migrated from vesoft-inc#3846 Co-authored-by: Doodle <13706157+critical27@users.noreply.github.com>

Add snapshot for prefix iterate.

c3f1ee5

SuperYoko changed the title ~~TMP ADD SNAPSHOT~~ Add snapshot for prefix iterate. Jan 17, 2022

SuperYoko marked this pull request as ready for review January 17, 2022 03:57

Merge branch 'master' into prefix_a_snap

b210b27

SuperYoko added ready-for-testing PR: ready for the CI test and removed ready-for-testing PR: ready for the CI test labels Jan 17, 2022

SuperYoko added 2 commits January 17, 2022 13:37

Add header comment

4794ea9

Add happy test

bedb514

SuperYoko added the ready-for-testing PR: ready for the CI test label Jan 17, 2022

SuperYoko requested review from critical27, cangfengzhs, darionyaphet, liwenhui-soul, panda-sheep, pengweisong and a team January 17, 2022 06:27

SuperYoko added 2 commits January 17, 2022 14:50

Fix lint

8c30df2

Modify engine snap argument position

492bd01

SuperYoko added the cherry-pick-v3.0 PR: need cherry-pick to this version label Jan 17, 2022

SuperYoko added 4 commits January 17, 2022 21:22

Merge branch 'master' into prefix_a_snap

21a4036

Fix other subclass

3d96060

Fix code

50bd476

Merge branch 'master' into prefix_a_snap

9097239

darionyaphet reviewed Jan 18, 2022

View reviewed changes

src/kvstore/RocksEngine.h Show resolved Hide resolved

darionyaphet reviewed Jan 18, 2022

View reviewed changes

critical27 approved these changes Jan 20, 2022

View reviewed changes

Merge branch 'master' into prefix_a_snap

15c0b41

darionyaphet approved these changes Jan 20, 2022

View reviewed changes

Sophie-Xie and others added 2 commits January 20, 2022 19:31

Merge branch 'master' into prefix_a_snap

8505e74

Merge branch 'master' into prefix_a_snap

6cdc51b

critical27 merged commit e14aba8 into vesoft-inc:master Jan 21, 2022

jamieliu1023 mentioned this pull request Jan 22, 2022

Weekly Report 2022-01-21 vesoft-inc/nebula-community#86

Closed

critical27 mentioned this pull request Jan 29, 2022

Use the same rocksdb snapshot when sending snapshot #3846

Merged

11 tasks

SuperYoko deleted the prefix_a_snap branch September 28, 2022 02:44

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add snapshot for prefix iterate. #3717

Add snapshot for prefix iterate. #3717

SuperYoko commented Jan 13, 2022

critical27 commented Jan 13, 2022

critical27 commented Jan 13, 2022

darionyaphet Jan 18, 2022 •

edited

Loading

SuperYoko Jan 18, 2022 •

edited

Loading

darionyaphet Jan 20, 2022

SuperYoko Jan 20, 2022 •

edited

Loading

liwenhui-soul commented Jan 18, 2022

SuperYoko commented Jan 18, 2022 •

edited

Loading

HarrisChu commented Jan 22, 2022

panda-sheep commented Jan 28, 2022

Add snapshot for prefix iterate. #3717

Add snapshot for prefix iterate. #3717

Conversation

SuperYoko commented Jan 13, 2022

What type of PR is this?

What problem(s) does this PR solve?

How do you solve it?

Special notes for your reviewer, ex. impact of this fix, design document, etc:

Checklist:

Release notes:

critical27 commented Jan 13, 2022

critical27 commented Jan 13, 2022

darionyaphet Jan 18, 2022 • edited Loading

Choose a reason for hiding this comment

SuperYoko Jan 18, 2022 • edited Loading

Choose a reason for hiding this comment

darionyaphet Jan 20, 2022

Choose a reason for hiding this comment

SuperYoko Jan 20, 2022 • edited Loading

Choose a reason for hiding this comment

liwenhui-soul commented Jan 18, 2022

SuperYoko commented Jan 18, 2022 • edited Loading

HarrisChu commented Jan 22, 2022

panda-sheep commented Jan 28, 2022

darionyaphet Jan 18, 2022 •

edited

Loading

SuperYoko Jan 18, 2022 •

edited

Loading

SuperYoko Jan 20, 2022 •

edited

Loading

SuperYoko commented Jan 18, 2022 •

edited

Loading