-
Notifications
You must be signed in to change notification settings - Fork 44
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Run coordinator together with router #287
Run coordinator together with router #287
Conversation
91681a2
to
477ed04
Compare
6082496
to
d5bb66e
Compare
We do not need coordinator console, since we only interact with coordinator using grpc-API in code, and i think that it is acceptable that final users can access spqr installation for different type of requests though router console. We need to run admin console in proxy way for several type of queries:
|
Sounds great. I will do it ASAP |
b0231ec
to
5375a2e
Compare
5375a2e
to
787a576
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Overall its ok
lgtm
Have enhanced commit message(squashed) with detailed explaining of this PR. |
* Fix release * Get tag in Makefile Create coordinator.md (pg-sharding#225) * Create coordinator.md * update docs * fix typo --------- Co-authored-by: denchick <denchick@yandex-team.ru> Run each feature in seperate job (pg-sharding#268) Unite admin-console/psql-console pre-routing logic. (pg-sharding#259) * Unite admin-console/psql-console pre-routing logic. essentially fixes pg-sharding#255 * Fix tests Update Router.md (pg-sharding#210) * Update Router.md resolves pg-sharding#201 * fix typo --------- Co-authored-by: denchick <denchick@yandex-team.ru> fix init.sql + backup.json logic (pg-sharding#262) * fix init.sql + backup logic * fix docs * add new scenario * fix spelling Coordinator Show clients/pools/backend_connections funcs (pg-sharding#265) * implement pools for coordinator * `show pools` & `show backend connections ` now shows router names * added units for coord pool * added feature for show pools etc. Show router status in `SHOW ROUTERS` command response (pg-sharding#266) * Show router status in `SHOW ROUTERS` command responce ``` spqr-consolds=> show routers; show routers | status --------------------------+-------- router r1-localhost:7010 | OPENED (1 row) ``` * Linter fix * Fix expected router out * fixes --------- Co-authored-by: denchick <denchick@yandex-team.ru> Fix dataraces in memqdb (pg-sharding#260) * fix datarace with memqdb.go? * fix linter + ci * add missing methods * resolve conflicts * fix deadlock: sharekey with unlockkey * fix deadlock: dropkeyrange with unlockkeyrange * fix Makefile * refactor method * fix comments * fix error message fix double lock key range on router when moving data (pg-sharding#271) fix lock/unlock key range wrong namespace in qdb (pg-sharding#270) * fix lock/unlock key range in coordinator * fix e2e tests test sync router after registration in coordinator (pg-sharding#267) * test sync router after registration in coordinator * check shard id in sync router test scenario * fix sql result comparison in coordinator feature tests Upload to existing release instead of creating a new one (pg-sharding#273) add some checks to Split/Unite (pg-sharding#275) * add some checks to UniteKeyRange in coordinator * add some checks on bound to Split * minor test scenarios titles fixes Refactor QDB interfaces, add comments to code (pg-sharding#274) check if sharding rule already exists, implement CheckKeyRangeLocked in etcdQDB (pg-sharding#278) fix key range intersection checks (pg-sharding#276) * fix moving key range to the same shardID * fix adding key range with existing krid by splitting * fix qdb key range intersection checks in qdb operations * minor pr fixes * fix tests * extended tests * cannot split range by lower or upper bound because range cannot be of 1 point * fix comment ping router before adding to qdb (pg-sharding#277) * ping router before adding to qdb * wrap ping router error bump dependencies (pg-sharding#281) fix coordinator crash when moving key range twice (pg-sharding#282) Single coord (pg-sharding#280) * locking coordinator * made transactions atomic * added tests * fixed tests and linters * fixed features * enhansed coordinator lock * fix * requested changes fix missing log level on router reload (pg-sharding#284) Support X protocol in frontend interactions: Part 1 (pg-sharding#286) * Support X protocol in frontend interactions * Add mockgen and add test to Check TSA * Add unit test to frontend function * Fix linter * Enhance makefile * Bump lyx * Comment frontend test as there is work to do, to make in work Fix typo in spqr_test.go (pg-sharding#289) Refactor Client, Relay and PoolMgr interfaces (pg-sharding#288) * Refactor Client, Relay and PoolMgr interfaces * Add test * Add copy test Parallelize scenarios (pg-sharding#285) * Execute scenarios in feature in parallel * not do second split * cache images from wal-g * try without docker install * don't build spqr-shard-image if docker/shard doesn't changes * reuse running jobs * revert "cache images" * refactor add checks to move key range (pg-sharding#283) * add checks to move key range * change shardID from sh3 to non-existent in move feature test Run regress tests on 12-15 Postgresql (pg-sharding#290) * run regress on multiple version of postgres * stop qdb after regress * remove 16 and 17 version Ignore feature tests artifacts (pg-sharding#293) Added SCRAM auth feature tests (pg-sharding#292) * Added "I run commandon host" step * Added .feature file * Some fixes * Switch back to "regress" user * Frontend auth test * Added "Command output should match" step * Added psql output check * Added backend auth test * Moved psql installation to Dockerfile fix possible deadlock on big move (pg-sharding#291) Bump lyx (pg-sharding#295) * Bump lyx * Regen make gogen (pg-sharding#296) Add gram rules for strart trace request (pg-sharding#297) Run coordinator together with router (pg-sharding#287) Dispatch coordinator goroutines with router binary start. Proxy all modifying requests to Coordinator via grpc. Run admin console in proxy way for several type of queries: All modify requests to router console should be proxied to coordinator via particular gRPC-call. It is very dangerous to allow user to modify router metadata, like key-ranges or sharding rules, etc. This only should by done by coordinator or via coord. API. All show requests, like SHOW CLIENTS etc should be processed locally. In future, we can support installation-wide show requests processing using additional syntax, like SHOW CLIENTS is processed by router but SHOW GLOBAL CLIENTS to be proccesed by coordinator (coord. can process such queries already). Regen yacc (pg-sharding#299) Logproxy in router (pg-sharding#298) Add workload recording infrastructure, workload recording API and bindings. Provide SQL interface to access new subsystem. Replay test workload (pg-sharding#303) Replay recorded workload in router. revert config param (pg-sharding#300) Update .gitignore (pg-sharding#304) route begin transaction read-only to replica (pg-sharding#305) * route begin transaction read-only to replica * Bump lyx * Bump Bump lyx (pg-sharding#307) bump lyx (pg-sharding#308) Update main.go (pg-sharding#310) Bump lyx (pg-sharding#311) Xproto: extended query protocol interactions. (pg-sharding#302) * Extended protocol interaction basic * Fix * Fix * util+gomod * Bump * Fix * Fix2 * Fix3 Bump lyx (pg-sharding#312) * Bump luyx * Fix * Better Added support for systemd notifications (pg-sharding#314) * Added notifier type * Added notifications to router instance * Added alive notifications * Added debug logging for systemd notifications * Added reloading notifications support * Added systemd notifier settings to config * Fixed golangci-lint Add client_encoding to router console (pg-sharding#315) Add dataStyle to router console (pg-sharding#316) Accept routing hints in comments (pg-sharding#317) * Accept routing hints in comments add dataspaces support Change gram.y: Add DROP DATASPACE support to parser Gen Bugfix Gen delete DROP DATASPACE support gen Add show dataspace regress test Small fix fix regress tests and add tests to dataspace Xproto: extended query protocol interactions. (pg-sharding#302) * Extended protocol interaction basic * Fix * Fix * util+gomod * Bump * Fix * Fix2 * Fix3 Accept routing hints in comments (pg-sharding#317) * Accept routing hints in comments add dataspaces support gen hz tests fix add dataspaces support Change gram.y: Add DROP DATASPACE support to parser Gen Bugfix delete DROP DATASPACE support fix regress tests and add tests to dataspace Xproto: extended query protocol interactions. (pg-sharding#302) * Extended protocol interaction basic * Fix * Fix * util+gomod * Bump * Fix * Fix2 * Fix3 Accept routing hints in comments (pg-sharding#317) * Accept routing hints in comments add dataspaces support gen tests fix fix go.sum gen Delete .DS_Store Delete .DS_Store files add dataspaces support Change gram.y: Add DROP DATASPACE support to parser Gen Bugfix delete DROP DATASPACE support Add show dataspace regress test fix regress tests and add tests to dataspace Xproto: extended query protocol interactions. (pg-sharding#302) * Extended protocol interaction basic * Fix * Fix * util+gomod * Bump * Fix * Fix2 * Fix3 Accept routing hints in comments (pg-sharding#317) * Accept routing hints in comments add dataspaces support gen tests fix Delete .DS_Store gen
Dispatch coordinator goroutines with router binary start. Proxy all modifying requests to Coordinator via grpc. Run admin console in proxy way for several type of queries: All modify requests to router console should be proxied to coordinator via particular gRPC-call. It is very dangerous to allow user to modify router metadata, like key-ranges or sharding rules, etc. This only should by done by coordinator or via coord. API. All show requests, like SHOW CLIENTS etc should be processed locally. In future, we can support installation-wide show requests processing using additional syntax, like SHOW CLIENTS is processed by router but SHOW GLOBAL CLIENTS to be proccesed by coordinator (coord. can process such queries already).
Coordinator can be run either separately or with router using
with_coordinator
flag