Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Online DDL via VReplication #7419

Merged
merged 74 commits into from
Mar 1, 2021
Merged
Show file tree
Hide file tree
Changes from 56 commits
Commits
Show all changes
74 commits
Select commit Hold shift + click to select a range
c67648f
Towards VReplication based Online DDL
shlomi-noach Jan 28, 2021
56cb0d6
testing for DDLStrategyOnline
shlomi-noach Jan 28, 2021
379d7ec
import from gh-ost and iterating
shlomi-noach Jan 31, 2021
52d3e85
towards vreplication based online DDL
shlomi-noach Jan 31, 2021
4f3fdc4
cleanup
shlomi-noach Jan 31, 2021
df0905c
call generalized 'analyze'
shlomi-noach Jan 31, 2021
29895c6
kick schema screation as soon as Open()
shlomi-noach Jan 31, 2021
c37f29a
generate vreplication insert and start statements
shlomi-noach Jan 31, 2021
b8409d6
communicating to VREngine through gRPC
shlomi-noach Jan 31, 2021
56b7a59
refactor parsing alterOptions; in the future this will go through sql…
shlomi-noach Jan 31, 2021
da96194
simplified queries in onlineddl
shlomi-noach Jan 31, 2021
d6d8eed
init schema in InitDBConfig()
shlomi-noach Feb 1, 2021
c6f758d
mutex protestion
shlomi-noach Feb 1, 2021
54428fe
whoops; mutex was already there
shlomi-noach Feb 1, 2021
6784448
fix chech type. Backport of https://github.com/vitessio/vitess/pull/7422
shlomi-noach Feb 1, 2021
5a620da
engine's early schema creation did not work as expected. Not improtan…
shlomi-noach Feb 1, 2021
82a0aa0
check vreplication liveness, cancel vreplication migration
shlomi-noach Feb 1, 2021
f825e56
check if vreplication migration is ready to cut-over
shlomi-noach Feb 2, 2021
1781640
added index on 'workflow' column in vreplication
shlomi-noach Feb 2, 2021
24b3899
refactor schemaMigrationsTableName outside vexec and into go/vt/schema
shlomi-noach Feb 3, 2021
5693c13
new interface for VExec executors on tablet
shlomi-noach Feb 3, 2021
5d1cad0
use new vexec.Executor interface
shlomi-noach Feb 3, 2021
691096a
VReplStream struct
shlomi-noach Feb 3, 2021
c6487c1
refactor SchemaMigrationsTableName to go/vt/schema. Generate table sw…
shlomi-noach Feb 3, 2021
9c2dd6c
check if vreplication DDL is ready for cutover; cut-over by stopping …
shlomi-noach Feb 3, 2021
3ba8845
WithDDL: force apply the schema on first use
shlomi-noach Feb 3, 2021
6ce7564
smaller index
shlomi-noach Feb 3, 2021
b770b0b
reuse exec function
shlomi-noach Feb 3, 2021
50e2468
cleanup
shlomi-noach Feb 3, 2021
065c677
with_ddl: fix test
shlomi-noach Feb 3, 2021
215678f
fix with_ddl_test
shlomi-noach Feb 3, 2021
ce425fd
rename
shlomi-noach Feb 3, 2021
bad3a92
support withDDLs()
shlomi-noach Feb 3, 2021
7666035
fixed engine_test: expect withDDL DDLs
shlomi-noach Feb 3, 2021
f19a80c
reverted with_ddl changes -- extracted to a different branch. This se…
shlomi-noach Feb 4, 2021
58ba93b
tabs to spaces
shlomi-noach Feb 4, 2021
e8548ab
minor refactor in VRepl, bls is a member
shlomi-noach Feb 4, 2021
2c44de6
handling vreplication migrations executed from a previous tablet
shlomi-noach Feb 4, 2021
e0dbc6d
renaming test 26 as onlineddl_ghost
shlomi-noach Feb 4, 2021
245f900
renamed workflow
shlomi-noach Feb 4, 2021
a7b277c
fixed path for onlineddl_ghost test
shlomi-noach Feb 4, 2021
30fe306
adding endtoend onlineddl_vrepl test workflow
shlomi-noach Feb 4, 2021
0ff5423
renamed test file
shlomi-noach Feb 4, 2021
81d0a1c
clarifying where online DDL tests are
shlomi-noach Feb 4, 2021
8d6705e
renamed a couple functions
shlomi-noach Feb 4, 2021
3d41289
adding onlineddl_vrepl tests
shlomi-noach Feb 4, 2021
05676f1
abort cancel when migration is in complete/failed statis
shlomi-noach Feb 4, 2021
29fa14b
Reload schema before starting vreplication migration
shlomi-noach Feb 4, 2021
d3d3e31
fixing endtoend tests
shlomi-noach Feb 4, 2021
c27ea9d
cut-over timeout for wait-for-pos and stop writes
shlomi-noach Feb 4, 2021
40063e8
fix ctx with cancel()
shlomi-noach Feb 4, 2021
86f6d2f
schema_migrations.message column
shlomi-noach Feb 8, 2021
ca5a214
merge master
shlomi-noach Feb 10, 2021
187f93d
update workflo gen
shlomi-noach Feb 10, 2021
32987c6
restored shard 26
shlomi-noach Feb 10, 2021
b358297
endtoend test: validate table data is as expected
shlomi-noach Feb 10, 2021
56b96a7
towards online ddl/vreplication mini stress tests in CI
shlomi-noach Feb 10, 2021
28e1e93
mini stress testing for vreplication based online DDL
shlomi-noach Feb 15, 2021
80e9f04
eliminating deadlock with point UPDATE and point DELETE
shlomi-noach Feb 15, 2021
a1bff1c
more metrics, count metrics in goroutine
shlomi-noach Feb 15, 2021
80c119d
multiple iterations for 'workload without ALTER TABLE'
shlomi-noach Feb 15, 2021
e5ab8b3
using waitgroup
shlomi-noach Feb 15, 2021
e5dabb4
typo
shlomi-noach Feb 15, 2021
114642b
remove transactionTimestamp evaluation
shlomi-noach Feb 15, 2021
323562b
fixes error message case
shlomi-noach Feb 15, 2021
8650ef9
no need for replicas, make test more lightweight
shlomi-noach Feb 16, 2021
f134469
wait for runMultipleConnections() to complete
shlomi-noach Feb 16, 2021
534201e
using context.WithCancel, simplify logic
shlomi-noach Feb 16, 2021
942b5be
async ReloadScema at cut-over
shlomi-noach Feb 16, 2021
61454be
Merge branch 'vreplication-online-ddl-mini-stress-test' into vreplica…
shlomi-noach Feb 16, 2021
1428eea
restore transaction_timestamp test
shlomi-noach Feb 16, 2021
7e2a074
inject dummy statements when vreplication transaction_timestamp is stale
shlomi-noach Feb 16, 2021
ed4207e
minor naming change
shlomi-noach Feb 17, 2021
7f6a800
revert 7e2a0748a50d79ca51f14bb69b65765e87376ffd: no need for dummy in…
shlomi-noach Feb 18, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 40 additions & 0 deletions .github/workflows/cluster_endtoend_onlineddl_ghost.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
# DO NOT MODIFY: THIS FILE IS GENERATED USING "make generate_ci_workflows"

name: Cluster (onlineddl_ghost)
on: [push, pull_request]
jobs:

build:
name: Run endtoend tests on Cluster (onlineddl_ghost)
runs-on: ubuntu-latest

steps:
- name: Set up Go
uses: actions/setup-go@v1
with:
go-version: 1.15

- name: Check out code
uses: actions/checkout@v2

- name: Get dependencies
run: |
sudo apt-get update
sudo apt-get install -y mysql-server mysql-client make unzip g++ etcd curl git wget eatmydata
sudo service mysql stop
sudo service etcd stop
sudo ln -s /etc/apparmor.d/usr.sbin.mysqld /etc/apparmor.d/disable/
sudo apparmor_parser -R /etc/apparmor.d/usr.sbin.mysqld
go mod download

wget https://repo.percona.com/apt/percona-release_latest.$(lsb_release -sc)_all.deb
sudo apt-get install -y gnupg2
sudo dpkg -i percona-release_latest.$(lsb_release -sc)_all.deb
sudo apt-get update
sudo apt-get install percona-xtrabackup-24

- name: Run cluster endtoend test
timeout-minutes: 30
run: |
source build.env
eatmydata -- go run test.go -docker=false -print-log -follow -shard onlineddl_ghost
40 changes: 40 additions & 0 deletions .github/workflows/cluster_endtoend_onlineddl_vrepl.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
# DO NOT MODIFY: THIS FILE IS GENERATED USING "make generate_ci_workflows"

name: Cluster (onlineddl_vrepl)
on: [push, pull_request]
jobs:

build:
name: Run endtoend tests on Cluster (onlineddl_vrepl)
runs-on: ubuntu-latest

steps:
- name: Set up Go
uses: actions/setup-go@v1
with:
go-version: 1.15

- name: Check out code
uses: actions/checkout@v2

- name: Get dependencies
run: |
sudo apt-get update
sudo apt-get install -y mysql-server mysql-client make unzip g++ etcd curl git wget eatmydata
sudo service mysql stop
sudo service etcd stop
sudo ln -s /etc/apparmor.d/usr.sbin.mysqld /etc/apparmor.d/disable/
sudo apparmor_parser -R /etc/apparmor.d/usr.sbin.mysqld
go mod download

wget https://repo.percona.com/apt/percona-release_latest.$(lsb_release -sc)_all.deb
sudo apt-get install -y gnupg2
sudo dpkg -i percona-release_latest.$(lsb_release -sc)_all.deb
sudo apt-get update
sudo apt-get install percona-xtrabackup-24

- name: Run cluster endtoend test
timeout-minutes: 30
run: |
source build.env
eatmydata -- go run test.go -docker=false -print-log -follow -shard onlineddl_vrepl
Loading