Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support running stability test out of cluster #397

Merged
merged 9 commits into from
Apr 16, 2019

Conversation

aylei
Copy link
Contributor

@aylei aylei commented Apr 16, 2019

Deploy & witness flow can be tedious when developing stability-test, this PR introduce the ability to run stability-test out of the cluster(your local machine, usually) while still operating the remote cluster.

How-to: generally we have three problems to solve:

TL;DR:

$ telepresence --new-deployment ${POD_NAME}
$ go build -o stability ./tests/cmd/stability/main.go
$ ./stability --operator-repo-dir=${ABITRARY_EMPTY_DIR_TO_CLONE_OPERATOR_REPO}

Explained

  1. Out of cluster client: Now we try to load configs in the following order:
    • if kubeconfig command line option provided, use it
    • if KUBECONFIG env variable set, use it
    • try loading InClusterConfig()
    • if no InClusterConfig() provided, try loading kubeconfig file from default location (~/.kube/config)
      so typically you will get right client configs if you have proper ~/.kube/config in your local environment.
  2. Privilege issue: If you don't want to or cannot run stability test with root privilege, change the working dir or create it in advance:
    • git repo dir can be overridden by option --git-repo-dir=xxxx, but helm dir must be created manually.
# helm dir
$ mkdir /charts
$ chmod 777 /charts
# git repo dir if you don't set command line option
$ mkdir /tidb-operator
$ chmod 777 /tidb-operator
  1. DNS and network issue: Two-way proxy using Telepresence. We cannot resolve cluster dns name and access cluster ip easily, telepresence helps with that, it creates a proxy pod in the cluster and open a vpn connection to kubernetes cluster via this pod. Just run (full documentations):
$ brew cask install osxfuse
$ brew install datawire/blackbird/telepresence
$ telepresence --new-deployment ${POD_NAME}

PS: If you cannot resolve cluster dns names after set up, try clear DNS cache.
PSS: Typically you can't use telepresence VPN mode with other VPNs (of course SSR is ok).

Signed-off-by: Aylei rayingecho@gmail.com

Signed-off-by: Aylei <rayingecho@gmail.com>
Signed-off-by: Aylei <rayingecho@gmail.com>
tests/config.go Outdated Show resolved Hide resolved
tests/config.go Outdated Show resolved Hide resolved
tests/config.go Outdated Show resolved Hide resolved
Signed-off-by: Aylei <rayingecho@gmail.com>
@aylei
Copy link
Contributor Author

aylei commented Apr 16, 2019

@cofyc @onlymellb @xiaojingchen PTAL again

@aylei aylei requested review from cofyc, xiaojingchen and onlymellb and removed request for cofyc April 16, 2019 06:08
tests/config.go Outdated Show resolved Hide resolved
Signed-off-by: Aylei <rayingecho@gmail.com>
@aylei
Copy link
Contributor Author

aylei commented Apr 16, 2019

I've updated the docs too

Signed-off-by: Aylei <rayingecho@gmail.com>
@aylei aylei requested a review from xiaojingchen April 16, 2019 06:23
Signed-off-by: Aylei <rayingecho@gmail.com>
onlymellb
onlymellb previously approved these changes Apr 16, 2019
Copy link
Contributor

@onlymellb onlymellb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

func checkoutTag(tagName string) error {
cmd := fmt.Sprintf(`cd /tidb-operator &&
func (oa *operatorActions) checkoutTag(tagName string) error {
cmd := fmt.Sprintf(`cd %s &&
git stash -u &&
git checkout %s &&
mkdir -p /charts/%s &&
cp -rf charts/tidb-operator /charts/%s/tidb-operator &&
cp -rf charts/tidb-cluster /charts/%s/tidb-cluster &&
cp -rf charts/tidb-backup /charts/%s/tidb-backup`,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(do not block this PR) Is it possible to copy them to temporary directories which are always writable in the future? I guess the path does not matter here. Then we can simplify the workflow (do not need to prepare /charts).

Copy link
Contributor Author

@aylei aylei Apr 16, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed for future work, I've tried but found that /charts is hard coded in many places😅

…y provided

Signed-off-by: Aylei <rayingecho@gmail.com>
@aylei
Copy link
Contributor Author

aylei commented Apr 16, 2019

@cofyc @xiaojingchen @weekface @onlymellb I've updated the doc, PTAL again.

Copy link
Contributor

@onlymellb onlymellb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@weekface
Copy link
Contributor

/run-e2e-tests

1 similar comment
@aylei
Copy link
Contributor Author

aylei commented Apr 16, 2019

/run-e2e-tests

@aylei aylei merged commit b1c5ec6 into pingcap:master Apr 16, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants