Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cover bottlerocket cluster test. #1096

Merged
merged 9 commits into from
Jul 21, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .circleci/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -73,7 +73,7 @@ jobs:
- run:
name: Run the integration tests
command: ./scripts/run-integration-tests.sh
no_output_timeout: 15m
no_output_timeout: 20m
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, it's that slow?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

eksctl can sometimes take longer than 15m to create the cluster and mng, so it did timeout once. Haven't seen any problems after moving it to 20m!

- save_cache:
key: dependency-packages-store-{{ checksum "test/integration/go.mod" }}
paths:
Expand Down
5 changes: 5 additions & 0 deletions scripts/lib/aws.sh
Original file line number Diff line number Diff line change
Expand Up @@ -26,3 +26,8 @@ ensure_aws_k8s_tester() {
chmod +x $TESTER_PATH
fi
}

ensure_eksctl() {
curl --silent --location "https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz" | tar xz -C /tmp
sudo mv -v /tmp/eksctl /usr/local/bin
}
35 changes: 26 additions & 9 deletions scripts/run-integration-tests.sh
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ ARCH=$(go env GOARCH)
: "${BUILD:=true}"
: "${RUN_CONFORMANCE:=false}"
: "${RUN_KOPS_TEST:=false}"
: "${RUN_BOTTLEROCKET_TEST:=false}"

__cluster_created=0
__cluster_deprovisioned=0
Expand All @@ -29,13 +30,15 @@ on_error() {
# Make sure we destroy any cluster that was created if we hit run into an
# error when attempting to run tests against the cluster
if [[ $__cluster_created -eq 1 && $__cluster_deprovisioned -eq 0 && "$DEPROVISION" == true ]]; then
echo "Cluster was provisioned already. Deprovisioning it..."
__cluster_deprovisioned=1
if [[ $RUN_KOPS_TEST == true ]]; then
__cluster_deprovisioned=1
echo "Cluster was provisioned already. Deprovisioning it..."
down-kops-cluster
elif [[ $RUN_BOTTLEROCKET_TEST == true ]]; then
eksctl delete cluster bottlerocket
else
# prevent double-deprovisioning with ctrl-c during deprovisioning...
__cluster_deprovisioned=1
echo "Cluster was provisioned already. Deprovisioning it..."
down-test-cluster
fi
Expand Down Expand Up @@ -151,10 +154,18 @@ mkdir -p "$TEST_CLUSTER_DIR"
mkdir -p "$TEST_CONFIG_DIR"

START=$SECONDS
if [[ "$PROVISION" == true && "$RUN_KOPS_TEST" == true ]]; then
up-kops-cluster
elif [[ "$PROVISION" == true ]]; then
up-test-cluster
if [[ "$PROVISION" == true ]]; then
START=$SECONDS
if [[ "$RUN_BOTTLEROCKET_TEST" == true ]]; then
ensure_eksctl
eksctl create cluster --config-file ./testdata/bottlerocket.yaml
elif [[ "$RUN_KOPS_TEST" == true ]]; then
up-kops-cluster
else
up-test-cluster
fi
UP_CLUSTER_DURATION=$((SECONDS - START))
echo "TIMELINE: Upping test cluster took $UP_CLUSTER_DURATION seconds."
fi
__cluster_created=1

Expand All @@ -170,11 +181,15 @@ sed -i'.bak' "s,:$MANIFEST_IMAGE_VERSION,:$TEST_IMAGE_VERSION," "$TEST_CONFIG_PA
sed -i'.bak' "s,602401143452.dkr.ecr.us-west-2.amazonaws.com/amazon-k8s-cni-init,$INIT_IMAGE_NAME," "$TEST_CONFIG_PATH"
sed -i'.bak' "s,:$MANIFEST_IMAGE_VERSION,:$TEST_IMAGE_VERSION," "$TEST_CONFIG_PATH"

if [[ $RUN_KOPS_TEST != true ]]; then
export KUBECONFIG=$KUBECONFIG_PATH
if [[ $RUN_KOPS_TEST == true || $RUN_BOTTLEROCKET_TEST == true ]]; then
KUBECTL_PATH=kubectl
export KUBECONFIG=~/.kube/config
Comment on lines +184 to +186
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, so both Kops and eksctl needs these to be set.

else
export KUBECONFIG=$KUBECONFIG_PATH
fi

if [[ $RUN_KOPS_TEST == true ]]; then
run_kops_conformance
KUBECTL_PATH=kubectl
fi
ADDONS_CNI_IMAGE=$($KUBECTL_PATH describe daemonset aws-node -n kube-system | grep Image | cut -d ":" -f 2-3 | tr -d '[:space:]')

Expand Down Expand Up @@ -239,6 +254,8 @@ if [[ "$DEPROVISION" == true ]]; then

if [[ "$RUN_KOPS_TEST" == true ]]; then
down-kops-cluster
elif [[ "$RUN_BOTTLEROCKET_TEST" == true ]]; then
eksctl delete cluster bottlerocket
else
down-test-cluster
fi
Expand Down
23 changes: 23 additions & 0 deletions testdata/bottlerocket.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
---
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
name: bottlerocket
region: us-west-2
version: '1.15'

nodeGroups:
- name: ng-bottlerocket
instanceType: m5.large
desiredCapacity: 4
amiFamily: Bottlerocket
iam:
attachPolicyARNs:
- arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy
- arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy
- arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly
- arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore
bottlerocket:
settings:
motd: "Hello from eksctl!"