Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Track driver deploy time in e2e test pipeline #815

Merged
merged 1 commit into from
Mar 24, 2021
Merged

Track driver deploy time in e2e test pipeline #815

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions hack/e2e/run.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ source "${BASE_DIR}"/util.sh

DRIVER_NAME=${DRIVER_NAME:-aws-ebs-csi-driver}
CONTAINER_NAME=${CONTAINER_NAME:-ebs-plugin}
DRIVER_START_TIME_THRESHOLD_SECONDS=60
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice :)


TEST_ID=${TEST_ID:-$RANDOM}
CLUSTER_NAME=test-cluster-${TEST_ID}.k8s.local
Expand Down Expand Up @@ -104,18 +105,29 @@ if [[ $? -ne 0 ]]; then
fi

loudecho "Deploying driver"
startSec=$(date +'%s')
"${HELM_BIN}" upgrade --install "${DRIVER_NAME}" \
--namespace kube-system \
--set image.repository="${IMAGE_NAME}" \
--set image.tag="${IMAGE_TAG}" \
-f "${HELM_VALUES_FILE}" \
--wait \
./charts/"${DRIVER_NAME}"

if [[ -r "${EBS_SNAPSHOT_CRD}" ]]; then
loudecho "Deploying snapshot CRD"
kubectl apply -f "$EBS_SNAPSHOT_CRD"
# TODO deploy snapshot controller too instead of including in helm chart
fi
endSec=$(date +'%s')
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

have to wait for the pods to become ready. not so easy in bash to be honest.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah agree, I did some research but no luck to find any tool to track container start up time.
But one thing is if we use helm --wait flag, helm will wait for containersReady condition before exit. IMO It would be good enough as for now.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And open for any suggestion to track this info :-)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ooh yeah let's use helm built-in functionality --wait flag, that is cool if they have it.

Otherwise I was thinking of how to call a python/go script from here and use the kube python/go client, which would be really ugly...

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we can use cloudwatch? The tests run in an AWS-internal account. And if running locally, i don't mind pushing metrics to my own account. We can fail-open in case for whatever reason the metric push fails (transient cloudwatch issue or whatever)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's a good find. Does it wait for everything deployed by the chart to be ready?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ideally there is a way to expose the metric publicly though, cloudwatch wont be visible like https://testgrid.k8s.io/provider-aws-efs-csi-driver#e2e-test&width=20 is.

Copy link
Contributor Author

@AndyXiangLi AndyXiangLi Mar 24, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it will wait for all the containers to be ready
https://helm.sh/docs/intro/using_helm/

secondUsed=$(( (endSec-startSec)/1 ))
# Set timeout threshold as 20 seconds for now, usually it takes less than 10s to startup
if [ $secondUsed -gt $DRIVER_START_TIME_THRESHOLD_SECONDS ]; then
loudecho "Driver start timeout, took $secondUsed but the threshold is $DRIVER_START_TIME_THRESHOLD_SECONDS. Fail the test."
exit 1
fi
loudecho "Driver deployment complete, time used: $secondUsed seconds"


loudecho "Testing focus ${GINKGO_FOCUS}"
eval "EXPANDED_TEST_EXTRA_FLAGS=$TEST_EXTRA_FLAGS"
Expand Down