Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use workload identity for azure cli when Federated token file is present #4939

Merged
merged 2 commits into from
Jul 10, 2024
Merged

Use workload identity for azure cli when Federated token file is present #4939

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
89d2566
login using Azure Workload ID when available
nawazkh Jun 24, 2024
c3f6e88
Remove storage key for azure storage commands
jsturtevant Jul 2, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 3 additions & 8 deletions hack/ensure-acr-login.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,16 +22,11 @@ set +o xtrace
REPO_ROOT=$(dirname "${BASH_SOURCE[0]}")/..
cd "${REPO_ROOT}" || exit 1

if [[ "${REGISTRY:-}" =~ capzci\.azurecr\.io ]]; then
if [[ "${REGISTRY:-}" =~ \.azurecr\.io ]]; then
nawazkh marked this conversation as resolved.
Show resolved Hide resolved
# if we are using the prow Azure Container Registry, login.
"${REPO_ROOT}/hack/ensure-azcli.sh"
: "${AZURE_SUBSCRIPTION_ID:?Environment variable empty or not defined.}"
az account set -s "${AZURE_SUBSCRIPTION_ID}"
az acr login --name capzci
# TODO(mainred): When using ACR, `az acr login` impacts the authentication of `docker buildx build --push` when the
# ACR, capzci in our case, has anonymous pull enabled.
# Use `docker login` as a suggested workaround and remove this target when the issue is resolved.
# Issue link: https://github.com/Azure/acr/issues/582
# Failed building link: https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cloud-provider-azure/974/pull-cloud-provider-azure-e2e-ccm-capz/1480459040440979456
docker login -u "${AZURE_CLIENT_ID}" -p "${AZURE_CLIENT_SECRET}" capzci.azurecr.io
acrname="${REGISTRY%%.*}"
az acr login --name "$acrname"
fi
13 changes: 12 additions & 1 deletion hack/ensure-azcli.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,5 +25,16 @@ if [[ -z "$(command -v az)" ]]; then
AZ_REPO=$(lsb_release -cs)
echo "deb [arch=amd64] https://packages.microsoft.com/repos/azure-cli/ ${AZ_REPO} main" | tee /etc/apt/sources.list.d/azure-cli.list
apt-get update && apt-get install -y azure-cli
az login --service-principal -u "${AZURE_CLIENT_ID}" -p "${AZURE_CLIENT_SECRET}" --tenant "${AZURE_TENANT_ID}" > /dev/null

if [[ -n "${AZURE_FEDERATED_TOKEN_FILE:-}" ]]; then
# AZURE_CLIENT_ID has been overloaded with Azure Workload ID in the preset-azure-cred-wi.
# This is done to avoid exporting Azure Workload ID as AZURE_CLIENT_ID in the test scenarios.
az login --service-principal -u "${AZURE_CLIENT_ID}" -t "${AZURE_TENANT_ID}" --federated-token "$(cat "${AZURE_FEDERATED_TOKEN_FILE}")" > /dev/null

# Use --auth-mode "login" in az storage commands.
ENABLE_AUTH_MODE_LOGIN="true"
export ENABLE_AUTH_MODE_LOGIN
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can assign and export in one line, just fyi:

export ENABLE_AUTH_MODE_LOGIN="true"

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I got a diff to use in the upcoming CAPZ test migration PR :)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sometimes linter complains about assign + export in one line though!

else
az login --service-principal -u "${AZURE_CLIENT_ID}" -p "${AZURE_CLIENT_SECRET}" --tenant "${AZURE_TENANT_ID}" > /dev/null
fi
fi
17 changes: 8 additions & 9 deletions scripts/ci-build-azure-ccm.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,6 @@ source "${REPO_ROOT}/hack/ensure-go.sh"
source "${REPO_ROOT}/hack/parse-prow-creds.sh"

: "${AZURE_STORAGE_ACCOUNT:?Environment variable empty or not defined.}"
: "${AZURE_STORAGE_KEY:?Environment variable empty or not defined.}"
: "${REGISTRY:?Environment variable empty or not defined.}"

# cloud controller manager image
Expand Down Expand Up @@ -69,16 +68,16 @@ main() {
echo "Building and pushing Linux and Windows amd64 Azure ACR credential provider"
make -C "${AZURE_CLOUD_PROVIDER_ROOT}" bin/azure-acr-credential-provider bin/azure-acr-credential-provider.exe

if [[ "$(az storage container exists --name "${AZURE_BLOB_CONTAINER_NAME}" --query exists --output tsv)" == "false" ]]; then
if [[ "$(az storage container exists ${ENABLE_AUTH_MODE_LOGIN:+"--auth-mode login"} --name "${AZURE_BLOB_CONTAINER_NAME}" --query exists --output tsv)" == "false" ]]; then
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

${ENABLE_AUTH_MODE_LOGIN:+"--auth-mode login"} is the penultimate workaround in the list of exceptions.
SC2086#exceptions

Note that the value of ENABLE_AUTH_MODE_LOGIN immaterial as long as it is set for ${ENABLE_AUTH_MODE_LOGIN:+"--auth-mode login"} evaluate to "--auth-mode login"

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TIL

echo "Creating ${AZURE_BLOB_CONTAINER_NAME} storage container"
az storage container create --name "${AZURE_BLOB_CONTAINER_NAME}" > /dev/null
az storage container set-permission --name "${AZURE_BLOB_CONTAINER_NAME}" --public-access container > /dev/null
az storage container create ${ENABLE_AUTH_MODE_LOGIN:+"--auth-mode login"} --name "${AZURE_BLOB_CONTAINER_NAME}" > /dev/null
az storage container set-permission ${ENABLE_AUTH_MODE_LOGIN:+"--auth-mode login"} --name "${AZURE_BLOB_CONTAINER_NAME}" --public-access container > /dev/null
fi

az storage blob upload --overwrite --container-name "${AZURE_BLOB_CONTAINER_NAME}" --file "${AZURE_CLOUD_PROVIDER_ROOT}/bin/azure-acr-credential-provider" --name "${IMAGE_TAG_ACR_CREDENTIAL_PROVIDER}/azure-acr-credential-provider"
az storage blob upload --overwrite --container-name "${AZURE_BLOB_CONTAINER_NAME}" --file "${AZURE_CLOUD_PROVIDER_ROOT}/bin/azure-acr-credential-provider.exe" --name "${IMAGE_TAG_ACR_CREDENTIAL_PROVIDER}/azure-acr-credential-provider.exe"
az storage blob upload --overwrite --container-name "${AZURE_BLOB_CONTAINER_NAME}" --file "${AZURE_CLOUD_PROVIDER_ROOT}/examples/out-of-tree/credential-provider-config.yaml" --name "${IMAGE_TAG_ACR_CREDENTIAL_PROVIDER}/credential-provider-config.yaml"
az storage blob upload --overwrite --container-name "${AZURE_BLOB_CONTAINER_NAME}" --file "${AZURE_CLOUD_PROVIDER_ROOT}/examples/out-of-tree/credential-provider-config-win.yaml" --name "${IMAGE_TAG_ACR_CREDENTIAL_PROVIDER}/credential-provider-config-win.yaml"
az storage blob upload ${ENABLE_AUTH_MODE_LOGIN:+"--auth-mode login"} --overwrite --container-name "${AZURE_BLOB_CONTAINER_NAME}" --file "${AZURE_CLOUD_PROVIDER_ROOT}/bin/azure-acr-credential-provider" --name "${IMAGE_TAG_ACR_CREDENTIAL_PROVIDER}/azure-acr-credential-provider"
az storage blob upload ${ENABLE_AUTH_MODE_LOGIN:+"--auth-mode login"} --overwrite --container-name "${AZURE_BLOB_CONTAINER_NAME}" --file "${AZURE_CLOUD_PROVIDER_ROOT}/bin/azure-acr-credential-provider.exe" --name "${IMAGE_TAG_ACR_CREDENTIAL_PROVIDER}/azure-acr-credential-provider.exe"
az storage blob upload ${ENABLE_AUTH_MODE_LOGIN:+"--auth-mode login"} --overwrite --container-name "${AZURE_BLOB_CONTAINER_NAME}" --file "${AZURE_CLOUD_PROVIDER_ROOT}/examples/out-of-tree/credential-provider-config.yaml" --name "${IMAGE_TAG_ACR_CREDENTIAL_PROVIDER}/credential-provider-config.yaml"
az storage blob upload ${ENABLE_AUTH_MODE_LOGIN:+"--auth-mode login"} --overwrite --container-name "${AZURE_BLOB_CONTAINER_NAME}" --file "${AZURE_CLOUD_PROVIDER_ROOT}/examples/out-of-tree/credential-provider-config-win.yaml" --name "${IMAGE_TAG_ACR_CREDENTIAL_PROVIDER}/credential-provider-config-win.yaml"
fi
}

Expand All @@ -102,7 +101,7 @@ can_reuse_artifacts() {
fi

for BINARY in azure-acr-credential-provider azure-acr-credential-provider.exe credential-provider-config.yaml credential-provider-config-win.yaml; do
if [[ "$(az storage blob exists --container-name "${AZURE_BLOB_CONTAINER_NAME}" --name "${IMAGE_TAG_ACR_CREDENTIAL_PROVIDER}/${BINARY}" --query exists --output tsv)" == "false" ]]; then
if [[ "$(az storage blob exists ${ENABLE_AUTH_MODE_LOGIN:+"--auth-mode login"} --container-name "${AZURE_BLOB_CONTAINER_NAME}" --name "${IMAGE_TAG_ACR_CREDENTIAL_PROVIDER}/${BINARY}" --query exists --output tsv)" == "false" ]]; then
echo "false" && return
fi
done
Expand Down
15 changes: 7 additions & 8 deletions scripts/ci-build-kubernetes.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,6 @@ source "${REPO_ROOT}/hack/parse-prow-creds.sh"
source "${REPO_ROOT}/hack/util.sh"

: "${AZURE_STORAGE_ACCOUNT:?Environment variable empty or not defined.}"
: "${AZURE_STORAGE_KEY:?Environment variable empty or not defined.}"
: "${REGISTRY:?Environment variable empty or not defined.}"

declare -a BINARIES=("kubeadm" "kubectl" "kubelet" "e2e.test")
Expand Down Expand Up @@ -80,10 +79,10 @@ setup() {
}

main() {
if [[ "$(az storage container exists --name "${AZURE_BLOB_CONTAINER_NAME}" --query exists --output tsv)" == "false" ]]; then
if [[ "$(az storage container exists ${ENABLE_AUTH_MODE_LOGIN:+"--auth-mode login"} --name "${AZURE_BLOB_CONTAINER_NAME}" --query exists --output tsv)" == "false" ]]; then
echo "Creating ${AZURE_BLOB_CONTAINER_NAME} storage container"
az storage container create --name "${AZURE_BLOB_CONTAINER_NAME}" > /dev/null
az storage container set-permission --name "${AZURE_BLOB_CONTAINER_NAME}" --public-access container > /dev/null
az storage container ${ENABLE_AUTH_MODE_LOGIN:+"--auth-mode login"} create --name "${AZURE_BLOB_CONTAINER_NAME}" > /dev/null
az storage container ${ENABLE_AUTH_MODE_LOGIN:+"--auth-mode login"} set-permission --name "${AZURE_BLOB_CONTAINER_NAME}" --public-access container > /dev/null
fi

if [[ "${KUBE_BUILD_CONFORMANCE:-}" =~ [yY] ]]; then
Expand Down Expand Up @@ -116,7 +115,7 @@ main() {
for BINARY in "${BINARIES[@]}"; do
BIN_PATH="${KUBE_GIT_VERSION}/bin/linux/amd64/${BINARY}"
echo "uploading ${BIN_PATH}"
az storage blob upload --overwrite --container-name "${AZURE_BLOB_CONTAINER_NAME}" --file "${KUBE_ROOT}/_output/dockerized/bin/linux/amd64/${BINARY}" --name "${BIN_PATH}"
az storage blob upload ${ENABLE_AUTH_MODE_LOGIN:+"--auth-mode login"} --overwrite --container-name "${AZURE_BLOB_CONTAINER_NAME}" --file "${KUBE_ROOT}/_output/dockerized/bin/linux/amd64/${BINARY}" --name "${BIN_PATH}"
done

if [[ "${TEST_WINDOWS:-}" == "true" ]]; then
Expand All @@ -129,7 +128,7 @@ main() {
for BINARY in "${WINDOWS_BINARIES[@]}"; do
BIN_PATH="${KUBE_GIT_VERSION}/bin/windows/amd64/${BINARY}.exe"
echo "uploading ${BIN_PATH}"
az storage blob upload --overwrite --container-name "${AZURE_BLOB_CONTAINER_NAME}" --file "${KUBE_ROOT}/_output/dockerized/bin/windows/amd64/${BINARY}.exe" --name "${BIN_PATH}"
az storage blob upload ${ENABLE_AUTH_MODE_LOGIN:+"--auth-mode login"} --overwrite --container-name "${AZURE_BLOB_CONTAINER_NAME}" --file "${KUBE_ROOT}/_output/dockerized/bin/windows/amd64/${BINARY}.exe" --name "${BIN_PATH}"
done
fi
fi
Expand All @@ -144,14 +143,14 @@ can_reuse_artifacts() {
done

for BINARY in "${BINARIES[@]}"; do
if [[ "$(az storage blob exists --container-name "${AZURE_BLOB_CONTAINER_NAME}" --name "${KUBE_GIT_VERSION}/bin/linux/amd64/${BINARY}" --query exists --output tsv)" == "false" ]]; then
if [[ "$(az storage blob exists ${ENABLE_AUTH_MODE_LOGIN:+"--auth-mode login"} --container-name "${AZURE_BLOB_CONTAINER_NAME}" --name "${KUBE_GIT_VERSION}/bin/linux/amd64/${BINARY}" --query exists --output tsv)" == "false" ]]; then
echo "false" && return
fi
done

if [[ "${TEST_WINDOWS:-}" == "true" ]]; then
for BINARY in "${WINDOWS_BINARIES[@]}"; do
if [[ "$(az storage blob exists --container-name "${AZURE_BLOB_CONTAINER_NAME}" --name "${KUBE_GIT_VERSION}/bin/windows/amd64/${BINARY}.exe" --query exists --output tsv)" == "false" ]]; then
if [[ "$(az storage blob exists ${ENABLE_AUTH_MODE_LOGIN:+"--auth-mode login"} --container-name "${AZURE_BLOB_CONTAINER_NAME}" --name "${KUBE_GIT_VERSION}/bin/windows/amd64/${BINARY}.exe" --query exists --output tsv)" == "false" ]]; then
echo "false" && return
fi
done
Expand Down
22 changes: 17 additions & 5 deletions scripts/kind-with-registry.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -93,22 +93,27 @@ function checkAZWIENVPreReqsAndCreateFiles() {
echo "AZWI_RESOURCE_GROUP environment variable required - Azure resource group to store required Workload Identity artifacts"
exit 1
fi

if [ "$(az group exists --name "${AZWI_RESOURCE_GROUP}" --output tsv)" == 'false' ]; then
echo "Creating resource group '${AZWI_RESOURCE_GROUP}' in '${AZWI_LOCATION}'"
az group create --name "${AZWI_RESOURCE_GROUP}" --location "${AZWI_LOCATION}" --output none --only-show-errors --tags creationTimestamp="${TIMESTAMP}" jobName="${JOB_NAME}" buildProvenance="${BUILD_PROVENANCE}"
fi

# Ensure that our connection to storage is inherited from the existing Azure login context
unset AZURE_STORAGE_KEY
unset AZURE_STORAGE_ACCOUNT

if ! az storage account show --name "${AZWI_STORAGE_ACCOUNT}" --resource-group "${AZWI_RESOURCE_GROUP}" > /dev/null 2>&1; then
echo "Creating storage account '${AZWI_STORAGE_ACCOUNT}' in '${AZWI_RESOURCE_GROUP}'"
az storage account create --resource-group "${AZWI_RESOURCE_GROUP}" --name "${AZWI_STORAGE_ACCOUNT}" --output none --only-show-errors --tags creationTimestamp="${TIMESTAMP}" jobName="${JOB_NAME}" buildProvenance="${BUILD_PROVENANCE}"
az storage blob service-properties update --account-name "${AZWI_STORAGE_ACCOUNT}" --static-website
az storage blob service-properties ${ENABLE_AUTH_MODE_LOGIN:+"--auth-mode login"} update --account-name "${AZWI_STORAGE_ACCOUNT}" --static-website
fi

if ! az storage container show --name "${AZWI_STORAGE_CONTAINER}" --account-name "${AZWI_STORAGE_ACCOUNT}" > /dev/null 2>&1; then
echo "Creating storage container '${AZWI_STORAGE_CONTAINER}' in '${AZWI_STORAGE_ACCOUNT}'"
az storage container create --name "${AZWI_STORAGE_CONTAINER}" --account-name "${AZWI_STORAGE_ACCOUNT}" --output none --only-show-errors
az storage container ${ENABLE_AUTH_MODE_LOGIN:+"--auth-mode login"} create --name "${AZWI_STORAGE_CONTAINER}" --account-name "${AZWI_STORAGE_ACCOUNT}" --output none --only-show-errors
fi

SERVICE_ACCOUNT_ISSUER=$(az storage account show --name "${AZWI_STORAGE_ACCOUNT}" -o json | jq -r .primaryEndpoints.web)
export SERVICE_ACCOUNT_ISSUER
AZWI_OPENID_CONFIG_FILEPATH="${REPO_ROOT}/openid-configuration.json"
Expand All @@ -131,30 +136,37 @@ EOF
openssl rsa -in "${SERVICE_ACCOUNT_SIGNING_KEY_FILEPATH}" -pubout -out "${SERVICE_ACCOUNT_SIGNING_PUB_FILEPATH}"
AZWI_JWKS_JSON_FILEPATH="${REPO_ROOT}/jwks.json"
"${AZWI}" jwks --public-keys "${SERVICE_ACCOUNT_SIGNING_PUB_FILEPATH}" --output-file "${AZWI_JWKS_JSON_FILEPATH}"

echo "Uploading openid-configuration document to '${AZWI_STORAGE_ACCOUNT}' storage account"
upload_to_blob "${AZWI_OPENID_CONFIG_FILEPATH}" ".well-known/openid-configuration"

echo "Uploading jwks document to '${AZWI_STORAGE_ACCOUNT}' storage account"
upload_to_blob "${AZWI_JWKS_JSON_FILEPATH}" "openid/v1/jwks"
echo "Removing key access on storage account as no further data writes are required"
az storage account update -n "${AZWI_STORAGE_ACCOUNT}" -g "${AZWI_RESOURCE_GROUP}" --subscription "${AZURE_SUBSCRIPTION_ID}" --allow-shared-key-access=false --output none --only-show-errors
fi

if [ -z "${AZURE_CLIENT_ID_USER_ASSIGNED_IDENTITY}" ]; then
if [ -z "${USER_IDENTITY}" ]; then
echo "USER_IDENTITY environment variable required if not bringing your own identity via AZURE_CLIENT_ID_USER_ASSIGNED_IDENTITY"
exit 1
fi

az identity create -n "${USER_IDENTITY}" -g "${AZWI_RESOURCE_GROUP}" -l "${AZWI_LOCATION}" --output none --only-show-errors --tags creationTimestamp="${TIMESTAMP}" jobName="${JOB_NAME}" buildProvenance="${BUILD_PROVENANCE}"
AZURE_IDENTITY_ID=$(az identity show -n "${USER_IDENTITY}" -g "${AZWI_RESOURCE_GROUP}" --query clientId -o tsv)
AZURE_IDENTITY_ID_PRINCIPAL_ID=$(az identity show -n "${USER_IDENTITY}" -g "${AZWI_RESOURCE_GROUP}" --query principalId -o tsv)

echo "${AZURE_IDENTITY_ID}" > "${AZURE_IDENTITY_ID_FILEPATH}"
until az role assignment create --assignee-object-id "${AZURE_IDENTITY_ID_PRINCIPAL_ID}" --role "Owner" --scope "/subscriptions/${AZURE_SUBSCRIPTION_ID}" --assignee-principal-type ServicePrincipal --output none --only-show-errors; do
sleep 5
done

echo "Creating federated credentials for capz-federated-identity"
az identity federated-credential create -n "capz-federated-identity" \
--identity-name "${USER_IDENTITY}" \
-g "${AZWI_RESOURCE_GROUP}" \
--issuer "${SERVICE_ACCOUNT_ISSUER}" \
--subject "system:serviceaccount:capz-system:capz-manager" --output none --only-show-errors

echo "Creating federated credentials for aso-federated-identity"
az identity federated-credential create -n "aso-federated-identity" \
--identity-name "${USER_IDENTITY}" \
-g "${AZWI_RESOURCE_GROUP}" \
Expand All @@ -168,7 +180,7 @@ function upload_to_blob() {
local blob_name=$2

echo "Uploading ${file_path} to '${AZWI_STORAGE_ACCOUNT}' storage account"
az storage blob upload \
az storage blob upload ${ENABLE_AUTH_MODE_LOGIN:+"--auth-mode login"} \
--container-name "${AZWI_STORAGE_CONTAINER}" \
--file "${file_path}" \
--name "${blob_name}" \
Expand Down