diff --git a/docs/clusterdefinition.md b/docs/clusterdefinition.md index a12583561e..e1bfc22534 100644 --- a/docs/clusterdefinition.md +++ b/docs/clusterdefinition.md @@ -497,8 +497,9 @@ We consider `kubeletConfig`, `controllerManagerConfig`, `apiServerConfig`, and ` | imageReference.resourceGroup | no | Resource group that contains the Linux OS image. Needs to be used in conjunction with name, above | | distro | no | Select Master(s) Operating System (Linux only). Currently supported values are: `ubuntu`, `aks` and `coreos` (CoreOS support is currently experimental). Defaults to `aks` if undefined. `aks` is a custom image based on `ubuntu` that comes with pre-installed software necessary for Kubernetes deployments (Azure Public Cloud only for now). Currently supported OS and orchestrator configurations -- `ubuntu` and `aks`: DCOS, Docker Swarm, Kubernetes; `RHEL`: OpenShift; `coreos`: Kubernetes. [Example of CoreOS Master with CoreOS Agents](../examples/coreos/kubernetes-coreos.json) | | customFiles | no | The custom files to be provisioned to the master nodes. Defined as an array of json objects with each defined as `"source":"absolute-local-path", "dest":"absolute-path-on-masternodes"`.[See examples](../examples/customfiles) | -| availabilityProfile | no | Supported values are `AvailabilitySet` (default) and `VirtualMachineScaleSets` (requires Kubernetes clusters version 1.10+ and agent pool availabilityProfile must also be `VirtualMachineScaleSets`). | +| availabilityProfile | no | Supported values are `AvailabilitySet` (default) and `VirtualMachineScaleSets` (requires Kubernetes clusters version 1.10+ and agent pool availabilityProfile must also be `VirtualMachineScaleSets`). When MasterProfile is using `VirtualMachineScaleSets`, to SSH into a master node, you need to use `ssh -p 50001` instead of port 22. | | agentVnetSubnetId | only required when using custom VNET and when MasterProfile is using `VirtualMachineScaleSets` | Specifies the Id of an alternate VNET subnet for all the agent pool nodes. The subnet id must specify a valid VNET ID owned by the same subscription. ([bring your own VNET examples](../examples/vnet)). When MasterProfile is using `VirtualMachineScaleSets`, this value should be the subnetId of the subnet for all agent pool nodes. | +| [availabilityZones](../examples/kubernetes-zones/README.md) | no | To protect your cluster from datacenter-level failures, you can enable the Availability Zones feature for your cluster by configuring `"availabilityZones"` for the master profile and all of the agentPool profiles in the cluster definition. Check out [Availability Zones README](../examples/kubernetes-zones/README.md) for more details. | ### agentPoolProfiles @@ -508,7 +509,7 @@ A cluster can have 0 to 12 agent pool profiles. Agent Pool Profiles are used for | ---------------------------- | -------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | availabilityProfile | no | Supported values are `VirtualMachineScaleSets` (default, except for Kubernetes clusters before version 1.10) and `AvailabilitySet`. | | count | yes | Describes the node count | -| availabilityZones | no | To protect your cluster from datacenter-level failures, you can enable the Availability Zones feature for your cluster by configuring `"availabilityZones"` for the master profile and all of the agentPool profiles in the cluster definition. This feature only applies to Kubernetes clusters version 1.12+. Supported values are arrays of strings, each representing a supported availability zone in a region for your subscription. e.g. `"availabilityZones": ["1","2"]` represents zone 1 and zone 2 can be used. To get supported zones for a region in your subscription, run `az vm list-skus --location centralus --query "[?name=='Standard_DS2_v2'].[locationInfo, restrictions"] -o table`. You should see values like `'zones': ['2', '3', '1']` appear in the first column. If `NotAvailableForSubscription` appears in the output, then create an Azure support ticket to enable zones for that region. NOTE: To ensure high availability, each profile must define at least two nodes per zone. e.g. An agent pool profile with 2 zones: `"availabilityZones": ["1","2"]` must have at least 4 nodes total with `"count": 4`. When `"availabilityZones"` is configured, the `"loadBalancerSku"` will default to `Standard` as Standard LoadBalancer is required for availability zones. | +| [availabilityZones](../examples/kubernetes-zones/README.md) | no | To protect your cluster from datacenter-level failures, you can enable the Availability Zones feature for your cluster by configuring `"availabilityZones"` for the master profile and all of the agentPool profiles in the cluster definition. Check out [Availability Zones README](../examples/kubernetes-zones/README.md) for more details. | | singlePlacementGroup | no | Supported values are `true` (default) and `false`. Only applies to clusters with availabilityProfile `VirtualMachineScaleSets`. `true`: A VMSS with a single placement group and has a range of 0-100 VMs. `false`: A VMSS with multiple placement groups and has a range of 0-1,000 VMs. For more information, check out [virtual machine scale sets placement groups](https://docs.microsoft.com/en-us/azure/virtual-machine-scale-sets/virtual-machine-scale-sets-placement-groups). | | scaleSetPriority | no | Supported values are `Regular` (default) and `Low`. Only applies to clusters with availabilityProfile `VirtualMachineScaleSets`. Enables the usage of [Low-priority VMs on Scale Sets](https://docs.microsoft.com/en-us/azure/virtual-machine-scale-sets/virtual-machine-scale-sets-use-low-priority). | | scaleSetEvictionPolicy | no | Supported values are `Delete` (default) and `Deallocate`. Only applies to clusters with availabilityProfile of `VirtualMachineScaleSets` and scaleSetPriority of `Low`. | diff --git a/examples/kubernetes-zones/README.md b/examples/kubernetes-zones/README.md new file mode 100644 index 0000000000..e3d0b50df1 --- /dev/null +++ b/examples/kubernetes-zones/README.md @@ -0,0 +1,74 @@ +# Availability Zones + +To protect your cluster from datacenter-level failures, you can enable the Availability Zones feature for your cluster by configuring `"availabilityZones"` for the master profile and all of the agentPool profiles in the cluster definition. + + - This feature only applies to Kubernetes clusters version 1.12+. + - Supported values are arrays of strings, each representing a supported availability zone in a region for your subscription. For example, `"availabilityZones": ["1","2"]` indicates zone 1 and zone 2 can be used. + + > To get supported zones for a region in your subscription, run `az vm list-skus --location centralus --query "[?name=='Standard_DS2_v2'].[locationInfo, restrictions"] -o table`. You should see values like `'zones': ['2', '3', '1']` appear in the first column. If `NotAvailableForSubscription` appears in the output, then create an Azure support ticket to enable zones for that region. + +- To ensure high availability, each profile must define at least two nodes per zone. For example, an agent pool profile with 2 zones must have at least 4 nodes total: `"availabilityZones": ["1","2"],"count": 4`. +- When `"availabilityZones"` is configured, the `"loadBalancerSku"` will default to `Standard` as Standard LoadBalancer is required for availability zones. + +Here is an [example of a Kubernetes cluster with Availability Zones support](../e2e-tests/kubernetes/zones/definition.json) + +```json +{ + "apiVersion": "vlabs", + "properties": { + "orchestratorProfile": { + "orchestratorType": "Kubernetes", + "orchestratorRelease": "1.12" + }, + "masterProfile": { + "count": 5, + "dnsPrefix": "", + "vmSize": "Standard_DS2_v2", + "availabilityProfile": "VirtualMachineScaleSets", + "availabilityZones": [ + "1", + "2" + ] + }, + "agentPoolProfiles": [ + { + "name": "agentpool", + "count": 4, + "vmSize": "Standard_DS2_v2", + "availabilityProfile": "VirtualMachineScaleSets", + "availabilityZones": [ + "1", + "2" + ] + } + ], + "linuxProfile": { + "adminUsername": "azureuser", + "ssh": { + "publicKeys": [ + { + "keyData": "" + } + ] + } + }, + "servicePrincipalProfile": { + "clientId": "", + "secret": "" + } + } +} + +``` + +To validate availability zones are working as expected, run the following commands: + +```bash +kubectl get nodes --show-labels | grep failure-domain.beta.kubernetes.io/zone + +...,failure-domain.beta.kubernetes.io/zone=eastus2-1, ... +...,failure-domain.beta.kubernetes.io/zone=eastus2-2, ... + +``` + +Each node in the cluster should have `REGION-ZONE` as values for the `failure-domain.beta.kubernetes.io/zone` label. \ No newline at end of file