Skip to content

Commit

Permalink
67
Browse files Browse the repository at this point in the history
  • Loading branch information
Houssem Dellai committed May 27, 2024
1 parent 5ea02ef commit 4aa763e
Show file tree
Hide file tree
Showing 6 changed files with 27 additions and 170 deletions.
192 changes: 26 additions & 166 deletions 67_egress_proxy_httponly/README.md
View file
Original file line number Diff line number Diff line change
@@ -1,10 +1,12 @@
# Controlling AKS egress using an HTTP Proxy
# AKS HTTP Proxy config ignores TCP connections

Azure Kubernetes Service (AKS) clusters, whether deployed into a managed or custom virtual network, have certain outbound dependencies necessary to function properly. Previously, in environments requiring internet access to be routed through HTTP proxies, this was a problem. Nodes had no way of bootstrapping the configuration, environment variables, and certificates necessary to access internet services.
When AKS is configured to egress traffic to an `HTTP Proxy`, all HTTP connections will traverse the HTTP Proxy like connections to web services and web sites.

This feature adds HTTP proxy support to AKS clusters, exposing a straightforward interface that cluster operators can use to secure AKS-required network traffic in proxy-dependent environments.
But, what about non-HTTP connections like `TCP` or `AMQP` ? Will they be forced through the HTTP Proxy ?

Both AKS nodes and Pods will be configured to use the HTTP proxy.
The short answer is no. End of the test :)

The rest is a lab where an environment is created and used to get the answer.

## Architecture

Expand All @@ -13,76 +15,9 @@ Both AKS nodes and Pods will be configured to use the HTTP proxy.
## Deploy demo using Terraform

You will create an environment where AKS egress traffic go through an HTTP Proxy server.
You will use MITM-Proxy as an HTTP Proxy server for AKS. Note you can use another proxy servers like `Squidhead` or `Zscaler`.

### Generate Certificate for MITM-Proxy server

By default, MITM6-Proxy generates a certificate when it starts. You can get this certificate from `~\.mitmproxy\` folder and use it with AKS.
But, for an enterprise use case, they will create their own certificate and then import it into MITM-Proxy.
That is what you will do here.

Refer to the script ``generate-cert.sh` to generate a certificate for MITM-Proxy and print it as base64 encoded.

```sh
openssl genrsa -out cert.key 2048

# (Specify the mitm domain as Common Name, e.g. \*.google.com or for all: *)

openssl req -new -x509 -key cert.key -out mitmproxy-ca-cert.pem

cat cert.key mitmproxy-ca-cert.pem > mitmproxy-ca.pem

openssl pkcs12 -export -inkey cert.key -in mitmproxy-ca-cert.pem -out mitmproxy-ca-cert.p12

cat mitmproxy-ca-cert.pem | base64 -w0
# sample output
# LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURnekNDQW11Z0F3SUJBZ0lVUzJTOHNMblQ1bi8vNkM3QTErMG01WXJUejhRd0RRWUpLb1pJaHZjTkFRRUwKQlFBd1VURUxNQWtHQTFVRUJoTUNSbEl4RXpBUkJnTlZCQWdNQ2xOdmJXVXRVM1JoZEdVeElUQWZCZ05WQkFvTQpHRWx1ZEdWeWJtVjBJRmRwWkdkcGRITWdVSFI1SUV4MFpERUtNQWdHQTFVRUF3d0JLakFlRncweU5EQXpNVFl3Ck9UUTVNemxhRncweU5EQTBNVFV3T1RRNU16bGFNRkV4Q3pBSkJnTlZCQVlUQWtaU01STXdFUVlEVlFRSURBcFQKYjIxbExWTjBZWFJsTVNFd0h3WURWUVFLREJoSmJuUmxjbTVsZENCWGFXUm5hWFJ6SUZCMGVTQk1kR1F4Q2pBSQpCZ05WQkFNTUFTb3dnZ0VpTUEwR0NTcUdTSWIzRFFFQkFRVUFBNElCRHdBd2dnRUtBb0lCQVFDWjVsUncvZFVlCkJsNXFjSzZSUUUrM1RwdTV5bWgxZDVDR0RwYkt2RDZ0djUwRjc5Y0JuUDJYODJ4aVJWU2R2TXJYZEx4MWJkek4KMVBnbjY4cVloSHVSOSt6TVdUN2VZUUtMZi9FYm9mSUEzbWhhS0xsVXFnTjNIRTNaMDU0RUdkQ0RrTlB3c3QyUAp6ckdBM3dVeDJyYkhXRzRpcC9SN1MvN0hIamtHdWh4QXFYZEdUM1BZdnBvKzh6RGVVeTdVRUxWYXg5VS9zdUFOCmhOMktweWxUZThLQmNVNnNFclNjUjdxYU8xLzdJYmVFRW9oQXhpblJ5SFQzaHJQZlY3WktjR0Q3NWtZUkJyRUMKWUdVL203bUsyeDJwek4zNmpad012ckxWZ3dkQkFieHpTSkxFSkR2YlVBWmZZalg3Y2w2SDNqL3ozYW1sTVdMbgpvU2NBeStkVTBFVkRBZ01CQUFHalV6QlJNQjBHQTFVZERnUVdCQlN1Y2VBWXQ2NE96Wk1XUXp3Q3BvZWVvRHk4ClVEQWZCZ05WSFNNRUdEQVdnQlN1Y2VBWXQ2NE96Wk1XUXp3Q3BvZWVvRHk4VURBUEJnTlZIUk1CQWY4RUJUQUQKQVFIL01BMEdDU3FHU0liM0RRRUJDd1VBQTRJQkFRQkswdFdybDZ3b1dDUCs1bS81VWx4SWl3MnE2d1QvdVQwVgpCR2J5QllYTGZKcms5L1lXQVBZR05yaFdmekhVQU8vaEIrbVY5TDU2UlU3NHAvYk51MXdqdGZuT0phRjl5YmUwCmhyMFNsaDlkdFdvRnBHeFVzMGlFVVFHNmhEVzM5bDg2TTlweVJ6NFYrWjVGVHMvMEkya2NTUk1ySk9PZk5JZm4KMkJiVSs4Z1FUV0U5L3gvcThOcWJocUZxSUQybkZXWjl4aUlvWG1GSmt5T3hNeU1ZS2RyTERERUlHa2ZEWHhqNQphUHp1Y3l4S0ZBVzNtbWEwd1Y3WEZFdE8yYjVDMkh1YjdEN2RlbDBkSzFmZUsveWR6Z2szaTdIREFvaFZKSFlLCmZxVzVZWlpNMjkyLzY1VThPaWJmNmtjYTNZOGRFTFRPYzkxRUdPdkt2SVBJQVQvdTFFTmgKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
```

An Azure VM `vm-linux-proxy-mitm.tf` will be used to install MITM-Proxy and import the generated certificate. Note how it runs the script `install-mitmproxy.sh` as custom data. For simplicity, the certificate will be imported from a Git repository.

```sh
#!/bin/bash

# 1. install MITM proxy from official package

wget https://downloads.mitmproxy.org/10.2.4/mitmproxy-10.2.4-linux-x86_64.tar.gz

tar -xvf mitmproxy-10.2.4-linux-x86_64.tar.gz

# [Other option] install MITM proxy using Python pip

# sudo apt install python3-pip -y
# pip3 install mitmproxy
# sudo apt install wget -y # install if not installed

# MITM proxy can create a certificate for us on starting, but we will use our own certificate
# 2. download the certificate files

wget 'https://raw.githubusercontent.com/HoussemDellai/docker-kubernetes-course/main/67_egress_proxy/certificate/mitmproxy-ca-cert.pem'
wget 'https://raw.githubusercontent.com/HoussemDellai/docker-kubernetes-course/main/67_egress_proxy/certificate/mitmproxy-ca.pem'
wget 'https://raw.githubusercontent.com/HoussemDellai/docker-kubernetes-course/main/67_egress_proxy/certificate/mitmproxy-ca-cert.p12'

# 3. start MITM proxy with the certificate and expose the web interface

./mitmweb --listen-port 8080 --web-host 0.0.0.0 --web-port 8081 --set block_global=false --certs *=./mitmproxy-ca.pem --set confdir=./
```

To configure AKS with an HTTP Proxy, you should use the following configuration sample.

```json
{
"httpProxy": "http://20.73.245.90:8080/",
"httpsProxy": "https://20.73.245.90:8080/",
"noProxy": [
"localhost",
"127.0.0.1",
"docker.io",
"docker.com"
],
"trustedCA": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURnekNDQW11Z0F3SUJBZ0lVUzJTOHNMblQ1bi8vNkM3QTErMG01WXJUejhRd0RRWUpLb1pJaHZjTkFRRUwKQlFBd1VURUxNQWtHQTFVRUJoTUNSbEl4RXpBUkJnTlZCQWdNQ2xOdmJXVXRVM1JoZEdVeElUQWZCZ05WQkFvTQpHRWx1ZEdWeWJtVjBJRmRwWkdkcGRITWdVSFI1SUV4MFpERUtNQWdHQTFVRUF3d0JLakFlRncweU5EQXpNVFl3Ck9UUTVNemxhRncweU5EQTBNVFV3T1RRNU16bGFNRkV4Q3pBSkJnTlZCQVlUQWtaU01STXdFUVlEVlFRSURBcFQKYjIxbExWTjBZWFJsTVNFd0h3WURWUVFLREJoSmJuUmxjbTVsZENCWGFXUm5hWFJ6SUZCMGVTQk1kR1F4Q2pBSQpCZ05WQkFNTUFTb3dnZ0VpTUEwR0NTcUdTSWIzRFFFQkFRVUFBNElCRHdBd2dnRUtBb0lCQVFDWjVsUncvZFVlCkJsNXFjSzZSUUUrM1RwdTV5bWgxZDVDR0RwYkt2RDZ0djUwRjc5Y0JuUDJYODJ4aVJWU2R2TXJYZEx4MWJkek4KMVBnbjY4cVloSHVSOSt6TVdUN2VZUUtMZi9FYm9mSUEzbWhhS0xsVXFnTjNIRTNaMDU0RUdkQ0RrTlB3c3QyUAp6ckdBM3dVeDJyYkhXRzRpcC9SN1MvN0hIamtHdWh4QXFYZEdUM1BZdnBvKzh6RGVVeTdVRUxWYXg5VS9zdUFOCmhOMktweWxUZThLQmNVNnNFclNjUjdxYU8xLzdJYmVFRW9oQXhpblJ5SFQzaHJQZlY3WktjR0Q3NWtZUkJyRUMKWUdVL203bUsyeDJwek4zNmpad012ckxWZ3dkQkFieHpTSkxFSkR2YlVBWmZZalg3Y2w2SDNqL3ozYW1sTVdMbgpvU2NBeStkVTBFVkRBZ01CQUFHalV6QlJNQjBHQTFVZERnUVdCQlN1Y2VBWXQ2NE96Wk1XUXp3Q3BvZWVvRHk4ClVEQWZCZ05WSFNNRUdEQVdnQlN1Y2VBWXQ2NE96Wk1XUXp3Q3BvZWVvRHk4VURBUEJnTlZIUk1CQWY4RUJUQUQKQVFIL01BMEdDU3FHU0liM0RRRUJDd1VBQTRJQkFRQkswdFdybDZ3b1dDUCs1bS81VWx4SWl3MnE2d1QvdVQwVgpCR2J5QllYTGZKcms5L1lXQVBZR05yaFdmekhVQU8vaEIrbVY5TDU2UlU3NHAvYk51MXdqdGZuT0phRjl5YmUwCmhyMFNsaDlkdFdvRnBHeFVzMGlFVVFHNmhEVzM5bDg2TTlweVJ6NFYrWjVGVHMvMEkya2NTUk1ySk9PZk5JZm4KMkJiVSs4Z1FUV0U5L3gvcThOcWJocUZxSUQybkZXWjl4aUlvWG1GSmt5T3hNeU1ZS2RyTERERUlHa2ZEWHhqNQphUHp1Y3l4S0ZBVzNtbWEwd1Y3WEZFdE8yYjVDMkh1YjdEN2RlbDBkSzFmZUsveWR6Z2szaTdIREFvaFZKSFlLCmZxVzVZWlpNMjkyLzY1VThPaWJmNmtjYTNZOGRFTFRPYzkxRUdPdkt2SVBJQVQvdTFFTmgKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo="
}
```
You will use `MITM-Proxy` as an HTTP Proxy server for AKS. Note you can use another proxy servers like `Squidhead` or `Zscaler`.
An `Azure SQL Server database` was created to test the TCP connection.
The connection will be initiated by a Pod in AKS.

Now you can deploy the Terraform template using the following commands.

Expand All @@ -96,105 +31,30 @@ terraform apply tfplan

![](images/resources.png)

## Testing the HTTP Proxy
## AKS and HTTP Proxy config

Deploy a sample Nginx pod and check the injected environment variables for the Proxy server.
Note the variables: `httpProxy`, `httpsProxy`, `noProxy` and `trustedCa`.
Configuring an AKS cluster with an HTTP Proxy was covered during the previous lab under this folder [67_egress_proxy](../67_egress_proxy).

```sh
kubectl run nginx --image=nginx

kubectl exec -it nginx -- env
# http_proxy=http://10.0.0.4:8080/
# HTTP_PROXY=http://10.0.0.4:8080/
# https_proxy=https://10.0.0.4:8080/
# HTTPS_PROXY=https://10.0.0.4:8080/
# no_proxy=localhost,aks-8v0n0swv.hcp.westeurope.azmk8s.io,10.10.0.0/24,10.0.0.0/16,169.254.169.254,docker.com,127.0.0.1,docker.io,konnectivity,10.10.0.0/16,168.63.129.16
# NO_PROXY=localhost,aks-8v0n0swv.hcp.westeurope.azmk8s.io,10.10.0.0/24,10.0.0.0/16,169.254.169.254,docker.com,127.0.0.1,docker.io,konnectivity,10.10.0.0/16,168.63.129.16
```

>Note: Note how AKS injected other CIDR ranges and domain names for the `NO_PROXY` environment variable. These are needed by the platform.
## Test the TCP connection

Check also these environment variables are injected into the cluster nodes.
To proceed with the test, you will deploy a pod `mcr.microsoft.com/mssql-tools` that wil initiate a connection to the SQL Server database.

```sh
kubectl get nodes
# NAME STATUS ROLES AGE VERSION
# aks-systempool-48300357-vmss000000 Ready <none> 11m v1.29.0
# aks-systempool-48300357-vmss000001 Ready <none> 11m v1.29.0
# aks-systempool-48300357-vmss000002 Ready <none> 11m v1.29.0

kubectl debug node/aks-systempool-48300357-vmss000000 -it --image=ubuntu

root@aks-systempool-48300357-vmss000000:/# chroot /host

env
# http_proxy=http://10.0.0.4:8080/
# HTTP_PROXY=http://10.0.0.4:8080/
# https_proxy=https://10.0.0.4:8080/
# HTTPS_PROXY=https://10.0.0.4:8080/
# no_proxy=localhost,aks-8v0n0swv.hcp.westeurope.azmk8s.io,10.10.0.0/24,10.0.0.0/16,169.254.169.254,docker.com,127.0.0.1,docker.io,konnectivity,10.10.0.0/16,168.63.129.16
# NO_PROXY=localhost,aks-8v0n0swv.hcp.westeurope.azmk8s.io,10.10.0.0/24,10.0.0.0/16,169.254.169.254,docker.com,127.0.0.1,docker.io,konnectivity,10.10.0.0/16,168.63.129.16
# ... removed for brievety
kubectl apply -f pod-mssql-tools.yaml
```

## Bypass HTTP Proxy

By default all egress traffic for nodes and pods will go through the Proxy because the environment variables for proxy are injected into all pods and nodes.
However, if you need to bypass the proxy, you just need to not inject these environment variables.
In kubernetes, this could be done declaratively using the annotation `"kubernetes.azure.com/no-http-proxy-vars": "true"`.

```yaml
apiVersion: v1
kind: Pod
metadata:
name: nginx-noproxy
annotations:
"kubernetes.azure.com/no-http-proxy-vars": "true"
spec:
containers:
- image: nginx
name: nginx
```
Then you will need to connect to the pod and use `sqlcmd` command line tool to start a connection to the database.
Make sure to use the correct values for database URL, login and password.
The query `select @@version` will check the version of the database. It is used to verify the connection was successful.

```sh
kubectl apply -f noproxy-pod.yaml
kubectl exec -it mssql-tools -- /bin/bash
# root@mssql-tools:/#
# root@mssql-tools:/#
sqlcmd -S mssql-server-67-dev.database.windows.net -U azureuser -P @Aa123456789 -d ProductsDB -Q 'select @@version'
# -----------------------------------------------------------------------------------------------------------------
# Microsoft SQL Azure (RTM) - 12.0.2000.8
# Apr 19 2024 18:03:25
# Copyright (C) 2022 Microsoft Corporation
# (1 rows affected)
```

Now if you check the environment variables for this pod, you will notice that the environment variables for the proxy were not injected.
And if you try to connect to internet, the egress traffic will be carried through the cluster Load Balancer and it's Public IP address.

```sh
kubectl exec -it nginx-noproxy -- env
# PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
# HOSTNAME=nginx-noproxy
# NGINX_VERSION=1.25.4
# NJS_VERSION=0.8.3
# PKG_RELEASE=1~bookworm
# KUBERNETES_PORT=tcp://10.0.0.1:443
# KUBERNETES_PORT_443_TCP=tcp://10.0.0.1:443
# KUBERNETES_PORT_443_TCP_PROTO=tcp
# KUBERNETES_PORT_443_TCP_PORT=443
# KUBERNETES_PORT_443_TCP_ADDR=10.0.0.1
# KUBERNETES_SERVICE_HOST=10.0.0.1
# KUBERNETES_SERVICE_PORT=443
# KUBERNETES_SERVICE_PORT_HTTPS=443
# TERM=xterm
# HOME=/root

kubectl exec -it nginx-noproxy -- curl ifconf.me
# 4.245.123.106 # this is cluster LB
```

## Updating Proxy configuration

You can update a cluster with existing proxy settings, but could not enable proxy for existing cluster.

```sh
az aks update -n aks -g rg-aks --http-proxy-config aks-proxy-config.json
```

An aks update for httpProxy, httpsProxy, and/or NoProxy will automatically inject new environment variables into pods with the new httpProxy, httpsProxy, or NoProxy values.
Pods must be rotated for the apps to pick it up.

For components under kubernetes, like containerd and the node itself, this won't take effect until a node image upgrade is performed.
1 change: 0 additions & 1 deletion 67_egress_proxy_httponly/aks.tf
View file
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,6 @@ resource "azurerm_kubernetes_cluster" "aks" {
node_count = 3
vm_size = "standard_b2als_v2"
vnet_subnet_id = azurerm_subnet.snet-aks.id
temporary_name_for_rotation = "systempool2"
}

identity {
Expand Down
Binary file modified 67_egress_proxy_httponly/images/architecture.png
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified 67_egress_proxy_httponly/images/resources.png
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
1 change: 0 additions & 1 deletion 67_egress_proxy_httponly/mssql.tf
View file
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,5 @@ output "mssql_server_fqdn" {

output "sqlcmd" {
value = "sqlcmd -S ${azurerm_mssql_server.mssql-server.fully_qualified_domain_name} -U ${azurerm_mssql_server.mssql-server.administrator_login} -P @Aa123456789 -d ${azurerm_mssql_database.database.name} -Q 'select @@version'"
# value = "sqlcmd -S ${azurerm_mssql_server.mssql-server.fully_qualified_domain_name} -U ${azurerm_mssql_server.mssql-server.administrator_login} -P ${azurerm_mssql_server.mssql-server.administrator_login_password} -d ${azurerm_mssql_database.database.name} -Q 'select @@version'"
sensitive = false
}
3 changes: 1 addition & 2 deletions 67_egress_proxy_httponly/pod-mssql-tools.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,5 +6,4 @@ spec:
containers:
- name: mssql-tools
image: mcr.microsoft.com/mssql-tools
args: [/bin/sh, -c, 'while true; do echo "$(date)"; sleep 10; done']
# args: [/bin/sh, -c, 'i=0; while true; do echo "This is demo log $i: $(date)"; i=$((i+1)); sleep 10; done']
args: [/bin/sh, -c, 'while true; do echo "$(date)"; sleep 10; done']\

0 comments on commit 4aa763e

Please sign in to comment.