Skip to content

Latest commit

 

History

History
382 lines (273 loc) · 14.1 KB

README.md

File metadata and controls

382 lines (273 loc) · 14.1 KB
Raw

Validate Scenarios

Now that we have implemented everything, let's go back and revisit our requirements and make sure they have been met.

Requirements

This is a quick recap of the requirements:

  1. Leverage Existing Identity Mgmt Solution

  2. Implement Security Least Privilege Principle

  3. Log Everything for Audit Reporting purposes

  4. Ensure Security Controls are being met (No Drifting)

  5. Monitoring and Alerting Events

    • Alert when SSH into Container
    • Alert AKS Cluster has IP whitelisting set

  6. Integrate with Existing SIEM

  7. Deploy into Existing VNET with Ingress and Egress Restrictions

  8. Resources can only be created in specific regions due to data sovereignty

  9. Container Registry Whitelisting

  10. Ability to Chargeback to Line of Business

  11. Secrets Mgmt

  12. Container Image Mgmt

  13. Restrict Creation of Public IPs

  14. Implement & Deploy Image Processing Application

  15. Easily rollout new versions of Application

Requirements to Technology Matrix

Requirement Technology Used
1. Leverage Existing Identity Mgmt Solution Azure AD (AAD)
2. Implement Security Least Privilege Principle AAD + ARM RBAC + K8s RBAC per Namespace
3. Log Everything for Audit Reporting purposes Azure Monitor for Containers, Azure Storage
4. Ensure Security Controls are being met (No Drifting) Flux, Git Repo
5a. Alert when SSH into Container Falco + Azure Monitor for Containers
5b. Alert AKS Cluster has IP whitelisting set Azure Security Center + Azure Monitor for Containers
6. Integrate with Existing SIEM Azure Security Center or Azure Sentinel
7. Deploy into Existing VNET with Ingress and Egress Restrictions Azure VNET, Azure Firewall
8. Resources can only be created in specific regions due to data sovereignty Azure Policy
9. Container Registry Whitelisting Open Policy Agent + OSS Version of Gatekeeper
10. Ability to Chargeback to Line of Business KubeCost
11. Secrets Mgmt Azure AD Pod Identity + Azure Key Vault
12. Container Image Mgmt Anchore
13. Restrict Creation of Public IPs Azure Policy
14. Implement & Deploy Image Processing Application Azure Monitor for Containers, Application Insights
15. Easily rollout new versions of Application Kubernetes Built-in

The rest of these sections shows how we can validate the requirements above.

1. Validate - Leverage Existing Identity Mgmt Solution

  • Log into AKS Cluster with Azure AD Credentials

Pull down the cluster configuration file and try to execute a command, you should get prompted to authenticate to Azure AD.

# Grab K8s Config
az aks get-credentials -g $RG -n $PREFIX-aks

# Execute a Command
kubectl get nodes

Azure AD Authentication)

2. Validate - Implement Security Least Privilege Principle

  • Validate Cluster Reader cannot Create Resources

Authenticate to AKS using the cluster reader credentials and then try and execute a create command.

# Try to create a pod in the default namespace
kubectl run --generator=run-pod/v1 -it --rm centos2 --image=centos

Cluster Reader)

3. Validate - Log Everything for Audit Reporting purposes

  • Run log analytics query against AzureActivity table

Go into Azure Monitor for Containers -> Logs and get a summary of activity logs by Resource Provider.

AzureActivity
| summarize AggregatedValue = count() by ResourceProvider

Activity Logs Summary by Resource Provider)

Azure Monitor Logs Activity Log Query)

4. Validate - Ensure Security Controls are being met (No Drifting)

  • Look at Flux Logs for GitOps style Drift Configuration. As you can see from the screenshot it detects whether something has changed or not.

Flux Logs)

  • Delete a resource and see it get re-created. Delete the Production NS and then wait upwards of 5 mins to see that Flux re-creates the resource.
kubectl delete ns production
kubectl logs -l name=flux -n flux -f

5. Validate - Monitoring and Alerting Events

  • View Azure Security Center Compliance Dashboard

Azure Security Center Compliance Dashboard)

  • Exec into a Container and get an Alert
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: centos
spec:
  containers:
  - name: centoss
    image: centos
    ports:
    - containerPort: 80
    command:
    - sleep
    - "3600"
EOF
# Check that Pod is Created and then Exec In
kubectl get po -o wide
kubectl exec -it centos -- /bin/bash
curl www.ubuntu.com
exit

Wait a few minutes for the log data to get processed and then go into the Azure Monitor Logs Workspace and execute the following query to see the log entry.

let startTimestamp = ago(1d);
let ContainerIDs = KubePodInventory
| where TimeGenerated > startTimestamp
| where ClusterId =~ "/subscriptions/${SUBID}/resourceGroups/${RG}/providers/Microsoft.ContainerService/managedClusters/${CLUSTER_NAME}"
| where Name contains "sysdig-falco"
| distinct ContainerID;
ContainerLog
| where ContainerID in (ContainerIDs)
| where LogEntry contains "Notice A shell was spawned"
| project LogEntrySource, LogEntry, TimeGenerated, Computer, Image, Name, ContainerID
| order by TimeGenerated desc
| limit 200

Azure Monitor Logs SSH Query)

  • AKS Cluster missing IP Whitelisting

Similiar to the above we will run the following Azure Monitor queries against the Azure Security Center data to see which Policies are being met from a security perspective.

AzureActivity
| where CategoryValue == 'Policy' and Level != 'Informational'
| where ResourceProvider == "Microsoft.ContainerService"
| extend p=todynamic(Properties)
| extend policies=todynamic(tostring(p.policies))
| mvexpand policy = policies
| summarize resource_count=count() by tostring(policy.policyDefinitionName),tostring(policy.policyDefinitionReferenceId)

Azure Monitor Logs Policy Out of Compliance Query)

// Check for Authorized IP Policy
let policyDefId = '0e246bcf-5f6f-4f87-bc6f-775d4712c7ea';
AzureActivity
| where CategoryValue == 'Policy' and Level != 'Informational'
| where ResourceProvider == "Microsoft.ContainerService"
| extend p=todynamic(Properties)
| extend policies=todynamic(tostring(p.policies))
| mvexpand policy = policies
| where policy.policyDefinitionName in (policyDefId)
| distinct ResourceId

Azure Monitor Logs Authorized IP Query)

6. Validate - Integrate with Existing SIEM

  • View Azure Security Center Security Solutions

Go to the Azure Portal, click on Security Center, then on the Security Solutions blade.

Azure Security Center SIEM Integration)

7. Validate - Deploy into Existing VNET with Ingress and Egress Restrictions

  • Validate Traffic In & Out of Cluster (North/South)
kubectl exec -it centos -- /bin/bash
curl http://superman.com

North/South)

  • Validate Traffic Restriction between Namespaces (East/West)
kubectl exec -it centos -- /bin/bash
curl http://imageclassifierweb.dev.svc.cluster.local
exit

East/West)

8. Validate - Resources can only be created in specific regions due to data sovereignty

  • Try to create resource outside of allowed region locations
az storage account create --sku Standard_LRS --kind StorageV2 --location westus -g notinallowedregions-rg -n niarsa

Create Storage in West US (Not in East US)

9. Validate - Container Registry Whitelisting

  • Try to pull from a non-whitelisted Container Registry
# Test out Allowed Registry Policy Against production Namespace
kubectl run --generator=run-pod/v1 -it --rm centosprod --image=centos -n production

Gatekeeper Allowed Registries

10. Validate - Ability to Chargeback to Line of Business

  • View Chargeback Dashboard
# Do a port-forward to see Kubecost Dashboard
kubectl port-forward deployment/kubecost-cost-analyzer -n kubecost 9090
# Open in Browser
open "http://localhost:9090"

Sample Kubecost Dashboard

11. Validate - Secrets Mgmt

  • Check that there is no sensitive data stored in the container image or in a configuration file in plain text.

  • The first place to start is looking at the application manifest file and we can see from looking at it that it is not storing credentials, it simply points to an Azure Key Vault Name.

      containers:
      - name: imageclassifierweb
        image: kevingbb/imageclassifierweb:v3
        imagePullPolicy: Always
        env:
        - name: KeyVault__Vault
          valueFrom:
            secretKeyRef:
              name: image-akv-secret
              key: KeyVault__Vault
  • The next step is to look inside the container to see if there is any configuration information.
# Exec into Web Container for Example
kubectl get pods -n dev

# Grab Pod Name (from above) and use to exec into Pod
kubectl exec -it imageclassifierweb-d7fdcd8bf-dphbm -n dev -c imageclassifierweb /bin/sh
# OR
kubectl exec -it $(k get po -l=app=imageclassifierweb -n dev -o jsonpath="{.items[0].metadata.name}") -n dev -c imageclassifierweb /bin/sh

# Once inside of Pod Look Around
ls -al

# Exit Out
exit

Secrets in Container

  • Check for Sensitive values in Key Vault
# Look at Secret Value in Key Vault
az keyvault secret show --name "AppSecret" --vault-name "${PREFIX}akv"
az keyvault secret show --name "AppInsightsInstrumentationKey" --vault-name "${PREFIX}akv"

AppSecret in Key Vault

AppSecret in App

  • So how does the secret get into the application then? Great question, it relies on Azure AD Pod Identity, or what we like to call Managed Pod Identity. Click here for more details.

12. Validate - Container Image Mgmt

  • Check that container images in ACR are passing image scanning policy check.
# Get ACR Variable Name
echo ACR_NAME=${PREFIX}acr.azurecr.io
# Exec into Container
kubectl exec -it $(kubectl get po -l app=anchore-anchore-engine -l component=analyzer -n anchore -o jsonpath='{.items[0].metadata.name}') -n anchore bash
# Setup Variables
ANCHORE_CLI_USER=admin
ANCHORE_CLI_PASS=foobar
ANCHORE_CLI_URL=http://anchore-anchore-engine-api.anchore.svc.cluster.local:8228/v1/

# Wait for all images to be "analyzed"
anchore-cli image list

# Check for Active Subscriptions
anchore-cli subscription list

# Get Policies
anchore-cli policy list

# Evaluate against Policy (Pass or Fail) Using ACR_NAME From Above
ACR_NAME=...
anchore-cli evaluate check $ACR_NAME/imageclassifierweb:v1
anchore-cli evaluate check $ACR_NAME/imageclassifierworker:v1

# Exit out of Pod
exit

Anchore Pass or Fail

13. Validate - Restrict Creation of Public IPs

This is similar to #8 in that Azure Policy can be used to restrict the creation of Public IPs exception in certain Resource Groups. This was not implemented in the workshop so that Public IPs were possible to be able to test and see endpoints.

14. Validate - Implement & Deploy Image Processing Application

  • Does the Application Run, Visit Public IP

Running Application)

15. Validate - Easily roll out new versions of Application

  • Ensure the app successfully rolls out a new version of the application and does not cause any downtime.
# Check Deployment History
kubectl rollout history deploy imageclassifierweb -n dev

# Make a change to the Manifest File & Apply a new version of the application
# Hint: Change registry to ACR
kubectl apply -f appv3msi.yaml

# Watch the Rollout
kubectl rollout status deploy imageclassifierweb -n dev

# Check Deployment History Again
kubectl rollout history deploy imageclassifierweb -n dev

Deployment Rollout

  • Test the app to make sure it continues to work.

Next Steps

Thought Leadership

Key Links