datapains-argo-cd-k8s

Workshops

Trino on K8S

This workshop is a continuation off: Deploy Trino on Kubernetes: Helm Setup with Hive Metastore & MinIO Integration

Medium Article Covering this repo.

Pre requisites

• docker-desktop with kubernetes enabled

Setup

Local

Set for your environment ~/.baschrc or ~/.zshrc or in the session where you run this.

export HELM_EXPERIMENTAL_OCI=1

helm repo add argo https://argoproj.github.io/argo-helm

List version

helm search repo argo-cd

Time of writing this using 7.3.4.

Core

make deploy-local-argocd 

Get password for UI.

kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d

Run

kubectl port-forward service/datapains-argocd-server -n argocd 8080:443

You can now login into the UI with:

ArgoCD UI

• user: admin
• password:

Applications

Airbyte

make create-secret REPO_URL=git@github.com:Thelin90/datapains-airbyte.git SECRET_NAME=datapains-airbyte-creds SSH_KEY_PATH=<path-to-id_rsa>

make apply-argocd-app REPO_NAME=datapains-airbyte APP_NAME=airbyte

Note deployment can take a few minutes, check in argocd UI!.

Access UI

UI is now accessible via: wwww.localhost:32767

Apache Superset

make create-secret REPO_URL=git@github.com:Thelin90/datapains-bi-tools.git SECRET_NAME=datapains-bi-tools-creds SSH_KEY_PATH=<path-to-id_rsa>

make apply-argocd-app REPO_NAME=datapains-bi-tools APP_NAME=superset

Port Forward

kubectl port-forward svc/superset 8088:8088 -n superset

Argo Workflow

make create-secret REPO_URL=git@github.com:Thelin90/datapains-argo-workflow.git SECRET_NAME=datapains-argo-workflow-creds SSH_KEY_PATH=<path-to-id_rsa>

make apply-argocd-app REPO_NAME=datapains-argo-workflow APP_NAME=argo-workflow

Note deployment can take a few minutes, check in argocd UI!.

Access UI

UI is now accessible via: wwww.localhost:32767

Prometheus Community Helm Chart

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

helm repo update

kube-prometheus-stack

• prometheus operator
• grafana

make apply-argocd-app REPO_NAME=datapains-monitoring APP_NAME=kube-prometheus-stack

Note deployment can take a few minutes, check in argocd UI!.

Prometheus

make create-secret REPO_URL=git@github.com:Thelin90/datapains-prometheus-k8s.git SECRET_NAME=datapains-prometheus-k8s-creds SSH_KEY_PATH=<path-to-id_rsa>

make apply-argocd-app REPO_NAME=datapains-monitoring APP_NAME=prometheus

Note deployment can take a few minutes, check in argocd UI!.

Example Prometheus App

make apply-argocd-app REPO_NAME=datapains-monitoring APP_NAME=example-application

Spark Operator

First add spark namespace.

kubectl create namespace spark

make apply-argocd-app REPO_NAME=datapains-spark-operator APP_NAME=spark-operator

DataPains Trino K8S

We will deploy trino through argocd, the repo we have available on datapains-trino-k8s

You will have to setup a secret key.

Create your secret based on your github SSH key.

 make create-secret REPO_URL=git@github.com:Thelin90/datapains-trino-k8s.git SECRET_NAME=datapains-trino-k8s-creds SSH_KEY_PATH=<path-to-id_rsa>

NOTE!

Remember to build the hive metastore image, instructions here.

Now deploy the metastore:

make apply-argocd-app REPO_NAME=datapains-trino-k8s APP_NAME=metastore

Now deploy trino, once you see it being healthy in ArgoCD UI.

make apply-argocd-app REPO_NAME=datapains-trino-k8s APP_NAME=trino