My name is José Carlos and I live in Belo Horizonte - MG - BR. I work with Data for about nine years and along this journey I had the opportunity of solving many kinds of Data issues, using a bunch of different technologies.
The last two years have been intense since I decided digging deep in Data Engineering world.
The bar chart bellow was developed by using plotly and it shows a resume of my skills.
Currently I'm working on a personal projects that get data from open sources DataEngineering_Kubernetes.
I have been working on these repositories separately with the aim of understanding different architectures. But currently I am focused on DataEngineering_Kubernetes
I am working on creating a Data Lakehouse, using AWS Glue. Also created a Kubernetes Cluster to orchestrate the Python Container and now I m focused on cretaing the Airflow Enviroment on AWS using EKS.
- On Data_Quality_On_Airflow repository I applied SODA framework to implement Data Quality layer on Airflow based on Astro-python-SDK.
- On DadosEconomicosBR repository I applied Terraform IAC to create AWS resources [S3 bucket, RDS], and Docker to encapsulate the Python script for ELT.
-
I'm going to be glad getting help with EKS - VPC - Subnets and its best practices. I have grew some knowledge on how to build the yaml manifests and how to monitor deployments, pods, namespaces and services by using eksctl and kubectl. But I'm still puzzled on provisioning it and running my dockerfile.
-
Reach me out on josecarlos.ico@gmail.com
