Apache DolphinScheduler is the modern data orchestration platform. Agile to create high performance workflow with low-code

DataSphereStudio is a one stop data application development& management portal, covering scenarios including data exchange, desensitization/cleansing, analysis/mining, quality measurement, visualization, and task scheduling.

Compass is a task diagnosis platform for bigdata

Define and schedule workflow, support Flink Jar/SQL, ClickHouse/Hive/Mysql SQL, Shell, etc.

This repo consists of all important concepts for data engineers.

A tool to find podcast metadata over an external api, store them, get their rss feeds and run ETL using Airflow, Kafka, Spark, and Cassandra. The particular Cassandra distribution used is Elassandra, which allows seamless integration with Elasticsearch. Displayed using a Gatsby app, served using Flask

HVAC Engine: Psychrometrics (Humid Air) analysis Java library. Humid air properties and thermodynamic processes, flows, heating, cooling, air mixing and more. Immutable, thread-safe, very accurate.

open source based development related contents

First academic big data project to implement analysis using MapReduce and Hive platform

PaddleSoft Backbone

aiflow-dashboard

Use K8s CustomResourceDefinition to define and create Airflow Dags

Data pipeline for multi-model databases using Microsoft Azure Cosmos DB SQL and Gremlin APIs and orchestration using Apache Airflow.

Text similarity based on Word2Vec on Twitter and Reddit updates, using Spark, Spring, MongoDB and Apache Airflow

Examples for ETL pipeline services infrastructure

Analyze OS Logs using a Kafka, Airflow, and Spark pipeline

Unveiling Cinematic Brilliance: Illuminating the Future of Movie Production through Data-Driven Insights