Skip to content

Commit

Permalink
1.7.1 (#181)
Browse files Browse the repository at this point in the history 
* Updated K8S Delete Job API
* Added GCP and K8S config tests in CI
* Updated documentation

Co-authored-by: Hari Krishnan <rite2hari347@gmail.com>
  • Loading branch information
@tharwaninitin @Hari3
tharwaninitin and Hari3 authored Jan 23, 2023
1 parent e9c6b2a commit af6e0f9
Show file tree
Hide file tree
Showing 22 changed files with 388 additions and 101 deletions.
14 changes: 5 additions & 9 deletions .github/workflows/ci.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -154,18 +154,14 @@ jobs:
java-version: [ 11, 17 ] # removed version 8 temporarily to avoid sbt-updates failing because of the latest version of LogbackVersion
fail-fast: false
services:
# Label used to access the service container
redis:
# Docker Hub image
image: redis
# Set health checks to wait until redis has started
options: >-
--health-cmd "redis-cli ping"
--health-interval 10s
--health-timeout 5s
--health-retries 5
ports:
# Maps tcp port 5432 on service container to the host
- 6379:6379
steps:
- uses: actions/checkout@v3
Expand Down Expand Up @@ -216,8 +212,8 @@ jobs:
java-version: ${{ matrix.java-version }}
distribution: 'adopt'
cache: 'sbt'
- name: Compile K8S Test
run: sbt ";project k8s; +Test/compile"
- name: Test K8S Module
run: sbt ";project k8s; +Test/compile; testOnly etlflow.RunConfigTests"
email:
runs-on: ubuntu-latest
strategy:
Expand All @@ -238,7 +234,7 @@ jobs:
runs-on: ubuntu-latest
strategy:
matrix:
java-version: [ 8, 11, 17 ]
java-version: [ 11, 17 ]
fail-fast: false
steps:
- uses: actions/checkout@v3
Expand All @@ -248,5 +244,5 @@ jobs:
java-version: ${{ matrix.java-version }}
distribution: 'adopt'
cache: 'sbt'
- name: Compile Gcp Module
run: sbt ";project gcp; +Test/compile"
- name: Test GCP Module
run: sbt ";project gcp; +Test/compile; testOnly etlflow.RunConfigTests"
67 changes: 49 additions & 18 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,18 @@
[![Maven Central](https://maven-badges.herokuapp.com/maven-central/com.github.tharwaninitin/etlflow-core_2.12/badge.svg)](https://mvnrepository.com/artifact/com.github.tharwaninitin/etlflow-core)
[![javadoc](https://javadoc.io/badge2/com.github.tharwaninitin/etlflow-core_2.12/javadoc.svg)](https://javadoc.io/doc/com.github.tharwaninitin/etlflow-core_2.12)

**EtlFlow** is an ecosystem of functional libraries in Scala based on ZIO for writing various different tasks, jobs on GCP and AWS.
**EtlFlow** is an ecosystem of functional libraries in Scala based on ZIO for running complex **_Auditable_** workflows which can interact with Google Cloud Platform, AWS, Kubernetes, Databases and more.

Below are some salient features of this library.

- **Functional**. Rapidly compose complex **workflows** from simple tasks.
- **Fibers**. Built on non-blocking fibers that never waste or leak resources, which lets you build scalable, resilient, and reactive applications
- **Resource-safe**. Build workflows that never leak resources (including threads!), even when they fail.
- **Concurrent**. Easily build concurrent workflows without deadlocks, race conditions, or complexity.
- **Asynchronous**. Write sequential code that looks the same whether it's asynchronous or synchronous.
- **Type-safe**. Use the full power of the Scala compiler to catch bugs at compile time.
- **Testable**. Inject test services into your workflow for fast, deterministic, and type-safe testing.
- **Resilient**. Build workflows that never lose errors, and which respond to failure locally and flexibly.

[//]: # (## Documentation)
[//]: # ()
Expand All @@ -28,13 +39,13 @@

| Module | Latest Version | Documentation | Scala Versions | Java Version |
|--------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------:|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------|
| Core | [![Latest Version](https://maven-badges.herokuapp.com/maven-central/com.github.tharwaninitin/etlflow-core_2.12/badge.svg)](https://mvnrepository.com/artifact/com.github.tharwaninitin/etlflow-core) | [![javadoc](https://javadoc.io/badge2/com.github.tharwaninitin/etlflow-core_2.12/javadoc.svg)](https://javadoc.io/doc/com.github.tharwaninitin/etlflow-core_2.12) | [![etlflow-core Scala version support](https://index.scala-lang.org/tharwaninitin/etlflow/etlflow-core/latest-by-scala-version.svg)](https://index.scala-lang.org/tharwaninitin/etlflow/etlflow-core) | Java 8 + |
| Gcp | [![Latest Version](https://maven-badges.herokuapp.com/maven-central/com.github.tharwaninitin/etlflow-gcp_2.12/badge.svg)](https://mvnrepository.com/artifact/com.github.tharwaninitin/etlflow-gcp) | [![javadoc](https://javadoc.io/badge2/com.github.tharwaninitin/etlflow-gcp_2.12/javadoc.svg)](https://javadoc.io/doc/com.github.tharwaninitin/etlflow-gcp_2.12) | [![etlflow-gcp Scala version support](https://index.scala-lang.org/tharwaninitin/etlflow/etlflow-gcp/latest-by-scala-version.svg)](https://index.scala-lang.org/tharwaninitin/etlflow/etlflow-gcp) | Java 8 + |
| JDBC | [![Latest Version](https://maven-badges.herokuapp.com/maven-central/com.github.tharwaninitin/etlflow-jdbc_2.12/badge.svg)](https://mvnrepository.com/artifact/com.github.tharwaninitin/etlflow-jdbc) | [![javadoc](https://javadoc.io/badge2/com.github.tharwaninitin/etlflow-jdbc_2.12/javadoc.svg)](https://javadoc.io/doc/com.github.tharwaninitin/etlflow-jdbc_2.12) | [![etlflow-jdbc Scala version support](https://index.scala-lang.org/tharwaninitin/etlflow/etlflow-jdbc/latest-by-scala-version.svg)](https://index.scala-lang.org/tharwaninitin/etlflow/etlflow-jdbc) | Java 8 + |
| Http | [![Latest Version](https://maven-badges.herokuapp.com/maven-central/com.github.tharwaninitin/etlflow-http_2.12/badge.svg)](https://mvnrepository.com/artifact/com.github.tharwaninitin/etlflow-http) | [![javadoc](https://javadoc.io/badge2/com.github.tharwaninitin/etlflow-http_2.12/javadoc.svg)](https://javadoc.io/doc/com.github.tharwaninitin/etlflow-http_2.12) | [![etlflow-http Scala version support](https://index.scala-lang.org/tharwaninitin/etlflow/etlflow-http/latest-by-scala-version.svg)](https://index.scala-lang.org/tharwaninitin/etlflow/etlflow-http) | Java 11 + |
| K8S | [![Latest Version](https://maven-badges.herokuapp.com/maven-central/com.github.tharwaninitin/etlflow-k8s_2.12/badge.svg)](https://mvnrepository.com/artifact/com.github.tharwaninitin/etlflow-k8s) | [![javadoc](https://javadoc.io/badge2/com.github.tharwaninitin/etlflow-k8s_2.12/javadoc.svg)](https://javadoc.io/doc/com.github.tharwaninitin/etlflow-k8s_2.12) | [![etlflow-k8s Scala version support](https://index.scala-lang.org/tharwaninitin/etlflow/etlflow-k8s/latest-by-scala-version.svg)](https://index.scala-lang.org/tharwaninitin/etlflow/etlflow-k8s) | Java 11 + |
| Email | [![Latest Version](https://maven-badges.herokuapp.com/maven-central/com.github.tharwaninitin/etlflow-email_2.12/badge.svg)](https://mvnrepository.com/artifact/com.github.tharwaninitin/etlflow-email) | [![javadoc](https://javadoc.io/badge2/com.github.tharwaninitin/etlflow-email_2.12/javadoc.svg)](https://javadoc.io/doc/com.github.tharwaninitin/etlflow-email_2.12) | [![etlflow-email Scala version support](https://index.scala-lang.org/tharwaninitin/etlflow/etlflow-email/latest-by-scala-version.svg)](https://index.scala-lang.org/tharwaninitin/etlflow/etlflow-email) | Java 8 + |
| Aws | [![Latest Version](https://maven-badges.herokuapp.com/maven-central/com.github.tharwaninitin/etlflow-aws_2.12/badge.svg)](https://mvnrepository.com/artifact/com.github.tharwaninitin/etlflow-aws) | [![javadoc](https://javadoc.io/badge2/com.github.tharwaninitin/etlflow-aws_2.12/javadoc.svg)](https://javadoc.io/doc/com.github.tharwaninitin/etlflow-aws_2.12) | [![etlflow-aws Scala version support](https://index.scala-lang.org/tharwaninitin/etlflow/etlflow-aws/latest-by-scala-version.svg)](https://index.scala-lang.org/tharwaninitin/etlflow/etlflow-aws) | Java 8 + |
| Core | [![Latest Version](https://maven-badges.herokuapp.com/maven-central/com.github.tharwaninitin/etlflow-core_2.12/badge.svg)](https://mvnrepository.com/artifact/com.github.tharwaninitin/etlflow-core) | [![javadoc](https://javadoc.io/badge2/com.github.tharwaninitin/etlflow-core_3/javadoc.svg)](https://javadoc.io/doc/com.github.tharwaninitin/etlflow-core_3) | [![etlflow-core Scala version support](https://index.scala-lang.org/tharwaninitin/etlflow/etlflow-core/latest-by-scala-version.svg)](https://index.scala-lang.org/tharwaninitin/etlflow/etlflow-core) | Java 8 + |
| GCP | [![Latest Version](https://maven-badges.herokuapp.com/maven-central/com.github.tharwaninitin/etlflow-gcp_2.12/badge.svg)](https://mvnrepository.com/artifact/com.github.tharwaninitin/etlflow-gcp) | [![javadoc](https://javadoc.io/badge2/com.github.tharwaninitin/etlflow-gcp_3/javadoc.svg)](https://javadoc.io/doc/com.github.tharwaninitin/etlflow-gcp_3) | [![etlflow-gcp Scala version support](https://index.scala-lang.org/tharwaninitin/etlflow/etlflow-gcp/latest-by-scala-version.svg)](https://index.scala-lang.org/tharwaninitin/etlflow/etlflow-gcp) | Java 8 + |
| JDBC | [![Latest Version](https://maven-badges.herokuapp.com/maven-central/com.github.tharwaninitin/etlflow-jdbc_2.12/badge.svg)](https://mvnrepository.com/artifact/com.github.tharwaninitin/etlflow-jdbc) | [![javadoc](https://javadoc.io/badge2/com.github.tharwaninitin/etlflow-jdbc_3/javadoc.svg)](https://javadoc.io/doc/com.github.tharwaninitin/etlflow-jdbc_3) | [![etlflow-jdbc Scala version support](https://index.scala-lang.org/tharwaninitin/etlflow/etlflow-jdbc/latest-by-scala-version.svg)](https://index.scala-lang.org/tharwaninitin/etlflow/etlflow-jdbc) | Java 8 + |
| Http | [![Latest Version](https://maven-badges.herokuapp.com/maven-central/com.github.tharwaninitin/etlflow-http_2.12/badge.svg)](https://mvnrepository.com/artifact/com.github.tharwaninitin/etlflow-http) | [![javadoc](https://javadoc.io/badge2/com.github.tharwaninitin/etlflow-http_3/javadoc.svg)](https://javadoc.io/doc/com.github.tharwaninitin/etlflow-http_3) | [![etlflow-http Scala version support](https://index.scala-lang.org/tharwaninitin/etlflow/etlflow-http/latest-by-scala-version.svg)](https://index.scala-lang.org/tharwaninitin/etlflow/etlflow-http) | Java 11 + |
| K8S | [![Latest Version](https://maven-badges.herokuapp.com/maven-central/com.github.tharwaninitin/etlflow-k8s_2.12/badge.svg)](https://mvnrepository.com/artifact/com.github.tharwaninitin/etlflow-k8s) | [![javadoc](https://javadoc.io/badge2/com.github.tharwaninitin/etlflow-k8s_3/javadoc.svg)](https://javadoc.io/doc/com.github.tharwaninitin/etlflow-k8s_3) | [![etlflow-k8s Scala version support](https://index.scala-lang.org/tharwaninitin/etlflow/etlflow-k8s/latest-by-scala-version.svg)](https://index.scala-lang.org/tharwaninitin/etlflow/etlflow-k8s) | Java 8 + |
| Email | [![Latest Version](https://maven-badges.herokuapp.com/maven-central/com.github.tharwaninitin/etlflow-email_2.12/badge.svg)](https://mvnrepository.com/artifact/com.github.tharwaninitin/etlflow-email) | [![javadoc](https://javadoc.io/badge2/com.github.tharwaninitin/etlflow-email_3/javadoc.svg)](https://javadoc.io/doc/com.github.tharwaninitin/etlflow-email_3) | [![etlflow-email Scala version support](https://index.scala-lang.org/tharwaninitin/etlflow/etlflow-email/latest-by-scala-version.svg)](https://index.scala-lang.org/tharwaninitin/etlflow/etlflow-email) | Java 8 + |
| AWS | [![Latest Version](https://maven-badges.herokuapp.com/maven-central/com.github.tharwaninitin/etlflow-aws_2.12/badge.svg)](https://mvnrepository.com/artifact/com.github.tharwaninitin/etlflow-aws) | [![javadoc](https://javadoc.io/badge2/com.github.tharwaninitin/etlflow-aws_3/javadoc.svg)](https://javadoc.io/doc/com.github.tharwaninitin/etlflow-aws_3) | [![etlflow-aws Scala version support](https://index.scala-lang.org/tharwaninitin/etlflow/etlflow-aws/latest-by-scala-version.svg)](https://index.scala-lang.org/tharwaninitin/etlflow/etlflow-aws) | Java 8 + |
| Redis | [![Latest Version](https://maven-badges.herokuapp.com/maven-central/com.github.tharwaninitin/etlflow-redis_2.12/badge.svg)](https://mvnrepository.com/artifact/com.github.tharwaninitin/etlflow-redis) | [![javadoc](https://javadoc.io/badge2/com.github.tharwaninitin/etlflow-redis_2.12/javadoc.svg)](https://javadoc.io/doc/com.github.tharwaninitin/etlflow-redis_2.12) | [![etlflow-redis Scala version support](https://index.scala-lang.org/tharwaninitin/etlflow/etlflow-redis/latest-by-scala-version.svg)](https://index.scala-lang.org/tharwaninitin/etlflow/etlflow-redis) | Java 8 + |
| Spark | [![Latest Version](https://maven-badges.herokuapp.com/maven-central/com.github.tharwaninitin/etlflow-spark_2.12/badge.svg)](https://mvnrepository.com/artifact/com.github.tharwaninitin/etlflow-spark) | [![javadoc](https://javadoc.io/badge2/com.github.tharwaninitin/etlflow-spark_2.12/javadoc.svg)](https://javadoc.io/doc/com.github.tharwaninitin/etlflow-spark_2.12) | [![etlflow-spark Scala version support](https://index.scala-lang.org/tharwaninitin/etlflow/etlflow-spark/latest-by-scala-version.svg)](https://index.scala-lang.org/tharwaninitin/etlflow/etlflow-spark) | Java 8 + |

Expand All @@ -48,29 +59,33 @@ Add the below latest release as a dependency to your project

__SBT__
```scala
libraryDependencies += "com.github.tharwaninitin" %% "etlflow-core" % "1.7.0"
libraryDependencies += "com.github.tharwaninitin" %% "etlflow-gcp" % "1.7.0"
libraryDependencies += "com.github.tharwaninitin" %% "etlflow-jdbc" % "1.7.0"
libraryDependencies += "com.github.tharwaninitin" %% "etlflow-spark" % "1.7.0"
libraryDependencies += "com.github.tharwaninitin" %% "etlflow-k8s" % "1.7.0"
libraryDependencies += "com.github.tharwaninitin" %% "etlflow-http" % "1.7.0"
libraryDependencies += "com.github.tharwaninitin" %% "etlflow-redis" % "1.7.0"
libraryDependencies += "com.github.tharwaninitin" %% "etlflow-aws" % "1.7.0"
libraryDependencies += "com.github.tharwaninitin" %% "etlflow-email" % "1.7.0"
libraryDependencies += "com.github.tharwaninitin" %% "etlflow-core" % "1.7.1"
libraryDependencies += "com.github.tharwaninitin" %% "etlflow-gcp" % "1.7.1"
libraryDependencies += "com.github.tharwaninitin" %% "etlflow-jdbc" % "1.7.1"
libraryDependencies += "com.github.tharwaninitin" %% "etlflow-spark" % "1.7.1"
libraryDependencies += "com.github.tharwaninitin" %% "etlflow-k8s" % "1.7.1"
libraryDependencies += "com.github.tharwaninitin" %% "etlflow-http" % "1.7.1"
libraryDependencies += "com.github.tharwaninitin" %% "etlflow-redis" % "1.7.1"
libraryDependencies += "com.github.tharwaninitin" %% "etlflow-aws" % "1.7.1"
libraryDependencies += "com.github.tharwaninitin" %% "etlflow-email" % "1.7.1"
```
__Maven__
```
<dependency>
<groupId>com.github.tharwaninitin</groupId>
<artifactId>etlflow-core_2.12</artifactId>
<version>1.7.0</version>
<version>1.7.1</version>
</dependency>
```

# Etlflow Modules
<!-- TOC -->
- [Etlflow Modules](#etlflow-modules)
- [Core](#core)
- [Task](#task)
- [Audit](#audit)
- [Config](#config)
- [Json](#json)
- [GCP](#gcp)
- [Dataproc](#dataproc)
- [K8S](#k8s)
Expand All @@ -86,6 +101,22 @@ __Maven__
```scala
// Todo
```
### Task
```scala
// Todo
```
### Audit
```scala
// Todo
```
### Config
```scala
// Todo
```
### Json
```scala
// Todo
```
## GCP
```shell
# To run all below GCP examples set the GOOGLE_APPLICATION_CREDENTIALS environment variable to the location of the service account json key.
Expand Down
Loading

0 comments on commit af6e0f9

Please sign in to comment.