v1.8.9

What's Changed

• #410 Improve Spark sink to allow output to table and connection data targets in #412
• #398 Do not correct decimals in schema if it is correct already in #413
• #415 Extend notification target interface to include table definition of the table in #416

Full Changelog: v1.8.8...v1.8.9

v1.8.8

What's Changed

• #407 Signal handling always stopping the pipeline, forward the signal to the original handler instead in #408

Full Changelog: v1.8.7...v1.8.8

v1.8.7

What's Changed

• #403 Add the ability to get currently completed tasks from anywhere from user's code in #404
• #405 Update the version of the channels library in #405

Full Changelog: v1.8.6...v1.8.7

v1.8.6

What's Changed

• #372 Add global notification targets in #400
• #399 Add ability to use date expressions and formatting in JDBC Native sql expressions in #401

Full Changelog: v1.8.5...v1.8.6

v1.8.5

What's Changed

• #394 Fix PostgreSQL with numeric fields with no permission and scale in #395
• #349 Allow add partition for updating hive tables in #396

Full Changelog: v1.8.4...v1.8.5

v1.8.4

What's Changed

• #388 Handle OS signals and include thread stack traces in email notifications in #390
• #389 Add Spark application id to jobs that fail before running a task in #391
• #387 Implement handling DATE overflows that can happen with PotgreSQL infinity date values in #392

Full Changelog: v1.8.3...v1.8.4

v1.8.3

What's Changed

• Hotfix: handle null values when sanitizing timestamps in #385

Full Changelog: v1.8.2...v1.8.3

v1.8.2

What's Changed

• #375 Add workaround for timestamp overflows in #377
• #376 Add support for 'input.table' when JDBC Native is used with supported SQL dialects in #380
• #378 Add support for MySQL databases in the JDBC source in #381
• #382 Update versions of Scala, Spark and sbt for builds and CI/CD in #383

⚠️ A bug is introduced in this version: Ingestion jobs that use JDBCNative source will fail if a column having timestamp type contains nulls. Please, use Pramen 1.8.3 instead.

Full Changelog: v1.8.1...v1.8.2

v1.8.1

What's Changed

• #362 Get column descriptions from JDBC connection when possible in #364
• #361 Add the ability of EnceladusSink to send a REST request to delete temporary version files on S3 in #363
• #369 Fix lazy sinks incorrectly made lazy if they depend on transient tables in #370
• #367 Do not create directories on object stores unless strictly needed in #368
• #366 Track raw files at sources and in the metastore by size instead by count in #371

Full Changelog: v1.8.0...v1.8.1

v1.8.0

What's Changed

• #245 Make transient transformers are now executed on-demand in #341
• #301 Make SQL generators extensible so that custom generators could be plugged in without Pramen recompilation in #352
• #353 Add the option to disable count query to optimize performance of very large JDBC datasets by @yruslan in #355
• #351 Add a method to SQL generators interface to return an alias expression. by @yruslan in #357
• #354 Add Denodo dialect to support 'timestamp with timezone' data type by @yruslan in #356
• #338 Do not include records per seconds in email notification for tables that do not have enough records in #338
• #343 Add a GitHub action for automatic releases in #343

Full Changelog: v1.7.4...v1.8.0