-
Notifications
You must be signed in to change notification settings - Fork 3
/
CHANGELOG
101 lines (92 loc) · 4.06 KB
/
CHANGELOG
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
Version 0.5.0 (2024-11-01)
--------------------------
Disable asynchronous deletes for Hudi (#94)
Upgrade dependencies (#92)
DynamoDB runtime dependencies for Delta S3 multi-writer
PDP-1323 Handle initialization errors for GCP
PDP-1324 - Handle initialization errors for lake loader Delta Azure
Upgrade common-streams to 0.8.0-M5 (#83)
Delete files asynchronously (#82)
Upgrade common-streams 0.8.0-M4 (#81)
Avoid error on duplicate view name (#80)
Add option to exit on missing Iglu schemas (#79)
common-streams 0.8.x with refactored health monitoring (#78)
Create table concurrently with subscribing to stream of events (#77)
Iceberg fail fast if missing permissions on the catalog (#76)
Make alert messages more human-readable (#75)
Hudi loader should fail early if missing permissions on Glue catalog (#72)
Add alert & retry for delta/s3 initialization (#74)
Implement alerting and retrying mechanisms
Bump aws-hudi to 1.0.0-beta2 (#71)
Bump hudi to 0.15.0 (#70)
Allow disregarding Iglu field's nullability when creating output columns (#66)
Extend health probe to report unhealthy on more error scenarios (#69)
Fix bad rows resizing (#68)
Version 0.4.1 (2024-06-04)
--------------------------
Fix missing s3 dependency for Delta
Version 0.4.0 (2024-06-03)
--------------------------
Improve default Iceberg table properties (#62)
Bad rows must not exceed the maximum size allowed by the sink (#62)
Trap SIGTERM and start graceful shutdown (#62)
Improve default Hudi configuration settings (#62)
Enable syncing Hudi metadata to the Glue Catalog (#62)
Turn on metrics logging for Iceberg (#62)
Improve default Delta table properties (#62)
Bump common-streams and iceberg to latest versions (#62)
Fix AWS dependencies (#61)
Optimizations for larger compute instances (#60)
Bump spark and lake libraries to latest versions (#59)
Bump common-streams to 0.6.0 (#58)
Version 0.3.0 (2024-03-20)
--------------------------
Add AWS Glue as an Iceberg catalog (#55)
Bump common-streams to 0.5.0 (#56)
Fix crashing on invalid view name (#54)
Remove Snowflake as an Iceberg catalog (#53)
Fix and simplify BigLake as Iceberg catalog (#52)
Change default log level of Delta libraries (#50)
Adjust default spark memory configuration (#51)
Azure: authenticate with Event Hubs using OAuth2 (#49)
Version 0.2.0 (2024-01-29)
--------------------------
Fix regression: it is taking longer to commit a batch to the lake (#46)
Use SLF4J for Cats Effect starvation warning message (#45)
Custom AWS credentials provider should fall back to default credentials chain (#42)
Bump dependencies to latest versions for the 0.2.0 release (#41)
Seperate docker builds for minority lake formats (#40)
Integration tests for Hudi and Iceberg output formats (#38)
Parse events from bytes instead of string (#44)
Custom AWS credentials provider with support for external ID (#35)
Health probe should not be healthy until after table is created (#34)
Set user-agent headers when using GCP SDKs (#25)
Add Hudi as a table format (#32)
Refactor table writers so we can extend to more table formats (#31)
AWS variant of the Lake Loader (#20)
Bump common-streams to 0.2.0 (#30)
Add latency statsd metrics (#23)
Import common loaders lib from maven (#18)
Downgrade to scala 2.12 (#21)
Update the license to Snowplow Limited Use License Agreement 1.0 (#43)
Version 0.1.4 (2023-11-14)
--------------------------
Bug: pubsub source should nack, not ack, unhandled events during shutdown (#27)
Version 0.1.3 (2023-09-21)
--------------------------
Improvements to example config files (#14)
Use TIMESTAMP_MICROS as parquet timestamp type (#13)
Bump scala analytics sdk to 3.1.0 (#12)
Remove delayThreshold as a config option of pubsub sink (#11)
Fix GCP app hanging when event size exceeds buffer size (#10)
Version 0.1.2 (2023-09-18)
--------------------------
Delta: Support data skipping on tstamp columns (#9)
Retry writing delta after concurrent modification exceptions (#7)
Version 0.1.1 (2023-09-13)
--------------------------
Add license file to packaged jar (#5)
Fix loading when a schema needs recovery columns (#4)
Version 0.1.0 (2023-09-10)
--------------------------
Initial release