From b6d276311d7a28543cb6b3ed9310285b837e29a4 Mon Sep 17 00:00:00 2001 From: ian-at-airbyte Date: Fri, 10 Jan 2025 17:38:59 -0800 Subject: [PATCH 1/6] Added draft of release notes --- docs/release_notes/v-1.4.md | 64 +++++++++++++++++++++++++++++++++++++ docusaurus/sidebars.js | 2 ++ 2 files changed, 66 insertions(+) create mode 100644 docs/release_notes/v-1.4.md diff --git a/docs/release_notes/v-1.4.md b/docs/release_notes/v-1.4.md new file mode 100644 index 000000000000..0663edbc83b8 --- /dev/null +++ b/docs/release_notes/v-1.4.md @@ -0,0 +1,64 @@ +# Airbyte 1.4.0 + +Happy new year! Airbyte version 1.4.0 was released on January 16, 2025. We’re excited to share new improvements and changes to the Airbyte platform. + +## 🚀 Platform Changes + +Platform changes improve Airbyte for everyone with a self-managed instance. + +### Configure the schema refresh rate + +Use the new environment variable, `DISCOVER_REFRESH_WINDOW_MINUTES`, to set how often Airbyte refreshes schemas, in minutes. The default is once per day in self-managed instances. The maximum is once per minute. Set this to 0 to disable automatic schema refreshes. [Learn more about configuring Airbyte](../operator-guides/configuring-airbyte.md). + +```yml title="values.yaml" +worker: + env_vars: + DISCOVER_REFRESH_WINDOW_MINUTES: 1440 +``` + +### Connectors support custom image registries + +Connectors can now use custom image registries rather than Airbyte’s public Docker registry. If you configure Airbyte to use a custom image registry, it now automatically uses that registry for connector images as well. Previously, only platform images supported this. In this example, we set Airbyte’s `values.yaml` file to pull all images from GitHub. + +```yml title="values.yaml" +global: + image: + registry: ghcr.io/NAMESPACE +``` + +[Learn how to set up custom image registries](../deploying-airbyte/integrations/custom-image-registries). + +:::note +If you have custom internal connectors that specify an image using a fully qualified domain name (for example, `example.com/airbyte/source-postgres`), Airbyte ignores your configured image registry and pulls images from the domain specified by that connector. +::: + +### Reduced resource consumption + +Instances of Airbyte running multiple connections at a time now consume fewer resources, thanks to optimizations to the `airbyte-worker` pod. + +## 🚀 Self-Managed Enterprise Changes + +To learn more about Airbyte Self-Managed Enterprise, [talk to our sales team](https://airbyte.com/talk-to-sales). + +### Audit logging for role-based access control (RBAC) permissions changes + +We are excited to share that Self-Managed Enterprise now supports audit logging. This initial release focuses on providing you with full visibility into permission changes. This data will ensure you have records of any unauthorized changes and insider threats, making it easy to continue meeting your compliance obligations while using Airbyte. + +Audit logging requires you to configure Airbyte to read from and write to a blob storage solution (S3, GCS, Azure Blob Storage). This is configured in Airbyte’s `values.yaml` file. To enable audit logging, add the following environment variables: + +```yml title="values.yaml" +server: + env_vars: + AUDIT_LOGGING_ENABLED: true + STORAGE_BUCKET_AUDIT_LOGGING: # your-audit-logging-bucket +``` + +Once enabled, audit logs are written to the `/audit-logging/` directory as JSON files. These files have the following naming convention: `__`. + +## 🐛 Bug fixes + +- **Canceled syncs do not rerun until the next scheduled sync**: If you previously cancelled an in-progress sync, and the next sync as configured in the connection frequency was behind schedule, Airbyte would immediately start a new sync. In practice, many users had to choose to ‘Cancel Sync’ twice in a row to stop moving data. Airbyte now automatically waits until the next scheduled sync to move data. + +- **Reduce rate limit errors from the `airbyte-cron` service**: We fixed an [issue](https://github.com/airbytehq/airbyte/issues/30691) reported by the community that caused excessive rate limit errors on the `airbyte-cron` pod when users scheduled connections using the Airbyte CRON capability. + +- **Autorecovery for hanging connections**: Airbyte now packages a service to detect connections blocked by the unlikely event that a sync becomes stuck, and remains in a perpetual ‘in-progress’ state without moving data. This heartbeat service will detect syncs that are hanging, and automatically create a new job attempt. diff --git a/docusaurus/sidebars.js b/docusaurus/sidebars.js index 760d7b5eb075..57225e35d678 100644 --- a/docusaurus/sidebars.js +++ b/docusaurus/sidebars.js @@ -726,8 +726,10 @@ module.exports = { label: "Release Notes", link: { type: "generated-index", + description: "We release new self-managed versions of Airbyte regularly. Airbyte Cloud customers always have the latest enhancements.", }, items: [ + "release_notes/v-1.4", "release_notes/v-1.3", "release_notes/v-1.2", "release_notes/v-1.1", From d5e24afaf39135f2be93787f6da1819aab5d5391 Mon Sep 17 00:00:00 2001 From: ian-at-airbyte Date: Tue, 14 Jan 2025 15:21:51 -0800 Subject: [PATCH 2/6] Apply edits to release notes as suggested by Alex --- docs/release_notes/v-1.4.md | 24 ++++++++++++++++-------- 1 file changed, 16 insertions(+), 8 deletions(-) diff --git a/docs/release_notes/v-1.4.md b/docs/release_notes/v-1.4.md index 0663edbc83b8..e735ca4ad274 100644 --- a/docs/release_notes/v-1.4.md +++ b/docs/release_notes/v-1.4.md @@ -8,17 +8,27 @@ Platform changes improve Airbyte for everyone with a self-managed instance. ### Configure the schema refresh rate -Use the new environment variable, `DISCOVER_REFRESH_WINDOW_MINUTES`, to set how often Airbyte refreshes schemas, in minutes. The default is once per day in self-managed instances. The maximum is once per minute. Set this to 0 to disable automatic schema refreshes. [Learn more about configuring Airbyte](../operator-guides/configuring-airbyte.md). +Before starting a sync, Airbyte can run discovery to refresh schemas. However, running discovery has a performance cost. For large databases with thousands of tables, discovery can take a while, and it might not be necessary before every sync. + +Use the new environment variable, `DISCOVER_REFRESH_WINDOW_MINUTES`, to set the minimum number of minutes Airbyte will wait to refresh a schema. By setting a larger number, you delay automatic schema refreshes and improve sync performance. The default in self-managed instances is 1440 (once per day). The lowest interval you can set is 1 (once per minute). + +```yml title="values.yaml" +worker: + env_vars: + DISCOVER_REFRESH_WINDOW_MINUTES: 1440 # Airbyte automatically refreshes schemas no more than once per day (1440 minutes). +``` + + Set this to 0 to disable automatic schema refreshes. ```yml title="values.yaml" worker: env_vars: - DISCOVER_REFRESH_WINDOW_MINUTES: 1440 + DISCOVER_REFRESH_WINDOW_MINUTES: 0 # Airbyte does not automatically detect schema changes. ``` ### Connectors support custom image registries -Connectors can now use custom image registries rather than Airbyte’s public Docker registry. If you configure Airbyte to use a custom image registry, it now automatically uses that registry for connector images as well. Previously, only platform images supported this. In this example, we set Airbyte’s `values.yaml` file to pull all images from GitHub. +Connectors can now use custom image registries rather than Airbyte’s public Docker registry. If you configure Airbyte to use a custom image registry, it now automatically uses that registry for connector images as well as platform images. Previously, only platform images supported this. In this example, we set Airbyte’s `values.yaml` file to pull all images from GitHub. ```yml title="values.yaml" global: @@ -26,11 +36,9 @@ global: registry: ghcr.io/NAMESPACE ``` -[Learn how to set up custom image registries](../deploying-airbyte/integrations/custom-image-registries). +You must ensure copies of platform and connector images are available in your custom image registry. [Learn how to set up custom image registries](../deploying-airbyte/integrations/custom-image-registries). -:::note -If you have custom internal connectors that specify an image using a fully qualified domain name (for example, `example.com/airbyte/source-postgres`), Airbyte ignores your configured image registry and pulls images from the domain specified by that connector. -::: +[Custom Docker connectors](../operator-guides/using-custom-connectors/) in your workspace that specify an image using a fully qualified domain name (for example, `example.com/airbyte/your-custom-source`) ignore your configured custom image registry and pull images from the domain specified by that connector. ### Reduced resource consumption @@ -44,7 +52,7 @@ To learn more about Airbyte Self-Managed Enterprise, [talk to our sales team](ht We are excited to share that Self-Managed Enterprise now supports audit logging. This initial release focuses on providing you with full visibility into permission changes. This data will ensure you have records of any unauthorized changes and insider threats, making it easy to continue meeting your compliance obligations while using Airbyte. -Audit logging requires you to configure Airbyte to read from and write to a blob storage solution (S3, GCS, Azure Blob Storage). This is configured in Airbyte’s `values.yaml` file. To enable audit logging, add the following environment variables: +Audit logging requires you to configure a blob storage solution (S3, GCS, Azure Blob Storage). Then, configure Airbyte to read from and write to it with Airbyte’s `values.yaml` file. To enable audit logging, add the following environment variables: ```yml title="values.yaml" server: From f416a92058f91732a85ab2dea562bb7a92d806ba Mon Sep 17 00:00:00 2001 From: Ian Alton Date: Wed, 15 Jan 2025 09:05:22 -0800 Subject: [PATCH 3/6] Apply suggestions from code review Applied Alex's edits Co-authored-by: Alexandre Cuoci --- docs/release_notes/v-1.4.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/release_notes/v-1.4.md b/docs/release_notes/v-1.4.md index e735ca4ad274..99567a2dfe3f 100644 --- a/docs/release_notes/v-1.4.md +++ b/docs/release_notes/v-1.4.md @@ -8,9 +8,9 @@ Platform changes improve Airbyte for everyone with a self-managed instance. ### Configure the schema refresh rate -Before starting a sync, Airbyte can run discovery to refresh schemas. However, running discovery has a performance cost. For large databases with thousands of tables, discovery can take a while, and it might not be necessary before every sync. +At the start of a sync, Airbyte occasionally re-runs schema discovery to detect changes in your source, such as new fields or a change in column type. But running schema discovery has a performance cost, and may not always be necessary before every sync. -Use the new environment variable, `DISCOVER_REFRESH_WINDOW_MINUTES`, to set the minimum number of minutes Airbyte will wait to refresh a schema. By setting a larger number, you delay automatic schema refreshes and improve sync performance. The default in self-managed instances is 1440 (once per day). The lowest interval you can set is 1 (once per minute). +You may now use a new toggle, `DISCOVER_REFRESH_WINDOW_MINUTES`, to set the minimum number of minutes Airbyte will wait to refresh the schema for your sources. By setting a larger number, you run automatic schema detection less frequently, which can result in gains in sync performance. The default configuration in Airbyte Open Source is `1440`, which refreshes schemas every 24 hours. The lowest interval you can set is `1`, which refreshes schemas before every sync. ```yml title="values.yaml" worker: @@ -18,7 +18,7 @@ worker: DISCOVER_REFRESH_WINDOW_MINUTES: 1440 # Airbyte automatically refreshes schemas no more than once per day (1440 minutes). ``` - Set this to 0 to disable automatic schema refreshes. + Set this to 0 to disable automatic schema refreshes. All schema refreshes will need to be done manually via the Airbyte UI or API. ```yml title="values.yaml" worker: From b8e46430c38563c97c8b2cb33dbe6f7f553986f6 Mon Sep 17 00:00:00 2001 From: ian-at-airbyte Date: Wed, 15 Jan 2025 09:07:37 -0800 Subject: [PATCH 4/6] Moved reduced resource consumption to the bug fix section --- docs/release_notes/v-1.4.md | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/docs/release_notes/v-1.4.md b/docs/release_notes/v-1.4.md index e735ca4ad274..a32fc7fc8c34 100644 --- a/docs/release_notes/v-1.4.md +++ b/docs/release_notes/v-1.4.md @@ -40,10 +40,6 @@ You must ensure copies of platform and connector images are available in your cu [Custom Docker connectors](../operator-guides/using-custom-connectors/) in your workspace that specify an image using a fully qualified domain name (for example, `example.com/airbyte/your-custom-source`) ignore your configured custom image registry and pull images from the domain specified by that connector. -### Reduced resource consumption - -Instances of Airbyte running multiple connections at a time now consume fewer resources, thanks to optimizations to the `airbyte-worker` pod. - ## 🚀 Self-Managed Enterprise Changes To learn more about Airbyte Self-Managed Enterprise, [talk to our sales team](https://airbyte.com/talk-to-sales). @@ -65,6 +61,8 @@ Once enabled, audit logs are written to the `/audit-logging/` directory as JSON ## 🐛 Bug fixes +- **Reduced resource consumption**: Instances of Airbyte running multiple connections at a time now consume fewer resources, thanks to optimizations to the `airbyte-worker` pod. + - **Canceled syncs do not rerun until the next scheduled sync**: If you previously cancelled an in-progress sync, and the next sync as configured in the connection frequency was behind schedule, Airbyte would immediately start a new sync. In practice, many users had to choose to ‘Cancel Sync’ twice in a row to stop moving data. Airbyte now automatically waits until the next scheduled sync to move data. - **Reduce rate limit errors from the `airbyte-cron` service**: We fixed an [issue](https://github.com/airbytehq/airbyte/issues/30691) reported by the community that caused excessive rate limit errors on the `airbyte-cron` pod when users scheduled connections using the Airbyte CRON capability. From 9828d979ca7be8fb4d141c417758bb084dbe471c Mon Sep 17 00:00:00 2001 From: ian-at-airbyte Date: Thu, 16 Jan 2025 11:34:04 -0800 Subject: [PATCH 5/6] Add bucket auth informaton --- docs/release_notes/v-1.4.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/release_notes/v-1.4.md b/docs/release_notes/v-1.4.md index 45c628bc12e2..7198928c518b 100644 --- a/docs/release_notes/v-1.4.md +++ b/docs/release_notes/v-1.4.md @@ -48,7 +48,9 @@ To learn more about Airbyte Self-Managed Enterprise, [talk to our sales team](ht We are excited to share that Self-Managed Enterprise now supports audit logging. This initial release focuses on providing you with full visibility into permission changes. This data will ensure you have records of any unauthorized changes and insider threats, making it easy to continue meeting your compliance obligations while using Airbyte. -Audit logging requires you to configure a blob storage solution (S3, GCS, Azure Blob Storage). Then, configure Airbyte to read from and write to it with Airbyte’s `values.yaml` file. To enable audit logging, add the following environment variables: +Audit logging requires you to configure a blob storage solution (S3, GCS, Azure Blob Storage). This bucket must be accessible by Airbyte's pods using the same credentials it uses to access your log and state storage. It's easiest to reuse your existing bucket, but you can create a new bucket as long as the [authentication is identical](../enterprise-setup/implementation-guide#configuring-external-logging). + +Then, you configure Airbyte to read from and write to your bucket with Airbyte’s `values.yaml` file: ```yml title="values.yaml" server: From 313dd4d9f067bfc397f1e1a9dea137acd977e236 Mon Sep 17 00:00:00 2001 From: ian-at-airbyte Date: Fri, 17 Jan 2025 08:24:12 -0800 Subject: [PATCH 6/6] Remove audit logging from v1.4 --- docs/release_notes/v-1.4.md | 23 +---------------------- 1 file changed, 1 insertion(+), 22 deletions(-) diff --git a/docs/release_notes/v-1.4.md b/docs/release_notes/v-1.4.md index 7198928c518b..773dda3e022b 100644 --- a/docs/release_notes/v-1.4.md +++ b/docs/release_notes/v-1.4.md @@ -1,6 +1,6 @@ # Airbyte 1.4.0 -Happy new year! Airbyte version 1.4.0 was released on January 16, 2025. We’re excited to share new improvements and changes to the Airbyte platform. +Happy new year! Airbyte version 1.4.0 was released on January 17, 2025. We’re excited to share new improvements and changes to the Airbyte platform. ## 🚀 Platform Changes @@ -40,27 +40,6 @@ You must ensure copies of platform and connector images are available in your cu [Custom Docker connectors](../operator-guides/using-custom-connectors/) in your workspace that specify an image using a fully qualified domain name (for example, `example.com/airbyte/your-custom-source`) ignore your configured custom image registry and pull images from the domain specified by that connector. -## 🚀 Self-Managed Enterprise Changes - -To learn more about Airbyte Self-Managed Enterprise, [talk to our sales team](https://airbyte.com/talk-to-sales). - -### Audit logging for role-based access control (RBAC) permissions changes - -We are excited to share that Self-Managed Enterprise now supports audit logging. This initial release focuses on providing you with full visibility into permission changes. This data will ensure you have records of any unauthorized changes and insider threats, making it easy to continue meeting your compliance obligations while using Airbyte. - -Audit logging requires you to configure a blob storage solution (S3, GCS, Azure Blob Storage). This bucket must be accessible by Airbyte's pods using the same credentials it uses to access your log and state storage. It's easiest to reuse your existing bucket, but you can create a new bucket as long as the [authentication is identical](../enterprise-setup/implementation-guide#configuring-external-logging). - -Then, you configure Airbyte to read from and write to your bucket with Airbyte’s `values.yaml` file: - -```yml title="values.yaml" -server: - env_vars: - AUDIT_LOGGING_ENABLED: true - STORAGE_BUCKET_AUDIT_LOGGING: # your-audit-logging-bucket -``` - -Once enabled, audit logs are written to the `/audit-logging/` directory as JSON files. These files have the following naming convention: `__`. - ## 🐛 Bug fixes - **Reduced resource consumption**: Instances of Airbyte running multiple connections at a time now consume fewer resources, thanks to optimizations to the `airbyte-worker` pod.