Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Xcom req as per 1.3.3 and airflow 2.5 #1483

Merged
merged 11 commits into from
Dec 23, 2022
7 changes: 4 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,15 +47,16 @@ pip install astro-sdk-python[amazon,google,snowflake,postgres]
airflow db init
```

> **Note:** `AIRFLOW__CORE__ENABLE_XCOM_PICKLING` no longer needs to be enabled from `astro-sdk-python` release 1.2 and above. This functionality is now deprecated as our custom xcom backend handles serialization.
Users can either use a custom XCom backend [`AstroCustomXcomBackend`](https://astro-sdk-python.readthedocs.io/en/latest/guides/xcom_backend.html) with Xcom pickling disabled (or) enable Xcom pickling.
Currently, custom XCom backends are limited to data types that are json serializable. Since Dataframes are not json serializable, we need to enable XCom pickling to store dataframes.
> **Note:** `AIRFLOW__CORE__ENABLE_XCOM_PICKLING` no longer needs to be enabled from astro-sdk-python release 1.2 and above. This functionality is now deprecated.
> - For airflow version < 2.5 and astro-sdk-python release < 1.2 Users can either use a custom XCom backend [AstroCustomXcomBackend](https://astro-sdk-python.readthedocs.io/en/latest/guides/xcom_backend.html#xcom-backend) with Xcom pickling disabled (or) enable Xcom pickling.
kaxil marked this conversation as resolved.
Show resolved Hide resolved
> - For airflow version >= 2.5 and astro-sdk-python release >= 1.3.3 Users can either use [Airflow's Xcom backend](https://astro-sdk-python.readthedocs.io/en/latest/guides/xcom_backend.html#airflow_xcom_backend) with Xcom pickling disabled (or) enable Xcom pickling.

The data format used by pickle is Python-specific. This has the advantage that there are no restrictions imposed by external standards such as JSON or XDR (which can’t represent pointer sharing); however it means that non-Python programs may not be able to reconstruct pickled Python objects.

Read more: [enable_xcom_pickling](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#enable-xcom-pickling) and [pickle](https://docs.python.org/3/library/pickle.html#comparison-with-json):



2. Create a SQLite database for the example to run with:

```shell
Expand Down
27 changes: 27 additions & 0 deletions python-sdk/docs/guides/xcom_backend.rst
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,9 @@
XCom Backend
============

.. note::
Recommended to be used with airflow < 2.5

The custom XCom backend adds special handling to Astro's custom constructs (see :ref:`concepts`) so they can
be used without enabling XCom picking (the ``xcom_pickling`` configuration). When the custom constructs are
not accessed, this is simply a wrapper around Airflow's default XCom backend, so a migration from the default
Expand Down Expand Up @@ -48,3 +51,27 @@ metadatabase instead, by setting

or the environment variable ``AIRFLOW__ASTRO_SDK__STORE_DATA_LOCAL_DEV`` instead. Note that this is considered
suboptimal, and should not be used in a production environment.

.. _airflow_xcom_backend:

======================
Airflow's XCom Backend
======================

.. note::
Recommended to be used with airflow >= 2.5

You don't need to use a Custom XCom backend or XCom pickling for Airflow >=2.5 and SDK 1.3+. From Airflow 2.5 onwards, airflow can internally handle serialization and deserialization of custom constructs of SDK. All we need to do to enable this feature is set a few configs in airflow’s config file as shown below.

.. code-block:: ini

[core]
enable_xcom_pickling = false
allowed_deserialization_classes = airflow.* astro.*

or we can also set env variables like

.. code-block:: ini

AIRFLOW__CORE__ENABLE_XCOM_PICKLING = false
kaxil marked this conversation as resolved.
Show resolved Hide resolved
AIRFLOW__CORE__ALLOWED_DESERIALIZATION_CLASSES=airflow.* astro.*