Fernet InvalidToken

[hpatel-higi]

Apache Airflow version

2.1.3 (latest released)

Operating System

Linux

Versions of Apache Airflow Providers

apache-airflow-providers-apache-hive==2.0.2
apache-airflow-providers-ftp==2.0.0
apache-airflow-providers-http==2.0.0
apache-airflow-providers-imap==2.0.0
apache-airflow-providers-jdbc==2.0.0
apache-airflow-providers-postgres==2.0.0
apache-airflow-providers-slack==4.0.0
apache-airflow-providers-snowflake==2.1.0
apache-airflow-providers-sqlite==2.0.0
apache-airflow-providers-ssh==2.1.0

Deployment

Docker-Compose

Deployment details

version: "2.1"
services:
postgres:
image: postgres:13.4
container_name: "airflow_postgres_qa"
environment:
- POSTGRES_HOST=postgres
- POSTGRES_PORT:5432
- POSTGRES_USER=airflow
- POSTGRES_PASSWORD=airflow
- POSTGRES_DB=airflow
webserver:
image: airflow_docker
container_name: "airflow_webserver_qa"
entrypoint: "/entrypoint.sh"
command: "webserver"
restart: always
depends_on:
- postgres
ports:
- "9003:9003"
# - "2222:2222"
# - "8080:80"
environment:
EXECUTOR: "Local"
AIRFLOW__WEBSERVER__BASE_URL: "http://localhost:9003"
volumes:
- ../../keys:/usr/local/airflow/keys
- ../pipes:/usr/local/airflow/configs/snowpipe
- ../../support_files:/usr/local/airflow/support_files
- ../../dags:/usr/local/airflow/dags
- ../../sql:/usr/local/airflow/sql
- ../access_control.json:/usr/local/airflow/configs/access_control/access_control.json
healthcheck:
test: ["CMD-SHELL", "[ -f /usr/local/airflow/airflow-webserver.pid ]"]
interval: 30s
timeout: 30s
retries: 3

What happened

Adding or deleting connections using CLI in entrypoint.sh file throws error:

Traceback (most recent call last):

File "/usr/local/bin/airflow", line 8, in <module>

sys.exit(main())

File "/usr/local/lib/python3.9/site-packages/airflow/__main__.py", line 40, in main

args.func(args)

File "/usr/local/lib/python3.9/site-packages/airflow/cli/cli_parser.py", line 48, in command

return func(*args, **kwargs)

File "/usr/local/lib/python3.9/site-packages/airflow/utils/cli.py", line 91, in wrapper

return f(*args, **kwargs)

File "/usr/local/lib/python3.9/site-packages/airflow/cli/commands/connection_command.py", line 229, in connections_delete

to_delete = session.query(Connection).filter(Connection.conn_id == args.conn_id).one()

File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/query.py", line 3490, in one

ret = self.one_or_none()

File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/query.py", line 3459, in one_or_none

ret = list(self)

File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/loading.py", line 100, in instances

cursor.close()

File "/usr/local/lib/python3.9/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__

compat.raise_(

File "/usr/local/lib/python3.9/site-packages/sqlalchemy/util/compat.py", line 182, in raise_

raise exception

File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/loading.py", line 80, in instances

rows = [proc(row) for row in fetch]

File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/loading.py", line 80, in <listcomp>

rows = [proc(row) for row in fetch]

File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/loading.py", line 601, in _instance

state.manager.dispatch.load(state, context)

File "/usr/local/lib/python3.9/site-packages/sqlalchemy/event/attr.py", line 322, in __call__

fn(*args, **kw)

File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/mapper.py", line 3397, in _event_on_load

instrumenting_mapper._reconstructor(state.obj())

File "/usr/local/lib/python3.9/site-packages/airflow/models/connection.py", line 150, in on_db_load

if self.password:

File "/usr/local/lib/python3.9/site-packages/sqlalchemy/orm/attributes.py", line 365, in __get__

retval = self.descriptor.__get__(instance, owner)

File "/usr/local/lib/python3.9/site-packages/airflow/models/connection.py", line 235, in get_password

return fernet.decrypt(bytes(self._password, 'utf-8')).decode()

File "/usr/local/lib/python3.9/site-packages/cryptography/fernet.py", line 194, in decrypt

raise InvalidToken

cryptography.fernet.InvalidToken

What you expected to happen

Expected the connections to be deleted and readded again.

How to reproduce

• Dont set FERNET_KEY value in airflow.cfg file
• Set FERNET_KEY from inside entrypoint.sh file
• Run container first time
  • First time it should work fine
• Run container again and it will throw Invalid Token errors

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[mik-laj]

In case you do not set the fernet key, Airflow will generate a new key, which means that if you restart the container, you may lose access to the key. You should explicitly define the key in docker-compose.yaml. You can also set an empty string to disable encryption.
Here is example:

airflow/docs/apache-airflow/start/docker-compose.yaml

Line 58 in 2fd3f27

AIRFLOW__CORE__FERNET_KEY: ''

[hpatel-higi]

In my entrypoint.sh file i have this line of code to generate the fernet key

: "${AIRFLOW__CORE__FERNET_KEY:=${FERNET_KEY:=$(python -c "from cryptography.fernet import Fernet; FERNET_KEY = Fernet.generate_key().decode(); print(FERNET_KEY)")}}"

Are you saying that i should not generate it in the entrypoint.sh but instead just provide it in the docker-compose file??

This worked like this in 1.10.8. We're upgrading to 2.1.3.

[mik-laj]

I don't understand why it worked in Airflow 1.10, but generating a new key every time the container is started will definitely not allow the data to be decrypted. You must use the same key for encryption and decryption.

You may not have had Fernet at all in Airflow 1.10 as it requires the installation of the crypto package. In Airflow 2.0, the fernet is turned on by default and you can turn it off if you want.
https://github.com/apache/airflow/blob/main/UPDATING.md#fernet-is-enabled-by-default

[hpatel-higi]

I actually have the crypto package installed in my airflow 1.10.8 docker container.

here is the pip freeze:

alembic==1.7.1
amqp==2.6.1
apache-airflow==1.10.8
apispec==1.3.3
argcomplete==1.12.3
asn1crypto==1.4.0
attrs==19.3.0
azure-common==1.1.27
azure-core==1.18.0
azure-data-tables==12.1.0
azure-storage-blob==2.1.0
azure-storage-common==2.1.0
Babel==2.9.1
bcrypt==3.2.0
billiard==3.6.4.0
boto3==1.13.26
botocore==1.16.26
cached-property==1.5.2
cachelib==0.3.0
cachetools==4.2.2
cattrs==0.9.2
celery==4.4.7
certifi==2020.12.5
cffi==1.13.2
chardet==3.0.4
click==7.1.2
cloudpickle==1.6.0
colorama==0.4.4
colorlog==4.0.2
configparser==3.5.3
contextvars==2.4
croniter==0.3.37
cryptography==2.9.2
dask==2021.3.0
dataclasses==0.8
datadog==0.42.0
defusedxml==0.7.1
dill==0.3.4
distributed==1.28.1
dnspython==2.1.0
docutils==0.15.2
email-validator==1.1.3
Flask==1.1.4
Flask-Admin==1.5.4
Flask-AppBuilder==2.3.4
Flask-Babel==1.0.0
Flask-Bcrypt==0.7.1
Flask-Caching==1.3.3
Flask-JWT-Extended==3.25.1
Flask-Login==0.4.1
Flask-OAuthlib==0.9.6
Flask-OpenID==1.3.0
Flask-SQLAlchemy==2.5.1
flask-swagger==0.2.13
Flask-WTF==0.14.3
flower==0.9.7
funcsigs==1.0.2
furl==2.1.2
future==0.16.0
google-api-core==1.31.2
google-auth==1.35.0
googleapis-common-protos==1.53.0
graphviz==0.17
gunicorn==19.10.0
HeapDict==1.0.1
hmsclient==0.1.1
humanize==3.11.0
idna==2.9
ijson==2.6.1
immutables==0.16
importlib-metadata==4.8.1
importlib-resources==5.2.2
iso8601==0.1.16
isodate==0.6.0
itsdangerous==1.1.0
JayDeBeApi==1.2.3
Jinja2==2.10.3
jmespath==0.10.0
JPype1==1.3.0
json-merge-patch==0.2
jsonschema==3.2.0
kombu==4.6.11
lazy-object-proxy==1.6.0
lockfile==0.12.2
Mako==1.1.5
Markdown==2.6.11
MarkupSafe==2.0.1
marshmallow==2.21.0
marshmallow-enum==1.5.1
marshmallow-sqlalchemy==0.23.1
msgpack==1.0.2
msrest==0.6.21
mysqlclient==1.3.14
natsort==7.1.1
ndg-httpsclient==0.5.1
numpy==1.19.5
oauthlib==2.1.0
opencensus==0.7.13
opencensus-context==0.1.2
opencensus-ext-azure==1.0.8
orderedmultidict==1.0.1
oscrypto==1.2.1
packaging==21.0
pandas==0.25.3
pendulum==1.4.4
prison==0.2.1
prometheus-client==0.8.0
protobuf==3.17.3
psutil==5.8.0
psycopg2-binary==2.9.1
pyarrow==0.17.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycparser==2.20
pycryptodomex==3.10.1
Pygments==2.10.0
PyHive==0.6.4
PyJWT==1.7.1
pymssql==2.1.4
pyodbc==4.0.30
pyOpenSSL==19.1.0
pyparsing==2.4.7
pyrsistent==0.18.0
python-daemon==2.1.2
python-dateutil==2.8.2
python3-openid==3.2.0
pytz==2020.1
pytzdata==2020.1
PyYAML==5.4.1
redis==2.10.6
requests==2.23.0
requests-oauthlib==1.1.0
rsa==4.7.2
s3transfer==0.3.7
setproctitle==1.2.2
six==1.16.0
slackclient==1.3.2
snowflake-connector-python==2.2.6
snowflake-ingest==1.0.2
snowflake-sqlalchemy==1.2.3
sortedcontainers==2.4.0
SQLAlchemy==1.3.15
SQLAlchemy-JSONField==0.9.0
SQLAlchemy-Utils==0.37.8
tabulate==0.8.9
tblib==1.7.0
tenacity==4.12.0
termcolor==1.1.0
text-unidecode==1.2
thrift==0.13.0
toolz==0.11.1
tornado==5.1.1
typing==3.7.4.3
typing-extensions==3.10.0.2
tzlocal==1.5.1
unicodecsv==0.14.1
urllib3==1.25.11
vine==1.3.0
websocket-client==0.54.0
Werkzeug==0.16.0
WTForms==2.3.3
zict==2.0.0
zipp==3.5.0
zope.deprecation==4.4.0

[hpatel-higi]

Here is a screenshot of the connections page from the airflow 1.10.8 website. You can see the connections are marked as Encrypted.