Version 2.9.2 airflow db migrate unsupported queries for mysql

[EnableServices]

Apache Airflow version

2.9.2

If "Other Airflow 2 version" selected, which one?

No response

What happened?

Running airflow db migrate after upgrade to version 2.9.2, experienced duplicate key errors adding constraints

What you think should happen instead?

The script execution should not halt with the following error:

sqlalchemy.exc.OperationalError: (MySQLdb.OperationalError) (1061, "Duplicate key name 'connection_conn_id_uq'")
[SQL: ALTER TABLE connection ADD CONSTRAINT connection_conn_id_uq UNIQUE (conn_id)]

How to reproduce

Install Airflow 2.9.1 using MySQL backend
Upgrade to Airflow 2.9.2
Run airflow db migrate

Operating System

Debian bookworm

Versions of Apache Airflow Providers

N/A

Deployment

Official Apache Airflow Helm Chart

Deployment details

No response

Anything else?

Within a code block intended for MySQL, the migration script https://github.com/apache/airflow/blob/2.9.2/airflow/migrations/versions/0142_2_9_2_fix_inconsistency_between_ORM_and_migration_files.py uses the statement

DROP CONSTRAINT

which is not supported on MySQL

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[kaxil]

cc @jedcunningham @phanikumv

[jedcunningham]

@EnableServices what version of MySQL are you on? I am not able to reproduce this failure, and MySQL 8 does support DROP CONSTRAINT (search for it in this page - can't deep link 😞).

[jedcunningham]

#40314 will fix 8.0.0-8.0.19. However, if you are using 8.0.0-8.0.19, I'd highly recommend you upgrade since it's over 4 years old at this point.

[potiuk]

#40314 will fix 8.0.0-8.0.19. However, if you are using 8.0.0-8.0.19, I'd highly recommend you upgrade since it's over 4 years old at this point.

Yet another reason why we should consider dropping mysql :) ?

I guess this is from (Changes in MySQL 8.0.19 (2020-01-13, General Availability)

When CHECK constraints were implemented in MySQL 8.0.16, ALTER TABLE supported DROP CHECK and
ALTER CHECK syntax as MySQL extensions to standard SQL for modifying check constraints,
but did not support the more general (and SQL standard) DROP CONSTRAINT and 
ALTER CONSTRAINT syntax for modifying existing constraints of any type. That syntax
is now supported; the constraint type is determined from the constraint name. (WL #12787)

[EnableServices]

That feels like a low blow @potiuk but on balance I think it's fair :)

On re-reading this page (https://airflow.apache.org/docs/apache-airflow/stable/howto/set-up-database.html), I can see that although it does suggest support for all versions of MySQL 8.0 - it does say "Old versions may not support all SQL statements."

Thank you @jedcunningham for the PR

[potiuk]

That feels like a low blow @potiuk but on balance I think it's fair :)

:) I think we got quite a few "low blows" from MySQL over the past 4 years (starting from bad decisions about encoding, limiting the index sizes, locking far too many things during queries and causing deadlocks, bad handling and expiring of MySQL certificates in their apt repository that necessitated manually updating all past images we released for the last 4 years - the last one took 2 days of my life out ). I'd say they deserve some kind of retaliation :D

[jedcunningham]

Full disclosure: I have no idea if there are other queries that'll break as you start actually running Airflow. I only tested the migrations on an empty db :)