With Apache Airflow you can design your ETL
as elegant Python code you would love to maintain and debug.
Usually we use Apache Airflow for bulk DB updates. So this is highly optimized SQL queries and so on.
But from time to time you would like to use SQLAlchemy models
inside your DAG for some not so massive
but complex operations with DB.
And Apache Airflow even based on SQLAlchemy!
For example this is how to get Apache Airflow Connections which id’s started with my_prefix_:
In common DAG you would not use SQLAlchemy - for bulk operations that would be just too slow.
If you do need SQLAlchemy model inside DAG you can get SQLAlchemy session for example from
But if you are going to do that in many Apache Airflow tasks this code will unnecessary complicate
you business logic code. Moreover you should close the DB connection to prevent connection leakage.
So this is additional try-finally around your code and it will became even more obscure.
Luckily you can easily create SQLAlchemy Operator for Apache Airflow and encapsulate all this code in it.
For example this is SQLAlchemy Operator for Postgres.