Upgrading Apache Airflow in production without downtime

Upgrading Airflow is the kind of task that looks like a version bump in the ticket and feels like defusing something in practice. Once orchestration is load-bearing — when 100+ jobs and every downstream dashboard depend on the scheduler waking up — a failed upgrade isn't an inconvenience, it's a data outage with a long tail of backfills.

Here's the process I used for a major-version upgrade, and why almost all the work happened before the cutover.

Let deprecation warnings write your migration checklist

The single most useful thing you can do starts months earlier: stop ignoring deprecation warnings. Airflow is unusually good about warning you in the minor release before it removes something. Those warnings are a free, personalised migration checklist.

Rather than reading them one by one in the scheduler logs, collect them across every DAG:

# Surface every deprecation warning raised during DAG parsing
python -W error::DeprecationWarning -c "
from airflow.models import DagBag
bag = DagBag(include_examples=False)
for path, err in bag.import_errors.items():
    print(f'{path}: {err}')
"

Turning warnings into errors during parsing is deliberately aggressive. You want the noise now, on your laptop, not at 2am after the upgrade.

Pin providers separately from core

The thing that bites people on modern Airflow is that provider packages version independently of core. apache-airflow and apache-airflow-providers-amazon are on separate release trains, and a core upgrade can quietly drag a provider forward with it.

Constrain both explicitly:

AIRFLOW_VERSION=2.9.0
PYTHON_VERSION=3.11
CONSTRAINT_URL="https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-${PYTHON_VERSION}.txt"

pip install "apache-airflow==${AIRFLOW_VERSION}" --constraint "${CONSTRAINT_URL}"

The constraints file is the officially tested dependency set for that release. Installing without it is how you end up debugging a transitive dependency conflict that has nothing to do with your DAGs.

Rehearse against a copy of the real metadata database

Testing against a clean database proves very little. The interesting failures live in your metadata DB: years of task instances, serialized DAGs, XComs, and rows written by versions that no longer exist.

So the rehearsal is: restore a recent snapshot of production metadata into an isolated environment, and run the migration against that.

# Always inspect pending migrations before applying them
airflow db check-migrations
airflow db upgrade

Time it. A schema migration over a large task_instance table can take considerably longer than you'd expect, and knowing whether it's 40 seconds or 40 minutes is the difference between a planned window and an incident.

This rehearsal is also where you find out whether your DB backup actually restores — a fact worth confirming before you need it.

Test DAG compatibility as a build step

Every DAG should be import-tested before it ever reaches the upgraded scheduler. Import errors are the most common upgrade failure and the cheapest to catch:

# tests/test_dag_integrity.py
import pytest
from airflow.models import DagBag

def test_no_import_errors():
    bag = DagBag(include_examples=False)
    assert not bag.import_errors, f"DAG import failures: {bag.import_errors}"

def test_every_dag_has_an_owner_and_retries():
    bag = DagBag(include_examples=False)
    for dag_id, dag in bag.dags.items():
        assert dag.default_args.get("owner"), f"{dag_id} has no owner"
        assert dag.default_args.get("retries") is not None, f"{dag_id} has no retry policy"

If you only adopt one thing from this post, make it this test. It's about fifteen lines and it catches the failure mode that actually happens.

Drain, don't interrupt

The cutover itself should be uneventful if the rehearsal was honest. The ordering that matters:

1. Pause the schedulers, don't kill them. Let running tasks finish rather than orphaning them mid-write.
2. Wait for the queue to drain. Tasks killed mid-flight leave state that has to be reconciled by hand.
3. Snapshot the metadata DB immediately before migrating — this is your rollback point.
4. Run the migration, then start a single scheduler and watch it parse before scaling back up.
5. Unpause in waves, starting with low-stakes DAGs.

Picking a genuinely low-traffic window matters more than it sounds. Know your own schedule: if most pipelines fire between 01:00 and 05:00, a Sunday afternoon is nearly free.

Have a rollback plan you'd actually execute

"We'll roll back if needed" isn't a plan unless you've written down the steps and know the constraint: Airflow schema migrations are not reliably reversible. airflow db downgrade exists, but restoring the pre-migration snapshot is the path you can trust.

That means your real rollback is: stop schedulers, restore the snapshot, redeploy the previous image. Which in turn means the snapshot must be recent and verified — hence step 3.

What I'd tell someone doing this next week

The upgrade succeeded because the interesting parts happened weeks earlier: warnings triaged, providers pinned, a rehearsal against real data, DAG integrity in CI. By the time we touched production, we'd already done the migration once and knew how long it took.

Orchestration platforms earn trust slowly and lose it instantly. The goal of an upgrade isn't to ship a version number — it's for nobody to notice it happened.