Building an Airflow platform from zero to 100+ production jobs

When I joined, there was no shared orchestration layer. Pipelines ran ad hoc — cron jobs here, a manual script run there, no consistent story for retries, scheduling, or "did last night's job actually finish." As the number of data workflows grew (ingestion, ESG scoring, analytics), that stopped being sustainable.

Starting from nothing

Standing up Airflow from scratch meant making a handful of decisions early that are annoying to change later:

• Deployment shape. We run on Kubernetes, so the executor and worker story had to fit the same cluster our other services live in, not a bolted-on VM.
• DAG conventions. Naming, tagging, and ownership conventions so that a DAG failing at 2am points clearly at who owns it and what it touches.
• Monitoring first, not later. Alerting on failed and long-running tasks was part of the initial rollout, not a follow-up ticket.

Getting these right early is what let the platform scale to what it runs today without a rewrite.

Where it ended up

The platform now orchestrates 100+ scheduled jobs — ingestion pipelines pulling in thousands of ESG filings, the DAGs feeding the ESG scoring platform, and downstream analytics jobs. It's the backbone the rest of the data platform runs on, not a side tool.

Upgrading it without breaking anything

Once a platform is load-bearing, upgrading it stops being optional and starts being risky. I planned and executed a major-version upgrade of Airflow in production — the kind of change where the honest goal isn't "add features," it's "nobody notices this happened." That meant testing DAG compatibility ahead of time, having a rollback plan, and picking a low-traffic window, rather than treating it as a routine pip install --upgrade.

The lesson that stuck with me: orchestration platforms earn trust slowly and lose it instantly. The upgrade going smoothly mattered more than the version bump itself.