← back to projects
cat airflow-platform/README.md
Airflow Orchestration Platform, Zero to Production
DSP Investment Managers · 2025
Built the company's Airflow platform from scratch — now runs 100+ jobs
Apache AirflowPythonDockerKubernetesAWS S3
Problem
There was no orchestration platform in place — pipelines ran ad hoc, with no shared scheduling, retry, or monitoring story as the number of data workflows grew.
Approach
- ▸Stood up Apache Airflow from the ground up: deployment architecture, DAG conventions, environment configuration, and monitoring.
- ▸Onboarded ingestion, ESG scoring, and analytics workflows onto the platform as the primary orchestration layer.
- ▸Planned and executed a major-version upgrade of Airflow in production with no disruption to scheduled jobs.
Impact
- ✓Now orchestrates 100+ production jobs across ingestion, ESG scoring, and analytics.
- ✓Became the backbone that the ESG platform and Capitaline ingestion pipelines run on.
- ✓Successfully upgraded to the latest major Airflow version in production.