Apache Airflow
Orchestration for hundreds of pipelines with owners, SLAs and honest failure handling.
Airflow is where our pipeline estates are orchestrated. Its value is not the scheduling — cron schedules things — but everything around it: dependency management between tasks, retries with backoff, backfills as a first-class operation, SLA monitoring, and a user interface where an operator can see what ran, what failed and why.
What separates an estate of two thousand DAGs that is manageable from one that is chaos is convention. Every DAG we ship has a named owner, an SLA appropriate to its business use, an alerting route that reaches a human who can act, and tasks that are idempotent so a rerun is always safe.
The most common thing we fix in an inherited Airflow estate is not the code but the operating model: no ownership, no SLAs, a single alert channel nobody monitors, and tasks that cannot safely be rerun — which means nobody dares backfill, and gaps become permanent.
Where it fits
What Apache Airflow genuinely gives us
Dependencies expressed properly
A task waits for its input to exist rather than running at a time when the input is usually there.
Backfill as a supported operation
Reprocessing a historical range is a command, not a bespoke script written under pressure.
Visibility an operator can use
A UI that shows run history, duration trends and failure context, so overnight support does not require the original engineer.
Python-native
DAGs are code, so they are reviewed, tested and version controlled like everything else.
What to orchestrate, and what to leave alone
Airflow is frequently over-applied. Once a team has an orchestrator, there is a pull to put everything in it, including jobs with no dependencies and no need for backfill that ran perfectly well on a simple schedule. Each one added carries a cost in DAG parsing, in complexity and in the attention of whoever maintains the deployment.
Our rule is that a job belongs in Airflow when at least one of three things is true: it depends on another job completing, it needs to be backfilled or rerun across historical dates, or its failure requires a defined escalation. A standalone hourly job that emails a report and matters little if it is late does not meet that bar and is better left where it is.
The corollary matters more. Anything that does meet the bar should be there completely, not half in and half out. The most damaging pattern we encounter is a pipeline where some steps are orchestrated and others run from cron on a separate machine, because the dependency graph is then partly fictional and the orchestrator reports success while a downstream step has silently failed. Either a pipeline is modelled honestly or the visibility Airflow provides is worse than none.
Orchestration is where data platforms are actually won or lost
Every organisation with data reaches the same moment: the number of scheduled jobs passes about fifteen, the dependencies between them become real, and cron stops being adequate. The symptoms are recognisable — a job that runs before its input is ready and silently produces yesterday’s numbers, a failure nobody notices until a manager queries a figure in a meeting, and a rerun procedure that exists only in one engineer’s head.
Airflow addresses that by making dependencies explicit and failures visible, but it rewards discipline and punishes casual use. The most common problem in the deployments we inherit is tasks that are not idempotent: rerunning them duplicates rows or corrupts a partition, so the team becomes afraid to rerun anything, which defeats the point of having an orchestrator at all. The second most common is business logic written inside the DAG file, which makes it untestable and means the scheduler is parsing heavy code every few seconds.
We build Airflow deployments where DAGs are thin declarations of dependency and schedule, the actual work lives in tested, importable modules, every task can be rerun safely for any date, data quality checks sit between stages so bad data fails loudly rather than propagating, and alerting distinguishes between a transient retry and a genuine failure that needs a person.
Sensors instead of hopeful schedules
A job scheduled at 03:00 because the source file usually lands by 02:30 will eventually process yesterday's file, and nobody will notice for a week. We use sensors that wait for actual data availability, with a timeout that raises an alert rather than allowing the DAG to proceed on missing input.
This one change eliminates a whole category of silent data quality incident — the kind where the pipeline reports success and the numbers are simply stale.
Observability beyond pass and fail
Every run records rows in, rows out, rows rejected, duration and the code version that produced it. Anomalies against historical norms — a table that usually receives forty thousand rows receiving four hundred — trigger investigation before the number reaches a report.
Duration trends matter too. A task that has been growing five per cent a week is going to breach its window in a month, and seeing that early is considerably cheaper than discovering it at 6 AM on a reporting day.
How Apache Airflow projects usually go wrong
These are the failure modes we look for first when we are called in to rescue somebody else's implementation.
Apache Airflow questions we get asked
Including where we would recommend something other than Apache Airflow. Call +91 70033 91355 and you will get the same answer from an engineer.
Airflow for most clients — the ecosystem, operator library and available talent are considerably deeper, and managed offerings exist on every cloud. Dagster has a genuinely better developer experience and asset-oriented model, and we use it where a team is starting fresh and values that. Prefect suits lighter workloads. The deciding factor is usually who will operate it in three years.
Managed — Cloud Composer, MWAA or Astronomer — unless you have platform engineering capacity or an on-premise requirement. Airflow operations are non-trivial: scheduler tuning, worker autoscaling, database maintenance and version upgrades. The managed premium is usually less than the cost of doing it well yourself.
Usually, and it is often the right answer. The recurring problems are consistent: non-idempotent tasks, clock-based scheduling, no SLAs, no ownership and unmonitored alerting. Those are fixable incrementally without a migration. We start with an audit and give you a ranked list — several clients have executed it themselves.
Services built on Apache Airflow
Technologies we pair it with
Tell us what is slowing your business down.
A 30-minute call with a senior engineer — not a salesperson. You leave with an architecture sketch and an honest cost range, whether or not you hire us.
Direct line
+91 70033 91355Mon–Sat · 9:30 AM – 7:30 PM IST · Sealdah, Kolkata