• Pipelines exploded in count. Single jobs became thousands of interdependent tasks across teams.
• Cron couldn't express dependencies. "Run B only after A succeeded for that date" needed custom glue everywhere.
• Failures were silent. A bad partition at 2am surfaced at 9am in a Monday standup.
• Backfills were terrifying. Re-running a week of history meant praying you knew every downstream dependency.
• Observability was a wiki page. "What ran? When? Did it succeed?" lived in tribal knowledge.

from airflow.sdk import dag, task
from datetime import datetime

@dag(
    schedule="@hourly",
    start_date=datetime(2026, 1, 1),
    catchup=False,
    tags=["rag", "ai"],
)
def rag_ingest():

    @task
    def fetch_docs() -> list[str]:
        return list_new_documents_from_s3()

    @task
    def chunk_and_embed(docs) -> list[dict]:
        return [embed(c) for d in docs
                         for c in split(d)]

    @task
    def upsert_to_vector_db(records):
        pinecone.upsert(records)

    upsert_to_vector_db(chunk_and_embed(fetch_docs()))

rag_ingest()

• Retries with backoff — per-task or per-DAG, with exponential delay. Transient errors heal themselves.
• SLAs & alerting — declare "this task must finish in 30 min" and get paged when it doesn't.
• Sensors — wait for a file, a partition, an API, or another DAG before continuing.
• Deferrable operators — long waits without burning a worker slot; freed back to the pool.
• Callbacks — on_failure, on_success, on_retry, sla_miss — wire into Slack, PagerDuty, anything.
• Backfilling — re-run any historical window through the UI or CLI, with full dependency awareness.
• Catchup & max_active_runs — control how aggressively missed runs are recovered.
• Manual retry from the UI — clear a single failed task and let Airflow figure out what to re-run.

@dag(
    schedule="@hourly",
    start_date=datetime(2026, 1, 1),
    catchup=False,
    max_active_runs=1,
    default_args={
        "retries": 3,
        "retry_delay": timedelta(minutes=5),
        "retry_exponential_backoff": True,
    },
    on_failure_callback=page_oncall,
    sla_miss_callback=alert_slack,
)
def critical_pipeline():

    @task(
        retries=5,
        execution_timeout=timedelta(minutes=15),
        sla=timedelta(minutes=30),
    )
    def call_external_api():
        return requests.get("https://...").json()

    @task.sensor(poke_interval=30,
                  timeout=3600,
                  mode="reschedule")
    def wait_for_upstream():
        return s3_object_exists("s3://lake/today.parquet")

    wait_for_upstream() >> call_external_api()

critical_pipeline()

@dag(schedule="@daily",
      start_date=datetime(2026, 1, 1))
def ml_eval():

    @task
    def list_models() -> list[str]:
        return ["claude-opus-4-7",
                "gpt-5",
                "gemini-2-pro"]

    @task
    def evaluate(model: str) -> dict:
        return run_eval_suite(model)

    @task.branch
    def pick_winner(scores: list[dict]):
        best = max(scores, key=lambda s: s["acc"])
        return ("promote" if best["acc"] > 0.9
                else "skip")

    @task
    def promote(): ...

    @task
    def skip(): ...

    # expand = one task instance per item
    scores = evaluate.expand(model=list_models())
    pick_winner(scores) >> [promote(), skip()]

ml_eval()

from airflow.sdk import asset, dag, task, Asset
from datetime import datetime

# --- Producer: declares a data asset ---
@asset(schedule="@hourly",
        uri="s3://lake/customer_events/")
def customer_events(context):
    df = extract_events()
    write_parquet("s3://lake/customer_events/", df)
    # asset is "materialized" on success

# --- Consumer: triggered BY the asset ---
@dag(
    schedule=[customer_events],   # data-aware
    start_date=datetime(2026, 1, 1),
)
def feature_store_refresh():
    @task
    def build_features():
        ...
    build_features()

# --- Or combine multiple assets ---
@dag(schedule=(customer_events & orders),
      start_date=datetime(2026, 1, 1))
def analytics_rollup(): ...

• Human-in-the-loop — first-class pause/resume so agents and humans share workflows: approvals, content review, escalation steps.
• AI-assisted operations — natural-language debugging, DAG authoring, and triage built on top of Airflow's metadata and logs.
• Multi-language tasks (AIP-108) — run Java, Kotlin, Go, and more alongside Python, on the same orchestrator.
• Event-driven scheduling — data-aware and asset-aware triggering, not just cron. Pipelines react to upstream data, not the wall clock.
• Task SDK — a stable, language-agnostic contract between task code and Airflow core. The foundation for everything above.

• You learn from the best. Free mentorship from world-class engineers, in public, on real code.
• It compounds your career. Public proof of work travels with you across every job, indefinitely.
• You shape the tools you use. Stop filing tickets — fix it upstream once, benefit forever.
• You build a global network. Contributors become collaborators, references, co-founders, and friends.
• You get paid in optionality. Maintainer reputation opens doors that interviews never will.
• It's a gift back. Every line of code you wrote today stood on someone else's open source work.

• Started as a user. Hit a paper cut, opened the repo to see if I could fix it myself.
• First PR was tiny. A small documentation correction. It got merged. That hooked me.
• Bugs → features. Each PR taught me more of the codebase than any tutorial could.
• Reviews → design discussions. Slowly moved from "fix this" to "should we do this".
• Committer invite. Months later, after consistent contributions and reviews.
• The pattern: show up, ship small things, be useful to other contributors.