From a0988b52a589d9c327200e670ce487c39d224756 Mon Sep 17 00:00:00 2001 From: Elad Kalif <45845474+eladkal@users.noreply.github.com> Date: Tue, 4 Aug 2026 10:57:25 +0300 Subject: [PATCH] [v3-3-test] Update description on What is Airflow (#70547) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * Update description on What is Airflow * change link to registry * Update airflow-core/docs/index.rst Co-authored-by: Przemysław Mirowski <17602603+Miretpl@users.noreply.github.com> * fix --------- (cherry picked from commit ee2e15e5e5108756925ab3cc76dfa4efe8552e2d) Co-authored-by: Elad Kalif <45845474+eladkal@users.noreply.github.com> Co-authored-by: Rahul Vats <43964496+vatsrahul1001@users.noreply.github.com> Co-authored-by: Przemysław Mirowski <17602603+Miretpl@users.noreply.github.com> --- README.md | 2 ++ airflow-core/docs/index.rst | 6 ++++-- 2 files changed, 6 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 1ab5c182f5fd2..d63db7c421835 100644 --- a/README.md +++ b/README.md @@ -84,6 +84,8 @@ Use Airflow to author workflows (Dags) that orchestrate tasks. The Airflow sched Airflow works best with workflows that are mostly static and slowly changing. When the Dag structure is similar from one run to the next, it clarifies the unit of work and continuity. Other similar projects include [Luigi](https://github.com/spotify/luigi), [Oozie](https://oozie.apache.org/) and [Azkaban](https://azkaban.github.io/). +Beyond traditional data pipelines, Airflow is widely used to orchestrate machine learning workflows — training, retraining, evaluation, and deployment — and increasingly to orchestrate agentic and LLM-based workloads, coordinating the steps of an AI pipeline (data prep, tool calls, model invocation, evaluation) rather than acting as the agent itself. This isn't a new direction: teams have run ML and AI workloads on Airflow for years, and the ecosystem of providers supporting these use cases (see the [AI & ML section of the Airflow registry](https://airflow.apache.org/registry/providers/?category=ai-ml)) continues to grow. + Airflow is commonly used to process data, but has the opinion that tasks should ideally be idempotent (i.e., results of the task will be the same, and will not create duplicated data in a destination system), and should not pass large quantities of data from one task to the next (though tasks can pass metadata using Airflow's [XCom feature](https://airflow.apache.org/docs/apache-airflow/stable/concepts/xcoms.html)). For high-volume, data-intensive tasks, a best practice is to delegate to external services specializing in that type of work. Airflow is not a streaming solution, but it is often used to process real-time data, pulling data off streams in batches. diff --git a/airflow-core/docs/index.rst b/airflow-core/docs/index.rst index 4280171e42dc9..879ccb50e0871 100644 --- a/airflow-core/docs/index.rst +++ b/airflow-core/docs/index.rst @@ -19,8 +19,10 @@ What is Airflow®? ========================================= `Apache Airflow® `_ is an open-source platform for developing, scheduling, -and monitoring batch-oriented workflows. Airflow's extensible Python framework enables you to build workflows -connecting with virtually any technology. A web-based UI helps you visualize, manage, and debug your workflows. +and monitoring workflows, such as traditional time based or event-triggered batch-oriented data pipelines, machine learning, model training, +and agentic or LLM-based workloads. Airflow's extensible Python framework enables you to build workflows +connecting with virtually any technology, with a growing set of providers for orchestrating AI and agentic +tools alongside the rest of your pipeline. A web-based UI helps you visualize, manage, and debug your workflows. You can run Airflow in a variety of configurations — from a single process on your laptop to a distributed system capable of handling massive workloads.