feat: Add [logging] task_log_to_stdout config to forward task logs to worker stdout

[mtraynham]

Summary

• Adds a new [logging] task_log_to_stdout boolean config option (default False) that forwards task subprocess log messages to the worker's stdout, in addition to the task log file and any configured remote log handler.
• Passes the config value as subprocess_logs_to_stdout in both LocalExecutor and CeleryExecutor call sites to supervise().

Motivation

In Airflow 3, tasks run in a forked subprocess managed by WatchedSubprocess. The subprocess uses structlog over a JSON socket channel back to the supervisor, which writes logs only to the task log file by default. The user's LOGGING_CONFIG dict (via AIRFLOW__LOGGING__LOGGING_CONFIG_CLASS) is never applied inside the task subprocess, so the Airflow 2 pattern of adding a StreamHandler to the airflow.task logger no longer works (https://stackoverflow.com/a/52261504).

The subprocess_logs_to_stdout flag already exists on WatchedSubprocess and supervise(), but defaults to False and is not exposed as a config option. Neither CeleryExecutor nor LocalExecutor passes True. There is no way for users to enable it without monkey-patching.

Container-based deployments (ECS, Kubernetes, Cloud Run, etc.) commonly rely on stdout-based log collection (CloudWatch, Fluentd, Fluent Bit, Datadog, etc.). Without this option, task execution logs are invisible to these collectors.

In case of an existing issue, reference it using one of the following:

• related: Celery executor does not log to stdout for Elasticsearch #49863
• related: [v3.0.4] Task logs are no longer visible in stdout only in files #54501
• related: Custom logging handler is ignored #58053

---

Was generative AI tooling used to co-author this PR?

• Yes (please specify the tool below)

---

• Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
• For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
• When adding dependency, check compliance with the ASF 3rd Party License Policy.
• For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

[mtraynham]

This does have the side-effect that all logs are tagged with supervisor.py and lose their scope of source file.

[potiuk]

@mtraynham This PR has been converted to draft because it does not yet meet our Pull Request quality criteria.

Issues found:

• ❌ mypy (type checking): Failing: CI image checks / MyPy checks (mypy-providers). Run prek --stage manual mypy-providers --all-files locally to reproduce. You need breeze ci-image build --python 3.10 for Docker-based mypy. See mypy (type checking) docs.
• ❌ Provider tests: Failing: provider distributions tests / Compat 2.11.1:P3.10:, provider distributions tests / Compat 3.0.6:P3.10:, provider distributions tests / Compat 3.1.8:P3.10:. Run provider tests with breeze run pytest <provider-test-path> -xvs. See Provider tests docs.
• ❌ Other failing CI checks: Failing: Postgres tests: core / DB-core:Postgres:14:3.10:Core...Serialization, MySQL tests: core / DB-core:MySQL:8.0:3.10:Core...Serialization, Sqlite tests: core / DB-core:Sqlite:3.10:Core...Serialization, Low dep tests:core / All-core:LowestDeps:14:3.10:Core...Serialization. Run prek run --from-ref main locally to reproduce. See static checks docs.

What to do next:

• The comment informs you what you need to do.
• Fix each issue, then mark the PR as "Ready for review" in the GitHub UI - but only after making sure that all the issues are fixed.
• There is no rush — take your time and work at your own pace. We appreciate your contribution and are happy to wait for updates.
• Maintainers will then proceed with a normal review.

Converting a PR to draft is not a rejection — it is an invitation to bring the PR up to the project's standards so that maintainer review time is spent productively. There is no rush — take your time and work at your own pace. We appreciate your contribution and are happy to wait for updates. If you have questions, feel free to ask on the Airflow Slack.

[Copilot]

Pull request overview

Adds a new [logging] task_log_to_stdout configuration option to allow forwarding task subprocess logs to the worker’s stdout (in addition to existing task log sinks), wiring it through both LocalExecutor and CeleryExecutor by passing subprocess_logs_to_stdout into the task supervisor.

Changes:

• Add [logging] task_log_to_stdout boolean config option (default False) to config.yml.
• Pass the config through to supervise() from LocalExecutor task execution.
• Pass the config through to supervise() from Celery’s execute_workload task execution.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File	Description
providers/celery/src/airflow/providers/celery/executors/celery_executor_utils.py	Wires the new config into Celery worker task execution by passing subprocess_logs_to_stdout to supervise().
airflow-core/src/airflow/executors/local_executor.py	Wires the new config into LocalExecutor task execution by passing subprocess_logs_to_stdout to supervise().
airflow-core/src/airflow/config_templates/config.yml	Documents the new [logging] task_log_to_stdout option in the generated config schema.

[Copilot]

This new wiring is not covered by a regression/unit test. Please add a test that sets [logging] task_log_to_stdout to True/False (e.g., via conf_vars) and asserts _execute_work()/supervise() is called with subprocess_logs_to_stdout matching the config value, so future refactors don’t silently break stdout log forwarding.

[Copilot]

Please add/extend a unit test for execute_workload that toggles [logging] task_log_to_stdout and asserts supervise() receives subprocess_logs_to_stdout accordingly. Without this, the new config-to-supervisor plumbing is easy to regress (especially since supervise is mocked in existing tests).

[Copilot]

This introduces a new user-facing config option; please add a Towncrier newsfragment in airflow-core/newsfragments/ describing [logging] task_log_to_stdout so it appears in the release notes (the PR template indicates this is expected for user-facing changes).