Skip to content

feat: Add [logging] task_log_to_stdout config to forward task logs to worker stdout #64481

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
mtraynham wants to merge 2 commits into
base: main
Choose a base branch
from
+12 −0
Draft

feat: Add [logging] task_log_to_stdout config to forward task logs to worker stdout #64481

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
4cea1b7
Add flag for logging tasks to stdout
mtraynham Mar 30, 2026
5967acd
Merge branch 'main' into stdout_logging
mtraynham Mar 31, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions airflow-core/src/airflow/config_templates/config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1076,6 +1076,16 @@ logging:
type: boolean
example: ~
default: "False"
task_log_to_stdout:
description: |
When True, task log messages are duplicated to the worker process's stdout
in addition to the task log file (and any configured remote log handler).
This is useful for container-based deployments where a log shipper (e.g.
CloudWatch agent, Fluentd, Fluent Bit) captures stdout.
version_added: 3.2.1
type: boolean
example: "True"
default: "False"
Comment on lines +1079 to +1088
Copy link

Copilot AI Apr 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This introduces a new user-facing config option; please add a Towncrier newsfragment in airflow-core/newsfragments/ describing [logging] task_log_to_stdout so it appears in the release notes (the PR template indicates this is expected for user-facing changes).

Copilot uses AI. Check for mistakes.
color_log_error_keywords:
description: |
A comma separated list of keywords related to errors whose presence should display the line in red
Expand Down
1 change: 1 addition & 0 deletions airflow-core/src/airflow/executors/local_executor.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -149,6 +149,7 @@ def _execute_work(log: Logger, workload: workloads.ExecuteTask, team_conf) -> No
token=workload.token,
server=team_conf.get("core", "execution_api_server_url", fallback=default_execution_api_server),
log_path=workload.log_path,
subprocess_logs_to_stdout=team_conf.getboolean("logging", "task_log_to_stdout", fallback=False),
)
Comment on lines 149 to 153
Copy link

Copilot AI Apr 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This new wiring is not covered by a regression/unit test. Please add a test that sets [logging] task_log_to_stdout to True/False (e.g., via conf_vars) and asserts _execute_work()/supervise() is called with subprocess_logs_to_stdout matching the config value, so future refactors don’t silently break stdout log forwarding.

Copilot uses AI. Check for mistakes.


Expand Down
1 change: 1 addition & 0 deletions providers/celery/src/airflow/providers/celery/executors/celery_executor_utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -223,6 +223,7 @@ def execute_workload(input: str) -> None:
token=workload.token,
server=conf.get("core", "execution_api_server_url", fallback=default_execution_api_server),
log_path=workload.log_path,
subprocess_logs_to_stdout=conf.getboolean("logging", "task_log_to_stdout", fallback=False),
)
Comment on lines 223 to 227
Copy link

Copilot AI Apr 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add/extend a unit test for execute_workload that toggles [logging] task_log_to_stdout and asserts supervise() receives subprocess_logs_to_stdout accordingly. Without this, the new config-to-supervisor plumbing is easy to regress (especially since supervise is mocked in existing tests).

Copilot uses AI. Check for mistakes.
elif isinstance(workload, workloads.ExecuteCallback):
success, error_msg = execute_callback_workload(workload.callback, log)
Expand Down
Loading