Skip to content

[GSoC-273] Feat: Integrate TestPubsubContext to prevent Pub/Sub resource leaks and expand stale cleaner scope - #39826

Open
HansMarcus01 wants to merge 4 commits into
apache:masterfrom
HansMarcus01:feat-extended-cleanup-manager
Open

[GSoC-273] Feat: Integrate TestPubsubContext to prevent Pub/Sub resource leaks and expand stale cleaner scope#39826
HansMarcus01 wants to merge 4 commits into
apache:masterfrom
HansMarcus01:feat-extended-cleanup-manager

Conversation

@HansMarcus01

Copy link
Copy Markdown
Contributor

Overview

This pull request improves the lifecycle management of Pub/Sub resources in our integration tests to actively reduce GCP consumption. It replaces manual teardown methods with the robust TestPubsubContext manager and expands the automated garbage collector's scope to catch previously untracked test prefixes.

Key Changes

  • Integration Test Refactoring (pubsub_integration_test.py, streaming_wordcount_it_test.py, streaming_wordcount_debugging_it_test.py):
    • Replaced manual and error-prone teardown logic (test_utils.cleanup_subscriptions and test_utils.cleanup_topics) with the TestPubsubContext monitor.
    • Explicitly registered temporary test topics and subscriptions (including ordering topics) to ensure guaranteed, graceful teardown on test completion.
  • Stale Resource Cleaner Expansion (.test-infra/tools/stale_cleaner.py):
    • Appended missing test prefixes to the cleanup tracking list (e.g., wc_topic_*, psit_*, leader_board_*).
    • Ensures any orphaned resources that slip through the cracks during local or CI test failures are properly detected and deleted by the automated stale cleaner.

…yond just creating Pub/Sub resources, and preventing resource leaks in the GCP environment.

- Deletion redundancy was increased by adding subscription prefixes to `stale_clener.py` that might have remained active had `TestPubSubManager` failed.
@github-actions

Copy link
Copy Markdown
Contributor

Assigning reviewers:

R: @claudevdm for label python.

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

  • stop reviewer notifications - opt out of the automated review tooling
  • remind me after tests pass - tag the comment author after tests pass
  • waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

-Add the Apache Beam project licenses.
- applying correct indentation using yapf formatting to the pubsub_integration/py and streaming_wordcount_it_test files
Updated the testing README to include the GCP resource leak audit log, justifying the implementation of the cleanup handler. It also documents the specific integration tests (PubSub and Streaming Wordcount) currently managed by the context.
@codecov

codecov Bot commented Aug 21, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 57.19%. Comparing base (9451007) to head (98a4019).
⚠️ Report is 4 commits behind head on master.

Additional details and impacted files
@@             Coverage Diff              @@
##             master   #39826      +/-   ##
============================================
+ Coverage     57.10%   57.19%   +0.08%     
  Complexity     3643     3643              
============================================
  Files          1191     1193       +2     
  Lines        192542   192979     +437     
  Branches       3816     3816              
============================================
+ Hits         109957   110378     +421     
- Misses        79027    79043      +16     
  Partials       3558     3558              
Flag Coverage Δ
python 79.93% <ø> (+0.08%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant