Skip to content

Refactor shared cache and debug deadline exceeded#38612

Draft
shunping wants to merge 8 commits into
apache:masterfrom
shunping:refactor-shared-cache
Draft

Refactor shared cache and debug deadline exceeded#38612
shunping wants to merge 8 commits into
apache:masterfrom
shunping:refactor-shared-cache

Conversation

@shunping
Copy link
Copy Markdown
Collaborator

No description provided.

shunping added 8 commits May 22, 2026 19:24
When a SubprocessServer fails to start (e.g., due to a process exit or
startup error), the server process could leak if standard purging
is blocked by other active owners sharing the cached subprocess.

To fix this:
- Implement `_SharedCache.force_remove()` to immediately remove a key
  from the cache and run its destructor regardless of active owners.
- Add `SubprocessServer.stop_force()` which calls `force_remove()` to
  completely terminate the server's process.
- Call `stop_force()` in the `except` block of `SubprocessServer.start()`
This ensures we can download pre-built wheels for environment staging
rather than relying on tarball building, which is sometimes slow.
@codecov
Copy link
Copy Markdown

codecov Bot commented May 23, 2026

Codecov Report

❌ Patch coverage is 17.30769% with 43 lines in your changes missing coverage. Please review.
✅ Project coverage is 58.06%. Comparing base (0c9b272) to head (8af7b3c).
⚠️ Report is 17 commits behind head on master.

Files with missing lines Patch % Lines
...o/pkg/beam/runners/prism/internal/worker/worker.go 15.78% 16 Missing ⚠️
sdks/go/pkg/beam/runners/prism/internal/execute.go 0.00% 9 Missing ⚠️
sdks/go/pkg/beam/core/runtime/harness/datamgr.go 0.00% 6 Missing ⚠️
sdks/go/pkg/beam/core/runtime/harness/statemgr.go 0.00% 6 Missing ⚠️
...go/pkg/beam/runners/prism/internal/environments.go 25.00% 2 Missing and 1 partial ⚠️
sdks/go/pkg/beam/runners/prism/internal/stage.go 57.14% 2 Missing and 1 partial ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##             master   #38612      +/-   ##
============================================
- Coverage     58.07%   58.06%   -0.01%     
  Complexity    12941    12941              
============================================
  Files          2511     2511              
  Lines        262223   262236      +13     
  Branches      10652    10651       -1     
============================================
- Hits         152282   152276       -6     
- Misses       104243   104257      +14     
- Partials       5698     5703       +5     
Flag Coverage Δ
go 28.72% <17.30%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant