ci: Remove benchmark workflow that comments on PRs - #5972
Merged
Conversation
The `app-metrics` job in integration-tests-benchmarks.yml ran the plain and Sentry test apps through SauceLabs on every PR and posted a "Performance metrics" comment with startup-time and size diffs. The numbers were not actionable: recent runs reported 0 B for both app sizes, and the startup diff swung between 22 ms and 143 ms across unrelated changes. Removing the whole workflow also drops its SauceLabs `test` job, which leaves several things with no remaining caller. Removed here as well: - `test-app-plain` and `test-app-sentry`, built only by `app-metrics` - `metrics-test.yml`, the app-metrics test specification - `.sauce/sentry-uitest-android-benchmark.yml` and its `-lite` variant, passed to saucectl only by the `test` job - the `make assembleBenchmarks` alias, called only by the `test` job The `sentry-uitest-android-benchmark` module itself is kept. It no longer runs in CI, but it still builds and can be run locally.
📲 Install BuildsAndroid
|
romtsn
approved these changes
Aug 17, 2026
runningcode
marked this pull request as ready for review
August 25, 2026 09:47
runningcode
requested review from
0xadam-brown,
adinauer and
markushi
as code owners
August 25, 2026 09:47
The Espresso benchmark module was only exercised by the SauceLabs `test` job in the workflow removed in the previous commit. Nothing runs it any more, and the numbers it produced were the same noisy startup-time and app-size measurements that made the PR comment unactionable, so keeping it around only costs maintenance. Also drops its `settings.gradle.kts` / `build.gradle.kts` entries, the `make assembleBenchmarkTestRelease` target, and the README references.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
📜 Description
Deletes
.github/workflows/integration-tests-benchmarks.yml.Its
app-metricsjob was the one posting the "Performance metrics 🚀" comment on every PR, viagetsentry/action-app-sdk-overhead-metrics. Removing the whole file also drops the workflow's other job, the SauceLabstestjob (nightly full benchmark + per-PR lite benchmark).That leaves a handful of things with no remaining caller, so they go too:
sentry-android-integration-tests/test-app-plainapp-metricssentry-android-integration-tests/test-app-sentryapp-metricssentry-android-integration-tests/metrics-test.ymlapp-metrics.sauce/sentry-uitest-android-benchmark.ymltestjob.sauce/sentry-uitest-android-benchmark-lite.ymltestjobsentry-android-integration-tests/sentry-uitest-android-benchmarktestjobmake assembleBenchmarkstestjobmake assembleBenchmarkTestReleasetestjobPlus the corresponding
settings.gradle.kts/build.gradle.ktsmodule entries and the README links insentry-android-integration-tests/andsentry-uitest-android/.The Espresso benchmark module (
sentry-uitest-android-benchmark) goes in the second commit. It produced the same noisy startup-time and app-size numbers that made the PR comment unactionable, and with thetestjob gone nothing invokes it — keeping it would only be maintenance cost.Left alone:
sentry-uitest-android(SauceLabs UI tests and the AGP compatibility matrix),sentry-uitest-android-critical(Maestro), andsentry-uitest-android-macrobenchmark.💡 Motivation and Context
The PR comment wasn't actionable. Recent runs reported 0 B for both app sizes, and the startup-time diff swung between 22 ms and 143 ms across unrelated changes — wide enough that a real regression would be invisible in it.
💚 How did you test it?
./gradlew projectsand./gradlew spotlessApply apiDumpboth succeed; the removed modules are gone from the project list and no.apifiles changed. Grepped the tree for references to every deleted path — none remain.📝 Checklist
sendDefaultPIIis enabled.#skip-changelog