ci: drop unused Chrome/Microsoft apt sources before apt-get update

[wen-coding]

Summary

• GitHub ubuntu-* runners ship Google Chrome (and Microsoft) apt sources. apt-get update then fetches dl.google.com even when the job only installs Ubuntu packages (build-essential, RocksDB deps).
• That repo periodically returns a Packages.gz whose hash does not match the signed Release file, failing Cross-Architecture Build and RocksDB Tests (e.g. attempts on (Autobahn) Prove commit-committee identity on giga handshake and learn live dial addresses (CON-358) #4108).
• Drop those unused sources before apt-get update. We do not install or need Chrome.

Test plan

• Confirm Cross-Architecture Build Test → Linux AMD64 Install system dependencies no longer hits dl.google.com/linux/chrome-stable.
• Confirm sei-db → RocksDB Tests Install RocksDB dependencies succeeds without hash-sum mismatch.
• Confirm both jobs still install build-essential (and RocksDB deps) from Ubuntu.

Made with Cursor

[github-actions]

The latest Buf updates on your PR. Results from workflow Buf / buf (pull_request).

Build	Format	Lint	Breaking	Updated (UTC)
✅ passed	✅ passed	✅ passed	✅ passed	Sep 9, 2026, 7:06 PM

[seidroid]

Straightforward CI flake mitigation: a guarded helper drops unused Chrome/Microsoft apt sources before every apt-get update in the repo's workflows. Correct and consistent with the existing .github/scripts/ci-*.sh conventions; one non-blocking robustness suggestion about the hardcoded filename list.

Findings: 0 blocking | 1 non-blocking | 1 posted inline

Blockers

• None at the file/PR level.

Non-blocking

• None at the file/PR level.
• 1 suggestion(s)/nit(s) flagged inline on specific lines.

[seidroid]

[suggestion] Matching exact filenames makes this fix silently expire: if a future runner image renames the file (e.g. google-chrome-stable.list, or a new microsoft-*.sources variant), every entry misses, the loop no-ops, and the hash-sum-mismatch flake returns with no signal that the mitigation stopped applying. A glob over the same directory would be durable against that layout change while still being narrowly scoped:

for src in /etc/apt/sources.list.d/*google*.list /etc/apt/sources.list.d/*google*.sources \
           /etc/apt/sources.list.d/*microsoft*.list /etc/apt/sources.list.d/*microsoft*.sources; do
  [ -e "$src" ] || continue
  echo "Removing $src"
  sudo rm -f "$src"
done

(Non-blocking — the current list matches today's ubuntu-latest/ubuntu-24.04-arm images, so the fix works as-is.)

[wen-coding]

@seidroid re-review please

[seidroid]

Claude finished @wen-coding's task in 1m 26s —— View job

---

Re-review: APPROVED

The follow-up commit (dd72530) adopts the glob suggestion from the prior round, and it's implemented correctly. The mitigation no longer expires silently if the runner image renames google-chrome.list / microsoft-prod.list.

• Fetch PR diff and new commit
• Verify the glob follow-up addresses the prior inline suggestion
• Check remaining apt-get update call sites in the repo
• Check .github/scripts/ conventions and shell correctness
• Post verdict

Findings: 0 blocking | 1 non-blocking

What changed since the last review

.github/scripts/ci-drop-third-party-apt-sources.sh swapped the hardcoded sources=(...) array + [ -e "$src" ] || continue for four globs plus shopt -s nullglob. That's the right shape:

• nullglob is what makes the guard work here — without it an unmatched pattern expands to the literal string and rm -f would silently "remove" a path that never existed (noisy log, no failure). With it, the loop body simply never runs. This correctly replaces the [ -e ] test the previous version needed, so there's no redundant guard left behind.
• Both .list and deb822 .sources variants are covered, which is the layout change most likely to land next in the runner images.
• The header comment was updated to match the new behavior rather than left describing the old list.

Coverage check

I grepped every apt-get update in .github/:

Call site	Guarded
.github/workflows/cross-arch-build.yml:35 (linux-amd64, ubuntu-latest)	✅
.github/workflows/cross-arch-build.yml:75 (linux-arm64, ubuntu-24.04-arm)	✅
.github/workflows/rocksdb-unit_tests.yml:30 (ubuntu-latest)	✅

.github/workflows/integration-test.yml:189,346 run sudo apt-get install -y jq with no preceding apt-get update, so they never fetch dl.google.com's Packages.gz and are not exposed to this flake. No gap.

I also confirmed no step in either changed workflow installs anything from a Google or Microsoft repo after the script runs (build-essential and the RocksDB deps are all Ubuntu main/universe), so widening from google-chrome to *google* — which would also catch e.g. a google-cloud-sdk.list — costs these jobs nothing. Invocation style (run: bash .github/scripts/ci-*.sh) matches the four existing ci-*.sh scripts, and the file landed with mode 100755.

Non-blocking

[nit] The header's "can never fail the job" claim is now slightly stronger than the code, vs. its sibling script. nullglob guards absence, but sudo rm -f itself is unguarded under set -euo pipefail — if sudo were unavailable or the removal were refused, this mitigation script would become a new source of job failure. ci-free-disk.sh makes exactly the same promise in its header and backs it with sudo rm -rf "$dir" || true. Matching that would make the two scripts read the same way:

  sudo rm -f "$src" || true

Low practical risk on ubuntu-* runners (passwordless sudo, root-owned /etc/apt/sources.list.d), so this is purely consistency with the guard convention the neighbouring script already sets. Fix this →

Verification note

I could not execute bash -n or shellcheck against the script in this environment (command approval denied), so the shell correctness above is from reading all 20 lines rather than from a run. The repo has no shellcheck gate in CI, so the real signal remains the test plan in the PR body — worth confirming the two jobs' install steps no longer reference dl.google.com before merging.

--- • Branch: wen/ci-drop-flaky-apt-sources

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 59.97%. Comparing base (a665891) to head (02d5dea).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4114      +/-   ##
==========================================
- Coverage   60.94%   59.97%   -0.98%     
==========================================
  Files        2192     2087     -105     
  Lines      192187   179513   -12674     
==========================================
- Hits       117131   107654    -9477     
+ Misses      63684    61576    -2108     
+ Partials    11372    10283    -1089     

Flag	Coverage Δ
sei-db	69.80% <ø> (?)
sei-db-state-db	?

Flags with carried forward coverage won't be shown. Click here to find out more.
see 115 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[wen-coding]

@seidroid re-review please

[seidroid]

Claude finished @wen-coding's task in 1m 16s —— View job

---

Re-review: APPROVED

02d5dea adopts the one non-blocking nit from the previous round. Both prior suggestions (glob instead of pinned filenames, || true on the removal) are now in, and the script matches ci-free-disk.sh's guard convention exactly. Nothing left outstanding from me.

• Fetch PR diff and the new commit (02d5dea)
• Verify the follow-up addresses the prior non-blocking nit
• Compare against .github/scripts/ci-free-disk.sh convention
• Re-check apt-get call sites and Google/Microsoft package usage
• Post verdict

Findings: 0 blocking | 0 non-blocking

What changed since the last review

One line, .github/scripts/ci-drop-third-party-apt-sources.sh:19:

-  sudo rm -f "$src"
+  sudo rm -f "$src" || true

That closes the gap I flagged: the header promises "every removal is guarded so a change to the runner image layout can never fail the job," but nullglob only guarded absence — a removal that was refused (no passwordless sudo, immutable mount) would still have tripped set -e and turned a flake mitigation into a new failure mode. With || true the header claim is now literally true for both paths.

It also lands on the same shape as its sibling, .github/scripts/ci-free-disk.sh:22 (sudo rm -rf "$dir" || true), which is the point — two scripts making the same "can never fail the job" promise in their headers now back it the same way, so the next person adding a ci-*.sh cleanup script has one pattern to copy rather than two.

Final state of the script

All 20 lines read cleanly:

• set -euo pipefail + shopt -s nullglob — unmatched patterns drop out of the loop rather than expanding to literal paths, so no [ -e ] test is needed and none is left behind.
• Globs cover *google*/*microsoft* × .list/.sources, so both the current layout and a deb822 migration are handled.
• The removal is guarded, and the guard lives in the single script every workflow step calls — one choke point, not a convention repeated at three call sites.

Coverage re-check

Every apt-get update in .github/ is preceded by the script:

Call site	Guarded
.github/workflows/cross-arch-build.yml:35 (linux-amd64, ubuntu-latest)	✅
.github/workflows/cross-arch-build.yml:75 (linux-arm64, ubuntu-24.04-arm)	✅
.github/workflows/rocksdb-unit_tests.yml:30 (ubuntu-latest)	✅

.github/workflows/integration-test.yml:189,346 run sudo apt-get install -y jq with no preceding apt-get update, so they never fetch the third-party Packages.gz and aren't exposed. No gap.

I also re-grepped .github/ for anything sourced from those repos — azure-cli, powershell/pwsh, dotnet, gcloud, google-cloud — and the only hit is ci-free-disk.sh deleting /usr/share/dotnet. Nothing in CI installs from a Google or Microsoft apt repo, so the widened globs cost these jobs nothing. File mode is 100755 in the index, matching the other executable ci-*.sh scripts.

Minor observation (not a finding)

echo "Removing $src" prints before the rm, so with || true a refused removal now logs "Removing …" and moves on silently. The failure it hides is benign — the source stays, and you're back to today's behavior — and ci-free-disk.sh:21-22 has exactly the same ordering, so I'd leave it rather than diverge from the sibling for a log-wording improvement.

Verification note

bash -n and shellcheck were again denied in this environment, so shell correctness above comes from reading all 20 lines, not from executing them. The repo has no shellcheck gate in CI either, so the real signal is still the PR's test plan — worth confirming on the next run that the three install steps no longer reference dl.google.com and that build-essential and the RocksDB deps still resolve.
• Branch: wen/ci-drop-flaky-apt-sources