probe round 2: is the 12 KB/s budget per-connection or per-host? #2

Workflow file for this run

.github/workflows/probe-gitcode-upload.yml at 5420962

	name: probe-gitcode-upload

	# TEMPORARY diagnostic workflow — delete with this branch.
	#
	# ROUND 1 (run 30374882109) already settled the "why":
	#
	# US-East runner -> file.gitcode.com upload is a FLAT ~0.012 MB/s (12 KB/s),
	# independent of file size AND of transport:
	# urllib 1MB 88.8s \| 4MB failed \| 16MB 1529.9s (0.010-0.011 MB/s)
	# curl 1MB 83.1s \| 4MB 330.7s \| 16MB 1218.4s (0.012-0.013 MB/s)
	# 4x8MB serial: 672/650/675/599s, wall 2618s
	# Controls from the SAME runner:
	# download from gitcode.com 3.87 MB/s (~320x the upload rate)
	# upload to github.com 16 MB/s
	# Control from a mainland-CN host: 1.84 MB/s (~150x the runner's rate).
	#
	# So it is neither the Python client (curl matches it) nor cross-border
	# bandwidth in general (the same runner downloads from the same host at
	# 3.87 MB/s). It is inbound-upload rate limiting on file.gitcode.com for
	# this egress. At 12 KB/s the 34.8MB asset needs ~48 MINUTES; the 180s cap
	# never had a chance, which is why "raise/loosen the cap" failed four times.
	#
	# ROUND 2 (this file) answers the one question that decides the FIX, which
	# round 1 ran out of wall-clock before reaching: is the 12 KB/s budget
	# PER-CONNECTION or PER-HOST?
	# per-connection -> N parallel uploads give ~N x aggregate, and
	# parallelising mirror_res.sh's serial per-asset loop is
	# a real (if partial) fix.
	# per-host -> parallelism buys nothing and the mirror must move off
	# GitHub-hosted runners entirely.
	# `scaling` measures exactly that. `variance` bounds how much the rate moves
	# over time, which decides whether ANY fixed timeout can be safe.

	on:
	push:
	branches: [ 'probe/**' ]
	workflow_dispatch:

	permissions:
	contents: read

	concurrency:
	group: probe-gitcode-${{ github.ref }}
	cancel-in-progress: true

	env:
	GTC_REPO: xlings-res/mcpp

	jobs:
	# ── THE decisive test: does concurrency multiply the budget? ─────────────
	scaling:
	name: scaling — 1 vs 4 vs 8 concurrent 1MB uploads
	runs-on: ubuntu-latest
	timeout-minutes: 50
	env:
	GITCODE_TOKEN: ${{ secrets.GITCODE_TOKEN }}
	TAG: probe-${{ github.run_id }}-scale
	steps:
	- uses: actions/checkout@v4
	- name: Create probe release
	run: python3 .github/tools/gtc release create "$GTC_REPO" --tag "$TAG" --name "$TAG"

	- name: Inspect the upload_url contract (multipart/resumable available?)
	run: \|
	# If the response carries multipart/part-size fields, a resumable
	# chunked upload is possible and a stalled transfer could be resumed
	# instead of restarted from byte zero.
	python3 - <<'PY'
	import json, os, urllib.request, urllib.parse
	repo, tag = os.environ["GTC_REPO"], os.environ["TAG"]
	url = (f"https://api.gitcode.com/api/v5/repos/{repo}/releases/{tag}"
	f"/upload_url?file_name=probe-contract.bin")
	req = urllib.request.Request(url, headers={
	"PRIVATE-TOKEN": os.environ["GITCODE_TOKEN"], "Accept": "application/json"})
	info = json.loads(urllib.request.urlopen(req, timeout=60).read())
	# Redact the signature so the log stays shareable.
	u = urllib.parse.urlparse(info["url"])
	print("keys: ", sorted(info.keys()))
	print("obs host: ", u.netloc)
	print("obs path: ", u.path)
	print("query params:", sorted(urllib.parse.parse_qs(u.query).keys()))
	print("headers: ", {k: ("<redacted>" if "auth" in k.lower() else v)
	for k, v in (info.get("headers") or {}).items()})
	PY

	- name: N=1 (baseline for this run)
	run: \|
	head -c 1048576 /dev/urandom > s1-1.bin
	start=$SECONDS
	python3 .github/tools/probe_gtc_upload.py --repo "$GTC_REPO" --tag "$TAG" \
	--file s1-1.bin --method curl --label "n1" >> results.ndjson
	echo "N1_WALL=$((SECONDS - start))" \| tee -a wall.txt

	- name: N=4 concurrent
	run: \|
	for i in 1 2 3 4; do head -c 1048576 /dev/urandom > "s4-$i.bin"; done
	start=$SECONDS
	for i in 1 2 3 4; do
	python3 .github/tools/probe_gtc_upload.py --repo "$GTC_REPO" --tag "$TAG" \
	--file "s4-$i.bin" --method curl --label "n4-$i" > "s4-$i.json" &
	done
	wait
	echo "N4_WALL=$((SECONDS - start))" \| tee -a wall.txt
	cat s4-*.json >> results.ndjson

	- name: N=8 concurrent
	run: \|
	for i in 1 2 3 4 5 6 7 8; do head -c 1048576 /dev/urandom > "s8-$i.bin"; done
	start=$SECONDS
	for i in 1 2 3 4 5 6 7 8; do
	python3 .github/tools/probe_gtc_upload.py --repo "$GTC_REPO" --tag "$TAG" \
	--file "s8-$i.bin" --method curl --label "n8-$i" > "s8-$i.json" &
	done
	wait
	echo "N8_WALL=$((SECONDS - start))" \| tee -a wall.txt
	cat s8-*.json >> results.ndjson

	- name: Verdict
	if: always()
	run: \|
	# N4_WALL ~= N1_WALL -> per-connection budget: parallelism is the fix.
	# N4_WALL ~= 4*N1_WALL -> per-host budget: parallelism buys nothing.
	{ echo '### scaling'; echo '```'; cat wall.txt; echo '```';
	echo '```json'; cat results.ndjson; echo '```'; } >> "$GITHUB_STEP_SUMMARY"
	cat wall.txt

	# ── Can ANY fixed timeout be safe? Depends on the spread, not the mean. ──
	variance:
	name: variance — 8 sequential 1MB uploads
	runs-on: ubuntu-latest
	timeout-minutes: 50
	env:
	GITCODE_TOKEN: ${{ secrets.GITCODE_TOKEN }}
	TAG: probe-${{ github.run_id }}-var
	steps:
	- uses: actions/checkout@v4
	- name: Create probe release
	run: python3 .github/tools/gtc release create "$GTC_REPO" --tag "$TAG" --name "$TAG"
	- name: 8 sequential 1MB uploads
	run: \|
	for i in 1 2 3 4 5 6 7 8; do
	head -c 1048576 /dev/urandom > "v-$i.bin"
	python3 .github/tools/probe_gtc_upload.py --repo "$GTC_REPO" --tag "$TAG" \
	--file "v-$i.bin" --method curl --label "v$i" >> results.ndjson
	rm -f "v-$i.bin"
	done
	- name: Summary
	if: always()
	run: \|
	{ echo '### variance'; echo '```json'; cat results.ndjson; echo '```'; } \
	>> "$GITHUB_STEP_SUMMARY"

	cleanup:
	name: cleanup probe tags
	needs: [scaling, variance]
	if: always()
	runs-on: ubuntu-latest
	timeout-minutes: 10
	env:
	GITCODE_TOKEN: ${{ secrets.GITCODE_TOKEN }}
	steps:
	- name: Delete GitCode probe tags (deleting the tag deletes the release)
	run: \|
	for t in "probe-${{ github.run_id }}-scale" "probe-${{ github.run_id }}-var"; do
	code=$(curl -sS -o /dev/null -w '%{http_code}' -X DELETE \
	-H "PRIVATE-TOKEN: $GITCODE_TOKEN" \
	"https://api.gitcode.com/api/v5/repos/${GTC_REPO}/tags/${t}" \|\| echo ERR)
	echo "delete $t -> $code"
	done

Navigation Menu

Uh oh!

probe round 2: is the 12 KB/s budget per-connection or per-host? #2

Workflow file

probe round 2: is the 12 KB/s budget per-connection or per-host? #2

Uh oh!

Workflow file for this run