Skip to content

Commit d1a9f34

Browse files
committed
ci: macOS 全量也装不下一片 —— 扇出改为逐平台按实测总量定
上一轮 linux 三片全过(1h17/1h09/44m),macOS 单片在 1h30m20s 撞顶。 原注释只从墙钟论证 macOS 不该分片(并发为 1,分了也串行),漏了另 一半:**每个分片是独立 job、各有各的 timeout-minutes**,所以分片同时 也是「让活装得下」的手段。串行不要紧,跑不完才要紧。 macOS 全量实测 6922s = 115 分钟,本来就超 90 上限;它此前能过,只是 因为总量刚好卡在线下,而 grpc-codegen(macOS 1715s)把它推了过去。 改为三平台同一条规则:从 tests/member-timings.tsv 求本次成员在该平台 的实测总和,每 ~70 分钟一片(在 90 上限下留 ~20 分钟冷缓存余量), 上限按各自 runner 并发定(linux 3 / macOS 2 / windows 2)。 本机核验: 全量 linux 3 片 ~75min · macOS 2 片 ~57min · windows 2 片 ~67min 本 PR 四成员 linux 2 片 ~44min · macOS 1 片 · windows 1 片 linux 全量是最紧的一档(实测最慢片 77/90),注释已写明后续两个杠杆。
1 parent d46afa7 commit d1a9f34

1 file changed

Lines changed: 52 additions & 27 deletions

File tree

.github/workflows/validate.yml

Lines changed: 52 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -404,40 +404,65 @@ jobs:
404404
"$1" "$2" "$3" "$4" "$5" "$6" "$i" "$7"
405405
done
406406
}
407-
# Shard count follows the WORK, not the full/partial flag.
407+
# Shard count follows the WORK, per platform, measured.
408408
#
409-
# It used to be binary — full run: linux 3, anything else: linux 1 —
410-
# which reads "a partial run is small". It is not: touching a widely
411-
# consumed descriptor selects every member that consumes it. This
412-
# PR's `pkgs/c/compat.openssl.lua` selected four (grpc-codegen,
413-
# grpc-module, asio-ssl, openssl), one shard got all of them, and
414-
# linux was cancelled at exactly 1h30m — the job cap — with
415-
# grpc-codegen (3563s) and grpc-module (1701s) back to back. Nothing
416-
# was broken; the plan just could not fit.
409+
# It used to be binary — full run: linux 3 / macos 1 / windows 2,
410+
# anything else: 1 each — which asserts two things that are not
411+
# true. "A partial run is small": touching a widely consumed
412+
# descriptor selects every member that consumes it, and this PR's
413+
# `pkgs/c/compat.openssl.lua` selected four; one linux shard took
414+
# grpc-codegen (3563s) and grpc-module (1701s) back to back and was
415+
# cancelled at exactly 1h30m, the job cap. "A full macOS run fits in
416+
# one shard": at 6922s measured it does not — it fit only while the
417+
# total sat just under the cap, and the run that added grpc-codegen
418+
# (1715s on macOS) pushed it to 1h30m20s.
417419
#
418-
# tests/member-timings.tsv already holds the measured cost of every
419-
# member, and plan_shards.lua already packs by it (LPT). So size the
420-
# fan-out from the same table: sum this run's members and give linux
421-
# a shard per ~45 minutes of work, capped at 3 (above that, runner
422-
# concurrency and the per-shard setup eat the gain — see the
423-
# measurement in the comment above). macOS stays at 1: its
424-
# concurrency is 1, so extra shards there run back to back.
425-
if [ "$full" = 1 ]; then
426-
ln=3; mn=1; wn=2
427-
else
428-
secs=$(lua5.4 -e '
429-
local want = {}
430-
for m in (os.getenv("MEMBERS") or ""):gmatch("%S+") do want[m] = true end
420+
# Sharding is not only a wall-clock lever. Each shard is its own job
421+
# with its own `timeout-minutes`, so it is also how the work is made
422+
# to FIT. That is the half the "macOS concurrency is 1, so extra
423+
# shards run back to back" note above left out: back to back is fine
424+
# when the alternative is not finishing.
425+
#
426+
# tests/member-timings.tsv already holds every member's measured
427+
# cost and plan_shards.lua already packs by it (LPT), so the fan-out
428+
# comes from the same table: sum this run's members for that
429+
# platform and take one shard per ~70 minutes, which leaves ~20
430+
# minutes of headroom under the 90-minute cap for a cold cache.
431+
#
432+
# The caps are where runner concurrency comes back in: linux 3 (the
433+
# measured concurrency — a 4th shard would queue), macOS 2, windows
434+
# 2. Past those, more shards buy fit that is already there and pay
435+
# another checkout + mcpp download + cache restore.
436+
#
437+
# Full run, from the table: linux 13570s over 3 -> ~75min/shard
438+
# (observed slowest 77), macOS 6922s over 2 -> ~57, windows 8043s
439+
# over 2 -> ~67. linux is the tight one — its cap binds before the
440+
# ~70-minute target does, so it is the first place to look if a
441+
# cold full run starts brushing 90 again. The levers, in order:
442+
# raise the linux cap to 4 (costs a queued runner), then the job
443+
# timeout.
444+
shards_for() { # platform cap -> shard count
445+
local secs
446+
secs=$(XPLAT="$1" lua5.4 -e '
447+
local plat = os.getenv("XPLAT")
448+
local sel, all = {}, (os.getenv("MEMBERS") == "__ALL__")
449+
if not all then
450+
for m in (os.getenv("MEMBERS") or ""):gmatch("%S+") do sel[m] = true end
451+
end
431452
local total = 0
432453
for line in io.lines("tests/member-timings.tsv") do
433454
local p, m, s = line:match("^(%S+)\t(%S+)\t(%d+)$")
434-
if p == "linux" and m and want[m] then total = total + tonumber(s) end
455+
if p == plat and m and (all or sel[m]) then total = total + tonumber(s) end
435456
end
436457
print(total)')
437-
ln=$(( secs / 2700 + 1 )); [ "$ln" -gt 3 ] && ln=3
438-
mn=1; wn=1
439-
echo "linux work: ${secs}s (measured) -> $ln shard(s)"
440-
fi
458+
local n=$(( secs / 4200 + 1 ))
459+
[ "$n" -gt "$2" ] && n="$2"
460+
echo "$1 work: ${secs}s (measured) -> $n shard(s)" >&2
461+
echo "$n"
462+
}
463+
ln=$(shards_for linux 3)
464+
mn=$(shards_for macos 2)
465+
wn=$(shards_for windows 2)
441466
{
442467
printf '{"include":['
443468
emit linux ubuntu-latest linux-x86_64 tar.gz bin/mcpp registry/bin/xlings "$ln"

0 commit comments

Comments
 (0)