Export DOTNET_GCHeapHardLimit from MEMORY_LIMIT to fix cgroup v2 memory handling - #1317
Export DOTNET_GCHeapHardLimit from MEMORY_LIMIT to fix cgroup v2 memory handling#1317rkoster wants to merge 4 commits into
Conversation
Fixes cloudfoundry#1100. Computes the .NET GC's heap hard limit from CF's MEMORY_LIMIT env var (always authoritative, cgroup-version-agnostic) at container startup, rather than relying on the .NET runtime's own cgroup v1/v2/hybrid detection, which is documented (dotnet/runtime#34334) to sometimes pick the wrong cgroup hierarchy on hybrid hosts and treat the container as unlimited. Forces base-10 interpretation (10#...) of MEMORY_LIMIT and DOTNET_GCHeapHardLimitPercent digit strings so that leading-zero values (e.g. 08, 09, 050) are never misinterpreted as octal or rejected by bash's arithmetic context, bounds the percent string to 1-3 digits to reject overflow-prone inputs before they reach arithmetic, and adds a floor so MEMORY_LIMIT=0m is a no-op instead of exporting a zero-byte heap limit.
Adds a /env/dotnet-gc-heap-hard-limit debug endpoint to the shared 'simple' fixture app and a new MemoryLimit integration suite that deploys it and asserts the exported DOTNET_GCHeapHardLimit value, both with the default 75% and with a caller-provided DOTNET_GCHeapHardLimitPercent override.
Clarifies (per final code review) that a plain decimal-looking DOTNET_GCHeapHardLimit override, e.g. "268435456", is parsed as hex rather than decimal - matching .NET's own native parsing convention for this env var.
Replaces reading back the raw DOTNET_GCHeapHardLimit environment variable with querying GC.GetConfigurationVariables()["GCHeapHardLimit"] (available since .NET 7). This proves the CLR itself parsed and applied the value into its live GC configuration, rather than merely confirming the shell exported the variable into the process environment. Verified against the real dotnet SDK (8.0 and 10.0 images) that the "GCHeapHardLimit" key is always present as a System.Int64 (0 when unconfigured), and re-ran the MemoryLimit integration suite end-to-end against the docker switchblade backend - both cases still pass with the same expected hex values (0x30000000 default, 0x20000000 with a DOTNET_GCHeapHardLimitPercent=50 override).
Update: this workaround is unnecessary once #1318 lands — closing in favor of #1318While investigating the related .NET 10 segfault issue, I found that the root cause behind this PR ("GC can fail to detect the real container memory limit and treat memory as effectively unlimited") is the same underlying upstream bug fixed in #1318/#1319: dotnet/runtime#130092 — VerificationI built a test buildpack with the dependency bump from #1318/#1319 (dotnet-runtime/dotnet-aspnetcore 10.0.11, dotnet-sdk 10.0.400) — without this PR's Result:
This shows the CLR correctly read the real container cgroup v2 memory limit on its own and applied its standard 75% default — with no ConclusionOnce #1318/#1319's dependency bump to (This approach could still have narrow value as a stopgap for teams pinned to older, unpatched .NET 8/9/10 runtime patch versions that predate the fix — but the clean, permanent fix is the version bump in #1318.) Thanks @benjaminguttmann-avtq for the pointers (on slack). |
|
Closing in favor of #1318, which fixes the actual root cause via the upstream .NET 10.0.11 dependency bump. See comment above for verification details. |
1 participant
Summary
Fixes #1100 ("Dotnet Core Buildpack does not handle cgroup v2 memory limits correctly – app does not release memory properly").
On cgroup v2 (and "hybrid" v1+v2) Diego cells, the .NET GC can fail to detect the real container memory limit and treat memory as effectively unlimited. This is a documented, unresolved ambiguity in coreclr's own cgroup detection (see dotnet/runtime#34334: "It is possible to have both cgroup v1 and v2 to both be enabled on a host... this change will pick one, which may not be the correct one"). Note the cgroup hierarchy in play is a property of the BOSH stemcell running the Diego cell (the host kernel/OS), not the CF stack —
cflinuxfs4/cflinuxfs5are just rootfs images bind-mounted into the app container and are orthogonal to which cgroup hierarchy the host kernel exposes; any stack can run on a Diego cell backed by any stemcell.Rather than reimplementing cgroup-file parsing in the buildpack (which would just move the same ambiguity elsewhere), this PR has the buildpack read Cloud Foundry's own
$MEMORY_LIMITenvironment variable — which Cloud Controller computes and injects independently of any cgroup layout (seecloud_controller_ng/lib/cloud_controller/diego/environment.rb) — and uses it to explicitly setDOTNET_GCHeapHardLimitat container startup, bypassing the runtime's own detection entirely.profile.d/startup.shscript (written byFinalizer.WriteProfileD), which runs at every container start (not baked in at staging time, sincecf scale -mchanges$MEMORY_LIMITwithout restaging).GCHeapHardLimitPercent), leaving headroom for non-GC-heap memory.DOTNET_GCHeapHardLimitorDOTNET_GCHeapHardLimitPercentyourself (e.g. viacf set-env) is honored, with a safety clamp (+ logged warning) if an explicit override would exceed the real container limit.DOTNET_GCHeapHardLimit/DOTNET_GCHeapHardLimitPercentvariables.$MEMORY_LIMITis absent (e.g. non-CF environments), behavior is unchanged (no-op).GC.GetConfigurationVariables()["GCHeapHardLimit"](available since .NET 7) rather than reading back the raw environment variable, proving the CLR itself parsed and applied the value into its live GC configuration.Test Plan
src/dotnetcore/finalize/finalize_test.gocover the script's generated content and, viaexec.Command("bash", ...), its actual runtime arithmetic across all documented scenarios (default 75%, custom percent, invalid/out-of-range percent fallback, user override respected, user override clamped with warning,MEMORY_LIMITabsent/malformed no-op, leading-zero octal edge cases,MEMORY_LIMIT=0mfloor).go test -mod=vendor ./src/dotnetcore/finalize/... -run TestFinalize -v— 18/18 passing.go vet -mod=vendor ./src/dotnetcore/...clean.src/dotnetcore/integration/memory_limit_test.go) via a debug endpoint on the sharedsimplefixture app that reportsGC.GetConfigurationVariables()["GCHeapHardLimit"].MemoryLimitintegration suite end-to-end against theswitchbladedocker backend on a real deployed app (both the default-75% andDOTNET_GCHeapHardLimitPercent=50override cases pass, confirmed against both .NET 8 and .NET 10 SDK images).