Repository navigation
HPCAgent-Bench v0.2 - #76
Open
ThrudPrimrose wants to merge 232 commits into
Open
ThrudPrimrose wants to merge 232 commits into
ThrudPrimrose wants to merge 232 commits into
Conversation
… on the default stream, transients pooled parallel keeps the classic CPU generator and now runs the new GPU one, as canon does (canon keeps the new generators on both); both offload with the new pass. max_concurrent_streams is pinned to -1, DaCe's "default stream only" (every stream nullptr) -- 1 created one stream, and canon's offload already assumes the default stream. Every device transient is allocated from the stream-ordered pool (MallocAsync/FreeAsync). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The CPF skills split by device and by form (source page vs tool), each with its own packet key; the old keys stay as aliases. A CPU page now says why a loop the analysis proved parallel runs sequentially. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
dace flavors: parallel on the new GPU generator; default stream only; pooled device transients
Four CPF skills: cpf-cpu-src, cpf-gpu-src, cpf-tool-cpu, cpf-tool-gpu
- The schema is the v0.2 results schema (PRAGMA user_version = 2); every other file is refused. All v6 handling goes: fold_legacy_grade, upgrade_legacy, grade_columns, LEGACY_*, grade_under.requests_select and promote_unsubmitted's legacy request columns. - episodes: rep dropped; job NOT NULL DEFAULT 0 (0 = a local run), keyed on (job, label). job_tag/job_of_dir return LOCAL_JOB; readers show it as "". - upsert keeps the first copy; only episodes fill columns still at their default (token counts recorded later), so a real temperature 1.0 or a credited speedup of 0 is never overwritten by merge order. - copy_foreign skips sweep rows equal in every column but id: two files carrying one sweep no longer double its samples. - merge keeps the destination's WAL journal (a judge may hold it open). - requests_of_row reads a sqlite3.Row as well as a dict. - copy_grade refuses a re-time of a re-time instead of a KeyError. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…story config.yaml: one comment per key with type, unit, valid values, reader and effect; header states the full precedence chain (set_override > scoped env > os.environ > file > reader default) and that whole-map keys ignore per-entry env names. Corrected: mpi.residency default (host), service.input_mode value set, measurement.baseline (no service.baseline), record.db_path (no schema version), runtime.backend value set incl. native and podman Harbor provider, fuzz anchor (fuzz.anchor, not always XL), judge.gpus_per_node null semantics, limits units; removed the orphaned drain-timeout paragraph. YAML values are unchanged. docs/configuration.md: new section on config.yaml resolution, env names, coercion and the null rule; SBATCH_ACCOUNT sentence fixed. docs/runtime.md, measurement_statistics.md: backend default and db_path stated as configured; quiescence calibration moved here from config.yaml. docs/serving: no job ids, dates or node names; readiness keys attributed to their layers. experiments: /score protocol, LAUNCH sizing table (72000 s), README anchor, layer comments (common.env, model-glm53.env, hardware-mi200.env), setups.yaml comments without job ids. Docstrings: history, run anecdotes and measurements replaced by the contract in cluster/, harness/, support/, agent driver and runner modules. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tests/test_docs_no_stories.py fails on "used to", "previously", "we found", "since PR", "historically", "job 6xxxxx" and "nid<digits>" in docstrings under hpcagent_bench/ and agent/, docs/**/*.md and config.yaml comments, except matches named in a per-match allowlist (stale entries fail too). Registers the pytest marker `smoke`; the scan is marked smoke. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
submit.sh now stages every setup before submitting any. submit_waves (submit_common.sh) splits the staged setups, in staging order, into waves whose setup_nodes counts sum to at most MAX_NODES (plan_waves) and submits each wave afterany on every job id of the previous wave; the first wave depends on DEPEND_ON. MAX_NODES empty (default) is one wave. A MAX_NODES that is not a positive integer, or a setup larger than it, is refused before any sbatch call. Each wave prints "wave <i>/<count>: <n> nodes" before its setups, under SUBMIT=1 and in the dry run. submit_setup_job sets SUBMITTED_JOB_ID (the --parsable id without its ";<cluster>" suffix). Registers the pytest marker smoke. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…n TODO LAUNCH.md section 0 documents MAX_NODES waves and REPEAT (agents per kernel, default the base's SUBMIT_REPEAT). gitscicomp10.md names SUBMIT_REPEAT: 3 as its agents per kernel, REPEAT as the per-submission override, and a TODO: paper rerun with five agents per kernel (REPEAT=5 MAX_NODES=32). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
New config block `requests:` (precisions: [], layouts: false, sparse_formats: false) read by requests_api.RequestAllowances (pydantic, frozen; precisions also accept a comma-separated environment value). - service.requests_refusal refuses, before the kernel checks, a cast to a precision the setup does not list, a single-node digit operation or any zip/unzip without requests.layouts, and a format operation without requests.sparse_formats (HTTP 400, submission not spent). startup.load stays gated by grading.allow_agent_build_tokens; startup.grid and the distributed layout over it are not gated. - The prompt's requests section names only the allowed precisions (with their C types), layouts and zip/unzip, and says every other request is refused; sections/sparse.j2 states only the default format, its symbol and index semantics unless requests.sparse_formats is on. - pydantic is now a direct dependency. - Tests: tests/test_request_allowances.py (unit + one smoke test against a live in-process judge); `smoke` marker registered in pyproject; tests that send gated requests opt in via the every_request_allowed fixture. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…, slowdowns as 1/Yx - figures/temperature.py -> figures/kernel_runs.py: every run (slot) per kernel and model with the CONDITIONS it ran under stacked side by side (a served temperature t<T>, a packet, or the default); statistics/plot_kernel_runs.py replaces plot_temperature.py and plot_repeats.py (per_kernel.runs_figure goes). `--conditions t0,default` reproduces the paper's temperature3_t0_t1.pdf. - figures/signed.py -> figures/kernel_speedups.py, drawn by the new statistics/plot_kernel_speedups.py (split out of plot_score_change --per-kernel). - Deleted: plot_speedup.py and figures/results.py (no paper uses them), plot_setup_summary.py, palette.framework_colors (unused). - Ratio labels: a slowdown reads as its reciprocal, 1/2x, not 0.5x. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Public information block, negative symbols, nonnegative_symbols helper
The machine_learning kernels are graded in bf16, but their canonical
parallel form was parsed with dc_float = float32 (bind_precision had no
bf16 entry) and rendered and looked up at the run's service.datatype.
- bind_precision("bf16") binds dace.bfloat16; DaCe's CPF renders it as
the dialect's own bfloat16 (__hip_bfloat16 / __nv_bfloat16 / __bf16).
- A form is rendered, served and verified at the datatype its kernel is
graded in (cpf_bridge.form_fptype, graded_datatype): prerender,
render_on_demand, the canonical_parallel_form route, make_problems and
cpf_verify agree on the key.
- The entry keeps the stub's spelling (__npb_bf16) and the unit names it
as the native type (typedef __hip_bfloat16 __npb_bf16;), two bytes in
the stub's uint16_t storage layout: the ABI check that compiles the
stub prototype after the unit passes, and the judge passes its storage
pointers unchanged. C has no bfloat16; CPF refuses it and names C++.
Checked: gemm_add_relu at bf16 rendered in C++ and called through the
judge's _call_native matches numpy; rendered for hip and called on the
MI300A through a uint16_t stub declaration, it matches a float32 numpy
reference within bf16 accumulation (max err 0.92 at max |out| 20.8).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The combined dace pin for v0.2: native bf16/fp16 CPF forms, the canon and CPF fixes of the bad-actor triage, parallel StateScan lowering, the no-pivot LU lift, the main PR ports; Alloy pins the same dace commit. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
v0.2: native bf16 CPF forms + dace/Alloy pin bump
…ic change removes the dependence The CPF skills and the canonical_parallel_form tool now say a sequential loop is proven sequential with its carried dependence named, so re-scheduling cannot parallelize it and only an algorithmic change can (break an anti dependence, privatize, reformulate the recurrence); a parallel loop is proven safe, possibly under a stated guard. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CPF loop verdicts: sequential loops are proven sequential
grading: cut a grade's peak memory (cloudsc_sedimentation XL fits 4 ranks/node)
The ubuntu-toolchain-r PPA answered 503 / timed out mid-download on run 38962821730, failing the mpi job's setup; with no venv every later phase then failed with 'No module named pytest'. Only add-apt-repository was retried. Retry the installs (apt keeps fetched debs) and let apt retry individual downloads. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CI setup: retry the GCC 16 / LLVM 22 apt installs
…R ROCm runtime) on the image's LLVM 22 A spack overlay package, installed against the environment's llvm by hash and pushed to the buildcache, in its own layer last in the agent stage. Source: randreshg/Polygeist@e503e554 (Polygeist migrated to LLVM 23-dev) with llvm22.patch: the LLVM 22.1 API, Polymer ported, raise-scf-to-affine raising loops nested in affine loops, malloc of a VLA row pointer, the C-style VLA rewrite kept out of the descriptor lowering, and no lit suites. Polymer's Pluto pins built without pet, so no LLVM 10. libmlir_rocm_runtime.so is built from the same LLVM release's source. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… built the way the Pluto column is cgeist lowers the C target's _pluto_input.c to affine MLIR; polygeist-opt parallelizes it with MLIR's own passes (CPU: affine-parallelize -> OpenMP; GPU: affine-parallelize -> 16x16x1 tiles -> gpu.launch_func -> ROCDL gfx942:xnack+, serialized by the image's mlir-opt); the image's clang compiles the LLVM IR. cgeist types a VLA parameter as memref<?x?xT>, which only the memref-descriptor ABI passes, so a generated C shim gives the column the scop's own signature (the pluto binding's order and shapes, so PlutoFramework.call_args applies unchanged). Every failure declines; a GPU nest that does not map is declined, never timed on the host. polygeist_gpu runs on host-resident arrays under HSA_XNACK=1. Polymer is built into the image but not run: its schedule computed heat_3d wrong. Pluto's oracle gate stays Pluto's; the Polygeist columns keep the harness's own --validate, as the PPCG ones do. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…gain Ubuntu's linux-headers-6.8.0-147 (pulled by the perf fallback's linux-tools-generic) triggers DKMS to rebuild the base image's amdgpu module, which fails for that kernel; the fallback's `|| true` hides it, dpkg stays half-configured, and the ROCm layer's apt-get install fails (build 684214). A container loads no kernel module. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…timization flags come from flags.py) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Polygeist columns (polygeist_cpu, polygeist_gpu) + Polygeist in the AMD image
manifest(**overrides: Any) was the one new ANN401 on release-v0.2, failing pre-commit's annotation ratchet. Every caller overrides only parameters, so take that as a typed keyword. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
test_public_info: type the manifest override (annotation ratchet)
…ction The 512 MiB negative control raised SIGSEGV only where gcc probes the stack by default (Ubuntu). The image's spack GCC 16.2 does not: the stack pointer jumps 800 MiB past the one-page guard into other mappings and the kernel runs clean, so the test failed in the judge image. Probe every page explicitly; all 53 tests pass in the judge image. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
test_kernel_memory_cap: build the VLA kernel with -fstack-clash-protection
…legacy dict Benchmark.initialize re-parsed BenchSpec from the legacy info dict, which carries no public block (nor config space, mpi, constraints, ...). Since public.negative validation runs in from_dict, every kernel declaring a negative symbol (mandelbrot1/2, smith_waterman, needleman_wunsch, cp2k_density_matrix_trs4, ...: 12 at S) raised ValueError and could not generate inputs. Use self.spec, which the legacy dict is derived from and never mutated. S data of all 712 other kernels is bit-identical (sha256). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
get_data: initialize from the loaded manifest (12 kernels could not generate inputs)
…tions away, a few far support/icon_connectivity.local_neighbours draws every (idx, blk) table of the ICON kernels (velocity_tendencies, zekin_gather, zekin_scatter, zekin_gather_scatter, icon_gather, icon_scatter, icon_edge_grad_w) around the owner's position in the flat blk * nproma + idx numbering, as ICON's space-filling-curve numbering places them: 1 + floor(|Normal(0, 0.75)|) positions away with a random sign (about 80 % one away, 17 % two), and 3 % of the slots uniformly anywhere in the patch. The address wraps, so every slot stays in range; all draws come from the counter generator. The manifests state the locality in public.description. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- spec: from_dict checks public.negative only when the dict carries a public block; from_yaml checks every manifest. The bridge's legacy dict has none, so test_spec_roundtrip's re-parse raised for the 12 kernels with negatives. - native_call.RepCheckServer declares __slots__ (test_class_hygiene). - test_aa_calibration: the fake C reference computes on the inputs the real one does (the timing's canonical follow-up, a rep's own draw). - test_optimize: optimize reads the kernel's small symbols, so pass a bench. - test_jobs: baseline.sbatch runs its verb through job_step's array. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CI on release-v0.2: legacy dict re-parse, RepCheckServer slots, three stale tests
ICON neighbour tables are mesh-local
…ll-time fallback njit compiles lazily, so a numpy reference numba cannot type passed the wrap-time check and fell back to the interpreter on its first call: the agreement test and helpers/scripts/njit_oracle_gate.py both read it as 'agree'. NJIT_INTERPRETED named 24 kernels; the gate, now catching the fallback, measures 272 (np.pad, matmul, max/mean(axis=...), einsum, fft, ...) on release-v0.2 in the judge image. - njit_oracle_gate.classify and test_njit_reference_agrees fail on the guard's 'failed to compile on call' record. - NJIT_INTERPRETED regenerated from that run (contour_integral kept for its LAPACK disagreement); it is keyed by short name, as the gate and the tests name it (njit_reference looked up module_name, which differs for e.g. bicg_solvers/sp_bicg). - conv_2d: its reference spells the edge padding out instead of np.pad, so numba compiles it: the interpreted loop nest took hours at L and stalled a canon sweep rank. Bitwise identical to np.pad(mode='edge'). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
pythran 0.18's CLI takes the header after a bare -include for a second
input file ('Unsupported file extension: .h') and dies with an IndexError,
so every pythran build in the judge image failed. -Wp,-include,<h> hands
the preprocessor the same -include; verified in the judge image (the
module builds and runs; jacobi_2d validates; tests/test_vecmath.py 17 passed).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
flags: pass the vecmath header to pythran as one -Wp, token
njit oracle: list every reference numba cannot type; catch the call-time fallback; conv_2d compiles
The NumpyToNumba emit keeps the numpy body's 2-d boolean-mask update (Z[I] = Z[I]*Z[I] + C[I]), which numba cannot lower (AssertionError in its BooleanArrayIndexer: 1-d masks only), so the numba column failed for mandelbrot1. Walk each pixel through the same sequence instead, rows in parallel. Judge image, S: validates (0.066 ms vs 0.535 ms C); test_numba_reference_overrides[mandelbrot1] passes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mandelbrot1: hand-written numba reference
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
HPCAgent-Bench v0.2: every fix and design change that must land before the paper experiments are rerun. Merged here: the v0.2 workstreams (grading decisions, smoke suite, hardcoded paths, prompt requests, config and docs precision, submit waves), the final results schema, the relayout kernels (#74), the CPF skills from
dev, and the pydantic port.Results database
hb-results-v1.0(PRAGMA user_version = 10). A file with any other schema is refused; there are no migrations and no compatibility readers.record.db_pathsets the database file (defaultresults/hpcagent_bench.db).grades.task_hash).Pydantic for every JSON format HB writes and reads
config.yamlconfig.Settingsvalidates it; a missing or unknown key is an error. Resolution order:config.yaml, thenHPCAGENT_BENCH_*, then runtime overrides.Submission; an unknown field is refusedservice.RequestFields; an unknown field is a 400;presetis no longer a body fieldgrade_under.Item/Scalingproblem_file.ProblemLinetokens.jsonepisode recordsepisodes.EpisodeRecordViewHeader,Pointer,Verificationnumpyto_c.bindings.BindingFileSeedsShardPlan/ShardResultDistPlan/DistResultTypeAdaptersThe
agent/package,fused_split.pyandgang_relay.pystay on the standard library: they run inside the agent images and on the batch host's own Python.Compatibility code removed
init.shapesspelling: sparse buffers are declared underinit.arrays, andparse_initrefuses any unknowninitkey.tokens.jsonrecords.presetplumbing in the judge client.Pins
f23d625.1422d49(pydantic records); both pin the same dace.Checks
pre-commit run --all-filesis clean.🤖 Generated with Claude Code