plotting.md

Plotting

How an experiment's run directories become a paper figure: extract once, then draw every figure from the extracted observations. Statistics behind the corpus figures (speedup heatmap, per-kernel distribution grid): measurement_statistics.md. Token cost and cards: token_accounting.md.

Design contract

Every figure in the HPCAgent-Bench papers follows these rules. A figure that breaks one is wrong.

1. Library, not script. Every figure is a function in hpcagent_bench.stats (figures.efficacy, figures.per_kernel, figures.kernel_speedups, figures.scaling, summary, palette, style). statistics/plot_*.py only parse arguments. A missing capability goes into the library with a test, never into a script or a paper repository.
2. Speedup axis = log2 of the ratio (summary.log2_change): 2x at +1, 0.5x at -1, 0 = no change, ticks labeled back in ratios (style.ratio_tick_label). Never signed_change (ratio - 1), never a bare ratio axis.
3. Statistics per Hoefler and Belli (SC15), as stats.rules encodes them: paired per kernel. A ratio (speedup) summarizes with the geomean of per-kernel ratios and its 95% BCa bootstrap interval (summary.geomean_ci); a cost (tokens) with the ARITHMETIC mean, so a cost change is a ratio of arithmetic means with a paired bootstrap interval (summary.mean_ratio_ci). Nothing is joined by a trend line (rule 12); the emitted table carries raw milliseconds and token counts (rule 4). Speedup and token cost never share one axis.
4. Colour = model, shape = treatment. One registry, one lookup per channel, the same answer in every figure:
   • Colour is the LLM: palette.model_color. A model drawn several times in one figure (with and without a packet, on several devices, in several pairs) wears close shades of its colour, palette.model_shade(model, step); its no-packet control is palette.CONTROL_SHADE (one step) lighter. Compilers and libraries take palette.framework_color, never a model colour.
   • Shape is the treatment, from the registry's one shapes: pool: its first harness_shapes free shapes are the harnesses', the rest the packets' (skill, tool, method), each kind taking its part in slot order (palette.shape_table, palette.harness_marker, palette.packet_marker). A packet may pin one with marker: (the paper's Skills diamond, CPF square, Terse triangle, Git X). Two treatments never share a shape; the pool outgrown is a registry error. Registering a treatment gives it a shape without reshaping any other.
   • Control is always the hollow circle palette.CONTROL_MARKER in its model's control shade, whichever packet, harness or task form it is the control of. No treatment is ever given it.
   • Statistics (a median line, a fit, a reference) are not entities: they use style's neutral inks and statistic inks, never a model or treatment colour. No figure carries a hex literal.
   • Standalone optimizers (DaCe, CPF as a compiler, Pluto, PPCG) keep palette.marker shapes in the compiler figures, where the optimizer is the entity.
   • Any other registry entry (a framework column above all) takes its colour and marker from its registry slot (palette.look): colour = hues()[slot % H], marker = markers()[(slot // H) % M]. An entry registered at runtime or by a plugin plots with no edit to the plot code; an unregistered name takes a stable CRC slot, never a shared grey.
5. Efficacy figure (hpcagent-bench plot efficacy, dot rows only). One row of panels, each an intervention against its control; rows: speedup (log2, over each kernel's own baseline), solved rate (%, no interval: a census), billed token cost. The x axis groups by delivery (C | Fortran, HIP | Triton | OpenMP): one tick per language, its models side by side in their colours, a light rule between languages. Several packets in one panel (intervention=packets) sit under their language's tick in their own shapes; a harness panel (intervention=harness) gives each harness its own group and shape. Speedup is a geometric mean and token cost an arithmetic mean over kernels, each with a 95% bootstrap interval; --mode relative draws each as the change over the LLM's own no-packet baseline. --mark scatter (default) draws each estimate as a marker, --mark bar as a bar from 1x (from the axis floor on the absolute token row); the interval is the same under both (figures.marks). Intervals from summary.MIN_PAIRS_FOR_INTERVAL (6) kernels. Compiler/framework comparators (comparators=) sit after the models of their delivery on the speedup and solved rows only, in palette.framework_color and an optimizer shape no packet wears (figures.efficacy.comparator_shapes). The paper key has four columns (PAPER_CONFIG.legend_ncol, compact spacing).
6. Per-kernel figure (hpcagent-bench plot kernel-speedups): wide, two rows on one kernel axis: log2 speedup per kernel with its interval on top, tokens per kernel below. Past a dashed separator, one summary slot per series: speedup = geomean with 95% bootstrap interval over the SOLVED kernels, tokens = median over the served kernels. The tokens row is omitted when no series carries tokens and is labelled by the priced cost card (efficacy.cost_axis_label, e.g. Billed Tokens (1, 0.1, 1)). A setup is coloured by its model and shaped by its packet, hollow for no packet (palette.model_color, palette.packet_marker), as every model x packet figure is. The speedup axis names the observations' own denominator (best-of(numba,c) reads "Best of GCC and Numba"; --baseline overrides it, numba for a canon-only figure), and without --canon-db only the setups are drawn, over the kernels of --tag-file.
7. Missing value = hollow cross. A kernel without a verified answer enters at 1x (population.NOT_DELIVERED), is drawn crossed, named style.NOT_DELIVERED_LABEL in the key, and left out of every summary. Hollow alone means control. A ? marks data not run yet (--mark-pending).
8. Sizing and type. A paper figure is drawn at the width it is placed at (style.ICLR_TEXT_WIDTH_IN, style.ICLR_WRAP_WIDTH_IN, ACM_*) on the print scale style.PRINT_SCALE: ticks and point labels 7 pt, axis labels and panel names 8 pt, keys 6 pt (PRINT_LEGEND_PT), nothing fitted below 5.5 pt (PRINT_MIN_PT). Authoring figures use style.AUTHOR_SCALE; a module never mixes the two. Save with style.save(fig, stem, width_in=...) (or print_size=True for a self-sized canvas): it refuses a canvas of another width and type off the print range. statistics/check_paper_figures.py <paper> checks every \includegraphics places its PDF at scale 1.0.
9. One legend per figure, below it (style.legend_below), never ax.legend. It names the interval method, n and the undelivered cross.
10. Minor ticks. Log axes, linear axes in log2 units and the tasks-completed row carry unlabelled minor ticks and a faint minor grid (style.minor_ticks, style.MinorLocator). Other linear axes and category axes carry none. A count over a fixed tag is a census: a mark with no interval.
11. Deliverable = PDF, 150 dpi PNG beside it, the CSV behind every mark, and the exact CLI. A bad figure is saved, shown with bad-vs-good, and asked about; never silently redrawn.

Further conventions:

• Identity keys by name. palette.model_color(name), palette.packet_marker(name), palette.framework_color(name) key by entity, never by list position. Key order in the explicit order of a registered class is its slot (hpcagent_bench/models.py, hpcagent_bench/skill_packets.py); renumbering one recolours or reshapes every published figure (tests/test_palette.py pins the rules: one shape per treatment, never the control circle, shades stay the model's hue).
• Names come from the registry through hpcagent_bench.study_tags (display_name, model_name, packet_name, framework_name), never literals. Serving details (sglang, -FP8) stay out of names.
• Baseline is a property of the data: population.one_denominator reads the column the judge stamped and refuses a mix, and a slice that records none.
• Costs. A kernel's tokens come from its episode row (population.kernel_tokens), priced with the billed card unless --cost-model names another (stats.cost.add_arguments). A summary over kernels is the geometric mean, never a median, and never over episodes in a cell.
• Intervals. No normality is assumed (Hoefler and Belli Rule 6). A summary interval is the 95% BCa bootstrap over kernels (summary.geomean_interval); a paired one inverts the configured paired test (sign-flip by default, the test registry). Both are withheld below summary.MIN_PAIRS_FOR_INTERVAL (6) values.
• Labels. Title Case (identifiers keep their spelling). Ticks at 0 or 90 degrees. Values print with one decimal (style.ratio_label: 6.3x, 0.04x below 0.1x); tokens with style.decade_label (35.5K). Labels beside marks are tagged style.CLEAR_GID and settled clear at save (style.settle_clear_labels). If a figure caps coverage, it prints what was dropped.
• A connector is a pair link, never a trend: it joins one setup's control and treated marks, and the legend names it Pair Link.

Extract once, plot from the observations

hpcagent-bench extract --runs "$RUN_ROOT/llrblind-*" --runs "$RUN_ROOT/6[0-9][0-9][0-9][0-9][0-9]" \
    --setup-prefix llrblind --benchmarks hpcagent_bench/benchmarks \
    --out data/llrblind --db data/observations.db --no-sources

--runs is a run-root glob (repeatable) and --setup-prefix keeps the setups of one experiment. Keep waves in a suffix (<experiment>-w2) so one prefix matches every wave. Check the printed summary: a missing setup means a wrong prefix. From Python: hpcagent_bench.studies.read_observations(path) reads the table back.

Registered studies (hpcagent_bench.experiments) extract by name, reading their experiments' run roots and the owed waves' owed-<study>-<date> roots, and fuse final-grade rows:

python -m hpcagent_bench.dataset --study llr-focus40-blind \
    --regrades "$RUN_ROOT/regrades/regrade-*.db" --out data/llrblind.db --csv data/llrblind.csv

From the results databases instead of run roots, --db names one or more results databases, read as one (hpcagent_bench.stats.databases.union). Several databases merge by natural key, so their row ids never collide; a setup two of them hold with different rows is refused.

python -m hpcagent_bench.dataset --study llr40 --db hpcagent-bench-v1-final2.db --out data/llr40.db

Regrade precedence, exempt submissions and promotion: measurement_statistics.md and experiments/README.md. Speedup comes from submission rows, cost from episode rows, both reduced by hpcagent_bench.stats.population (latest valid submission per kernel; the task's final-attempt tokens); the one-reduction checks run over each episode's answer only.

The figures

script	figure	library
hpcagent-bench plot efficacy	efficacy: speedup, tasks completed and token cost per comparison	figures.efficacy.figure_dot_row
hpcagent-bench plot kernel-speedups	every kernel of a tag: canon columns (compilers) and setups over one baseline, tokens below	figures.kernel_speedups.kernel_comparison
hpcagent-bench plot scaling	distributed track: eta(P), sigma(P), per-kernel, per-setup summary	figures.scaling
hpcagent-bench plot kernel-runs	every run (slot) per kernel and model, conditions (temperatures, packets) stacked side by side: speedup, token cost and solved runs; box or violin	figures.kernel_runs.kernel_runs_figure, stats.reliability

Run any script with -h for its flags. Worked commands for every figure, with the default protocol behind the numbers: statistics/README.md. Every command writes the PDF, a PNG beside it and the CSV behind every mark; open the PNG and check the CSV's solved column before quoting a figure.

Kernel runs figure (designed repeats, conditions stacked)

hpcagent-bench plot kernel-runs (figures.kernel_runs.kernel_runs_figure) draws every run (slot) of a designed repeat (population.designed_runs, spec R8) per kernel and model, the conditions it ran under stacked side by side (repeat5: 20 slots per setup and kernel; temperature3: 3 kernels x 20 slots x 3 temperatures per model). Every run is one dot; nothing is averaged away.

Conditions. A condition is read off the setup name (kernel_runs.condition_of): -t<T> is a served sampling temperature (condition t<T>), a packet suffix is that packet, and a setup with neither ran the default ("": the model's own generation_config.json temperature, DEFAULT_TEMPERATURE 1.0, and no packet). --conditions t0,default picks and orders the groups (default names the default); unset, every condition the runs ran is drawn, temperatures by value with the default among them, then packets in registry order. One group per condition, named above it (Temperature = 0, Temperature = 1 (Default), or the packet's name) and separated by dashed rules; a single condition is one unnamed group. Inside a group one column per kernel of --tag, its short name horizontal on up to three lines; inside a column the models side by side in their registry colours, close together (DODGE_SPAN).

Size. Up to two groups (SQUARE_GROUPS) make a square figure one ACM column wide, its group titles short (T = 0) and its key entries compact; more groups are ACM text width. --width overrides the width.

Rows (the efficacy dot row's labels and height fractions, efficacy.MEASURE_LABELS, MEASURE_HEIGHT x DOT_ROW_HEIGHT_IN, at print type; the square figure scales the rows to its height):

Row	Value per run	Notes
Speedup	final-grade speedup over the kernel's baseline	solved filled; unsolved hollow at 1x (no cross); log2 axis
Billed Tokens (1, 0.1, 1)	the episode's token total priced by --cost-model	weights under the label: (fresh input, cached input, output); log10 axis
Solved (%)	solved / graded runs per cell	a census mark, solved/graded beside it; owed runs counted apart (+N?)

Summary under the dots (--style):

• box (default): median and quartiles of the graded runs, whiskers to the lowest and highest run (WHISKERS = (0, 100) percentiles, not a 1.5 IQR fence, since every run is drawn anyway). Width BOX_STEP_SHARE of a model's dodge step.
• violin: matplotlib's Gaussian kernel density estimate (Scott's bandwidth) of the graded runs, computed over their log10 so the shape is not skewed by the log axis, mapped back onto it. It is a density estimate, not a bootstrap. On top, a line a shade darker than the model (CI_DARKEN): the median as a tick and its 95% bootstrap interval (summary.median_ci: 9999 percentile resamples, seed 0, no outlier rejection, withheld below 5 runs). The speedup interval includes the unsolved runs at 1x. A cell whose runs are all equal (every run at 1x) draws no violin.

Runs still owed a final grade or a rerun refuse the figure and the --table statistics; --allow-owed draws them as ? for a preview of an unfinished study. Every run drawn goes to the CSV beside the PDF, with its condition, state, speedup and tokens; --table writes the per-cell reliability statistics.

python -m hpcagent_bench.dataset --study temperature3 --out data/temperature3.db
hpcagent-bench plot kernel-runs data/temperature3.db --experiment temperature3 --tag temperature3 \
    --conditions t0,default --out figures/temperature3_t0_t1.pdf          # the paper's figure: square, one column
hpcagent-bench plot kernel-runs data/temperature3.db --experiment temperature3 --tag temperature3 \
    --style violin --out figures/temperature3-violin.pdf
hpcagent-bench plot kernel-runs data/repeat5.db --tag repeat5 --out figures/repeat5-runs.pdf \
    --table data/repeat5-reliability.csv

Efficacy figure

Drawn by figures.efficacy.figure_dot_row in one of two modes (--mode, FigureConfig.mode):

• absolute (default): each column is one model and delivery; the control is a hollow circle, named "Baseline ()" in the key, the treated setup the packet's shape.
• relative: each LLM's no-packet control is its own 1x line ("Baseline () = 1x"); the one mark per column is the treated setup's change over it (speedup: geomean of paired ratios; tokens: ratio of arithmetic means), and comparators are not drawn (they have no control). A baseline band at the foot of the speedup and token rows prints each control's absolute level in its model's colour: its geomean speedup over the experiment baseline (5.4x) and its arithmetic-mean tokens per kernel (1.15M); the solved row draws the control's rate as a dash. The emitted -pairs.csv carries both levels with their intervals (control_speedup*, control_mean_tokens*, treated_*) beside the ratios, so a caption can quote them.

Panels are tagged (a), (b), ... as LaTeX subfigures; row names are folded onto two lines and set at one size for every row; tick and key text is in title case (efficacy.title_case). Rows:

• Speedup: geomean over the kernels both setups solved; a wrong answer is left out, a correct slower answer keeps its sub-1 ratio. --speedup-over served draws every kernel with a failure at 1x.
• Solved Rate: kernels solved per setup on a 0-100% axis, no interval (census). --no-success-row drops it.
• Token cost: every served kernel, failed ones included, priced with --cost-model (the library prices with the billed card when called without one: figures.efficacy.paired_kernels).

* marks a significant speedup change and + a significant token-cost change after Benjamini-Hochberg correction within the panel's family: one family per panel, exactly the tests it draws. An interval is cut at FigureConfig.interval_reach past the outermost mark with an arrowhead; a mark over fewer than FigureConfig.min_interval_kernels kernels has none.

--comparison is a key=value;... spec, one per panel. Either treatment=<packet> (split one experiment on a recorded packet) or pairs=<csv> (pairs from paired_setups.py, whose corrected verdicts are the stars; the figure recomputes only the drawn point through figures.efficacy.reduce_pair). Other keys:

key	effect
title, intervention	panel title; registered packet key the treated side wears (shape, name); packets/harness: each column wears its own packet's or harness's shape
observations=a.db,b.db	observations for this panel; default the positional files
control-label=...	legend name of a control that is not "no packet"
placeholders=Fortran	empty column for a leg with no data yet
pending=kimi27sglang,qwen38	empty column per model with no pair yet; ? with --mark-pending
difference=HIP:qwen38,...	grey bar between a named pair's two marks, with its factor
comparators=<csv>	compiler/framework marks from a kernel,comparator,device,numba_ms,ms,speedup table (one row per tag kernel, speedup blank where invalid)
comparator-set=pluto:C,jax_cpu:C	which comparators the panel draws and under which delivery; no :group = the panel's first delivery. One mark each: geomean of speedup over the valid kernels, 95% bootstrap interval from summary.MIN_PAIRS_FOR_INTERVAL kernels; solved row = valid / tag; nothing on the cost row. Numbers go to <table>-comparators.csv

A single comparison can also use top-level flags:

hpcagent-bench plot efficacy scored.csv blind.csv \
    --pairs-csv blind_vs_scored.csv --intervention no-score --control-label "Score Tool" \
    --out figures/blind.pdf --table data/blind.csv

--row-width {natural,iclr,iclr-wrap,acm-column,acm-text} sizes a joined row to a page budget. --no-success-row drops the solved row.

Per-kernel figure

hpcagent-bench plot kernel-speedups draws one row per --canon-columns column of the canon DB and per setup of --experiment (<experiment>-<model>-<language>[-<packet>], narrowed by --setups and --conditions), every kernel of --tag-file on one axis:

• Numba is the denominator (the 1x line; --baseline changes it, --baseline-fallback times a kernel the baseline did not verify). Compiler columns are comparators.
• Filled mark = measured. Hollow crossed mark = no validated result, drawn at 1x, kept as a row of -kernels.csv (canon.tag_speedups), left out of the summary; read the n column of -summary.csv before quoting a geomean.
• Without observations only the compiler columns are drawn. --offset spreads a kernel's rows across its slot; 0 stacks them. --mark-pending draws a kernel a row has not attempted yet as ?.
• When both DaCe device columns appear, each falls back to its frameworks name, which carries the device (kernel_speedups.distinct_canon_labels).

Scaling figures

hpcagent_bench/stats/commands/scaling.py draws the distributed track from the same observations, rows with record == "scaling". Required columns: ranks (P), ranked_ns (T(P)), single_rank_ns (T(1)); optional: scaling_mode (weak/strong), nodes, work_ratio (r; missing on a weak row means r = P), scaling_note. The judge persists scaling_points (harness.recording.record_scaling); extraction turns them into scaling rows.

Every point goes through harness.metric.scaling_point, the function the grade uses: eta(P) = T(1)/(P T(P)) strong, r T(1)/(P T(P)) weak. A row whose recorded efficiency disagrees is refused (figures.scaling.disagreements). A P the sweep could not measure is a hole, never a zero, and is listed with its reason in <table>-dropped.csv. P is a log2 axis with ticks at the rank counts run and no grid. Weak and strong are panels; colour and shape are the model.

torch.distributed baseline curve. The scaling grade (harness.scaling_grade, the gang shape of hpcagent-bench job grade-under) also times the kernel's own reference_dist at every (kernel, law, P) point of the sweep, independent of any submission (harness.torch_dist_curve: torch.compile under the one-GPU baseline's autotune config, eager only when the compile fails), and stores it once per (kernel, law, P, params, GPU arch, image) in the grade DB's reference_scaling_points table under source = 'torch_dist'. Extraction reads those rows as scaling rows under the pseudo-setup torch_dist, and every overlay panel draws it in the control's grey, dashed, beside the models; --no-torch-dist leaves it out.

OBS=data/mlscale20.db  # python -m hpcagent_bench.dataset --study mlscale20 --out data/mlscale20.db
hpcagent-bench plot scaling "$OBS" --experiment mlscale --out figures/scaling --table data/scaling.csv
hpcagent-bench plot scaling "$OBS" --experiment mlscale --figure efficiency --out figures/scaling
hpcagent-bench plot scaling "$OBS" --experiment mlscale --figure speedup --out figures/scaling
hpcagent-bench plot scaling "$OBS" --experiment mlscale --figure per-kernel --mode strong \
    --quantity efficiency --out figures/scaling
hpcagent-bench plot scaling "$OBS" --experiment mlscale --figure summary --out figures/scaling
hpcagent-bench plot scaling "$OBS" --setups 'mlscale-qwen38-hip' --width 5.5 --out figures/scaling-qwen38

A new figure

Add a function under hpcagent_bench/stats/figures/ with a test, then a thin script in statistics/. Call style.apply() before importing pyplot, take colours and shapes from palette (model_color, model_markers), names from study_tags, and save with style.save (PDF and PNG). hpcagent_bench/stats/commands/kernel_speedups.py is a short example.