Add codameter-bench: shardable config-sweep CLI for Fargate/Batch#13
Open
mdenolle wants to merge 1 commit into
Open
Add codameter-bench: shardable config-sweep CLI for Fargate/Batch#13mdenolle wants to merge 1 commit into
mdenolle wants to merge 1 commit into
Conversation
Scale the config sweep (case x config grid -> recovery RMS) across a cluster. The work is deterministic and embarrassingly parallel, so it fans out over an array of tasks with no coordination. - src/codameter/bench.py: `plan` (size the job, no compute), `sweep` (score this shard's cells), `aggregate` (merge shard files + per-case best config). Grids (compact/multiverse/wide) are built per use case around its recommended config, so each case keeps a physical band/window. `--shard k/N` round-robins the work (even load despite slow moving/inversion references); `--jobs` uses local vCPUs; `--out` writes one shard-<k>-of-<N>.jsonl to a local dir or s3://. On AWS Batch the shard is read from AWS_BATCH_JOB_ARRAY_INDEX + CODAMETER_SHARDS so one image runs every array task. Bad cells are caught per cell, never kill the shard. - docker/: Dockerfile (ENTRYPOINT codameter-bench), an AWS Batch Fargate job definition, and a run guide. - console script `codameter-bench`; optional `aws` extra (boto3) for s3 output; pixi task `bench`; README section. - tests/test_bench.py: grid shapes, disjoint/covering shards, env shard parsing, cell scoring (recovery + error handling), sweep round-trip, best-config. plan: multiverse = 6480 cells (648/case) over 10 cases, balanced across 64 shards. Verified natively and in the built container (plan + full sweep). Full suite: 199 passed, 1 skipped. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Adds a new codameter-bench CLI and supporting Docker/AWS Batch assets to shard and scale the “golden case × config grid” recovery-RMS sweep across local CPUs or an AWS Batch/Fargate array, plus tests and documentation.
Changes:
- Introduce
src/codameter/bench.pywithplan,sweep, andaggregatesubcommands, sharding (k/N) and optional multiprocessing. - Add a console script entry point + optional
awsextra (boto3) and apixitask for running the benchmark. - Add Docker image + AWS Batch job definition examples and pytest coverage for the new bench functionality.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
src/codameter/bench.py |
New shardable benchmark CLI (plan/sweep/aggregate), local + S3 JSONL IO, multiprocessing runner. |
tests/test_bench.py |
New tests covering grid construction, sharding, env parsing, scoring, and JSONL round-trip. |
README.md |
Docs section explaining how to scale sweeps via shards / Batch array jobs. |
pyproject.toml |
Adds codameter-bench console script and aws optional extra for S3 output. |
pixi.toml |
Adds pixi run bench task to run the module as a CLI. |
docker/README.md |
Step-by-step guide for running locally and on AWS Batch (Fargate). |
docker/Dockerfile |
Container image for running codameter-bench (installs .[aws]). |
docker/aws-batch-job-def.json |
Example AWS Batch job definition for an array sweep. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+140
to
+142
| if spec: | ||
| k, n = spec.split("/") | ||
| return int(k), int(n) |
| drop_err=m["drop_err"], n_valid=int(np.sum(valid)), ok=True, | ||
| error=None) | ||
| except Exception as exc: # a bad cell must not kill the shard | ||
| row.update(rms=None, drop_err=None, n_valid=0, ok=False, error=str(exc)) |
Comment on lines
+271
to
+273
| case_ids = _all_case_ids(args.cases) | ||
| items = work_items(case_ids, args.grid) | ||
| k, n = parse_shard(args.shard) |
Comment on lines
+246
to
+253
| for obj in s3.get_paginator("list_objects_v2").paginate( | ||
| Bucket=bucket, Prefix=prefix): | ||
| for it in obj.get("Contents", []): | ||
| if it["Key"].endswith(".jsonl"): | ||
| body = s3.get_object(Bucket=bucket, Key=it["Key"])["Body"].read() | ||
| for line in body.decode().splitlines(): | ||
| if line.strip(): | ||
| yield json.loads(line) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Scale the config sweep (case x config grid -> recovery RMS) across a cluster. The work is deterministic and embarrassingly parallel, so it fans out over an array of tasks with no coordination.
What
src/codameter/bench.py— three subcommands:plan(size the job, no compute),sweep(score this shard's cells),aggregate(merge shard files + per-case best config). Grids (compact/multiverse/wide) are built per use case around its recommended config, so each case keeps a physical band/window.--shard k/Nround-robins the work (even load despite slow moving/inversion references);--jobsuses local vCPUs;--outwrites oneshard-<k>-of-<N>.jsonlto a local dir ors3://.AWS_BATCH_JOB_ARRAY_INDEX+CODAMETER_SHARDS, so one image runs every array task. Bad cells are caught per cell, never kill the shard.docker/— Dockerfile (ENTRYPOINT codameter-bench), an AWS Batch Fargate job definition, and a run guide.codameter-bench; optionalawsextra (boto3) for s3 output; pixi taskbench; README section.tests/test_bench.py— grid shapes, disjoint/covering shards, env shard parsing, cell scoring (recovery + error handling), sweep round-trip, best-config.Verification
plan: multiverse = 6480 cells (648/case) over 10 cases, balanced across 64 shards.plan+ a full in-containersweep).tests/test_bench.py: 7 passed.Depends on the golden/advisor layer merged in #11 (
golden.CASES,use_cases).🤖 Generated with Claude Code