From 33b5dc4014c377f5bca7f453d889e1881d6e7026 Mon Sep 17 00:00:00 2001
From: "Sebastian (Tiedtke) Huckleberry" <sebastiantiedtke@gmail.com>
Date: Mon, 13 Jul 2026 13:19:42 -0700
Subject: [PATCH] docs: document eval task scaffolding

Signed-off-by: Sebastian (Tiedtke) Huckleberry <sebastiantiedtke@gmail.com>
---
 docs/eval/quickstart.md | 30 ++++++++++++++++++++++++++++++
 docs/eval/reference.md  | 39 ++++++++++++++++++++++++++++++++++++++-
 2 files changed, 68 insertions(+), 1 deletion(-)

diff --git a/docs/eval/quickstart.md b/docs/eval/quickstart.md
index fa968e15..b209ed2c 100644
--- a/docs/eval/quickstart.md
+++ b/docs/eval/quickstart.md
@@ -90,6 +90,36 @@ OPENAI_API_KEY="$OPENAI_API_KEY" runme eval examples/harbor/datasets/runme-rewar
   --env docker
 ```
 
+## Create your own task scaffold
+
+After running the included examples, use `runme eval task new` to start a task for your own workflow. The scaffold command is part of the `runme` CLI. Running evals still requires the Harbor adapter above.
+
+Start a new task in the default eval dataset:
+
+```sh
+runme eval task new runmedev/my-task \
+  --description "Evaluate whether an agent can complete my task"
+```
+
+This writes `evals/tasks/my-task/` by default. Bare task names are also supported when you pass `--org`; Harbor uses it as the task namespace for datasets and published registry packages:
+
+```sh
+runme eval task new my-task --org runmedev
+```
+
+Before running the scaffold as a real eval:
+
+- Replace `instruction.md` with the task prompt.
+- Implement `tests/test.sh` so it writes a meaningful reward to `$RUNME_REWARD_PATH`.
+- Fill in `solution/solve.sh` if you want an oracle or reference solution.
+- Add Docker setup in `environment/Dockerfile` only when the task needs `--env docker`.
+
+Run the scaffold after editing:
+
+```sh
+runme eval --task-dir my-task --agent claude-code
+```
+
 ## View eval jobs
 
 Open the eval dashboard for the default jobs directory:
diff --git a/docs/eval/reference.md b/docs/eval/reference.md
index ec93ba6b..46d7bb44 100644
--- a/docs/eval/reference.md
+++ b/docs/eval/reference.md
@@ -5,7 +5,7 @@ title: Command reference
 
 # Task eval command reference
 
-`runme eval` is an experimental command group for running repeatable task datasets against local coding agents.
+`runme eval` is an experimental command group for running repeatable task datasets against local coding agents. `runme eval task new` helps authors create a starter task under the default dataset layout.
 
 It builds on [Harbor's task, dataset, trial, and job model](https://www.harborframework.com/docs), but Runme presents the workflow through the `runme eval` CLI.
 
@@ -82,6 +82,43 @@ Staging depends on the selected environment:
 
 Add `**/environment/workdir/` to `.gitignore` when Docker workdir staging is used. The staged copy is generated eval state, not source.
 
+## `runme eval task new`
+
+Create a Harbor eval task scaffold for Runme.
+
+```sh
+runme eval task new <org/name> [flags]
+```
+
+When `--tasks-dir` is omitted, Runme writes under `./evals/tasks`.
+
+Important flags:
+
+- `--tasks-dir`: eval tasks directory. Defaults to `./evals/tasks`.
+- `--org`: organization namespace for bare task names, such as `runme eval task new my-task --org runmedev`. Harbor uses it for dataset membership and published registry packages.
+- `--description`: task description written to `task.toml`.
+- `--author`: author in `Name <email>` or `Name` format. Can be repeated. When omitted, Runme uses `git config user.name` and `git config user.email` when available.
+- `--no-solution`: do not include `solution/solve.sh`.
+- `--force`: overwrite scaffold-owned files in an existing task directory.
+
+Generated files:
+
+- `README.md`: task-local run hint.
+- `task.toml`: Harbor task metadata and Runme-compatible environment defaults.
+- `instruction.md`: placeholder task instruction.
+- `environment/Dockerfile`: optional Docker environment starter.
+- `workdir/`: Git-ignored workspace for task-local files.
+- `tests/test.sh`: executable verifier stub that writes reward JSON.
+- `solution/solve.sh`: executable reference solution stub, unless `--no-solution` is passed.
+
+Behavior:
+
+- Qualified names use Harbor's `org/name` task identifier format and create a local directory named `name`.
+- Bare names require `--org`, which supplies the Harbor task namespace.
+- The scaffold maps the task workdir to `/app/<dataset>/<task>/workdir`.
+- Existing task directories require `--force`.
+- `--force` overwrites scaffold-owned files. Review hand-edited tasks before using it.
+
 ## `runme eval view`
 
 View eval jobs in the dashboard.