diff --git a/PhyAgentOS/skills/forge-skill/robodojo/README.md b/PhyAgentOS/skills/forge-skill/robodojo/README.md new file mode 100644 index 000000000..dd3a4a09f --- /dev/null +++ b/PhyAgentOS/skills/forge-skill/robodojo/README.md @@ -0,0 +1,152 @@ +# robodojo Skill + +`robodojo` is a Forge Skill with one official profile, `g05`. It runs the RoboDojo +`put_bottles_into_dustbin` task through Isaac Sim and exposes two Gateway tools to +PAOS: `bench.describe` and `bench.run`. + +This Skill ships the description, manifest, g05 profile, dataflow and install +material only. The benchmark node and the G05 policy node are Forge Nodes whose +source lives in `framework/forge_runtime`; the profile reaches them through the +installed Node entrypoints under `${FORGE_RUNTIME_BIN}`. + +## Layout + +```text +skill.yaml manifest (manifest_version 2, profile g05) +SKILL.md Skill document referenced by skill_document +profiles/g05/dataflow.yaml gateway -> bench_endpoint -> robodojo_benchmark -> robodojo_policy +profiles/g05/*.yaml gateway / bench_endpoint / benchmark config +profiles/g05/env.template.sh host environment template +scripts/install_robodojo.py per-run TOS presign + `paos skill install --index` +scripts/check_bundle_inventory.py verify the template inventory against this tree +docs/ unsigned schema-v3 index template + 0.1.0 delivery record +``` + +There is no `nodes/` tree and no `profiles/g05/bin/` launcher here. An earlier +revision of this branch vendored the benchmark/policy sources and two in-bundle shell +launchers inside the Skill, which duplicated the Node sources and let the two copies +drift (four files had already diverged through lint-only edits). The sources of +truth are: + +| node | source of truth | build entry | +| --- | --- | --- | +| benchmark + G05 policy | `framework/forge_runtime`, `packages/nodes/benchmarks/robodojo/` | `scripts/build_node_artifacts.py` (`robodojo_benchmark`) | +| gateway | `framework/forge_gateway` tag `v1.0.2` | same script (`forge_runtime`) | +| bench endpoint | `framework/forge_runtime`, `packages/runtime/bench_endpoint/` | same script (`robodojo_endpoint`) | + +## Node bundles + +| node_id | artifact_id | entrypoints | node_digest | +| --- | --- | --- | --- | +| `forge_runtime` | `forge-runtime-0.1.0` | `gateway` | `cee927eb587f3f958c1803086e59692f6f7ae373477b3cfd15118d01d76c408e` | +| `robodojo_endpoint` | `robodojo-endpoint-0.1.0` | `bench_endpoint` | `b9a66fb07b04ee1f76bb8221f68f96b35d1f1ce136d3d6cf967cb7a34da8e159` | +| `robodojo_benchmark` | `robodojo-benchmark-0.1.0` | `robodojo_benchmark`, `robodojo_policy` | `e48cf4e838ba73512ac1eb0c144ea17ea177bb56ad41e524c8332f9a38c19ff2` | + +The benchmark node and the G05 policy node ship as **one** digest bundle with two +entrypoints: they share `_core/`, `config.py` and `adapters/` and run in the same +Python environment, so a second archive would only duplicate the same tree. + +## Source / artifact correspondence + +- The published `0.1.0` TOS objects (Skill archive sha256 `5c7806843e5cff16a8a407254ffe7736c9b1d5d13982bd1b82195c7e4f5bc907` + plus the `forge-runtime` / `robodojo-endpoint` Node archives) are frozen snapshots + of the **pre-split** layout: that Skill archive still contains `nodes/` and + `profiles/g05/bin/`. They stay untouched. +- This branch is the **post-split** candidate. The Skill archive is rebuilt with the + official packager (`scripts/package_skill.py`) and needs a new object key or version + before it can be published. Its sha256 is recorded in the pull request description: + a file cannot carry the hash of the archive it is packed into. +- Node implementations come from `framework/forge_runtime`. The recorded base is branch + `qinhan/robodojo-g05-node` at `1630e75a450464307cdab08de6e7d4536d04af86`; this revision adds + `93c2659` (lint-only port that makes the Node source byte-identical to the copy that had been + vendored in this Skill) and a `scripts/build_node_artifacts.py` fix that first sorts the + payload inventory by POSIX path, so one source tree yields one `node_digest` on Windows and on + Linux alike. +- That branch (`qinhan/robodojo-node-bundles`) is committed locally only. + `framework/forge_runtime` has no personal fork and the GitLab project cannot be forked without + an API session, so pushing it — as a topic branch or as a fork — is a maintainer decision + under the fork + MR contribution flow. +- `node_digest` is content-derived and reproducible across hosts; the Node *archive* sha256 is + not, because the tarball records file mtimes and ownership. + +## Runtime requirement + +`skill.yaml` uses two manifest features that must be present in the PAOS runtime: + +- `profiles..startup_timeout_s` — a per-profile Dora readiness timeout + (the generic LIBERO `lingbot_va` profile uses `900`). +- digest-locked multi-file Node bundles — `artifacts.nodes..digest` pins the + installed `node-manifest.json` digest, and the installed Node manifest declares the + `entrypoints` that provide `gateway`, `bench_endpoint`, `robodojo_benchmark` and + `robodojo_policy`. + +`profiles..startup_timeout_s` is already supported by upstream `dev`. +Digest-locked bundles come from `feature/skill-runtime-node-bundles` (PR #112); this +branch depends on it. The legacy single-executable `artifact_type: executable_tar_gz` +lock form keeps working unchanged. + +## Build + +```bash +python scripts/package_skill.py PhyAgentOS/skills/forge-skill/robodojo \ + --output-dir dist/skills +``` + +The Node bundles are built from `framework/forge_runtime`: + +```bash +python scripts/build_node_artifacts.py --out dist/nodes --forge-src /path/to/forge_gateway +``` + +## Install + +Requires read permission for TOS bucket `phyagentos-resource-inner` and `tosutil`. + +```bash +export PAOS_CONFIG=/abs/instance/config.json +python3 scripts/install_robodojo.py --tosutil /path/to/tosutil --vp 1d +``` + +`install_robodojo.py` regenerates fresh presigned download URLs on every run, verifies +each downloaded object against the sha256 recorded in the template, writes a private +temporary schema-v3 index, and calls: + +```bash +paos skill install robodojo --version 0.1.0 --index +``` + +The historical `paos-forge-packages.0.1.0.yaml` carries expired links and is not the +recommended entry point. + +## Verify, start, stop + +```bash +paos skill list +paos skill inspect robodojo +paos forge-node verify robodojo forge_runtime +paos forge-node verify robodojo robodojo_endpoint +paos forge-node verify robodojo robodojo_benchmark + +paos skill start robodojo --profile g05 +paos skill status robodojo +paos skill stop robodojo +``` + +`paos skill start` needs `dora` on `PATH`. A running Skill leaves Dora up after the +benchmark finishes; the pending item is that an Agent must actually stop and confirm +when the user asks. + +## External environment + +The manifest declares `FORGE_GATEWAY_PYTHON`, `BENCH_ENDPOINT_PYTHON`, `ROBODOJO_ENV`, +`ROBODOJO_ROOT`, `XPOLICYLAB_ROOT`, `G05_CKPT_PATH`, `G05_PROCESSOR_PATH`, `CUDA_HOME`, +`CUDA_VISIBLE_DEVICES`, `ROBODOJO_CACHE_ROOT`, `ROBODOJO_DEVICE_ID` and +`ROBODOJO_RESULT_DIR`. `profiles/g05/env.template.sh` lists the same values with +descriptions. + +`TASK_NAME=put_bottles_into_dustbin` is also exported by `env.template.sh` and read by +`policy_node.py`; it is not part of the frozen `skill.yaml` `required_environment` list, +so the host must set it explicitly until a new candidate manifest is published. + +Checkpoint/processor and Isaac Sim are external and are not bundled here. The G05 +checkpoint is ~33.9 GB; do not copy it into the Skill package. diff --git a/PhyAgentOS/skills/forge-skill/robodojo/SKILL.md b/PhyAgentOS/skills/forge-skill/robodojo/SKILL.md new file mode 100644 index 000000000..c9a164b08 --- /dev/null +++ b/PhyAgentOS/skills/forge-skill/robodojo/SKILL.md @@ -0,0 +1,144 @@ +--- +name: robodojo +description: Run the RoboDojo G05 benchmark and report its aggregate score. +metadata: {"PhyAgentOS":{"always":false,"requires":{"runtime":["robodojo"]}}} +--- + +# RoboDojo Benchmark + +Run a RoboDojo benchmark batch in Isaac Sim and report the score the benchmark +produced. Use only these stable Tool IDs: + +- `bench.describe`: Query; return the live task inventory, defaults, and episode + budget limits. Does not touch the simulator state. +- `bench.run`: Action; run one batch of episodes and return its aggregate result. + Long running — minutes per episode. + +Use the PAOS bridge tools `forge_tool_context`, `forge_tool_query`, +`forge_tool_start_action`, `forge_tool_action_status`, `forge_tool_action_result`, +and `forge_tool_cancel_action`. Always pass the stable Forge Tool ID explicitly. +Do not use shell commands and do not construct Gateway HTTP requests directly. + +## What this Skill must not do + +Scoring belongs entirely to the benchmark node. This Skill selects and sequences +Tools, and reports what came back. It must not: + +- compute, recompute, adjust, average, or extrapolate `success_rate` or any other + metric; +- decide whether an individual episode succeeded, failed, or timed out; +- aggregate results across separate `bench.run` invocations into one number; +- convert a partial or failed batch into a score, or present one as if the batch + had completed; +- estimate a score when the result is unavailable. + +If a number is needed and no terminal result exists, say so. Do not produce one. + +## 1. Always describe before running + +Call `bench.describe` first, every time. Never fill `bench.run` arguments from +memory or from an earlier conversation — the installed suite, task count, and +limits are properties of the running profile, not of this document. + +`bench.describe` returns: + +- `benchmark`, `suite`, `policy_id`, `env_cfg_type`, `action_type` — fixed by the + profile. They are read-only context, not parameters you can change. +- `tasks[]` — each with `task_id`, `name`, `language`, `num_init_states`. The + init-state count differs per task; do not assume one task's count applies to + another. +- `defaults` — `num_runs`, `seed`, `max_steps` to use when the user gave no value. +- `limits` — `max_total_episodes`, `max_max_steps`. Hard caps; see §2. + +If `bench.describe` fails with `BENCH_NOT_READY`, the simulator is still starting. +Wait and retry; do not proceed to `bench.run`. + +## 2. Build the bench.run arguments + +All five fields are required. There are no defaults on the wire. + +- `task_ids`: an array of ids taken from `bench.describe`, or `null` for every + task. `null` and `[]` are different — `[]` is always an error, never "all". +- `init_state_ids`: an array, or `null` for every init state shared by all + selected tasks. Validated per task. +- `num_runs`: repeats per (task, init_state) pair. Use `defaults.num_runs` unless + the user asked for more. +- `seed`: base seed. The same seed and the same ids reproduce the same episodes. + Change it only when the user asks for a different sample. +- `max_steps`: a **hard cap on control steps**. The episode is terminated as + `timed_out` if this many steps are reached before the environment ends. The + task native `step_lim` defines its normal end (success / partial / failed); set `max_steps` to at least the task horizon (from `bench.describe` defaults) to allow completion, or lower to force an explicit step-limit stop. + +Episode count is `len(task_ids) x len(init_state_ids) x num_runs`. Keep it at or +below `limits.max_total_episodes`; exceeding it is rejected, not clamped. When the +user asks for something larger, report the cap and the requested count and ask +before splitting the work into several batches. + +Nothing else is selectable. Policy weights, suite, camera set, and GPU are fixed by +the profile. If the user asks to change one of those, say it requires a different +profile — do not try to express it through these arguments. + +## 3. Account for the Action lifecycle + +`bench.run` is an Action. Accepted is not completed. + +1. `forge_tool_start_action` returns an acknowledgement with an `invocation_id` and + `total_episodes`. The batch has not run yet. +2. `forge_tool_action_status` reports `accepted`, `running`, or a terminal phase. + Status is **advisory** — never report a score from it. Poll at a human pace; + a single episode takes minutes. +3. `forge_tool_action_result` is the only authoritative source of the outcome. + +Treat the four terminal outcomes as four different things: + +- `succeeded` — the batch ran to completion. Report per §4. +- `failed` — the batch ended abnormally. Report the error code and the episode + counts; do not present `success_rate` as the headline. +- `cancelled` — stopped on request. Partial counts only. +- `unknown` — **the outcome cannot be recovered**. Side effects have already + happened: GPU time was consumed and a `result.json` may exist on disk. Do not + blindly retry. Report that the outcome is unknown, give the `invocation_id`, and + ask how to proceed. + +Use `forge_tool_cancel_action` only when the user asks to stop. Cancel is accepted, +not immediate — keep polling until a terminal result appears. + +## 4. Reporting rules + +Report these together, always, in this order: + +1. `status` of the batch. +2. `success_rate`, immediately followed by `successes` / `total_episodes`. +3. `completed_episodes` / `total_episodes`. + +`success_rate` is computed over `total_episodes`, not over +`completed_episodes`. When a batch ends early these differ, and `success_rate` +alone reads as "the policy scored low" when the real story is "the batch did not +finish". If `completed_episodes < total_episodes`, say that first, before the +score. + +Also report: + +- `result_path` — the authoritative record on disk. Always include it. +- `timed_out_episodes` when non-zero. +- `obs_republished` when non-zero: observations were re-sent during the run. The + score is still valid, but mention it. +- `episodes[]` per-episode outcomes when the user asks for detail. When + `episodes_truncated` is true the list was too large to inline — point at + `result_path` instead of summarising from an empty array. + +Never present a single number without its denominator. + +## 5. Errors and replanning + +| code | retryable | what to do | +|---|---|---| +| `BENCH_NOT_READY` | yes | Simulator still starting. Wait, retry `bench.describe`. | +| `FORGE_BUSY` | yes | Another batch is running; one simulator per machine. Report which `invocation_id` holds it. Do not queue. | +| `BENCH_INVALID_ARGUMENT` | no | Arguments are wrong. Re-read `bench.describe` and rebuild them. `details` carries the valid range and the requested vs allowed episode count. Never retry unchanged. | +| `BENCH_NODE_UNAVAILABLE` | yes | The benchmark node stopped reporting. The outcome is unknown — see §3. Do not restart the batch without asking. | +| `BENCH_UNKNOWN_INVOCATION` | no | The id is not known here. Do not invent one; start a new run if the user wants one. | +| `FORGE_DEADLINE_EXCEEDED` | no | The gateway stopped tracking; the batch may still be running. Do not start another — that would collide. Report and ask. | + +When two errors are plausible, prefer the more conservative reading: an unknown +outcome is not a failure, and a rejected batch is not a zero score. diff --git a/PhyAgentOS/skills/forge-skill/robodojo/docs/DELIVERY_0.1.0.md b/PhyAgentOS/skills/forge-skill/robodojo/docs/DELIVERY_0.1.0.md new file mode 100644 index 000000000..3b29b2de0 --- /dev/null +++ b/PhyAgentOS/skills/forge-skill/robodojo/docs/DELIVERY_0.1.0.md @@ -0,0 +1,173 @@ +# RoboDojo 0.1.0 — Installation (release candidate) + +**Status**: candidate, post-split layout. The schema-v3 static-index route is verified +(fresh-instance install on host 120 passed for the pre-split archive; a fresh isolated +instance install of the post-split layout was also checked locally without GPU). No HTTP +Resource Registry is used; no default Skill version changed. Final functional acceptance +is pending. + +## What changed: node ownership + +This revision removes the vendored benchmark/policy sources and the two in-bundle shell +launchers from the Skill. The nodes are maintained in `framework/forge_runtime` and are +installed as digest Node bundles; the profile reaches them through +`${FORGE_RUNTIME_BIN}/`. + +- The published pre-split objects stay untouched: Skill `robodojo-0.1.0.tar.gz` sha256 + `5c7806843e5cff16a8a407254ffe7736c9b1d5d13982bd1b82195c7e4f5bc907`, Node + `forge-runtime-0.1.0`, Node `robodojo-endpoint-0.1.0`. +- This revision adds a third Node bundle, `robodojo-benchmark-0.1.0` (node_digest + `e48cf4e838ba73512ac1eb0c144ea17ea177bb56ad41e524c8332f9a38c19ff2`, entrypoints + `robodojo_benchmark` and `robodojo_policy`). It is **not uploaded yet**. +- Because the Skill payload changed, its archive must be rebuilt and published under a new + object key or version before this revision can be installed from TOS. Versioning and + publishing are maintainer decisions; this revision overwrote nothing in TOS and did not + register anything in a public index. + +## Recommended entry point (credential-enabled, per-run presign) +Use the **unsigned template + refresh helper** — NOT the historical index that carries +pre-signed download URLs. The helper presigns fresh HTTPS download links on the target machine +on every run, so it never depends on a link lifetime. The signed index is written to a private +temp dir and is not tracked in git. This per-run presign route is **established**; a durable +HTTP(S)-hosted index / HTTP Resource Registry is **not a blocker** for this route. + +## Verified baseline +- Skill `robodojo-0.1.0.tar.gz` sha256 `5c7806843e5cff16a8a407254ffe7736c9b1d5d13982bd1b82195c7e4f5bc907` + (pre-split archive; the post-split candidate is rebuilt locally and needs a new object key) +- Node `forge-runtime-0.1.0` node_digest `cee927eb587f3f958c1803086e59692f6f7ae373477b3cfd15118d01d76c408e` (entry `bin/gateway`, 60 files) +- Node `robodojo-endpoint-0.1.0` node_digest `b9a66fb07b04ee1f76bb8221f68f96b35d1f1ce136d3d6cf967cb7a34da8e159` (entry `bin/bench_endpoint`, 27 files) +- Node `robodojo-benchmark-0.1.0` node_digest `e48cf4e838ba73512ac1eb0c144ea17ea177bb56ad41e524c8332f9a38c19ff2` (entries `bin/robodojo_benchmark`, `bin/robodojo_policy`, 20 files) — candidate, not uploaded +- Runtime source commit `aac600574a6d9dba225216e1f2271e069be2e4e2` plus the lint-port commit that makes the Node source byte-identical to the copy previously vendored in the Skill +- TOS bucket `phyagentos-resource-inner`; Skill and Node bundles live under + `skill-bundles/robodojo/0.1.0/` and + `node-bundles/{forge_runtime,robodojo_endpoint,robodojo_benchmark}/0.1.0/`. + +The `node_digest` values above are reproducible from the source trees on any host (the builder +sorts the payload inventory by POSIX path, so a Windows checkout cannot produce a different +digest than an LF checkout); the *archive* sha256 is not, because the tarball records file +mtimes and ownership. Pin and compare `node_digest`. + +## Post-split local verification (no GPU, this revision) + +Node sources come from `framework/forge_runtime` and are built with +`scripts/build_node_artifacts.py` (branch `qinhan/robodojo-node-bundles`: base `1630e75`, lint +port `93c2659`, plus the inventory-ordering fix). The Skill is packed with the official +`scripts/package_skill.py`. On a Linux host, with no GPU, no Isaac Sim, no Dora and no Agent: + +- Fresh isolated instance, verified empty of Skill / Node / lock / cache before the run. +- Installed with `install_robodojo.py --index ` served over loopback; the + loopback server replaced TOS presigning because the rebuilt archives are still candidates. + The run exercised the `prepare_instance` fix (`config.json` + `py/sitecustomize.py`), the + digest bundle install path and the lock write, and ended with + `✓ Installed Skill robodojo 0.1.0`. +- `paos skill list` reports `robodojo 0.1.0 / g05 / not started`. +- `paos forge-node verify robodojo {forge_runtime,robodojo_endpoint,robodojo_benchmark}` all + report "verified against Skill lock". +- The profile environment materialises `bin/{gateway,bench_endpoint,robodojo_benchmark,robodojo_policy}` + as relative symlinks into `/nodes//versions//bin/...`, and the + rendered `launch/profiles/g05/dataflow.yaml` resolves every node `path` to that expanded + `${FORGE_RUNTIME_BIN}` directory — never to a copy inside the Skill tree. The installed Skill + contains no `nodes/` and no `profiles/g05/bin/`. +- `python3 scripts/check_bundle_inventory.py --check` reports every Skill file matching the + template inventory. The inventory is not read at install time, so it drifts silently when a + Skill file changes; run the checker (or `--write`) on the pack host before registering or + publishing this candidate. +- Still pending: a presign run against real TOS for this revision, and GPU / Isaac Sim / + Agent functional acceptance. + +## Prerequisites +- Linux x86_64, CUDA GPU (>=16 GB for G05), Isaac Sim host, EGL/GL (`MUJOCO_GL=egl`, + `PYOPENGL_PLATFORM=egl`), `ffmpeg`, `dora` coordinator/daemon. +- PAOS runtime: `startup_timeout_s` is already supported by upstream `dev`. Digest-locked + multi-file Node bundle support is provided by `feature/skill-runtime-node-bundles` (PR #112), + which this Skill revision depends on. `paos skill install --index` supports the schema-v3 + static index. `forge-node install ` needs an HTTP Resource Registry URL and is + **not** configured — use the `--index` route. +- **TOS read permission for `phyagentos-resource-inner`** plus `tosutil` (this is a + credential-required install, not a public credential-less one). If the target machine cannot + obtain this permission, "download-authorization method missing" is a delivery blocker. + +## Install helper (regenerates download links on each run) +The template `paos-forge-packages.template.yaml` records a `bucket_key` and an +`expected_sha256` per package. `install_robodojo.py` presigns a fresh HTTPS link for each key +(via `tosutil presign`), downloads each object once to measure `size` and confirm +`expected_sha256`, writes a temporary schema-v3 index with `direct_download_url` / `sha256` / +`size` filled in (`paos skill install --index` rejects an index without them), and then runs the +verified install command. + +The helper is **self-contained**: it finds the template next to itself (or in the repo layout +`