Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
58 changes: 43 additions & 15 deletions .github/workflows/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -214,11 +214,18 @@ docker run --gpus all motphys/motrixlab:test scripts/view.py env=cartpole

### File: `pypi-publish.yml`

Publishes the `motrix-env-core`, `motrix-deploy`, and `motrix-rl` distributions
(the dependency closure of the public packages) to a Python package index.
Publishes six distributions to a Python package index: `motrix-env-core`,
`motrix-deploy`, `motrix-rl`, `motrix-rl-builtin`, `motrix-env-motrixsim`, and
`motrixlab`. The `motrixlab` workspace member is an integration package with a
thin, lazy `mxlab` CLI, distinct from the development-only root project `motrix-lab`.
Its five commands are `train`, `play`, `export-onnx`, `export-deploy`, and `deploy`.
Command implementations remain in `motrix_rl.commands` and `motrix_deploy.cli`.
It requires Python `>=3.10,<3.12` and pins all five component distributions to its own version.
It does not include the full development workspace, built-in tasks/robot assets,
SKRL/RSLRL providers, or deployment backend plugins.

- **Release trigger**: publishing a GitHub Release whose `v`-prefixed tag matches the
workspace package version uploads all three packages to pypi.org.
workspace package version uploads all six packages to pypi.org.
- **Manual trigger**: `workflow_dispatch` uploads to TestPyPI by default, or to
pypi.org with `index=pypi`. Use a TestPyPI run to verify the full install
closure before creating the production release. A manual formal-PyPI run has
Expand All @@ -229,18 +236,39 @@ For production publication, the selected commit must be an ancestor of `stable`
and the matching immutable tag must point exactly at that commit. Every workspace
package must carry the same version before any distribution is built. A dedicated
build job uploads one wheel per package as a workflow artifact, and one publishing
job per package then uploads them in dependency order (env-core → deploy → rl).
Existing immutable distributions are skipped during recovery, so a partial
failure can be retried without replacing an already-published package.

Authentication uses PyPI Trusted Publishing (OIDC) — no API tokens are stored
in repository secrets. Before the first run, a maintainer must register a
pending publisher for each of the three project names on both pypi.org and
test.pypi.org, pointing at `Motphys/MotrixLab` with the workflow file
`pypi-publish.yml` and the matching job environment (`pypi-env-core`,
`pypi-deploy`, or `pypi-rl`). The distinct environments are required because
PyPI binds at most one project to a repository/workflow/environment
combination.
job per package then uploads them in dependency order:

```text
env-core → deploy → rl → [rl-builtin, env-motrixsim] → motrixlab
```

The two bracketed jobs can run in parallel; the integration package waits for both.
Existing immutable distributions are skipped with `skip-existing: true` during
recovery, so a partial failure can be retried without replacing an
already-published package. Publishing is restricted to `Motphys/MotrixLab`,
and only publishing jobs receive `id-token: write`.

Authentication uses PyPI Trusted Publishing (OIDC), without PyPI API-token
secrets. Before the first run, a maintainer must create the matching GitHub
environments and register Trusted Publishers separately on **both pypi.org and
test.pypi.org**, using owner `Motphys`, repository `MotrixLab`, and workflow
filename `pypi-publish.yml` (pending publishers for new projects):

| Project | GitHub environment |
| --- | --- |
| `motrix-env-core` | `pypi-env-core` |
| `motrix-deploy` | `pypi-deploy` |
| `motrix-rl` | `pypi-rl` |
| `motrix-rl-builtin` | `pypi-rl-builtin` |
| `motrix-env-motrixsim` | `pypi-env-motrixsim` |
| `motrixlab` | `pypi-motrixlab` |

The distinct environments are required because PyPI binds at most one project
to a repository/workflow/environment combination. These external registrations
and environment protections are not configured or verified by this repository
change. See the [release guide's clean installation verification](RELEASE.md#clean-installation-verification)
for testing all six TestPyPI wheels in a fresh Python 3.10/3.11 environment,
resolving third-party dependencies from public PyPI without mixing indexes.

# Release Pipeline

Expand Down
109 changes: 98 additions & 11 deletions .github/workflows/RELEASE.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ wheels. Docker publishing remains a separate manual operation.
| Workflow | Purpose | Trigger |
| --- | --- | --- |
| `release.yml` | Bump workspace versions, open release PRs, tag `stable`, create GitHub Releases, and open stable-patch back-merge PRs | Manual dispatch and pushes to `main`/`stable` |
| `pypi-publish.yml` | Build and publish wheels for `motrix-env-core`, `motrix-deploy`, and `motrix-rl` | GitHub Release publication or manual dispatch |
| `pypi-publish.yml` | Build and publish wheels for `motrix-env-core`, `motrix-deploy`, `motrix-rl`, `motrix-rl-builtin`, `motrix-env-motrixsim`, and `motrixlab` | GitHub Release publication or manual dispatch |
| `docker-build.yml` | Build and push the Docker image | Manual dispatch from `stable` |

**Stable branch patch release** is the preferred name for a patch cut from
Expand All @@ -25,15 +25,22 @@ not a separate version class.
- **Pull requests: read and write**
3. Store the App client ID as the `RELEASE_APP_CLIENT_ID` repository variable
and its private key as the `RELEASE_APP_PRIVATE_KEY` secret.
4. Register PyPI Trusted Publishers for all three publishable projects on both
pypi.org and test.pypi.org:
4. Create the six GitHub environments below and configure their protection
rules as appropriate. Register a Trusted Publisher for each project on both
**pypi.org and test.pypi.org** (use a pending publisher for a new project):
- `motrix-env-core` with the environment `pypi-env-core`
- `motrix-deploy` with the environment `pypi-deploy`
- `motrix-rl` with the environment `pypi-rl`

Use repository `Motphys/MotrixLab` and workflow `pypi-publish.yml`. The
distinct GitHub environments are required because PyPI binds at most one
project to a repository/workflow/environment combination.
- `motrix-rl-builtin` with the environment `pypi-rl-builtin`
- `motrix-env-motrixsim` with the environment `pypi-env-motrixsim`
- `motrixlab` with the environment `pypi-motrixlab`

Use owner `Motphys`, repository `MotrixLab`, and workflow filename
`pypi-publish.yml`. The distinct GitHub environments are required because
PyPI binds at most one project to a repository/workflow/environment
combination. These registrations and environment settings are external
maintainer actions; committing this workflow does not configure or verify
them. No PyPI API token is needed.
5. For Docker publishing, configure `DOCKER_USERNAME` and `DOCKER_PASSWORD`.

The workflows exchange the App credentials for a short-lived installation
Expand Down Expand Up @@ -257,13 +264,28 @@ verifies that:
- the release tag matches the workspace version;
- every workspace package has the same version.

Only wheels are built. The three public distributions are published in
dependency order:
Only wheels are built. The six public distributions are published in this
order; builtin and the MotrixSim backend can publish in parallel after RL,
and the integration package waits for both:

```text
motrix-env-core → motrix-deploy → motrix-rl
├─ motrix-rl-builtin ──────┐
└─ motrix-env-motrixsim ───┴─ motrixlab
```

`motrixlab` is the integration workspace member, distinct from the development-only
root project `motrix-lab`. It requires Python `>=3.10,<3.12` and depends on
`motrix-env-core`, `motrix-deploy`, `motrix-rl`, `motrix-rl-builtin`, and
`motrix-env-motrixsim`, each pinned to the same version as the integration package.
Its `motrixlab` import namespace provides a thin, lazy dispatcher for the sole `mxlab`
CLI, with `train`, `play`, `export-onnx`, `export-deploy`, and `deploy` commands.
Command implementations remain in `motrix_rl.commands` and `motrix_deploy.cli`.
Its installation scope does not include built-in tasks/robot assets, SKRL/RSLRL providers, or
deployment backend plugins; use the source workspace for those components.
MotrixSim and other third-party dependencies are resolved from public PyPI,
not built by this workflow.

Publishing uses `skip-existing: true`: if one package job succeeds and a later
job fails, re-running the failed path skips immutable distributions that are
already present and publishes only the missing package. It never replaces a
Expand All @@ -289,6 +311,71 @@ from `stable`, and the matching immutable `v*` tag must point exactly at that
commit. Normal production publishing should still occur through the reviewed
GitHub Release event.

### Clean installation verification

After all six TestPyPI jobs succeed, test outside the checkout in a fresh
Python 3.10 or 3.11 virtual environment. Set `VERSION` to the exact version
from that run (including any beta suffix):

```bash
VERSION=0.4.0b1 # Replace with the published version.
workdir="$(mktemp -d)"
cd "$workdir"
python3.10 -m venv .venv
. .venv/bin/activate
python -m pip install --upgrade pip
python -m pip download --no-deps --only-binary=:all: \
--index-url https://test.pypi.org/simple/ --dest wheels \
"motrix-env-core==$VERSION" "motrix-deploy==$VERSION" \
"motrix-rl==$VERSION" "motrix-rl-builtin==$VERSION" \
"motrix-env-motrixsim==$VERSION" "motrixlab==$VERSION"
python -m pip install --index-url https://pypi.org/simple/ wheels/*.whl
python -m pip check
python - <<'PY'
from importlib.metadata import version

names = (
"motrix-env-core", "motrix-deploy", "motrix-rl",
"motrix-rl-builtin", "motrix-env-motrixsim", "motrixlab",
)
versions = {name: version(name) for name in names}
assert len(set(versions.values())) == 1, versions
print(versions)
import motrix_env_core
import motrix_deploy
import motrix_rl
import motrix_rl_builtin
import motrix_env_motrixsim
from motrix_rl.frameworks import get_framework
from motrix_env_core.sim.registry import list_sim_backends, default_sim_backend_name

# The builtin entry point registers the framework under its public name.
assert get_framework("motrix").name == "motrix"
assert "motrixsim" in list_sim_backends()
assert default_sim_backend_name() == "motrixsim"
PY
mxlab --help
mxlab train --help
mxlab play --help
mxlab export-onnx --help
mxlab export-deploy --help
mxlab deploy --help
```

Downloading the six exact wheels with `--no-deps` ensures these projects come
from TestPyPI; the subsequent installation resolves third-party dependencies
from PyPI. Do not use `--extra-index-url` to mix the two indexes, or an older
production package may mask a missing test artifact. This is an actual install
and import check, not a GPU simulation or training test. Record the platform,
Python version, commands, and results; repeat with Python 3.11 for both supported
interpreters.

After production publication, use another fresh environment outside the
checkout and install `"motrixlab==$VERSION"` with
`python -m pip install --index-url https://pypi.org/simple/`, then repeat
`pip check` and the metadata/import checks above. Neither editable workspace
installs nor `--no-deps` installs validate the published dependency closure.

## Automation and human gates

| Stage | Automation |
Expand Down Expand Up @@ -337,8 +424,8 @@ Before merging anything into `stable`:
After publication:

- [ ] Verify the GitHub Release and immutable tag.
- [ ] Verify all three Python publishing jobs succeeded.
- [ ] Install or inspect the published wheels.
- [ ] Verify all six Python publishing jobs succeeded.
- [ ] Run the clean published-wheel installation verification above.
- [ ] Run Docker publishing manually if this release needs a new image.
- [ ] For a stable branch patch release, review and merge the automatic
`stable → main` back-merge PR.
74 changes: 73 additions & 1 deletion .github/workflows/pypi-publish.yml
Original file line number Diff line number Diff line change
Expand Up @@ -96,6 +96,9 @@ jobs:
uv build --wheel motrix_env_core --out-dir dist/env-core
uv build --wheel motrix_deploy --out-dir dist/deploy
uv build --wheel motrix_rl --out-dir dist/rl
uv build --wheel motrix_rl_builtin --out-dir dist/rl-builtin
uv build --wheel motrix_env_motrixsim --out-dir dist/env-motrixsim
uv build --wheel motrixlab --out-dir dist/motrixlab

- name: Upload distribution packages
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
Expand All @@ -109,7 +112,7 @@ jobs:
needs: build
if: github.repository == 'Motphys/MotrixLab'
runs-on: ubuntu-latest
# Distinct GitHub environments keep the three Trusted Publisher registrations
# Distinct GitHub environments keep the six Trusted Publisher registrations
# unique: PyPI binds at most one project to a repository/workflow/environment.
environment: pypi-env-core
timeout-minutes: 10
Expand Down Expand Up @@ -174,3 +177,72 @@ jobs:
packages-dir: dist/rl
repository-url: ${{ (github.event_name == 'release' || inputs.index == 'pypi') && env.PYPI_URL || env.TESTPYPI_URL }}
skip-existing: true

publish-rl-builtin:
name: Publish motrix-rl-builtin
needs: publish-rl
if: github.repository == 'Motphys/MotrixLab'
runs-on: ubuntu-latest
environment: pypi-rl-builtin
timeout-minutes: 10
permissions:
contents: read
id-token: write # PyPI Trusted Publishing OIDC token.
steps:
- name: Download distribution packages
uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
with:
name: release-dists
path: dist/
- name: Publish to the target index
uses: pypa/gh-action-pypi-publish@dc37677b2e1c63e2034f94d8a5b11f265b73ba33 # v1.14.2
with:
packages-dir: dist/rl-builtin
repository-url: ${{ (github.event_name == 'release' || inputs.index == 'pypi') && env.PYPI_URL || env.TESTPYPI_URL }}
skip-existing: true

publish-env-motrixsim:
name: Publish motrix-env-motrixsim
needs: publish-rl
if: github.repository == 'Motphys/MotrixLab'
runs-on: ubuntu-latest
environment: pypi-env-motrixsim
timeout-minutes: 10
permissions:
contents: read
id-token: write # PyPI Trusted Publishing OIDC token.
steps:
- name: Download distribution packages
uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
with:
name: release-dists
path: dist/
- name: Publish to the target index
uses: pypa/gh-action-pypi-publish@dc37677b2e1c63e2034f94d8a5b11f265b73ba33 # v1.14.2
with:
packages-dir: dist/env-motrixsim
repository-url: ${{ (github.event_name == 'release' || inputs.index == 'pypi') && env.PYPI_URL || env.TESTPYPI_URL }}
skip-existing: true

publish-motrixlab:
name: Publish motrixlab
needs: [publish-rl-builtin, publish-env-motrixsim]
if: github.repository == 'Motphys/MotrixLab'
runs-on: ubuntu-latest
environment: pypi-motrixlab
timeout-minutes: 10
permissions:
contents: read
id-token: write # PyPI Trusted Publishing OIDC token.
steps:
- name: Download distribution packages
uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
with:
name: release-dists
path: dist/
- name: Publish to the target index
uses: pypa/gh-action-pypi-publish@dc37677b2e1c63e2034f94d8a5b11f265b73ba33 # v1.14.2
with:
packages-dir: dist/motrixlab
repository-url: ${{ (github.event_name == 'release' || inputs.index == 'pypi') && env.PYPI_URL || env.TESTPYPI_URL }}
skip-existing: true
Loading
Loading