diff --git a/.agents/skills/testing-sysml-repl/SKILL.md b/.agents/skills/testing-sysml-repl/SKILL.md index 6268636036..d4c67287d1 100644 --- a/.agents/skills/testing-sysml-repl/SKILL.md +++ b/.agents/skills/testing-sysml-repl/SKILL.md @@ -6309,6 +6309,36 @@ None for these local REPL/gRPC checks. - Validate saved Turtle independently with rdflib; check real triples and declarations rather than trusting the save confirmation. + +## Migration and state-rendering comparisons + +- Migrate real Cameo archives through `bin/sysml model.mdzip -migrate sysml + -o output.sysml -migration-report output.report.txt`; source XMI is normally + `com.nomagic.magicdraw.uml_model.model` inside the archive. +- Compare reports by element XMI ID, not line number or name: repeated names + can move records into the approximated section even when their notation was + successfully emitted. Preserve counts and inspect the notation as well. +- For direction checks, resolve `informationSource`, `informationTarget`, + `realizingConnector`, connector ends' `partWithPort`/`role`, and part/port + types from XMI. A conveyed item's emitted path should end in the actual + flow property of the connector's port type. Include a reversed connector + order fixture and an unrelated-end negative control. +- A compiler/lowering fix may repair an old migrated `.sysml` without + changing its state declarations. Comparing old and new notation under the + same new binary is not a negative control: use a separately built + pre-change binary to demonstrate the old empty graph. +- Use `-render '#state:M' -render-form text` for a small state definition, + and `-graphs M` to inspect inherited vertices, parent IDs and transitions. + Unsupported inherited members may produce an empty rendering plus a + `cannot be inherited` note while the CLI still exits zero. Exit status and + strict validation alone do not establish a nonempty graph. +- For positioned migrated views, default DOT may omit unpositioned inherited + nodes. Read `not represented` comments; use `-render-unplaced strip` to + include those nodes below the original drawing. Do not confuse omitted + placement with failed graph lowering. Named rendering may also print + validation warnings from other views; separate requested-view warnings + from whole-model warnings. + ### Devin Secrets Needed None for native local CLI/REPL tests. diff --git a/.circleci/config.yml b/.circleci/config.yml index a8548d60b1..f4375f7b34 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -23,6 +23,13 @@ executors: - image: cimg/python:3.11 resource_class: medium + # The -node variant of the same image: the Jupyter kernel package builds + # its JupyterLab extension with Node before it builds the wheels. + python-node-executor: + docker: + - image: cimg/python:3.11-node + resource_class: medium + rust-executor: docker: - image: cimg/rust:1.83 @@ -1682,7 +1689,7 @@ jobs: # platform bundles that platform's kernel, so `pip install` alone registers # it; the sdist carries none and downloads by the pins. build-jupyter-kernel-package: - executor: python-executor + executor: python-node-executor steps: - checkout - rehearsal-tag @@ -1691,9 +1698,11 @@ jobs: - attach_workspace: at: . + # jupyterlab provides `jupyter labextension build`, which bundles the + # extension every distribution ships (make jupyterlab-build). - run: name: Install build tooling - command: python -m pip install --upgrade pip build twine packaging + command: python -m pip install --upgrade pip build twine packaging jupyterlab==4.6.4 - run: name: Resolve the version from the tag @@ -1778,6 +1787,25 @@ jobs: # The kernelspec `install` would write names the binary it copies in. /tmp/verify-kernel-wheel/bin/python -m jupyter_opensysml_kernel kernelspec + # The same wheel, installed alone, put the prebuilt JupyterLab extension + # where JupyterLab 4 and Notebook 7 load federated extensions from. + - run: + name: Check the installed wheel registered the JupyterLab extension + command: | + /tmp/verify-kernel-wheel/bin/python - \<<'EOF' + import json + import os + import sys + + root = os.path.join(sys.prefix, "share", "jupyter", "labextensions", "jupyterlab-opensysml") + with open(os.path.join(root, "package.json")) as f: + manifest = json.load(f) + load = manifest["jupyterlab"]["_build"]["load"] + if not os.path.isfile(os.path.join(root, load)) or not os.path.isfile(os.path.join(root, "install.json")): + raise SystemExit(f"Error: {root} is not a loadable federated extension (load={load!r})") + print(f"jupyterlab-opensysml {manifest['version']} installed at {root}, loading {load}") + EOF + # A distribution whose table lacks any of the five kernel assets for the # tag, or pins a digest other than the one its binary hashes to, fails # the job here: every platform wheel and the sdist. The installed wheel diff --git a/.github/workflows/nightly.yml b/.github/workflows/nightly.yml index 5ca81ab0aa..88f9b865d9 100644 --- a/.github/workflows/nightly.yml +++ b/.github/workflows/nightly.yml @@ -332,9 +332,18 @@ jobs: - name: Build the Jupyter kernel sdist and platform wheels env: PYPI_VERSION: ${{ steps.version.outputs.pypi }} + NPM_VERSION: ${{ steps.version.outputs.npm }} run: | - # The sdist, then one wheel per platform bundling the kernel from dist/jupyter; - # the script opens each and checks what it carries. + # jupyterlab provides `jupyter labextension build`, which bundles the + # JupyterLab extension every distribution ships (make jupyterlab-build). + python -m pip install --only-binary ":all:" jupyterlab==4.6.4 + # `jupyter labextension list` reports the extension's manifest version, so the + # manifest is stamped with the snapshot's npm version first; `npm version` keeps + # the lock file in step, as `npm ci` requires. + (cd editors/jupyterlab && npm version --no-git-tag-version "$NPM_VERSION") + git diff --stat -- editors/jupyterlab/package.json editors/jupyterlab/package-lock.json + # The extension, the sdist, then one wheel per platform bundling the kernel + # from dist/jupyter; the script opens each and checks what it carries. "$KERNEL_DIST_SCRIPT" dist/jupyter client/jupyter-kernel/dist # The version is PEP 440 in canonical form and the build tools normalize the # name, so these are the files. @@ -344,6 +353,13 @@ jobs: echo "::error::expected five platform wheels for ${PYPI_VERSION}, found $#: $*" exit 1 fi + stamped=$(unzip -p "$1" '*.data/data/share/jupyter/labextensions/jupyterlab-opensysml/package.json' \ + | python -c 'import json, sys; print(json.load(sys.stdin)["version"])') + if [ "$stamped" != "$NPM_VERSION" ]; then + echo "::error::the labextension in $1 carries version '$stamped', not the '$NPM_VERSION' it is being published as." + exit 1 + fi + echo "ok: the labextension carries $stamped" python -m twine check --strict client/jupyter-kernel/dist/* # Like the client's wheel: this platform's wheel must import at the snapshot diff --git a/.github/workflows/pr.yml b/.github/workflows/pr.yml index cb74289213..5672191a35 100644 --- a/.github/workflows/pr.yml +++ b/.github/workflows/pr.yml @@ -55,6 +55,7 @@ jobs: julia: ${{ steps.areas.outputs.julia }} matlab: ${{ steps.areas.outputs.matlab }} vscode: ${{ steps.areas.outputs.vscode }} + jupyterlab: ${{ steps.areas.outputs.jupyterlab }} mdk: ${{ steps.areas.outputs.mdk }} syson: ${{ steps.areas.outputs.syson }} steps: @@ -77,10 +78,11 @@ jobs: needs: changes if: needs.changes.outputs.go == 'true' # The shards split the suite's slowest packages across runners so each gets its own job cap (see scripts/race-shard.sh). + # internal/exec/runtime alone outruns a job under -race, so it is two shards split by test name (see make test-shard). strategy: fail-fast: false matrix: - shard: [runtime, model, export, rest] + shard: [runtime, runtime-corpus, model, export, rest] runs-on: ubuntu-latest permissions: contents: read @@ -251,8 +253,8 @@ jobs: - name: Download the Reference-FMUs run: ./scripts/download-reference-fmus.sh - # Per-package timeout: under -race the runtime package runs 31-47 minutes on - # these runners. Matches `make test-shard`. + # Per-package timeout: under -race the whole runtime package ran 31-57 minutes + # on these runners, hence its two shards. Matches `make test-shard`. - name: Run Go race tests env: SHARD: ${{ matrix.shard }} @@ -272,8 +274,12 @@ jobs: permissions: contents: read timeout-minutes: 5 - # The shard package sets are disjoint, so their blocks concatenate under one mode line. + # The shards' blocks concatenate under one mode line; the two runtime shards instrument + # the same package, so their duplicate blocks are collapsed as make coverage does. steps: + - name: Checkout + uses: actions/checkout@v4 + - name: Download shard profiles uses: actions/download-artifact@v4 with: @@ -285,12 +291,12 @@ jobs: set -euo pipefail profiles=(shards/coverage-*/coverage.txt) # One per race-tests shard. - if [ "${#profiles[@]}" -ne 4 ]; then echo "error: expected 4 shard profiles, found ${#profiles[@]}" >&2; exit 1; fi + if [ "${#profiles[@]}" -ne 5 ]; then echo "error: expected 5 shard profiles, found ${#profiles[@]}" >&2; exit 1; fi for f in "${profiles[@]}"; do [ "$(head -n 1 "$f")" = "mode: atomic" ] || { echo "error: $f is not an atomic-mode profile" >&2; exit 1; } done { echo "mode: atomic"; for f in "${profiles[@]}"; do tail -n +2 "$f"; done; } > coverage.txt - wc -l coverage.txt + python3 scripts/dedupe-coverage.py coverage.txt - name: Upload coverage artifact uses: actions/upload-artifact@v4 @@ -797,6 +803,7 @@ jobs: - rust-test - static-and-integrity - vscode-extension + - jupyterlab-extension - wasm-gate - xpect-oracle runs-on: ubuntu-latest @@ -823,6 +830,7 @@ jobs: RUST_TEST_RESULT: ${{ needs.rust-test.result }} STATIC_AND_INTEGRITY_RESULT: ${{ needs.static-and-integrity.result }} VSCODE_EXTENSION_RESULT: ${{ needs.vscode-extension.result }} + JUPYTERLAB_EXTENSION_RESULT: ${{ needs.jupyterlab-extension.result }} WASM_GATE_RESULT: ${{ needs.wasm-gate.result }} XPECT_ORACLE_RESULT: ${{ needs.xpect-oracle.result }} run: | @@ -844,6 +852,7 @@ jobs: ! ok "$RUST_TEST_RESULT" || ! ok "$STATIC_AND_INTEGRITY_RESULT" || ! ok "$VSCODE_EXTENSION_RESULT" || + ! ok "$JUPYTERLAB_EXTENSION_RESULT" || ! ok "$WASM_GATE_RESULT" || ! ok "$XPECT_ORACLE_RESULT"; then echo "changes=$CHANGES_RESULT" @@ -863,6 +872,7 @@ jobs: echo "rust-test=$RUST_TEST_RESULT" echo "static-and-integrity=$STATIC_AND_INTEGRITY_RESULT" echo "vscode-extension=$VSCODE_EXTENSION_RESULT" + echo "jupyterlab-extension=$JUPYTERLAB_EXTENSION_RESULT" echo "wasm-gate=$WASM_GATE_RESULT" echo "xpect-oracle=$XPECT_ORACLE_RESULT" exit 1 @@ -978,6 +988,49 @@ jobs: path: editors/vscode/opensysml-sysml.vsix if-no-files-found: error + jupyterlab-extension: + name: JupyterLab extension + needs: changes + if: needs.changes.outputs.jupyterlab == 'true' + runs-on: ubuntu-latest + permissions: + contents: read + timeout-minutes: 10 + steps: + - name: Check out repository + uses: actions/checkout@v4 + + - name: Set up Node.js + uses: actions/setup-node@v4 + with: + node-version: '22' + cache: npm + cache-dependency-path: editors/jupyterlab/package-lock.json + + - name: Set up Python + uses: actions/setup-python@v5 + with: + python-version: '3.11' + + # jupyterlab provides `jupyter labextension build`, which bundles the + # extension the kernel package ships. The generated syntax table is + # gated by go test ./editors/... in the Go suite. + - name: Install JupyterLab + run: pip install --only-binary ":all:" jupyterlab==4.6.4 + + - name: Test the tokenizer + run: make jupyterlab-test + + - name: Build the prebuilt extension + run: make jupyterlab-build + + - name: Upload JupyterLab extension artifact + uses: actions/upload-artifact@v4 + with: + name: jupyterlab-extension + path: client/jupyter-kernel/labextension + if-no-files-found: error + mdk-plugin: name: OpenSysML MDK plugin needs: [changes, build] diff --git a/.gitignore b/.gitignore index 169ea37cd1..4188d3517a 100644 --- a/.gitignore +++ b/.gitignore @@ -70,3 +70,4 @@ cmd/*/rsrc_windows_*.syso # The kernel a platform wheel bundles, staged by the release pipeline (setup.py) /client/jupyter-kernel/jupyter_opensysml_kernel/bin/ +/client/jupyter-kernel/labextension/ diff --git a/CHANGELOG.md b/CHANGELOG.md index f057ad70bc..80126a28f5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,7 +8,6 @@ release is described in [docs/project/releasing.md](docs/project/releasing.md). ## Unreleased ## 0.10.0 — 2026-10-08 - ### Added - **The action view draws the action's own parameters as pins on its frame, and its parameter bindings as edges.** An `action def` or action usage with `in`, `inout`, `out` or `return` parameters — declared, or a usage's inherited from its type — has them as pins on the frame boundary, inputs on the input side and outputs on the output side, as the SysML v2 action-flow notation shows them; a nested parameter's value naming the action's parameter (`in b = bread;`, `in bread = ToastBread::bread;`, `out x :>> x = y;`), an explicit `bind pack.boxed = toast;` and a `bind heat.t = pack.t;` between two nodes' pins are binding edges between the pins, in `dot`, `text`, `mermaid` and `plantuml` and through the service's render surfaces. The lowered `ActionGraph` carries the action's `Parameters` and its pins' `ValueBindings` for every consumer. The text form now lists each node's pins under it and names the pins an edge joins; Mermaid and PlantUML say which pins they cannot draw in a `not represented` notice. @@ -258,6 +257,17 @@ release is described in [docs/project/releasing.md](docs/project/releasing.md). - **Imports that make a namespace's memberships indistinguishable are reported.** Two imports bringing members of one name or short name whose metaclasses conform — `import A::*; import B::*;` where each package owns a `part def Engine`, or one recursive `import V::**` reaching two nested `vb` — now warn once per name on the import bringing the later membership, naming every colliding member and the import it came through: `Duplicate of imported member name 'Engine': A::Engine (import A::*), B::Engine (import B::*)` (code `name-conflict`, beside the owned and inherited duplicate-name warnings). An imported member repeating one the type inherits warns the same way (`Base::x (inherited)`). Resolution is unchanged and still takes the first membership. One membership two imports both reach, an alias beside the element it names, an imported name an owned member hides, a pair whose metaclasses are unrelated and the standard library's own members are not reported. +- **SysML syntax highlighting in JupyterLab 4 and Notebook 7, shipped with the kernel's pip package.** `pip install jupyter-opensysml-kernel` now installs `jupyterlab-opensysml`, a prebuilt JupyterLab extension (`editors/jupyterlab`), as shared data under `share/jupyter/labextensions`, in every platform wheel and the sdist — no Node.js, no `jupyter labextension install`. It registers the `sysml` (`text/x-sysml`, `.sysml`) and `kerml` (`text/x-kerml`, `.kerml`) CodeMirror languages, so the cells of a `sysml` notebook, `.sysml`/`.kerml` files in the editor and Markdown fences tagged `sysml` are highlighted: keywords, contextual keywords, comments and `doc` bodies, strings, numbers, unrestricted and qualified names, operators, and a cell's leading `%command`. The keyword tables are generated from the lexer by the VS Code grammar generator (`make vscode-grammar`) and gated by a test, and the kernel now reports `language_info.codemirror_mode = "sysml"`. + +- **`sysml model.sysml -list documents|views|diagrams|pseudo-views|all` names what a model declares by the names `-render-document` and `-render` read.** Each document or view is listed by its qualified name as the notation writes it — quoted segments, `::` inside a segment and a line break escaped as `\n` included — so it can be passed back unchanged. `views` marks a view whose kind is not produced as unsupported with the reason; `diagrams` keeps the graph-shaped views (`tree`, `interconnection`, `state`, `action`, `case`, `mixed`, `sequence`, `requirement`, `definition` and `package`); `-list-kind state,action` filters by kind; `-list-form text|tsv|json` writes aligned text, tab-separated values with a header, or a JSON array carrying each item's kind, support, reason, file and 1-based line. The REPL's `%documents` and `%views [diagrams] [...]` print the same text form, and the workspace's `AllViews` lists the views of all its own documents, each naming its declaring document. + +- **`%load` reads Jupyter notebooks.** A path ending in `.ipynb` — named at the prompt, on the `sysml` command line, or found under a directory or pattern beside `.sysml` and `.kerml` files — loads the notebook's code cells in order, each a source of its own, so a diagnostic names `.ipynb cell 3` and its line in the cell, and re-loading the notebook redeclares it. A cell is loaded for what it declares: its `%` command lines and its expressions are skipped and the report says how many, so a loaded notebook runs no one's `%sweep` or `%save`. `--cells 1,3-5` or `--cells tag:` picks cells; a cell tagged `skip-load` is never loaded; a notebook written for another kernel language, or in a format other than nbformat 4, is refused with the reason. + +- **The REPL and the Jupyter kernel read and write a SysML v2 API repository with the OMG pilot kernel's four commands.** `%repo []` shows or sets the server (seeded from `FLEXO_SYSMLV2_URL`, the token from `FLEXO_INTEROP_TOKEN`, never printed); `%projects` lists every project, paged to the end; `%load [--id=] [--name=] [--branch=] []` reads a project's model at a branch head into the session as notation and remembers where it came from; `%publish [-d] [--project=] [--branch=] ` writes the elements rooted in a session element as a new project or as a commit of what changed on the branch, reporting the commit id and the counts created, updated and deleted; `-d` includes derived properties. Any server speaking the standard API serves — the pilot's `SysML-v2-API-Services` without a token, Flexo MMS with one — and the kernel answers argument problems as `UsageError`. +- **The standard SysML v2 API client is its own package.** `internal/translate/interop/sysmlapi` drives projects, branches, commits with `change` payloads, elements and roots with optional bearer authentication and follows a server's `Link: rel="next"` paging to the end; `internal/translate/interop/flexo` keeps Flexo's extras — Layer 1 Turtle, organizations, ETag pushes — over it, and `internal/translate/interop/modelsync` holds what `sysml -sync` and the REPL share. + +- `%viz [--view=] [--style=