Skip to content

feat(migration): sync plugin for the legacy dfx 0.32.0 assetstorage canister#71

Draft
lwshang wants to merge 3 commits into
mainfrom
migration
Draft

feat(migration): sync plugin for the legacy dfx 0.32.0 assetstorage canister#71
lwshang wants to merge 3 commits into
mainfrom
migration

Conversation

@lwshang

@lwshang lwshang commented Jun 2, 2026

Copy link
Copy Markdown
Collaborator

⚠️ This branch is never merged into main

The migration branch is a permanently separate line of work. Do not merge it.

Instead, we will git-tag commits on this branch and cut special GitHub releases that host the built sync-plugin wasm module. icp-cli projects (recipes, templates) reference that hosted wasm directly.

The driving use case: a companion assets recipe that swaps the sync mechanism from the old type: assets to type: plugin while keeping the canister wasm unchanged (the legacy dfx 0.32.0 assetstorage canister). This minimizes the upgrade effort for existing assets-recipe users when we drop type: assets sync in the upcoming icp-cli release.

What this branch does

Retargets sync-core / sync-plugin from this repo's V2 certified-assets canister to the legacy assetstorage canister shipped with dfx 0.32.0 (ic-certified-assets, which reports api_version == 2). Per-asset metadata is read from .ic-assets.json5 (full ic-asset parity) instead of the _headers / _redirects files used on main.

The legacy canister's wire protocol is nearly identical to this repo's V2 canister; the differences are narrow and handled here:

Concern main (new) legacy (0.32.0)
Redirect rules get_redirect_rules / SetRedirectRules absent (removed)
CreateAssetArguments adds enable_aliasing
SetAssetPropertiesArguments adds is_aliased
AssetProperties adds is_aliased
Per-asset config _headers + _redirects .ic-assets.json5
HTML routing synthesised redirect rules native enable_aliasing

sync-core

  • new config.rs.ic-assets.json5 / .ic-assets.json parser ported from ic-asset: nested config inheritance, glob match, ignore, headers, cache.max_age, enable_aliasing, allow_raw_access, encodings, security_policy.
  • new security_policy.rsstandard / hardened / disabled CSP presets.
  • scan.rs rewritten as gather_asset_descriptors (walkdir + config-driven ignore; .well-known kept; config files excluded).
  • canister.rs — add enable_aliasing / is_aliased; drop RedirectRule / RulePattern / SetRedirectRules / get_redirect_rules.
  • sync.rs — drive CreateAsset / SetAssetProperties from AssetConfig; drop redirect & html-handling; keep chunk packing + commit staging.
  • delete redirects.rs, headers.rs, html_handling.rs, glob.rs.
  • idempotency fix: the legacy canister injects a Set-Cookie: ic_env=... header into every HTML asset (driven by icp-cli's env-var step) and returns it from get_asset_properties. update_properties now ignores that canister-managed cookie so repeat syncs report "up to date" instead of clobbering it every run.

e2e

  • build.rs fetches the real assetstorage.wasm.gz from the dfx 0.32.0 release (sha256-pinned to the GH release digest, installed gzipped — no decompression) instead of building this repo's canister.
  • fixtures/tests converted to .ic-assets.json5; redirect / header / multi-dir cases removed; new config.rs end-to-end test (headers, max_age, aliasing, security_policy CSP, config-file-not-uploaded).

Test plan

  • cargo test -p sync-core — 75 unit tests pass.
  • cargo build -p sync-plugin --target wasm32-wasip2 --release — links under WASI.
  • cargo test -p e2e — deploys the real 0.32.0 assetstorage.wasm and exercises the plugin via icp deploy; config test + all sync tests (incl. idempotent no_op_sync) pass.

🤖 Generated with Claude Code

Retarget sync-core/sync-plugin from this repo's V2 certified-assets
canister to the legacy `assetstorage` canister shipped with dfx 0.32.0
(`ic-certified-assets`, api_version 2). Per-asset metadata now comes from
`.ic-assets.json5` instead of `_headers`/`_redirects`.

sync-core:
- new `config.rs`: full `.ic-assets.json5`/`.ic-assets.json` parser ported
  from ic-asset (nested inheritance, glob `match`, `ignore`, `headers`,
  `cache.max_age`, `enable_aliasing`, `allow_raw_access`, `encodings`,
  `security_policy`)
- new `security_policy.rs`: standard/hardened/disabled CSP presets
- `scan.rs` rewritten as gather_asset_descriptors (walkdir + config-driven
  ignore; `.well-known` kept; config files excluded)
- `canister.rs`: add `enable_aliasing` (CreateAsset) and `is_aliased`
  (SetAssetProperties + AssetProperties); drop RedirectRule/RulePattern/
  SetRedirectRules/get_redirect_rules
- `sync.rs`: drive CreateAsset/SetAssetProperties from AssetConfig; drop
  redirect/html-handling; keep chunk packing + commit staging
- delete redirects.rs, headers.rs, html_handling.rs, glob.rs
- update_properties ignores the canister-injected `Set-Cookie: ic_env=...`
  header (added by icp-cli's env-var step) so syncs stay idempotent

e2e:
- build.rs fetches the real 0.32.0 assetstorage.wasm.gz (sha256-pinned,
  installed gzipped, no decompress) instead of building this repo's canister
- convert fixtures/tests to `.ic-assets.json5`; remove redirect/header/
  multi-dir cases; add config.rs end-to-end test

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
lwshang added a commit to dfinity/icp-cli-recipes that referenced this pull request Jun 3, 2026
Switch to `sync_plugin` without changing the canister wasm.

### Context

This swaps the asset-canister recipe's sync step from the built-in
`type: assets` mechanism to `type: plugin`, **without changing the
canister wasm** — it remains the legacy dfx `assetstorage` canister. The
goal is to ease the upgrade path for existing asset-canister users once
`type: assets` sync is dropped in the upcoming icp-cli release: their
canister stays the same, only the sync mechanism changes.

The plugin wasm is pinned to a pre-release hosted on the
certified-assets repo:
- URL:
`https://github.com/dfinity/certified-assets/releases/download/migration-v2.2.0-209d688/sync_plugin.wasm`
- sha256:
`297c2ef05680d47ac70688d6cebed9bc3a41b2302f400739f894f4f413e6a5ee`

That plugin is built from the **`migration`** branch of
`dfinity/certified-assets`, a permanently separate branch (never merged
into `main`) dedicated to producing a sync plugin that targets the
*legacy* assetstorage canister and reads `.ic-assets.json5` config:
- Branch: https://github.com/dfinity/certified-assets/tree/migration
- Plugin source / rationale:
dfinity/certified-assets#71
- Release hosting this wasm:
https://github.com/dfinity/certified-assets/releases/tag/migration-v2.2.0-209d688

The `v2.2.0` in the tag tracks this recipe version (current latest is
`asset-canister-v2.1.0`); the `-209d688` suffix pins the exact commit
the wasm was built from. Future recipe bumps will get a matching
`migration-vX.Y.Z-<hash>` release.

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The sync plugin's scan step called `Path::canonicalize` on the manifest
`dir`. On wasm32-wasip2 that calls `realpath`, which returns ENOENT for
any path under a preopen whose guest name has more than one component
(e.g. `src/frontend/dist`), even though ordinary access works. Nested
`dir`s therefore failed with "canonicalize src/frontend/dist: No such
file or directory"; single-component dirs like `dist` happened to work.

Build the scan root by prepending `/` to the (host-validated, relative,
`..`-free) manifest dir instead. This matches the shape canonicalize
produced for single-component dirs and satisfies the absolute-root
requirement of AssetSourceDirectoryConfiguration::load, while using only
plain wasi:filesystem ops the preopen supports.

Adds an e2e fixture + test using a nested `dirs` entry, which reproduces
the reported failure without the fix.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@lwshang

lwshang commented Jun 7, 2026

Copy link
Copy Markdown
Collaborator Author

Pushed a fix for a sync failure reported on the forum (#97): a user with dirs: [src/frontend/dist] hit

failed to run plugin
  caused by: plugin returned error: canonicalize src/frontend/dist: No such file or directory (os error 44)

Root cause. scan.rs called Path::canonicalize on the manifest dir. On wasm32-wasip2 that calls realpath, and wasi-libc's realpath returns ENOENT for any path under a preopen whose WASI guest name has more than one component (e.g. src/frontend/dist) — even though read_dir/metadata/read through that preopen work fine. The host preopens each dir under its literal manifest string, so single-component dirs (dist) canonicalized to /dist and worked, while nested dirs failed. Our e2e fixtures and the example plugin only ever used single-component dirs, so this went uncaught. Reproduced in isolation with a minimal wasip2 binary under wasmtime.

Fix. Drop canonicalize in the scan step; build the root by prepending / to the (host-validated, relative, ..-free) manifest dir. This yields the same shape canonicalize produced for single-component dirs, satisfies AssetSourceDirectoryConfiguration::load's absolute-root requirement, and uses only plain wasi:filesystem ops the preopen supports. The walked entry paths are already rooted there, so config lookup and strip_prefix work directly.

Test. Added an e2e fixture + nested_dir_deploy test using dirs: [src/frontend/dist] with a nested subdir asset. Verified it fails with the exact reported error when the fix is reverted, and passes with it. sync-core units, e2e (sync + config), and clippy all green.

PR #71 (migration) is usually in a conflicting/dirty merge state against
main, so the `pull_request`-triggered CI never starts (GitHub can't build
the test-merge commit it runs against). Add a `push` trigger for the
branch, which runs against the branch head directly and is unaffected by
conflicts.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
lwshang added a commit to dfinity/icp-cli-recipes that referenced this pull request Jun 7, 2026
Bumps the asset-canister recipe's sync plugin to the new
`migration-v2.2.1-6b48585` release on
[dfinity/certified-assets](https://github.com/dfinity/certified-assets/releases/tag/migration-v2.2.1-6b48585).

## Why
A user reported that `icp deploy` failed during sync when the recipe's
`dir` was a **nested** path such as `src/frontend/dist`:

```
failed to run plugin
  caused by: plugin returned error: canonicalize src/frontend/dist: No such file or directory (os error 44)
```

The v2.2.0 plugin called `canonicalize` during its scan step. Under WASI
that calls `realpath`, which returns `ENOENT` for any path beneath a
preopened directory whose guest name has more than one component (e.g.
`src/frontend/dist`) — even though ordinary access works.
Single-component dirs like `dist` happened to work, which is why our
examples didn't catch it. The v2.2.1 plugin drops `canonicalize`; fix +
regression test in dfinity/certified-assets#71.

## Change
Only `recipes/asset-canister/recipe.hbs` — the plugin `url` + `sha256`:

| | old (v2.2.0) | new (v2.2.1) |
|---|---|---|
| url | `…/migration-v2.2.0-209d688/sync_plugin.wasm` |
`…/migration-v2.2.1-6b48585/sync_plugin.wasm` |
| sha256 | `297c2ef0…` | `ca7cb566…` |

The canister wasm is unchanged (still the legacy dfx `assetstorage`
canister), so this is a drop-in patch bump for existing users.

## Releasing
After merge, push the `asset-canister-v2.2.1` tag to trigger
`release-recipe.yml` and publish the recipe release.

Ref:
https://forum.dfinity.org/t/icp-cli-announcements-and-feedback-discussion/60410/97
lwshang added a commit that referenced this pull request Jun 7, 2026
Ports the WASI `canonicalize` fix from the `migration` branch (#71) to
`main`.

## Problem
The sync plugin's scan step canonicalized the manifest `dir`. On
`wasm32-wasip2`, `Path::canonicalize` calls `realpath`, which returns
`ENOENT` for **any** path beneath a preopen whose WASI guest name has
more than one component (e.g. `src/frontend/dist`) — even though
`read_dir`/`metadata`/`read` through that preopen work fine. So a nested
`dir` failed:

```
failed to run plugin
  caused by: plugin returned error: canonicalize src/frontend/dist: No such file or directory (os error 44)
```

Single-component dirs like `dist` happened to canonicalize to `/dist`
and worked, which is why the existing fixtures never caught it.
(Originally reported on the
[forum](https://forum.dfinity.org/t/icp-cli-announcements-and-feedback-discussion/60410/97)
against the migration recipe; same root cause lives here.)

## Fix
Build the scan root by prepending `/` to the (host-validated, relative,
`..`-free) manifest dir instead of canonicalizing. This yields the same
shape canonicalize produced for single-component dirs (`/dist`,
`/src/frontend/dist`) and uses only plain `wasi:filesystem` ops the
preopen supports. `main`'s `scan()` had a single `canonicalize` call;
`walk` already uses `read_dir` paths rooted at that dir, and
`_headers`/`_redirects` load via relative paths — none of which need
canonicalization.

## Test
Adds an e2e fixture + `nested_dir_deploy` test using `dirs:
[src/frontend/dist]` with a nested subdir asset. Verified it fails with
the exact error above when the fix is reverted, and passes with it.

- `cargo test -p sync-core` — 188 pass
- `cargo test -p e2e --test sync` — 7 pass (incl. nested regression)
- `cargo clippy -p sync-core` — clean

Note: this is the same fix already shipped on `migration` as
`migration-v2.2.1` (consumed by `asset-canister-v2.2.1`).

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant