Reduce stored record size with opt-in reference sharing

.github/workflows/release.yml

@@ -3,6 +3,10 @@ name: Release candidate
 on:
   workflow_dispatch:
     inputs:
+      release_id:
+        description: Unique Framework Release ID (published as framework-<release_id>)
+        required: true
+        type: string
       source_id:
         description: Logical Release source ID
         required: true
@@ -31,6 +35,7 @@ jobs:
       ADF_RELEASE_SOURCE_ID: ${{ inputs.source_id }}
       ADF_RELEASE_SIGNER_KEY_ID: ${{ inputs.signer_key_id }}
       ADF_RELEASE_OUTPUT_DIR: ${{ github.workspace }}/dist/framework
+      ADF_RELEASE_ID: ${{ inputs.release_id }}
 
     steps:
       - name: Check out reviewed source

README.md

@@ -417,6 +417,68 @@ cargo build --locked --bin adf-claude-runner
 The signed binary release currently continues to publish only `adf`; runner
 distribution is a later compatibility milestone.
 
+## Reducing stored Record size
+
+ADF can store identical input and freshness reference maps once within each Result.
+Each file remains self-contained JSON. Reading it restores the exact logical Record,
+including explicit versus omitted fields, before validation and digest calculation.
+Record IDs, evidence, explanations and the pinned Framework identity are preserved.
+Small Records remain plain JSON when sharing would cost more space.
+
+Existing projects keep their current write format until explicitly migrated. Before
+migration, stop agents, MCP sessions and CI writers for that working directory and
+update every reader/writer to a version supporting `adf-record-refmaps-v1`. Older
+CLIs reject the new project configuration on most paths, but some old execution
+commands bypass that check; configuration alone cannot stop a running old writer.
+
+```sh
+adf project storage inspect --format json
+adf project storage migrate --to adaptive-refmaps-v1 --dry-run
+adf project storage migrate --to adaptive-refmaps-v1
+adf project storage verify
+adf project storage export --record result.<id> --format json
+```
+
+These commands accept `--project <root>` and the existing offline `--release <root>`
+option. Inspect, dry-run, verify and export leave Records, configuration and derived
+indexes unchanged; they acquire a small maintenance lock under `.adf/cache/locks/`.
+Reports describe Result/Evidence JSON bytes only, excluding execution events,
+non-JSON evidence, caches, Git history and other working directories. A skipped-file
+list identifies non-JSON files and interrupted temporary files.
+
+Migration validates Records against the pinned signed release and checks lossless
+roundtrips before writing. It enables project config version 2, atomically replaces
+individual files, and keeps a small Git-ignored recovery journal under
+`.adf/local/storage-migrations/`. A local `.gitignore` is created there if needed;
+existing ignore rules are never overwritten. Configuration YAML may be reformatted.
+Normal operations cooperate with the migration lock. Do not edit files, switch Git
+revisions, or remove lock files while migration is running.
+
+Repeat the same migration command after interruption. If a Record changed since an
+interrupted migration, the command stops instead of overwriting it. Inspect that
+conflict before continuing. After verifying the changed Records, move the recovery
+journal aside and run dry-run again to review a new plan; do not rewrite its hashes
+to conceal the conflict. To restore compatibility with older CLIs, expand all
+Records before downgrading the configuration:
+
+```sh
+adf project storage migrate --to plain-json-v1 --dry-run
+adf project storage migrate --to plain-json-v1
+```
+
+Rollback restores equivalent JSON, including untracked Records, without keeping
+second copies of their contents. Original whitespace is not retained. Free space is
+checked before either direction. Completed replacement files are synced before
+publication; an interruption during a temporary write can leave a temporary file,
+which is reported and must be reviewed before removal. A restored Record may be up
+to 256 MiB; a sharing envelope may be up to 64 MiB. Excessive expansion is rejected
+before copying shared maps. JSON nesting is subject to the parser's depth limit.
+
+No commits, Git history rewrites, cache deletion or automatic legacy Kit migration
+are performed. Existing indexes rebuild from restored Records when their physical
+source changes; their own reference duplication is a separate capacity cost. A Git
+revision or source change still triggers the normal freshness rules.
+
 ## Commands
 
 | Command | What it does |

docs/concepts.ja.md

@@ -252,6 +252,35 @@ Feature Contractは、上位Contractの写しでも上書きでもありませ
 
 現在有効な規範は`docs/`に留めず、`contracts/`へ上げます。`docs/adf/`は案内と索引です。
 
+## 記録の保存容量を減らす
+
+作業結果の中で同じ入力・鮮度確認用の一覧が繰り返される場合、一件の記録内で一覧を共有できます。読込み時に元と同じJSONの内容へ戻してから検証するため、記録ID、根拠、説明文、項目の省略・空の区別を保持します。一覧を共有しても小さくならない記録は通常のJSONを使います。
+
+既存プロジェクトは、バイナリを更新しただけでは保存形式を変更しません。まず、その作業ディレクトリへ書き込むエージェント、MCP、CIを止め、新形式に対応するバイナリへ揃えてください。旧CLIの一部経路は設定の版を検査しないため、設定変更だけで旧プロセスを止められるわけではありません。
+
+```sh
+adf project storage inspect
+adf project storage migrate --to adaptive-refmaps-v1 --dry-run
+adf project storage migrate --to adaptive-refmaps-v1
+adf project storage verify
+adf project storage export --record result.<id>
+```
+
+対象は作業結果と証拠のJSONです。結果に表示する容量は、この対象のバイト数です。実行イベント、JSON以外の証拠、キャッシュ、Git履歴、別作業ディレクトリは含みません。対象外のファイルと停止時の一時ファイルは別途一覧に表示します。参照系の操作も小さな移行ロックを取得しますが、正規記録・設定・索引を変更しません。
+
+移行は設定をversion 2へ切り替え、一件ずつ検証して置き換えます。本文の複製は残さず、再開に必要なハッシュなどをGit管理外の`.adf/local/storage-migrations/`へ保存します。必要なら、この専用ディレクトリに自身を無視する`.gitignore`を新規作成します。既存の無視設定は上書きしません。設定YAMLの整形は変わる場合があります。
+
+停止した場合は同じ移行コマンドで再開できます。停止後に記録が変わっていれば、上書きせず止まります。内容を確認して新しい計画に切り替える場合は、再開用の`current.json`を別名で保管し、dry-runからやり直します。通常JSONへ戻す場合は、次を実行します。
+
+```sh
+adf project storage migrate --to plain-json-v1 --dry-run
+adf project storage migrate --to plain-json-v1
+```
+
+復元には追加の空き容量が必要です。旧CLIへ戻すのは、通常JSONへの展開と設定の復元が完了してからです。元の空白・改行までは復元しません。途中の一時書込みで停止した場合は、一時ファイルが残ることがあるため、報告された内容を確認してから整理します。移行中の手動編集、Git操作、ロックファイルの削除は避けてください。
+
+新形式でもGit revisionやコード変更による証拠の失効は維持します。Git履歴やキャッシュを削除して容量を小さく見せる操作は行いません。索引の重複削減と、旧Kitを使うプロジェクトの移行は別の改善です。
+
 ## 制御基盤・エージェント・人の分担
 
 | 担当 | 役割 |

docs/publishing.md

@@ -75,6 +75,17 @@ linked into them, and uploads the whole set as workflow artifacts.
 
 Nothing is published. Stop here and look at what it produced.
 
+Set `release_id` to a new identifier, for example `adf-2026-09-11-storage`.
+The publication tag will be `framework-adf-2026-09-11-storage`. Never reuse an
+existing identifier for updated binaries. Set `source_id` and `signer_key_id`
+to the configured repository values so existing projects retain their trust pins.
+Local candidate builds accept the same identifier through `ADF_RELEASE_ID`;
+the default `adf-dev` is retained for regression fixtures.
+Signed release names identify archives; compatibility still requires exact
+protocol versions and Rule/Schema digests. The development lock retains its
+fixed `adf-dev` identity. Changing a signed release name does not relax signature
+or pinned archive verification.
+
 ### 2. Publish it
 
 Run the **Publish a Framework Release** workflow with the candidate run's ID

schemas/storage/project-config-v2.schema.json

@@ -0,0 +1,17 @@
+{
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "$id": "adf://schemas/storage/project-config-v2",
+  "type": "object",
+  "required": ["schema_version", "record_storage", "project_sources", "repository_observation"],
+  "properties": {
+    "schema_version": {"const": "2"},
+    "record_storage": {"const": "adaptive-refmaps-v1"},
+    "project_sources": {
+      "type": "object", "required": ["contracts", "decisions"],
+      "properties": {"contracts": {"type": "string"}, "decisions": {"type": "string"}},
+      "additionalProperties": false
+    },
+    "repository_observation": {"type": "string"}
+  },
+  "additionalProperties": false
+}

schemas/storage/record-refmaps-v1.schema.json

@@ -0,0 +1,24 @@
+{
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "$id": "adf://schemas/storage/record-refmaps-v1",
+  "type": "object",
+  "required": ["storage_format", "record_kind", "record_digest", "record", "reference_maps", "map_bindings"],
+  "properties": {
+    "storage_format": {"const": "adf-record-refmaps-v1"},
+    "record_kind": {"enum": ["result", "evidence"]},
+    "record_digest": {"type": "string", "pattern": "^sha256:[0-9a-f]{64}$"},
+    "record": {"type": "object"},
+    "reference_maps": {
+      "type": "object", "minProperties": 1,
+      "propertyNames": {"pattern": "^sha256:[0-9a-f]{64}$"},
+      "additionalProperties": {"type": "object", "additionalProperties": {"type": "string"}}
+    },
+    "map_bindings": {
+      "type": "object", "minProperties": 1,
+      "propertyNames": {"pattern": "^/(input_refs|freshness_refs|payload/outcomes/(0|[1-9][0-9]*)/(input_refs|freshness_refs))$"},
+      "additionalProperties": {"type": "string", "pattern": "^sha256:[0-9a-f]{64}$"}
+    }
+  },
+  "additionalProperties": false,
+  "$comment": "The codec also verifies map hashes, allowed paths by record kind, missing/unused maps, no overwrites, resource limits and restored record_digest. Validate the restored Record with the pinned logical Schema. This storage Schema does not change the Framework identity."
+}

scripts/release-ci.sh

@@ -12,6 +12,13 @@ BASE_LOCK=$KIT_ROOT/testdata/fixtures/db-sqs/framework-lock.yaml
 OUTPUT_DIR=${ADF_RELEASE_OUTPUT_DIR:-"$KIT_ROOT/dist/framework"}
 SOURCE_ID=${ADF_RELEASE_SOURCE_ID:-remote:official}
 SIGNER_KEY_ID=${ADF_RELEASE_SIGNER_KEY_ID:-framework.release.prototype}
+RELEASE_ID=${ADF_RELEASE_ID:-adf-dev}
+case "$RELEASE_ID" in
+  ''|*[!A-Za-z0-9._-]*)
+    echo "ADF_RELEASE_ID contains unsupported characters" >&2
+    exit 2
+    ;;
+esac
 PUBLIC_KEY=${ADF_RELEASE_SIGNING_PUBLIC_KEY_HEX:?ADF_RELEASE_SIGNING_PUBLIC_KEY_HEX is required}
 : "${ADF_RELEASE_SIGNING_KEY_HEX:?ADF_RELEASE_SIGNING_KEY_HEX is required}"
 
@@ -34,6 +41,19 @@ cleanup() {
 trap cleanup EXIT HUP INT TERM
 
 mkdir -p "$WORK_ROOT/source/schemas" "$WORK_ROOT/first" "$WORK_ROOT/second"
+python3 - "$BASE_LOCK" "$WORK_ROOT/framework-lock.yaml" "$RELEASE_ID" <<'PY'
+import re
+import sys
+from pathlib import Path
+
+source, output, release_id = sys.argv[1:]
+text, count = re.subn(r'^framework_release:.*$', 'framework_release: "' + release_id + '"',
+                      Path(source).read_text(), flags=re.MULTILINE)
+if count != 1:
+    raise SystemExit('Expected exactly one Framework Release ID in the base lock')
+Path(output).write_text(text)
+PY
+BASE_LOCK=$WORK_ROOT/framework-lock.yaml
 cp "$SOURCE_RULES" "$WORK_ROOT/source/rules.yaml"
 cp "$SOURCE_FRAMEWORK_CATALOG" "$WORK_ROOT/source/framework-catalog.yaml"
 cp -R "$SOURCE_SCHEMAS" "$WORK_ROOT/source/schemas/v1"

scripts/tests/test-release-ci.sh

@@ -32,6 +32,23 @@ test -s "$OUTPUT/candidate-framework.lock"
 test -s "$OUTPUT/distribution-trust.json"
 test -s "$OUTPUT/publish-receipt.json"
 
+# A new CLI release must not collide with the development release tag.
+VERSIONED_OUTPUT=$TEST_ROOT/versioned
+ADF_RELEASE_ID=adf-storage-test run_release_ci "$VERSIONED_OUTPUT" "$PUBLIC_KEY"
+python3 - "$VERSIONED_OUTPUT" <<'PY'
+import json
+import sys
+from pathlib import Path
+root = Path(sys.argv[1])
+assert json.loads((root / 'publish-receipt.json').read_text())['release_id'] == 'adf-storage-test'
+assert 'adf-storage-test' in (root / 'candidate-framework.lock').read_text()
+PY
+if ADF_RELEASE_ID='../invalid' run_release_ci "$TEST_ROOT/invalid" "$PUBLIC_KEY" >/dev/null 2>&1; then
+  echo "Release CI accepted an unsafe release ID" >&2
+  exit 1
+fi
+test ! -e "$TEST_ROOT/invalid/framework-release.tar"
+
 # A rerun must not replace an already reviewed candidate.
 if run_release_ci "$OUTPUT" "$PUBLIC_KEY" >/dev/null 2>&1; then
   echo "Release CI unexpectedly overwrote existing outputs" >&2

skill-src/adf-analyst/SKILL.md

@@ -207,3 +207,11 @@ measurements remain unknown.
 
 If submission is rejected as stale, the inputs moved under you. Call `adf_next`
 again and redo the work against the fresh action - do not retry the old payload.
+
+## Reading stored Records
+
+CLI and MCP return logical Records even when files use shared reference maps. To
+inspect a file-backed Result or Evidence as ordinary JSON, run
+`adf project storage export --record <id>`. Do not hand-edit storage envelopes,
+reference tables, or digests. Storage migration is an explicit maintenance task;
+continue using the issued Context and normal submission tools for development.

skill-src/adf-builder/SKILL.md

@@ -103,3 +103,11 @@ collect these metrics. ADF records Context size without an extra model invocatio
 
 If submission is rejected as stale, the inputs moved under you. Call `adf_next`
 again and work from the fresh action rather than retrying the old payload.
+
+## Reading stored Records
+
+CLI and MCP return logical Records even when files use shared reference maps. To
+inspect a file-backed Result or Evidence as ordinary JSON, run
+`adf project storage export --record <id>`. Do not hand-edit storage envelopes,
+reference tables, or digests. Storage migration is an explicit maintenance task;
+continue using the issued Context and normal submission tools for development.

skill-src/adf-challenger/SKILL.md

@@ -94,3 +94,11 @@ or another model call solely to collect them.
 
 If submission is rejected as stale, the work moved under you. Call `adf_next` again
 and challenge the fresh state rather than retrying the old payload.
+
+## Reading stored Records
+
+CLI and MCP return logical Records even when files use shared reference maps. To
+inspect a file-backed Result or Evidence as ordinary JSON, run
+`adf project storage export --record <id>`. Do not hand-edit storage envelopes,
+reference tables, or digests. Storage migration is an explicit maintenance task;
+continue using the issued Context and normal submission tools for development.

src/delivery.rs

@@ -289,6 +289,8 @@ pub fn switch_framework_lock(
     project_root: &Path,
     candidate_lock_path: &Path,
 ) -> Result<SwitchReceipt, DeliveryError> {
+    let _storage_guard =
+        crate::storage_io::StorageGuard::shared(project_root).map_err(delivery_error)?;
     switch_framework_lock_for(project_root, candidate_lock_path, TrustUse::NewActivation)
 }
 
@@ -355,6 +357,8 @@ pub fn rollback_framework_lock(
     project_root: &Path,
     backup_lock_path: &Path,
 ) -> Result<SwitchReceipt, DeliveryError> {
+    let _storage_guard =
+        crate::storage_io::StorageGuard::shared(project_root).map_err(delivery_error)?;
     let project_root = canonical_project_root(project_root)?;
     let backups_root = project_root.join(".adf/cache/framework-lock-backups");
     let backup_path = absolute_from(&project_root, backup_lock_path)

src/execution_record.rs

@@ -146,6 +146,7 @@ impl ExecutionEvent {
 
 #[derive(Debug, Clone)]
 pub struct ExecutionEventStore {
+    _storage_guard: crate::storage_io::StorageGuard,
     project_root: PathBuf,
 }
 
@@ -155,7 +156,14 @@ impl ExecutionEventStore {
             .as_ref()
             .canonicalize()
             .map_err(|error| format!("cannot resolve project root: {error}"))?;
-        Ok(Self { project_root })
+        let storage_guard = crate::storage_io::StorageGuard::shared(&project_root)?;
+        if project_root.join(".adf/config.yaml").exists() {
+            crate::project_config::load_project_config(&project_root).map_err(|e| e.to_string())?;
+        }
+        Ok(Self {
+            project_root,
+            _storage_guard: storage_guard,
+        })
     }
 
     pub fn begin(
@@ -291,8 +299,9 @@ impl ExecutionEventStore {
         paths.sort();
         for path in paths {
             let bytes = read_regular_file(&path)?;
-            let result: Value = serde_json::from_slice(&bytes)
-                .map_err(|error| format!("{}: {error}", path.display()))?;
+            let result =
+                crate::record_storage::decode(&bytes, crate::record_storage::RecordKind::Result)
+                    .map_err(|error| format!("{}: {error}", path.display()))?;
             if result["id"].as_str() == Some(result_id) {
                 if result["action_id"].as_str() == Some(started.action_id.as_str())
                     && result["context_digest"].as_str() == Some(started.context_digest.as_str())