feat: complete data-ml integration — sql schemas, scripts, docs, tests

.github/workflows/validate.yml

@@ -6,11 +6,37 @@ on:
     branches: [ main ]
 
 jobs:
-  readme:
+  validate:
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v4
-      - name: Ensure scaffold files exist
+
+      - name: Ensure required files exist
         run: |
           test -f README.md
           test -f LICENSE
+          test -f .release-versions.env
+          test -f sql/schema-model-memory.sql
+          test -f sql/schema-sae.sql
+          test -f sql/schema-semantic-cache.sql
+          test -f sql/schema-manifests.sql
+          test -f sql/schema-proof-bundles.sql
+
+      - name: Validate SQL schemas
+        run: bash tests/validate-schemas.sh
+
+      - name: Validate scripts (no akai dependency steps)
+        run: |
+          # Run validate-scripts but skip the doctor-deep step (no akai in CI)
+          DB=$(mktemp /tmp/aurekai-ci-XXXXXX.db)
+          bash scripts/init-db.sh "$DB"
+          sqlite3 "$DB" "INSERT INTO manifests (schema_version, name, version, operator_count, raw_json) VALUES ('aurekai.deploy.v1','aurekai','0.8.0-alpha.4',89,'{}');"
+          bash scripts/manifest-verify.sh "$DB" aurekai 0.8.0-alpha.4
+          sqlite3 "$DB" "INSERT INTO sae_dictionaries (artifact_name, artifact_ext, layer_index, size_bytes, sha256) VALUES ('t.aksae','.aksae',0,1,'abc');"
+          bash scripts/sae-audit.sh "$DB"
+          sqlite3 "$DB" "INSERT INTO proof_bundles (recipe_name, status, version, schema_version, proof_json) VALUES ('r.json','ok','0.8.0-alpha.4','aurekai.deploy.v1','{\"status\":\"ok\"}');"
+          bash scripts/proof-bundle-export.sh "$DB" /tmp/pbe-out
+          bash scripts/semantic-cache-bench.sh "$DB" 100
+          sqlite3 "$DB" "INSERT INTO semantic_cache (cache_key, value) VALUES ('k','v');"
+          sqlite3 "$DB" "INSERT INTO model_memory (artifact_name, artifact_ext) VALUES ('m.akmodel','.akmodel');"
+          bash scripts/release-gate.sh "$DB"

.release-versions.env

@@ -0,0 +1,4 @@
+AUREKAI_VERSION=0.8.0-alpha.4
+AKAI_PACKAGE_VERSION=0.8.0-alpha.4
+AUREKAI_MANIFEST_SCHEMA=aurekai.deploy.v1
+HELM_CHART_VERSION=0.8.1

README.md

@@ -4,20 +4,65 @@
 
 # aurekai-sqlite
 
-Aurekai integration surface for Sqlite.
+Aurekai integration surface for SQLite — lightweight, serverless, embedded data-ml storage for model memory, SAE dictionaries, semantic cache, manifests, and proof bundles.
 
-Status: planned
-Type: data
+Status: active
+Type: data-ml
+
+## Overview
+
+`aurekai-sqlite` provides DDL schemas and scripts to persist and query Aurekai artifacts using SQLite (`sqlite3`). No server required — everything runs embedded. Works with `akai` CLI, the Python `aurekai` SDK, and any standard SQLite tooling.
 
 ## Core Template Set
 
-- doctor-deep
-- manifest-verify
-- model-memory-pack
-- sae-audit
-- semantic-cache-bench
-- proof-bundle-export
-- release-gate
+| Template | Description |
+|---|---|
+| `doctor-deep` | Connectivity check, schema integrity, and `akai doctor --deep` |
+| `manifest-verify` | Validate `aurekai.deploy.v1` manifest row in the database |
+| `model-memory-pack` | Insert `.akmodel` / `.bfmodel` artifacts into `model_memory` table |
+| `sae-audit` | Audit SAE dictionary rows in `sae_dictionaries` table |
+| `semantic-cache-bench` | Benchmark semantic cache read/write throughput against SQLite |
+| `proof-bundle-export` | Export proof bundle rows to `out/proof-bundle.json` |
+| `release-gate` | Verify all artifact tables are populated before publish |
+
+## Quick Start
+
+```bash
+# 1. Install Aurekai runtime
+npm install -g @aurekai/runtime@0.8.0-alpha.4
+
+# 2. Init the database
+bash scripts/init-db.sh /tmp/aurekai.db
+
+# 3. Run doctor deep
+bash scripts/doctor-deep.sh /tmp/aurekai.db
+
+# 4. Pack a model memory artifact
+bash scripts/model-memory-pack.sh /tmp/aurekai.db path/to/artifact.akmodel
+
+# 5. Run the release gate
+bash scripts/release-gate.sh /tmp/aurekai.db
+```
+
+## Directory Layout
+
+```
+sql/                   DDL schema files
+scripts/               Template scripts (doctor-deep, model-memory-pack, …)
+examples/              Sample inputs and usage
+tests/                 Validation scripts
+docs/                  Quickstart, schema reference, script reference
+```
+
+## SQL Schemas
+
+| File | Tables |
+|---|---|
+| `sql/schema-model-memory.sql` | `model_memory` |
+| `sql/schema-sae.sql` | `sae_dictionaries` |
+| `sql/schema-semantic-cache.sql` | `semantic_cache` |
+| `sql/schema-manifests.sql` | `manifests` |
+| `sql/schema-proof-bundles.sql` | `proof_bundles` |
 
 ## Canonical References
 

docs/quickstart.md

@@ -0,0 +1,73 @@
+# Quickstart
+
+## 1. Clone the repo
+
+```bash
+git clone https://github.com/aurekai/aurekai-sqlite.git
+cd aurekai-sqlite
+```
+
+## 2. Install Aurekai runtime
+
+```bash
+npm install -g @aurekai/runtime@0.8.0-alpha.4
+```
+
+## 3. Initialize the database
+
+```bash
+bash scripts/init-db.sh /path/to/aurekai.db
+```
+
+This applies all five schemas (`model_memory`, `sae_dictionaries`, `semantic_cache`, `manifests`, `proof_bundles`).
+
+## 4. Run doctor deep
+
+```bash
+bash scripts/doctor-deep.sh /path/to/aurekai.db
+```
+
+Verifies all required tables exist, prints row counts, and runs `akai doctor --deep`.
+
+## 5. Insert a manifest
+
+```python
+import sqlite3, json
+
+db = sqlite3.connect("/path/to/aurekai.db")
+manifest = json.load(open("examples/sample-aurekai.manifest.json"))
+db.execute(
+    """INSERT INTO manifests (schema_version, name, version, operator_count, raw_json)
+       VALUES (?,?,?,?,?)""",
+    (manifest["schema_version"], manifest["name"], manifest["version"],
+     manifest["operator_count"], json.dumps(manifest))
+)
+db.commit()
+```
+
+## 6. Pack a model memory artifact
+
+```bash
+bash scripts/model-memory-pack.sh /path/to/aurekai.db path/to/artifact.akmodel
+```
+
+## 7. Run the release gate
+
+```bash
+bash scripts/release-gate.sh /path/to/aurekai.db
+```
+
+Checks that all artifact tables have at least one row and no `proof_bundles` row has `status=fail`.
+
+## 8. End-to-end example
+
+```bash
+bash examples/sample-db-init.sh
+```
+
+## 9. Run the test suite
+
+```bash
+bash tests/validate-schemas.sh
+bash tests/validate-scripts.sh
+```

docs/schema.md

@@ -0,0 +1,99 @@
+# Schema Reference
+
+## model_memory
+
+Stores `.akmodel`, `.bfmodel`, `.akfpqx`, `.bffpqx` artifact metadata and optional binary payload.
+
+| Column | Type | Notes |
+|---|---|---|
+| `id` | INTEGER PK | Auto-increment |
+| `artifact_name` | TEXT | File name of the artifact |
+| `artifact_ext` | TEXT | `.akmodel` / `.bfmodel` / `.akfpqx` / `.bffpqx` |
+| `operator_id` | TEXT | Aurekai operator that produced the artifact |
+| `version` | TEXT | Aurekai release version |
+| `schema_version` | TEXT | Always `aurekai.deploy.v1` |
+| `size_bytes` | INTEGER | File size |
+| `sha256` | TEXT | Hex SHA-256 checksum |
+| `payload` | BLOB | Raw artifact bytes (optional) |
+| `created_at` | TEXT | ISO-8601 UTC |
+| `updated_at` | TEXT | ISO-8601 UTC |
+
+---
+
+## sae_dictionaries
+
+Stores `.aksae` / `.bfsae` Sparse Autoencoder dictionary artifacts.
+
+| Column | Type | Notes |
+|---|---|---|
+| `id` | INTEGER PK | Auto-increment |
+| `artifact_name` | TEXT | File name |
+| `artifact_ext` | TEXT | `.aksae` / `.bfsae` |
+| `operator_id` | TEXT | Producing operator |
+| `version` | TEXT | Aurekai release version |
+| `schema_version` | TEXT | Always `aurekai.deploy.v1` |
+| `layer_index` | INTEGER | Network layer index |
+| `feature_count` | INTEGER | Number of learned features |
+| `size_bytes` | INTEGER | File size |
+| `sha256` | TEXT | Hex SHA-256 checksum |
+| `payload` | BLOB | Raw bytes (optional) |
+| `created_at` | TEXT | ISO-8601 UTC |
+| `updated_at` | TEXT | ISO-8601 UTC |
+
+---
+
+## semantic_cache
+
+Stores semantic cache key→value pairs with optional embedding blobs.
+
+| Column | Type | Notes |
+|---|---|---|
+| `id` | INTEGER PK | Auto-increment |
+| `cache_key` | TEXT UNIQUE | Hash key (e.g. SHA-256 of input) |
+| `embedding` | BLOB | Optional embedding vector |
+| `value` | TEXT | Cached response |
+| `hit_count` | INTEGER | Number of times this entry was served |
+| `version` | TEXT | Aurekai release version |
+| `expires_at` | TEXT | ISO-8601 UTC expiry (nullable) |
+| `created_at` | TEXT | ISO-8601 UTC |
+| `updated_at` | TEXT | ISO-8601 UTC |
+
+---
+
+## manifests
+
+Stores `aurekai.deploy.v1` deploy manifest records.
+
+| Column | Type | Notes |
+|---|---|---|
+| `id` | INTEGER PK | Auto-increment |
+| `schema_version` | TEXT | Always `aurekai.deploy.v1` |
+| `name` | TEXT | Manifest name (e.g. `aurekai`) |
+| `version` | TEXT | Release version |
+| `operator_count` | INTEGER | Number of operators in release |
+| `runtime_ref` | TEXT | Native runtime git ref |
+| `helm_chart_ver` | TEXT | Helm chart version |
+| `pypi_ver` | TEXT | PyPI package version |
+| `npm_ver` | TEXT | npm package version |
+| `jsr_ver` | TEXT | JSR package version |
+| `raw_json` | TEXT | Full manifest JSON |
+| `created_at` | TEXT | ISO-8601 UTC |
+
+Unique constraint on `(name, version)`.
+
+---
+
+## proof_bundles
+
+Stores proof bundle export records from `akai run --sae-audit`.
+
+| Column | Type | Notes |
+|---|---|---|
+| `id` | INTEGER PK | Auto-increment |
+| `recipe_name` | TEXT | Recipe file that was run |
+| `status` | TEXT | `ok` / `fail` / `partial` |
+| `operator_id` | TEXT | Producing operator |
+| `version` | TEXT | Aurekai release version |
+| `schema_version` | TEXT | Always `aurekai.deploy.v1` |
+| `proof_json` | TEXT | Proof bundle JSON payload |
+| `exported_at` | TEXT | ISO-8601 UTC |

docs/scripts.md

@@ -0,0 +1,101 @@
+# Scripts Reference
+
+All scripts in `scripts/` accept `<db_path>` as their first argument.
+
+---
+
+## `init-db.sh`
+
+```
+bash scripts/init-db.sh <db_path>
+```
+
+Applies all five DDL schema files in order. Safe to run on an existing database (uses `CREATE TABLE IF NOT EXISTS`).
+
+---
+
+## `doctor-deep.sh`
+
+```
+bash scripts/doctor-deep.sh <db_path>
+```
+
+1. Verifies the database file exists.
+2. Checks all five required tables are present.
+3. Prints row counts per table.
+4. Runs `akai doctor --deep`.
+
+Exit 1 on any failure.
+
+---
+
+## `manifest-verify.sh`
+
+```
+bash scripts/manifest-verify.sh <db_path> <name> <version>
+```
+
+Queries `manifests` for a row matching `(name, version, schema_version='aurekai.deploy.v1')`. Prints the matched row and exits 1 if not found.
+
+---
+
+## `model-memory-pack.sh`
+
+```
+bash scripts/model-memory-pack.sh <db_path> <artifact_path> [operator_id]
+```
+
+Inserts an `.akmodel`, `.bfmodel`, `.akfpqx`, or `.bffpqx` artifact file into the `model_memory` table, including `size_bytes`, `sha256`, and raw payload bytes via SQLite's `readfile()`.
+
+---
+
+## `sae-audit.sh`
+
+```
+bash scripts/sae-audit.sh <db_path>
+```
+
+Prints total row count, breakdown by extension and operator, and warns if any rows are missing `sha256`.
+
+---
+
+## `semantic-cache-bench.sh`
+
+```
+bash scripts/semantic-cache-bench.sh <db_path> [iterations]
+```
+
+Runs `iterations` (default 1000) write + read operations against `semantic_cache` and prints throughput in ops/s.
+
+---
+
+## `proof-bundle-export.sh`
+
+```
+bash scripts/proof-bundle-export.sh <db_path> [output_dir]
+```
+
+Exports all `proof_bundles` rows to `<output_dir>/proof-bundle.json` (default `out/`). The JSON structure is:
+
+```json
+{
+  "status": "ok",
+  "count": 3,
+  "bundles": [ ... ]
+}
+```
+
+---
+
+## `release-gate.sh`
+
+```
+bash scripts/release-gate.sh <db_path>
+```
+
+Checks:
+1. Every table has at least one row.
+2. All manifests use `schema_version = 'aurekai.deploy.v1'`.
+3. No `proof_bundles` row has `status = 'fail'`.
+
+Exits 1 if any check fails — suitable as a CI pre-publish gate.