Date: 2026-05-24
Branch / commit at run time: feature/data-layer @ bcd29ec + uncommitted data-layer completed_at fix
Account / profile: denon82 (active default per gflow auth list)
Spend: 1 Imagen credit (t2i) + 1 Veo credit (omni-flash, 4s, count=1, landscape)
Driver: tests/e2e/test_data_layer_e2e.py
Spec: docs/superpowers/specs/2026-05-24-data-layer-design.md
Plan: docs/superpowers/plans/2026-05-24-data-layer.md
Doc: docs/DATA_LAYER.md
This document is the credit-spending evidence ledger for PR #58. Mirrors the project's existing pattern (see LIVE_VERIFICATION_v0.8.1, LIVE_VERIFICATION_video_download). Per [[verification-ledger-5-layer]] every credit-spending feature must have file-cardinality + magic-bytes + Pillow-dimensions + structlog-invariants + user-gallery confirmation — for the data layer we add a 6th layer: persisted-row verification.
| Layer | t2i | t2v |
|---|---|---|
| Subprocess exit code | 0 ✅ | 0 ✅ |
| File present | 1 PNG, 735 434 B ✅ | 1 MP4, 621 429 B ✅ |
| Magic bytes | 89 50 4E 47 (PNG) ✅ |
66 74 79 70 69 73 6F 6D (ftypisom) ✅ |
| Pillow dimensions | 1024 × 1024 (1:1) ✅ | n/a (video) |
| Data-layer profile row | denon82 ✅ | denon82 ✅ |
| Data-layer project row | source=generated ✅ | source=generated ✅ |
| Data-layer asset row | kind=image, status=ready, 1024×1024 ✅ | kind=video, status=MEDIA_GENERATION_STATUS_SUCCESSFUL, duration=4.0s ✅ |
| Data-layer operation row | status=succeeded, completed_at set ✅ | status=succeeded, completed_at set ✅ |
| Prompt round-trip | full text + sha256 ✅ | full text + sha256 ✅ |
| Operation_assets link | output, position 0 ✅ | output, position 0 ✅ |
| local_files row | absolute path + sha256 + 735 KB ✅ | absolute path + sha256 + 621 KB ✅ |
gflow data media <id> round-trip |
media_id + project_id + kind in output ✅ | media_id + project_id + kind in output ✅ |
gflow data media <unknown> exit |
16 (DataStoreError) ✅ | 16 (DataStoreError) ✅ |
Result: ALL six layers pass for both image and video.
Wall-clock: t2i ~52 s, t2v ~52 s, both well below their generous timeouts (240 s / 600 s).
The first live t2i run failed on the assertion op["completed_at"] is not None. The data layer had successfully persisted the started_at (the moment recording began) but left completed_at NULL for image operations.
Root cause: OperationRecorder.record_generated_images and record_upload_image inserted the SUCCEEDED operation row in one shot via insert_operation(...), which only stamps started_at. completed_at was reserved for the video lifecycle's later update_operation_status(...) call. But for image generation, recording happens AFTER the file is downloaded — the operation is already terminal at insert time. Leaving completed_at NULL would have broken any query like SELECT * FROM operations WHERE completed_at IS NULL (which should surface only pending operations).
Fix: After the SUCCEEDED insert_operation(...), immediately call repo.update_operation_status(op_id, SUCCEEDED, _now_utc_iso(), None, None) to stamp completed_at = now. Lifted the datetime/UTC import to module scope and extracted a _now_utc_iso() helper. Two operations affected: record_upload_image, record_generated_images.
Without the E2E live test, this gap would have shipped silently — every existing unit test passed because they didn't assert on completed_at round-trip.
The Task 7 transport refactor wired operation_name_from_generate_response(body) into the on_started callback so flow_operation_id could be stored separately from flow_media_id (per spec). The live omni-flash t2v response did NOT carry operations[0].operation.name — the helper returned None, which the recorder persisted as NULL.
This is consistent with the spec's wording: "In current captures this appears to match the generated media ID, but it should be stored separately when observed." The recorder correctly stores None when not observed. Verified for omni-flash; behavior on veo-quality / veo-fast / veo-lite is not yet observed live. The E2E test's assertion was loosened to "if present, equals media_id" so future model coverage can tighten without API changes.
This is not a data-layer bug; the data layer stored what the transport surfaced. Filed for follow-up investigation when a richer Flow capture is available.
gflow image t2i honors GFLOW_CLI_OUTPUT_DIR. gflow video t2v does NOT — it defaults to ./tmp and requires --out-dir explicitly. The E2E test passes --out-dir for the video path. Not part of this PR's scope, but worth noting for future env-var-coverage work.
$ pytest tests/e2e/test_data_layer_e2e.py::test_t2i_records_full_provenance -m e2e -v
tests/e2e/test_data_layer_e2e.py::test_t2i_records_full_provenance PASSED [100%]
============================= 1 passed in 38.00s ==============================
out/images/2026-05-24/57f8a03c-e46b-4ef3-9339-73096e78cd9b_1.png
size: 735 434 bytes
magic: 89 50 4E 47 (PNG)
Pillow: 1024 × 1024 (1:1 within ±2%)
profiles
name=denon82
profile_dir=C:\Users\ffrol\AppData\Local\ffroliva\gflow-cli\profile_denon82
projects
flow_project_id=404f2af8-a0af-4161-8d80-7daeb4cb92b4
source=generated
assets
flow_media_id=57f8a03c-e46b-4ef3-9339-73096e78cd9b
flow_project_id=404f2af8-a0af-4161-8d80-7daeb4cb92b4
kind=image status=ready
width=1024 height=1024
model=NARWHAL aspect_ratio=IMAGE_ASPECT_RATIO_SQUARE
operations
mode=t2i status=succeeded
prompt="a single red apple on a wooden table, soft daylight"
prompt_hash=1ff72c3b…687642 (SHA-256, 64 hex)
prompt_redacted=0 (GFLOW_CLI_HISTORY_PROMPTS=store)
model=NARWHAL aspect_ratio=IMAGE_ASPECT_RATIO_SQUARE
started_at=2026-05-24T11:33:00.054Z
completed_at=2026-05-24T11:33:00.055Z ← post-fix; was NULL before
flow_operation_id=NULL flow_batch_id=NULL
operation_assets
role=output position=0
local_files
path=…\57f8a03c-e46b-4ef3-9339-73096e78cd9b_1.png (absolute)
sha256=<64-char hex>
bytes=735434
media_type=image/png
$ pytest tests/e2e/test_data_layer_e2e.py::test_t2v_records_started_and_completed_lifecycle -m e2e -v
tests/e2e/test_data_layer_e2e.py::test_t2v_records_started_and_completed_lifecycle PASSED [100%]
============================= 1 passed in 52.39s ==============================
out/c0608f5e-1af8-4113-8cfe-859aea9a951f.mp4
size: 621 429 bytes
magic: 00 00 00 20 66 74 79 70 69 73 6F 6D (ftypisom — ISO BMFF mp4)
projects
flow_project_id=35e0dac8-6922-49ad-a8a9-4d37beb66b56
source=generated
assets
flow_media_id=c0608f5e-1af8-4113-8cfe-859aea9a951f
flow_project_id=35e0dac8-6922-49ad-a8a9-4d37beb66b56
kind=video
status=MEDIA_GENERATION_STATUS_SUCCESSFUL ← raw Flow status string
duration_seconds=4.0
operations
mode=t2v status=succeeded
prompt="a single red apple on a wooden table, soft daylight"
prompt_hash=1ff72c3b…687642
model=omni_flash aspect_ratio=landscape
started_at=2026-05-24T11:31:22.506Z
completed_at=2026-05-24T11:31:56.414Z
flow_operation_id=NULL ← see "Known observation" above
operation_assets
role=output position=0
local_files
path=…\c0608f5e-1af8-4113-8cfe-859aea9a951f.mp4 (absolute)
sha256=<64-char hex>
bytes=621429
media_type=video/mp4
Both live tests assert that gflow data media <flow_media_id> --profile denon82 returns the persisted row with media_id, project_id, and kind present in stdout. Both pass.
Negative-path: gflow data media does-not-exist --profile denon82 exits 16 (DataStoreError) with the message No local media record found: does-not-exist. The CLI structured log emits error_raised carrying the RFC 9457 Problem Details payload.
export GFLOW_CLI_E2E_PROFILE=denon82
export PYTHONUTF8=1
uv run python -m pytest tests/e2e/test_data_layer_e2e.py -m e2e -v --basetemp=tmp/e2e-evidenceFor the cheap subset (no Veo spend), set GFLOW_CLI_E2E_RUN_VIDEO=0.
DB inspection after the run:
sqlite3 tmp/e2e-evidence/test_*_records_*/gflow.db \
-header -column \
"SELECT mode, status, started_at, completed_at FROM operations;"- DATA_LAYER.md — user-facing reference for the data layer.
- USAGE.md § exit codes — exit code 16 (DataStoreError) row.
- tests/e2e/test_data_layer_e2e.py — the verifier itself.
- Memory: [[verification-ledger-5-layer]] · [[data-layer-overview]] · [[on-started-callback-recorder-safety]].