Skip to content

LIVE_VERIFICATION_image_batch.md

Latest commit

 

History

History
355 lines (269 loc) · 22.8 KB

LIVE_VERIFICATION_image_batch.md

File metadata and controls

355 lines (269 loc) · 22.8 KB
Raw

Live verification — gflow image batch jitter matrix

Status: Skeleton — sessions not yet recorded. Spec: docs/superpowers/specs/2026-05-21-multi-image-prompt-design.md §8 Profile (planned): ui_automation (Chrome strategy, mandatory per project memory)

Why this exists

gflow image batch --same-project inserts a 3–7 s random delay between submissions. The rationale is anti-bot-detection (avoid Flow throttling) but it has never been verified empirically. Spec §8 mandates a systematic check before committing to keep or drop the jitter.

Method — the 3 × 2 × 3 matrix

Three cells × two sessions × N=3 reps per cell. Sessions are separated by ≥ 2 hours to defeat account-warmth confounders.

Cell --same-project --jitter env What it tests
R1 1 0 Same-project, no sleep → if pass, jitter is unnecessary in same-project mode.
R2 1 1 (default 3–7 s) Current behaviour baseline.
R3 0 0 Different-project, no sleep → control. Isolates "rapid-fire across projects" from jitter.

Cell pass criteria (all must hold)

  • CLI exit code 0.
  • ui_automation.batch_response_seen count == manifest row count.
  • ui_automation.batch_response_dropped_project_id_mismatch count == 0.
  • ui_automation.overlay_dismiss_failed count == 0.
  • No row times out (each row produces an image within the configured row timeout).

Otherwise it fails. If a failure is suspected to be the open listener-miss flake (memory phase-b-followups), classify the cell as inconclusive, not failed.

Decision rule

Outcome Action (commit 5b)
R1 passes 3/3 in both sessions AND R3 passes 3/3 in both sessions Drop jitter — refactor(image): drop unconditional jitter from --same-project (live-verified safe).
R1 fails any time with non-listener-miss failure Keep jitter — docs(image): document anti-detection jitter rationale on --same-project. Cite the failing run.
R1 mixed (some pass, some inconclusive/fail) Default conservative: keep jitter. File a follow-up issue to make it configurable.
Matrix incomplete (< 2 sessions) Default conservative: keep jitter.

Mid-matrix abort: if R1 first session fails non-listener-miss, the matrix may abort early. The conclusion is "keep jitter"; remaining cells are not run. The abort reason is recorded below.

Environment

Property Session 1 Session 2
Date / UTC time 2026-05-22T16:12:27Z (pending)
gflow-cli git rev 81eb012 (pending)
Python version 3.13.3 (pending)
Playwright version 1.59.0 (pending)
Chromium build bundled with Playwright 1.59.0 (pending)
UTC hour 16 (pending)
Account-warmth proxy cold (first matrix run today on profile denon82; last profile use 2026-05-21 16:00 UTC, > 24 h prior) (pending)
Profile used denon82 (substituted for ui_automation — see Aborted runs section for rationale) (pending)

Matrix runs

Session Cell Rep Exit batch_response_seen dropped_pid overlay_fail Notes
1 R1 1 1 — — — Aborted pre-flight — AuthExpiredError: HTTP 401 from project.createProject after 12 s. Stale denon82 cookies. Did not reach Flow submission. See Aborted runs section.
1 R1 1 (retry) 1 8 0 (inferred) 0 (inferred) Test-assertion bug, not a jitter verdict. 162.5 s. Flow submission succeeded: 4 images generated, all quality assertions (status, file count, magic bytes, aspect ratio) passed. Then crashed on assertion 5 len(batch_response_seen) == len(prompts) — got 8 events for 3 prompts. Manifest has a count=2 row, and same_project=1 multiplexes events per project, so 1-per-row invariant doesn't hold. See Aborted runs.
1 R1 1 (retry 2, after assertion fix) 1 (not captured) (not captured) (not captured) Dropped image under jitter=0 + same_project=1 + count=2. 324 s. Only 3 files produced when 4 expected; the second image of the count=2 row (prompt_1_1.png) is missing. Crashed on assertion 2 (file cardinality). All 3 returned outcomes had status=ok. Cannot distinguish "Flow generated 3" from "Flow generated 4 but listener missed 1" without an additional debug-dump rerun — but either way, this is exactly the rapid-fire failure jitter is designed to prevent.

Verdict

Matrix invalidated — no verdict reachable from the data collected. Earlier drafts of this section asserted KEEP with a partial-data argument; that conclusion is retracted because the premise of every cell turned out to be false.

Why retracted

The matrix was designed to compare image generation behaviour with vs without jitter while all prompts ran inside one shared Flow project (--same-project=1). Inspection of the user's Flow gallery after both R1 runs (Run 2 at 16:39 local, Run 3 at 16:57 local) shows each prompt landed in its own separate Flow project — not in the shared project the orchestration code created. The shared project (gflow-cli e2e at 05:01 PM) was created and then never used.

Root cause: src/gflow_cli/api/transports/ui_automation.py::generate_images accepts a project_id argument for Protocol-parity reasons but explicitly discards it (_ = project_id # accepted for Protocol parity; UI creates its own project) and runs the full "gallery → + New project → editor → submit → close" navigation on every call. So the --same-project=1 mode does not actually exist at the transport layer; every prompt creates a new Flow project regardless of the flag.

Consequences for this matrix:

  • Every cell would have been measuring rapid-fire across separate projects, not within one project. There is no shared-project scenario to compare against.
  • The R1 rep 1 second-retry observation ("only 3 of 4 images delivered") is unreliable evidence for or against jitter, because the prompts were not in the same project and the missing image was in its own separate project that no longer existed in our local references when the test crashed.
  • All decision-rule paths in spec §8 assume both cells run with the same shared-project semantics; that assumption does not hold.

What is actually decided (separate from the matrix)

The design intent the user articulated mid-session — that jitter exists for submission-cadence (anti-bot) rather than completion-wait — does stand on its own merits and is preserved in the project-memory record batch-submission-cadence. That rationale is being applied to the next branch scope, not as a verdict on this matrix.

What this matrix did NOT verify

Everything it was meant to verify. The data collected cannot answer the jitter question because the same-project condition was never satisfied.

Cumulative credit spend this matrix

~7 Flow image-credits (4 from Run 2, 3 from Run 3). Session 2 not entered. Five Flow projects exist on profile denon82 (denon82@gmail.com) from these runs; the user has been pointed at them to inspect manually.

Next step (no more matrix runs against this codebase)

The multi-image-prompt branch's scope is being revised: drop the --same-project=0 mode entirely, refactor ui_automation.generate_images to keep the editor mounted across all prompts in a batch (so all of them actually share one project), and treat jitter as a documented submission-cadence control. Spec and plan are being updated in the same session. The jitter matrix as designed is not being re-run; if a future investigation needs cadence tuning, it will be sized against the real same-project implementation.

Reproduce

Per-rep prompt variants prevent Flow-side caching of identical inputs:

Copy-Item test_assets/sample_batch.tsv tmp/sample_batch_rep1.tsv
Copy-Item test_assets/sample_batch.tsv tmp/sample_batch_rep2.tsv
Copy-Item test_assets/sample_batch.tsv tmp/sample_batch_rep3.tsv
(Get-Content tmp/sample_batch_rep1.tsv) -replace 'kitten', 'kitten #r1' | Set-Content tmp/sample_batch_rep1.tsv
(Get-Content tmp/sample_batch_rep2.tsv) -replace 'kitten', 'kitten #r2' | Set-Content tmp/sample_batch_rep2.tsv
(Get-Content tmp/sample_batch_rep3.tsv) -replace 'kitten', 'kitten #r3' | Set-Content tmp/sample_batch_rep3.tsv

$env:GFLOW_CLI_E2E_PROFILE = "ui_automation"

# R1 cell, rep 1 (same_project=1, jitter=0)
$env:GFLOW_CLI_E2E_BATCH_SAME_PROJECT = "1"
$env:GFLOW_CLI_E2E_BATCH_JITTER = "0"
$env:GFLOW_CLI_E2E_BATCH_MANIFEST = "tmp/sample_batch_rep1.tsv"
uv run pytest -q tests/e2e/test_image_batch_e2e.py 2>&1 | Tee-Object -FilePath tmp/r1_session1_rep1.log
# (repeat for rep 2, rep 3, then R2, then R3)

After the matrix:

$env:GFLOW_CLI_E2E_PROFILE = $null

Tested

(filled after runs — list every cell + rep with the relevant log path)

Invariants asserted (from tests/e2e/test_image_batch_e2e.py)

  • All outcomes status == "ok".
  • File cardinality == sum of manifest row counts.
  • Magic bytes: PNG, JPEG, or WebP (else fails loud).
  • Pillow dimensions ± 2 % of declared aspect_ratio.
  • ui_automation.batch_response_seen count == manifest row count.
  • image_batch.row_completed count == total image count.
  • image_batch.submission_attempt event present per row.
  • --same-project=1: single project ID across rows. --same-project=0: distinct project ID per row.

Correlation IDs

(filled after runs — project IDs and SHA256 prefixes captured from image_batch.row_completed events per cell)

NOT verified

  • Behaviour outside profile ui_automation.
  • Behaviour outside Chrome strategy.
  • Behaviour with count > 1 per row across --same-project=1 mode for prompts other than test_assets/sample_batch.tsv row 2.
  • Long-running rate-limit windows beyond the 2-hour cross-session gap.

Outputs

(filled after runs — pytest tmp_path artefacts; not committed; SHA256 prefixes recorded above)

Aborted runs (e2e bug or non-listener-miss failure)

2026-05-22T16:19:25Z — Session 1, R1 rep 1: AuthExpiredError: HTTP 401.

  • Cell config: same_project=1, jitter=0, manifest=tmp/sample_batch_rep1.tsv, profile=denon82 (substituted for ui_automation).
  • Symptom: gflow_cli.errors.AuthExpiredError: Authentication expired: HTTP 401 raised from client.create_project → _post_json("project.createProject"). Test failed in 12 s; no credits spent at Flow.
  • Classification: Pre-flight / infrastructure failure, not a jitter-cell verdict. The matrix never reached Flow submission. The §8 abort gate (which assumes the failure is observed under jitter=0 conditions) does not apply because no submission occurred.
  • Likely cause: denon82 profile cookies expired (last interactive use 2026-05-21 16:00 UTC, > 24 h ago); tests/e2e/test_image_batch_e2e.py invokes run_manifest_image_batch(..., transport=None, ...) so the default transport (API client via stored cookies) is used. Per memory image-generation-401-next, the v0.7.0 fix routed image generation through ui_automation transport — but the e2e test does not opt into it.
  • Resolution path (not done in this session):
    1. Refresh auth on the chosen profile via gflow auth login --profile <name> (interactive Chrome window) or swap to a freshly-logged-in profile.
    2. Optionally: verify whether run_manifest_image_batch's default transport is API-client or ui_automation. If API-client, consider whether the e2e test should be updated to pass transport="ui_automation" to mirror v0.7.0's resolution path. That is a code change, not a matrix-run decision.
    3. Re-run session 1 from R1 rep 1 with refreshed credentials. Session 2 timer (≥ 2 h after session 1 completes) starts at that point, not now.
  • Verdict impact: None yet. Matrix incomplete. Per §8 decision rule "Matrix incomplete (< 2 sessions) → default conservative: keep jitter", the conservative default still applies and #5b would be the docs-update KEEP variant if no further runs land.
  • Log: tmp/r1_session1_rep1.log (gitignored).

2026-05-22T16:42:13Z — Session 1, R1 rep 1 (retry after auth refresh): batch_response_seen over-count.

  • Cell config: identical to abort above.
  • Auth status: Refreshed successfully at 16:39 UTC (auth_flow_session_verified for denon82@gmail.com).
  • Run duration: 162.5 s. Flow submission completed; Flow billed for 4 image generations.
  • Test outcome: Quality assertions 1-4 passed (status == "ok", file count == 4 == sum of prompt counts, all PNG/JPEG/WebP magic bytes valid, all images within ±2 % of declared aspect ratio). Failed assertion 5: len(batch_response_seen) == len(prompts) — got 8 events, expected 3.
  • Classification: Test-assertion bug, not a jitter verdict. This is the inverse of the listener-miss flake (over-count, not under-count). All 8 events share the same filter_project_id (same-project mode), suggesting Flow emits multiple in-flight/complete events per image. With one count=2 row in the manifest, the actual image-event count is at least 4, plus per-image lifecycle events.
  • Why this blocks the matrix: Every R1/R2/R3 cell will hit the same assertion failure regardless of jitter setting. The matrix cannot distinguish "jitter unnecessary" from "test invariant wrong" while this assertion is over-strict.
  • Resolution path (not done in this session):
    1. Relax assertion 5 in tests/e2e/test_image_batch_e2e.py line 184 — likely to >= len(prompts) or >= sum(p.count for p in prompts). Tightening the lower bound preserves the "did we observe responses" signal while tolerating per-image / per-status multiplexing.
    2. Re-run R1 rep 1 with the relaxed assertion. If pass, continue the matrix.
    3. Or: simplify the manifest to all-count=1 rows for the matrix runs only (changes the credit cost from 4 images/rep to 3, but isolates the jitter signal from the count-mux question).
  • Credit accounting: ~4 images burned on this run. Cumulative session-1 cost so far: ~4 images.
  • Log: tmp/r1_session1_rep1.log (gitignored — contains the full assertion error and 8 captured event payloads).

Post-#5b verification

(filled in commit #5b's amend OR a follow-up edit — the e2e re-run under the verdict's chosen cell config per AC6)

Post-refactor live verification — Phase 7 (PARTIAL PASS, 2026-05-22)

Status: PARTIAL — race-condition fix verified; count-tab selector verification blocked by Issue #24 (locale-agnostic selectors).

What was verified ✅

  • Race-condition fix in c759c90 ("fix(image): batch-transport race in await loop short-circuit"). The Phase 3 await loop's if not captured and submit_error is None short-circuit incorrectly caught in-flight successful submissions. The fix changes the sentinel to detach is _noop_detach. Live e2e on profile denon82:

    • First attempt (with the bug): 30s wall time, 3 fail outcomes returned without raising, all-fail with no actual response-await. Visible at tmp/v3_7_live_verification.log and pytest-790.
    • After race fix: 49s wall time, submissions actually waited for responses, count=2 row generated 2 images correctly, count=1 rows generated 1+1 images each (1 expected + 1 from count drift — see blocked items below). Visible at tmp/v3_7_live_verification_retry.log and pytest-793.

  • Stay-mounted editor session works structurally. A single Flow project (gflow-cli e2e) was created once and reused for all prompts, with the editor page staying mounted across all submissions. Confirmed visually via the user's Flow gallery and via the image_batch.submission_attempt structlog events sharing one project_id per batch.

What's blocked by #24 ⏸️

  • Count selector read-back fails on non-English locales. The user's browser runs Flow in Portuguese (labs.google/fx/pt/tools/flow). Count tab labels render as "1 imagem", "2 imagens", etc. — not the x1/x2 English patterns the current selectors match. The instrumented _set_count in 18d184b returned 'imageImagem' (icon ligature image

    • Portuguese label Imagem) and exhausted its 3-attempt retry loop on the very first prompt, raising RuntimeError and triggering the orchestrator's BatchPartialError salvage path. Visible at tmp/v3_7_live_verification_retry3.log.

    Two earlier attempts to fix the count stickiness without locale-invariance also failed in different ways: 401aaf5 (force-reset by clicking x1 first) silently no-op'd and produced 5 files for 3 prompts (pytest-793); 18d184b (read-back verify with retry) raised the clear error above. Both are kept as commits — the read-back instrumentation in 18d184b was load-bearing in diagnosing the locale root cause and stays.

  • Final all-green e2e awaits #24 closing. The deliverable for #24 is locale-agnostic count-tab selectors (e.g., position-in-tablist, leading-digit regex, or aria attribute). Once #24 ships, a Phase 7b rerun on profile denon82 should pass cleanly.

Cumulative credit spend across Phase 7 attempts

~15-18 Flow image generations on profile denon82 (denon82@gmail.com), across the three e2e iterations referenced above. No production credits spent beyond what was already authorised in the matrix and Phase 7 budgets.

Conclusion

The v3-3 stay-mounted refactor lands the core bug fix for the --same-project=1 no-op: all prompts of a batch now actually share one Flow project, the editor page stays mounted, and per-prompt responses are captured correctly with race-immune detach semantics. The remaining gap is a pre-existing locale-portability issue (#24) that was previously unobserved because earlier verification ran on English-locale profiles. The branch is ready for review; a Phase 7b follow-up will close the loop once #24 lands.

Post-mode-switch-fix verification — 2026-05-23 (FULL PASS on ffroliva)

Status: FULL PASS — all eight test assertions green; user-confirmed gallery shows one project with four real images of the expected aspect ratios.

Bug discovered and fixed in this commit chain

Earlier Phase 7 conclusions implicitly assumed the editor always opened in Image mode. That assumption broke today on ffroliva: a prior unrelated session had left the editor in Video mode, and the image transport had no equivalent of ui_automation_video.py's _switch_to_video_mode. Submissions silently routed to the video endpoint, no batchGenerateImages response was observed, and the listener timed out after 3 minutes. This also explains the historical "first-attempt listener-miss flake" recorded in phase-b-followups memory item #1.

The fix mirrors the video-side pattern: UiAutomationTransport._switch_to_image_mode opens the 2-step mode dropdown via the shared MODE_SWITCH_TRIGGER_SELECTORS (imported from ui_automation_video), clicks the Image tab via the new IMAGE_TAB_IN_MENU_SELECTORS cascade (aria-controls*='IMAGE' first, locale text fallbacks after), presses Escape to close the menu, and logs ui_automation.image_mode_entered. Called from generate_images and _generate_images_batch_locked after _dismiss_blocking_overlays. The batch path wraps the call in the same orphaned_project_warning try/except as overlay dismissal.

Retraction of earlier-session claim

An earlier session in this branch's history claimed "Phase 7e produced 4 unique images end-to-end" based on file-size differences alone. That claim is hereby retracted. Inspection of the surviving pytest-825 / pytest-826 artifacts shows zero prompt_N_M.png files were written in those runs; the only PNGs present are count-tab diagnostic screenshots in _diagnostics/. The runs the earlier claim referenced were either reaped pytest tmp dirs or non-batch unit-test fixtures (8-byte magic-only PNGs in test_*0/out/). The first actually-verified 4-image batch run on this branch is pytest-836 / pytest-837 from 2026-05-23, recorded below.

Live run 1 — gflow image t2i (smoke test, 1 credit)

Layer Result Evidence
Exit code 0 tmp/ffroliva_postfix_t2i/run.log
File 2366d8f2-1109-487f-bcbf-0a35c260430d_1.png, 894,106 bytes magic ffd8ffe000104a46 (JPEG)
Pillow dims 768×1376, aspect 0.5581 (±0.79% of 9:16) within 2% threshold
image_mode_entered event fired structlog at 08:56:56Z after selector_matched probe=image_mode_tab [aria-controls*='IMAGE']
batch_response_seen status 200 OK on flowMedia:batchGenerateImages project 21910786-df7b-4005-9188-52717fe9960b
User gallery confirmation one project, one image (not a video) confirmed

Live run 2 — tests/e2e/test_image_batch_e2e.py (4 credits)

Assertion Result
Exit code 0
1. All outcomes status == "ok" passed
2. File cardinality == sum(p.count) == 4 passed (with _diagnostics/ excluded from rglob)
3. Magic bytes valid (PNG/JPEG/WebP) passed — all four JPEG
4. Pillow aspect-ratio per row, ±2% passed
5. ui_automation.batch_response_seen count ≥ sum(p.count) passed
6. image_batch.row_completed count == sum(p.count) (4) passed
7. image_batch.submission_attempt count == len(prompts) (3) passed
8. Shared project_id across submission_attempt events passed

pytest-837/out/ artifacts (4 distinct JPEGs in one batch):

File Size Dims Aspect Row config
prompt_0_0.png 828 KB 768×1376 9:16 ✓ row 0, count=1, PORTRAIT
prompt_1_0.png 1000 KB 1376×768 16:9 ✓ row 1, count=2, LANDSCAPE
prompt_1_1.png 908 KB 1376×768 16:9 ✓ row 1, count=2 (2nd)
prompt_2_0.png 1203 KB 1024×1024 1:1 ✓ row 2, count=1, SQUARE

User-confirmed: one project on ffroliva's Flow gallery contains exactly four real images of the correct aspect ratios.

Test-side fix

The e2e file-count assertion at tests/e2e/test_image_batch_e2e.py:142 counted every .png/.jpg under out/. Commit c5c8d4a relocated count-tab diagnostic screenshots into out/_diagnostics/, which broke the assertion (counted 10 instead of 4). The assertion now filters "_diagnostics" not in f.parts. No production-code change.

Still open

  • denon82 WAF/reCAPTCHA 403 on batchGenerateImages is unrelated to the mode-switch bug and untouched by this commit chain. Today's evidence (pytest-825/pytest-826/v3_7f/v3_7g) showed PUBLIC_ERROR_UNUSUAL_ACTIVITY from denon82's reCAPTCHA score — separate from the Image/Video mode confusion. The fact that ffroliva passed end-to-end today says WAF can be passed by a sufficiently fresh chrome-strategy profile; whether denon82 recovers with time or needs a new profile is a separate investigation.