[codex] Add forkable Design Studio lineages

.env.example

@@ -20,6 +20,14 @@ ETSY_SHOP_ID=
 ETSY_REDIRECT_URI=http://localhost:8000/api/etsy/oauth/callback
 ETSY_OAUTH_STATE_TTL_SECONDS=600
 
+# Optional GA4 read-only source for first-party traffic validation.
+GOOGLE_CLIENT_ID=
+GOOGLE_CLIENT_SECRET=
+# Generate once with the same Fernet command used for Etsy above.
+GOOGLE_TOKEN_ENCRYPTION_KEY=
+GOOGLE_ANALYTICS_REDIRECT_URI=http://localhost:8000/api/research-sources/google-analytics/oauth/callback
+GOOGLE_OAUTH_STATE_TTL_SECONDS=600
+
 PRINTFUL_ACCESS_TOKEN=
 PRINTFUL_STORE_ID=
 PRINTFUL_ETSY_STORE_ID=
@@ -48,3 +56,7 @@ OPENAI_IMAGE_WIDTH=1024
 OPENAI_IMAGE_HEIGHT=1024
 OPENAI_IMAGE_MAX_COST_USD=0.50
 OPENAI_IMAGE_REFERENCE_COST_RESERVE_USD=0.06
+
+RESEARCH_TREND_LIMIT=8
+RESEARCH_POLL_SECONDS=5
+RESEARCH_MAX_TOOL_CALLS=12

.githooks/pre-commit

@@ -6,3 +6,4 @@ cd "$root"
 
 make lint
 make format-check
+make typecheck

README.md

@@ -2,12 +2,12 @@
 
 Python 3.12 MVP for a reviewable Etsy and Printful product workflow:
 
-`trend research -> topic approval -> design -> design approval -> listing copy approval -> Printful product -> mock Etsy draft -> manual publish -> monitoring`
+`trend discovery -> optional research report -> design -> design approval -> listing copy approval -> Printful product -> Etsy draft -> manual publish -> monitoring`
 
-Etsy remains mocked. Printful supports real authenticated catalog reads and an
-operator-driven Workshop for curation, mockups, and sync-product creation; all real
-writes require a separate feature flag and S3-compatible asset storage. OpenAI image
-generation can still be enabled independently.
+Etsy supports authenticated draft-listing workflows. Printful supports real catalog
+reads and an operator-driven Workshop for curation, mockups, and sync-product
+creation; all real writes require separate safeguards. OpenAI image generation and
+web-grounded research can be enabled independently.
 
 The architecture rationale and the decisions behind it are documented in
 [ARCHITECTURE_REVIEW.md](ARCHITECTURE_REVIEW.md).
@@ -112,11 +112,15 @@ The MVP is local-only and has no authentication. Do not expose ports remotely. A
 
 ## Data Model
 
-Ten tables, all created by Alembic:
+Core tables created by Alembic include:
 
 | Table | Holds |
 |---|---|
-| `trends` | One row per researched trend candidate, with embedded scores and a review status |
+| `trend_discovery_runs` | Durable manual scans for emerging apparel trends |
+| `trends` | Trend candidates with evidence, apparel scores, confidence, and risk warnings |
+| `research_runs` | In-depth research tasks, structured output, and rendered Markdown |
+| `research_source_configs` | Selected optional research sources such as GA4 |
+| `etsy_stats_imports` | Deduplicated manual imports of Etsy shopper search terms |
 | `products` | The full concept → design → copy → listings → publish lifecycle of one product |
 | `design_assets` | Every generated design revision with its brief, checksum, and review outcome |
 | `approval_events` | Append-only human decisions with unique idempotency keys |
@@ -158,6 +162,41 @@ before generation when its configured output-cost estimate exceeds the per-image
 limit. Dollar amounts are estimates because the generation response reports token
 usage rather than the final billed amount.
 
+## Trend Research And Reports
+
+The Trend Inbox starts manual discovery scans. In real OpenAI mode, scans and reports
+use Responses API web search in background mode, store verified source URLs, and
+continue through the durable job queue while the dashboard is closed. Mock mode
+provides deterministic local results.
+
+Each trend includes a short explanation, 1–5 apparel score, confidence, evidence, and
+separate trademark, copyright, cultural, and marketplace-policy warnings. Selecting
+**Research** opens an editable topic form. Manual topics use the same form. Completed
+reports render consistently in the dashboard and download as Markdown.
+
+Research context can include Etsy API listing and transaction signals, manually
+imported Etsy Stats search terms, an optional read-only GA4 property, and cached
+Printful products, prices, placements, and techniques.
+
+To connect GA4, create a Google OAuth web client, enable the Google Analytics Admin
+and Data APIs, add the callback as an authorized redirect URI, and set:
+
+```text
+GOOGLE_CLIENT_ID=...
+GOOGLE_CLIENT_SECRET=...
+GOOGLE_TOKEN_ENCRYPTION_KEY=...  # Fernet key
+GOOGLE_ANALYTICS_REDIRECT_URI=https://api.workshop.example.com/api/research-sources/google-analytics/oauth/callback
+```
+
+Then connect Google Analytics and select a property from Research Reports. GA4 is a
+first-party validation signal, not broad market discovery.
+
+Etsy shopper search terms remain in Shop Manager Stats rather than the current public
+API. Research Reports explains how to open **Shop Manager → Stats → How shoppers
+found you → Etsy search**, download Workshop's CSV template, and upload the copied
+terms and visit counts. Google Trends is disabled because its official API remains
+limited-access alpha.
+
 ## Tests
 
 ```bash

TODO.md

@@ -140,7 +140,11 @@ duplicate actions or unexpected charges.
   persist bytes through `AssetStore`; record model, size, usage, cost, checksum.
 - [x] Per-image budget, top-level kill switch, provider mode, safe transient retries,
   and explicit unknown-outcome handling for ambiguous timeouts.
-- [x] Durable standalone Design Studio for prompt iteration and generation history.
+- [x] `APP_ENV=test` rejects `OPENAI_MODE=real`, so local smoke tests cannot inherit
+  paid OpenAI settings from `.env` by accident.
+- [x] Durable standalone Design Studio for prompt iteration, collapsed lineage,
+  shared design titles, v1/v2/v3 tracking, prompt previews, and configurable aspect
+  ratios.
 - [ ] Daily image-generation budget and promotion of a selected playground image into
   the product design queue.
 - [x] Promotion of Design Studio and approved workflow assets into an independent
@@ -152,6 +156,30 @@ duplicate actions or unexpected charges.
 - [ ] Basic print-readiness checks (dimensions, transparency, DPI) and human review of
   every asset (already enforced by the design gate).
 
+### Trend research and reports
+
+- [x] Durable manual trend-discovery runs with emerging-theme blurbs, evidence,
+  confidence, 1–5 apparel recommendations, and separate IP/policy warnings.
+- [x] Shared editable research form for manual topics and trend-prefilled topics.
+- [x] Durable background research runs using structured output, verified web sources,
+  bounded transient-rate-limit retries, consistent Markdown rendering, in-app display,
+  and download.
+- [x] Live-source validation keeps only URLs recorded by OpenAI web search, rejects
+  results with no verified sources, and reuses completed provider responses when a
+  failed report is retried.
+- [x] Research context from Etsy API signals, imported Etsy Stats search terms, cached
+  Printful products, and optional read-only GA4 data.
+- [x] Etsy Stats CSV template, validation, deduplication, import history, and in-app
+  instructions for collecting terms from Shop Manager.
+- [x] Google Analytics OAuth, token refresh, accessible-property selection, source
+  status, and recent-versus-prior reporting.
+- [x] Google Trends integration is documented as disabled while the official API
+  remains limited-access alpha; it is not a research dependency.
+- [ ] Review discovery quality and OpenAI cost/latency after several real shop runs;
+  tune the trend limit, tool-call limit, and model only from observed results.
+- [ ] Add scheduled trend discovery after manual scan quality and operating cost are
+  understood.
+
 ### Notifications (Slack, after the dashboard works remotely)
 
 - [ ] `BLOCKED` Single-workspace Slack app; bot token + signing secret stored securely.

migrations/versions/0009_research_subsystem.py

@@ -0,0 +1,245 @@
+"""Add durable trend discovery and research reports.
+
+Revision ID: 0009
+Revises: 0008
+Create Date: 2026-06-15
+"""
+
+import sqlalchemy as sa
+from alembic import op
+from sqlalchemy.dialects import postgresql
+
+revision = "0009"
+down_revision = "0008"
+branch_labels = None
+depends_on = None
+
+JSON = postgresql.JSONB(astext_type=sa.Text())
+
+
+def _identity_columns(*, versioned: bool = False) -> list[sa.Column]:
+    columns = [
+        sa.Column("id", sa.Uuid(), nullable=False),
+        sa.Column(
+            "created_at",
+            sa.DateTime(timezone=True),
+            server_default=sa.text("now()"),
+            nullable=False,
+        ),
+        sa.Column(
+            "updated_at",
+            sa.DateTime(timezone=True),
+            server_default=sa.text("now()"),
+            nullable=False,
+        ),
+    ]
+    if versioned:
+        columns.append(sa.Column("version", sa.Integer(), nullable=False))
+    columns.append(sa.PrimaryKeyConstraint("id"))
+    return columns
+
+
+def upgrade() -> None:
+    op.alter_column(
+        "oauth_credentials",
+        "provider",
+        type_=sa.String(32),
+        existing_type=sa.String(8),
+        existing_nullable=False,
+    )
+    op.create_table(
+        "google_oauth_states",
+        sa.Column("state_digest", sa.String(64), nullable=False),
+        sa.Column("code_verifier_ciphertext", sa.Text(), nullable=False),
+        sa.Column("redirect_uri", sa.Text(), nullable=False),
+        sa.Column("scopes", JSON, nullable=False),
+        sa.Column("expires_at", sa.DateTime(timezone=True), nullable=False),
+        sa.Column("consumed_at", sa.DateTime(timezone=True)),
+        *_identity_columns(),
+        sa.UniqueConstraint("state_digest"),
+    )
+    op.create_index(
+        "ix_google_oauth_states_state_digest",
+        "google_oauth_states",
+        ["state_digest"],
+    )
+    op.create_index(
+        "ix_google_oauth_states_expires_at",
+        "google_oauth_states",
+        ["expires_at"],
+    )
+
+    op.create_table(
+        "trend_discovery_runs",
+        sa.Column("status", sa.String(16), nullable=False),
+        sa.Column("provider_response_id", sa.String(255)),
+        sa.Column("input_snapshot", JSON, nullable=False),
+        sa.Column("structured_result", JSON, nullable=False),
+        sa.Column("error_message", sa.Text()),
+        sa.Column("started_at", sa.DateTime(timezone=True)),
+        sa.Column("finished_at", sa.DateTime(timezone=True)),
+        *_identity_columns(),
+    )
+    op.create_index("ix_trend_discovery_runs_status", "trend_discovery_runs", ["status"])
+
+    op.add_column("trends", sa.Column("discovery_run_id", sa.Uuid()))
+    op.add_column(
+        "trends",
+        sa.Column("blurb", sa.Text(), server_default=sa.text("''"), nullable=False),
+    )
+    op.add_column("trends", sa.Column("apparel_score", sa.Numeric(3, 2)))
+    op.add_column(
+        "trends",
+        sa.Column(
+            "score_breakdown",
+            JSON,
+            server_default=sa.text("'{}'::jsonb"),
+            nullable=False,
+        ),
+    )
+    op.add_column("trends", sa.Column("confidence", sa.Numeric(3, 2)))
+    op.add_column(
+        "trends",
+        sa.Column(
+            "verified_sources",
+            JSON,
+            server_default=sa.text("'[]'::jsonb"),
+            nullable=False,
+        ),
+    )
+    op.add_column(
+        "trends",
+        sa.Column(
+            "risk_warnings",
+            JSON,
+            server_default=sa.text("'[]'::jsonb"),
+            nullable=False,
+        ),
+    )
+    op.create_foreign_key(
+        "fk_trends_discovery_run_id",
+        "trends",
+        "trend_discovery_runs",
+        ["discovery_run_id"],
+        ["id"],
+    )
+    op.create_index("ix_trends_discovery_run_id", "trends", ["discovery_run_id"])
+
+    connection = op.get_bind()
+    rows = connection.execute(
+        sa.text(
+            "SELECT research_batch_id, MIN(created_at), MAX(updated_at) "
+            "FROM trends GROUP BY research_batch_id"
+        )
+    ).fetchall()
+    for batch_id, created_at, updated_at in rows:
+        connection.execute(
+            sa.text(
+                "INSERT INTO trend_discovery_runs "
+                "(id, status, input_snapshot, structured_result, created_at, updated_at, "
+                "finished_at) VALUES "
+                "(:id, 'completed', '{\"legacy\": true}', '{}', :created_at, :updated_at, "
+                ":updated_at)"
+            ),
+            {
+                "id": batch_id,
+                "created_at": created_at,
+                "updated_at": updated_at,
+            },
+        )
+        connection.execute(
+            sa.text("UPDATE trends SET discovery_run_id = :id WHERE research_batch_id = :id"),
+            {"id": batch_id},
+        )
+
+    op.create_table(
+        "research_runs",
+        sa.Column("trend_id", sa.Uuid()),
+        sa.Column("topic_title", sa.String(255), nullable=False),
+        sa.Column(
+            "additional_context",
+            sa.Text(),
+            server_default=sa.text("''"),
+            nullable=False,
+        ),
+        sa.Column("status", sa.String(16), nullable=False),
+        sa.Column("provider_response_id", sa.String(255)),
+        sa.Column("input_snapshot", JSON, nullable=False),
+        sa.Column("structured_result", JSON, nullable=False),
+        sa.Column("markdown_report", sa.Text()),
+        sa.Column("error_message", sa.Text()),
+        sa.Column("started_at", sa.DateTime(timezone=True)),
+        sa.Column("finished_at", sa.DateTime(timezone=True)),
+        *_identity_columns(),
+        sa.ForeignKeyConstraint(["trend_id"], ["trends.id"]),
+    )
+    op.create_index("ix_research_runs_trend_id", "research_runs", ["trend_id"])
+    op.create_index("ix_research_runs_status", "research_runs", ["status"])
+
+    op.create_table(
+        "research_source_configs",
+        sa.Column("provider", sa.String(50), nullable=False),
+        sa.Column("enabled", sa.Boolean(), nullable=False),
+        sa.Column("config", JSON, nullable=False),
+        *_identity_columns(versioned=True),
+        sa.UniqueConstraint("provider"),
+    )
+
+    op.create_table(
+        "etsy_stats_imports",
+        sa.Column("filename", sa.String(255), nullable=False),
+        sa.Column("checksum", sa.String(64), nullable=False),
+        sa.Column("period_start", sa.DateTime(timezone=True), nullable=False),
+        sa.Column("period_end", sa.DateTime(timezone=True), nullable=False),
+        sa.Column("row_count", sa.Integer(), nullable=False),
+        sa.Column("rows", JSON, nullable=False),
+        sa.Column("warnings", JSON, nullable=False),
+        *_identity_columns(),
+        sa.UniqueConstraint("checksum"),
+    )
+    op.create_index("ix_etsy_stats_imports_checksum", "etsy_stats_imports", ["checksum"])
+
+    op.add_column("jobs", sa.Column("available_at", sa.DateTime(timezone=True)))
+    op.create_index("ix_jobs_available_at", "jobs", ["available_at"])
+    op.drop_index("uq_jobs_active", table_name="jobs")
+    op.create_index(
+        "uq_jobs_active",
+        "jobs",
+        ["job_type", "subject_id"],
+        unique=True,
+        postgresql_where=sa.text("status IN ('queued', 'waiting', 'running')"),
+    )
+
+
+def downgrade() -> None:
+    op.drop_index("uq_jobs_active", table_name="jobs")
+    op.create_index(
+        "uq_jobs_active",
+        "jobs",
+        ["job_type", "subject_id"],
+        unique=True,
+        postgresql_where=sa.text("status IN ('queued', 'running')"),
+    )
+    op.drop_index("ix_jobs_available_at", table_name="jobs")
+    op.drop_column("jobs", "available_at")
+    op.drop_table("etsy_stats_imports")
+    op.drop_table("research_source_configs")
+    op.drop_table("research_runs")
+    op.drop_index("ix_trends_discovery_run_id", table_name="trends")
+    op.drop_constraint("fk_trends_discovery_run_id", "trends", type_="foreignkey")
+    op.drop_column("trends", "risk_warnings")
+    op.drop_column("trends", "verified_sources")
+    op.drop_column("trends", "confidence")
+    op.drop_column("trends", "score_breakdown")
+    op.drop_column("trends", "apparel_score")
+    op.drop_column("trends", "blurb")
+    op.drop_column("trends", "discovery_run_id")
+    op.drop_table("trend_discovery_runs")
+    op.drop_table("google_oauth_states")
+    op.alter_column(
+        "oauth_credentials",
+        "provider",
+        type_=sa.String(8),
+        existing_type=sa.String(32),
+        existing_nullable=False,
+    )