make `--upstream_sync` also sync the gap analysis graph progressively

[northdpole]

Status

Maintainer-approved design — opportunistic gap-analysis (GA) fetch from production on access, replacing the prior idea of bulk-downloading the full ~9 GB GA graph via --upstream_sync. Implementation spec will land in a follow-up PR; this issue tracks the feature and acceptance criteria.

Supersedes bulk GA download as the primary local-dev path. Prateek's JIT Neo4j compute path remains an advanced fallback documented separately (not the default UI/REST path).

---

Problem

The gap-analysis cache is very large (~9 GB). Downloading it wholesale during --upstream_sync is unreliable and unnecessary for most local development: contributors typically need only the GA pairs they actually view.

Production (opencre.org) already serves precomputed GA rows from Postgres (cache-only; cache miss → 404). Local instances should reuse that upstream cache opportunistically instead of requiring a full bulk sync.

---

Proposed solution

When a user requests a GA pair (via UI or local REST GA API — same backend code path):

1. If the pair exists in the local GA store, serve it.
2. If not, and auto-fetch is enabled, fetch the full material row from prod (read-only HTTP client), upsert into the local GA store, then serve it.
3. While a fetch is in flight for a pair, return a "download in progress" state; concurrent requests for the same pair share a single in-flight fetch (frontend polls/waits).
4. On prod 404 (pair not in prod cache): show documented instructions for local backfill (links to opencre.org and/or GitHub docs — to be written as part of this work). Do not compute GA in the request path from the UI.
5. On prod unreachable or fetch failure: fall through to local cached row if present; otherwise show Try again (not local compute in UI).

GA auto-fetch is independent of --upstream_sync CRE graph sync. Locally cached pairs are preserved and not overwritten by a failed fetch.

---

Design decisions

1. Trigger: Both UI and local REST GA API use the same backend code path.
2. Prod URL: Configurable env var (default https://opencre.org). Prod down / miss behavior:
   • Prod unreachable (timeout, network error, exhausted retries): serve local cached row if it exists.
   • Not in local cache: show link to compute/backfill instructions (no UI compute).
   • Do not silently compute on prod-down in the UI/request path. Local compute remains CI/docs-only (advanced JIT Neo4j fallback), not a hidden fallback when prod is down.
3. Persist: Fetch the full material row from prod; upsert into the local GA store (gap_analysis_results / equivalent).
4. Concurrency: Single fetch per pair; backend returns "download in progress" while in-flight; frontend polls/waits.
5. 404 miss: Exact UX copy plus links to local backfill documentation (opencre.org and/or GitHub). Block local compute from UI for now.
6. Instructions: Link to docs to be written as part of this work — do not prescribe specific Makefile targets in this ticket.
7. Soft message: When both constituent standards exist locally and prod confirms they are valid GA participants, but the specific pair is missing on prod, show a soft CTA to file a ticket after the backfill instructions. Detection logic (all must hold):
   • Both standards exist locally in the CRE graph (nodes present for the pair's two standard names).
   • Both are valid GA participants — either:
     • the standards list endpoint includes both standards as GA-eligible, or
     • individual GET per standard succeeds against prod (or local graph metadata when prod standards list is unavailable).
   • GA request for that specific pair returns 404 or an error from prod (after retries exhausted).
   • When all three conditions hold, append the soft "file a ticket" CTA; otherwise show only the standard backfill instructions.
8. Refresh: Backend owns fetch lifecycle; frontend refresh sees the same state. Log when a local instance pulls a pair from prod.
9. Env disable: CRE_GA_NO_UPSTREAM=1 (or similar name) documented in .env.example — disables auto-fetch. On fetch failure: Try again button (not local calc in UI).
10. Independence: GA auto-fetch is independent of --upstream_sync CRE graph sync. Locally cached pairs are preserved (not overwritten by failed fetch).
11. Security: No auth on prod fetch; read-only HTTP client to prod GA endpoints; no writes to prod. State explicitly in implementation spec.
12. Configurable prod fetch (env vars): GA upstream HTTP client uses CRE_GA_UPSTREAM_* env vars, scoped parallel to the existing CRE_UPSTREAM_* pattern from upstream CRE graph sync (feat: retry transient failures during upstream sync #951):
    • CRE_GA_UPSTREAM_TIMEOUT_SECONDS — default 30 (Heroku-typical; mirrors CRE_UPSTREAM_TIMEOUT_SECONDS)
    • CRE_GA_UPSTREAM_MAX_ATTEMPTS — default 4 (mirrors CRE_UPSTREAM_MAX_ATTEMPTS)
    • CRE_GA_UPSTREAM_RETRY_BACKOFF_SECONDS — default 2 (mirrors CRE_UPSTREAM_RETRY_BACKOFF_SECONDS; optional but recommended for parity)
    • Exhaust retries before surfacing Try again in UI.

Cache policy

• V1: Once cached locally, always serve from local cache — no automatic re-fetch from prod on subsequent requests.
• V2 (future): Add TTL-based re-fetch from prod (e.g. env-controlled staleness window). Document as Version 2; out of scope for v1 implementation.

Open items (TBD — do not block implementation)

Item	Notes
V2 TTL / force-refresh	Deferred to Version 2 (see Cache policy)

---

Out of scope

• Bulk download of the full GA graph via --upstream_sync as the primary path (superseded by this design).
• Computing GA on production (prod remains cache-only).
• Computing GA in the UI/request path on cache miss (blocked; backfill via docs/CLI only).
• Prescribing exact Makefile targets in this ticket (docs PR will cover operational steps).
• JIT Neo4j compute as the default path (advanced fallback in docs only).
• V2: TTL-based automatic re-fetch from prod for locally cached pairs (see Cache policy).

---

Acceptance criteria

• UI and local REST GA API share one backend fetch/cache code path.
• Configurable prod base URL (default opencre.org); read-only prod HTTP client (no auth, no writes).
• On cache miss with auto-fetch enabled: fetch full material row from prod and upsert locally.
• V1 cache policy: once a pair is cached locally, subsequent requests serve from local cache without automatic prod re-fetch.
• On prod unreachable (after CRE_GA_UPSTREAM_MAX_ATTEMPTS retries): serve local cached row if present; otherwise Try again with compute/backfill instructions link (no UI-triggered compute).
• On prod 404 for a pair not in local cache: documented backfill instructions (opencre.org + GitHub links); no local compute from UI.
• Single in-flight fetch per pair; concurrent requests get "download in progress"; frontend polls/waits.
• Soft CTA to file a ticket when soft-message detection (make frontend show Graph #7) conditions are met (both standards local + valid GA participants + prod pair 404/error).
• CRE_GA_NO_UPSTREAM=1 (or agreed name) disables auto-fetch; documented in .env.example.
• CRE_GA_UPSTREAM_TIMEOUT_SECONDS default 30; CRE_GA_UPSTREAM_MAX_ATTEMPTS default 4; documented in .env.example (and implementation spec).
• Failed fetch does not overwrite existing local cached row.
• Logging when local instance pulls a pair from prod.
• Tests: mocked prod HTTP, cache upsert, in-progress lock, 404 UX copy, env disable flag, read-only prod client behavior, logging, env defaults (CRE_GA_UPSTREAM_TIMEOUT_SECONDS, CRE_GA_UPSTREAM_MAX_ATTEMPTS), soft-message detection logic (positive/negative cases), V1 no re-fetch (cached pair served without second prod call).

---

Dependencies

• Production GA REST API serving precomputed material rows (existing on opencre.org).
• Local GA store schema (gap_analysis_results / equivalent) supporting upsert of material rows.
• Documentation PR: local backfill instructions, env vars, advanced JIT Neo4j fallback (not default).
• Detailed implementation spec PR (referenced from this issue).

---

Note

A detailed implementation spec will land in a follow-up PR linked from this issue. Volunteers should read this body (and the spec PR when available) before claiming the issue.

[kart-u]

hello @northdpole is this issue still open for work , I would love to contribute in it!
I like issues involving optimizations :-)

[zenowinged]

@northdpole i'd like to work on this issue. could you please assign it to me?

[sd2604]

Hey if this issue is still available, could you pls assign this to me, I'd realy like to work on this issue...

[IND-Anshuman]

Hi! @northdpole I'd like to work on this issue. I've analyzed the codebase and have a clear plan to solve the problem.

The Problem I identified: Currently, --upstream_sync only downloads the CRE graph structure but ignores the gap analysis data, which is ~9GB and impractical to download reliably for local development.

My Solution: First, I'll extend the donwload_graph_from_upstream() function in application/cmd/cre_main.py to progressively sync gap analysis data after downloading the CRE structure. It will get the list of standards from upstream, go through standard pairs, and download their gap analysis results incrementally while skipping already-cached data. I can also implement lazy loading by modifying the schedule() function in application/utils/gap_analysis.py to get the gap analysis from upstream on-demand before falling back to expensive local calculations. This means the system will first try to grab pre-calculated results from upstream , only calculating locally if upstream is unavailable or doesn't have the data.

Could you please assign this issue to me? Thanks!

[ashutosh-engineer]

@northdpole While exploring and fixing the Issue of #469 I am also exploring the Solution for this issue.
Will Share the report soon.