P1: Unblock Heroku prod deploy — split requirements + aggressive slug strip

[northdpole]

Summary

Prod deploy to opencreorg is blocked: Heroku rejects the slug at 3.1G (max 1000M). Root cause is sentence-transformers pulling CUDA PyTorch into the Heroku build via requirements.txt.

Latest failure: https://github.com/OWASP/OpenCRE/actions/runs/29030616421

Compiled slug size: 3.1G is too large (max is 1000M).
Push rejected to opencreorg.

Goal

Unblock Heroku prod deploy with a tiny prod slug: no torch/CUDA, and no importer/LLM/test tooling on the dyno. Prod serves cache-only GA + REST + auth + worker queue.

Product decisions (confirmed for this ticket)

1. Chat/completion on opencreorg — OK if it 500s without litellm (compute/chat tooling is local/dev).
2. OSCAL export / Google Sheets import on prod — CLI/local only; not required on the dyno.
3. Worker jobs that call embeddings/import on Heroku may fail — matches “compute locally, prod is cache-only”.

Plan (approved)

Files

Action	Path
Rewrite	requirements.txt (prod, minimal)
Create	requirements-dev.txt (-r requirements.txt + everything else)
Update	Makefile, Dockerfile, Dockerfile-dev, .github/workflows/linter.yml, scripts/run-local.sh, scripts/update-cwe.sh
Fix boot imports	cre.py, application/web/web_main.py, application/cmd/cre_main.py, application/utils/spreadsheet.py (and any other top-level pulls of heavy deps)

Prod requirements.txt (target)

Keep only what web/worker/release need after import fixes:

# server
gunicorn
Flask
Flask-Caching
flask-compress
Flask-Cors
Flask-Migrate
Flask-SQLAlchemy
SQLAlchemy
psycopg2-binary
setuptools
click
python-dotenv

# queue / cache
redis
rq

# graph / DB helpers used by db.py
neo4j
neomodel
networkx
PyYAML==6.0.1

# auth + HTTP
requests
oauthlib
google
google-api-core
google_auth_oauthlib

# OpenAPI / schemas used by web
marshmallow
apispec
pydantic>=2.4.0,<3
dacite
semver
python-markdown-maker

# optional analytics (env-gated)
posthog

typing_extensions

Dev requirements-dev.txt

-r requirements.txt

# ML / Librarian
sentence-transformers>=5.0.0,<6.0.0

# embeddings / chat / import tooling (not on Procfile)
litellm
openai
vertexai
google-cloud-aiplatform
google-cloud-trace
grpcio
grpcio_status
playwright
nltk
numpy
scipy
scikit-learn
beautifulsoup4
pypdf
alive-progress
compliance-trestle
gspread
GitPython
PyGithub
xmltodict
openpyxl
Pillow
# … remaining freeze noise + lint/test (black, mypy, pytest*, coverage, etc.)

Import fixes (required for aggressive strip)

Without these, dynos fail on missing modules at boot:

1. cre.py — lazy-import coverage (keep module load free of test-only deps)
2. web_main.py — stop top-level import cre_main / alive_progress / OSCAL / spreadsheet-gspread; lazy where used
3. cre_main.py — stop top-level prompt_client import; lazy inside register/embeddings/librarian
4. spreadsheet.py — don’t import gspread at module top if CSV export is the only prod path

Why: today gunicorn cre:app → web_main → cre_main → prompt_client eagerly loads nltk / playwright / sklearn. Chat’s “lazy import” comment is wrong relative to current boot.

Steps

1. Break eager import chains (above)
2. Write slim prod + fat dev requirement files
3. Point local/CI/Docker-dev at requirements-dev.txt; Heroku stays on requirements.txt (buildpack default)
4. Run lint/mypy/tests
5. Redeploy — expect slug ≪ 1 GB

Success criteria

• requirements.txt has no sentence-transformers, torch, playwright, nltk, scikit-learn, litellm, Google AI SDKs, test/lint tools
• make lint / make mypy / make test pass with requirements-dev.txt
• App imports cleanly with only prod deps (smoke: python -c "from application import create_app; create_app()" in a prod-only venv)
• Heroku deploy slug < 1000M (Deploy to OPENCREORG green)

Risks

• Chat / OSCAL / Sheets paths on prod will 500 until those deps are present — accepted for this ticket
• Over-stripping a transitive dep → add back one package
• Worker embeddings/import on Heroku will fail — accepted (local compute)

Out of scope

• CPU-only torch pin
• Changing Librarian feature behavior beyond import laziness
• Removing neo4j from prod (still imported by db.py)
• Expanding .slugignore as the primary fix (nice-to-have only)

Context / prior work

• Chat plan approved 2026-07-10; action first thing next day
• Related: Module C Librarian added sentence-transformers to the shared requirements file
• Pattern: Heroku uses requirements.txt by default; no deploy.yml change needed if prod file stays named that way

Verification commands

make lint
make mypy
make test
# after merge / deploy:
# confirm Actions Deploy to OPENCREORG succeeds and slug < 1000M

[northdpole]

Follow-on for lighter prod chat matching (pgvector instead of numpy/scipy/sklearn on the dyno): #977.