Skip to content

Speed up deploy preparation and add opt-in frontend build reuse - #7054

Open
Alek99 wants to merge 3 commits into
mainfrom
codex/faster-deploy
Open

Alek99 wants to merge 3 commits into
mainfrom
codex/faster-deploy

Conversation

@Alek99

@Alek99 Alek99 commented Sep 6, 2026

Copy link
Copy Markdown
Member

Repeated deploys redo frontend dependency and archive work, and unchanged production frontends always rerun Vite. This PR reuses matching installed framework pins, avoids unnecessary manifest rewrites and sidecar recompression, and adds opt-in reuse of deterministic production frontend builds.

Changes

  • Retain equivalent package manifests without rewriting them, preserving main's deterministic dependency-only install fingerprint.
  • Reuse matching framework pins after a successful persisted-lock install. Verify npm caret declarations against both lock metadata and installed package versions, use --save-dev, and include development dependencies under NODE_ENV=production.
  • Cache archive exclusion identities and avoid recompressing frontend gzip/Brotli/Zstandard sidecars. Keep normal frontend/backend files deflated.
  • Preserve main's per-package 24-hour version-check cache and one-hour failure cooldown; handle legacy/local and UTC timestamps, reject failed HTTP responses, and skip provider lookup when no provider prompt is needed. Required hosting compatibility checks remain active.
  • Add REFLEX_FRONTEND_BUILD_CACHE=true, disabled by default. Cache pristine Vite output and replay post-build plugins, fallback handling, compression, and frontend-path processing. Validate inputs/output, invalidate changed inputs, fall back on unsupported/corrupt snapshots, and publish only after a successful complete build.

Current rebase validation (87ea648d)

Rebased onto main at 77cfe61b on September 8, 2026. The rebase preserves main's lazy CLI startup, dependency-only install fingerprint, per-package version checks, and failure cooldown.

  • Production workspace locking now also works on Windows, even with build reuse disabled. It waits through contention, propagates other acquisition failures, supports nested same-thread calls, and releases after failed builds. Opt-in build reuse itself remains macOS/Linux only.
  • Build fingerprints ignore the new per-package version-check and attempt timestamps, avoiding needless cache misses after routine metadata updates.
  • Regression failures were captured before fixing Windows lock bypass and timestamp-driven cache invalidation. Separate spawned-process locking tests run on all CI platforms with caching both enabled and disabled.
  • The final 87ea648d follow-up only consolidates documentation and adds duration comments. Its Python AST is identical to the fully tested b9230726 revision.
  • Full final local unit suite: 8,479 passed, 18 skipped; 76.45% coverage on Python 3.14.5. An earlier run alongside other checks hit a Redis lock-expiry timing failure; the isolated final suite passed without changes to the Redis implementation or that test.
  • Focused rebase/cache/deploy suite: 341 passed; final cache/prerequisites check after the platform-guard adjustment: 204 passed.
  • Real production build with caching enabled: 9 passed, covering gzip/Brotli/Zstandard sidecars and HTTP content negotiation; 9 development-mode cases intentionally skipped.
  • All repository pre-commit hooks passed: Ruff, codespell, stub generation, Pyright, ty, and Biome.

Historical local deploy measurements (93c8683, before rebase)

These measurements predate this rebase. Main now includes independent startup and dependency-cache improvements, so these numbers are not a performance comparison against current main.

Same nine-page dashboard, two warmups and three fresh-process samples per variant, rotating run order. Times include imports, preparation, compilation/dependency handling, build or restore, postprocessing/compression, and both ZIPs. They exclude cloud authentication/provider APIs, upload, remote builds/rollout, optional version requests, telemetry delivery, interpreter teardown, and artifact verification.

Unchanged frontend Original New default Build cache enabled
npm 5.792 s 2.533 s (56.3% less time) 1.248 s (78.4% less time)
Bun 2.551 s 2.464 s (3.4% less time) 1.244 s (51.2% less time)

All unchanged candidate runs skipped dependency-manager commands; enabled-cache runs also skipped Vite. Python hash seeds vary between interpreters while the child build environment remains constant.

Changed API/deploy URLs Original New default Build cache enabled, miss
npm 5.787 s 3.390 s 4.188 s
Bun 2.541 s 2.510 s 3.299 s

Cache misses add about 0.8 seconds over the new default. The cache is for repeat deterministic builds on local macOS/Linux filesystems with reliable file timestamps; Windows bypasses it. External/clock/remote build inputs and excluded generated caches require a fresh build. Set REFLEX_FRONTEND_BUILD_CACHE=false to discard the snapshot and rebuild. Source/snapshot contents are hashed; installed dependencies use identity, size, mtime, and ctime. Documentation covers ignored private telemetry timestamps and symlink handling.

Measured on macOS arm64, Python 3.14.5, Node 26.0.0, npm 11.12.1, Bun 1.4.0 against base c49a85df9c41ddffbe9fc2ac37d495d363f4e313. No competing builds/tests ran during timing. These are warm local results, not total cloud deploy latency. Small differences can be noise. The original npm changed-URL samples were 8.489, 5.787, and 5.776 seconds; all are retained in the median. Avoiding sidecar recompression increased the earlier dashboard ZIP by 0.063%; upload impact was not measured.

Earlier validation before rebase

  • Full unit suite: 8,441 passed, 18 skipped, 76.38% coverage (72% required).
  • Production integration: 14 passed each with Bun and npm, with caching enabled; real JS/CSS compression and HTTP negotiation plus Chromium chart rendering/state updates. Each manager skipped 9 development cases and deselected 5 development browser cases.
  • Fixed-clock real exports cover unchanged repeats, URL/Python/CSS/SVG mutations, and dependency addition/removal. 96 cross-variant archive comparisons plus 48 reference checks verify raw extracted payloads, ordered names, counts, modes, current assets/content, and actual cache hit/miss commands. npm comparisons permit dependency-section moves and dev/devOptional flags while requiring other manifest/lock metadata and resolved versions to agree.
  • The archive matrix preceded the last symlink-redirect invalidation fix; its new failing-before/passing-after regression and the final full unit/production suites cover that correction. Earlier validation also covered clean .web restoration and npm NODE_ENV=production.
  • Ruff, formatting, pyright reflex tests, whitespace checks, and patch application against the base pass. All applicable pre-commit hooks pass, including codespell, stub generation, Pyright, and ty. Independent review identified and resolved serializer and symlink edge cases. Regression tests, documentation, and news fragments for each touched package are included.

Full cloud deployment and arbitrary concurrent workspace writers (initialization/dev hot reload/external tools) were not exercised. The new concurrency regressions use separate processes with controlled compiler/build/ZIP test doubles; production behavior is also checked by the Bun/npm integration tests.

Review in cubic

Earlier review follow-up (4da5154, before rebase)

  • Serialize initialized production exports from environment/URL configuration through compilation, setup, build/cache operations, and ZIP creation. Production/preview startup uses the same reentrant POSIX lock through frontend build and releases it before serving; direct builds and cache-disabled calls participate too. Forked children drop inherited lock ownership safely. Lock acquisition failures stop the operation. Initialization, dev hot reload, and unrelated workspace writers remain outside this scope.
  • Bump the cache schema to discard snapshots created before locking.
  • Expire successful version checks after 24 hours; expired, malformed, and future timestamps refresh. Preserve legacy local timestamps, write UTC for new checks, and keep network/HTTP/parse failures retryable.
  • Added regressions for process overlap, distinct export URLs, ZIP interleaving, mixed cache settings, reentrancy/fork cleanup, unsafe/failed locks, expiry boundaries, timezone handling, and retries. 8,441 unit tests, 28 production integration checks, and all applicable pre-commit hooks pass.

Candidate-only follow-up on the same warmed unchanged-frontend fixture (two warmups, three fresh-process runs per variant, alternating order):

Manager Default Build reuse enabled
npm 2.601 s 1.253 s
bun 2.527 s 1.268 s

All measured repeats skipped dependency-manager commands; enabled-cache repeats skipped Vite and reran compression. The original baseline and changed-URL cases were not rerun in this follow-up; the earlier table remains the initial revision's comparison. Timing still covers local preparation through ZIP creation, excluding cloud and interpreter teardown.

@Alek99
Alek99 requested a review from a team as a code owner September 6, 2026 22:58
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

@codspeed

codspeed Bot commented Sep 6, 2026

Copy link
Copy Markdown

Merging this PR will not alter performance

✅ 40 untouched benchmarks
⏩ 8 skipped benchmarks1


Comparing codex/faster-deploy (87ea648) with main (77cfe61)

Open in CodSpeed

Footnotes

  1. 8 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

@greptile-apps

greptile-apps Bot commented Sep 6, 2026

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

The PR appears safe to merge; no actionable new issue or outstanding previous finding remains.

Summary

  • Serializes production frontend workspace operations across exports and production/preview startup.
  • Validates and safely restores cached pristine frontend output before replaying post-build processing.
  • Reuses verified frontend dependency pins and preserves development dependencies under production npm environments.
  • Avoids unnecessary provider lookups and recompression of precompressed frontend sidecars.
  • The changes since the previous review resolve the outstanding duration-comment and locking-documentation concerns.

Comment thread reflex/utils/build.py
Comment thread reflex/utils/prerequisites.py Outdated

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 21 files

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread reflex/utils/build.py
Comment thread reflex/utils/prerequisites.py Outdated
Comment thread reflex/utils/prerequisites.py Outdated

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 11 files (changes from recent commits).

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread reflex/utils/build_cache.py Outdated
@Alek99
Alek99 force-pushed the codex/faster-deploy branch from 4da5154 to b923072 Compare September 8, 2026 22:28
Comment thread reflex/utils/prerequisites.py
Comment thread docs/hosting/self-hosting.md Outdated
constants.Reflex.VERSION,
[str(arg) for arg in command],
runtimes,
sorted(os.environ.items()),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hash only the environment that reaches Vite, not all of os.environ.

With the full environment in the key, ordinary variation between two runs of the same command defeats the cache. Reproduced on an unchanged app, each case rebuilt and replaced the snapshot, so the next plain run missed again:

  • reflex export --loglevel debug (sets REFLEX_LOGLEVEL)
  • OLDPWD=/tmp reflex export
  • the same command in the foreground and in a background subshell (SHLVL differs)

In CI, a per-run variable such as a job id makes the cache never hit. Hash an allowlist (for example PATH, NODE_*, NPM_CONFIG_*, VITE_*, REFLEX_* minus REFLEX_LOGLEVEL), or the explicit env dict that _build_frontend passes to the subprocess.

raise ValueError(msg)
digest = hashlib.sha256()

def visit(directory: Path) -> None:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Split visit by entry type.

_tree_digest measures cyclomatic complexity 15 and visit 13 (ruff --select C901 at max 8). One helper per entry kind (symlink, node_modules metadata, reflex.json, regular file) keeps each path readable and testable on its own.

return Path(package_manager).stem.lower() == "bun"


def _npm_installed_package_sections(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Extract the per-package verification into a helper.

This function measures cyclomatic complexity 13. The loop body is a chain of guard clauses that reads better as _verified_installed_version(name, declaration, entry) -> str | None, with this function reduced to the loop and the assignment.

Comment thread reflex/utils/build.py
progress.advance(task)
zipf.write(file, Path(file).relative_to(root_directory))
# Sidecars are already compressed for serving the frontend.
compress_type = (

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Move the compress-type choice into a small helper.

_zip now measures cyclomatic complexity 9. It was under 8 on main. A helper such as _zip_compress_type(component_name, file) keeps the write loop flat.

Comment thread reflex/utils/export.py
include_db_file=upload_db_file,
backend_excluded_dirs=backend_excluded_dirs,
)
with build_cache.frontend_build_lock(prerequisites.get_web_dir()):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Take the lock only when frontend is true.

A backend-only export in a project with no .web now creates a .web directory that contains only .reflex-build.lock. Reproduced with reflex export --backend-only after removing .web.

{
_version_check_timestamp_key(package_name): datetime.now(
timezone.utc
).isoformat()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use isoformat() for the attempt timestamp as well.

The attempt key at line 193 still writes str(now), so reflex.json mixes 2026-09-10 18:19:33+00:00 and 2026-09-10T18:19:34+00:00. Both parse, but one format is simpler to reason about.

@FarhanAliRaza FarhanAliRaza left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested with a two-page counter app that has one static asset, built against this branch.

Ran reflex export with and without zip, with the cache enabled and disabled, with bun and with npm. Repeat exports with the cache enabled skipped Vite and restored the snapshot. Setting the variable to false rebuilt and removed the cache directory. Two concurrent exports serialized on the lock, and the second one reused the first one's snapshot when the environment was identical. The frontend zip stored the .gz sidecars and deflated the other entries.

Ran reflex run --env prod with the cache enabled. The restart restored the cached build. In the browser the counter updated over the websocket, the asset loaded, page navigation worked, and gzip content negotiation returned compressed assets. No console errors.

The unit tests for the touched files pass.

The inline comments hold the requested changes. The first one, about the cache key, is the one that matters most: the same command with a different log level, a different OLDPWD, or a different SHLVL rebuilt every time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants