Skip to content

OUT-4176: support Dropbox uploads over 150 MiB via upload sessions - #153

Merged
SandipBajracharya merged 5 commits into
mainfrom
OUT-4176
Sep 14, 2026
Merged

SandipBajracharya merged 5 commits into
mainfrom
OUT-4176

Conversation

@SandipBajracharya

@SandipBajracharya SandipBajracharya commented Sep 11, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Files larger than 150 MiB failed to upload to Dropbox with payload_too_large because uploadFileInDropbox used the single-shot /files/upload endpoint (150 MiB cap). Large files now stream through a Dropbox upload session (OUT-4176).

What changed

  • chunkStream helper — reads a byte stream into fixed-size buffers (last one smaller); peak memory ≈ one chunk.
  • DropboxClient upload session — _uploadFileSession drives start → append_v2 (per chunk) → finish, exposed as retry-wrapped uploadFileSession. getBody yields a fresh stream so a retry restarts the whole session; the streamed byte count is checked against the expected size before finish, so a short/expired download can never commit a partial file.
  • Routing — uploadFileInDropbox branches on file.size: over 150 MiB (or unknown size) → session, else the unchanged single-shot path. A shared download step fails on a non-ok response so an error page is never uploaded as content. A log line records which path each file took.
  • Constants — DROPBOX_SINGLE_UPLOAD_MAX_BYTES (150 MiB), DROPBOX_UPLOAD_CHUNK_BYTES (16 MiB, a multiple of 4 MiB per Dropbox guidance).
  • Logging — raised the logger inspect depth 2 → 4 (with a LOG_DEPTH override) so nested payloads stop printing as [Object]. Cross-cutting fix, bundled here.

Design notes

  • 16 MiB is the chunk size, not per file: peak memory ≈ one chunk per in-flight upload (so chunk size × queue concurrency overall).
  • On any failure the whole session restarts with a fresh session_id; the orphaned partial session expires on Dropbox's side (no duplicate, mode: 'add').
  • Strict > boundary: a file of exactly 150 MiB stays on the single-shot path.

Testing

  • pnpm typecheck, pnpm lint, pnpm test — 311 unit tests pass.
  • pnpm test:integration — 189 integration tests pass (22 files), including a new large-file flow (assembly-to-dropbox.large-file.integration.test.ts) that drives the session path end-to-end and asserts start/finish ran and the mapping row was created. The single-shot upload is left unmocked, so a routing regression trips msw.
  • Unit coverage: chunkStream edges, session sequence/offsets/error, empty-body, retry re-fetch, routing (single/session/unknown), non-ok download, byte-count guard, logger depth.

Before merge

  • Confirm file.size is actually populated from Assembly on real files (log it in staging). If it's ever missing, files still upload correctly but always take the slower session path and skip the byte-count guard.

🤖 Generated with Claude Code

SandipBajracharya and others added 4 commits September 11, 2026 20:18
…ffers

Reads a byte stream into fixed-size buffers (last one may be smaller) so a
large file can be fed to a Dropbox upload session one chunk per append.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add _uploadFileSession (start, append_v2, finish) for files too large for
/files/upload, exposed as the retry-wrapped uploadFileSession. getBody yields a
fresh stream so a retry restarts the whole session, and the streamed byte count
is checked against the expected size before finishing so a short download can
never commit a partial file.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
uploadFileInDropbox now branches on the Assembly file size: files over the
150 MiB single-request limit (or with an unknown size) go through the chunked
upload session, everything else keeps the single-shot /files/upload. Add a
shared download step that fails on a non-ok response so an error page is never
uploaded as file content, plus a log line showing which path each file took.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Object logs were capped at depth 2, so nested payloads printed as [Object]
(e.g. a batched item's opts/file). Raise the default depth to 4 and add a
LOG_DEPTH override, keeping the array and string caps so a line stays bounded.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@vercel

vercel Bot commented Sep 11, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
dropbox-integration Ready Ready Preview Sep 11, 2026 2:46pm UTC

Request Review

@linear-code

linear-code Bot commented Sep 11, 2026

Copy link
Copy Markdown

OUT-4176

@greptile-apps

greptile-apps Bot commented Sep 11, 2026 •

Copy link
Copy Markdown

RetriggerConfidence Score: 5/5

The PR appears safe to merge with no outstanding correctness, security, or repository-rule violations.

Summary

  • Streams large or unknown-size files through Dropbox session start, append, and finish requests.
  • Restarts the complete session with a fresh Assembly download when retrying.
  • Verifies known file sizes before committing and rejects unsuccessful Assembly downloads.
  • Adds unit and integration coverage for chunking, upload-session sequencing, routing, retries, and durable file mappings.
  • Makes logger inspection depth configurable while retaining bounded output.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Assembly file] --> B{Download URL present?}
    B -- No --> X[Fail upload]
    B -- Yes --> C{Size known and ≤ 150 MiB?}
    C -- Yes --> D[Fetch Assembly body]
    D --> E[Dropbox files/upload]
    C -- No --> F[Fetch fresh Assembly body]
    F --> G[Upload-session start with first chunk]
    G --> H[Append remaining chunks with offsets]
    H --> I{Known size matches streamed bytes?}
    I -- No --> X
    I -- Yes --> J[Finish and commit Dropbox file]
    E --> K[Complete durable file mapping]
    J --> K
Loading

Reviews (2) · Last reviewed commit: "test(OUT-4176): add large-file upload-se..."

Comment thread src/lib/dropbox/__tests__/DropboxClient.uploadSession.test.ts
Drive an Assembly -> Dropbox sync for a file with no reported size, which routes
through the upload session, and assert the session start/finish ran and the
mapping row was created. The single-shot upload is left unmocked so a routing
regression trips msw. Adds a mockDropboxUploadSession helper.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@SandipBajracharya

Copy link
Copy Markdown
Collaborator Author

@greptileai rereview

@SandipBajracharya SandipBajracharya changed the title feat(OUT-4176): support Dropbox uploads over 150 MiB via upload sessions OUT-4176: support Dropbox uploads over 150 MiB via upload sessions Sep 14, 2026

@priosshrsth priosshrsth left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@SandipBajracharya
SandipBajracharya merged commit 0682d15 into main Sep 14, 2026
11 checks passed

This branch was successfully deployed

1 active deployment
Preview — 33774488 Deployed Sep 11, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants