feat(files): preview .csv and .xlsx as data grids in the docked pane - #1134
Merged
Merged
Conversation
Adds a third viewer to the preview pane. `.csv` is an allowed upload type but had no preview at all — a `.md` attachment opens a modal and a csv card did nothing, so the only way to look at a data file in the app was to ask the agent to read it back, which is the most expensive way to look at a table. Deliberately a data grid, not a spreadsheet view. Cells stay the strings the file contained: no type inference, no locale-aware numbers, no date parsing. A preview that renders 007 as 7 is answering a different question than the one being asked. This does not reopen the .xlsx decision. That was a decline of the client-side *renderers* — SheetJS frozen on npm with unfixed advisories, ExcelJS throwing on any workbook with a native chart. A delimited file needs a parser, not a renderer, and the parser here is 200 lines with no new dependency. - `csv-parse.ts`: RFC 4180 field splitting (quoted delimiters, embedded newlines, doubled quotes), delimiter sniffing that scores consistency across rows rather than raw frequency, BOM handling, all three line endings, and byte/row/column caps. - `csv-viewer.component.ts`: virtualised with `cdk-virtual-scroll- viewport`, so a 40k-row file scrolls rather than showing its first screenful. `@angular/cdk` was already a dependency. Virtualising costs the `<table>` element, so the grid carries explicit ARIA grid roles and reports `aria-rowcount` from the data rather than the rendered window. - `PREVIEW_KIND_MIMES` becomes a list per kind: the recorded MIME is whatever the browser reported at upload, and Windows reports `application/vnd.ms-excel` for a .csv whenever Excel is the registered handler. The extension still chooses the viewer, so .xls stays out. Verified in the browser against a 4,001-row file: virtual scroller exact at depth (row 1001 at scrollTop 32000, 35 rows in the DOM), header and body sharing one track list, horizontal scroll synced, and every text pair clearing WCAG AA in both themes (min 4.63:1). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Completes the spreadsheet preview. The `.csv` grid shipped in 2c6aebdf
parses in the browser; a workbook cannot, so app-api reads it and sends
rows. The distinction that unblocked this is that the earlier `.xlsx`
decline was of client-side *renderers* — SheetJS frozen on npm with
unfixed advisories, ExcelJS raising on any workbook with a native chart,
which is what `create_excel_spreadsheet` produces. A grid of values
needs a reader, and openpyxl is already what our own spreadsheet tools
drive inside Code Interpreter.
New dependency: openpyxl==3.1.5 (MIT, pure Python, one dependency).
Already present in the rag-ingestion image via docling; this pin brings
it into app-api's own closure, where it was not before.
Backend
- `sheet_preview.py` reads every visible sheet into display strings,
streaming with `read_only=True` and stopping at the caps rather than
walking a whole sheet to discard most of it.
- It reads the workbook TWICE, and that is the point. `data_only=True`
returns None for any formula Excel never cached — the normal state of
a workbook openpyxl itself wrote, since openpyxl does not evaluate —
so a values-only preview would show blanks exactly where the totals
belong. The second pass supplies the formula text instead.
- Formatting decisions are the server's so the two readers cannot drift:
floats lose binary noise (0.1+0.2 renders as 0.3), a date-formatted
cell does not gain the midnight openpyxl invents for it, and a numeric
string stays a string.
- `GET /files/{id}/sheet-preview`, cookie-authenticated like every other
user-facing app-api route, distinguishing 404/413/415/422 because each
means something different to the user. The size cap is checked against
recorded metadata, so an oversized workbook costs a metadata read
rather than a transfer into memory.
Frontend
- `DataGridComponent` extracted from the csv viewer: one virtualised
grid now serves both readers, so the ARIA roles, column sizing, header
scroll sync and contrast work are shared rather than duplicated.
- `XlsxViewerComponent` takes an upload id rather than bytes, with a tab
per sheet. `previewFetchesBytes()` keeps the two paths apart, and
`fetchDocument` now refuses a server-read kind outright — letting one
through would pull a whole workbook into the browser and discard it.
Also fixes two things found while verifying in the browser: the
truncation notice measured 3.2:1 on the light grid and failed WCAG AA
(-700 clears it at 5.03:1), and the per-character column estimate
clipped symbol-heavy cells like formula text by a few px.
Adds `inline-template-backtick-guard.spec.ts`. A backtick inside an
inline template:/styles: block ends the literal early; tsc passes and
only the Angular compiler fails, naming the wrong cause. It has cost
debugging time three times, twice in this branch.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…oller
Two problems, found by questioning why the xlsx preview stopped at 500
rows when the csv one shows 50,000.
**The cap was guessed, not measured.** It is not a limit on what the
grid can draw — the viewer virtualises and recycles row elements, so the
DOM is bounded whatever it is handed. It bounds the server: one JSON
response, and an openpyxl parse in front of it. Measured on a 20-column
sheet:
cap parse JSON gzip
500 0.04s 0.06 MB 0.02 MB
5,000 0.53s 0.68 MB 0.15 MB
50,000 3.80s 7.17 MB 1.51 MB
There is no gzip middleware on app-api, so the JSON column is the wire
cost. 5,000 buys ten times the rows for half a second.
Fetch-on-scroll was considered and rejected on measurement: openpyxl's
read-only mode is a streaming parser, so `min_row` does not seek.
Reaching row 45,000 of a 50,000-row sheet costs 2.63s against 2.97s for
a complete pass, so paging is O(n²) over a scroll — ~119s of CPU where
one pass costs 3s. For this format, read once and send more.
Two further fixes the measurements exposed:
- The cell budget divided by MAX_COLUMNS_PER_SHEET rather than the
sheet's own width, charging a 7-column sheet as though it were 64
columns wide and cutting its rows by an order of magnitude.
- The formula pass is now lazy. It doubles the parse and earns nothing
on a sheet with no empty cells, since an empty cell is the only thing
a cached-value-less formula can look like. A 50,000-row sheet with no
formulas went 0.53s → 0.09s.
**The scroller froze on any second dataset.** One viewport serves every
dataset the grid is handed — switching worksheet tabs, or previewing a
second file without closing the pane. CDK picked up the new length (the
spacer grew to full height) but never recomputed the rendered range, so
the body stayed pinned to the first screenful while the scrollbar moved
over the whole sheet. Measured in the browser: at scrollOffset 10000 of
a 1,200-row sheet the rendered range was still {start: 0, end: 32}, and
checkViewportSize() corrected it to {start: 309, end: 345} on the spot.
The grid now re-measures and rewinds when its rows change — which is the
right behaviour anyway, since a newly chosen sheet should start at its
first row rather than inherit the previous one's offset.
Verified live against dev: the 1,200-row sheet now returns whole in a
311ms round trip and scrolls to row 1,200 with 29-36 rows in the DOM.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds spreadsheet and delimited-data previews to the docked pane, alongside the
.docxand.pptxviewers from #1122..csvis an allowed upload type that had no preview at all — a.mdattachment opened a modal and a csv card did nothing, so the only way to look at a data file in the app was to ask the agent to read it back, which is the most expensive way to look at a table.This does not reverse the earlier
.xlsxdecisionThat decline was of client-side renderers — reproducing Excel's own layout. It still stands: the npm build of SheetJS is frozen on a 2022 release with unfixed advisories, and ExcelJS raises outright on any workbook containing a native chart, which is exactly what
create_excel_spreadsheetproduces.A grid of values needs a reader, not a renderer. So:
.csvparses in the browser (csv-parse.ts, ~200 lines, no new dependency)..xlsxis read by app-api with openpyxl and sent as rows. The workbook never reaches the browser.One
DataGridComponentdraws both.What reviewers should weigh
New dependency:
openpyxl==3.1.5(MIT, pure Python, one dependencyet-xmlfile). It is already what our owncreate_excel_spreadsheettool drives inside Code Interpreter, and the rag-ingestion image pulls it via docling — this pin brings it into app-api's own closure, where it was not before. Note upstream is slow-moving: 3.1.5 dates from June 2024 and is still the latest release.New route:
GET /files/{upload_id}/sheet-previewon app-api, cookie-authenticated viaget_current_user_from_sessionlike every other user-facing route. It distinguishes 404 / 413 / 415 / 422 because each means something different to the user. The size cap is checked against recorded metadata, so an oversized workbook costs a metadata read rather than a transfer into memory, and the parse runs inasyncio.to_threadbecause openpyxl is blocking CPU.The non-obvious part: openpyxl is read twice
data_only=TruereturnsNonefor any formula Excel never cached — and openpyxl does no evaluation, so every formula in a workbook our own tools wrote reads back as None. A values-only preview would show blanks exactly where the totals belong. The second pass supplies the formula text, so such a cell renders as=SUM(B2:B10)rather than empty. That is both more useful and more honest: the file really does not carry that number yet.The second pass is lazy — it only runs for a sheet that came back with an empty cell, since that is the only thing a value-less formula can look like. A 50,000-row sheet with no formulas went 0.53s → 0.09s.
Formatting decisions live on the server so the two readers cannot drift: floats lose binary noise (
0.1 + 0.2renders0.3), a date-formatted cell does not gain the midnight openpyxl invents for it, booleans renderTRUE/FALSE, and007stays007. Neither reader does type inference — a preview that prettifies007into7is answering a different question than the one being asked.Row caps, measured rather than guessed
The caps are not a limit on what the grid can draw — it virtualises with
cdk-virtual-scroll-viewportand recycles row elements, so the DOM is bounded regardless. They bound the server: one JSON response with an openpyxl parse in front of it. On a 20-column sheet:There is no gzip middleware on app-api, so the JSON column is the wire cost. The cap is 5,000 rows per sheet, under a global cell budget spent against each sheet's own width.
Fetch-on-scroll was considered and rejected on measurement: openpyxl's read-only mode is a streaming parser, so
min_rowdoes not seek. Reaching row 45,000 of a 50,000-row sheet costs 2.63s against 2.97s for a complete pass, making paging O(n²) over a scroll — roughly 119s of CPU where one pass costs 3s. For this format, read once and send more.Two bugs found while verifying in the browser
scrollOffset10000 of a 1,200-row sheet the rendered range was still{start: 0, end: 32}, andcheckViewportSize()corrected it to{start: 309, end: 345}on the spot. This affected the.csvpath too.text-gray-500 dark:text-gray-400. Every text pair in the grid now clears AA in both themes (minimum 4.84:1).Also included
inline-template-backtick-guard.spec.ts— a backtick inside an inlinetemplate:/styles:block ends the template literal early;tsc --noEmitpasses and only the Angular compiler fails, naming the wrong cause ("Cannot find name 'text'"). It has cost debugging time three times. Now it is a test failure with a message that says what actually happened.Verification
test_sheet_preview_route.py, which drives route → real service → openpyxl with only S3 and the repository stubbed, and asserts the zip magicPK\x03\x04never appears in the response body.isolate: falseclass of flake — develop's442e574cmemo fix holds).=SUM(...), the hidden sheet is omitted,0.1 + 0.2reads0.3, and a 1,200-row sheet returns whole in a 311ms round trip and scrolls to row 1,200 with 29–36 rows in the DOM.Follow-ups (not in this PR)
GZipMiddlewareto app-api would take that 0.68 MB response to 0.15 MB and help every other endpoint. It needstext/event-streamexcluded first — CloudFront already setscompress: falseon/api/*for exactly that reason.text-state-warning-600 dark:text-state-warning-400fails AA in light mode (3.2:1) and appears in ~36 places app-wide. Fixed here in the one instance this PR touches; the sweep is separate.🤖 Generated with Claude Code