Repository navigation
perf(coding-agent): images within the limits skip photon; resizes run in Bun.Image - #67
Conversation
… in Bun.Image Pi-Bolt runs photon's WebAssembly in the interpreter. Reading a 1800x1100 PNG that needs no resize cost 1.16 s of CPU (a full decode to learn its size), and a 2442x1438 screenshot 36.7 s (decode, Lanczos3 resize, a PNG and five JPEG encodes, then a second decode when the tool result was normalized). - Read width and height from the PNG, JPEG, GIF or WebP structure. An image within the limits is returned as it is, without photon or a worker. The parsers accept only intact files (every PNG chunk CRC, end markers, frame bounds); anything else takes the decode path as before. - Encode candidates one at a time and stop at the first that fits: the JPEG qualities are encoded only when the PNG is too large. Same output. - The read tool marks the image blocks it produced; tool-result normalization skips them when the options match instead of processing them again. - When the runtime has Bun.Image (Pi-Bolt's runtime, Bun 1.4), resize there: native decoders, SIMD Lanczos3, libspng/libjpeg-turbo encoders, off the main thread. Same target sizes, candidate order and size limit; different bytes. Photon remains the path under Node and the fallback when it fails. CPU per `pi -p` read (0.9.5 runtime, medians of 3): 1800x1100 PNG 1162 -> 56 ms (a text read is 57 ms), 1600x1200 JPEG 3651 -> 68 ms, 2442x1438 PNG 36.7 -> 0.52 s, 3024x1964 PNG 39.9 -> 0.70 s, 4032x3024 JPEG 75.5 -> 1.09 s.
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
If your organization's extra usage balance is empty, an organization admin can add extra usage credits at claude.ai/admin-settings/usage. If its monthly spend limit was reached, an admin can raise it on the same page. If neither applies, contact Anthropic support.
Once extra usage is available, someone with write access to this repository can comment @claude review on this pull request to trigger a review.
There was a problem hiding this comment.
Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.
An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: b660514a8a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…re sent as they are
The header check alone let through files whose structure is intact but whose
image data is damaged, for example a JPEG with corrupt scan data ending in
FF D9. 0.9.5 decoded every image with photon and omitted those; the previous
commit sent them as they were, which a provider can reject for the whole
request.
- PNG and WebP within the limits skip photon only when Bun.Image decodes them
(a full native decode to 1x1: about 5 ms for an 1800x1100 PNG, 25 ms for an
1800x1100 WebP). When it fails, photon decodes them as in 0.9.5.
- JPEG and GIF within the limits always go through photon: libjpeg-turbo and
Bun's GIF decoder accept damaged data that photon rejects.
- Without Bun.Image (Node, photon with a JIT), every format goes through
photon, as in 0.9.5.
- resizeImageInProcess (photon path) no longer skips the decode.
CPU per `pi -p` read against 0.9.5 (medians of 3): 1800x1100 PNG 996 -> 47 ms,
1800x1100 WebP 6328 -> 68 ms, 1600x1200 JPEG 2094 -> 1052 ms (one photon
decode instead of two). Requests are identical to 0.9.5 for these files and for
a damaged JPEG ("Image omitted") and a damaged WebP.
Reading an image with the
readtool, or attaching one, costs seconds to over a minute of CPU on Pi-Bolt. After this change it costs about 0 ms for an image that needs no resize, and 0.5-1.1 s for one that does.Problem
Pi-Bolt runs with the JIT off, so JSC interprets photon's WebAssembly. Pi sends every image through photon:
readresult then goes throughnormalizeToolResultImages, which decodes the result again.A 2442x1438 screenshot: 36.7 s of CPU. By operation, interpreted (from the review): decode 2.0 s, resize 11.4 s, PNG 3.7 s, five JPEGs about 4 s each, second decode 1.7 s.
Fix
image-header.tsreads width and height from the PNG, JPEG, GIF or WebP structure. With EXIF orientations 5-8 the two are swapped, as photon's rotation does. An image within the limits is returned as it is: no photon, no worker. The bytes are the same as before. The parsers accept only intact files. Anything else takes the decode path as before. They check:normalizeToolResultImagesskips a marked block when its data and options are unchanged. Processing the block again would return it unchanged anyway.Bun.Image: libspng, libjpeg-turbo and libwebp codecs, and a SIMD Lanczos3 resize, on Bun's own worker threads. WhenBun.Imageexists, an image over the limits is resized there. It keeps the same target sizes, candidate order (PNG, then JPEG 80, 85, 70, 55, 40, then 75% steps) and size limit. The EXIF orientation is applied withrotate/flip/flop, which matches photon's rotation pixel for pixel for orientations 2-8. IfBun.Imagethrows, the photon path runs as before. Under Node, photon runs as before.Before / after (CPU per image, Pi-Bolt build, 0.9.5 runtime)
pi -preads one file against a local fake model. base = 0.9.5 (origin/pi-bolt). Each figure is the median of 3 interleaved runs, user+sys CPU. The Mac was shared, with a load average of 30-140.Wall time for the screenshot: 45.6 s to 1.06 s. Peak RSS: 98 to 66 MB for the 1800x1100 PNG, because photon is never loaded; 113 to 108 MB for the screenshot.
Without
Bun.Image(Node, or a native failure), points 1-3 still apply. This is photon in process with the JIT off, one call, the same output bytes as before:What the model receives
resizeImageInProcessgave the same output on 7 cases, including the JPEG fallback and the 75% shrink.maxBytes200 KB, the 4032x3024 photo is now sent at 2000x1500; photon had to go down to 1500x1125.readreports "Image omitted" when the decode fails. An in-limit image skips photon only in these cases:Tests
image-header.test.ts:resizeImageInProcess: returns in-limit images without loading photon; uses the displayed size for EXIF orientation 6; falls back to photon when the header cannot be read.image-resize-encode.test.ts: no JPEG encode when the PNG fits; qualities in order until one fits; the shrink loop; the native pipeline with a stand-in forBun.Image(orientation, Lanczos3/fill, stops at the first fit, errors reach the caller).suite/agent-session-read-images.test.ts: a sessionreadof an in-limit PNG is byte-identical and processed once; an over-limit PNG is resized once, with the dimension note.npm run checkis clean. These pass: the existing image, read, tool-result and clipboard tests (17 files, 295 tests).Not in this PR
pi-tui components/image.ts) still converts non-PNG images to PNG with photon, synchronously insiderender().image-convert.ts) still uses photon.Each of these could move to
Bun.Image.