Skip to content

Think: workspace read emits file-data for images, breaking image reads on the ai@6 peer #2080

Description

@jbreite

Describe the bug

Think's workspace read tool classifies images correctly as kind: "image", but toModelOutput emits them as a file-data content part — the same part type it uses for PDFs. On the ai@7 peer this is harmless, because AI SDK core rewrites deprecated parts and the provider re-derives routing from mediaType. On the ai@6 peer there is no normalization step: file-data maps straight to a document part, so images are sent to the model as documents.

@cloudflare/think declares "ai": "^6.0.0 || ^7.0.0" as a peer, so ai@6 is a supported configuration. On it, reading an image from the workspace fails — loudly on OpenAI, silently on Anthropic. PDF and text reads are unaffected on both.

To Reproduce

  1. Install @cloudflare/think@0.15.1 with ai@6.x and an OpenAI-backed model.
  2. Write a PNG into the workspace: workspace.writeFileBytes("/screenshot.png", bytes, "image/png").
  3. Ask the agent to read that path.
  4. The turn fails:
Invalid file data: 'input[3].output[1].file_data'. Expected a base64-encoded data URL
with a valid file MIME type (e.g. 'data:text/plain;base64,SGVsbG8sIFdvcmxkIQ=='),
but got unsupported MIME type 'image/png'.

With an Anthropic model instead, step 4 produces no error — its file-data case only handles application/pdf and pushes an unsupported tool content part type warning for anything else, dropping the part. The model receives the text note with no image and answers as though nothing was attached, which is the more dangerous of the two failures.

Changing only the emitted part type from file-data to image-data fixes both.

Expected behavior

read on an image returns it to the model as image content, so a vision-capable model can see it — the behaviour that shipped in #1435 and is still described by the tool's own description ("Images and PDFs are passed to capable models as file content") and by the docs table ("pass images and PDFs to multimodal models").

Screenshots

N/A — error output included above.

Version:

@cloudflare/think 0.15.1 (latest)
agents 0.20.1
ai 6.0.208
provider @ai-sdk/gateway 3.x → OpenAI and Anthropic
runtime Cloudflare Workers + Durable Objects

Additional context

Root cause

Pinned to main at e70f54ddcfaa55383fa1986d031be8a215282d85.

execute derives the distinction correctly — image at workspace.ts#L321-L330, PDF at #L331-L338:

if (mediaType.startsWith("image/")) {
  return { kind: "image", ... };
}
if (mediaType === "application/pdf") {
  return { kind: "file", ... };
}

toModelOutput then discards it. isModelFileReadOutput admits both kinds, and everything past that gate emits one shape at #L415-L429:

{
  // `file-data` (base64 string `data`) is accepted by both AI SDK v6
  // and v7. v7 also offers the newer `{ type: "file", data: { type:
  // "data", data } }` shape, but that does not exist in v6, so we use
  // the cross-major form here.
  type: "file-data",
  data,
  mediaType: replayOutput.mediaType,
  filename: replayOutput.name
}

The comment is accurate that both majors accept file-data. The gap is that they interpret it differently.

ai@7 rewrites deprecated tool-result parts in convert-to-language-model-prompt.ts before the provider sees them, preserving mediaType:

case 'file-data': {
  return { type: 'file', data: { type: 'data', data: item.data },
           filename: item.filename, mediaType: item.mediaType, ... };
}

The provider then routes on media type, so image/* still becomes input_image:

case "file": {
  const topLevel = getTopLevelMediaType(item.mediaType);
  if (topLevel === "image") return { type: "input_image", image_url: ... };
  return { type: "input_file", filename: ..., file_data: ... };
}

ai@6 has no normalization step. image-data and file-data are distinct part types that map straight through, and the caller chooses:

  • image-datainput_image (OpenAI) / image block (Anthropic)
  • file-datainput_file (OpenAI) / document, PDF-only (Anthropic)

So on v6 the media type never gets a chance to influence routing, and an image emitted as file-data is a document by construction.

How it regressed

This worked when multimodal read landed. #1435 emitted image-data for images and file-data for PDFs, with tests asserting the split.

#1922 (feat: update AI SDK to v7) changed the image branch to file-data and updated both test expectations from image-data to file-data in the same commit, which is why CI stayed green. The PR notes describe the intent:

Prefer file over legacy image / image-data / media / file-data content parts. AI SDK v7 docs show multimodal tool results as { type: "file", mediaType, data: { type: "data", data } }.

That is correct for v7, where consolidating is safe precisely because the provider re-derives routing from mediaType. The regression is in the fallback: since { type: "file" } does not exist in v6, the code falls back to file-data on both majors — and on v6 that is document semantics, not a neutral container.

Suggested fix

Restoring the pre-#1922 branch is correct on both majors, so no peer detection is needed:

const isImage = replayOutput.kind === "image";
return {
  type: "content",
  value: [
    { type: "text", text: note },
    isImage
      ? { type: "image-data", data, mediaType: replayOutput.mediaType }
      : { type: "file-data", data, mediaType: replayOutput.mediaType,
          filename: replayOutput.name }
  ]
};

On ai@7, image-data normalizes to { type: "file", mediaType } with the media type preserved (convert-to-language-model-prompt.ts, case 'image-data'), so the provider still produces input_image — same result as today, plus a deprecation warning. On ai@6 it produces input_image directly, which is the fix.

If you would rather not reintroduce a deprecated part type, the alternative is to emit v7's { type: "file", data: { type: "data", data }, mediaType } when the peer is v7 and image-data/file-data when it is v6 — but that needs version detection for no behavioural gain, since v7 accepts the deprecated forms until v8.

The two tests changed in #1922 are the ones that would have caught this, so reverting their expectations alongside the fix restores the guard.

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions