From be92084ddb724d3a6c06b31c17ae416966cd56b2 Mon Sep 17 00:00:00 2001 From: L4Ph Date: Fri, 14 Aug 2026 08:19:22 +0900 Subject: [PATCH 1/3] fix(ai-gemini): stop dropping modelOptions on the native image path GeminiImageModelProviderOptionsByName mapped every image model to the Imagen-shaped GeminiImageProviderOptions, so safetySettings, thinkingConfig, imageConfig and systemInstruction were compile errors on the four Gemini-native models -- even though those models are served by generateContent, whose GenerateContentConfig accepts all four. The adapter compensated by forwarding only `seed` and silently dropping the rest, which its own comment documented as deliberate. Split the map native vs Imagen, mirroring the split that GeminiImageModelSizeByName and GeminiImageModelInputModalitiesByName already use. Both API paths now pick their config fields by name rather than spreading modelOptions wholesale, so neither endpoint can receive a field shaped for the other. responseModalities stays a protected adapter default and is deliberately absent from the new type. Runtime routing moves off a `gemini-` prefix test onto membership in GEMINI_NATIVE_IMAGE_MODELS, so the route and the type-level split cannot drift apart. An unknown id now reaches the Imagen endpoint and fails there instead of taking the native path with Imagen-shaped option types -- the same class of mismatch this change exists to remove. --- .../gemini-native-image-model-options.md | 11 + docs/adapters/gemini.md | 24 ++ docs/config.json | 4 +- docs/media/image-generation.md | 36 ++- packages/ai-gemini/src/adapters/image.ts | 124 +++++--- .../src/image/image-provider-options.ts | 129 ++++++++- packages/ai-gemini/src/index.ts | 11 + .../ai-gemini/tests/image-adapter.test.ts | 274 ++++++++++++++++-- .../tests/image-per-model-type-safety.test.ts | 159 ++++++++++ 9 files changed, 700 insertions(+), 72 deletions(-) create mode 100644 .changeset/gemini-native-image-model-options.md create mode 100644 packages/ai-gemini/tests/image-per-model-type-safety.test.ts diff --git a/.changeset/gemini-native-image-model-options.md b/.changeset/gemini-native-image-model-options.md new file mode 100644 index 0000000000..6b22f36b0b --- /dev/null +++ b/.changeset/gemini-native-image-model-options.md @@ -0,0 +1,11 @@ +--- +'@tanstack/ai-gemini': minor +--- + +Type Gemini-native image models with their own provider options. `GeminiImageModelProviderOptionsByName` mapped **every** image model to the Imagen-shaped `GeminiImageProviderOptions`, so `modelOptions: { safetySettings, thinkingConfig, imageConfig, systemInstruction }` was a compile error on `gemini-3.1-flash-image-preview`, `gemini-3.1-flash-lite-image`, `gemini-3-pro-image-preview`, and `gemini-2.5-flash-image` — even though those models are served by `generateContent`, whose `GenerateContentConfig` accepts all of them. The adapter compensated by forwarding only `seed`, silently dropping anything else. + +The map now splits native vs Imagen, mirroring the split already used by `GeminiImageModelSizeByName` and `GeminiImageModelInputModalitiesByName`: native models get the new `GeminiNativeImageProviderOptions` (`seed`, `safetySettings`, `thinkingConfig`, `imageConfig`, `systemInstruction`), Imagen models keep `GeminiImageProviderOptions`. Both API paths now pick their config fields by name — never a wholesale spread — so neither shape's fields can reach the other's endpoint. + +`responseModalities` stays a protected adapter default (`['TEXT', 'IMAGE']`) and is deliberately absent from the new type. `modelOptions.imageConfig` merges **over** the `imageConfig` derived from the portable `size` option, per field — passing only `imageConfig.imageSize` keeps the `aspectRatio` that `size` implied. `HarmCategory` and `HarmBlockThreshold` are now re-exported so `safetySettings` can be written without adding `@google/genai` to your own dependencies. + +**BREAKING (types only):** Imagen fields no longer compile on the four Gemini-native image models — `aspectRatio`, `negativePrompt`, `personGeneration`, `safetyFilterLevel`, `addWatermark`, `language`, `outputMimeType`, `outputCompressionQuality`, `guidanceScale`, `enhancePrompt`, `includeSafetyAttributes`, `includeRaiReason`, `outputGcsUri`, `labels`. They previously type-checked but were already dropped at runtime (only `seed` was ever forwarded to `generateContent`), so no request behaviour changes — the compiler now reports what was already happening. Migrate `aspectRatio` to the portable `size` option (`'16:9_4K'`) or to `modelOptions.imageConfig`, and drop the rest. `GeminiImageAdapter.generateImages` (and its `~types.providerOptions`) also widens from `ImageGenerationOptions` to `ImageGenerationOptions`, which affects code structurally annotated against the old signature. diff --git a/docs/adapters/gemini.md b/docs/adapters/gemini.md index 31da9cf6e2..c137309e11 100644 --- a/docs/adapters/gemini.md +++ b/docs/adapters/gemini.md @@ -494,6 +494,10 @@ const result = await generateImage({ ### Image Model Options +`modelOptions` is typed per model family, because the two families hit different APIs. + +Imagen models (`generateImages`) take `GenerateImagesConfig` fields: + ```typescript ignore import { generateImage } from "@tanstack/ai"; import { geminiImage } from "@tanstack/ai-gemini"; @@ -509,6 +513,26 @@ const result = await generateImage({ }); ``` +Gemini native models (`generateContent`) take `GenerateContentConfig` fields instead — `seed`, `safetySettings`, `thinkingConfig`, `imageConfig`, and `systemInstruction`: + +```typescript +import { generateImage } from "@tanstack/ai"; +import { geminiImage } from "@tanstack/ai-gemini"; + +const result = await generateImage({ + adapter: geminiImage("gemini-3.1-flash-image-preview"), + prompt: "...", + size: "16:9_4K", + modelOptions: { + thinkingConfig: { thinkingBudget: 512 }, + // Merged over the imageConfig derived from `size`, per field. + imageConfig: { imageSize: "2K" }, + }, +}); +``` + +See [Image Generation](../media/image-generation) for the full native option list. + ## Text-to-Speech (Experimental) > **Note:** Gemini TTS is experimental and may require the Live API for full functionality. diff --git a/docs/config.json b/docs/config.json index e5c98bc877..67f5fd0085 100644 --- a/docs/config.json +++ b/docs/config.json @@ -437,7 +437,7 @@ "label": "Image Generation", "to": "media/image-generation", "addedAt": "2026-04-15", - "updatedAt": "2026-08-13" + "updatedAt": "2026-08-14" }, { "label": "Video Generation", @@ -796,7 +796,7 @@ "label": "Google Gemini", "to": "adapters/gemini", "addedAt": "2026-04-15", - "updatedAt": "2026-07-22" + "updatedAt": "2026-08-14" }, { "label": "Ollama", diff --git a/docs/media/image-generation.md b/docs/media/image-generation.md index c1312e8122..24c8ab8734 100644 --- a/docs/media/image-generation.md +++ b/docs/media/image-generation.md @@ -634,7 +634,7 @@ const result = await generateImage({ #### Gemini Native Model Options (NanoBanana) -Gemini native image models accept `GenerateContentConfig` options directly in `modelOptions`: +Gemini native image models are served by `generateContent`, so their `modelOptions` are `GenerateContentConfig` fields — a different shape from the Imagen options above: ```typescript import { generateImage } from "@tanstack/ai"; @@ -644,9 +644,43 @@ const result = await generateImage({ adapter: geminiImage("gemini-3.1-flash-image-preview"), prompt: "A beautiful garden", size: "16:9_4K", + modelOptions: { + seed: 42, + thinkingConfig: { thinkingBudget: 512 }, + systemInstruction: "Always render in watercolor.", + // Merged over the imageConfig derived from `size`, per field: this keeps + // the 16:9 aspect ratio and overrides only the resolution tier. + imageConfig: { imageSize: "2K" }, + }, +}); +``` + +`safetySettings` takes the SDK's `HarmCategory` / `HarmBlockThreshold` enums, so plain strings won't type-check. Both are re-exported from `@tanstack/ai-gemini` — you don't need `@google/genai` in your own dependencies: + +```typescript +import { generateImage } from "@tanstack/ai"; +import { + HarmBlockThreshold, + HarmCategory, + geminiImage, +} from "@tanstack/ai-gemini"; + +const result = await generateImage({ + adapter: geminiImage("gemini-3.1-flash-image-preview"), + prompt: "A beautiful garden", + modelOptions: { + safetySettings: [ + { + category: HarmCategory.HARM_CATEGORY_HATE_SPEECH, + threshold: HarmBlockThreshold.BLOCK_ONLY_HIGH, + }, + ], + }, }); ``` +`responseModalities` is not accepted — the adapter always requests `['TEXT', 'IMAGE']`, so nothing can silently disable image output. + ### Response Format The image generation result includes: diff --git a/packages/ai-gemini/src/adapters/image.ts b/packages/ai-gemini/src/adapters/image.ts index 85284d282e..abbfa38cec 100644 --- a/packages/ai-gemini/src/adapters/image.ts +++ b/packages/ai-gemini/src/adapters/image.ts @@ -7,6 +7,7 @@ import { } from '../utils' import { buildGeminiUsage } from '../usage' import { + isGeminiNativeImageModel, parseNativeImageSize, sizeToAspectRatio, validateImageSize, @@ -15,10 +16,11 @@ import { } from '../image/image-provider-options' import type { GEMINI_IMAGE_MODELS } from '../model-meta' import type { + GeminiAnyImageProviderOptions, GeminiImageModelInputModalitiesByName, GeminiImageModelProviderOptionsByName, GeminiImageModelSizeByName, - GeminiImageProviderOptions, + GeminiNativeImageProviderOptions, } from '../image/image-provider-options' import type { GeneratedImage, @@ -35,6 +37,7 @@ import type { GenerateImagesConfig, GenerateImagesResponse, GoogleGenAI, + ImageConfig, Part, } from '@google/genai' import type { GeminiClientConfig } from '../utils/client' @@ -65,7 +68,7 @@ export class GeminiImageAdapter< TModel extends GeminiImageModel, > extends BaseImageAdapter< TModel, - GeminiImageProviderOptions, + GeminiAnyImageProviderOptions, GeminiImageModelProviderOptionsByName, GeminiImageModelSizeByName, GeminiImageModelInputModalitiesByName @@ -75,7 +78,7 @@ export class GeminiImageAdapter< // Type-only property - never assigned at runtime declare '~types': { - providerOptions: GeminiImageProviderOptions + providerOptions: GeminiAnyImageProviderOptions modelProviderOptionsByName: GeminiImageModelProviderOptionsByName modelSizeByName: GeminiImageModelSizeByName modelInputModalitiesByName: GeminiImageModelInputModalitiesByName @@ -89,7 +92,7 @@ export class GeminiImageAdapter< } async generateImages( - options: ImageGenerationOptions, + options: ImageGenerationOptions, ): Promise { const { model, logger } = options @@ -121,7 +124,7 @@ export class GeminiImageAdapter< ) } - if (this.isGeminiImageModel(model)) { + if (isGeminiNativeImageModel(model)) { return await this.generateWithGeminiApi(options, resolved) } @@ -155,29 +158,40 @@ export class GeminiImageAdapter< } } - private isGeminiImageModel(model: string): boolean { - return model.startsWith('gemini-') - } - private async generateWithGeminiApi( - options: ImageGenerationOptions, + options: ImageGenerationOptions, resolved: ResolvedMediaPrompt, ): Promise { const { model, size, numberOfImages, modelOptions } = options const parsedSize = size ? parseNativeImageSize(size) : undefined - // GeminiImageProviderOptions is Imagen-shaped — most fields - // (personGeneration, safetyFilterLevel, addWatermark, outputMimeType, - // outputCompressionQuality, guidanceScale, enhancePrompt, - // includeSafetyAttributes, includeRaiReason, outputGcsUri, labels, - // negativePrompt, language) are only valid on GenerateImagesConfig and - // would be rejected by the Gemini-native generateContent path. Pick only - // the fields that are valid on GenerateContentConfig instead of spreading - // the whole options object. - const nativeConfig: GenerateContentConfig = {} - if (modelOptions?.seed !== undefined) { - nativeConfig.seed = modelOptions.seed + // The portable `size` option is the baseline; modelOptions.imageConfig is + // the provider escape hatch and wins per field, so a caller passing only + // `imageConfig.imageSize` keeps the aspectRatio derived from `size`. + const imageConfig: ImageConfig = { + ...(parsedSize?.aspectRatio && { aspectRatio: parsedSize.aspectRatio }), + ...(parsedSize?.resolution && { imageSize: parsedSize.resolution }), + ...modelOptions?.imageConfig, + } + + // Named picks, never a wholesale spread: the Imagen-shaped fields of + // GeminiImageProviderOptions (personGeneration, safetyFilterLevel, + // addWatermark, outputMimeType, …) are only valid on GenerateImagesConfig + // and would be rejected by generateContent. Picking by name means no + // Imagen field can reach this path even if one slips past the per-model + // provider-options map. + const nativeConfig: GenerateContentConfig = { + ...(modelOptions?.seed !== undefined && { seed: modelOptions.seed }), + ...(modelOptions?.safetySettings !== undefined && { + safetySettings: modelOptions.safetySettings, + }), + ...(modelOptions?.thinkingConfig !== undefined && { + thinkingConfig: modelOptions.thinkingConfig, + }), + ...(modelOptions?.systemInstruction !== undefined && { + systemInstruction: modelOptions.systemInstruction, + }), } const config: GenerateContentConfig = { @@ -186,16 +200,7 @@ export class GeminiImageAdapter< // IMPORTANT: responseModalities is a protected default — set it AFTER // nativeConfig so nothing can silently disable image output. responseModalities: ['TEXT', 'IMAGE'], - ...(parsedSize && { - imageConfig: { - ...(parsedSize.aspectRatio && { - aspectRatio: parsedSize.aspectRatio, - }), - ...(parsedSize.resolution && { - imageSize: parsedSize.resolution, - }), - }, - }), + ...(Object.keys(imageConfig).length > 0 && { imageConfig }), } const contents = this.buildContents(resolved, numberOfImages) @@ -321,7 +326,7 @@ export class GeminiImageAdapter< } private buildImagenConfig( - options: ImageGenerationOptions, + options: ImageGenerationOptions, ): GenerateImagesConfig { const { size, numberOfImages, modelOptions } = options @@ -329,11 +334,62 @@ export class GeminiImageAdapter< // vendor `GenerateImagesConfig` fields are `field?: T` (no `| undefined`), // so we can only assign the property when we actually have a value. const sizeAspectRatio = size ? sizeToAspectRatio(size) : undefined + + // Named picks, never a wholesale spread — the mirror image of the native + // path below. A native-only field (safetySettings, thinkingConfig, + // imageConfig, systemInstruction) belongs to GenerateContentConfig and is + // rejected by generateImages with 400 INVALID_ARGUMENT, so it must not be + // able to reach here even when the caller's `modelOptions` was typed + // against both shapes at once (e.g. an adapter inferred from a union of + // model names). return { numberOfImages: numberOfImages ?? 1, - // Map size to aspect ratio if provided (modelOptions.aspectRatio will override) + // Map size to aspect ratio if provided; modelOptions.aspectRatio, + // picked after it, overrides. ...(sizeAspectRatio !== undefined && { aspectRatio: sizeAspectRatio }), - ...modelOptions, + ...(modelOptions?.aspectRatio !== undefined && { + aspectRatio: modelOptions.aspectRatio, + }), + ...(modelOptions?.personGeneration !== undefined && { + personGeneration: modelOptions.personGeneration, + }), + ...(modelOptions?.safetyFilterLevel !== undefined && { + safetyFilterLevel: modelOptions.safetyFilterLevel, + }), + ...(modelOptions?.seed !== undefined && { seed: modelOptions.seed }), + ...(modelOptions?.addWatermark !== undefined && { + addWatermark: modelOptions.addWatermark, + }), + ...(modelOptions?.language !== undefined && { + language: modelOptions.language, + }), + ...(modelOptions?.negativePrompt !== undefined && { + negativePrompt: modelOptions.negativePrompt, + }), + ...(modelOptions?.outputMimeType !== undefined && { + outputMimeType: modelOptions.outputMimeType, + }), + ...(modelOptions?.outputCompressionQuality !== undefined && { + outputCompressionQuality: modelOptions.outputCompressionQuality, + }), + ...(modelOptions?.guidanceScale !== undefined && { + guidanceScale: modelOptions.guidanceScale, + }), + ...(modelOptions?.enhancePrompt !== undefined && { + enhancePrompt: modelOptions.enhancePrompt, + }), + ...(modelOptions?.includeSafetyAttributes !== undefined && { + includeSafetyAttributes: modelOptions.includeSafetyAttributes, + }), + ...(modelOptions?.includeRaiReason !== undefined && { + includeRaiReason: modelOptions.includeRaiReason, + }), + ...(modelOptions?.outputGcsUri !== undefined && { + outputGcsUri: modelOptions.outputGcsUri, + }), + ...(modelOptions?.labels !== undefined && { + labels: modelOptions.labels, + }), } } diff --git a/packages/ai-gemini/src/image/image-provider-options.ts b/packages/ai-gemini/src/image/image-provider-options.ts index 84fd66da76..364fad9f56 100644 --- a/packages/ai-gemini/src/image/image-provider-options.ts +++ b/packages/ai-gemini/src/image/image-provider-options.ts @@ -1,12 +1,24 @@ import type { GeminiImageModels } from '../model-meta' import type { + ContentUnion, + ImageConfig, ImagePromptLanguage, PersonGeneration, SafetyFilterLevel, + SafetySetting, + ThinkingConfig, } from '@google/genai' // Re-export SDK types so users can use them directly -export type { ImagePromptLanguage, PersonGeneration, SafetyFilterLevel } +export type { + ContentUnion, + ImageConfig, + ImagePromptLanguage, + PersonGeneration, + SafetyFilterLevel, + SafetySetting, + ThinkingConfig, +} /** * Gemini Imagen aspect ratio options @@ -121,11 +133,77 @@ export interface GeminiImageProviderOptions { } /** - * Model-specific provider options mapping - * Currently all Imagen models use the same options structure + * Provider options for Gemini native image models (Nano Banana and friends). + * + * These models are served by `generateContent`, not `generateImages`, so they + * are configured by @google/genai's `GenerateContentConfig` — a different + * shape from the Imagen-only {@link GeminiImageProviderOptions} above. Only + * the `GenerateContentConfig` fields with clear image-generation semantics are + * surfaced; sampling knobs (`temperature`, `topK`, …) and chat-only plumbing + * (`tools`, `responseSchema`, …) are deliberately left out. + * + * `responseModalities` is intentionally absent: the adapter always requests + * `['TEXT', 'IMAGE']`, and letting a caller override it would silently disable + * image output on an image-generation call. + */ +export interface GeminiNativeImageProviderOptions { + /** + * Optional seed for reproducible image generation + * When the same seed is used with the same prompt and settings, + * you should get similar (though not identical) results + */ + seed?: number + + /** + * Per-category safety thresholds applied to the request + * Each entry pairs a HarmCategory with a HarmBlockThreshold + */ + safetySettings?: Array + + /** + * Controls the model's internal reasoning before it emits an image + * Use to raise or disable the thinking budget on models that support it + */ + thinkingConfig?: ThinkingConfig + + /** + * Native image output controls (aspect ratio, resolution tier, …) + * Merged over the values derived from the portable `size` option, so + * fields set here win per field while the rest of `size` is preserved + */ + imageConfig?: ImageConfig + + /** + * System-level instructions that steer the model for the whole request, + * e.g. a house art direction applied on top of the per-call prompt + */ + systemInstruction?: ContentUnion +} + +/** + * Every provider-option field this adapter understands, across both API + * paths. Used as the adapter's base (model-agnostic) option type; the + * per-model map below is what narrows a given model to the half that + * actually applies to it. + */ +export type GeminiAnyImageProviderOptions = GeminiImageProviderOptions & + GeminiNativeImageProviderOptions + +/** + * Model-specific provider options mapping. + * Gemini native image models go through `generateContent` and take + * `GenerateContentConfig` fields; Imagen models go through `generateImages` + * and take `GenerateImagesConfig` fields. Mirrors the native/Imagen split in + * {@link GeminiImageModelSizeByName} and + * {@link GeminiImageModelInputModalitiesByName}. */ export type GeminiImageModelProviderOptionsByName = { - [K in GeminiImageModels]: GeminiImageProviderOptions + [K in GeminiNativeImageModels]: GeminiNativeImageProviderOptions +} & { + [K in Exclude< + GeminiImageModels, + GeminiNativeImageModels + >]: GeminiImageProviderOptions } /** @@ -173,12 +251,47 @@ export type GeminiNativeImageSize = /** * Gemini native image models that use the generateContent API path. * These models support template literal sizes (aspectRatio_resolution). + * + * This array is the single source of truth for the native/Imagen split: the + * `GeminiNativeImageModels` union and the per-model option/size/modality maps + * all derive from it. The `satisfies` clause makes a typo (or a name that + * isn't a known image model) a build error rather than a phantom key on every + * per-model map. + * + * It is also the single source of truth for the adapter's *runtime* routing + * — see {@link isGeminiNativeImageModel}. Adding a new `gemini-*` image model + * means adding it here as well as to `GEMINI_IMAGE_MODELS` in model-meta; + * until it is listed here it routes to the Imagen API instead and fails + * loudly on the first call, rather than silently taking the wrong option + * shape. */ +export const GEMINI_NATIVE_IMAGE_MODELS = [ + 'gemini-3.1-flash-image-preview', + 'gemini-3.1-flash-lite-image', + 'gemini-3-pro-image-preview', + 'gemini-2.5-flash-image', +] as const satisfies ReadonlyArray + export type GeminiNativeImageModels = - | 'gemini-3.1-flash-image-preview' - | 'gemini-3.1-flash-lite-image' - | 'gemini-3-pro-image-preview' - | 'gemini-2.5-flash-image' + (typeof GEMINI_NATIVE_IMAGE_MODELS)[number] + +const NATIVE_IMAGE_MODEL_NAMES: ReadonlySet = new Set( + GEMINI_NATIVE_IMAGE_MODELS, +) + +/** + * Runtime counterpart to {@link GeminiNativeImageModels} — decides which of + * the two Gemini image APIs a model goes to. + * + * Membership in {@link GEMINI_NATIVE_IMAGE_MODELS}, not a `gemini-` prefix + * test, so the runtime route and the type-level split cannot drift apart. An + * id this package does not know about reaches the Imagen endpoint and fails + * there, which is the intended signal to add the model here rather than to + * have it silently take the native path with Imagen-shaped option types. + */ +export function isGeminiNativeImageModel(model: string): boolean { + return NATIVE_IMAGE_MODEL_NAMES.has(model) +} /** * Model-specific size options mapping. diff --git a/packages/ai-gemini/src/index.ts b/packages/ai-gemini/src/index.ts index e0c58922c0..9397ee5ae4 100644 --- a/packages/ai-gemini/src/index.ts +++ b/packages/ai-gemini/src/index.ts @@ -28,13 +28,24 @@ export { } from './adapters/image' export type { GeminiImageProviderOptions, + GeminiNativeImageProviderOptions, + GeminiAnyImageProviderOptions, GeminiImageModelProviderOptionsByName, GeminiAspectRatio, // Re-export SDK types for convenience PersonGeneration, SafetyFilterLevel, ImagePromptLanguage, + SafetySetting, + ThinkingConfig, + ImageConfig, + ContentUnion, } from './image/image-provider-options' +// `SafetySetting` is built from two SDK enums, and enums are values — they +// cannot travel through `export type`. Re-exported here so `safetySettings` +// is usable with only `@tanstack/ai-gemini` installed, without the consumer +// having to add `@google/genai` to their own dependencies. +export { HarmBlockThreshold, HarmCategory } from '@google/genai' // Embedding adapter - for embedding vectors export { diff --git a/packages/ai-gemini/tests/image-adapter.test.ts b/packages/ai-gemini/tests/image-adapter.test.ts index 7f07bb04f0..ac63d324f9 100644 --- a/packages/ai-gemini/tests/image-adapter.test.ts +++ b/packages/ai-gemini/tests/image-adapter.test.ts @@ -1,4 +1,11 @@ import { describe, it, expect, vi } from 'vitest' +import { + HarmBlockThreshold, + HarmCategory, + ImagePromptLanguage, + PersonGeneration, + SafetyFilterLevel, +} from '@google/genai' import { generateImage } from '@tanstack/ai' import { resolveDebugOption } from '@tanstack/ai/adapter-internals' import { GeminiImageAdapter, createGeminiImage } from '../src/adapters/image' @@ -10,6 +17,55 @@ import { validatePrompt, } from '../src/image/image-provider-options' +const mockImageResponse = { + candidates: [ + { + content: { + parts: [{ inlineData: { mimeType: 'image/png', data: 'out' } }], + }, + }, + ], +} + +/** + * A native-path adapter whose `client.models.generateContent` is stubbed, so + * tests can assert the exact config object handed to the SDK. + */ +function mockedNativeAdapter() { + const mockGenerateContent = vi.fn().mockResolvedValueOnce(mockImageResponse) + const adapter = createGeminiImage( + 'gemini-3.1-flash-image-preview', + 'test-api-key', + ) + ;( + adapter as unknown as { + client: { models: { generateContent: unknown } } + } + ).client = { + models: { generateContent: mockGenerateContent }, + } + return { adapter, mockGenerateContent } +} + +/** + * An Imagen-path adapter whose `client.models.generateImages` is stubbed, so + * tests can assert the exact GenerateImagesConfig handed to the SDK. + */ +function mockedImagenAdapter() { + const mockGenerateImages = vi.fn().mockResolvedValueOnce({ + generatedImages: [{ image: { imageBytes: 'imagen-b64' } }], + }) + const adapter = createGeminiImage('imagen-4.0-generate-001', 'test-api-key') + ;( + adapter as unknown as { + client: { models: { generateImages: unknown } } + } + ).client = { + models: { generateImages: mockGenerateImages }, + } + return { adapter, mockGenerateImages } +} + describe('Gemini Image Adapter', () => { describe('createGeminiImage', () => { it('creates an adapter with the provided API key', () => { @@ -721,35 +777,199 @@ describe('Gemini Image Adapter', () => { }) }) - describe('multimodal prompt (image-conditioned generation)', () => { - const testLogger = resolveDebugOption(false) - const mockImageResponse = { - candidates: [ - { - content: { - parts: [{ inlineData: { mimeType: 'image/png', data: 'out' } }], - }, + describe('native modelOptions (GenerateContentConfig)', () => { + // Regression: GeminiImageModelProviderOptionsByName used to map every + // image model — native ones included — to the Imagen-shaped + // GeminiImageProviderOptions, so these fields were a compile error and the + // adapter whitelisted only `seed`. + const safetySettings = [ + { + category: HarmCategory.HARM_CATEGORY_HATE_SPEECH, + threshold: HarmBlockThreshold.BLOCK_ONLY_HIGH, + }, + ] + + it('forwards safetySettings to generateContent', async () => { + const { adapter, mockGenerateContent } = mockedNativeAdapter() + + await generateImage({ + adapter, + prompt: 'A quiet harbour', + modelOptions: { safetySettings }, + }) + + const args = mockGenerateContent.mock.calls[0]![0] + expect(args.config.safetySettings).toEqual(safetySettings) + }) + + it('forwards thinkingConfig to generateContent', async () => { + const { adapter, mockGenerateContent } = mockedNativeAdapter() + + await generateImage({ + adapter, + prompt: 'A quiet harbour', + modelOptions: { thinkingConfig: { thinkingBudget: 512 } }, + }) + + const args = mockGenerateContent.mock.calls[0]![0] + expect(args.config.thinkingConfig).toEqual({ thinkingBudget: 512 }) + }) + + it('forwards systemInstruction to generateContent', async () => { + const { adapter, mockGenerateContent } = mockedNativeAdapter() + + await generateImage({ + adapter, + prompt: 'A quiet harbour', + modelOptions: { systemInstruction: 'Always render in watercolor.' }, + }) + + const args = mockGenerateContent.mock.calls[0]![0] + expect(args.config.systemInstruction).toBe('Always render in watercolor.') + }) + + it('merges modelOptions.imageConfig over the size-derived imageConfig', async () => { + // `size` is the portable API, `imageConfig` the provider escape hatch: + // the overriding field wins, the untouched one survives. + const { adapter, mockGenerateContent } = mockedNativeAdapter() + + await generateImage({ + adapter, + prompt: 'A quiet harbour', + size: '16:9_4K', + modelOptions: { imageConfig: { imageSize: '2K' } }, + }) + + const args = mockGenerateContent.mock.calls[0]![0] + expect(args.config.imageConfig).toEqual({ + aspectRatio: '16:9', + imageSize: '2K', + }) + }) + + it('applies modelOptions.imageConfig when no size is given', async () => { + const { adapter, mockGenerateContent } = mockedNativeAdapter() + + await generateImage({ + adapter, + prompt: 'A quiet harbour', + modelOptions: { imageConfig: { aspectRatio: '21:9' } }, + }) + + const args = mockGenerateContent.mock.calls[0]![0] + expect(args.config.imageConfig).toEqual({ aspectRatio: '21:9' }) + }) + + it('keeps Imagen models on GenerateImagesConfig with no native fields', async () => { + const { adapter, mockGenerateImages } = mockedImagenAdapter() + + await generateImage({ + adapter, + prompt: 'A quiet harbour', + size: '1920x1080', + modelOptions: { + personGeneration: PersonGeneration.ALLOW_ADULT, + negativePrompt: 'blurry', + // Native-only fields. The per-model type rejects them (hence the + // cast, as in the responseModalities regression test above), but an + // adapter inferred from a union of model names widens modelOptions + // to both shapes, so they can still arrive here at runtime — and + // generateImages answers a GenerateContentConfig field with + // 400 INVALID_ARGUMENT. The named picks must drop them. + safetySettings: [ + { + category: HarmCategory.HARM_CATEGORY_HATE_SPEECH, + threshold: HarmBlockThreshold.BLOCK_ONLY_HIGH, + }, + ], + thinkingConfig: { thinkingBudget: 512 }, + imageConfig: { imageSize: '2K' }, + systemInstruction: 'Always render in watercolor.', + } as unknown as never, + }) + + // Exact match: the Imagen path gets its own GenerateImagesConfig fields + // and nothing else. + expect(mockGenerateImages).toHaveBeenCalledWith({ + model: 'imagen-4.0-generate-001', + prompt: 'A quiet harbour', + config: { + numberOfImages: 1, + aspectRatio: '16:9', + personGeneration: PersonGeneration.ALLOW_ADULT, + negativePrompt: 'blurry', }, - ], - } + }) + }) - function mockedNativeAdapter() { - const mockGenerateContent = vi - .fn() - .mockResolvedValueOnce(mockImageResponse) - const adapter = createGeminiImage( - 'gemini-3.1-flash-image-preview', - 'test-api-key', - ) - ;( - adapter as unknown as { - client: { models: { generateContent: unknown } } - } - ).client = { - models: { generateContent: mockGenerateContent }, - } - return { adapter, mockGenerateContent } - } + it('forwards the whole Imagen option set to generateImages', async () => { + // The Imagen path picks fields by name, so this guards against a field + // being forgotten in that list. + const { adapter, mockGenerateImages } = mockedImagenAdapter() + + await generateImage({ + adapter, + prompt: 'A quiet harbour', + modelOptions: { + aspectRatio: '21:9', + personGeneration: PersonGeneration.ALLOW_ADULT, + safetyFilterLevel: SafetyFilterLevel.BLOCK_ONLY_HIGH, + seed: 42, + addWatermark: false, + language: ImagePromptLanguage.en, + negativePrompt: 'blurry', + outputMimeType: 'image/jpeg', + outputCompressionQuality: 80, + guidanceScale: 12, + enhancePrompt: true, + includeSafetyAttributes: true, + includeRaiReason: true, + outputGcsUri: 'gs://bucket/out', + labels: { team: 'design' }, + }, + }) + + expect(mockGenerateImages).toHaveBeenCalledWith({ + model: 'imagen-4.0-generate-001', + prompt: 'A quiet harbour', + config: { + numberOfImages: 1, + aspectRatio: '21:9', + personGeneration: PersonGeneration.ALLOW_ADULT, + safetyFilterLevel: SafetyFilterLevel.BLOCK_ONLY_HIGH, + seed: 42, + addWatermark: false, + language: ImagePromptLanguage.en, + negativePrompt: 'blurry', + outputMimeType: 'image/jpeg', + outputCompressionQuality: 80, + guidanceScale: 12, + enhancePrompt: true, + includeSafetyAttributes: true, + includeRaiReason: true, + outputGcsUri: 'gs://bucket/out', + labels: { team: 'design' }, + }, + }) + }) + + it('lets modelOptions.aspectRatio override the size-derived one', async () => { + const { adapter, mockGenerateImages } = mockedImagenAdapter() + + await generateImage({ + adapter, + prompt: 'A quiet harbour', + size: '1920x1080', + modelOptions: { aspectRatio: '9:16' }, + }) + + const args = mockGenerateImages.mock.calls[0]![0] + expect(args.config.aspectRatio).toBe('9:16') + }) + }) + + describe('multimodal prompt (image-conditioned generation)', () => { + const testLogger = resolveDebugOption(false) it('maps interleaved prompt parts onto multimodal contents in order', async () => { const { adapter, mockGenerateContent } = mockedNativeAdapter() diff --git a/packages/ai-gemini/tests/image-per-model-type-safety.test.ts b/packages/ai-gemini/tests/image-per-model-type-safety.test.ts new file mode 100644 index 0000000000..88e42d1299 --- /dev/null +++ b/packages/ai-gemini/tests/image-per-model-type-safety.test.ts @@ -0,0 +1,159 @@ +/** + * Per-model type-safety tests for Gemini generateImage() modelOptions. + * + * Gemini-native image models (generateContent) and Imagen models + * (generateImages) take different provider-option shapes, so + * `GeminiImageModelProviderOptionsByName` splits on the model family the same + * way the size and input-modality maps do. Positive cases compile cleanly; + * cross-family cases produce a `@ts-expect-error`. + * + * Compile-time only — `createImageOptions` builds the typed options object + * without issuing a request. + */ +import { beforeAll, describe, expectTypeOf, it } from 'vitest' +import { createImageOptions } from '@tanstack/ai' +import { geminiImage } from '../src' +import type { GeminiImageModelProviderOptionsByName } from '../src' + +// Set a dummy API key so adapter construction does not throw at runtime. +// These tests only exercise compile-time type gating; no network calls are made. +beforeAll(() => { + process.env['GOOGLE_API_KEY'] = 'sk-test-dummy' +}) + +describe('Gemini per-model image modelOptions gating', () => { + describe('gemini-3.1-flash-image-preview — native (GenerateContentConfig)', () => { + it('accepts the native option set', () => { + createImageOptions({ + adapter: geminiImage('gemini-3.1-flash-image-preview'), + prompt: 'a quiet harbour', + modelOptions: { + seed: 7, + safetySettings: [], + thinkingConfig: { thinkingBudget: 512 }, + imageConfig: { aspectRatio: '16:9', imageSize: '2K' }, + systemInstruction: 'Always render in watercolor.', + }, + }) + }) + + it('rejects Imagen-only options', () => { + // The probes are plain values that are structurally valid on + // GeminiImageProviderOptions (`negativePrompt: string`, + // `aspectRatio: GeminiAspectRatio`), so the errors below come from the + // native/Imagen split and nothing else — an enum-typed field such as + // `personGeneration` would reject a string literal on either shape and + // would still "pass" with the split reverted. + createImageOptions({ + adapter: geminiImage('gemini-3.1-flash-image-preview'), + prompt: 'a quiet harbour', + modelOptions: { + // @ts-expect-error - negativePrompt is a GenerateImagesConfig (Imagen) field + negativePrompt: 'blurry', + }, + }) + + createImageOptions({ + adapter: geminiImage('gemini-3.1-flash-image-preview'), + prompt: 'a quiet harbour', + modelOptions: { + // @ts-expect-error - aspectRatio is Imagen-only; native models use size / imageConfig + aspectRatio: '16:9', + }, + }) + }) + + it('rejects responseModalities — the adapter owns it', () => { + createImageOptions({ + adapter: geminiImage('gemini-3.1-flash-image-preview'), + prompt: 'a quiet harbour', + modelOptions: { + // @ts-expect-error - responseModalities is a protected adapter default + responseModalities: ['TEXT'], + }, + }) + }) + }) + + describe('imagen-4.0-generate-001 — Imagen (GenerateImagesConfig)', () => { + it('accepts the Imagen option set', () => { + createImageOptions({ + adapter: geminiImage('imagen-4.0-generate-001'), + prompt: 'a quiet harbour', + modelOptions: { + aspectRatio: '16:9', + negativePrompt: 'blurry', + addWatermark: true, + outputMimeType: 'image/png', + }, + }) + }) + + it('rejects native-only options', () => { + createImageOptions({ + adapter: geminiImage('imagen-4.0-generate-001'), + prompt: 'a quiet harbour', + modelOptions: { + // @ts-expect-error - safetySettings is a GenerateContentConfig (native) field + safetySettings: [], + }, + }) + }) + }) +}) + +describe('Gemini image provider options shape assertions', () => { + describe('native models take GenerateContentConfig fields', () => { + type Options = + GeminiImageModelProviderOptionsByName['gemini-3.1-flash-image-preview'] + + it('has safetySettings', () => { + expectTypeOf().toHaveProperty('safetySettings') + }) + it('has thinkingConfig', () => { + expectTypeOf().toHaveProperty('thinkingConfig') + }) + it('has imageConfig', () => { + expectTypeOf().toHaveProperty('imageConfig') + }) + it('has systemInstruction', () => { + expectTypeOf().toHaveProperty('systemInstruction') + }) + it('has seed', () => { + expectTypeOf().toHaveProperty('seed') + }) + }) + + describe('Imagen models keep GenerateImagesConfig fields', () => { + type Options = + GeminiImageModelProviderOptionsByName['imagen-4.0-generate-001'] + + it('has aspectRatio', () => { + expectTypeOf().toHaveProperty('aspectRatio') + }) + it('has personGeneration', () => { + expectTypeOf().toHaveProperty('personGeneration') + }) + it('has negativePrompt', () => { + expectTypeOf().toHaveProperty('negativePrompt') + }) + }) + + describe('every native model id resolves to the native shape', () => { + it('gemini-3.1-flash-lite-image', () => { + expectTypeOf< + GeminiImageModelProviderOptionsByName['gemini-3.1-flash-lite-image'] + >().toHaveProperty('imageConfig') + }) + it('gemini-3-pro-image-preview', () => { + expectTypeOf< + GeminiImageModelProviderOptionsByName['gemini-3-pro-image-preview'] + >().toHaveProperty('imageConfig') + }) + it('gemini-2.5-flash-image', () => { + expectTypeOf< + GeminiImageModelProviderOptionsByName['gemini-2.5-flash-image'] + >().toHaveProperty('imageConfig') + }) + }) +}) From ddc4bdae9da201895528a2c2037fba3f62db2fc5 Mon Sep 17 00:00:00 2001 From: L4Ph Date: Fri, 14 Aug 2026 09:47:17 +0900 Subject: [PATCH 2/3] docs(ai-gemini): describe image routing by model list, not prefix The prose and the changeset still said the adapter routes on a `gemini-` prefix. It routes on membership in GEMINI_NATIVE_IMAGE_MODELS, so an unlisted `gemini-*` id reaches generateImages and fails there. Raised by CodeRabbit on #1103. --- .changeset/gemini-native-image-model-options.md | 2 +- docs/adapters/gemini.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/.changeset/gemini-native-image-model-options.md b/.changeset/gemini-native-image-model-options.md index 6b22f36b0b..3479d3704e 100644 --- a/.changeset/gemini-native-image-model-options.md +++ b/.changeset/gemini-native-image-model-options.md @@ -4,7 +4,7 @@ Type Gemini-native image models with their own provider options. `GeminiImageModelProviderOptionsByName` mapped **every** image model to the Imagen-shaped `GeminiImageProviderOptions`, so `modelOptions: { safetySettings, thinkingConfig, imageConfig, systemInstruction }` was a compile error on `gemini-3.1-flash-image-preview`, `gemini-3.1-flash-lite-image`, `gemini-3-pro-image-preview`, and `gemini-2.5-flash-image` — even though those models are served by `generateContent`, whose `GenerateContentConfig` accepts all of them. The adapter compensated by forwarding only `seed`, silently dropping anything else. -The map now splits native vs Imagen, mirroring the split already used by `GeminiImageModelSizeByName` and `GeminiImageModelInputModalitiesByName`: native models get the new `GeminiNativeImageProviderOptions` (`seed`, `safetySettings`, `thinkingConfig`, `imageConfig`, `systemInstruction`), Imagen models keep `GeminiImageProviderOptions`. Both API paths now pick their config fields by name — never a wholesale spread — so neither shape's fields can reach the other's endpoint. +The map now splits native vs Imagen, mirroring the split already used by `GeminiImageModelSizeByName` and `GeminiImageModelInputModalitiesByName`: native models get the new `GeminiNativeImageProviderOptions` (`seed`, `safetySettings`, `thinkingConfig`, `imageConfig`, `systemInstruction`), Imagen models keep `GeminiImageProviderOptions`. Both API paths now pick their config fields by name — never a wholesale spread — so neither shape's fields can reach the other's endpoint. Runtime routing moves the same way, off a `gemini-` prefix test onto membership in `GEMINI_NATIVE_IMAGE_MODELS`: a `gemini-*` image model not present in that list now routes to `generateImages` instead of `generateContent`, so it fails against that endpoint rather than silently taking the native path. `responseModalities` stays a protected adapter default (`['TEXT', 'IMAGE']`) and is deliberately absent from the new type. `modelOptions.imageConfig` merges **over** the `imageConfig` derived from the portable `size` option, per field — passing only `imageConfig.imageSize` keeps the `aspectRatio` that `size` implied. `HarmCategory` and `HarmBlockThreshold` are now re-exported so `safetySettings` can be written without adding `@google/genai` to your own dependencies. diff --git a/docs/adapters/gemini.md b/docs/adapters/gemini.md index c137309e11..c933e017be 100644 --- a/docs/adapters/gemini.md +++ b/docs/adapters/gemini.md @@ -414,7 +414,7 @@ The Gemini adapter supports two types of image generation: - **Gemini native image models** (NanoBanana) — Use the `generateContent` API with models like `gemini-3.1-flash-image-preview`. These support extended resolution tiers (1K, 2K, 4K) and aspect ratio control. - **Imagen models** — Use the `generateImages` API with models like `imagen-4.0-generate-001`. These are dedicated image generation models with WIDTHxHEIGHT sizing. -The adapter automatically routes to the correct API based on the model name — models starting with `gemini-` use `generateContent`, while `imagen-` models use `generateImages`. +The adapter automatically routes to the correct API based on membership in the known list of Gemini native models (`GEMINI_NATIVE_IMAGE_MODELS`) — the Gemini native models listed above use `generateContent`, while Imagen models, and any model id this package doesn't know about, use `generateImages`. ### Example: Gemini Native Image Generation (NanoBanana) From b0dcc1dfd357c182422f2e39162d01a211c329ea Mon Sep 17 00:00:00 2001 From: L4Ph Date: Sat, 15 Aug 2026 10:29:54 +0900 Subject: [PATCH 3/3] test(e2e): cover Gemini native image modelOptions on the wire aimock's handleGemini has no image-response branch and its journal stores a lossy OpenAI-shaped translation that drops safetySettings and generationConfig, so neither fixtures nor /journal can see this path. Mount the endpoint directly instead -- the same escape hatch geminiTTSMount() already uses for the identical {model}:generateContent inlineData shape -- and reject the request if the fields are absent, so a dropped option fails the spec instead of passing silently. Verified revert-proof: with packages/ai-gemini/src reverted to the parent commit, the spec fails with "Missing top-level safetySettings". --- testing/e2e/global-setup.ts | 184 ++++++++++++++++++ testing/e2e/src/routeTree.gen.ts | 22 +++ .../routes/api.gemini-native-image-wire.ts | 71 +++++++ .../tests/gemini-native-image-wire.spec.ts | 43 ++++ 4 files changed, 320 insertions(+) create mode 100644 testing/e2e/src/routes/api.gemini-native-image-wire.ts create mode 100644 testing/e2e/tests/gemini-native-image-wire.spec.ts diff --git a/testing/e2e/global-setup.ts b/testing/e2e/global-setup.ts index 0502e4418c..25d8f419cb 100644 --- a/testing/e2e/global-setup.ts +++ b/testing/e2e/global-setup.ts @@ -55,6 +55,13 @@ export default async function globalSetup() { '/v1beta/models/gemini-3.1-flash-tts-preview:generateContent', geminiTTSMount(), ) + // Gemini native image generation hits the same generateContent endpoint + // shape, one model id over — see geminiNativeImageMount for why it needs + // a hand-mocked response and a raw-body wire-shape check of its own. + mock.mount( + '/v1beta/models/gemini-2.5-flash-image:generateContent', + geminiNativeImageMount(), + ) // Gemini Veo video generation. aimock 1.29 mocks Gemini's `:predict` // (Imagen) endpoint but not the long-running `:predictLongRunning` + // operations-polling pair Veo uses, so mount both here. Non-Veo paths @@ -265,6 +272,183 @@ function geminiTTSMount(): Mountable { } } +/** + * Gemini native image generation hits the standard Gemini generateContent + * endpoint too (POST /v1beta/models/{model}:generateContent) — the same + * shape geminiTTSMount above targets, just with `image/png` inlineData + * instead of PCM audio. aimock's native handleGemini recognizes only + * text / tool-call / text-with-tool-call / audio fixture response shapes: + * isImageResponse (helpers.js) is defined but never imported into + * gemini.js, so there is no image-response branch at all. A fixture shaped + * `{image}`/`{images}` matched against this endpoint falls through every + * isXResponse() check and hits the final fallback — a 500 "Fixture response + * did not match any known type." Native image generation needs a + * hand-mocked response for the same reason TTS does. + * + * There's a second, PR-specific reason this needs its own mount rather than + * a fixture even if one could match: aimock's request-journaling for this + * endpoint goes through geminiToCompletionRequest, which reshapes the raw + * Gemini request into an OpenAI-chat-shaped completionReq carrying only + * {model, messages, stream, temperature, max_tokens, top_p, top_k, tools} — + * it silently drops safetySettings, generationConfig.thinkingConfig, and + * generationConfig.imageConfig before anything is journaled. So GET /journal + * cannot show whether those fields reached the wire, even in principle. + * This mount reads the raw, untranslated body instead (see + * readJsonRequestBody, used the same way by the BytePlus mounts below) and + * validates it directly — the BytePlus mounts' house pattern of turning a + * dropped field into a failing spec rather than a silently green one. + * + * Mounted at the exact model+endpoint path (not the shared '/v1beta/models' + * prefix geminiVeoMount/geminiBatchEmbedMount use below) so it can never + * intercept an unrelated Gemini chat/text generateContent call for a + * different model. + * + * `/api/gemini-native-image-wire` (see that route) drives this with + * `modelOptions: { safetySettings, thinkingConfig }`. Reverting the + * ai-gemini fix that stops dropping modelOptions on the native image path + * removes both from the outgoing `nativeConfig`, so the request this mount + * receives is missing them, this mount answers 400, and the route surfaces + * that as `ok: false` — the companion spec's revert-detection mechanism. + */ +function geminiNativeImageMount(): Mountable { + // 1x1 transparent PNG — just enough for transformGeminiResponse's + // inlineData branch to produce a GeneratedImage. Mirrors FAKE_PCM_BYTES / + // FAKE_MP3_BYTES above: content fidelity isn't under test here. + const PNG_1X1_BASE64 = + 'iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mNk+A8AAQUBAScY42YAAAAASUVORK5CYII=' + + // Field names that only belong on Imagen's GenerateImagesConfig, never on + // generateContent's GenerateContentConfig. The adapter's `nativeConfig` + // picks fields by name specifically so modelOptions carrying both shapes + // (or a future regression back to a wholesale `...modelOptions` spread) + // can't let one through — mirrors the field list documented on + // GeminiImageProviderOptions in image-provider-options.ts, minus `seed` + // (legitimately shared by both configs and always forwarded) and `labels` + // (also a real GenerateContentConfig field name, but the Gemini Developer + // API rejects it outright rather than letting it reach the wire, so it + // can't appear in a body this mount ever sees). + const IMAGEN_ONLY_FIELDS = [ + 'personGeneration', + 'safetyFilterLevel', + 'addWatermark', + 'language', + 'negativePrompt', + 'outputMimeType', + 'outputCompressionQuality', + 'guidanceScale', + 'enhancePrompt', + 'includeSafetyAttributes', + 'includeRaiReason', + 'outputGcsUri', + // Imagen's own top-level `aspectRatio` (GenerateImagesConfig) is distinct + // from the native path's nested generationConfig.imageConfig.aspectRatio + // — its presence at either level here would mean the two configs got + // crossed. + 'aspectRatio', + ] + + return { + async handleRequest( + req: http.IncomingMessage, + res: http.ServerResponse, + // Exact-path mount — pathname is "/" for the one path this is + // registered on. + pathname: string, + ): Promise { + if (pathname !== '/' || req.method !== 'POST') return false + + const body = await readJsonRequestBody(req) + if (!body) { + return rejectGeminiImageRequest(res, 'Malformed JSON body.') + } + const generationConfig = asRecord(body.generationConfig) + + const leaked = IMAGEN_ONLY_FIELDS.find( + (name) => + name in body || (generationConfig && name in generationConfig), + ) + if (leaked) { + return rejectGeminiImageRequest( + res, + `Imagen-only field "${leaked}" reached generateContent — GenerateImagesConfig and GenerateContentConfig got crossed.`, + ) + } + + if ( + !Array.isArray(body.safetySettings) || + body.safetySettings.length === 0 + ) { + return rejectGeminiImageRequest( + res, + 'Missing top-level safetySettings (modelOptions.safetySettings did not reach the wire).', + ) + } + if ( + !generationConfig || + typeof generationConfig.thinkingConfig !== 'object' || + generationConfig.thinkingConfig === null + ) { + return rejectGeminiImageRequest( + res, + 'Missing generationConfig.thinkingConfig (modelOptions.thinkingConfig did not reach the wire).', + ) + } + + res.statusCode = 200 + res.setHeader('Content-Type', 'application/json') + res.end( + JSON.stringify({ + candidates: [ + { + content: { + role: 'model', + parts: [ + { + inlineData: { + mimeType: 'image/png', + data: PNG_1X1_BASE64, + }, + }, + ], + }, + finishReason: 'STOP', + index: 0, + }, + ], + usageMetadata: { + promptTokenCount: 8, + candidatesTokenCount: 1290, + totalTokenCount: 1298, + }, + }), + ) + return true + }, + } +} + +/** + * Rejects with Gemini's real MLDev error envelope shape + * (`{ error: { code, message, status } }`) — the same fallback shape + * @google/genai's own `throwErrorIfNotOK` builds for a non-JSON error body, + * so the thrown ApiError's `message` carries the actual validation failure + * (JSON.stringify'd) for debugging, the same way rejectArkRequest / + * rejectVoiceRequest do for BytePlus below. + */ +function rejectGeminiImageRequest( + res: http.ServerResponse, + message: string, +): true { + res.statusCode = 400 + res.setHeader('Content-Type', 'application/json') + res.end( + JSON.stringify({ + error: { code: 400, message, status: 'INVALID_ARGUMENT' }, + }), + ) + return true +} + function grokSTTMount(): Mountable { return { async handleRequest( diff --git a/testing/e2e/src/routeTree.gen.ts b/testing/e2e/src/routeTree.gen.ts index c6d9853490..5ef35eee9e 100644 --- a/testing/e2e/src/routeTree.gen.ts +++ b/testing/e2e/src/routeTree.gen.ts @@ -59,6 +59,7 @@ import { Route as ApiInterruptsTestRouteImport } from './routes/api.interrupts-t import { Route as ApiImageRouteImport } from './routes/api.image' import { Route as ApiGenerationPersistenceServerRouteImport } from './routes/api.generation-persistence-server' import { Route as ApiGenerationPersistenceResumeRouteImport } from './routes/api.generation-persistence-resume' +import { Route as ApiGeminiNativeImageWireRouteImport } from './routes/api.gemini-native-image-wire' import { Route as ApiForeignInterruptRouteImport } from './routes/api.foreign-interrupt' import { Route as ApiEmbeddingRouteImport } from './routes/api.embedding' import { Route as ApiDurableTakeoverRouteImport } from './routes/api.durable-takeover' @@ -336,6 +337,12 @@ const ApiGenerationPersistenceResumeRoute = path: '/api/generation-persistence-resume', getParentRoute: () => rootRouteImport, } as any) +const ApiGeminiNativeImageWireRoute = + ApiGeminiNativeImageWireRouteImport.update({ + id: '/api/gemini-native-image-wire', + path: '/api/gemini-native-image-wire', + getParentRoute: () => rootRouteImport, + } as any) const ApiForeignInterruptRoute = ApiForeignInterruptRouteImport.update({ id: '/api/foreign-interrupt', path: '/api/foreign-interrupt', @@ -453,6 +460,7 @@ export interface FileRoutesByFullPath { '/api/durable-takeover': typeof ApiDurableTakeoverRoute '/api/embedding': typeof ApiEmbeddingRoute '/api/foreign-interrupt': typeof ApiForeignInterruptRoute + '/api/gemini-native-image-wire': typeof ApiGeminiNativeImageWireRoute '/api/generation-persistence-resume': typeof ApiGenerationPersistenceResumeRoute '/api/generation-persistence-server': typeof ApiGenerationPersistenceServerRoute '/api/image': typeof ApiImageRouteWithChildren @@ -522,6 +530,7 @@ export interface FileRoutesByTo { '/api/durable-takeover': typeof ApiDurableTakeoverRoute '/api/embedding': typeof ApiEmbeddingRoute '/api/foreign-interrupt': typeof ApiForeignInterruptRoute + '/api/gemini-native-image-wire': typeof ApiGeminiNativeImageWireRoute '/api/generation-persistence-resume': typeof ApiGenerationPersistenceResumeRoute '/api/generation-persistence-server': typeof ApiGenerationPersistenceServerRoute '/api/image': typeof ApiImageRouteWithChildren @@ -592,6 +601,7 @@ export interface FileRoutesById { '/api/durable-takeover': typeof ApiDurableTakeoverRoute '/api/embedding': typeof ApiEmbeddingRoute '/api/foreign-interrupt': typeof ApiForeignInterruptRoute + '/api/gemini-native-image-wire': typeof ApiGeminiNativeImageWireRoute '/api/generation-persistence-resume': typeof ApiGenerationPersistenceResumeRoute '/api/generation-persistence-server': typeof ApiGenerationPersistenceServerRoute '/api/image': typeof ApiImageRouteWithChildren @@ -663,6 +673,7 @@ export interface FileRouteTypes { | '/api/durable-takeover' | '/api/embedding' | '/api/foreign-interrupt' + | '/api/gemini-native-image-wire' | '/api/generation-persistence-resume' | '/api/generation-persistence-server' | '/api/image' @@ -732,6 +743,7 @@ export interface FileRouteTypes { | '/api/durable-takeover' | '/api/embedding' | '/api/foreign-interrupt' + | '/api/gemini-native-image-wire' | '/api/generation-persistence-resume' | '/api/generation-persistence-server' | '/api/image' @@ -801,6 +813,7 @@ export interface FileRouteTypes { | '/api/durable-takeover' | '/api/embedding' | '/api/foreign-interrupt' + | '/api/gemini-native-image-wire' | '/api/generation-persistence-resume' | '/api/generation-persistence-server' | '/api/image' @@ -871,6 +884,7 @@ export interface RootRouteChildren { ApiDurableTakeoverRoute: typeof ApiDurableTakeoverRoute ApiEmbeddingRoute: typeof ApiEmbeddingRoute ApiForeignInterruptRoute: typeof ApiForeignInterruptRoute + ApiGeminiNativeImageWireRoute: typeof ApiGeminiNativeImageWireRoute ApiGenerationPersistenceResumeRoute: typeof ApiGenerationPersistenceResumeRoute ApiGenerationPersistenceServerRoute: typeof ApiGenerationPersistenceServerRoute ApiImageRoute: typeof ApiImageRouteWithChildren @@ -1258,6 +1272,13 @@ declare module '@tanstack/react-router' { preLoaderRoute: typeof ApiGenerationPersistenceResumeRouteImport parentRoute: typeof rootRouteImport } + '/api/gemini-native-image-wire': { + id: '/api/gemini-native-image-wire' + path: '/api/gemini-native-image-wire' + fullPath: '/api/gemini-native-image-wire' + preLoaderRoute: typeof ApiGeminiNativeImageWireRouteImport + parentRoute: typeof rootRouteImport + } '/api/foreign-interrupt': { id: '/api/foreign-interrupt' path: '/api/foreign-interrupt' @@ -1468,6 +1489,7 @@ const rootRouteChildren: RootRouteChildren = { ApiDurableTakeoverRoute: ApiDurableTakeoverRoute, ApiEmbeddingRoute: ApiEmbeddingRoute, ApiForeignInterruptRoute: ApiForeignInterruptRoute, + ApiGeminiNativeImageWireRoute: ApiGeminiNativeImageWireRoute, ApiGenerationPersistenceResumeRoute: ApiGenerationPersistenceResumeRoute, ApiGenerationPersistenceServerRoute: ApiGenerationPersistenceServerRoute, ApiImageRoute: ApiImageRouteWithChildren, diff --git a/testing/e2e/src/routes/api.gemini-native-image-wire.ts b/testing/e2e/src/routes/api.gemini-native-image-wire.ts new file mode 100644 index 0000000000..94c15a246a --- /dev/null +++ b/testing/e2e/src/routes/api.gemini-native-image-wire.ts @@ -0,0 +1,71 @@ +import { createFileRoute } from '@tanstack/react-router' +import { generateImage } from '@tanstack/ai' +import { createImageAdapter } from '@/lib/media-providers' + +/** + * Wire-format verification for Gemini-native `modelOptions` on the image + * generation path (fix/gemini-native-image-model-options). + * + * Before that fix, `GeminiImageAdapter`'s `generateWithGeminiApi` only ever + * forwarded `modelOptions.seed` into the `generateContent` request — + * `safetySettings`, `thinkingConfig`, `imageConfig`, and `systemInstruction` + * were silently dropped even though the adapter's provider-options type + * (`GeminiNativeImageProviderOptions`) already declared them. This route + * drives `generateImage()` against `gemini-2.5-flash-image` with + * `modelOptions: { safetySettings, thinkingConfig }` set, hitting + * `geminiNativeImageMount` in global-setup.ts — a hand-mocked + * `POST /v1beta/models/gemini-2.5-flash-image:generateContent` endpoint that + * reads the raw, untranslated request body (aimock's own journal cannot see + * these fields for this endpoint — see that mount's comment) and rejects + * with 400 unless `safetySettings` is present at the request root and + * `generationConfig.thinkingConfig` is present nested, and rejects unless no + * Imagen-only field (`personGeneration`, `negativePrompt`, a root-level + * `aspectRatio`, …) is present anywhere in the body. + * + * A regression that stops forwarding `modelOptions` on this path — reverting + * to only `seed`, or reverting to a wholesale `...modelOptions` spread that + * lets an Imagen field cross over — makes the mount reject the request, the + * adapter's `client.models.generateContent()` call throws, and this route + * returns `ok: false`. The companion spec asserts `ok: true`. + */ +export const Route = createFileRoute('/api/gemini-native-image-wire')({ + server: { + handlers: { + POST: async () => { + const adapter = createImageAdapter('gemini') + + try { + const result = await generateImage({ + adapter, + prompt: 'a guitar in a music store', + stream: false, + modelOptions: { + safetySettings: [ + { + category: 'HARM_CATEGORY_DANGEROUS_CONTENT', + threshold: 'BLOCK_ONLY_HIGH', + }, + ], + thinkingConfig: { thinkingBudget: 128 }, + }, + }) + return new Response( + JSON.stringify({ ok: true, images: result.images.length }), + { + status: 200, + headers: { 'Content-Type': 'application/json' }, + }, + ) + } catch (error) { + return new Response( + JSON.stringify({ + ok: false, + error: error instanceof Error ? error.message : String(error), + }), + { status: 200, headers: { 'Content-Type': 'application/json' } }, + ) + } + }, + }, + }, +}) diff --git a/testing/e2e/tests/gemini-native-image-wire.spec.ts b/testing/e2e/tests/gemini-native-image-wire.spec.ts new file mode 100644 index 0000000000..283638baf3 --- /dev/null +++ b/testing/e2e/tests/gemini-native-image-wire.spec.ts @@ -0,0 +1,43 @@ +import { test, expect } from './fixtures' + +/** + * Wire-format verification for Gemini-native `modelOptions` on the image + * generation path (fix/gemini-native-image-model-options). + * + * `/api/gemini-native-image-wire` drives `generateImage()` against + * `gemini-2.5-flash-image` with `modelOptions: { safetySettings, + * thinkingConfig }`. That request lands on `geminiNativeImageMount` in + * global-setup.ts, which reads the raw, untranslated request body (aimock's + * own journal normalises this endpoint's requests and drops these exact + * fields before journalling — see that mount's comment) and rejects with 400 + * unless `safetySettings` is present at the request root, + * `generationConfig.thinkingConfig` is present nested under + * `generationConfig`, and no Imagen-only field (`personGeneration`, + * `negativePrompt`, a root-level `aspectRatio`, …) appears anywhere in the + * body. + * + * Before the fix, `generateWithGeminiApi` only forwarded `modelOptions.seed` + * — `safetySettings` and `thinkingConfig` were silently dropped even though + * the adapter's own provider-options type already declared them. Reverting + * the fix reproduces that: the outgoing request loses both fields, the mount + * rejects it with 400, `client.models.generateContent()` throws, and the + * route returns `ok: false` — this spec's `ok` assertion fails. + */ +test.describe('gemini native image — modelOptions reach the generateContent wire', () => { + test('safetySettings and thinkingConfig survive to the request; no Imagen field does', async ({ + request, + }) => { + const res = await request.post('/api/gemini-native-image-wire') + expect(res.ok()).toBe(true) + + const { ok, images, error } = (await res.json()) as { + ok: boolean + images?: number + error?: string + } + + expect(error ?? null).toBeNull() + expect(ok).toBe(true) + expect(images).toBe(1) + }) +})