diff --git a/.changeset/gemini-native-image-model-options.md b/.changeset/gemini-native-image-model-options.md new file mode 100644 index 0000000000..3479d3704e --- /dev/null +++ b/.changeset/gemini-native-image-model-options.md @@ -0,0 +1,11 @@ +--- +'@tanstack/ai-gemini': minor +--- + +Type Gemini-native image models with their own provider options. `GeminiImageModelProviderOptionsByName` mapped **every** image model to the Imagen-shaped `GeminiImageProviderOptions`, so `modelOptions: { safetySettings, thinkingConfig, imageConfig, systemInstruction }` was a compile error on `gemini-3.1-flash-image-preview`, `gemini-3.1-flash-lite-image`, `gemini-3-pro-image-preview`, and `gemini-2.5-flash-image` — even though those models are served by `generateContent`, whose `GenerateContentConfig` accepts all of them. The adapter compensated by forwarding only `seed`, silently dropping anything else. + +The map now splits native vs Imagen, mirroring the split already used by `GeminiImageModelSizeByName` and `GeminiImageModelInputModalitiesByName`: native models get the new `GeminiNativeImageProviderOptions` (`seed`, `safetySettings`, `thinkingConfig`, `imageConfig`, `systemInstruction`), Imagen models keep `GeminiImageProviderOptions`. Both API paths now pick their config fields by name — never a wholesale spread — so neither shape's fields can reach the other's endpoint. Runtime routing moves the same way, off a `gemini-` prefix test onto membership in `GEMINI_NATIVE_IMAGE_MODELS`: a `gemini-*` image model not present in that list now routes to `generateImages` instead of `generateContent`, so it fails against that endpoint rather than silently taking the native path. + +`responseModalities` stays a protected adapter default (`['TEXT', 'IMAGE']`) and is deliberately absent from the new type. `modelOptions.imageConfig` merges **over** the `imageConfig` derived from the portable `size` option, per field — passing only `imageConfig.imageSize` keeps the `aspectRatio` that `size` implied. `HarmCategory` and `HarmBlockThreshold` are now re-exported so `safetySettings` can be written without adding `@google/genai` to your own dependencies. + +**BREAKING (types only):** Imagen fields no longer compile on the four Gemini-native image models — `aspectRatio`, `negativePrompt`, `personGeneration`, `safetyFilterLevel`, `addWatermark`, `language`, `outputMimeType`, `outputCompressionQuality`, `guidanceScale`, `enhancePrompt`, `includeSafetyAttributes`, `includeRaiReason`, `outputGcsUri`, `labels`. They previously type-checked but were already dropped at runtime (only `seed` was ever forwarded to `generateContent`), so no request behaviour changes — the compiler now reports what was already happening. Migrate `aspectRatio` to the portable `size` option (`'16:9_4K'`) or to `modelOptions.imageConfig`, and drop the rest. `GeminiImageAdapter.generateImages` (and its `~types.providerOptions`) also widens from `ImageGenerationOptions` to `ImageGenerationOptions`, which affects code structurally annotated against the old signature. diff --git a/docs/adapters/gemini.md b/docs/adapters/gemini.md index 31da9cf6e2..c933e017be 100644 --- a/docs/adapters/gemini.md +++ b/docs/adapters/gemini.md @@ -414,7 +414,7 @@ The Gemini adapter supports two types of image generation: - **Gemini native image models** (NanoBanana) — Use the `generateContent` API with models like `gemini-3.1-flash-image-preview`. These support extended resolution tiers (1K, 2K, 4K) and aspect ratio control. - **Imagen models** — Use the `generateImages` API with models like `imagen-4.0-generate-001`. These are dedicated image generation models with WIDTHxHEIGHT sizing. -The adapter automatically routes to the correct API based on the model name — models starting with `gemini-` use `generateContent`, while `imagen-` models use `generateImages`. +The adapter automatically routes to the correct API based on membership in the known list of Gemini native models (`GEMINI_NATIVE_IMAGE_MODELS`) — the Gemini native models listed above use `generateContent`, while Imagen models, and any model id this package doesn't know about, use `generateImages`. ### Example: Gemini Native Image Generation (NanoBanana) @@ -494,6 +494,10 @@ const result = await generateImage({ ### Image Model Options +`modelOptions` is typed per model family, because the two families hit different APIs. + +Imagen models (`generateImages`) take `GenerateImagesConfig` fields: + ```typescript ignore import { generateImage } from "@tanstack/ai"; import { geminiImage } from "@tanstack/ai-gemini"; @@ -509,6 +513,26 @@ const result = await generateImage({ }); ``` +Gemini native models (`generateContent`) take `GenerateContentConfig` fields instead — `seed`, `safetySettings`, `thinkingConfig`, `imageConfig`, and `systemInstruction`: + +```typescript +import { generateImage } from "@tanstack/ai"; +import { geminiImage } from "@tanstack/ai-gemini"; + +const result = await generateImage({ + adapter: geminiImage("gemini-3.1-flash-image-preview"), + prompt: "...", + size: "16:9_4K", + modelOptions: { + thinkingConfig: { thinkingBudget: 512 }, + // Merged over the imageConfig derived from `size`, per field. + imageConfig: { imageSize: "2K" }, + }, +}); +``` + +See [Image Generation](../media/image-generation) for the full native option list. + ## Text-to-Speech (Experimental) > **Note:** Gemini TTS is experimental and may require the Live API for full functionality. diff --git a/docs/config.json b/docs/config.json index e5c98bc877..67f5fd0085 100644 --- a/docs/config.json +++ b/docs/config.json @@ -437,7 +437,7 @@ "label": "Image Generation", "to": "media/image-generation", "addedAt": "2026-04-15", - "updatedAt": "2026-08-13" + "updatedAt": "2026-08-14" }, { "label": "Video Generation", @@ -796,7 +796,7 @@ "label": "Google Gemini", "to": "adapters/gemini", "addedAt": "2026-04-15", - "updatedAt": "2026-07-22" + "updatedAt": "2026-08-14" }, { "label": "Ollama", diff --git a/docs/media/image-generation.md b/docs/media/image-generation.md index c1312e8122..24c8ab8734 100644 --- a/docs/media/image-generation.md +++ b/docs/media/image-generation.md @@ -634,7 +634,7 @@ const result = await generateImage({ #### Gemini Native Model Options (NanoBanana) -Gemini native image models accept `GenerateContentConfig` options directly in `modelOptions`: +Gemini native image models are served by `generateContent`, so their `modelOptions` are `GenerateContentConfig` fields — a different shape from the Imagen options above: ```typescript import { generateImage } from "@tanstack/ai"; @@ -644,9 +644,43 @@ const result = await generateImage({ adapter: geminiImage("gemini-3.1-flash-image-preview"), prompt: "A beautiful garden", size: "16:9_4K", + modelOptions: { + seed: 42, + thinkingConfig: { thinkingBudget: 512 }, + systemInstruction: "Always render in watercolor.", + // Merged over the imageConfig derived from `size`, per field: this keeps + // the 16:9 aspect ratio and overrides only the resolution tier. + imageConfig: { imageSize: "2K" }, + }, +}); +``` + +`safetySettings` takes the SDK's `HarmCategory` / `HarmBlockThreshold` enums, so plain strings won't type-check. Both are re-exported from `@tanstack/ai-gemini` — you don't need `@google/genai` in your own dependencies: + +```typescript +import { generateImage } from "@tanstack/ai"; +import { + HarmBlockThreshold, + HarmCategory, + geminiImage, +} from "@tanstack/ai-gemini"; + +const result = await generateImage({ + adapter: geminiImage("gemini-3.1-flash-image-preview"), + prompt: "A beautiful garden", + modelOptions: { + safetySettings: [ + { + category: HarmCategory.HARM_CATEGORY_HATE_SPEECH, + threshold: HarmBlockThreshold.BLOCK_ONLY_HIGH, + }, + ], + }, }); ``` +`responseModalities` is not accepted — the adapter always requests `['TEXT', 'IMAGE']`, so nothing can silently disable image output. + ### Response Format The image generation result includes: diff --git a/packages/ai-gemini/src/adapters/image.ts b/packages/ai-gemini/src/adapters/image.ts index 85284d282e..abbfa38cec 100644 --- a/packages/ai-gemini/src/adapters/image.ts +++ b/packages/ai-gemini/src/adapters/image.ts @@ -7,6 +7,7 @@ import { } from '../utils' import { buildGeminiUsage } from '../usage' import { + isGeminiNativeImageModel, parseNativeImageSize, sizeToAspectRatio, validateImageSize, @@ -15,10 +16,11 @@ import { } from '../image/image-provider-options' import type { GEMINI_IMAGE_MODELS } from '../model-meta' import type { + GeminiAnyImageProviderOptions, GeminiImageModelInputModalitiesByName, GeminiImageModelProviderOptionsByName, GeminiImageModelSizeByName, - GeminiImageProviderOptions, + GeminiNativeImageProviderOptions, } from '../image/image-provider-options' import type { GeneratedImage, @@ -35,6 +37,7 @@ import type { GenerateImagesConfig, GenerateImagesResponse, GoogleGenAI, + ImageConfig, Part, } from '@google/genai' import type { GeminiClientConfig } from '../utils/client' @@ -65,7 +68,7 @@ export class GeminiImageAdapter< TModel extends GeminiImageModel, > extends BaseImageAdapter< TModel, - GeminiImageProviderOptions, + GeminiAnyImageProviderOptions, GeminiImageModelProviderOptionsByName, GeminiImageModelSizeByName, GeminiImageModelInputModalitiesByName @@ -75,7 +78,7 @@ export class GeminiImageAdapter< // Type-only property - never assigned at runtime declare '~types': { - providerOptions: GeminiImageProviderOptions + providerOptions: GeminiAnyImageProviderOptions modelProviderOptionsByName: GeminiImageModelProviderOptionsByName modelSizeByName: GeminiImageModelSizeByName modelInputModalitiesByName: GeminiImageModelInputModalitiesByName @@ -89,7 +92,7 @@ export class GeminiImageAdapter< } async generateImages( - options: ImageGenerationOptions, + options: ImageGenerationOptions, ): Promise { const { model, logger } = options @@ -121,7 +124,7 @@ export class GeminiImageAdapter< ) } - if (this.isGeminiImageModel(model)) { + if (isGeminiNativeImageModel(model)) { return await this.generateWithGeminiApi(options, resolved) } @@ -155,29 +158,40 @@ export class GeminiImageAdapter< } } - private isGeminiImageModel(model: string): boolean { - return model.startsWith('gemini-') - } - private async generateWithGeminiApi( - options: ImageGenerationOptions, + options: ImageGenerationOptions, resolved: ResolvedMediaPrompt, ): Promise { const { model, size, numberOfImages, modelOptions } = options const parsedSize = size ? parseNativeImageSize(size) : undefined - // GeminiImageProviderOptions is Imagen-shaped — most fields - // (personGeneration, safetyFilterLevel, addWatermark, outputMimeType, - // outputCompressionQuality, guidanceScale, enhancePrompt, - // includeSafetyAttributes, includeRaiReason, outputGcsUri, labels, - // negativePrompt, language) are only valid on GenerateImagesConfig and - // would be rejected by the Gemini-native generateContent path. Pick only - // the fields that are valid on GenerateContentConfig instead of spreading - // the whole options object. - const nativeConfig: GenerateContentConfig = {} - if (modelOptions?.seed !== undefined) { - nativeConfig.seed = modelOptions.seed + // The portable `size` option is the baseline; modelOptions.imageConfig is + // the provider escape hatch and wins per field, so a caller passing only + // `imageConfig.imageSize` keeps the aspectRatio derived from `size`. + const imageConfig: ImageConfig = { + ...(parsedSize?.aspectRatio && { aspectRatio: parsedSize.aspectRatio }), + ...(parsedSize?.resolution && { imageSize: parsedSize.resolution }), + ...modelOptions?.imageConfig, + } + + // Named picks, never a wholesale spread: the Imagen-shaped fields of + // GeminiImageProviderOptions (personGeneration, safetyFilterLevel, + // addWatermark, outputMimeType, …) are only valid on GenerateImagesConfig + // and would be rejected by generateContent. Picking by name means no + // Imagen field can reach this path even if one slips past the per-model + // provider-options map. + const nativeConfig: GenerateContentConfig = { + ...(modelOptions?.seed !== undefined && { seed: modelOptions.seed }), + ...(modelOptions?.safetySettings !== undefined && { + safetySettings: modelOptions.safetySettings, + }), + ...(modelOptions?.thinkingConfig !== undefined && { + thinkingConfig: modelOptions.thinkingConfig, + }), + ...(modelOptions?.systemInstruction !== undefined && { + systemInstruction: modelOptions.systemInstruction, + }), } const config: GenerateContentConfig = { @@ -186,16 +200,7 @@ export class GeminiImageAdapter< // IMPORTANT: responseModalities is a protected default — set it AFTER // nativeConfig so nothing can silently disable image output. responseModalities: ['TEXT', 'IMAGE'], - ...(parsedSize && { - imageConfig: { - ...(parsedSize.aspectRatio && { - aspectRatio: parsedSize.aspectRatio, - }), - ...(parsedSize.resolution && { - imageSize: parsedSize.resolution, - }), - }, - }), + ...(Object.keys(imageConfig).length > 0 && { imageConfig }), } const contents = this.buildContents(resolved, numberOfImages) @@ -321,7 +326,7 @@ export class GeminiImageAdapter< } private buildImagenConfig( - options: ImageGenerationOptions, + options: ImageGenerationOptions, ): GenerateImagesConfig { const { size, numberOfImages, modelOptions } = options @@ -329,11 +334,62 @@ export class GeminiImageAdapter< // vendor `GenerateImagesConfig` fields are `field?: T` (no `| undefined`), // so we can only assign the property when we actually have a value. const sizeAspectRatio = size ? sizeToAspectRatio(size) : undefined + + // Named picks, never a wholesale spread — the mirror image of the native + // path below. A native-only field (safetySettings, thinkingConfig, + // imageConfig, systemInstruction) belongs to GenerateContentConfig and is + // rejected by generateImages with 400 INVALID_ARGUMENT, so it must not be + // able to reach here even when the caller's `modelOptions` was typed + // against both shapes at once (e.g. an adapter inferred from a union of + // model names). return { numberOfImages: numberOfImages ?? 1, - // Map size to aspect ratio if provided (modelOptions.aspectRatio will override) + // Map size to aspect ratio if provided; modelOptions.aspectRatio, + // picked after it, overrides. ...(sizeAspectRatio !== undefined && { aspectRatio: sizeAspectRatio }), - ...modelOptions, + ...(modelOptions?.aspectRatio !== undefined && { + aspectRatio: modelOptions.aspectRatio, + }), + ...(modelOptions?.personGeneration !== undefined && { + personGeneration: modelOptions.personGeneration, + }), + ...(modelOptions?.safetyFilterLevel !== undefined && { + safetyFilterLevel: modelOptions.safetyFilterLevel, + }), + ...(modelOptions?.seed !== undefined && { seed: modelOptions.seed }), + ...(modelOptions?.addWatermark !== undefined && { + addWatermark: modelOptions.addWatermark, + }), + ...(modelOptions?.language !== undefined && { + language: modelOptions.language, + }), + ...(modelOptions?.negativePrompt !== undefined && { + negativePrompt: modelOptions.negativePrompt, + }), + ...(modelOptions?.outputMimeType !== undefined && { + outputMimeType: modelOptions.outputMimeType, + }), + ...(modelOptions?.outputCompressionQuality !== undefined && { + outputCompressionQuality: modelOptions.outputCompressionQuality, + }), + ...(modelOptions?.guidanceScale !== undefined && { + guidanceScale: modelOptions.guidanceScale, + }), + ...(modelOptions?.enhancePrompt !== undefined && { + enhancePrompt: modelOptions.enhancePrompt, + }), + ...(modelOptions?.includeSafetyAttributes !== undefined && { + includeSafetyAttributes: modelOptions.includeSafetyAttributes, + }), + ...(modelOptions?.includeRaiReason !== undefined && { + includeRaiReason: modelOptions.includeRaiReason, + }), + ...(modelOptions?.outputGcsUri !== undefined && { + outputGcsUri: modelOptions.outputGcsUri, + }), + ...(modelOptions?.labels !== undefined && { + labels: modelOptions.labels, + }), } } diff --git a/packages/ai-gemini/src/image/image-provider-options.ts b/packages/ai-gemini/src/image/image-provider-options.ts index 84fd66da76..364fad9f56 100644 --- a/packages/ai-gemini/src/image/image-provider-options.ts +++ b/packages/ai-gemini/src/image/image-provider-options.ts @@ -1,12 +1,24 @@ import type { GeminiImageModels } from '../model-meta' import type { + ContentUnion, + ImageConfig, ImagePromptLanguage, PersonGeneration, SafetyFilterLevel, + SafetySetting, + ThinkingConfig, } from '@google/genai' // Re-export SDK types so users can use them directly -export type { ImagePromptLanguage, PersonGeneration, SafetyFilterLevel } +export type { + ContentUnion, + ImageConfig, + ImagePromptLanguage, + PersonGeneration, + SafetyFilterLevel, + SafetySetting, + ThinkingConfig, +} /** * Gemini Imagen aspect ratio options @@ -121,11 +133,77 @@ export interface GeminiImageProviderOptions { } /** - * Model-specific provider options mapping - * Currently all Imagen models use the same options structure + * Provider options for Gemini native image models (Nano Banana and friends). + * + * These models are served by `generateContent`, not `generateImages`, so they + * are configured by @google/genai's `GenerateContentConfig` — a different + * shape from the Imagen-only {@link GeminiImageProviderOptions} above. Only + * the `GenerateContentConfig` fields with clear image-generation semantics are + * surfaced; sampling knobs (`temperature`, `topK`, …) and chat-only plumbing + * (`tools`, `responseSchema`, …) are deliberately left out. + * + * `responseModalities` is intentionally absent: the adapter always requests + * `['TEXT', 'IMAGE']`, and letting a caller override it would silently disable + * image output on an image-generation call. + */ +export interface GeminiNativeImageProviderOptions { + /** + * Optional seed for reproducible image generation + * When the same seed is used with the same prompt and settings, + * you should get similar (though not identical) results + */ + seed?: number + + /** + * Per-category safety thresholds applied to the request + * Each entry pairs a HarmCategory with a HarmBlockThreshold + */ + safetySettings?: Array + + /** + * Controls the model's internal reasoning before it emits an image + * Use to raise or disable the thinking budget on models that support it + */ + thinkingConfig?: ThinkingConfig + + /** + * Native image output controls (aspect ratio, resolution tier, …) + * Merged over the values derived from the portable `size` option, so + * fields set here win per field while the rest of `size` is preserved + */ + imageConfig?: ImageConfig + + /** + * System-level instructions that steer the model for the whole request, + * e.g. a house art direction applied on top of the per-call prompt + */ + systemInstruction?: ContentUnion +} + +/** + * Every provider-option field this adapter understands, across both API + * paths. Used as the adapter's base (model-agnostic) option type; the + * per-model map below is what narrows a given model to the half that + * actually applies to it. + */ +export type GeminiAnyImageProviderOptions = GeminiImageProviderOptions & + GeminiNativeImageProviderOptions + +/** + * Model-specific provider options mapping. + * Gemini native image models go through `generateContent` and take + * `GenerateContentConfig` fields; Imagen models go through `generateImages` + * and take `GenerateImagesConfig` fields. Mirrors the native/Imagen split in + * {@link GeminiImageModelSizeByName} and + * {@link GeminiImageModelInputModalitiesByName}. */ export type GeminiImageModelProviderOptionsByName = { - [K in GeminiImageModels]: GeminiImageProviderOptions + [K in GeminiNativeImageModels]: GeminiNativeImageProviderOptions +} & { + [K in Exclude< + GeminiImageModels, + GeminiNativeImageModels + >]: GeminiImageProviderOptions } /** @@ -173,12 +251,47 @@ export type GeminiNativeImageSize = /** * Gemini native image models that use the generateContent API path. * These models support template literal sizes (aspectRatio_resolution). + * + * This array is the single source of truth for the native/Imagen split: the + * `GeminiNativeImageModels` union and the per-model option/size/modality maps + * all derive from it. The `satisfies` clause makes a typo (or a name that + * isn't a known image model) a build error rather than a phantom key on every + * per-model map. + * + * It is also the single source of truth for the adapter's *runtime* routing + * — see {@link isGeminiNativeImageModel}. Adding a new `gemini-*` image model + * means adding it here as well as to `GEMINI_IMAGE_MODELS` in model-meta; + * until it is listed here it routes to the Imagen API instead and fails + * loudly on the first call, rather than silently taking the wrong option + * shape. */ +export const GEMINI_NATIVE_IMAGE_MODELS = [ + 'gemini-3.1-flash-image-preview', + 'gemini-3.1-flash-lite-image', + 'gemini-3-pro-image-preview', + 'gemini-2.5-flash-image', +] as const satisfies ReadonlyArray + export type GeminiNativeImageModels = - | 'gemini-3.1-flash-image-preview' - | 'gemini-3.1-flash-lite-image' - | 'gemini-3-pro-image-preview' - | 'gemini-2.5-flash-image' + (typeof GEMINI_NATIVE_IMAGE_MODELS)[number] + +const NATIVE_IMAGE_MODEL_NAMES: ReadonlySet = new Set( + GEMINI_NATIVE_IMAGE_MODELS, +) + +/** + * Runtime counterpart to {@link GeminiNativeImageModels} — decides which of + * the two Gemini image APIs a model goes to. + * + * Membership in {@link GEMINI_NATIVE_IMAGE_MODELS}, not a `gemini-` prefix + * test, so the runtime route and the type-level split cannot drift apart. An + * id this package does not know about reaches the Imagen endpoint and fails + * there, which is the intended signal to add the model here rather than to + * have it silently take the native path with Imagen-shaped option types. + */ +export function isGeminiNativeImageModel(model: string): boolean { + return NATIVE_IMAGE_MODEL_NAMES.has(model) +} /** * Model-specific size options mapping. diff --git a/packages/ai-gemini/src/index.ts b/packages/ai-gemini/src/index.ts index e0c58922c0..9397ee5ae4 100644 --- a/packages/ai-gemini/src/index.ts +++ b/packages/ai-gemini/src/index.ts @@ -28,13 +28,24 @@ export { } from './adapters/image' export type { GeminiImageProviderOptions, + GeminiNativeImageProviderOptions, + GeminiAnyImageProviderOptions, GeminiImageModelProviderOptionsByName, GeminiAspectRatio, // Re-export SDK types for convenience PersonGeneration, SafetyFilterLevel, ImagePromptLanguage, + SafetySetting, + ThinkingConfig, + ImageConfig, + ContentUnion, } from './image/image-provider-options' +// `SafetySetting` is built from two SDK enums, and enums are values — they +// cannot travel through `export type`. Re-exported here so `safetySettings` +// is usable with only `@tanstack/ai-gemini` installed, without the consumer +// having to add `@google/genai` to their own dependencies. +export { HarmBlockThreshold, HarmCategory } from '@google/genai' // Embedding adapter - for embedding vectors export { diff --git a/packages/ai-gemini/tests/image-adapter.test.ts b/packages/ai-gemini/tests/image-adapter.test.ts index 7f07bb04f0..ac63d324f9 100644 --- a/packages/ai-gemini/tests/image-adapter.test.ts +++ b/packages/ai-gemini/tests/image-adapter.test.ts @@ -1,4 +1,11 @@ import { describe, it, expect, vi } from 'vitest' +import { + HarmBlockThreshold, + HarmCategory, + ImagePromptLanguage, + PersonGeneration, + SafetyFilterLevel, +} from '@google/genai' import { generateImage } from '@tanstack/ai' import { resolveDebugOption } from '@tanstack/ai/adapter-internals' import { GeminiImageAdapter, createGeminiImage } from '../src/adapters/image' @@ -10,6 +17,55 @@ import { validatePrompt, } from '../src/image/image-provider-options' +const mockImageResponse = { + candidates: [ + { + content: { + parts: [{ inlineData: { mimeType: 'image/png', data: 'out' } }], + }, + }, + ], +} + +/** + * A native-path adapter whose `client.models.generateContent` is stubbed, so + * tests can assert the exact config object handed to the SDK. + */ +function mockedNativeAdapter() { + const mockGenerateContent = vi.fn().mockResolvedValueOnce(mockImageResponse) + const adapter = createGeminiImage( + 'gemini-3.1-flash-image-preview', + 'test-api-key', + ) + ;( + adapter as unknown as { + client: { models: { generateContent: unknown } } + } + ).client = { + models: { generateContent: mockGenerateContent }, + } + return { adapter, mockGenerateContent } +} + +/** + * An Imagen-path adapter whose `client.models.generateImages` is stubbed, so + * tests can assert the exact GenerateImagesConfig handed to the SDK. + */ +function mockedImagenAdapter() { + const mockGenerateImages = vi.fn().mockResolvedValueOnce({ + generatedImages: [{ image: { imageBytes: 'imagen-b64' } }], + }) + const adapter = createGeminiImage('imagen-4.0-generate-001', 'test-api-key') + ;( + adapter as unknown as { + client: { models: { generateImages: unknown } } + } + ).client = { + models: { generateImages: mockGenerateImages }, + } + return { adapter, mockGenerateImages } +} + describe('Gemini Image Adapter', () => { describe('createGeminiImage', () => { it('creates an adapter with the provided API key', () => { @@ -721,35 +777,199 @@ describe('Gemini Image Adapter', () => { }) }) - describe('multimodal prompt (image-conditioned generation)', () => { - const testLogger = resolveDebugOption(false) - const mockImageResponse = { - candidates: [ - { - content: { - parts: [{ inlineData: { mimeType: 'image/png', data: 'out' } }], - }, + describe('native modelOptions (GenerateContentConfig)', () => { + // Regression: GeminiImageModelProviderOptionsByName used to map every + // image model — native ones included — to the Imagen-shaped + // GeminiImageProviderOptions, so these fields were a compile error and the + // adapter whitelisted only `seed`. + const safetySettings = [ + { + category: HarmCategory.HARM_CATEGORY_HATE_SPEECH, + threshold: HarmBlockThreshold.BLOCK_ONLY_HIGH, + }, + ] + + it('forwards safetySettings to generateContent', async () => { + const { adapter, mockGenerateContent } = mockedNativeAdapter() + + await generateImage({ + adapter, + prompt: 'A quiet harbour', + modelOptions: { safetySettings }, + }) + + const args = mockGenerateContent.mock.calls[0]![0] + expect(args.config.safetySettings).toEqual(safetySettings) + }) + + it('forwards thinkingConfig to generateContent', async () => { + const { adapter, mockGenerateContent } = mockedNativeAdapter() + + await generateImage({ + adapter, + prompt: 'A quiet harbour', + modelOptions: { thinkingConfig: { thinkingBudget: 512 } }, + }) + + const args = mockGenerateContent.mock.calls[0]![0] + expect(args.config.thinkingConfig).toEqual({ thinkingBudget: 512 }) + }) + + it('forwards systemInstruction to generateContent', async () => { + const { adapter, mockGenerateContent } = mockedNativeAdapter() + + await generateImage({ + adapter, + prompt: 'A quiet harbour', + modelOptions: { systemInstruction: 'Always render in watercolor.' }, + }) + + const args = mockGenerateContent.mock.calls[0]![0] + expect(args.config.systemInstruction).toBe('Always render in watercolor.') + }) + + it('merges modelOptions.imageConfig over the size-derived imageConfig', async () => { + // `size` is the portable API, `imageConfig` the provider escape hatch: + // the overriding field wins, the untouched one survives. + const { adapter, mockGenerateContent } = mockedNativeAdapter() + + await generateImage({ + adapter, + prompt: 'A quiet harbour', + size: '16:9_4K', + modelOptions: { imageConfig: { imageSize: '2K' } }, + }) + + const args = mockGenerateContent.mock.calls[0]![0] + expect(args.config.imageConfig).toEqual({ + aspectRatio: '16:9', + imageSize: '2K', + }) + }) + + it('applies modelOptions.imageConfig when no size is given', async () => { + const { adapter, mockGenerateContent } = mockedNativeAdapter() + + await generateImage({ + adapter, + prompt: 'A quiet harbour', + modelOptions: { imageConfig: { aspectRatio: '21:9' } }, + }) + + const args = mockGenerateContent.mock.calls[0]![0] + expect(args.config.imageConfig).toEqual({ aspectRatio: '21:9' }) + }) + + it('keeps Imagen models on GenerateImagesConfig with no native fields', async () => { + const { adapter, mockGenerateImages } = mockedImagenAdapter() + + await generateImage({ + adapter, + prompt: 'A quiet harbour', + size: '1920x1080', + modelOptions: { + personGeneration: PersonGeneration.ALLOW_ADULT, + negativePrompt: 'blurry', + // Native-only fields. The per-model type rejects them (hence the + // cast, as in the responseModalities regression test above), but an + // adapter inferred from a union of model names widens modelOptions + // to both shapes, so they can still arrive here at runtime — and + // generateImages answers a GenerateContentConfig field with + // 400 INVALID_ARGUMENT. The named picks must drop them. + safetySettings: [ + { + category: HarmCategory.HARM_CATEGORY_HATE_SPEECH, + threshold: HarmBlockThreshold.BLOCK_ONLY_HIGH, + }, + ], + thinkingConfig: { thinkingBudget: 512 }, + imageConfig: { imageSize: '2K' }, + systemInstruction: 'Always render in watercolor.', + } as unknown as never, + }) + + // Exact match: the Imagen path gets its own GenerateImagesConfig fields + // and nothing else. + expect(mockGenerateImages).toHaveBeenCalledWith({ + model: 'imagen-4.0-generate-001', + prompt: 'A quiet harbour', + config: { + numberOfImages: 1, + aspectRatio: '16:9', + personGeneration: PersonGeneration.ALLOW_ADULT, + negativePrompt: 'blurry', }, - ], - } + }) + }) - function mockedNativeAdapter() { - const mockGenerateContent = vi - .fn() - .mockResolvedValueOnce(mockImageResponse) - const adapter = createGeminiImage( - 'gemini-3.1-flash-image-preview', - 'test-api-key', - ) - ;( - adapter as unknown as { - client: { models: { generateContent: unknown } } - } - ).client = { - models: { generateContent: mockGenerateContent }, - } - return { adapter, mockGenerateContent } - } + it('forwards the whole Imagen option set to generateImages', async () => { + // The Imagen path picks fields by name, so this guards against a field + // being forgotten in that list. + const { adapter, mockGenerateImages } = mockedImagenAdapter() + + await generateImage({ + adapter, + prompt: 'A quiet harbour', + modelOptions: { + aspectRatio: '21:9', + personGeneration: PersonGeneration.ALLOW_ADULT, + safetyFilterLevel: SafetyFilterLevel.BLOCK_ONLY_HIGH, + seed: 42, + addWatermark: false, + language: ImagePromptLanguage.en, + negativePrompt: 'blurry', + outputMimeType: 'image/jpeg', + outputCompressionQuality: 80, + guidanceScale: 12, + enhancePrompt: true, + includeSafetyAttributes: true, + includeRaiReason: true, + outputGcsUri: 'gs://bucket/out', + labels: { team: 'design' }, + }, + }) + + expect(mockGenerateImages).toHaveBeenCalledWith({ + model: 'imagen-4.0-generate-001', + prompt: 'A quiet harbour', + config: { + numberOfImages: 1, + aspectRatio: '21:9', + personGeneration: PersonGeneration.ALLOW_ADULT, + safetyFilterLevel: SafetyFilterLevel.BLOCK_ONLY_HIGH, + seed: 42, + addWatermark: false, + language: ImagePromptLanguage.en, + negativePrompt: 'blurry', + outputMimeType: 'image/jpeg', + outputCompressionQuality: 80, + guidanceScale: 12, + enhancePrompt: true, + includeSafetyAttributes: true, + includeRaiReason: true, + outputGcsUri: 'gs://bucket/out', + labels: { team: 'design' }, + }, + }) + }) + + it('lets modelOptions.aspectRatio override the size-derived one', async () => { + const { adapter, mockGenerateImages } = mockedImagenAdapter() + + await generateImage({ + adapter, + prompt: 'A quiet harbour', + size: '1920x1080', + modelOptions: { aspectRatio: '9:16' }, + }) + + const args = mockGenerateImages.mock.calls[0]![0] + expect(args.config.aspectRatio).toBe('9:16') + }) + }) + + describe('multimodal prompt (image-conditioned generation)', () => { + const testLogger = resolveDebugOption(false) it('maps interleaved prompt parts onto multimodal contents in order', async () => { const { adapter, mockGenerateContent } = mockedNativeAdapter() diff --git a/packages/ai-gemini/tests/image-per-model-type-safety.test.ts b/packages/ai-gemini/tests/image-per-model-type-safety.test.ts new file mode 100644 index 0000000000..88e42d1299 --- /dev/null +++ b/packages/ai-gemini/tests/image-per-model-type-safety.test.ts @@ -0,0 +1,159 @@ +/** + * Per-model type-safety tests for Gemini generateImage() modelOptions. + * + * Gemini-native image models (generateContent) and Imagen models + * (generateImages) take different provider-option shapes, so + * `GeminiImageModelProviderOptionsByName` splits on the model family the same + * way the size and input-modality maps do. Positive cases compile cleanly; + * cross-family cases produce a `@ts-expect-error`. + * + * Compile-time only — `createImageOptions` builds the typed options object + * without issuing a request. + */ +import { beforeAll, describe, expectTypeOf, it } from 'vitest' +import { createImageOptions } from '@tanstack/ai' +import { geminiImage } from '../src' +import type { GeminiImageModelProviderOptionsByName } from '../src' + +// Set a dummy API key so adapter construction does not throw at runtime. +// These tests only exercise compile-time type gating; no network calls are made. +beforeAll(() => { + process.env['GOOGLE_API_KEY'] = 'sk-test-dummy' +}) + +describe('Gemini per-model image modelOptions gating', () => { + describe('gemini-3.1-flash-image-preview — native (GenerateContentConfig)', () => { + it('accepts the native option set', () => { + createImageOptions({ + adapter: geminiImage('gemini-3.1-flash-image-preview'), + prompt: 'a quiet harbour', + modelOptions: { + seed: 7, + safetySettings: [], + thinkingConfig: { thinkingBudget: 512 }, + imageConfig: { aspectRatio: '16:9', imageSize: '2K' }, + systemInstruction: 'Always render in watercolor.', + }, + }) + }) + + it('rejects Imagen-only options', () => { + // The probes are plain values that are structurally valid on + // GeminiImageProviderOptions (`negativePrompt: string`, + // `aspectRatio: GeminiAspectRatio`), so the errors below come from the + // native/Imagen split and nothing else — an enum-typed field such as + // `personGeneration` would reject a string literal on either shape and + // would still "pass" with the split reverted. + createImageOptions({ + adapter: geminiImage('gemini-3.1-flash-image-preview'), + prompt: 'a quiet harbour', + modelOptions: { + // @ts-expect-error - negativePrompt is a GenerateImagesConfig (Imagen) field + negativePrompt: 'blurry', + }, + }) + + createImageOptions({ + adapter: geminiImage('gemini-3.1-flash-image-preview'), + prompt: 'a quiet harbour', + modelOptions: { + // @ts-expect-error - aspectRatio is Imagen-only; native models use size / imageConfig + aspectRatio: '16:9', + }, + }) + }) + + it('rejects responseModalities — the adapter owns it', () => { + createImageOptions({ + adapter: geminiImage('gemini-3.1-flash-image-preview'), + prompt: 'a quiet harbour', + modelOptions: { + // @ts-expect-error - responseModalities is a protected adapter default + responseModalities: ['TEXT'], + }, + }) + }) + }) + + describe('imagen-4.0-generate-001 — Imagen (GenerateImagesConfig)', () => { + it('accepts the Imagen option set', () => { + createImageOptions({ + adapter: geminiImage('imagen-4.0-generate-001'), + prompt: 'a quiet harbour', + modelOptions: { + aspectRatio: '16:9', + negativePrompt: 'blurry', + addWatermark: true, + outputMimeType: 'image/png', + }, + }) + }) + + it('rejects native-only options', () => { + createImageOptions({ + adapter: geminiImage('imagen-4.0-generate-001'), + prompt: 'a quiet harbour', + modelOptions: { + // @ts-expect-error - safetySettings is a GenerateContentConfig (native) field + safetySettings: [], + }, + }) + }) + }) +}) + +describe('Gemini image provider options shape assertions', () => { + describe('native models take GenerateContentConfig fields', () => { + type Options = + GeminiImageModelProviderOptionsByName['gemini-3.1-flash-image-preview'] + + it('has safetySettings', () => { + expectTypeOf().toHaveProperty('safetySettings') + }) + it('has thinkingConfig', () => { + expectTypeOf().toHaveProperty('thinkingConfig') + }) + it('has imageConfig', () => { + expectTypeOf().toHaveProperty('imageConfig') + }) + it('has systemInstruction', () => { + expectTypeOf().toHaveProperty('systemInstruction') + }) + it('has seed', () => { + expectTypeOf().toHaveProperty('seed') + }) + }) + + describe('Imagen models keep GenerateImagesConfig fields', () => { + type Options = + GeminiImageModelProviderOptionsByName['imagen-4.0-generate-001'] + + it('has aspectRatio', () => { + expectTypeOf().toHaveProperty('aspectRatio') + }) + it('has personGeneration', () => { + expectTypeOf().toHaveProperty('personGeneration') + }) + it('has negativePrompt', () => { + expectTypeOf().toHaveProperty('negativePrompt') + }) + }) + + describe('every native model id resolves to the native shape', () => { + it('gemini-3.1-flash-lite-image', () => { + expectTypeOf< + GeminiImageModelProviderOptionsByName['gemini-3.1-flash-lite-image'] + >().toHaveProperty('imageConfig') + }) + it('gemini-3-pro-image-preview', () => { + expectTypeOf< + GeminiImageModelProviderOptionsByName['gemini-3-pro-image-preview'] + >().toHaveProperty('imageConfig') + }) + it('gemini-2.5-flash-image', () => { + expectTypeOf< + GeminiImageModelProviderOptionsByName['gemini-2.5-flash-image'] + >().toHaveProperty('imageConfig') + }) + }) +}) diff --git a/testing/e2e/global-setup.ts b/testing/e2e/global-setup.ts index 0502e4418c..25d8f419cb 100644 --- a/testing/e2e/global-setup.ts +++ b/testing/e2e/global-setup.ts @@ -55,6 +55,13 @@ export default async function globalSetup() { '/v1beta/models/gemini-3.1-flash-tts-preview:generateContent', geminiTTSMount(), ) + // Gemini native image generation hits the same generateContent endpoint + // shape, one model id over — see geminiNativeImageMount for why it needs + // a hand-mocked response and a raw-body wire-shape check of its own. + mock.mount( + '/v1beta/models/gemini-2.5-flash-image:generateContent', + geminiNativeImageMount(), + ) // Gemini Veo video generation. aimock 1.29 mocks Gemini's `:predict` // (Imagen) endpoint but not the long-running `:predictLongRunning` + // operations-polling pair Veo uses, so mount both here. Non-Veo paths @@ -265,6 +272,183 @@ function geminiTTSMount(): Mountable { } } +/** + * Gemini native image generation hits the standard Gemini generateContent + * endpoint too (POST /v1beta/models/{model}:generateContent) — the same + * shape geminiTTSMount above targets, just with `image/png` inlineData + * instead of PCM audio. aimock's native handleGemini recognizes only + * text / tool-call / text-with-tool-call / audio fixture response shapes: + * isImageResponse (helpers.js) is defined but never imported into + * gemini.js, so there is no image-response branch at all. A fixture shaped + * `{image}`/`{images}` matched against this endpoint falls through every + * isXResponse() check and hits the final fallback — a 500 "Fixture response + * did not match any known type." Native image generation needs a + * hand-mocked response for the same reason TTS does. + * + * There's a second, PR-specific reason this needs its own mount rather than + * a fixture even if one could match: aimock's request-journaling for this + * endpoint goes through geminiToCompletionRequest, which reshapes the raw + * Gemini request into an OpenAI-chat-shaped completionReq carrying only + * {model, messages, stream, temperature, max_tokens, top_p, top_k, tools} — + * it silently drops safetySettings, generationConfig.thinkingConfig, and + * generationConfig.imageConfig before anything is journaled. So GET /journal + * cannot show whether those fields reached the wire, even in principle. + * This mount reads the raw, untranslated body instead (see + * readJsonRequestBody, used the same way by the BytePlus mounts below) and + * validates it directly — the BytePlus mounts' house pattern of turning a + * dropped field into a failing spec rather than a silently green one. + * + * Mounted at the exact model+endpoint path (not the shared '/v1beta/models' + * prefix geminiVeoMount/geminiBatchEmbedMount use below) so it can never + * intercept an unrelated Gemini chat/text generateContent call for a + * different model. + * + * `/api/gemini-native-image-wire` (see that route) drives this with + * `modelOptions: { safetySettings, thinkingConfig }`. Reverting the + * ai-gemini fix that stops dropping modelOptions on the native image path + * removes both from the outgoing `nativeConfig`, so the request this mount + * receives is missing them, this mount answers 400, and the route surfaces + * that as `ok: false` — the companion spec's revert-detection mechanism. + */ +function geminiNativeImageMount(): Mountable { + // 1x1 transparent PNG — just enough for transformGeminiResponse's + // inlineData branch to produce a GeneratedImage. Mirrors FAKE_PCM_BYTES / + // FAKE_MP3_BYTES above: content fidelity isn't under test here. + const PNG_1X1_BASE64 = + 'iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mNk+A8AAQUBAScY42YAAAAASUVORK5CYII=' + + // Field names that only belong on Imagen's GenerateImagesConfig, never on + // generateContent's GenerateContentConfig. The adapter's `nativeConfig` + // picks fields by name specifically so modelOptions carrying both shapes + // (or a future regression back to a wholesale `...modelOptions` spread) + // can't let one through — mirrors the field list documented on + // GeminiImageProviderOptions in image-provider-options.ts, minus `seed` + // (legitimately shared by both configs and always forwarded) and `labels` + // (also a real GenerateContentConfig field name, but the Gemini Developer + // API rejects it outright rather than letting it reach the wire, so it + // can't appear in a body this mount ever sees). + const IMAGEN_ONLY_FIELDS = [ + 'personGeneration', + 'safetyFilterLevel', + 'addWatermark', + 'language', + 'negativePrompt', + 'outputMimeType', + 'outputCompressionQuality', + 'guidanceScale', + 'enhancePrompt', + 'includeSafetyAttributes', + 'includeRaiReason', + 'outputGcsUri', + // Imagen's own top-level `aspectRatio` (GenerateImagesConfig) is distinct + // from the native path's nested generationConfig.imageConfig.aspectRatio + // — its presence at either level here would mean the two configs got + // crossed. + 'aspectRatio', + ] + + return { + async handleRequest( + req: http.IncomingMessage, + res: http.ServerResponse, + // Exact-path mount — pathname is "/" for the one path this is + // registered on. + pathname: string, + ): Promise { + if (pathname !== '/' || req.method !== 'POST') return false + + const body = await readJsonRequestBody(req) + if (!body) { + return rejectGeminiImageRequest(res, 'Malformed JSON body.') + } + const generationConfig = asRecord(body.generationConfig) + + const leaked = IMAGEN_ONLY_FIELDS.find( + (name) => + name in body || (generationConfig && name in generationConfig), + ) + if (leaked) { + return rejectGeminiImageRequest( + res, + `Imagen-only field "${leaked}" reached generateContent — GenerateImagesConfig and GenerateContentConfig got crossed.`, + ) + } + + if ( + !Array.isArray(body.safetySettings) || + body.safetySettings.length === 0 + ) { + return rejectGeminiImageRequest( + res, + 'Missing top-level safetySettings (modelOptions.safetySettings did not reach the wire).', + ) + } + if ( + !generationConfig || + typeof generationConfig.thinkingConfig !== 'object' || + generationConfig.thinkingConfig === null + ) { + return rejectGeminiImageRequest( + res, + 'Missing generationConfig.thinkingConfig (modelOptions.thinkingConfig did not reach the wire).', + ) + } + + res.statusCode = 200 + res.setHeader('Content-Type', 'application/json') + res.end( + JSON.stringify({ + candidates: [ + { + content: { + role: 'model', + parts: [ + { + inlineData: { + mimeType: 'image/png', + data: PNG_1X1_BASE64, + }, + }, + ], + }, + finishReason: 'STOP', + index: 0, + }, + ], + usageMetadata: { + promptTokenCount: 8, + candidatesTokenCount: 1290, + totalTokenCount: 1298, + }, + }), + ) + return true + }, + } +} + +/** + * Rejects with Gemini's real MLDev error envelope shape + * (`{ error: { code, message, status } }`) — the same fallback shape + * @google/genai's own `throwErrorIfNotOK` builds for a non-JSON error body, + * so the thrown ApiError's `message` carries the actual validation failure + * (JSON.stringify'd) for debugging, the same way rejectArkRequest / + * rejectVoiceRequest do for BytePlus below. + */ +function rejectGeminiImageRequest( + res: http.ServerResponse, + message: string, +): true { + res.statusCode = 400 + res.setHeader('Content-Type', 'application/json') + res.end( + JSON.stringify({ + error: { code: 400, message, status: 'INVALID_ARGUMENT' }, + }), + ) + return true +} + function grokSTTMount(): Mountable { return { async handleRequest( diff --git a/testing/e2e/src/routeTree.gen.ts b/testing/e2e/src/routeTree.gen.ts index c6d9853490..5ef35eee9e 100644 --- a/testing/e2e/src/routeTree.gen.ts +++ b/testing/e2e/src/routeTree.gen.ts @@ -59,6 +59,7 @@ import { Route as ApiInterruptsTestRouteImport } from './routes/api.interrupts-t import { Route as ApiImageRouteImport } from './routes/api.image' import { Route as ApiGenerationPersistenceServerRouteImport } from './routes/api.generation-persistence-server' import { Route as ApiGenerationPersistenceResumeRouteImport } from './routes/api.generation-persistence-resume' +import { Route as ApiGeminiNativeImageWireRouteImport } from './routes/api.gemini-native-image-wire' import { Route as ApiForeignInterruptRouteImport } from './routes/api.foreign-interrupt' import { Route as ApiEmbeddingRouteImport } from './routes/api.embedding' import { Route as ApiDurableTakeoverRouteImport } from './routes/api.durable-takeover' @@ -336,6 +337,12 @@ const ApiGenerationPersistenceResumeRoute = path: '/api/generation-persistence-resume', getParentRoute: () => rootRouteImport, } as any) +const ApiGeminiNativeImageWireRoute = + ApiGeminiNativeImageWireRouteImport.update({ + id: '/api/gemini-native-image-wire', + path: '/api/gemini-native-image-wire', + getParentRoute: () => rootRouteImport, + } as any) const ApiForeignInterruptRoute = ApiForeignInterruptRouteImport.update({ id: '/api/foreign-interrupt', path: '/api/foreign-interrupt', @@ -453,6 +460,7 @@ export interface FileRoutesByFullPath { '/api/durable-takeover': typeof ApiDurableTakeoverRoute '/api/embedding': typeof ApiEmbeddingRoute '/api/foreign-interrupt': typeof ApiForeignInterruptRoute + '/api/gemini-native-image-wire': typeof ApiGeminiNativeImageWireRoute '/api/generation-persistence-resume': typeof ApiGenerationPersistenceResumeRoute '/api/generation-persistence-server': typeof ApiGenerationPersistenceServerRoute '/api/image': typeof ApiImageRouteWithChildren @@ -522,6 +530,7 @@ export interface FileRoutesByTo { '/api/durable-takeover': typeof ApiDurableTakeoverRoute '/api/embedding': typeof ApiEmbeddingRoute '/api/foreign-interrupt': typeof ApiForeignInterruptRoute + '/api/gemini-native-image-wire': typeof ApiGeminiNativeImageWireRoute '/api/generation-persistence-resume': typeof ApiGenerationPersistenceResumeRoute '/api/generation-persistence-server': typeof ApiGenerationPersistenceServerRoute '/api/image': typeof ApiImageRouteWithChildren @@ -592,6 +601,7 @@ export interface FileRoutesById { '/api/durable-takeover': typeof ApiDurableTakeoverRoute '/api/embedding': typeof ApiEmbeddingRoute '/api/foreign-interrupt': typeof ApiForeignInterruptRoute + '/api/gemini-native-image-wire': typeof ApiGeminiNativeImageWireRoute '/api/generation-persistence-resume': typeof ApiGenerationPersistenceResumeRoute '/api/generation-persistence-server': typeof ApiGenerationPersistenceServerRoute '/api/image': typeof ApiImageRouteWithChildren @@ -663,6 +673,7 @@ export interface FileRouteTypes { | '/api/durable-takeover' | '/api/embedding' | '/api/foreign-interrupt' + | '/api/gemini-native-image-wire' | '/api/generation-persistence-resume' | '/api/generation-persistence-server' | '/api/image' @@ -732,6 +743,7 @@ export interface FileRouteTypes { | '/api/durable-takeover' | '/api/embedding' | '/api/foreign-interrupt' + | '/api/gemini-native-image-wire' | '/api/generation-persistence-resume' | '/api/generation-persistence-server' | '/api/image' @@ -801,6 +813,7 @@ export interface FileRouteTypes { | '/api/durable-takeover' | '/api/embedding' | '/api/foreign-interrupt' + | '/api/gemini-native-image-wire' | '/api/generation-persistence-resume' | '/api/generation-persistence-server' | '/api/image' @@ -871,6 +884,7 @@ export interface RootRouteChildren { ApiDurableTakeoverRoute: typeof ApiDurableTakeoverRoute ApiEmbeddingRoute: typeof ApiEmbeddingRoute ApiForeignInterruptRoute: typeof ApiForeignInterruptRoute + ApiGeminiNativeImageWireRoute: typeof ApiGeminiNativeImageWireRoute ApiGenerationPersistenceResumeRoute: typeof ApiGenerationPersistenceResumeRoute ApiGenerationPersistenceServerRoute: typeof ApiGenerationPersistenceServerRoute ApiImageRoute: typeof ApiImageRouteWithChildren @@ -1258,6 +1272,13 @@ declare module '@tanstack/react-router' { preLoaderRoute: typeof ApiGenerationPersistenceResumeRouteImport parentRoute: typeof rootRouteImport } + '/api/gemini-native-image-wire': { + id: '/api/gemini-native-image-wire' + path: '/api/gemini-native-image-wire' + fullPath: '/api/gemini-native-image-wire' + preLoaderRoute: typeof ApiGeminiNativeImageWireRouteImport + parentRoute: typeof rootRouteImport + } '/api/foreign-interrupt': { id: '/api/foreign-interrupt' path: '/api/foreign-interrupt' @@ -1468,6 +1489,7 @@ const rootRouteChildren: RootRouteChildren = { ApiDurableTakeoverRoute: ApiDurableTakeoverRoute, ApiEmbeddingRoute: ApiEmbeddingRoute, ApiForeignInterruptRoute: ApiForeignInterruptRoute, + ApiGeminiNativeImageWireRoute: ApiGeminiNativeImageWireRoute, ApiGenerationPersistenceResumeRoute: ApiGenerationPersistenceResumeRoute, ApiGenerationPersistenceServerRoute: ApiGenerationPersistenceServerRoute, ApiImageRoute: ApiImageRouteWithChildren, diff --git a/testing/e2e/src/routes/api.gemini-native-image-wire.ts b/testing/e2e/src/routes/api.gemini-native-image-wire.ts new file mode 100644 index 0000000000..94c15a246a --- /dev/null +++ b/testing/e2e/src/routes/api.gemini-native-image-wire.ts @@ -0,0 +1,71 @@ +import { createFileRoute } from '@tanstack/react-router' +import { generateImage } from '@tanstack/ai' +import { createImageAdapter } from '@/lib/media-providers' + +/** + * Wire-format verification for Gemini-native `modelOptions` on the image + * generation path (fix/gemini-native-image-model-options). + * + * Before that fix, `GeminiImageAdapter`'s `generateWithGeminiApi` only ever + * forwarded `modelOptions.seed` into the `generateContent` request — + * `safetySettings`, `thinkingConfig`, `imageConfig`, and `systemInstruction` + * were silently dropped even though the adapter's provider-options type + * (`GeminiNativeImageProviderOptions`) already declared them. This route + * drives `generateImage()` against `gemini-2.5-flash-image` with + * `modelOptions: { safetySettings, thinkingConfig }` set, hitting + * `geminiNativeImageMount` in global-setup.ts — a hand-mocked + * `POST /v1beta/models/gemini-2.5-flash-image:generateContent` endpoint that + * reads the raw, untranslated request body (aimock's own journal cannot see + * these fields for this endpoint — see that mount's comment) and rejects + * with 400 unless `safetySettings` is present at the request root and + * `generationConfig.thinkingConfig` is present nested, and rejects unless no + * Imagen-only field (`personGeneration`, `negativePrompt`, a root-level + * `aspectRatio`, …) is present anywhere in the body. + * + * A regression that stops forwarding `modelOptions` on this path — reverting + * to only `seed`, or reverting to a wholesale `...modelOptions` spread that + * lets an Imagen field cross over — makes the mount reject the request, the + * adapter's `client.models.generateContent()` call throws, and this route + * returns `ok: false`. The companion spec asserts `ok: true`. + */ +export const Route = createFileRoute('/api/gemini-native-image-wire')({ + server: { + handlers: { + POST: async () => { + const adapter = createImageAdapter('gemini') + + try { + const result = await generateImage({ + adapter, + prompt: 'a guitar in a music store', + stream: false, + modelOptions: { + safetySettings: [ + { + category: 'HARM_CATEGORY_DANGEROUS_CONTENT', + threshold: 'BLOCK_ONLY_HIGH', + }, + ], + thinkingConfig: { thinkingBudget: 128 }, + }, + }) + return new Response( + JSON.stringify({ ok: true, images: result.images.length }), + { + status: 200, + headers: { 'Content-Type': 'application/json' }, + }, + ) + } catch (error) { + return new Response( + JSON.stringify({ + ok: false, + error: error instanceof Error ? error.message : String(error), + }), + { status: 200, headers: { 'Content-Type': 'application/json' } }, + ) + } + }, + }, + }, +}) diff --git a/testing/e2e/tests/gemini-native-image-wire.spec.ts b/testing/e2e/tests/gemini-native-image-wire.spec.ts new file mode 100644 index 0000000000..283638baf3 --- /dev/null +++ b/testing/e2e/tests/gemini-native-image-wire.spec.ts @@ -0,0 +1,43 @@ +import { test, expect } from './fixtures' + +/** + * Wire-format verification for Gemini-native `modelOptions` on the image + * generation path (fix/gemini-native-image-model-options). + * + * `/api/gemini-native-image-wire` drives `generateImage()` against + * `gemini-2.5-flash-image` with `modelOptions: { safetySettings, + * thinkingConfig }`. That request lands on `geminiNativeImageMount` in + * global-setup.ts, which reads the raw, untranslated request body (aimock's + * own journal normalises this endpoint's requests and drops these exact + * fields before journalling — see that mount's comment) and rejects with 400 + * unless `safetySettings` is present at the request root, + * `generationConfig.thinkingConfig` is present nested under + * `generationConfig`, and no Imagen-only field (`personGeneration`, + * `negativePrompt`, a root-level `aspectRatio`, …) appears anywhere in the + * body. + * + * Before the fix, `generateWithGeminiApi` only forwarded `modelOptions.seed` + * — `safetySettings` and `thinkingConfig` were silently dropped even though + * the adapter's own provider-options type already declared them. Reverting + * the fix reproduces that: the outgoing request loses both fields, the mount + * rejects it with 400, `client.models.generateContent()` throws, and the + * route returns `ok: false` — this spec's `ok` assertion fails. + */ +test.describe('gemini native image — modelOptions reach the generateContent wire', () => { + test('safetySettings and thinkingConfig survive to the request; no Imagen field does', async ({ + request, + }) => { + const res = await request.post('/api/gemini-native-image-wire') + expect(res.ok()).toBe(true) + + const { ok, images, error } = (await res.json()) as { + ok: boolean + images?: number + error?: string + } + + expect(error ?? null).toBeNull() + expect(ok).toBe(true) + expect(images).toBe(1) + }) +})