extrait

High-level LLM text generation and structured JSON extraction with validation, repair, and streaming.

Features

Multi-candidate JSON extraction from LLM responses
Automatic repair with jsonrepair
Zod schema validation and coercion
Optional self-healing for validation failures
Streaming support
MCP tools
Vector embeddings (OpenAI-compatible + Voyage AI)
Decision models (llama.cpp /v1/systemone): typed choice, yes/no and score answers
Pass-through multimodal images from paths, URLs, bytes, data URLs, or blobs

Installation

Install extrait with your preferred package manager.

bun add extrait
# or
npm install extrait
# or
deno add npm:extrait

Quick Start

openai-compatible targets the Responses API. Use openai-compatible-legacy for servers that only expose Chat Completions.

import { createLLM, prompt, s } from "extrait";
import { z } from "zod";

const llm = createLLM({
  provider: "openai-compatible",
  model: "mistralai/ministral-3-3b",
  baseURL: "http://localhost:1234/v1",
  apiKey: process.env.LLM_API_KEY ?? "local-demo-key",
});

const text = "Beat two eggs, add flour and milk, then cook thin pancakes in a hot pan.";

const RecipeSchema = s.schema(
  "Recipe",
  z.object({
    title: s.string().min(1).describe("Short recipe title"),
    ingredients: s.array(s.string()).min(1).describe("Ingredient list"),
  })
);

const result = await llm.structured(
  RecipeSchema,
  prompt`Extract a simple recipe from this text: """${text}"""`
);

console.log(result.data);

Examples at a Glance

These examples cover the most common usage patterns in the repository.

examples/simple.ts - Basic structured output with streaming
examples/generate.ts - High-level text generation
examples/logprobs.ts - Token log probabilities and alternatives
examples/streaming.ts - Real-time partial output and snapshot updates
examples/calculator-tool.ts - Structured extraction with MCP tools
examples/streaming-turns-with-tools.ts - Streaming MCP turns, transitions, and reasoning blocks
examples/conversation.ts - Multi-turn prompts and multimodal content
examples/image-analysis.ts - Vision input with structured output
examples/embeddings.ts - Embeddings and similarity workflows
examples/decision.ts - Routing with a llama.cpp decision model

bun run dev simple "Bun.js runtime"
bun run dev generate "Bun.js runtime"
bun run dev streaming
bun run dev calculator-tool

API Reference

The sections below cover the main building blocks of the library.

Create an LLM Client

Use createLLM() to configure the provider, model, transport, and client defaults.

const llm = createLLM({
  provider: "openai-compatible" | "openai-compatible-legacy" | "anthropic-compatible",
  model: "gpt-5-nano",
  baseURL: "http://localhost:1234/v1",     // required
  apiKey: process.env.LLM_API_KEY,         // optional
  transport: {                             // optional advanced settings
    path: "/v1/responses",                 // optional provider endpoint override
    embeddingPath: "/v1/embeddings",       // optional embedding endpoint (openai-compatible only)
    decisionPath: "/v1/systemone",         // optional decision-model endpoint (openai-compatible only)
    headers: { "x-trace-id": "docs-demo" }, // optional extra headers
    defaultBody: { user: "docs-demo" },    // optional provider body defaults
    version: "2023-06-01",                 // anthropic-compatible only
    defaultMaxTokens: 4096,                // anthropic-compatible only (default: 1024)
    defaultMaxToolRounds: 10,              // optional cap on MCP tool rounds (default: 100)
    fetcher: fetch,                        // optional custom fetch implementation
  },
  defaults: {
    mode: "loose" | "strict",             // loose allows repair
    selfHeal: 1,                          // optional retry attempts
    debug: false,                         // optional structured debug output
    // or:
    // debug: { enabled: true, verbose: true },
    systemPrompt: "You are a helpful assistant.",
    timeout: {
      request: 30_000,
      tool: 10_000,
    },
  },
});

The endpoint and credentials always live at the top level (baseURL, apiKey); transport is reserved for advanced settings such as endpoint paths, headers, body defaults, and a custom fetcher.

baseURL is required for the built-in providers. There is no implicit https://api.openai.com / https://api.anthropic.com fallback, so a client that was never pointed at an endpoint fails at construction instead of shipping your key to a vendor host. For request-specific options such as stream, request, schemaInstruction, and parse tuning, see the sections below.

Common setup patterns:

// OpenAI-compatible gateway or local endpoint
const llm = createLLM({
  provider: "openai-compatible",
  model: "local-model",
  baseURL: process.env.LLM_BASE_URL ?? "http://localhost:1234/v1", // never left undefined
  apiKey: process.env.LLM_API_KEY ?? "local-demo-key",
});

// Chat Completions-only gateway
const legacy = createLLM({
  provider: "openai-compatible-legacy",
  model: "local-model",
  baseURL: "http://localhost:1234",
  apiKey: "local-demo-key",
});

// Anthropic-compatible endpoint with explicit API version
const anthropic = createLLM({
  provider: "anthropic-compatible",
  model: "claude-3-5-sonnet-latest",
  baseURL: "http://localhost:8080",
  apiKey: process.env.LLM_API_KEY,
  transport: {
    version: "2023-06-01",
  },
});

Defining Schemas

Use the s wrapper around Zod for schema names, descriptions, and a more ergonomic authoring flow.

import { s } from "extrait";
import { z } from "zod";

const Schema = s.schema(
  "SchemaName",
  z.object({
    // String fields
    text: s.string().min(1).describe("Field description"),
    optional: s.string().optional(),
    withDefault: s.string().default("value"),

    // Numbers
    count: s.number().int().min(0).max(100),
    score: s.number().min(0).max(1),

    // Arrays
    items: s.array(s.string()).min(1).max(10),

    // Nested objects
    nested: z.object({
      field: s.string(),
    }),

    // Enums (use native Zod)
    category: z.enum(["a", "b", "c"]),

    // Booleans
    flag: s.boolean(),
  })
);

Making Structured Calls

structured() accepts a schema plus either a tagged prompt, a fluent prompt builder, or a raw message payload.

// Simple prompt
const result = await llm.structured(
  Schema,
  prompt`Your prompt with ${variables}`
);

// Multi-part prompt
const result = await llm.structured(
  Schema,
  prompt()
    .system`You are an expert assistant.`
    .user`Analyze: """${input}"""`
);

// Multi-turn conversation
const conversationResult = await llm.structured(
  Schema,
  prompt()
    .system`You are an expert assistant.`
    .user`Hello`
    .assistant`Hi, how can I help?`
    .user`Analyze: """${input}"""`
);

// With options
const result = await llm.structured(
  Schema,
  prompt`Your prompt`,
  {
    mode: "loose",
    selfHeal: 1,
    debug: true,
    systemPrompt: "You are a helpful assistant.",
    stream: {
      to: "stdout",
      onData: (event) => {
        if (event.delta.text) {
          console.log("New visible text:", event.delta.text);
        }
        if (event.delta.reasoning) {
          console.log("New reasoning text:", event.delta.reasoning);
        }

        console.log("Current visible text:", event.snapshot.text);
        console.log("Current reasoning:", event.snapshot.reasoning);
        console.log("Current structured snapshot:", event.snapshot.data);

        if (event.done) {
          console.log("Streaming done.");
        }
      },
    },
    request: {
      signal: AbortSignal.timeout(30_000),  // optional AbortSignal
      reasoningEffort: "medium",            // optional reasoning effort hint
    },
    timeout: {
      request: 30_000,  // ms per LLM HTTP request
      tool: 10_000,     // ms per MCP tool call
    },
  }
);

prompt() builds an ordered messages payload. Use prompt`...` for a single string prompt, or the fluent builder for multi-turn conversations. The LLMMessage type is exported if you need to type your own message arrays.

In stream.onData, the event is split into two layers:

event.delta.text is only the newly received visible text since the previous event.
event.delta.reasoning is only the newly received reasoning text since the previous event.
event.snapshot.text is the full visible text accumulated so far.
event.snapshot.reasoning is the full normalized reasoning accumulated so far.
event.snapshot.data is the best structured JSON snapshot that can be parsed from the stream so far. It may stay unchanged while event.delta.text continues to grow.
event.resync is set in the rare case where text already delivered by a delta is withdrawn (e.g. a <think> tag split across chunks in an unexpected case, or a final response that differs from the streamed chunks). For each flagged field, the delta is then the whole stable value and replaces what the previous deltas built: view = event.resync?.text ? event.delta.text : view + event.delta.text. Folded this way, the deltas always rebuild the output exactly.

Recomputing event.snapshot.data means repairing and parsing the whole accumulated text, which gets expensive on long generations. It is only done when snapshot.data is read, so a consumer that never reads it never pays for it. By default this is adaptive: while the accumulated text is small (≤ 8 KB) every event reparses, then recomputations are coalesced to at most every 25 ms, events in between reuse the last parsed value, so snapshot.data can briefly lag behind snapshot.text. The final done event always reparses the complete output. Set stream.dataInterval (ms) to pick the interval yourself, or 0 to reparse on every event regardless of size:

stream: {
  dataInterval: 0, // exact preview: recompute snapshot.data on every event
  onData: (event) => render(event.snapshot.data),
},

Typical usage is:

render event.delta.text directly to a terminal or chat UI
optionally render event.delta.reasoning in a separate reasoning panel
use event.snapshot.data to drive partial structured UI state
use event.snapshot.text / event.snapshot.reasoning when you need the full accumulated state instead of only the latest increment

The stream is processed incrementally: each chunk is read once, and deltas never re-read the accumulated output. The snapshot strings are handed out without being copied, so a consumer that only reads delta stays linear however long the output grows. Reading snapshot.text (or comparing it) on every event costs its full length each time: keep a snapshot read for when you need it (e.g. the done event), and render from deltas in between.

You can also pass provider request options through request:

const result = await llm.structured(
  Schema,
  prompt`Summarize this document: """${text}"""`,
  {
    request: {
      temperature: 0,
      maxTokens: 800,
      body: { user: "demo-user" },
    },
  }
);

Making Text Calls

generate() is the high-level API for non-structured generation. It accepts the same prompt shapes as structured(), but does not inject any schema or parse the output. The call is always generate(prompt, options?): the prompt, possibly a { prompt?, systemPrompt?, messages? } payload, comes first, and the options second.

The former single-object form generate({ prompt, ...options }) was removed: it could not be told apart from a prompt payload, whose messages it silently dropped. Calling it now throws; move the options to the second argument, e.g. llm.generate({ messages }, { stream }).

// Simple prompt
const result = await llm.generate(
  prompt`Write a short summary of ${topic}.`
);

// Multi-message prompt
const result = await llm.generate(
  prompt()
    .system`You are a concise assistant.`
    .user`Summarize: """${text}"""`
);

// Raw messages payload
const result = await llm.generate({
  messages: [
    { role: "user", content: "Say hello in one sentence." },
  ],
});

Streaming mirrors structured(), except the snapshot only contains text and reasoning:

const result = await llm.generate(
  prompt`Explain ${topic} in one short paragraph.`,
  {
    stream: {
      enabled: true,
      onData: (event) => {
        process.stdout.write(event.delta.text);

        console.log("Full text so far:", event.snapshot.text);
        console.log("Full reasoning so far:", event.snapshot.reasoning);

        if (event.done) {
          console.log("Streaming done.");
        }
      },
    },
  }
);

Provider request options and MCP tools still go through request:

const result = await llm.generate(
  prompt`Use tools if needed and answer the user clearly.`,
  {
    request: {
      temperature: 0,
      maxTokens: 800,
      reasoningEffort: "medium",
      mcpClients: [calculatorMCP],
      maxToolRounds: 10,
    },
  }
);

reasoningEffort is one provider-independent scale, ordered from least to most: none, minimal, low, medium, high, xhigh, max. Each adapter maps it onto its own vendor scale, and sends it as reasoning: { effort } on openai-compatible, reasoning_effort on openai-compatible-legacy, and output_config.effort on anthropic-compatible.

`reasoningEffort`	`openai-compatible` / `-legacy`	`anthropic-compatible`
`none`	`none`	thinking disabled, no `effort`
`minimal`	`minimal`	`low`
`low` / `medium` / `high` / `xhigh`	unchanged	unchanged
`max`	`xhigh`	`max`

OpenAI's scale tops out at xhigh, so max lands there: it is the same ceiling under the other vendor's name. Anthropic treats xhigh and max as different levels, so both are preserved.

Any effort other than none also auto-enables thinking: { type: "adaptive" } on anthropic-compatible; the thinking content comes back in result.reasoning.

For existing history or multi-turn conversations, pass messages directly:

const messages = conversation("You are a helpful assistant.", [
  { role: "user", text: "What is the speed of light?" },
  { role: "assistant", text: "Approximately 299,792 km/s in a vacuum." },
  { role: "user", text: "How long does light take to reach Earth from the Sun?" },
]);

const result = await llm.generate({ messages });

Use llm.adapter.complete(...) or llm.adapter.stream(...) only when you need the raw low-level provider interface.

When streaming through the adapter directly, createStreamNormalizer() does what generate() does with the raw chunks, incrementally: it hides <think> blocks, merges them with the dedicated reasoning, and returns stable deltas (a trailing fragment that may still become a tag is held back until it is decided):

import { createStreamNormalizer } from "extrait";

const normalizer = createStreamNormalizer();
const response = await llm.adapter.stream!(request, {
  onChunk: (chunk) => {
    const { delta } = normalizer.push({ text: chunk.textDelta, reasoning: chunk.reasoningDelta });
    ui.appendText(delta.text);
    ui.appendReasoning(delta.reasoning);
  },
});
// The final response is authoritative; this releases anything still held back.
const { delta, text, reasoning } = normalizer.finish({ text: response.text, reasoning: response.reasoning });

Token Logprobs

Set request.topLogprobs (0-20) to get token-level probabilities on openai-compatible and openai-compatible-legacy. The result exposes logprobs.content, one entry per generated token, each with its chosen token, logprob, and the requested number of alternatives.

const result = await llm.generate(prompt`Answer with one word: yes or no?`, {
  request: { topLogprobs: 3 },
});

for (const token of result.logprobs?.content ?? []) {
  const probability = Math.exp(token.logprob);
  console.log(token.token, probability, token.top_logprobs);
}

On openai-compatible (Responses API) this sends top_logprobs and automatically adds the message.output_text.logprobs include; on openai-compatible-legacy it sends logprobs: true + top_logprobs. Streaming accumulates logprobs across chunks. Anthropic has no logprobs API, so anthropic-compatible never returns them.

Some Responses-dialect servers (e.g. llama.cpp) also require the Chat Completions-style flag; pass it through transport.defaultBody: { logprobs: true }. See examples/logprobs.ts.

Images (multimodal)

extrait transports images to vision-capable models without decoding, resizing, recompressing, or otherwise transforming their pixel data. It exposes two helpers that produce LLMImageContent[] blocks ready to spread into a user or assistant message:

	Accepts	I/O
`images(source)`	data or `http(s)` URL, remote/data `URL` object, `Uint8Array`, `ArrayBuffer`, `Buffer`, or `{ base64, mimeType }`	Synchronous; no I/O
`await loadImages(source)`	Everything accepted by `images()`, plus file paths, `file:` URLs, `Blob`, and `File`	Reads local files and blobs

Both helpers accept one source or an array of sources and preserve their order.

import { loadImages, prompt } from "extrait";

const imageContent = await loadImages("photo.png");

const result = await llm.structured(
  Schema,
  prompt()
    .system`You are a vision assistant.`
    .user([
      { type: "text", text: "Describe this image." },
      ...imageContent,
    ]),
);

Use images() when the source is already available in memory or remotely:

import { images } from "extrait";

const remote = images("https://example.com/photo.png");
const encoded = images("data:image/png;base64,iVBORw0KG...");
const fromBytes = images(imageBytes);
const explicit = images({ base64, mimeType: "image/png" });
const multiple = images([firstImageBytes, secondImageBytes]);

Use loadImages() when a source must be read asynchronously:

import { loadImages } from "extrait";

const local = await loadImages("photo.png");
const fromFileURL = await loadImages(new URL("file:///tmp/photo.png"));
const fromBlob = await loadImages(new Blob([imageBytes], { type: "image/png" }));

const mixed = await loadImages([
  "first.png",
  "https://example.com/second.jpg",
  { base64, mimeType: "image/webp" },
]);

String sources are resolved as follows:

data: URLs are passed through unchanged.
http:// and https:// URLs are passed through unchanged; extrait does not fetch them.
Other strings are treated as local file paths and must be passed to loadImages().

A bare base64 string is ambiguous and is therefore treated as a file path. Use a data URL or { base64, mimeType } instead.

Raw bytes are wrapped in a base64 data URL. Their MIME type is detected from the file signature for PNG, JPEG, GIF, WebP, AVIF, HEIC, and BMP. For local files, the extension is used as a fallback. A Blob or { base64, mimeType } can provide the MIME type explicitly. The sniffMimeType(bytes) helper is also exported for direct use.

Remote and data URLs keep their original representation. Local files, blobs, and raw bytes are only read and wrapped as data URLs; pixel data is never processed. Make sure the source already meets the selected provider's supported formats and size limits.

See examples/image-analysis.ts.

Conversations (multi-turn history)

Use conversation() to build a LLMMessage[] from an existing conversation history. This is the idiomatic way to pass prior turns to the LLM.

import { conversation } from "extrait";

const messages = conversation("You are a helpful assistant.", [
  { role: "user",      text: "What is the speed of light?" },
  { role: "assistant", text: "Approximately 299,792 km/s in a vacuum." },
  { role: "user",      text: "How long does light take to reach Earth from the Sun?" },
]);

// High-level text generation
const response = await llm.generate({ messages });

// Or to structured extraction
const result = await llm.structured(Schema, { messages });

Consecutive tool_call entries are one assistant turn of parallel calls: they are merged into a single assistant message, together with the assistant text entry right before them, which is the shape providers require before the matching tool_result entries.

Entries with images produce multimodal content automatically:

const messages = conversation("You are a vision assistant.", [
  {
    role: "user",
    text: "What is in this image?",
    images: ["https://example.com/photo.png"], // any synchronous source
  },
]);

Result Object

Successful generate() calls return normalized text/reasoning plus request metadata:

{
  text: string,
  reasoning: string,
  attempts: GenerateAttempt[],
  usage?: {
    inputTokens?: number,      // billed prompt tokens, summed over every call (tool rounds included)
    outputTokens?: number,
    totalTokens?: number,
    cost?: number,
    contextTokens?: number,    // largest single prompt: context occupancy, comparable to the window
  },
  finishReason?: string,
  logprobs?: LLMLogprobs,          // when request.topLogprobs is set
  reasoningBlocks?: ReasoningBlock[], // per-turn reasoning for multi-round MCP streams
}

Each attempts entry includes:

{
  attempt: number,
  via: "complete" | "stream",
  text: string,
  reasoning: string,
  usage?: LLMUsage,
  finishReason?: string,
}

Successful structured() calls return validated data plus normalized text/reasoning and trace metadata.

{
  data: T,                      // Validated data matching schema
  text: string,                 // Visible model text, without inline <think> blocks
  reasoning: string,            // Normalized reasoning across dedicated fields and inline <think>
  json: unknown | null,         // Parsed JSON before validation
  attempts: StructuredAttempt<T>[], // One entry per parse / self-heal attempt
  usage?: {
    inputTokens?: number,      // billed prompt tokens, summed over every call (tool rounds included)
    outputTokens?: number,
    totalTokens?: number,
    cost?: number,
    contextTokens?: number,    // largest single prompt: context occupancy, comparable to the window
  },
  finishReason?: string,        // provider vocabulary: "completed" (Responses), "stop" (Chat Completions), "end_turn" (Anthropic)
  logprobs?: LLMLogprobs,          // when request.topLogprobs is set
  reasoningBlocks?: ReasoningBlock[], // per-turn reasoning for multi-round MCP streams
}

Each attempts entry includes:

{
  attempt: number,
  selfHeal: boolean,
  via: "complete" | "stream",
  text: string,
  reasoning: string,
  json: unknown | null,
  candidates: string[],
  repairLog: string[],
  zodIssues: z.ZodIssue[],
  success: boolean,
  usage?: LLMUsage,
  finishReason?: string,
  parsed: ParseLLMOutputResult<T>,
}

Legacy inline <think>...</think> blocks are still supported, but the high-level structured() API now folds them into reasoning internally instead of exposing block metadata.

Error Handling

Catch StructuredParseError when repair and validation still fail.

import { StructuredParseError } from "extrait";

try {
  const result = await llm.structured(Schema, prompt`...`);
} catch (error) {
  if (error instanceof StructuredParseError) {
    console.error("Validation failed");
    console.error("Attempt:", error.attempt);
    console.error("Zod issues:", error.zodIssues);
    console.error("Repair log:", error.repairLog);
    console.error("Candidates:", error.candidates);
  }
}

Embeddings

Generate vector embeddings using llm.embed(). It always returns number[][], one vector per input string.

// Create a dedicated embedder client (recommended)
const embedder = createLLM({
  provider: "openai-compatible",
  model: "text-embedding-3-small",
  baseURL: "http://localhost:1234/v1",
  apiKey: process.env.LLM_API_KEY,
});

// Single string
const { embeddings, model, usage } = await embedder.embed("Hello world");
const vector: number[] = embeddings[0];

// Multiple strings in one request
const { embeddings } = await embedder.embed(["text one", "text two", "text three"]);
// embeddings[0], embeddings[1], embeddings[2]: one vector each

// Optional: override model or request extra options per call
const { embeddings } = await embedder.embed("Hello", {
  model: "text-embedding-ada-002",
  dimensions: 512,              // supported by text-embedding-3-* models
  body: { user: "user-id" },    // pass-through to provider
  signal: AbortSignal.timeout(10_000), // optional cancellation / timeout
});

Result shape:

{
  embeddings: number[][];  // one vector per input
  model: string;
  usage?: { inputTokens?: number; totalTokens?: number };
  raw?: unknown;           // full provider response
}

Anthropic / Voyage AI

Anthropic does not provide a native embedding API. Their recommended solution is Voyage AI, which uses the same OpenAI-compatible format:

const embedder = createLLM({
  provider: "openai-compatible",
  model: "voyage-3",
  baseURL: "https://api.voyageai.com",
  apiKey: process.env.LLM_API_KEY,
});

const { embeddings } = await embedder.embed(["query", "document"]);

Requests default to encoding_format: "float". If your endpoint rejects that value, override it per call via body, or drop it entirely with transport.defaultBody: { encoding_format: undefined }.

Calling llm.embed() on an anthropic-compatible adapter throws a descriptive error pointing to Voyage AI.

Decision Models

Decision models answer by scoring the options you give them instead of generating text: one pass over the input, a probability per option, no output tokens. llm.decide() calls llama.cpp's POST /v1/systemone endpoint (openai-compatible and openai-compatible-legacy providers).

// llama serve -hf ggml-org/Kev-4B-GGUF
const llm = createLLM({
  provider: "openai-compatible",
  model: "ggml-org/Kev-4B-GGUF",
  baseURL: "http://localhost:8080",
});

const { answers } = await llm.decide(
  "Customer message: I was charged twice for my order last week and nobody has replied.",
  {
    route: {
      type: "choice",                     // pick one option
      instructions: "Which team should handle this?",
      criteria: {
        billing: "payments, charges, refunds, invoices",
        shipping: "delivery, tracking, lost or late parcels",
        technical: "bugs, errors, login problems",
      },
    },
    angry: {
      type: "noul",                       // yes/no
      instructions: "Is the customer angry?",
    },
    urgency: {
      type: "score",                      // 2 to 10 levels, lowest first
      instructions: "How urgent is this?",
      criteria: ["can wait", "this week", "today", "right now"],
    },
  },
);

answers.route.choice;          // "billing" (typed as "billing" | "shipping" | "technical")
answers.route.probabilities;   // { billing: 0.9049, shipping: 0.0275, technical: 0.0676 }
answers.route.confidence;      // 0.8574
answers.angry.noul;            // 0.8208 (probability of "yes")
answers.urgency.score;         // 2.2821 (expected level, 0 = "can wait")
answers.urgency.legend;        // { "0": "can wait", "1": "this week", "2": "today", "3": "right now" }

if (answers.route.confidence < 0.5) {
  // act on confident answers, send the rest to a human
}

Choice options without descriptions can be given as a list: criteria: ["invoice", "receipt", "other"] is sent as { invoice: null, receipt: null, other: null }.

Options:

await llm.decide(state, questions, {
  images: await loadImages("./document.png"), // or data URLs; multimodal models only (e.g. OpenJev)
  model: "ggml-org/OpenJev-GGUF",            // select a vision model in router mode
  body: { /* pass-through fields */ },
  signal: AbortSignal.timeout(5_000),
});

state accepts plain text, JSON objects/arrays (including nested values), or chat messages (LLMMessage[], e.g. from conversation()). The server renders JSON content as text; image_url parts inside messages are read as images too, and must contain data URLs. The top-level state cannot be null.

Yes/no questions can describe both alternatives:

await llm.decide({ message: "Please refund my order", order: { id: 4471 } }, {
  refund: {
    type: "noul",
    instructions: "Is a refund requested?",
    criteria: { true: "money back is asked", false: "no money back is asked" },
  },
});

Questions are checked before sending (at least one question, a non-empty choice, 2 to 10 score levels). Responses must match the question type, contain all expected probabilities and score labels, and keep probabilities/confidence within [0, 1] and scores within the requested scale. A choice must be one of the supplied options. Malformed answers throw instead of returning partial results. Calling llm.decide() on an anthropic-compatible adapter throws.

Result shape:

{
  answers: { [key]: DecisionChoiceAnswer | DecisionNoulAnswer | DecisionScoreAnswer }; // typed per question
  model: string;
  usage?: { inputTokens?: number; outputTokens?: number };
  raw?: unknown;           // full server response
}

MCP Tools

Attach MCP clients at request time to let the model call tools during structured generation.

import { createMCPClient } from "extrait";

const mcpClient = await createMCPClient({
  id: "calculator",
  transport: {
    type: "stdio",
    command: "bun",
    args: ["run", "examples/calculator-mcp-server.ts"],
  },
});

const result = await llm.structured(
  Schema,
  prompt`Calculate 14 + 8`,
  {
    request: {
      mcpClients: [mcpClient],
      maxToolRounds: 5,       // default: 100 (also configurable via transport.defaultMaxToolRounds)
      toolDebug: {
        enabled: true,
        includeRequest: true,
        includeResult: true,
      },
      onToolExecution: (execution) => {
        console.log(execution.name, execution.durationMs);
      },
      // Optional: transform tool output before it is sent back to the LLM
      transformToolOutput: (output, execution) => {
        return { ...output, source: execution.name };
      },
      // Optional: transform tool arguments before the tool is called
      transformToolArguments: (args, call) => args,
      // Optional: transform the full MCP call payload, including _meta
      transformToolCallParams: (params, call) => ({
        ...params,
        _meta: {
          source: "extrait-docs",
          clientId: call.clientId,
        },
      }),
      // Optional: custom error message when an unknown tool is called
      unknownToolError: (toolName) => `Tool "${toolName}" is not available.`,
    },
  }
);

await mcpClient.close?.();

transformToolArguments() only receives the tool input object. transformToolCallParams() runs after it and receives the full MCPCallToolParams payload that will be sent to the MCP client:

type MCPCallToolParams = {
  name: string;
  arguments?: Record<string, unknown>;
  _meta?: Record<string, unknown>;
};

MCP toolsets may change between tool rounds. Providers call listTools() before each round, so a client can expose additional tools after a previous callTool() result, or remove tools that are no longer available. A listTools() pagination that never ends (a repeated cursor, or more than 100 pages) fails the request instead of hanging it.

callTool(params, { signal }) receives the request's abort signal: once the caller aborts (or the request times out), the remaining calls of the round are not started, and clients created with createMCPClient() cancel the in-flight call on the server.

Use transformToolCallParams() when you need to attach MCP-specific metadata, override the final remote tool name, or otherwise change the full request passed to client.callTool(). This hook is exported as LLMToolCallParamsTransformer.

Timeouts

Use timeout to set per-request and per-tool-call time limits without managing AbortSignal manually.

const result = await llm.structured(Schema, prompt`...`, {
  timeout: {
    request: 30_000,  // abort the LLM HTTP request after 30s
    tool: 5_000,      // abort each MCP tool call after 5s
  },
});

Both fields are optional. timeout.request creates an AbortSignal.timeout internally; it is ignored if you also pass request.signal (your signal takes precedence). timeout.tool wraps each MCP client transparently.

You can also set defaults on the client:

const llm = createLLM({
  provider: "openai-compatible",
  model: "gpt-5-nano",
  baseURL: process.env.LLM_BASE_URL ?? "http://localhost:1234/v1",
  apiKey: process.env.LLM_API_KEY,
  defaults: {
    timeout: { request: 60_000 },
  },
});

Examples

Run repository examples with bun run dev <example-name>.

Available examples:

generate - High-level text generation (generate.ts)
logprobs - Token log probabilities and verbose debug output (logprobs.ts)
streaming - Real LLM streaming + snapshot self-check (streaming.ts)
streaming-with-tools - Real text streaming with MCP tools + self-check (streaming-with-tools.ts)
streaming-turns-with-tools - Streaming MCP turns, transitions, and reasoning blocks (streaming-turns-with-tools.ts)
abort-signal - Start a generation then cancel quickly with AbortSignal (abort-signal.ts)
timeout - Set per-request and per-tool timeouts via the timeout option (timeout.ts)
simple - Basic structured output with streaming (simple.ts)
sentiment-analysis - Enum validation, strict mode (sentiment-analysis.ts)
data-extraction - Complex nested schemas, self-healing (data-extraction.ts)
multi-step-reasoning - Chained structured calls (multi-step-reasoning.ts)
calculator-tool - MCP tool integration (calculator-tool.ts)
image-analysis - Multimodal structured extraction from an image file (image-analysis.ts)
conversation - Multi-turn conversation history and inline image messages (conversation.ts)
simulated-tools - Inject fake tool calls/results into conversation context without real execution (simulated-tools.ts)
embeddings - Vector embeddings, cosine similarity, and semantic comparison (embeddings.ts)
decision - Ticket routing with a llama.cpp decision model (decision.ts)

Pass arguments after the example name:

bun run dev generate "Why Bun is fast"
bun run dev logprobs "Answer with one word: yes or no?"
bun run dev streaming
bun run dev streaming-with-tools
bun run dev abort-signal 120 "JSON cancellation demo"
bun run dev timeout 5000
bun run dev simple "Bun.js runtime"
bun run dev sentiment-analysis "I love this product."
bun run dev multi-step-reasoning "Why is the sky blue?"
bun run dev embeddings "the cat sat on the mat" "a feline rested on the rug"
bun run dev decision "My parcel never arrived."

Environment Variables

These environment variables are used across the examples and common client setups.

LLM_PROVIDER - openai-compatible, openai-compatible-legacy, or anthropic-compatible
LLM_BASE_URL - API endpoint (required, no default)
LLM_MODEL - Model name (default: gpt-5-nano)
LLM_API_KEY - API key for the provider
STRUCTURED_DEBUG=1 - Enable debug output By default, structured debug prints text (public visible output) and reasoning (normalized reasoning). parseSource (the internal source used by parsing and self-heal) is only printed when debug.verbose is enabled.

Testing

Run the test suite with Bun.

bun run test

Check types and unused code across the source, tests, examples, and benchmarks with bun run lint. Use bun run format to apply the repository's Prettier formatting, or bun run format:check to verify it without changes. Run bun run build to generate the ESM/CJS bundles and TypeScript declarations.

Name		Name	Last commit message	Last commit date
bench		bench
examples		examples
src		src
tests		tests
.env.example		.env.example
.gitignore		.gitignore
.prettierignore		.prettierignore
.prettierrc.json		.prettierrc.json
AGENTS.md		AGENTS.md
CLAUDE.md		CLAUDE.md
LICENSE		LICENSE
README.md		README.md
bun.lock		bun.lock
package.json		package.json
tsconfig.build.json		tsconfig.build.json
tsconfig.json		tsconfig.json
tsconfig.lint.json		tsconfig.lint.json
tsconfig.typecheck.json		tsconfig.typecheck.json

Navigation Menu

Repository navigation

About

Repository files navigation

extrait

Features

Installation

Quick Start

Examples at a Glance

API Reference

Create an LLM Client

Defining Schemas

Making Structured Calls

Making Text Calls

Token Logprobs

Images (multimodal)

Conversations (multi-turn history)

Result Object

Error Handling

Embeddings

Decision Models

MCP Tools

Timeouts

Examples

Environment Variables

Testing

About

Releases

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

extrait

Features

Installation

Quick Start

Examples at a Glance

API Reference

Create an LLM Client

Defining Schemas

Making Structured Calls

Making Text Calls

Token Logprobs

Images (multimodal)

Conversations (multi-turn history)

Result Object

Error Handling

Embeddings

Decision Models

MCP Tools

Timeouts

Examples

Environment Variables

Testing

About

Topics

Resources

Stars

Watchers

Forks

Releases

Contributors

Languages