Add the deep fragmenter record-building interface

CONTEXT.md

@@ -28,6 +28,10 @@ _Avoid_: Custom field, pass-through field
 An allowlisted public Ghost source field or package-owned compatibility projection that projection configuration may include, omit, or expose under a validated alias. Enabled optional fields are repeated in every Algolia record derived from that Ghost content.
 _Avoid_: Custom field, arbitrary field
 
+**Ranking sibling**:
+An additional custom-ranking value carried beside the package-owned heading and position values, sourced from an allowlisted numeric or boolean Ghost field under a validated alias.
+_Avoid_: Custom ranking field, ranking attribute, sort field
+
 **Extraction fragment**:
 An ordered emitted unit of searchable rendered meaning with searchable fragment HTML, preserved source text, heading context, and a fragment source.
 _Avoid_: Chunk, paragraph record
@@ -44,6 +48,10 @@ _Avoid_: Outer HTML, raw attribute value
 A stable description of whether an extraction fragment came from element content or an attribute, including whether an element was selected as ordinary content or as a card-heading fallback.
 _Avoid_: Parser node, candidate ID, card adapter
 
+**Anchor group**:
+The ordered extraction fragments of one Ghost content item that share the same anchor, kept in first-seen anchor order. It is the unit that fixes an Algolia record's deep link, heading context, and identifier.
+_Avoid_: Heading group, section, chunk
+
 **HTML extractor**:
 The component that converts rendered HTML into ordered extraction fragments.
 _Avoid_: Fragmenter, transformer
@@ -56,6 +64,22 @@ _Avoid_: HTML extractor
 The final indexed object containing projected Ghost fields, grouped extracted content, and ranking metadata.
 _Avoid_: Extraction fragment, Algolia post
 
+**Fallback record**:
+The single Algolia record emitted for Ghost content that produces no extraction fragments. It carries the Ghost content projection with empty fragment content and the headingless rank.
+_Avoid_: Empty record, placeholder record, stub
+
+**Continuation record**:
+An Algolia record carrying the later whole extraction fragments of one anchor group that did not fit within the record byte ceiling. It repeats the same projection and deep link under a stable suffixed object ID.
+_Avoid_: Split record, overflow record, record page
+
+**Record byte ceiling**:
+The largest compact UTF-8 byte size allowed for one complete Algolia record, chosen so output stays valid on Algolia's smallest plan.
+_Avoid_: Size limit, 10 KB limit, character count
+
+**Preflight**:
+The offline check that validates caller policy, Ghost content, and every complete Algolia record before any Algolia request. Failing preflight produces no records at all.
+_Avoid_: Dry run (for this record check; the term still belongs to release tooling), validation pass, sanity check
+
 **Ghost-rendered fixture**:
 An immutable Content API response produced by Ghost from controlled source content and retained as deterministic test evidence.
 _Avoid_: Mock response, live fixture

packages/algolia-fragmenter/README.md

@@ -16,7 +16,59 @@ pnpm add @tryghost/algolia-fragmenter
 
 ## Usage
 
-Convert Ghost Content API posts, then reduce the resulting records into fragments:
+`createAlgoliaRecords` turns Ghost content into complete final Algolia records in one synchronous call. It owns projection, HTML extraction, heading-anchor grouping, fallback records, deep links, identifiers, ranking metadata, record-size handling, and validation:
+
+```js
+import {createAlgoliaRecords} from '@tryghost/algolia-fragmenter';
+
+const records = createAlgoliaRecords(posts);
+```
+
+Every record contains the package-owned fields `objectID`, `slug`, `url`, `html`, `title`, `headings`, `anchor`, and `customRanking` with its `position` and `heading` values. Ghost content that produces no extraction fragments emits one fallback record with empty `html`, no anchor, and the headingless rank. The required Ghost input fields are `id`, `slug`, `url`, `title`, and `html`.
+
+### Options
+
+```js
+const records = createAlgoliaRecords(posts, {
+    ignoreSlugs: ['secret-page'],
+    contentProjection: {
+        fields: ['image', 'tags', {source: 'reading_time', as: 'readingMinutes'}],
+        customRanking: [{source: 'featured', as: 'isFeatured'}]
+    }
+});
+```
+
+- `ignoreSlugs` excludes content by slug before the rest of its fields are validated.
+- `contentProjection.fields` is the complete optional field set; it replaces the default set rather than patching it, and `[]` selects no optional fields. Without `contentProjection`, the optional fields are `image`, `tags`, `authors`, and `excerpt`.
+- `contentProjection.customRanking` adds ranking siblings beside the package-owned `position` and `heading` values. Each sibling needs a validated alias.
+
+The optional source allowlist is `image`, `tags`, `authors`, `excerpt`, `custom_excerpt`, `feature_image_alt`, `feature_image_caption`, `canonical_url`, `featured`, `visibility`, `created_at`, `updated_at`, `published_at`, and `reading_time`. Ranking siblings may only be sourced from `featured` and `reading_time`. A field may be aliased with `{source, as}`, where `as` matches `^[A-Za-z][A-Za-z0-9_]*$` and changes only the output key.
+
+Enabled optional fields are repeated in every record derived from the same Ghost content. A missing scalar becomes `null`, missing `tags` or `authors` become `[]`, and meaningful `false`, `0`, and empty-string values are preserved. `image` reads Ghost's `feature_image`, and `tags` and `authors` keep the `{name, slug}` shape.
+
+### Record size
+
+Every complete record stays within 9,999 compact UTF-8 bytes. Whole extraction fragments are packed greedily and never truncated: the first record of an anchor group keeps `<content id>_<group index>` and continuations add `_<continuation index>`. An indivisible fragment, or required metadata that leaves no room for one, fails instead of being shortened.
+
+### Errors
+
+`createAlgoliaRecords` validates the whole batch and returns no records when any deterministic problem exists — it never returns a partial array. It throws one `FragmenterError` whose `code` is `INVALID_POLICY`, `INVALID_GHOST_CONTENT`, or `RECORD_TOO_LARGE`, and whose `issues` array lists every issue in input order:
+
+```js
+import {createAlgoliaRecords, FragmenterError} from '@tryghost/algolia-fragmenter';
+
+try {
+    createAlgoliaRecords(posts, options);
+} catch (error) {
+    if (error instanceof FragmenterError) {
+        console.error(error.code, error.issues);
+    }
+}
+```
+
+Policy issues carry the configuration `path` that must change. Content issues add the batch `index`, the Ghost content id, and the expected type. Size issues add the record's `objectID`, the anchor and source position when available, the measured `bytes`, the 9,999-byte `limit`, and the `excess`.
+
+### Deprecated wrappers
 
 ```js
 import {fragmentTransformer, transformToAlgoliaObject} from '@tryghost/algolia-fragmenter';
@@ -27,7 +79,7 @@ const fragments = records.reduce(fragmentTransformer, []);
 
 `transformToAlgoliaObject` accepts an optional array of post slugs to exclude as its second argument. `fragmentTransformer` is designed to be passed directly to `Array#reduce`.
 
-Both operations are deprecated compatibility wrappers. They remain available with their existing output while a deeper record-building API is introduced separately.
+Both operations are deprecated compatibility wrappers. They keep their existing output, do not receive the projection policy, and do not apply the record-size behaviour. New callers should use `createAlgoliaRecords`.
 
 This package is ESM-only and requires Node.js 24 or later.
 

packages/algolia-fragmenter/lib/create-algolia-records.d.mts

@@ -0,0 +1,16 @@
+import { type CreateAlgoliaRecordsOptions } from './policy.mjs';
+import { type GhostContent } from './projection.mjs';
+import { type AlgoliaRecord } from './records.mjs';
+/**
+ * Turns Ghost content into complete final Algolia records: projection, HTML extraction, legacy
+ * anchor grouping, fallback records, deep links, identifiers, ranking metadata, and
+ * deterministic record-size handling.
+ *
+ * The whole batch is validated before any record is returned. A deterministic policy, Ghost
+ * content, or record-size problem throws one {@link FragmenterError} carrying every issue in
+ * input order; a partial batch is never returned.
+ *
+ * @throws {FragmenterError} `INVALID_POLICY`, `INVALID_GHOST_CONTENT`, or `RECORD_TOO_LARGE`.
+ */
+export declare const createAlgoliaRecords: (ghostContent: readonly GhostContent[], options?: CreateAlgoliaRecordsOptions) => readonly AlgoliaRecord[];
+//# sourceMappingURL=create-algolia-records.d.mts.map

packages/algolia-fragmenter/lib/create-algolia-records.mjs

@@ -0,0 +1,40 @@
+import { extract } from '@tryghost/algolia-html-extractor';
+import { FragmenterError } from './errors.mjs';
+import { groupFragmentsByAnchor } from './grouping.mjs';
+import { resolvePolicy } from './policy.mjs';
+import { prepareGhostContent } from './projection.mjs';
+import { createContentRecords } from './records.mjs';
+/**
+ * Turns Ghost content into complete final Algolia records: projection, HTML extraction, legacy
+ * anchor grouping, fallback records, deep links, identifiers, ranking metadata, and
+ * deterministic record-size handling.
+ *
+ * The whole batch is validated before any record is returned. A deterministic policy, Ghost
+ * content, or record-size problem throws one {@link FragmenterError} carrying every issue in
+ * input order; a partial batch is never returned.
+ *
+ * @throws {FragmenterError} `INVALID_POLICY`, `INVALID_GHOST_CONTENT`, or `RECORD_TOO_LARGE`.
+ */
+export const createAlgoliaRecords = (ghostContent, options) => {
+    const policy = resolvePolicy(options);
+    if (!policy.ok) {
+        throw new FragmenterError('INVALID_POLICY', policy.issues);
+    }
+    const prepared = prepareGhostContent(ghostContent, policy.policy);
+    if (!prepared.ok) {
+        throw new FragmenterError('INVALID_GHOST_CONTENT', prepared.issues);
+    }
+    const records = [];
+    const issues = [];
+    for (const content of prepared.contents) {
+        const groups = groupFragmentsByAnchor(extract(content.html));
+        const contentRecords = createContentRecords(content, groups);
+        records.push(...contentRecords.records);
+        issues.push(...contentRecords.issues);
+    }
+    if (issues.length > 0) {
+        throw new FragmenterError('RECORD_TOO_LARGE', issues);
+    }
+    return records;
+};
+//# sourceMappingURL=create-algolia-records.mjs.map

packages/algolia-fragmenter/lib/errors.d.mts

@@ -0,0 +1,44 @@
+export type FragmenterErrorCode = 'INVALID_POLICY' | 'INVALID_GHOST_CONTENT' | 'RECORD_TOO_LARGE';
+export type PolicyIssueReason = 'invalid-shape' | 'unknown-property' | 'unknown-source' | 'repeated-source' | 'repeated-output' | 'invalid-alias' | 'protected-collision' | 'container-collision' | 'canonical-collision' | 'reserved-collision';
+export type PolicyIssue = Readonly<{
+    kind: 'policy';
+    reason: PolicyIssueReason;
+    path: string;
+    message: string;
+}>;
+export type GhostContentIssueReason = 'invalid-shape' | 'missing' | 'wrong-type';
+export type ExpectedValueType = 'string' | 'number' | 'boolean' | 'object' | 'array';
+export type GhostContentIssue = Readonly<{
+    kind: 'content';
+    reason: GhostContentIssueReason;
+    path: string;
+    index: number | null;
+    contentId: string | null;
+    expected: ExpectedValueType;
+    message: string;
+}>;
+export type RecordSizeIssue = Readonly<{
+    kind: 'size';
+    reason: 'record-too-large';
+    path: string;
+    index: number;
+    contentId: string;
+    objectID: string;
+    anchor: string | null;
+    position: number | null;
+    bytes: number;
+    limit: number;
+    excess: number;
+    message: string;
+}>;
+export type FragmenterIssue = PolicyIssue | GhostContentIssue | RecordSizeIssue;
+/**
+ * The single public error for every deterministic policy, Ghost content, or record size
+ * problem found while building Algolia records. It never carries a partial record batch.
+ */
+export declare class FragmenterError extends Error {
+    readonly code: FragmenterErrorCode;
+    readonly issues: readonly FragmenterIssue[];
+    constructor(code: FragmenterErrorCode, issues: readonly FragmenterIssue[]);
+}
+//# sourceMappingURL=errors.d.mts.map

packages/algolia-fragmenter/lib/errors.mjs

@@ -0,0 +1,23 @@
+const MESSAGE_ISSUE_LIMIT = 5;
+const describeIssues = (code, issues) => {
+    const listed = issues.slice(0, MESSAGE_ISSUE_LIMIT).map(issue => issue.message);
+    const remaining = issues.length - listed.length;
+    const suffix = remaining > 0 ? `; and ${remaining} more` : '';
+    const count = `${issues.length} issue${issues.length === 1 ? '' : 's'}`;
+    return `${code}: ${count}. ${listed.join('; ')}${suffix}`;
+};
+/**
+ * The single public error for every deterministic policy, Ghost content, or record size
+ * problem found while building Algolia records. It never carries a partial record batch.
+ */
+export class FragmenterError extends Error {
+    code;
+    issues;
+    constructor(code, issues) {
+        super(describeIssues(code, issues));
+        this.name = 'FragmenterError';
+        this.code = code;
+        this.issues = Object.freeze([...issues]);
+    }
+}
+//# sourceMappingURL=errors.mjs.map