Skip to content

Commit da553ac

Browse files
feat(knowledge): index Confluence image attachments (#8949)
* refactor(knowledge): share the processing checkpoint store OCR and embedding checkpoints each carried their own copy of the framed header + body format, the outbox cleanup reservation and the cleanup handler factory. Move them into processing-checkpoint-store so a third checkpoint kind can reuse them. * feat(knowledge): index Confluence image attachments with a vision model Confluence images (PNG, JPEG, WebP, GIF) were dropped at listing, so tables, scans and screenshots attached to pages never reached search. - Connectors declare an image policy; Confluence reads images with a dedicated Gemini adapter that returns the image's text, every other source keeps OCR unchanged. - A new Image Attachments setting on the Confluence connector, on by default, so existing knowledge bases pick images up on their next sync. - Decorative images (logos, icons, images under 64 px) complete with no chunks; files under 2 KB and SVGs are never listed. - Each sync admits at most KB_IMAGE_EXTRACTIONS_PER_SYNC new images (default 500); the rest wait for the next sync, which reports partial. - A vision checkpoint lets a retried pass reuse the extraction instead of paying for a second call; usage is billed per image. - Completing a pass now always replaces old embeddings, so a document that re-indexes to zero chunks drops its stale ones. - sharp is added to the Trigger.dev bundle for image preparation. * feat(knowledge): explain image documents with nothing to index A completed image with no chunks shows an empty state instead of bare table headers, and the connector and self-hosting docs cover image attachments and the two new environment variables. * fix(knowledge): bound image re-reads and handle image opt-out directly Review follow-ups for Confluence image indexing. - The per-sync cap now counts changed images as well as new ones, and a member sync shares one budget across its members. Held-back images that already exist are still marked seen, so they are never reconciled away. - A full resync no longer re-reads unchanged images; their bytes are pinned to the attachment version. - Without any Gemini key, images are left out of the sync instead of becoming failed documents. - Turning Image Attachments off removes the indexed images on the next sync, outside the soft-delete and mass-deletion hold meant for source deletions. - Gemini errors: a response cut off at the token limit, a rejected key or region, and an unknown model fail permanently with an accurate message instead of retrying or blaming the image; an image whose pixel data cannot be decoded fails as invalid. - The admission wait keeps the post-call time reserve, Retry-After uses the shared parser, and a configured model must be a catalog Gemini model so its pricing is known. * fix(knowledge): make image indexing opt-in for existing Confluence connectors - New connectors are created with Image Attachments on, through a server-side defaultSourceConfig on the connector meta that every create surface applies; an unset value now means off, so existing connectors only read images once their owner turns the setting on. - The Gemini call retries transient server and network errors within the extraction's time budget; throttling still defers the pass. - Gemini error bodies are reduced to their status and reason codes. - Images under the byte floor complete as decorative even when the source reported no size, and the extraction model must be a generateContent Gemini model. - A member sync that held images back forces a full listing next time, the image count survives a listing restart and is saved per batch, and turning images off removes them for member-credential connectors too. - The disabled-image removal skips rows the delete would refuse. - An integration suite covers the cap, held-back images staying indexed, full resync, opt-out removal, opt-in for existing connectors, zero-chunk re-indexing and the no-key hold-back against real PostgreSQL; the OCR input-failure suite now uses a source that still sends images to OCR. * fix(knowledge): retry malformed extractions in-call and clean up images on every member run - A Gemini reply that is not valid JSON is retried within the extraction's budget and fails permanently once the attempts are spent, instead of costing a whole document retry each time. - A member sync that skips its content pass (content not due yet) still removes images once Image Attachments is turned off. - The reconciliation integration suite runs with images on, so it keeps exercising reconciliation alone. * fix(knowledge): count image removals in member runs and bill every extraction attempt - Member runs add the lifecycle's purges to the run's deletion counts instead of overwriting them, so image removals after Image Attachments is turned off are reported and logged as purges. - A retried extraction reports the tokens of every attempt Gemini answered, since each one was billed. - The checkpoint header check uses isRecordLike. - The create-time default is tested through the real creation path in the integration suite, replacing a mock-only unit test. * fix(knowledge): resume a member run whose image removal hit its deadline An unfinished disabled-image removal now keeps the member run marked as having work remaining, so the next run starts promptly instead of waiting a full sync interval with images still searchable. * refactor(knowledge): declare image extraction once and remove images only when queued - Connectors declare image extraction once on their meta (`imageExtraction: { settingId }`), read by the shared `isImageExtractionEnabled`; drops `ConnectorConfig.imageExtractionEnabled` and the Confluence-only helper. - Turning the image setting off sets `knowledge_connector.image_removal_pending` (migration 0412); the next sync removes the indexed images and clears it, so ordinary syncs no longer scan a connector's documents for images. - Gemini specifics (key, Gemini 3 model resolution, error classification) live in the adapter. Requests send `thinkingLevel: LOW`, and any finish reason other than STOP fails without paid retries. - Image reads get their own admission budget (`vision`, KB_CONFIG_VISION_REQUESTS_PER_MINUTE, default 300) instead of sharing OCR's. - One zod schema validates the reply and generates the request's JSON schema. - The prompt is source-neutral and treats text in an image as content, never instructions; sharp loads lazily so it can only fail image documents. - Full relisting after held-back work is one named rule (`noticeRequiresFullListing`). * fix(knowledge): keep op narrowing in processDocOps and clean up Project fixtures * refactor(knowledge): tighten image extraction after a full review - Every Gemini attempt takes its own admission, and 429/503 records a provider cooldown for the key; the retry delay is read from the body's RetryInfo when no Retry-After header is sent. - The image is base64-encoded once per extraction, rotated upright even when it already fits, and flattened onto white before a JPEG fallback. - Cost goes through calculateBillableModelCost; both indexing charges share one recordIndexingUsage helper. - Reconciliation waits until queued image removal finishes, and the removal walks by id instead of rescanning from the start. - The per-sync image cap is at least 1, so a sync can always finish. - canExtractImages lives with the extractor; isSemanticImage treats an undeclared policy as OCR, so callers drop their fallbacks. - Tests: unique embedding vectors, restored pool keys, waits for dispatched syncs, fake timers in finally, native noise fixtures; drops mock-only assertions. --------- Co-authored-by: Waleed Latif <walif6@gmail.com>
1 parent 9939762 commit da553ac

52 files changed

Lines changed: 33342 additions & 414 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎apps/docs/content/docs/knowledgebase/connectors.mdx‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -86,7 +86,7 @@ Each connector has source-specific fields that control what gets synced. Example
8686
- **Notion** — sync an entire workspace, a specific database, or a single page tree
8787
- **GitHub** — specify a repository, branch, and optional file extension filter
8888
- **GitLab** — specify a project path or ID and your instance host (leave blank for GitLab.com), then choose repository files, wiki pages, issues, or merge requests. Each connector syncs one project; submodules need their own connectors.
89-
- **Confluence** — enter your Atlassian domain and choose spaces, or **All** for all spaces accessible at each sync. Optionally filter by content type or label. PDF, Word (`.docx`, Word 97–2003 `.doc`), Excel (`.xlsx`), and PowerPoint (`.pptx`) attachments on matching pages and blog posts are included as separate documents.
89+
- **Confluence** — enter your Atlassian domain and choose spaces, or **All** for all spaces accessible at each sync. Optionally filter by content type or label. PDF, Word (`.docx`, Word 97–2003 `.doc`), Excel (`.xlsx`), and PowerPoint (`.pptx`) attachments on matching pages and blog posts are included as separate documents, and so are PNG, JPEG, WebP, and GIF images when **Image Attachments** is on.
9090
- **Azure DevOps** — choose what to sync (wiki pages, work items, repository files, or all), with optional work item type/state filters, a custom WIQL query, and repository/branch/path filters
9191
- **Amazon S3** — point at a bucket with an optional key prefix and a customizable file extension allowlist; S3-compatible stores (Cloudflare R2, MinIO) are supported via a custom endpoint
9292
- **YouTube** — sync a channel (by `@handle` or ID) or playlist, with an optional published-after date filter and the option to exclude Shorts
@@ -98,6 +98,8 @@ Configuration is validated on save — if a repository doesn't exist or a domain
9898

9999
Confluence attachment indexing requires `read:attachment:confluence`. For a service account, include it when creating the scoped API token; see the [Confluence scope list](/search/confluence#using-a-service-account). Attachments are checked even when the parent page has not changed. Files over 100 MB appear as skipped; convert Word 6/95 files to `.docx`, and `.xls` and `.ppt` files to `.xlsx` and `.pptx`, before attaching them. Spaces that were already connected pick up newly supported formats on their next sync.
100100

101+
Each image is read once by a vision model, which writes out what it holds: a table becomes one line per row with its column headers, a scan becomes its full text, and a screenshot or diagram becomes its visible text plus a description of what it shows. That text is what search finds. Logos, icons, and other decorative images are kept as documents but have nothing to search, so they show 0 chunks. Images under 2 KB, SVG, and HEIC files are not indexed. Image reading uses a Gemini key: the workspace's own key when one is added, otherwise the platform's, and each image is billed as a model call. Without any Gemini key, images are left out of the sync until one is added. One sync reads at most 500 new or changed images, and the rest are picked up by the next sync; a full resync does not read unchanged images again. New Confluence connectors read images by default. Connectors created before image support keep images off until you set **Image Attachments** to **Read with AI** in the connector's settings; the next sync then reads them. To stop indexing images, set it to **Don't index**: the next sync removes the images already indexed, and switching back reads them again, billed per image.
102+
101103
</Step>
102104
<Step>
103105

‎apps/docs/content/docs/platform/self-hosting/environment-variables.mdx‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -192,6 +192,9 @@ See [Observability](/platform/self-hosting/observability).
192192

193193
| Variable | Description |
194194
|----------|-------------|
195+
| `KB_IMAGE_EXTRACTION_MODEL` | Gemini model that reads Confluence image attachments into searchable text. Defaults to `gemini-3.8-flash`; a value that is not a Gemini 3 model in the catalog falls back to the default. Needs `GEMINI_API_KEY` (or a workspace Gemini key); without one, image attachments are left out of syncs |
196+
| `KB_IMAGE_EXTRACTIONS_PER_SYNC` | New or changed images one connector sync sends for reading, at least `1`. Defaults to `500`; the rest wait for the next sync, which reports as partial |
197+
| `KB_CONFIG_VISION_REQUESTS_PER_MINUTE` | Image reads per minute per Gemini key, shared by every sync on the deployment. Defaults to `300`; lower it to match a key with a smaller Gemini quota |
195198
| `KB_EMBEDDING_MODEL` | Embedding model for new knowledge bases. Defaults to `text-embedding-3-small`; use `ollama/<model>` for a model on your own Ollama. An unrecognised hosted model id falls back to the default, but an `ollama/` id is taken at face value — if that model is not on the server, knowledge-base creation fails rather than falling back |
196199
| `EMBEDDING_OUTPUT_DIMS` | Vector width new knowledge bases are stored at: `384`, `768`, `1024`, `1536` (default), or `3072`. It must be a width the chosen model can emit; anything else falls back to `1536` with a warning |
197200
| `OPENROUTER_API_KEY` | Fallback route for the OpenAI embedding models — used when it is set and `OPENAI_API_KEY` is not the chosen path |

‎apps/sim/.env.example‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -259,6 +259,12 @@ CRON_SECRET=your_cron_secret # Use `openssl rand -hex 32` to generate. Authentic
259259
# keys in the same organization to one group using SHA-256 fingerprints, never raw keys.
260260
# MISTRAL_OCR_QUOTA_GROUPS={"<64-character lowercase key fingerprint>":"organization-id"}
261261

262+
# Knowledge base images read with Gemini (Confluence "Image Attachments: Read with AI").
263+
# Needs GEMINI_API_KEY or a workspace Gemini key; without one, images are left out of syncs.
264+
# KB_IMAGE_EXTRACTION_MODEL=gemini-3.8-flash
265+
# KB_IMAGE_EXTRACTIONS_PER_SYNC=500
266+
# KB_CONFIG_VISION_REQUESTS_PER_MINUTE=300
267+
262268
# Official Sim Search Slack app (optional; requires existing Search access)
263269
# Supply these through the deployment environment (for example, ECS Secrets Manager injection).
264270
# SLACK_SEARCH_APP_ID=

‎apps/sim/app/workspace/[workspaceId]/knowledge/[id]/[documentId]/document.tsx‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -402,6 +402,9 @@ export function Document({
402402
const DocumentIcon =
403403
ConnectorIcon || getDocumentIcon(documentData?.mimeType ?? '', effectiveDocumentName)
404404
const isCompleted = documentData?.processingStatus === 'completed'
405+
/** A completed image with no chunks held nothing searchable, such as a logo or an icon. */
406+
const isImageWithNothingToIndex =
407+
isCompleted && documentData.mimeType.startsWith('image/') && documentData.chunkCount === 0
405408

406409
/**
407410
* Kept separate from `documentError`: without the document there is no page to draw,
@@ -1284,6 +1287,12 @@ export function Document({
12841287
emptyState={
12851288
chunkError ? (
12861289
<ChunkLoadError message={chunkError} kind={initialError ? 'load' : 'search'} />
1290+
) : isImageWithNothingToIndex ? (
1291+
<EmptyState
1292+
graphic={<FileX className='size-[24px] text-[var(--text-icon)]' />}
1293+
title='Nothing to index'
1294+
description='This image holds no searchable information, such as a logo or an icon.'
1295+
/>
12871296
) : undefined
12881297
}
12891298
selectable={chunkError ? undefined : selectableConfig}

‎apps/sim/app/workspace/[workspaceId]/knowledge/[id]/components/add-connector-modal/add-connector-modal.tsx‎

Lines changed: 16 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -83,6 +83,17 @@ import { useConnectorSetupStore } from '@/stores/connector-setup/store'
8383

8484
const CONNECTOR_ENTRIES = Object.entries(CONNECTOR_META_REGISTRY)
8585

86+
/** The settings a new connector's form starts from; a Search source adds its own defaults. */
87+
function startingSourceConfig(
88+
meta: Pick<ConnectorMeta, 'defaultSourceConfig' | 'searchDefaultSourceConfig'> | null | undefined,
89+
isSearchIndex: boolean | undefined
90+
): Record<string, string> {
91+
return {
92+
...meta?.defaultSourceConfig,
93+
...(isSearchIndex ? meta?.searchDefaultSourceConfig : undefined),
94+
}
95+
}
96+
8697
interface AddConnectorModalProps {
8798
scope?: ResourceScope
8899
open: boolean
@@ -327,9 +338,10 @@ export function AddConnectorModal({
327338
} = useConnectorConfigFields({
328339
connectorConfig,
329340
accessMode: access.accessMode,
330-
initialSourceConfig: isSearchIndex
331-
? { ...connectorConfig?.searchDefaultSourceConfig, ...draft?.sourceConfig }
332-
: draft?.sourceConfig,
341+
initialSourceConfig: {
342+
...startingSourceConfig(connectorConfig, isSearchIndex),
343+
...draft?.sourceConfig,
344+
},
333345
initialCanonicalModes: draft?.canonicalModes,
334346
initialSelectionLabels: draft?.selectionLabels,
335347
})
@@ -475,9 +487,7 @@ export function AddConnectorModal({
475487
if (setupDraftKey) useConnectorSetupStore.getState().clearDraft(setupDraftKey)
476488
gitlabPermissions.reset()
477489
setSelectedType(type)
478-
setSourceConfig(
479-
isSearchIndex ? { ...CONNECTOR_META_REGISTRY[type]?.searchDefaultSourceConfig } : {}
480-
)
490+
setSourceConfig(startingSourceConfig(CONNECTOR_META_REGISTRY[type], isSearchIndex))
481491
setSelectedCredentialId(null)
482492
setContentCredentialId(null)
483493
setAccess({

‎apps/sim/connectors/confluence/attachments.test.ts‎

Lines changed: 102 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -16,8 +16,9 @@ vi.mock('@/lib/knowledge/documents/secure-fetch.server', async (importOriginal)
1616
}))
1717

1818
const CONFIG = { domain: 'example.atlassian.net', spaceKey: 'ENG' }
19+
const IMAGES_ON = { ...CONFIG, imageAttachments: 'semantic' }
1920
const CONTEXT = { cloudId: 'cloud', spaceId: '1' }
20-
const INPUT = { accessToken: 'token', cloudId: 'cloud', domain: CONFIG.domain }
21+
const INPUT = { accessToken: 'token', cloudId: 'cloud', domain: CONFIG.domain, includeImages: true }
2122
const fetchMock = vi.fn<typeof fetch>()
2223

2324
function file(overrides: Record<string, unknown> = {}) {
@@ -84,7 +85,10 @@ function fixture(attachment = file()) {
8485
})
8586
}
8687

87-
async function get(externalId = 'attachment:page:p1:att123', config = CONFIG) {
88+
async function get(
89+
externalId = 'attachment:page:p1:att123',
90+
config: Record<string, unknown> = CONFIG
91+
) {
8892
return confluenceConnector.getDocument('token', config, externalId, { ...CONTEXT })
8993
}
9094

@@ -404,6 +408,102 @@ describe('Confluence attachment listing', () => {
404408
})
405409
})
406410

411+
describe('Confluence image attachments', () => {
412+
async function listOne(attachment: Record<string, unknown>) {
413+
fetchMock.mockResolvedValue(Response.json({ results: [attachment] }))
414+
const result = await listConfluenceAttachments({
415+
...INPUT,
416+
listParents: async () => ({ documents: [parent()], hasMore: false }),
417+
})
418+
return result.documents.slice(1)
419+
}
420+
421+
it.each([
422+
{ title: 'Quarterly schedule.png', mediaType: 'image/png', mimeType: 'image/png' },
423+
{ title: 'image-20240101-000000', mediaType: 'image/jpeg', mimeType: 'image/jpeg' },
424+
{ title: 'Floor plan', mediaType: 'IMAGE/JPG', mimeType: 'image/jpeg' },
425+
{ title: 'Diagram.webp', mediaType: 'image/webp', mimeType: 'image/webp' },
426+
{ title: 'Spinner.gif', mediaType: 'image/gif', mimeType: 'image/gif' },
427+
])('lists $title by its media type', async ({ title, mediaType, mimeType }) => {
428+
const [image] = await listOne(file({ title, mediaType, fileSize: 40_000 }))
429+
expect(image).toMatchObject({ mimeType, contentDeferred: true })
430+
expect(image.contentHash).toBe('confluence:attachment-v1:att123:page:p1:2:image-extract-v1')
431+
})
432+
433+
it.each([
434+
{ title: 'Logo.svg', mediaType: 'image/svg+xml' },
435+
{ title: 'Photo.heic', mediaType: 'image/heic' },
436+
{ title: 'Photo.avif', mediaType: 'image/avif' },
437+
{ title: 'Archive', mediaType: 'application/zip' },
438+
{ title: 'Unknown', mediaType: undefined },
439+
])('drops $title', async ({ title, mediaType }) => {
440+
expect(await listOne(file({ title, mediaType, fileSize: 40_000 }))).toEqual([])
441+
})
442+
443+
it('drops images under the byte floor and keeps one whose size is unreported', async () => {
444+
expect(
445+
await listOne(file({ title: 'Icon.png', mediaType: 'image/png', fileSize: 900 }))
446+
).toEqual([])
447+
const [unsized] = await listOne(
448+
file({ title: 'Chart.png', mediaType: 'image/png', fileSize: undefined })
449+
)
450+
expect(unsized.mimeType).toBe('image/png')
451+
})
452+
453+
it('keeps document hashes unchanged and still lists small documents', async () => {
454+
const [pdf] = await listOne(file({ mediaType: 'application/pdf', fileSize: 12 }))
455+
expect(pdf.contentHash).toBe('confluence:attachment-v1:att123:page:p1:2')
456+
})
457+
458+
it('hydrates a listed image with its bytes and the same media type', async () => {
459+
fixture(file({ title: 'Quarterly schedule', mediaType: 'image/png', fileSize: 40_000 }))
460+
secureDownload.mockResolvedValue(new Response('png bytes'))
461+
const hydrated = await get('attachment:page:p1:att123', IMAGES_ON)
462+
expect(hydrated?.skippedReason).toBeUndefined()
463+
expect(hydrated?.sourceFile).toMatchObject({
464+
fileName: 'Quarterly schedule',
465+
mimeType: 'image/png',
466+
})
467+
})
468+
469+
it('lists no images when the connector turns image attachments off, and keeps documents', async () => {
470+
fetchMock.mockResolvedValue(
471+
Response.json({
472+
results: [
473+
file({ title: 'Quarterly schedule.png', mediaType: 'image/png', fileSize: 40_000 }),
474+
file({ id: 'att124', title: 'Guide.pdf', mediaType: 'application/pdf' }),
475+
],
476+
})
477+
)
478+
const result = await listConfluenceAttachments({
479+
...INPUT,
480+
includeImages: false,
481+
listParents: async () => ({ documents: [parent()], hasMore: false }),
482+
})
483+
expect(result.documents.slice(1).map((doc) => doc.title)).toEqual(['Guide.pdf'])
484+
})
485+
486+
it('replaces an indexed image once image attachments are turned off', async () => {
487+
fixture(file({ title: 'Quarterly schedule', mediaType: 'image/png', fileSize: 40_000 }))
488+
const hydrated = await get('attachment:page:p1:att123', { ...CONFIG, imageAttachments: 'off' })
489+
expect(hydrated?.skippedReason).toBe('Image attachments are turned off for this connector')
490+
expect(hydrated?.skippedExistingDisposition).toBe('replace')
491+
})
492+
493+
it('keeps images off for a connector created before the setting existed', async () => {
494+
fixture(file({ title: 'Quarterly schedule', mediaType: 'image/png', fileSize: 40_000 }))
495+
const hydrated = await get('attachment:page:p1:att123', CONFIG)
496+
expect(hydrated?.skippedReason).toBe('Image attachments are turned off for this connector')
497+
})
498+
499+
it('replaces an attachment that stopped being a supported image', async () => {
500+
fixture(file({ title: 'Logo', mediaType: 'image/svg+xml', fileSize: 40_000 }))
501+
const hydrated = await get()
502+
expect(hydrated?.skippedReason).toContain('supported image')
503+
expect(hydrated?.skippedExistingDisposition).toBe('replace')
504+
})
505+
})
506+
407507
describe('Confluence attachment hydration', () => {
408508
it.each(['/attachments/att123', '/download'])(
409509
'reports missing attachment scope without invalidating the credential at %s',

0 commit comments

Comments
 (0)