Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -123,6 +123,15 @@ OPENCODE_MODEL_ID=big-pickle
# If STT_API_URL is not set, voice messages will get a "not configured" reply.
# STT_API_URL=
# STT_API_KEY=

# Document Extractor API (optional)
# If set, the bot will extract text from PDF, DOCX, PPTX, and other documents
# using an external API when the current model doesn't natively support them.
# The API should accept multipart/form-data POST with a "file" field and
# return JSON with a "text" field containing the extracted content.
# Only DOC_EXTRACTOR_URL is required; API key is optional for self-hosted setups.
# DOC_EXTRACTOR_URL=
# DOC_EXTRACTOR_API_KEY=
# STT_MODEL=
# STT_LANGUAGE=
# STT_NOTE_PROMPT="The following text is transcribed from voice. It may contain homophone or phonetic errors. Infer the intended meaning from context."
Expand Down
16 changes: 16 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -239,6 +239,8 @@ When installed via npm, the configuration wizard handles the initial setup. The
| `STT_MODEL` | STT model name passed to `/audio/transcriptions` | No | `whisper-large-v3-turbo` |
| `STT_LANGUAGE` | Optional language hint (empty = provider auto-detect) | No | — |
| `STT_NOTE_PROMPT` | Optional note prepended to the LLM prompt as `[Note: ...]` for voice transcriptions; empty / `false` / `0` disable it | No | — |
| `DOC_EXTRACTOR_URL` | Document text extraction API URL (enables PDF/DOCX/PPTX extraction) | No | — |
| `DOC_EXTRACTOR_API_KEY` | API key for the document extractor (optional for self-hosted extractors) | No | — |
| `TTS_PROVIDER` | TTS provider: `openai` for OpenAI-compatible APIs, `elevenlabs` for ElevenLabs, or `google` for Google Cloud TTS | No | `openai` |
| `TTS_API_URL` | TTS API base URL for OpenAI-compatible APIs or ElevenLabs | No | — |
| `TTS_API_KEY` | TTS API key for OpenAI-compatible APIs or ElevenLabs | No | — |
Expand Down Expand Up @@ -365,6 +367,20 @@ Supported provider examples (Whisper-compatible):

If STT variables are not set, voice/audio transcription is disabled and the bot will ask you to configure STT.

### Document Text Extraction (Optional)

If `DOC_EXTRACTOR_URL` is set, the bot will extract text from PDF, DOCX, PPTX, and other document files using an external API when the current model does not natively support document input.

The API contract is:

- **Endpoint:** `POST {DOC_EXTRACTOR_URL}`
- **Content-Type:** `multipart/form-data`
- **Field:** `file` — the document binary
- **Authorization:** `Bearer {DOC_EXTRACTOR_API_KEY}` (only sent when a key is configured)
- **Response:** JSON `{ "text": "extracted content..." }`

If the extractor is not configured and the model doesn't support documents, the bot replies with a notice and forwards only the caption text.

### Model Configuration

The model picker uses OpenCode local model state (`favorite` + `recent`):
Expand Down
72 changes: 72 additions & 0 deletions src/app/services/document-extractor-service.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
import { config } from "../../config.js";
import { logger } from "../../utils/logger.js";

const REQUEST_TIMEOUT_MS = 60_000;

export interface DocExtractorResult {
text: string;
}

export function isDocExtractorConfigured(): boolean {
return Boolean(config.docExtractor.apiUrl);
}

export async function extractDocument(
fileBuffer: Buffer,
mimeType: string,
filename: string,
): Promise<DocExtractorResult> {
if (!isDocExtractorConfigured()) {
throw new Error(
"Document extractor is not configured: DOC_EXTRACTOR_URL is required",
);
}

const url = config.docExtractor.apiUrl!;

const formData = new FormData();
formData.append("file", new Blob([new Uint8Array(fileBuffer)], { type: mimeType }), filename);

logger.debug(
`[DocExtractor] Sending extraction request: url=${url}, mime=${mimeType}, filename=${filename}, size=${fileBuffer.length} bytes`,
);

const controller = new AbortController();
const timeout = setTimeout(() => controller.abort(), REQUEST_TIMEOUT_MS);

try {
const headers: Record<string, string> = {};
if (config.docExtractor.apiKey) {
headers["Authorization"] = `Bearer ${config.docExtractor.apiKey}`;
}

const response = await fetch(url, {
method: "POST",
headers,
body: formData,
signal: controller.signal,
});

if (!response.ok) {
const errorBody = await response.text().catch(() => "");
throw new Error(
`Document extractor API returned HTTP ${response.status}: ${errorBody || response.statusText}`,
);
}

const data = (await response.json()) as { text?: string };

if (typeof data.text !== "string") {
throw new Error("Document extractor API response does not contain a text field");
}

return { text: data.text };
} catch (err) {
if (err instanceof DOMException && err.name === "AbortError") {
throw new Error(`Document extractor request timed out after ${REQUEST_TIMEOUT_MS}ms`);
}
throw err;
} finally {
clearTimeout(timeout);
}
}
56 changes: 47 additions & 9 deletions src/bot/handlers/document-handler.ts
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ import {
isTextMimeType,
isFileSizeAllowed,
} from "../../app/services/file-download-service.js";
import { isDocExtractorConfigured, extractDocument } from "../../app/services/document-extractor-service.js";
import { getModelCapabilities, supportsInput } from "../../app/services/model-capabilities-service.js";
import { getStoredModel } from "../../app/services/model-selection-service.js";
import { logger } from "../../utils/logger.js";
Expand Down Expand Up @@ -112,18 +113,55 @@ export async function handleDocumentMessage(
return;
}

if (mimeType === "application/pdf") {
const DOCUMENT_MIME_TYPES = [
"application/pdf",
"application/msword",
"application/vnd.openxmlformats-officedocument.wordprocessingml.document",
"application/vnd.ms-powerpoint",
"application/vnd.openxmlformats-officedocument.presentationml.presentation",
"application/vnd.ms-excel",
"application/vnd.openxmlformats-officedocument.spreadsheetml.sheet",
"application/vnd.oasis.opendocument.text",
"application/vnd.oasis.opendocument.presentation",
"application/vnd.oasis.opendocument.spreadsheet",
"text/rtf",
];

if (DOCUMENT_MIME_TYPES.includes(mimeType)) {
const storedModel = getStored();
const capabilities = await getCapabilities(storedModel.providerID, storedModel.modelID);

if (!supportsInput(capabilities, "pdf")) {
logger.warn(
`[Document] Model ${storedModel.providerID}/${storedModel.modelID} doesn't support PDF input`,
);
await ctx.reply(t("bot.model_no_pdf"));

if (caption.trim().length > 0) {
await processPrompt(ctx, caption, deps);
if (isDocExtractorConfigured()) {
logger.warn(
`[Document] Model doesn't support PDF input, delegating document to DOC_EXTRACTOR_URL`,
);
await ctx.reply(t("bot.file_downloading"));
const downloadedFile = await downloadFile(ctx.api, doc.file_id);

try {
const result = await extractDocument(downloadedFile.buffer, mimeType, filename);
const promptWithFile = `--- Content of ${filename} ---\n${result.text}\n--- End of file ---\n\n${caption}`;
logger.info(
`[Document] Sending extracted document text from ${filename} (${result.text.length} chars) as prompt`,
);
await processPrompt(ctx, promptWithFile, deps);
} catch (extractErr) {
const errMsg = extractErr instanceof Error ? extractErr.message : String(extractErr);
logger.error(`[Document] Document extraction failed: ${errMsg}`);
await ctx.reply(t("bot.document_extraction_error"));
if (caption.trim().length > 0) {
await processPrompt(ctx, caption, deps);
}
}
} else {
logger.warn(
`[Document] Model doesn't support PDF input and DOC_EXTRACTOR_URL is not configured`,
);
await ctx.reply(t("bot.model_no_pdf"));
if (caption.trim().length > 0) {
await processPrompt(ctx, caption, deps);
}
}
return;
}
Expand All @@ -141,7 +179,7 @@ export async function handleDocumentMessage(
};

logger.info(
`[Document] Sending PDF (${downloadedFile.buffer.length} bytes, ${filename}) with prompt`,
`[Document] Sending document (${downloadedFile.buffer.length} bytes, ${filename}, ${mimeType}) with prompt`,
);

await processPrompt(ctx, caption, deps, [filePart]);
Expand Down
4 changes: 4 additions & 0 deletions src/config.ts
Original file line number Diff line number Diff line change
Expand Up @@ -181,6 +181,10 @@ export const config = {
language: getEnvVar("STT_LANGUAGE", false),
notePrompt: getEnvVar("STT_NOTE_PROMPT", false),
},
docExtractor: {
apiUrl: getEnvVar("DOC_EXTRACTOR_URL", false),
apiKey: getEnvVar("DOC_EXTRACTOR_API_KEY", false),
},
tts: (() => {
const provider = getOptionalTtsProviderEnvVar("TTS_PROVIDER", "openai");
const defaultVoice =
Expand Down
3 changes: 2 additions & 1 deletion src/i18n/ar.ts
Original file line number Diff line number Diff line change
Expand Up @@ -106,11 +106,12 @@ export const ar: I18nDictionary = {
"bot.file_too_large": "⚠️ حجم الملف أكبر من الحد المسموح ({maxSizeMb}MB)",
"bot.file_download_error": "🔴 تعذر تنزيل الملف",
"bot.file_type_unsupported":
"⚠️ نوع الملف غير مدعوم. أرسل صورة أو ملف PDF أو ملفًا نصيًا أو برمجيًا.",
"⚠️ نوع الملف غير مدعوم. أرسل صورة أو مستندًا (PDF، DOCX، PPTX) أو ملفًا نصيًا أو برمجيًا.",
"bot.media_group_not_processed":
"⚠️ تعذر معالجة ملف أو أكثر في هذه المجموعة. لم يتم إرسال أي ملف إلى OpenCode.",
"bot.media_group_download_error": "🔴 تعذر تنزيل أحد الملفات. لم يتم إرسال أي ملف إلى OpenCode.",
"bot.model_no_pdf": "⚠️ النموذج الحالي لا يدعم ملفات PDF. سيتم إرسال النص فقط.",
"bot.document_extraction_error": "🔴 فشل استخراج نص المستند.",
"bot.text_file_too_large": "⚠️ حجم الملف النصي أكبر من الحد المسموح ({maxSizeKb}KB)",

"status.header_running": "🟢 خادم OpenCode يعمل",
Expand Down
3 changes: 2 additions & 1 deletion src/i18n/de.ts
Original file line number Diff line number Diff line change
Expand Up @@ -110,12 +110,13 @@ export const de: I18nDictionary = {
"bot.file_too_large": "⚠️ Datei ist zu groß (max. {maxSizeMb}MB)",
"bot.file_download_error": "🔴 Datei konnte nicht heruntergeladen werden",
"bot.file_type_unsupported":
"⚠️ Dieser Dateityp wird nicht unterstützt. Sende ein Bild, PDF oder eine Text-/Code-Datei.",
"⚠️ Dieser Dateityp wird nicht unterstützt. Sende ein Bild, Dokument (PDF, DOCX, PPTX) oder eine Text-/Code-Datei.",
"bot.media_group_not_processed":
"⚠️ Eine oder mehrere Dateien in diesem Album können nicht verarbeitet werden. Es wurde nichts an OpenCode gesendet.",
"bot.media_group_download_error":
"🔴 Eine der Dateien konnte nicht heruntergeladen werden. Es wurde nichts an OpenCode gesendet.",
"bot.model_no_pdf": "⚠️ Das aktuelle Modell unterstützt keine PDF-Eingabe. Sende nur Text.",
"bot.document_extraction_error": "🔴 Dokumenttext konnte nicht extrahiert werden.",
"bot.text_file_too_large": "⚠️ Textdatei ist zu groß (max. {maxSizeKb}KB)",

"status.header_running": "🟢 OpenCode-Server läuft",
Expand Down
3 changes: 2 additions & 1 deletion src/i18n/en.ts
Original file line number Diff line number Diff line change
Expand Up @@ -103,12 +103,13 @@ export const en = {
"bot.file_too_large": "⚠️ File is too large (max {maxSizeMb}MB)",
"bot.file_download_error": "🔴 Failed to download file",
"bot.file_type_unsupported":
"⚠️ This file type is not supported. Send an image, PDF, or text/code file.",
"⚠️ This file type is not supported. Send an image, document (PDF, DOCX, PPTX), or text/code file.",
"bot.media_group_not_processed":
"⚠️ One or more files in this album cannot be processed. Nothing was sent to OpenCode.",
"bot.media_group_download_error":
"🔴 Failed to download one of the files. Nothing was sent to OpenCode.",
"bot.model_no_pdf": "⚠️ Current model doesn't support PDF input. Sending text only.",
"bot.document_extraction_error": "🔴 Failed to extract document text.",
"bot.text_file_too_large": "⚠️ Text file is too large (max {maxSizeKb}KB)",

"status.header_running": "🟢 OpenCode Server is running",
Expand Down
3 changes: 2 additions & 1 deletion src/i18n/es.ts
Original file line number Diff line number Diff line change
Expand Up @@ -111,12 +111,13 @@ export const es: I18nDictionary = {
"bot.file_too_large": "⚠️ El archivo es demasiado grande (max {maxSizeMb}MB)",
"bot.file_download_error": "🔴 No se pudo descargar el archivo",
"bot.file_type_unsupported":
"⚠️ Este tipo de archivo no es compatible. Envía una imagen, PDF o archivo de texto/código.",
"⚠️ Este tipo de archivo no es compatible. Envía una imagen, documento (PDF, DOCX, PPTX) o archivo de texto/código.",
"bot.media_group_not_processed":
"⚠️ Uno o más archivos de este álbum no se pueden procesar. No se envió nada a OpenCode.",
"bot.media_group_download_error":
"🔴 No se pudo descargar uno de los archivos. No se envió nada a OpenCode.",
"bot.model_no_pdf": "⚠️ El modelo actual no admite entrada PDF. Enviaré solo texto.",
"bot.document_extraction_error": "🔴 No se pudo extraer el texto del documento.",
"bot.text_file_too_large": "⚠️ El archivo de texto es demasiado grande (max {maxSizeKb}KB)",

"status.header_running": "🟢 OpenCode Server está en ejecución",
Expand Down
4 changes: 3 additions & 1 deletion src/i18n/fr.ts
Original file line number Diff line number Diff line change
Expand Up @@ -111,13 +111,15 @@ export const fr: I18nDictionary = {
"bot.file_too_large": "⚠️ Le fichier est trop volumineux (max {maxSizeMb}MB)",
"bot.file_download_error": "🔴 Impossible de télécharger le fichier",
"bot.file_type_unsupported":
"⚠️ Ce type de fichier n'est pas pris en charge. Envoyez une image, un PDF ou un fichier texte/code.",
"⚠️ Ce type de fichier n'est pas pris en charge. Envoyez une image, un document (PDF, DOCX, PPTX) ou un fichier texte/code.",
"bot.media_group_not_processed":
"⚠️ Un ou plusieurs fichiers de cet album ne peuvent pas être traités. Rien n'a été envoyé à OpenCode.",
"bot.media_group_download_error":
"🔴 Impossible de télécharger l'un des fichiers. Rien n'a été envoyé à OpenCode.",
"bot.model_no_pdf":
"⚠️ Le modèle actuel ne prend pas en charge les PDF. Envoi du texte uniquement.",
"bot.document_extraction_error":
"🔴 Échec de l'extraction du texte du document.",
"bot.text_file_too_large": "⚠️ Le fichier texte est trop volumineux (max {maxSizeKb}KB)",

"status.header_running": "🟢 Le serveur OpenCode est en cours d'exécution",
Expand Down
3 changes: 2 additions & 1 deletion src/i18n/ru.ts
Original file line number Diff line number Diff line change
Expand Up @@ -103,12 +103,13 @@ export const ru: I18nDictionary = {
"bot.file_too_large": "⚠️ Файл слишком большой (макс. {maxSizeMb}МБ)",
"bot.file_download_error": "🔴 Не удалось скачать файл",
"bot.file_type_unsupported":
"⚠️ Этот тип файла не поддерживается. Отправьте изображение, PDF или текстовый/кодовый файл.",
"⚠️ Этот тип файла не поддерживается. Отправьте изображение, документ (PDF, DOCX, PPTX) или текстовый/кодовый файл.",
"bot.media_group_not_processed":
"⚠️ Один или несколько файлов в альбоме нельзя обработать. В OpenCode ничего не отправлено.",
"bot.media_group_download_error":
"🔴 Не удалось скачать один из файлов. В OpenCode ничего не отправлено.",
"bot.model_no_pdf": "⚠️ Текущая модель не поддерживает PDF. Отправляю только текст.",
"bot.document_extraction_error": "🔴 Не удалось извлечь текст из документа.",
"bot.text_file_too_large": "⚠️ Текстовый файл слишком большой (макс. {maxSizeKb}КБ)",

"status.header_running": "🟢 OpenCode Server запущен",
Expand Down
4 changes: 3 additions & 1 deletion src/i18n/zh.ts
Original file line number Diff line number Diff line change
Expand Up @@ -93,11 +93,13 @@ export const zh: I18nDictionary = {
"bot.files_downloading": "⏳ 正在下载文件...",
"bot.file_too_large": "⚠️ 文件过大(最大 {maxSizeMb}MB)",
"bot.file_download_error": "🔴 下载文件失败",
"bot.file_type_unsupported": "⚠️ 不支持此文件类型。请发送图片、PDF 或文本/代码文件。",
"bot.file_type_unsupported":
"⚠️ 不支持此文件类型。请发送图片、文档(PDF、DOCX、PPTX)或文本/代码文件。",
"bot.media_group_not_processed":
"⚠️ 此相册中有一个或多个文件无法处理。未向 OpenCode 发送任何内容。",
"bot.media_group_download_error": "🔴 无法下载其中一个文件。未向 OpenCode 发送任何内容。",
"bot.model_no_pdf": "⚠️ 当前模型不支持PDF输入。将仅发送文本。",
"bot.document_extraction_error": "🔴 文档文本提取失败。",
"bot.text_file_too_large": "⚠️ 文本文件过大(最大 {maxSizeKb}KB)",

"status.header_running": "🟢 OpenCode 服务器正在运行",
Expand Down
Loading