Skip to content

feat(tui): prepend ReadMediaFile hint when video attachments are present #1778

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
bj456736 wants to merge 1 commit into
base: main
Choose a base branch
from
+33 −6
Open

feat(tui): prepend ReadMediaFile hint when video attachments are present #1778

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions apps/kimi-code/src/tui/utils/image-placeholder.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,17 @@ export function extractMediaAttachments(
const tail = text.slice(cursor);
pushText(parts, tail);

// When video attachments are present, prepend a prompt hint so the model
// knows to use ReadMediaFile rather than writing Python frame-extraction scripts.
if (videoAttachmentIds.length > 0) {
parts.unshift({
type: 'text',
text:
'Use the ReadMediaFile tool to read and analyze the attached video directly. ' +
'Do not write Python scripts or Bash commands to extract frames.',
Comment on lines +95 to +99

@chatgpt-codex-connector chatgpt-codex-connector Bot Jul 16, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Exclude injected video hints from prompt metadata

When a normal prompt contains a video, this prepends the internal tool hint as the first text part sent to session.prompt. The session metadata builders (promptMetadataTextFromPayload / promptMetadataTextFromContentParts) derive lastPrompt and the initial title from every text part and only strip image-compression captions, so a new session started with summarize [video] will be titled with this ReadMediaFile instruction and expose it in last_prompt instead of the user's prompt. Please route this hint through a hidden/system-reminder path or strip it from metadata/history projection like compression captions.

Useful? React with 👍 / 👎.

});
}

return {
// Text-only submissions drop the synthesised parts array — the
// caller's contract is "parts is meaningful iff hasMedia", and
Expand Down
28 changes: 22 additions & 6 deletions apps/kimi-code/test/tui/input/image-placeholder.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -103,12 +103,19 @@ describe('extractMediaAttachments', () => {
const r = extractMediaAttachments(text, store);
expect(r.imageAttachmentIds).toEqual([1]);
expect(r.videoAttachmentIds).toEqual([2]);
expect(r.parts[0]).toEqual({ type: 'text', text: 'first ' });
expect(r.parts[1]).toEqual({
// First part is the video hint (because video is present)
expect(r.parts[0]).toEqual({
type: 'text',
text:
'Use the ReadMediaFile tool to read and analyze the attached video directly. ' +
'Do not write Python scripts or Bash commands to extract frames.',
});
expect(r.parts[1]).toEqual({ type: 'text', text: 'first ' });
expect(r.parts[2]).toEqual({
type: 'image_url',
imageUrl: { url: 'data:image/png;base64,AQ==' },
});
const cachePath = videoPathFromParts(r.parts);
const cachePath = videoPathFromParts(r.parts.slice(3));
expect(cachePath.startsWith(getCacheDir())).toBe(true);
expect(readFileSync(cachePath, 'utf8')).toBe('video-bytes');
} finally {
Expand Down Expand Up @@ -145,8 +152,9 @@ describe('extractMediaAttachments', () => {
// The filename drives the cache label; `&` must be escaped in the attribute.
const att = store.addVideo('video/mp4', srcVideo, 'a&b.mp4');
const r = extractMediaAttachments(att.placeholder, store);
expect(r.parts).toHaveLength(1);
const text = (r.parts[0] as TextPart).text;
expect(r.parts).toHaveLength(2);
// First part is the video hint; second part is the actual tag.
const text = (r.parts[1] as TextPart).text;
expect(text).toMatch(/<video path="[^"]+a&amp;b\.mp4"><\/video>/);
} finally {
cleanup();
Expand All @@ -165,7 +173,15 @@ describe('extractMediaAttachments', () => {
const r = extractMediaAttachments(att.placeholder, store);
expect(r.hasMedia).toBe(true);
expect(r.videoAttachmentIds).toEqual([1]);
const cachePath = videoPathFromParts(r.parts);
expect(r.parts).toHaveLength(2);
// First part is the video hint
expect(r.parts[0]).toEqual({
type: 'text',
text:
'Use the ReadMediaFile tool to read and analyze the attached video directly. ' +
'Do not write Python scripts or Bash commands to extract frames.',
});
const cachePath = videoPathFromParts(r.parts.slice(1));
// The tag points at the cache, not the original source path.
expect(cachePath.startsWith(getCacheDir())).toBe(true);
expect(cachePath).not.toBe(srcVideo);
Expand Down
Loading