.NET: AgentFrameworkResponseHandler drops AI Search annotations (url_citation) from streamed responses

[graemefoster]

Description

When using FoundryAITool.CreateAzureAISearchTool() with a hosted agent via MapFoundryResponses, the response.output_text.annotation.added events are not emitted in the SSE stream to the client. The text content correctly contains citation markers (e.g. 【5:0†source】), but the annotation metadata (URL, title, start/end indices) is lost.

Expected Behaviour

When the underlying model returns annotations (e.g. url_citation from Azure AI Search grounding), the OutputConverter should emit response.output_text.annotation.added events in the SSE stream, matching what the Foundry Responses API returns when called directly.

Actual Behaviour

• The text delta events contain citation markers like 【5:0†source】
• The annotations array in response.content_part.done and response.output_item.done events is always empty: "annotations":[]
• No response.output_text.annotation.added events are emitted

Root Cause Analysis

OutputConverter.ConvertUpdatesToEventsAsync processes AgentResponseUpdate.Contents and handles TextContent, FunctionCallContent, TextReasoningContent, etc. — but has no case for annotation content.

When calling the Foundry /responses endpoint directly (bypassing the agent framework), response.output_text.annotation.added events ARE returned with full annotation metadata:

{
  "type": "response.output_text.annotation.added",
  "annotation": {
    "type": "url_citation",
    "end_index": 341,
    "start_index": 329,
    "title": "MO_Policy_KM_Agent_v3.2.docx",
    "url": "https://search-....search.windows.net/"
  },
  "annotation_index": 2,
  "content_index": 0,
  "item_id": "msg_...",
  "output_index": 2,
  "sequence_number": 72
}

But these are not present in the SSE output streamed from the agent to the web.

Reproduction

1. Create a hosted agent with FoundryAITool.CreateAzureAISearchTool()
2. Register via builder.Services.AddFoundryResponses(agent)
3. Send a query that triggers AI Search grounding
4. Observe that the SSE stream contains citation markers in text but no annotation events

Environment

• Microsoft.Agents.AI.Foundry.Hosting v1.10.0-preview.260610.1
• Azure.AI.Projects v2.1.0-beta.3
• .NET 10