[yuyixg]

Stacked on #1459. Until that merges this PR also lists its four commits; the changes here are the last two
(3e7973b0a, 644ef6944), touching only BotSharp.Plugin.Twilio.

Why

An inbound phone agent that creates work orders needs the same streamed ConversationRelay experience #1459 gives outbound
calls, and callers should not sit in silence while the model thinks or a slow tool (a ~13s create-work-order API) runs.

What

Inbound calls on ConversationRelay (TwilioInboundController)

• Agents listed in ConversationRelay.AgentIds now get ConversationRelay TwiML on twilio/inbound, instead of the Gather flow.
• ConversationRelay speaks text only, so the prompts session hooks queue in SpeechPaths become the welcome greeting through
  their Twilio:PromptTexts text (TwilioService.GetPromptText).
• Calls that arrive with an Intent keep the Gather flow, which already has their first message queued.
• Note: with Enabled: true and an empty AgentIds, inbound calls of every agent switch to ConversationRelay.

Response watchdog (ConversationRelaySession, off by default)

• ResponseWatchdogIntervals, e.g. [1.5, 3, 5]: seconds of silence before each filler while the agent works on a reply.
  The first counts from when the caller finished, each next one from when whatever was said last (a filler, a tool
  indication, a streamed reply) ends; the last value repeats until the reply comes.
• The first filler comes from ResponseWatchdogPrompts (default hold-on-1..5), later ones from
  ResponseWatchdogLongPrompts (default hold-on-long-1..10); no sentence repeats in a turn while unused ones remain.
• ResponseWatchdogAgentIds limits it to some agents.
• A filler is decided while holding the send lock, so it never lands inside a streamed reply, and nothing is said once
  the turn's reply is in.

Testing

• Unit tests (in the onebrain host, driving ConversationRelaySession over a fake WebSocket): slow reply gets one filler,
  fast reply none, unlisted agent none, an indication keeps it quiet, a filler never splits a streamed reply, and a long
  wait gets a different filler at each interval. Repeated runs are stable.
• End to end locally: signed twilio/inbound plus the relay WebSocket in text against a dev environment: greeting from
  the prompt text, fillers at the configured intervals with different sentences, and the expected handoff on transfer.

🤖 Generated with Claude Code

[qodo-code-review]

Code Review by Qodo

Sorry, something went wrong

We weren't able to complete the code review on our side. Please try again manually by commenting /agentic_review on this PR.

[qodo-code-review]

PR Summary by Qodo

Add Twilio ConversationRelay calls and a response watchdog

✨ Enhancement 🐞 Bug fix ⚙️ Configuration changes 🕐 40+ Minutes

AI Description

• Route eligible inbound and outbound calls through ConversationRelay for text-based, optionally
 streamed conversations.
• Add an opt-in response watchdog to speak varied fillers during slow agent turns.
• Support TwiML Say for generated speech and bound OpenAI speech-synthesis request time.

Diagram

sequenceDiagram
    actor Caller
    participant Twilio
    participant Entry as Call webhook
    participant Service as Twilio service
    participant Socket as Relay middleware
    participant Session as Relay session
    participant Agent as Conversation processor
    participant Action as Action webhook
    Caller->>Twilio: Starts call
    Twilio->>Entry: Requests instructions
    Entry->>Service: Selects relay for eligible agent
    Service-->>Twilio: Connect ConversationRelay
    Twilio->>Socket: Opens WebSocket
    Socket->>Session: Starts call session
    Twilio->>Session: Sends caller text
    Session->>Agent: Processes caller turn
    Agent-->>Session: Streams reply and indications
    Session->>Session: Watches for response silence
    Session-->>Twilio: Sends speech text or handoff
    Twilio->>Action: Reports session end
    Action-->>Twilio: Hangs up, transfers, or reconnects

Loading

High-Level Assessment

The following are alternative approaches to this PR:

1. Extend the Gather webhook flow

• ➕ Reuses the established call flow and its session storage.
• ➕ Avoids a new long-lived WebSocket session.
• ➖ Cannot deliver the same low-latency streamed text interaction.
• ➖ Repeated webhooks complicate silence-aware filler timing.

Recommendation: Keep ConversationRelay as an opt-in path and retain Gather for intent-bearing and non-eligible calls. Extracting a shared conversation processor limits duplicated agent logic; extending Gather alone would not meet the streaming goal. Review WebSocket middleware registration in the host and the concurrency boundaries carefully, since neither test files nor middleware registration are shown in this diff.

Files changed (22) +1575 / -146

Enhancement (16) +1276 / -62

ProviderHelper.csAllow optional OpenAI client timeout and retry policy +17/-3

Allow optional OpenAI client timeout and retry policy

• Accepts network timeout and retry-count overrides when creating an OpenAI client while leaving other callers on SDK defaults.

src/Plugins/BotSharp.Plugin.OpenAI/Providers/ProviderHelper.cs

TwilioConversationRelayController.csHandle ConversationRelay end callbacks +89/-0

Handle ConversationRelay end callbacks

• Authenticates session-end callbacks and returns TwiML to transfer, retry a failed relay session within limits, or hang up. Emits the corresponding session hooks for transfer and requested hang-up.

src/Plugins/BotSharp.Plugin.Twilio/Controllers/TwilioConversationRelayController.cs

TwilioInboundController.csRoute eligible inbound calls to ConversationRelay +16/-2

Route eligible inbound calls to ConversationRelay

• Uses queued prompt text as the relay greeting for eligible calls without an intent, preserving the existing realtime and intent-bearing paths. Plays or says voicemail according to its saved format.

src/Plugins/BotSharp.Plugin.Twilio/Controllers/TwilioInboundController.cs

TwilioOutboundController.csRoute eligible outbound calls to ConversationRelay +14/-2

Route eligible outbound calls to ConversationRelay

• Uses the stored initial assistant reply as the relay welcome greeting. Updates voicemail playback to support saved text as well as audio.

src/Plugins/BotSharp.Plugin.Twilio/Controllers/TwilioOutboundController.cs

TwilioReconnectController.csReconnect using format-aware speech output +1/-2

Reconnect using format-aware speech output

• Appends the saved initial speech as Say or Play rather than always playing an audio URL.

src/Plugins/BotSharp.Plugin.Twilio/Controllers/TwilioReconnectController.cs

AgentTurnCallbacks.csDefine agent-turn speech callbacks +24/-0

Define agent-turn speech callbacks

• Introduces callbacks for tool indications, streamed reply text, and the start of a new streamed response.

src/Plugins/BotSharp.Plugin.Twilio/Models/AgentTurnCallbacks.cs

ConversationRelayActionRequest.csModel relay termination callbacks +33/-0

Model relay termination callbacks

• Binds Twilio's session status, handoff, error, and reconnect fields for the relay action endpoint.

src/Plugins/BotSharp.Plugin.Twilio/Models/Relay/ConversationRelayActionRequest.cs

ConversationRelayHandoff.csDefine relay handoff actions +10/-0

Define relay handoff actions

• Names the hang-up and transfer actions exchanged between the relay session and its action callback.

src/Plugins/BotSharp.Plugin.Twilio/Models/Relay/ConversationRelayHandoff.cs

ConversationRelayParameters.csName the welcome-greeting length parameter +9/-0

Name the welcome-greeting length parameter

• Defines the custom TwiML parameter used to pass greeting length into relay setup for silence timing.

src/Plugins/BotSharp.Plugin.Twilio/Models/Relay/ConversationRelayParameters.cs

HangupPhoneCallFn.csSave closing speech in the configured format +3/-6

Save closing speech in the configured format

• Uses the Twilio speech helper for the optional ending message instead of always synthesizing an MP3.

src/Plugins/BotSharp.Plugin.Twilio/OutboundPhoneCallHandler/Functions/HangupPhoneCallFn.cs

LeaveVoicemailFn.csSave voicemail in the configured format +3/-6

Save voicemail in the configured format

• Delegates voicemail speech storage to the shared Twilio helper, supporting either saved text or synthesized audio.

src/Plugins/BotSharp.Plugin.Twilio/OutboundPhoneCallHandler/Functions/LeaveVoicemailFn.cs

OutboundPhoneCallFn.csSkip greeting synthesis for relay outbound calls +12/-12

Skip greeting synthesis for relay outbound calls

• Checks agent eligibility before placing the call, allowing ConversationRelay to speak the stored initial message. Non-relay calls save initial speech through the format-aware helper.

src/Plugins/BotSharp.Plugin.Twilio/OutboundPhoneCallHandler/Functions/OutboundPhoneCallFn.cs

TransferPhoneCallFn.csSave transfer speech in the configured format +3/-5

Save transfer speech in the configured format

• Uses the shared speech helper for the optional transfer message instead of always creating an MP3.

src/Plugins/BotSharp.Plugin.Twilio/OutboundPhoneCallHandler/Functions/TransferPhoneCallFn.cs

ConversationRelaySession.csManage relay turns, streaming, and silence watchdogs +765/-0

Manage relay turns, streaming, and silence watchdogs

• Processes WebSocket prompts and keypad input, serializes agent turns and speech sends, streams replies, and handles interruptions and call endings. Adds idle-call handling and an opt-in response watchdog that chooses nonrepeating fillers during slow turns.

src/Plugins/BotSharp.Plugin.Twilio/Services/ConversationRelaySession.cs

TwilioService.csBuild relay TwiML and support text-based speech +190/-24

Build relay TwiML and support text-based speech

• Builds ConversationRelay connection instructions and URLs, including welcome-greeting timing metadata. Centralizes saving and rendering speech as TwiML Say or Play and resolves configured text for prerecorded prompts.

src/Plugins/BotSharp.Plugin.Twilio/Services/TwilioService.cs

TwilioConversationRelayMiddleware.csAccept and validate relay WebSocket connections +87/-0

Accept and validate relay WebSocket connections

• Parses relay URL identifiers, optionally validates Twilio's handshake signature, and starts a session on the accepted socket. Creates a fresh service scope for each caller turn.

src/Plugins/BotSharp.Plugin.Twilio/TwilioConversationRelayMiddleware.cs

Bug fix (1) +12 / -1

AudioSynthesisProvider.csBound speech-synthesis attempts +12/-1

Bound speech-synthesis attempts

• Uses a 30-second per-attempt timeout and three retries by default, with per-model overrides, to limit stalled speech-generation calls.

src/Plugins/BotSharp.Plugin.OpenAI/Providers/Audio/AudioSynthesisProvider.cs

Refactor (2) +148 / -83

TwilioConversationProcessor.csShare agent-turn processing across phone flows +136/-0

Share agent-turn processing across phone flows

• Extracts conversation initialization, postback handling, and agent messaging for both queued webhook and relay calls. Exposes indication and optional streaming callbacks to the calling flow.

src/Plugins/BotSharp.Plugin.Twilio/Services/TwilioConversationProcessor.cs

TwilioMessageQueueService.csUse shared processing in the queued call flow +12/-83

Use shared processing in the queued call flow

• Replaces duplicated conversation processing with the shared processor while retaining queued indications, reply audio generation, hints, and session storage. Uses the format-aware speech helper for replies.

src/Plugins/BotSharp.Plugin.Twilio/Services/TwilioMessageQueueService.cs

Other (3) +139 / -0

LlmModelSetting.csAdd per-model audio synthesis settings +17/-0

Add per-model audio synthesis settings

• Adds optional timeout and retry settings under a model's audio synthesis configuration.

src/Infrastructure/BotSharp.Abstraction/MLTasks/Settings/LlmModelSetting.cs

ConversationRelaySetting.csConfigure relay eligibility and response timing +104/-0

Configure relay eligibility and response timing

• Adds agent selection, transcription and speech options, streaming, idle timeouts, reconnect limits, and signature validation. Configures opt-in watchdog intervals, eligible agents, and first-versus-later filler prompt lists.

src/Plugins/BotSharp.Plugin.Twilio/Settings/ConversationRelaySetting.cs

TwilioSetting.csExpose Say and ConversationRelay options +18/-0

Expose Say and ConversationRelay options

• Adds the Say mode switch, voice and prompt-text mappings, and nested ConversationRelay settings.

src/Plugins/BotSharp.Plugin.Twilio/Settings/TwilioSetting.cs