[RFC] Agentic Chat for OpenSearch Dashboards

[joshuali925]

Summary

This RFC proposes an agent-primary observability experience for OpenSearch Dashboards. A user states a goal in chat ("why did service X error rates spike?"); the agent investigates, streams its reasoning, and answers with interactive MCP App widgets the user can verify inline: a service map, a trace timeline, a firing-alerts view, a metric chart. When the user wants the familiar OpenSearch Dashboards app, the agent opens the relevant Dashboards page into a canvas beside the conversation. Chat and canvas together form a session the plugin persists, so the user starts an investigation from a question instead of hunting through pages and menus. This experience is observability-focused and available only in observability workspaces; every other workspace keeps the existing dashboards-assistant.

A category of commercial "AI SRE" products has emerged around the same thesis: the agent is the first line of response, it hands the engineer an evidence-backed answer instead of a raw alert, and it retains past incidents so its answers improve over time. This RFC delivers the open, verifiable interaction surface that experience needs, built on ml-commons AG-UI streaming, agentic memory, and MCP Apps rather than a proprietary stack, and it sets up the longer arc toward answer-first, alert-triggered on-call.

Motivation

Agentic UI as the primary experience

The existing dashboards-assistant renders as a flyout, dock, or sidecar panel. Its model keeps the page-based Dashboards UI primary: the user drives navigation through pages and menus, and chat sits beside the current page to assist with it. The assistant is a companion to a human-driven workflow.

This RFC proposes inverting that model so the agentic surface is primary. The user states a goal in chat; the agent investigates, renders MCP App widgets inline, and opens the relevant Dashboards pages into a canvas the agent drives on request. Pages become embedded tools the agent and user share, rather than the top-level surface the user navigates by hand. The chat and canvas together are the workspace.

Aligning with the emerging AI SRE category

A category of commercial "AI SRE" products has emerged that this proposal should be read against. They share a consistent thesis, and it sharpens both what we are building and where it is headed:

• The agent is the first line of response, not a companion to a human at a keyboard. A firing alert triggers the investigation, and the agent hands the engineer an answer (root cause, blast radius, supporting evidence, a suggested fix) instead of a raw alert to start digging from. "On-call starts from an answer, not from an alert" is the recurring promise.
• Investigation is evidence-backed and verifiable. The agent doesn't just assert a conclusion; it shows the signals it correlated (traces, logs, metrics, topology) so an engineer can confirm the reasoning rather than trust it blindly.
• Memory accumulates. The agent retains investigations, system topology, and human corrections, so it doesn't re-investigate the same incident from scratch and its answers improve over time. Captured tribal knowledge is a first-class asset.
• Deduplication gates cost and noise. Grouping alerts into unique incidents runs the expensive investigation loop once per real problem instead of once per duplicate, the primary lever on both LLM spend and human interrupt load.
• Human-in-the-loop with a safety gradient. Read/investigate is autonomous and safe by default; write actions and remediation are opt-in and gated. Engineers direct the agent and take the consequential actions.
• Proactive as well as reactive. Agents run operational workflows on a schedule or on trigger, not only in response to a user's question.
• Operator-controlled models and integrations. Bring-your-own LLM, and a wide, pluggable set of data sources and destinations reached over MCP, HTTP APIs, and reusable skills.

This RFC delivers the interaction surface that experience needs, on open primitives (ml-commons AG-UI streaming, agentic memory, MCP Apps) rather than a proprietary stack. The mapping is direct: streamed tool calls plus inline MCP App widgets are the "show your work" surface the category relies on, so the user confirms the agent's reasoning inline instead of trusting a text conclusion, and the canvas opens the real Dashboards page behind any widget for deeper verification; the agentic memory container is the substrate for investigations that build on prior work rather than restart; read-oriented presentation tools run server-side while consequential actions surface as approval/frontend tools, matching the "read-only by default, remediation opt-in" posture; and bring-your-own connector with MCP Apps as pluggable capabilities matches operator-controlled models and open integration.

Two deliberate differences frame the near-term scope honestly:

• We start from a question, they start from an alert. This RFC's entry point is a user stating a goal in chat. The larger opportunity, the direction this foundation enables, is alert-triggered autonomous triage, where a firing alert opens a thread, the agent investigates unattended, and the on-call engineer opens the persisted session with its widgets first. The chat/canvas/widget surface, the AG-UI event stream, and the memory container are the building blocks for that; wiring alerting as an investigation trigger is a follow-on rather than in scope here.
• Deduplication / incident grouping is not addressed. It is the primary cost and noise lever in the commercial products. If we pursue autonomous, alert-triggered investigation, a grouping stage in front of the agent loop matters so we investigate unique incidents rather than every duplicate.

The product intent follows from this: treat the work as the agentic front door for OpenSearch observability rather than a chat add-on, the surface an on-call engineer opens to an in-progress or completed investigation, and keep the presentation-tool / widget contract the stable seam, since it lets both an interactive user and (later) an autonomous trigger produce the same verifiable, evidence-backed session. The roadmap sequences toward answer-first on-call: user-initiated chat (this RFC) → persisted, shareable investigations → alert-triggered autonomous triage → incident grouping and remediation gating.

Background

OpenSearch Dashboards ships dashboards-assistant, a chat surface with a single core.chat implementation slot that entry points such as Ask-AI and PPL quick-fix route into. It predates the AG-UI agent and does not stream tool calls, render interactive widgets, or drive page navigation.

ml-commons 3.6.0 added an AG-UI agent that streams a conversation over Server-Sent Events from POST /_plugins/_ml/agents/<id>/_execute/stream, along with frontend-tool support, agentic memory containers, and token-usage tracking. These are the primitives an agentic frontend needs, and they are the foundation this RFC builds on.

The proposed experience has three surfaces:

• Chat: a thread UI (built on the assistant-ui library) that streams an AG-UI conversation, renders tool calls (plan, approval, terminal, data table, chart, option list, etc.), and persists threads.
• MCP App widgets: interactive observability visualizations (service map, trace health overview, trace details) rendered inline in the transcript. We'll reuse existing MCP App widgets available from https://docs.aws.amazon.com/opensearch-service/latest/developerguide/opensearch-observability-mcp-app.html.
• Canvas: a tabbed workspace that embeds real Dashboards observability pages (logs, traces, service map, dashboards, alerting, etc).

MCP Apps

The observability widgets in this RFC are MCP Apps: interactive UI applications that a tool call renders inside an MCP host. The capability is documented publicly at Agentic Observability with MCP Apps, where a local MCP server bridges an agentic IDE (Claude Desktop, VS Code with GitHub Copilot, Cursor, Goose, ChatGPT) to an OpenSearch observability workspace and renders MCP App widgets in the conversation.

Each MCP App is a composable unit that performs one observability job. The documented app families are:

• Triage and Response: alert correlation, incident timeline.
• Log Investigation: pattern search, log clustering.
• Trace Investigation: trace finder, span details, latency breakdown.
• Metric Investigation: PromQL exploration, threshold analysis.
• Service Performance: RED metrics, service-level views.
• Topology: service maps, dependency graphs.
• Visualizations: dynamic line, bar, area, and metric charts.
• Datasets and Correlations: cross-signal joins, data summaries.
• AI and Agent Observability: LLM call tracing, agent trace map and summary.
• Stack Health: cluster status, shard allocation.
• Instrumentation score: telemetry-based detection of instrumentation gaps.

As a prerequisite, the MCP apps needs to be open sourced and modified to support on-premise OpenSearch Dashboards. Future additions to app families will benefit both places.

Proposed Design

Existing Architecture

flowchart LR
  subgraph Browser
    UI[dashboards-assistant chat UI]
  end
  subgraph OSD[OpenSearch Dashboards]
    SLOT[core.chat implementation slot]
  end
  subgraph Cluster
    MLC[ml-commons AG-UI agent 3.6.0+]
    MODEL[Model connector - BYO]
  end
  UI --> SLOT --> MLC --> MODEL

Loading

The AG-UI agent, memory, and streaming exist in the cluster today; the gap is a Dashboards frontend that consumes them agentically, plus the widget library and server-side presentation tools.

New Architecture

Ship a new Dashboards plugin that provides the core.chat implementation within observability workspaces only, consumes the AG-UI endpoint in ml-commons, and renders MCP App widgets from a standalone widget library. Contribute the observability presentation tools to ml-commons so any cluster can run them server-side. The agentic chat is observability-focused: it is scoped to observability workspaces, and every other workspace keeps the existing dashboards-assistant chat unchanged. The two surfaces coexist permanently, each owning the core.chat slot in its own workspaces.

flowchart LR
  subgraph Browser
    UI[Chat UI]
    WGT[widget library - standalone pkg]
    CANVAS[Canvas - iframed OSD pages]
  end
  subgraph OSD[OpenSearch Dashboards]
    PLUGIN[agentic-chat plugin - core.chat impl]
  end
  subgraph Cluster
    MLC[ml-commons AG-UI agent 3.6.0+]
    TOOLS[Presentation tools - contributed to ml-commons]
    MEM[Agentic memory container]
    MODEL[Model connector - BYO Bedrock/HTTP]
  end
  UI --> PLUGIN
  WGT --> PLUGIN
  CANVAS -.same-origin.-> OSD
  PLUGIN -->|AG-UI SSE| MLC
  MLC --> TOOLS
  MLC --> MEM
  MLC --> MODEL

Loading

Core Concepts

• AG-UI conversation: the agent turn is an SSE stream of typed events (RUN_STARTED, TEXT_MESSAGE_START/CONTENT/END, TOOL_CALL_START/ARGS/END/RESULT, MESSAGES_SNAPSHOT, RUN_FINISHED, RUN_ERROR). ml-commons defines all of these event types today.
• Backend tool: a tool the agent executes server-side inside the ReAct loop; its result feeds the next model step.
• Frontend tool: a tool declared in the request that the agent must not run server-side; the browser runs it. ml-commons models this with AGUIFrontendTool (a placeholder so the model sees the tool) and pauses the run when the model calls one (see Backend Readiness). MCP App widgets and canvas navigation are frontend tools.
• Presentation tool: a backend tool whose structured result is a widget: { text, ...props }. The transcript shows text; the plugin caches the remaining fields as widget props and picks a View by tool name.
• Canvas: a tabbed set of same-origin iframes hosting real Dashboards pages, kept alive across tab switches, driven by typed navigation tools.

Gaps and prerequisites

• Version floor: the features require ml-commons >= 3.6.0. Document the minimum version and align the Dashboards release.
• Cluster settings: two experimental settings must be enabled, plugins.ml_commons.stream_enabled and plugins.ml_commons.ag_ui_enabled. Document them as prerequisites and surface a clear disabled-state error in the UI rather than a silent empty reply.
• Presentation tools: Contribute them as ml-commons built-in tools so any cluster can run them server-side.
• Provisioning: there is no bundled bootstrap for the agent, memory container, and connector. Ship a default agent template with config override, and document manual provisioning.
• Follow-up robustness: add a reproduction test upstream for follow-up completion over threads that contain prior tool errors or empty turns, and verify before GA.

Interfaces / APIs

Backend endpoint consumed:

POST /_plugins/_ml/agents/<agentId>/_execute/stream
Content-Type: application/json
Accept: text/event-stream

# AG-UI RunAgentInput body: { threadId, runId, messages[], tools[], ... }
# tools[] carries frontend tool descriptors (widgets, canvas navigation)

Plugin configuration (opensearch_dashboards.yml), illustrative:

agentic_chat.enabled: true
agentic_chat.agentId: ""            # optional; if empty, plugin bootstraps a default agent
agentic_chat.memoryContainerId: ""  # agentic memory container for thread history
agentic_chat.modelConnectorId: ""   # bring-your-own model connector

Widget library dependency:

// package.json
"dependencies": {
  "opensearch-mcp-apps-ui": "^<version>"
}

Scope

In Scope

• New Dashboards plugin providing agentic chat as the core.chat implementation within observability workspaces only; dashboards-assistant continues to serve every other workspace, and the two coexist.
• Chat thread UI, AG-UI SSE streaming, tool-call rendering (plan, approval, terminal, data table, chart, option list, parameter slider, preferences, question flow), and thread persistence.
• Inline MCP App widgets via the presentation-tool contract.
• Standalone widget library (opensearch-mcp-apps-ui).
• Canvas: same-origin iframe embedding of Dashboards observability pages, agent-driven navigation, tabs, sidebar nav.
• Contributing the observability presentation tools to ml-commons.
• Bring-your-own model connector; a bundled default agent template with config override.
• Ask-AI and PPL apply-and-test-fix integration through the core.chat slot.
• Documentation of prerequisites (version floor, cluster settings, provisioning).

Out of Scope

• Rewriting, replacing, or deprecating dashboards-assistant; it keeps serving all non-observability workspaces and coexists indefinitely.
• Changing chat in non-observability workspaces; the agentic experience is scoped to observability workspaces only.
• Direct-component embedding of observability pages (a later migration away from iframes; see Migration Strategy).
• Provisioning or shipping any specific LLM; operators bring their own connector.

Coexistence and Migration Strategy

• Coexistence with dashboards-assistant: the agentic chat provides the core.chat implementation only within observability workspaces; dashboards-assistant continues to own the slot in every other workspace. There is no deprecation and no rewrite; the two surfaces coexist indefinitely, scoped by workspace. Within observability workspaces, Ask-AI and PPL quick-fix route into the agent; elsewhere they behave exactly as today. No wire-format change; both consume the same AG-UI endpoint.
• From iframe canvas to direct-component import: long-term, gated on target observability plugins exposing mountable components. Keep the transport glue (postMessage) thin and throwaway; the typed tool contract survives the swap.

Implementation Plan

Phase 1: Backend readiness and presentation tools

• Confirm parity of the ml-commons >= 3.6.0 AG-UI agent (frontend-tool pause, memory/history, token tracking) against the plugin's needs; file any gaps upstream.
• Add an upstream reproduction test for follow-up robustness over prior tool errors and empty turns.
• Contribute observability presentation tools (traces_health_overview, service map, trace details) to ml-commons as built-in tools.
• Good first issues: documentation of the two cluster settings; a sample agent template.

Phase 2: Widget library

• Publish the widget library as a standalone package (opensearch-mcp-apps-ui) with its own prefix, tokens, WidgetHost, runRoute, and descriptor list.
• Version it independently; the plugin consumes it as a dependency.

Phase 3: Chat plugin MVP

• New plugin: thread UI, AG-UI streaming, tool-call rendering, thread persistence.
• Register as the core.chat implementation; bring-your-own connector; bundled default agent template with config override.

Phase 4: Canvas and widgets

• Same-origin iframe canvas for observability pages; tabs; sidebar nav; agent-driven navigation (pageLink); inline MCP App widgets via the presentation-tool contract.

Phase 5: Ask-AI / PPL-fix integration and hardening

• Reroute Ask-AI and PPL quick-fix through the core.chat slot; PPL apply/test-fix tools.
• Feature-flag / kill-switch config; disabled-state UX when cluster settings are off; guards for drift in the untyped DOM/CSS contracts the canvas depends on.

Alternatives Considered

A. Status quo

Ship no agentic frontend; leave dashboards-assistant as the only chat surface.

• Pros: no work; no new support surface.
• Cons: observability workspaces get no agent-primary experience.

B. Extend dashboards-assistant in place

Upgrade the existing assistant plugin to AG-UI, canvas, and widgets rather than shipping a new plugin.

• Pros: reuses existing users and the core.chat slot; one plugin.
• Cons: inherits a non-agentic architecture and a heavier, riskier review; the canvas and widget surfaces are large additions that do not fit its current shape; changing the shared plugin would affect all workspaces rather than just observability ones. Rejected in favor of a new plugin scoped to observability workspaces, leaving dashboards-assistant untouched everywhere else.

Risks and Mitigations

• Follow-up robustness over threads with prior tool errors is not fully verified upstream. Add a reproduction test in Phase 1 and block GA on it. The behavior client widgets depend on (the frontend-tool pause) is already merged.
• Streaming and AG-UI are gated behind experimental cluster settings that default off. Document the prerequisites and surface a clear disabled-state error rather than a silent empty reply.
• The canvas relies on untyped DOM/CSS contracts (core chrome and nav, EUI selectors). Centralize them in one registry with dev-time drift warnings; iframe isolation limits the blast radius.
• The iframe canvas is heavy (double React roots, memory). Keep the transport glue thin and throwaway, with a documented direct-import migration path once target plugins expose mountable components.
• Model-connector availability varies by deployment. Bring-your-own connector; ml-commons supports both Bedrock and generic HTTP streaming upstream, so there is no hard Bedrock dependency.
• The presentation tools may not be accepted upstream in ml-commons. Fallback: ship them as a separate tool extension registered with the agent.

Open Questions

1. Should the observability presentation tools be ml-commons built-in tools, or a separately versioned tool extension registered with the agent?
2. Where should the default bundled agent template live, and how should it be versioned against ml-commons prompt and tool changes?
3. What is the repo home for the standalone widget library and agentic chat plugin: a monorepo plugin, or independent repos?

References

• ml-commons AG-UI runners and events (main / 3.6.0): MLAGUIAgentRunner, MLChatAgentRunner, RestMLExecuteStreamAction, common/.../agui/*
• ml-commons 3.6.0 release notes
• Existing open-source assistant: opensearch-project/dashboards-assistant
• assistant-ui chat UI library (docs)
• Agentic Observability with MCP Apps

[joshuali925]

video concept

export-1787609729590.small.mp4