Enhancing Root Cause Analysis via LLM-based Multi-Agents with Human-in-the-Loop
ChatRCA is an advanced tool for root cause analysis of cloud events. It builds a multi-intelligent agent root cause analysis method with humans in the loop, simulating the collaborative model in real-world root cause analysis. Multiple domain experts (Architecture, Resource, Network) collaborate through a group chat, observe abnormal data, and produce a structured diagnosis with root cause category, service location, and evidence.
- Architecture Overview
- Multi-Agent Design
- Quick Start
- Configuration
- Datasets
- Project Structure
- Output Format
- RAG Knowledge Base
- Empirical Study
- License
User_proxy ──▶ GroupChatManager ──▶ OperationEngineer (final diagnosis)
│
├── Observable_engineer ──▶ Operator (tool execution)
├── ArchitectExpert
├── ResourceExpert
└── NetWorkExpert
- User_proxy initiates the chat with a fault description (time, affected service).
- Observable_engineer calls registered tools to read and filter abnormal data (log/metric/trace) and architecture information.
- Domain Experts (Architect / Resource / Network) analyze the observation data from their professional perspective, each producing a structured JSON finding.
- OperationEngineer collects all expert findings, compares evidence and confidence, and outputs the final structured diagnosis.
| Agent | Role | Key Behavior |
|---|---|---|
| User_proxy_agent | Human-in-the-loop | Initiates fault report, can intervene at any round |
| Observable_engineer_agent | Data observation | Reads fault data via tools, filters anomalies, provides structured evidence |
| Operator | Tool executor | Executes registered Python functions (code execution agent, no LLM) |
| ArchitectExpert | Architecture expert | Analyzes service topology, deployment position, upstream/downstream dependencies |
| ResourceExpert | Resource expert | Analyzes CPU/memory/node/pod exhaustion, runtime bottlenecks |
| NetWorkExpert | Network expert | Analyzes timeouts, connection resets, DNS/routing, inter-service communication failures |
| OperationEngineer | Final diagnosis | Synthesizes expert findings, resolves conflicts by evidence strength, outputs final JSON diagnosis |
The Observable_engineer_agent has the following tools registered (executed by Operator):
| Tool | Description |
|---|---|
data |
Read current fault observation data |
log_data_processing |
Read and filter log data (extract ERROR entries) |
trace_data_processing |
Read and filter trace data (slow spans exceeding duration threshold) |
knowledge_retrieval |
Retrieve advisory architecture/runbook knowledge for the current RCA query |
- Python > 3.10
- OpenAI API key (GPT-4o)
git clone https://github.com/leocache/ChatRCA.git
cd ChatRCA
pip install -r requirements.txtcp .env.example .env
# Edit .env and fill in your OpenAI API key- Select a fault instance in
config.yaml:
fault: fault_1- Run the main program:
python main.py- The system will start a multi-agent group chat. Example output:
User_proxy_agent (to chat_manager):
The current cloud system experienced a failure at 2023-01-29 09:25:39.
The current ts-basic-service service is affected. Please analyze the root cause.
--------------------------------------------------------------------------------
Next speaker: OperationEngineer
OperationEngineer (to chat_manager):
Let's proceed with the analysis step by step:
1. Observable Engineer - Please provide the current abnormal data.
2. NetWork Expert - Prepare to analyze network-related issues.
3. Architecture Expert - Be ready to provide architecture insights.
4. Resource Expert - Prepare to examine resource-related anomalies.
...
Edit main.py to switch the entry function:
if __name__ == '__main__':
# run_TrainTicket_fault()
run_GAIA_fault()| Field | Description | Example |
|---|---|---|
fault |
Fault instance directory name under the dataset folder | fault_1, F_6 |
| Variable | Description |
|---|---|
OPENAI_API_KEY |
Your OpenAI API key (required for GPT-4o) |
The LLM model and group chat parameters can be configured in agents.py:
config_list— model name, API key, timeoutGroup_chat.max_round— maximum conversation rounds (default: 20)human_input_mode— set to"ALWAYS"for human-in-the-loop,"NEVER"for fully autonomous
An open-source dataset from the medium-sized case system TrainTicket. Contains 45 fault instances covering real fault types: system resource exhaustion, network anomalies, application errors, etc.
- Location:
TrainTicket/fault_*/ - Each fault folder contains:
fault.txt— fault injection source information (JSON format)log.csv— log data before and after the faultmetric.csv— metric data before and after the faulttrace.csv— trace data before and after the fault
A private dataset collected from a core system on a large enterprise cloud platform. Due to data security requirements, the raw data cannot be open-sourced. Pre-processed data is provided in the GAIA/ directory.
- Location:
GAIA/F_*/ - Data cleaning scripts:
GAIA_DataCleaning/
Additional private fault data stored in PrivateFault/, not open-sourced.
ChatRCA/
├── main.py # Entry point: run_TrainTicket_fault() / run_GAIA_fault()
├── agents.py # Multi-agent definitions, tool registration, group chat setup
├── schemas.py # Pydantic schemas for structured diagnosis output
├── config.yaml # Fault instance selection
├── .env.example # API key template
├── requirements.txt # Python dependencies
│
├── utils/ # Utility modules
│ ├── __init__.py # Package init, exposes get_fault_info, get_fault_type_census
│ ├── read_file.py # File reading: YAML, CSV→Markdown, fault path resolution
│ ├── csv2md.py # CSV to Markdown table converter
│ ├── filter_tools.py # Data filters: error logs, abnormal metrics, slow traces
│ ├── location_tools.py # Service location normalization and inference from pod names
│ ├── observation_summary.py # Summarize filtered data into structured evidence
│ ├── skills4tt.py # Tools for TrainTicket dataset (registered as agent skills)
│ └── skills4gy.py # Tools for GAIA dataset (registered as agent skills)
│
├── architecture/ # Architecture dependency documents
│ ├── TrainTicket.md # TrainTicket microservice topology and dependencies
│ └── GAIA.md # GAIA system architecture
│
├── knowledge/ # Advisory RAG knowledge sources
│ ├── README.md # Knowledge-base policy and build instructions
│ └── runbooks/ # Diagnostic runbooks for common RCA patterns
│
├── build_knowledge_base.py # Builds the Chroma/local retrieval index
├── TrainTicket/ # D1 dataset: 45 fault instances
│ ├── fault_1/
│ │ ├── fault.txt
│ │ ├── log.csv
│ │ ├── metric.csv
│ │ └── trace.csv
│ ├── fault_2/
│ └── ...
│
├── GAIA/ # D2 dataset (pre-processed)
│ └── DataCleaning/ # Data cleaning scripts for GAIA
├── PrivateFault/ # D3 private fault data
│
├── empirical_study/ # Empirical research questionnaire responses
│ ├── Questionnaire_1.docx
│ └── ...
│
└── LICENSE # MIT License
ChatRCA includes a RAG layer for advisory knowledge retrieval. When chromadb is available, it builds and queries a persistent Chroma index under .knowledge_chroma/. The Chroma backend prefers sentence-transformers/all-MiniLM-L6-v2 for real semantic embeddings and falls back to deterministic token-hashing embeddings if the model is unavailable. If Chroma itself is unavailable, ChatRCA falls back to a local JSON retrieval index under .knowledge_index/. It indexes architecture documents and runbook knowledge, then exposes a knowledge_retrieval tool to the observation agent. Retrieved knowledge helps agents interpret incident evidence, but it must not replace evidence from the current fault's logs, traces, metrics, or architecture observations.
venv/bin/python build_knowledge_base.pyUse the project virtualenv when you want the Chroma backend, because it contains chromadb, sentence-transformers, and the compatible numerical dependencies. The generated Chroma index is stored under .knowledge_chroma/. If the sentence-transformer model cannot be loaded, the Chroma backend falls back to deterministic token-hashing embeddings. If Chroma is unavailable in the active Python environment, the script builds the fallback .knowledge_index/index.json file instead. Both generated index directories are ignored by git because they can be rebuilt from source documents.
architecture/*.md— service topology and dependency context.knowledge/runbooks/*.md— diagnostic playbooks for timeouts, resource exhaustion, slow traces, and cascading failures.
The knowledge base should not index fault.txt, groundtruth.txt, fault injection fields, answer labels, or raw evaluation ground truth. This keeps RCA evaluation evidence-driven and prevents label leakage.
Observable_engineer_agent can call knowledge_retrieval with a query built from service names, pod names, trace service names, error keywords, and slow-span clues. The tool uses backend="auto" by default: Chroma is preferred when available, and the local JSON index is used as a fallback. The retrieved snippets are returned separately from observed evidence. OperationEngineer is instructed to use retrieved knowledge only as advisory context.
The empirical_study/ directory contains 20 questionnaire response files (Questionnaire_1.docx ~ Questionnaire_20.docx) from the empirical evaluation described in our paper. These questionnaires assess the effectiveness and usability of the ChatRCA approach from domain practitioners.
This project is licensed under the MIT License. See LICENSE for details.