diff --git a/docs/agentic_detection_engineering.md b/docs/agentic_detection_engineering.md new file mode 100644 index 00000000..910fbe9b --- /dev/null +++ b/docs/agentic_detection_engineering.md @@ -0,0 +1,190 @@ +# Agentic Detection Engineering in Google SecOps + +Agentic Detection Engineering (ADE) enables security teams to automate and accelerate the end-to-end detection engineering lifecycle using Google Security Operations (SecOps) APIs and AI assistants. + +By integrating threat intelligence, automated Threat Detection Opportunity (TDO) extraction, synthetic telemetry simulation, and sandbox rule coverage evaluations, ADE transforms unstructured threat descriptions into tested, production-ready YARA-L 2.0 detection rules. + +--- + +## Overview + +Traditional detection engineering requires manual parsing of threat intelligence reports, manual drafting of adversary simulation commands or test logs, tedious cross-referencing against existing rule corpora, and extensive manual tuning to write YARA-L 2.0 detection rules. + +Agentic Detection Engineering in Google SecOps streamlines this into a continuous, automated lifecycle: + +``` ++-----------------------------+ +| Threat Intelligence Ingest | (Blogs, Reports, CVEs, TTPs) ++--------------+--------------+ + | + v ++-----------------------------+ +| TDO Generation | (generate_threat_detection_opportunity) ++--------------+--------------+ + | + v ++-----------------------------+ +| Synthetic Simulation | (generate_synthetic_events) ++--------------+--------------+ + | + v ++-----------------------------+ +| Rule Coverage Evaluation | (evaluate_rule_coverage_long_running) ++--------------+--------------+ + | + v ++-----------------------------+ +| Operation Polling & Results | (get_operation) ++--------------+--------------+ + | + +-----------------------+ + | | + [Coverage Confirmed] [Coverage Gap] + | | + v v + (No action needed) +-----------------------------+ + | Candidate Rule Generation | (generate_rules) + +--------------+--------------+ + | + v + +-----------------------------+ + | Human Review & Deployment | (test_rule, create_rule) + +-----------------------------+ +``` + +--- + +## The Detection Engineering Lifecycle + +### 1. Threat Detection Opportunity (TDO) Extraction +Detection engineers or autonomous agents analyze threat reports, advisories, or post-incident reviews to identify observable adversary behaviors. Using `generate_threat_detection_opportunity`, the input text is transformed into structured TDO objects containing: +- **TDO ID**: Unique identifier (e.g., `t01`, `t02`). +- **Summary**: Concise description of the attacker tactic or procedure. +- **MITRE ATT&CK Mapping**: Specific tactics and techniques (e.g., `T1059.001` PowerShell, `T1071.001` Web Protocols). +- **Log Types**: Relevant Chronicle log ingestion types (e.g., `WINEVTLOG`, `PROCESS_EXECUTION`, `GCP_CLOUDAUDIT`). + +### 2. Synthetic Event Simulation +To evaluate whether existing detection rules would catch the activity, `generate_synthetic_events` generates high-fidelity synthetic telemetry. This produces: +- Raw mock log lines matching the targeted log type formats. +- Structured Unified Data Model (UDM) events with appropriate entity metadata (`principal`, `target`, `network`, `about`). +- JSON-encoded UDM event strings (`udmJson`) formatted for direct consumption by Chronicle evaluation engines. + +### 3. SecOps UI: Synthetic Data Visibility +Synthetic events and the resulting detections can be displayed directly in the Google SecOps Web UI for interactive inspection and validation. + +To view synthetic telemetry in list and detail views: +1. Navigate to **Google SecOps**. +2. Click **Settings** (gear icon) in the navigation bar. +3. Select **User Preferences** > **Synthetic Data Visibility**. +4. Check **Show synthetic test data**. +5. Click **Save**. + +![Synthetic Data Visibility](img/synthetic_data_visibility.png) + +> **Note:** Enabling this setting displays synthetic test data (including events, detections, and alerts) in list and detail views across Chronicle. This does not affect data generated by Security Validation, which remains hidden by default. + +### 4. Rule Coverage Evaluation via Long-Running Operations (LRO) +Evaluating synthetic events against an organization's active ruleset is computationally intensive. The tool `evaluate_rule_coverage_long_running` initiates an asynchronous evaluation job via Chronicle's `:evaluateRuleCoverageLongRunning` API endpoint: +- **Sandboxed Execution:** Synthetic events are evaluated in an ephemeral sandbox without committing test records to permanent customer log storage. +- **Composite Coverage Control:** The `exclude_composite_coverage` parameter allows filtering out multi-event composite rules when testing single atomic behaviors. +- **Asynchronous Operation:** Returns a standard Google Long-Running Operation resource (e.g., `operations/dea-bkFXS0...`). + +### 5. Polling Operation Status +The `get_operation` tool polls the returned operation name until completion: +- **In-Progress:** Returns operation metadata including progress status and percentages. +- **Completed:** Returns the final evaluation result containing covered TDO IDs, uncovered TDO IDs, matching rule identifiers, and matched event counts. + +### 6. Candidate Rule Synthesis +For any TDO identified as having a coverage gap, `generate_rules` synthesizes candidate YARA-L 2.0 detection rules. The generated rules include: +- Informative `meta` section with author, description, severity, and MITRE ATT&CK tags. +- Precise `events` logic referencing UDM fields. +- Deduplication and aggregation logic in `match` and `condition` sections. + +### 7. Human-in-the-Loop Review and Deployment +Generated rules must never be automatically activated in production without human validation. Detection engineers follow these verification steps: +1. **Rule Logic Inspection:** Verify UDM field references and thresholds. +2. **Backtesting (`test_rule`):** Execute historical test queries over real tenant data to assess alert volume and detect potential false positives. +3. **Draft Rule Creation (`create_rule`):** Deploy rule in a disabled (`enabled=False`) or alerting-only state for staging observation. +4. **Activation:** Enable live evaluation once verified (via the SecOps console or rule management tools). + +--- + +## Available MCP Tools + +The `secops-mcp` server provides 5 purpose-built tools for Agentic Detection Engineering: + +| Tool | Purpose | Key Parameters | +|------|---------|----------------| +| `generate_threat_detection_opportunity` | Extracts structured TDOs from threat descriptions | `threat_description`, `log_types` | +| `generate_synthetic_events` | Synthesizes realistic raw logs and UDM test events | `threat_detection_opportunities` | +| `evaluate_rule_coverage_long_running` | Starts asynchronous rule coverage evaluation LRO | `threat_detection_opportunity_events`, `exclude_composite_coverage` | +| `get_operation` | Polls status and retrieves LRO evaluation results | `name` | +| `generate_rules` | Generates candidate YARA-L 2.0 rules for coverage gaps | `threat_detection_opportunities`, `background_context` | + +For detailed parameter schemas and API reference, see [SecOps MCP Tools](servers/secops_mcp.md). + +--- + +## Agent Skill: `detection-engineering-coverage-evaluation` + +The **Google SecOps Extension** packages this entire workflow into a turnkey agent skill: + +- **Trigger:** `/security:detect`, `"Evaluate coverage for [URL/Text]"`, `"Develop detections for [Threat]"`. +- **Location:** `extensions/google-secops/skills/detection-coverage/SKILL.md` (exposed via `.agent/skills/detection-coverage/`). +- **Prompt Injection Safeguards:** Threat intelligence articles and external blog URLs are treated as untrusted data. The skill enforces clear demarcation between ingested threat content and agent execution instructions. +- **Human Authorization Gate:** Explicit user confirmation is strictly required prior to saving or enabling any detection rules in production. + +--- + +## Example Workflow + +### Step 1: Ingest Threat Description +```python +tdo_response = generate_threat_detection_opportunity( + threat_description=""" + Adversaries execute encoded PowerShell commands to download secondary stage payloads + from external C2 servers and establish persistent scheduled tasks. + """, + log_types=["WINEVTLOG", "PROCESS_EXECUTION"] +) +``` + +### Step 2: Generate Synthetic UDM Events +```python +events_response = generate_synthetic_events( + threat_detection_opportunities=tdo_response["threat_detection_opportunities"] +) +``` + +### Step 3: Evaluate Coverage Sandbox +```python +lro_response = evaluate_rule_coverage_long_running( + threat_detection_opportunity_events=events_response["threat_detection_opportunity_events"], + exclude_composite_coverage=True +) +operation_name = lro_response["name"] +``` + +### Step 4: Poll LRO Until Done +```python +status = get_operation(name=operation_name) +# Poll until status["done"] is True +# Coverage results indicate uncovered TDOs +``` + +### Step 5: Generate YARA-L 2.0 Rule for Gaps +```python +rules_response = generate_rules( + threat_detection_opportunities=uncovered_tdos, + background_context="Enterprise Windows workstations with Defender and Sysmon telemetry." +) +``` + +--- + +## Related Documentation + +- [SecOps MCP Server Reference](servers/secops_mcp.md) +- [Detection Engineer Persona](personas/detection_engineer.md) +- [Google SecOps Extension Skills](google_secops_extension.md) +- [Official Google SecOps ADE Guide](https://docs.cloud.google.com/chronicle/docs/secops/agentic-detection-engineering) diff --git a/docs/google_secops_extension.md b/docs/google_secops_extension.md index 7b9c6807..71b3bb3b 100644 --- a/docs/google_secops_extension.md +++ b/docs/google_secops_extension.md @@ -81,6 +81,11 @@ You will be prompted for two environment variables for the MCP configuration: * **Trigger**: "Hunt for [Threat]", "Search for TTP [ID]". * **Function**: Assists in proactive threat hunting by generating hypotheses and constructing complex UDM queries for Chronicle. +### 6. Detection Engineering (`detection-engineering-coverage-evaluation`) +* **Trigger**: "Develop detections for [Threat]", "Evaluate coverage for [URL/Text]", `/security:detect`. +* **Function**: Orchestrates the end-to-end Detection Engineering lifecycle: extracts TDOs from threat intelligence, simulates synthetic UDM events, evaluates existing rule coverage with long-running operations, generates draft YARA-L 2.0 rules to close coverage gaps, and deploys approved rules. +* **Guide**: See [Agentic Detection Engineering Guide](agentic_detection_engineering.md). + ## How it Works These skills act as **Driver Agents** that: @@ -96,7 +101,7 @@ The skills employ an **Adaptive Execution** strategy to ensure robustness: 2. **Prioritize Remote**: If the **Remote MCP Server** is connected, the skill uses remote tools (e.g., `list_cases`, `udm_search`) for maximum capability. 3. **Fallback to Local**: If remote tools are unavailable, the skill automatically falls back to **Local Python Tools** (e.g., `search_security_events`). -For a detailed mapping of Remote vs. Local capabilities, see [`TOOL_MAPPING.md`](../TOOL_MAPPING.md). +For a detailed mapping of Remote vs. Local capabilities, see [`TOOL_MAPPING.md`](https://github.com/google/mcp-security/blob/main/extensions/google-secops/TOOL_MAPPING.md). ## Cross-Compatibility diff --git a/docs/img/synthetic_data_visibility.png b/docs/img/synthetic_data_visibility.png new file mode 100644 index 00000000..bb83a55e Binary files /dev/null and b/docs/img/synthetic_data_visibility.png differ diff --git a/docs/index.md b/docs/index.md index ed0ca5d2..25073ca1 100644 --- a/docs/index.md +++ b/docs/index.md @@ -20,6 +20,7 @@ If you're new to this project, we recommend starting with the [Usage Guide](usag ## Quick Links - **[Installation & Setup](usage_guide.md#getting-started)** - Get started quickly with installation instructions +- **[Agentic Detection Engineering](agentic_detection_engineering.md)** - Automate detection engineering with TDO extraction, synthetic events, and rule coverage evaluations - **[Configuration Reference](usage_guide.md#mcp-server-configuration-reference)** - Configure the MCP servers for your environment - **[Usage Examples](usage_guide.md#usage-examples)** - See examples of how to interact with the MCP servers - **[Development Guide](development_guide.md)** - Learn how to contribute to or extend the project diff --git a/docs/servers/secops_mcp.md b/docs/servers/secops_mcp.md index df99f877..6874ce7d 100644 --- a/docs/servers/secops_mcp.md +++ b/docs/servers/secops_mcp.md @@ -827,7 +827,59 @@ The service account or user credentials need the following Chronicle roles: - `project_id` (optional): Google Cloud project ID (defaults to environment config). - `customer_id` (optional): Chronicle customer ID (defaults to environment config). - `region` (optional): Chronicle region (defaults to environment config or 'us'). - - **Returns:** Dictionary containing investigation associations grouped by detection ID, with verdict and confidence information. +### Detection Engineering Agent Tools + +Tools for automating the Detection Engineering lifecycle using Chronicle's Agentic Detection Engineering (ADE) APIs: + +- **`generate_threat_detection_opportunity(threat_description, log_types, project_id=None, customer_id=None, region=None)`** + - **Description:** Generate structured Threat Detection Opportunities (TDOs) from a threat intelligence description and targeted log types. Returns extracted tactics, techniques, and procedures (TTPs) mapped to MITRE ATT&CK. + - **Parameters:** + - `threat_description` (required): Natural-language text describing the threat or adversary TTPs. + - `log_types` (required): List of Chronicle log types to consider (e.g. `["WINEVTLOG", "PROCESS_EXECUTION"]`). + - `project_id` (optional): Google Cloud project ID (defaults to environment config). + - `customer_id` (optional): Chronicle customer ID (defaults to environment config). + - `region` (optional): Chronicle region (defaults to environment config or 'us'). + - **Returns:** Dictionary containing generated `threat_detection_opportunities` with IDs, summaries, log types, and MITRE ATT&CK mappings. + +- **`generate_synthetic_events(threat_detection_opportunities, project_id=None, customer_id=None, region=None)`** + - **Description:** Generate high-fidelity synthetic telemetry (raw logs and structured UDM events) to simulate attacker behavior for given Threat Detection Opportunities. + - **Parameters:** + - `threat_detection_opportunities` (required): List of TDO dictionaries (or TDO objects returned by `generate_threat_detection_opportunity`). + - `project_id` (optional): Google Cloud project ID (defaults to environment config). + - `customer_id` (optional): Chronicle customer ID (defaults to environment config). + - `region` (optional): Chronicle region (defaults to environment config or 'us'). + - **Returns:** Dictionary containing `threat_detection_opportunity_events` with synthetic raw logs, structured UDM events, and JSON-encoded `udmJson` strings. + +- **`evaluate_rule_coverage_long_running(threat_detection_opportunity_events, exclude_composite_coverage=True, project_id=None, customer_id=None, region=None)`** + - **Description:** Initiate an asynchronous Long-Running Operation (LRO) via Chronicle's `:evaluateRuleCoverageLongRunning` endpoint to test synthetic events against active rulesets in a safe sandbox simulation. + - **Parameters:** + - `threat_detection_opportunity_events` (required): List of event bundles containing `threat_detection_opportunity_id` and list of `udms_json`. + - `exclude_composite_coverage` (optional): Whether to exclude multi-event composite rules from evaluation (default: `True`). + - `project_id` (optional): Google Cloud project ID (defaults to environment config). + - `customer_id` (optional): Chronicle customer ID (defaults to environment config). + - `region` (optional): Chronicle region (defaults to environment config or 'us'). + - **Returns:** Dictionary containing the Long-Running Operation resource with `name` (e.g., `operations/dea-...`). + +- **`get_operation(name, project_id=None, customer_id=None, region=None)`** + - **Description:** Poll the status of a Long-Running Operation (such as rule coverage evaluation). Returns progress metadata or final coverage results when `done` is `True`. + - **Parameters:** + - `name` (required): Full operation resource name returned by `evaluate_rule_coverage_long_running`. + - `project_id` (optional): Google Cloud project ID (defaults to environment config). + - `customer_id` (optional): Chronicle customer ID (defaults to environment config). + - `region` (optional): Chronicle region (defaults to environment config or 'us'). + - **Returns:** Dictionary containing operation state (`done`, `metadata`, and `response`). + +- **`generate_rules(threat_detection_opportunities, background_context=None, project_id=None, customer_id=None, region=None)`** + - **Description:** Synthesize candidate YARA-L 2.0 detection rules tailored to close coverage gaps identified for specific Threat Detection Opportunities. + - **Parameters:** + - `threat_detection_opportunities` (required): List of uncovered TDO dictionaries. + - `background_context` (optional): Additional organizational context, telemetry nuances, or detection conventions. + - `project_id` (optional): Google Cloud project ID (defaults to environment config). + - `customer_id` (optional): Chronicle customer ID (defaults to environment config). + - `region` (optional): Chronicle region (defaults to environment config or 'us'). + - **Returns:** Dictionary containing synthesized candidate YARA-L 2.0 rules with metadata and logic. + +For an end-to-end walkthrough, see the [Agentic Detection Engineering Guide](../agentic_detection_engineering.md). ## Usage Examples @@ -1123,6 +1175,68 @@ Entities previously on this watchlist will now use their base risk scores without the 4.0x multiplier. ``` +### Example 9: Agentic Detection Engineering Workflow + +``` +User: We received a threat intel advisory detailing a new PowerShell download cradle communicating with pastebin-like sites. Can you extract detection opportunities, test our rule coverage, and generate rules for any gaps? + +AI uses: generate_threat_detection_opportunity( + threat_description="Threat actors leverage obfuscated PowerShell commands with DownloadString to fetch scripts from paste sites and execute memory-resident payloads.", + log_types=["WINEVTLOG", "PROCESS_EXECUTION"] +) + +AI: Generated 2 Threat Detection Opportunities: +1. t01: Obfuscated PowerShell execution with WebClient DownloadString +2. t02: Memory-resident payload invocation via reflective loading + +AI uses: generate_synthetic_events( + threat_detection_opportunities=[ + {"threat_detection_opportunity_id": "t01", "summary": "Obfuscated PowerShell DownloadString", "log_types": ["WINEVTLOG", "PROCESS_EXECUTION"]}, + {"threat_detection_opportunity_id": "t02", "summary": "Reflective PE loading", "log_types": ["PROCESS_EXECUTION"]} + ] +) + +AI: Generated 8 synthetic events simulating the adversary techniques across PowerShell event logs and process creation telemetry. + +AI uses: evaluate_rule_coverage_long_running( + threat_detection_opportunity_events=[...], + exclude_composite_coverage=True +) + +AI: Initiated Long-Running Operation: operations/dea-xyz789... + +AI uses: get_operation( + name="operations/dea-xyz789..." +) + +AI: Evaluation completed. Coverage results: +- t01 (PowerShell DownloadString): COVERED by rule "win_powershell_downloadstring_webclient" +- t02 (Reflective PE loading): UNCOVERED (coverage gap identified) + +AI uses: generate_rules( + threat_detection_opportunities=[ + {"threat_detection_opportunity_id": "t02", "summary": "Reflective PE loading via suspicious memory allocation"} + ], + background_context="Enterprise Windows environment with Sysmon Event ID 7 and 8 telemetry." +) + +AI: Synthesized candidate YARA-L 2.0 rule: +rule ttp_reflective_pe_memory_loading { + meta: + author = "SecOps ADE" + description = "Detects suspicious memory allocation and reflective DLL injection" + severity = "HIGH" + events: + $e.metadata.event_type = "PROCESS_UNCATEGORIZED" + $e.target.process.file.full_path = /powershell\.exe/ nocase + $e.security_result.action = "ALLOW" + condition: + $e +} + +Would you like to run historical backtesting (`test_rule`) and create this rule in disabled mode for staging? +``` + ## Additional Resources - [Chronicle API Documentation](https://cloud.google.com/chronicle/docs/reference/rest) diff --git a/docs/toc.md b/docs/toc.md index 49b62cb9..4ccef395 100644 --- a/docs/toc.md +++ b/docs/toc.md @@ -2,6 +2,8 @@ * [Development Guide](development_guide.md) * [Usage Guide](usage_guide.md) +* [Agentic Detection Engineering](agentic_detection_engineering.md) +* [Google SecOps Extension](google_secops_extension.md) * [Servers](servers/index.md) * [Remote MCP Server](remote_server.md) * [Google Threat Intelligence](servers/gti_mcp.md)