diff --git a/.ai/skills/upload-custom-integration/SKILL.md b/.ai/skills/upload-custom-integration/SKILL.md new file mode 100644 index 000000000000..b6b3192fae06 --- /dev/null +++ b/.ai/skills/upload-custom-integration/SKILL.md @@ -0,0 +1,362 @@ +--- +name: upload-custom-integration +description: > + Automates uploading a custom integration to a + Cortex Platform tenant using `demisto-sdk upload-custom-integration`. Use + this skill whenever the user asks to "upload a custom integration", + "duplicate an integration and upload it", "push my integration copy to + Platform", or similar. Handles repo sync, demisto-sdk installation, + credential discovery, the `_copy` naming safety convention, and the upload + itself, pausing for human input only where CLI automation is not possible + (API key generation, ambiguous paths, or bypassing safety checks). +--- + +# Skill: Upload Custom Integration to Cortex Platform + +## 1. Agent Role & Hard Constraints + +You are acting as an automation agent for the Cortex Platform custom +integration upload workflow. Follow these constraints strictly: + +- **STRICT COMMAND CONSTRAINT:** Only ever run `demisto-sdk + upload-custom-integration` to perform the actual upload. **Never** use + plain `demisto-sdk upload` for this workflow — it lacks the `_copy` safety + validation. +- **IN-PLACE MUTATION ONLY:** Modify the target integration YAML directly. + Do not create a duplicate directory unless the user explicitly asks for + that instead. +- **`--force-id` GATE:** Only pass `--force-id` if the human explicitly and + unambiguously asks to bypass the `_copy` check in their prompt (e.g., "use + force-id", "bypass the copy check", "I don't want to rename the ID"). Never + infer this on your own. Always show the CLI's own warning text to the user + first (see Step 5). +- **NEVER silently overwrite `content/.env`.** Only append missing keys or + ask the human to fill them in; never delete or comment-out existing lines. +- **PAUSE FOR HUMAN INPUT** wherever a step genuinely requires a browser/UI + action (generating an API key) or an ambiguous decision (multiple YAML + files found, missing target path). Do not guess. + +--- + +## 2. Execution Workflow + +```mermaid +flowchart TD + A[Start] --> B[Step 1: Sync demisto content repo] + B --> C[Step 2: Set up Python venv & install demisto-sdk] + C --> D[Step 3: Verify or collect Cortex Platform credentials] + D --> E[Step 4: Apply the _copy marker to the target integration YAML] + E --> F[Step 5: Run upload-custom-integration] + F --> G[Step 6: Print verification summary] +``` + +--- + +## Step 1 — Repository Discovery & Synchronization + +Determine whether the current working directory is inside a clone of +`demisto/content`: + +```bash +git remote -v +``` + +**If already inside `demisto/content`:** + +```bash +git checkout master +git pull origin master --rebase +``` + +**If NOT inside `demisto/content`:** ask the user for the path to their +existing local clone first (most users already have one — check common +locations like `../content` relative to `demisto-sdk`, or ask). Only clone a +fresh copy if the user confirms they don't have one yet: + +```bash +ssh -T git@github.com 2>&1 # check SSH availability +# if SSH works: +git clone git@github.com:demisto/content.git +# otherwise: +git clone https://github.com/demisto/content.git +cd content +``` + +--- + +## Step 2 — Python Environment & demisto-sdk Dependency Check + +Assume a brand-new machine with nothing installed yet. `demisto-sdk` must +run in an **activated virtual environment** (or be otherwise available on +`PATH`) — never rely on a system-wide `pip install` without one, since that +can conflict with other Python tooling on the machine. + +### 2.1 Check for an existing virtual environment + +The `content` repo root contains a [`pyproject.toml`](../../../pyproject.toml) +and a [`poetry.toml`](../../../poetry.toml) with `in-project = true`, meaning +Poetry (if used) creates its virtualenv at `content/.venv`. Check what's +already there before creating anything new: + +```bash +# Is Poetry installed? +poetry --version + +# Is there already an in-project venv? +ls -d .venv 2>/dev/null && echo "Found existing .venv" +``` + +### 2.2 Decision logic — which path to take + +**If Poetry is installed (preferred — matches the repo's own tooling):** + +```bash +# Only if python --version reports < 3.10 or > 3.14 (pyproject.toml requires >=3.10,<3.15) +python3 --version + +# Install project dependencies into an in-project .venv (creates it if missing) +poetry install --with dev + +# Activate the venv for the current shell session +source .venv/bin/activate +``` + +**If Poetry is not installed and the user doesn't want to install it,** fall +back to a plain `venv` + `pip`: + +```bash +python3 -m venv .venv +source .venv/bin/activate +pip install --upgrade pip +pip install --upgrade demisto-sdk +``` + +> ⚠️ On Windows (non-WSL) shells, activation is +> `.venv\Scripts\activate` instead of `source .venv/bin/activate`. + +### 2.3 Confirm the venv is actually active before continuing + +Do not just assume activation succeeded — verify it: + +```bash +which python # should point inside content/.venv/bin/python +which demisto-sdk # should also resolve inside the same .venv +``` + +If either command resolves to a path **outside** `.venv`, the environment is +not active — stop and re-run the activation command from 2.2 before +proceeding. Every subsequent step in this skill (Step 5 in particular) must +be run from this same activated shell/session. + +### 2.4 Verify demisto-sdk itself + +```bash +demisto-sdk --version +``` + +**Decision logic:** + +- If `demisto-sdk` is **not installed**, or the reported version is **below + 1.39.10**, upgrade to the latest release from PyPI (inside the active venv): + + ```bash + pip install --upgrade demisto-sdk + ``` + +- After installing/upgrading, **verify the subcommand is actually + registered** (this is the real signal that the feature is available, + more reliable than the version number alone): + + ```bash + demisto-sdk upload-custom-integration --help + ``` + +--- + +## Step 3 — Cortex Platform Credentials (Human-in-the-Loop) + +The `upload-custom-integration` command reads `DEMISTO_BASE_URL`, +`DEMISTO_API_KEY`, and `XSIAM_AUTH_ID` from the environment. In this +workflow, they are expected to live in **`content/.env`**, which +`demisto-sdk` automatically loads on every command invocation. + +### 3.1 Locate and parse `content/.env` + +- Look for `.env` in the root of the `content` repo clone. +- If it does not exist, create an empty one. +- Parse the file, but **only consider active (non-commented) lines** — + ignore any line starting with `#`. + **Never uncomment, delete, or reorder these historical + profiles.** +- From the active (uncommented) lines only, extract current values (if any) + for: + - `DEMISTO_BASE_URL` + - `DEMISTO_API_KEY` + - `XSIAM_AUTH_ID` + +### 3.2 If all three active values are present and non-empty + +Proceed directly to Step 4 — no human interaction needed. + +### 3.3 If any value is missing or empty + +Pause and display this exact guidance to the user (this cannot be automated +because it requires UI interaction in the Cortex Platform console): + +``` +📋 [HUMAN ACTION REQUIRED]: Platform Credentials Configuration + +To allow demisto-sdk to authenticate with your Cortex Platform tenant, +please perform the following steps in your web console: + +1. Obtain API Base URL (DEMISTO_BASE_URL): + - Log in to your Cortex Platform instance. + - Navigate to: Settings → Configurations → API Keys. + - Click "Copy API URL" in the top right corner. + +2. Generate API Key (DEMISTO_API_KEY): + - Click "New Key" (top right corner). + - Set Key Type to "Standard". + - Set Role to "Instance Administrator". + - Click Generate and COPY THE KEY IMMEDIATELY from the modal popup + (it will not be shown again). + +3. Obtain Auth ID (XSIAM_AUTH_ID): + - Close the key popup to view the API Keys table. + - Find the row for your newly created key (check the "Created By" column). + - Copy the numeric value in the "ID" column (e.g., 4). + +4. Paste these three values back to me, or add them directly to + content/.env as a new (uncommented) block: + + DEMISTO_BASE_URL=https://api-your-tenant-url... + DEMISTO_API_KEY=your_copied_api_key_here + XSIAM_AUTH_ID=your_key_id_number_here + DEMISTO_VERIFY_SSL=False +``` + +Wait for the user's response. Once provided, append the new values as a new +block at the bottom of `content/.env` (do not touch existing lines). + +### 3.4 Verify before proceeding + +Re-parse `content/.env` and confirm `DEMISTO_BASE_URL`, `DEMISTO_API_KEY`, +and `XSIAM_AUTH_ID` are all present and non-empty before moving to Step 4. + +--- + +## Step 4 — Apply the `_copy` Marker to the Target Integration YAML + +1. **Resolve the target path.** Ask the user for the integration path if not + already given (e.g., `Packs/PolarSecurity/Integrations/PolarSecurity`). + - `upload-custom-integration` already accepts either a `.yml` file or a + directory (it resolves the single YAML inside automatically). You do + not need to duplicate that resolution logic — but if a directory + contains **more than one** `.yml` file, ask the user which one to + target rather than guessing. + +2. **Read the current values.** Parse the YAML (`ruamel.yaml` or `PyYAML`, + preserving formatting/comments if possible) and read: + - `commonfields.id` (fallback: top-level `id` for unified YAMLs) + - `name` + +3. **Apply the `_copy` suffix in place, only where missing:** + - If `commonfields.id` (or `id`) does not already end with `_copy`, + rename it to `{original_id}_copy`. + - If `name` does not already end with `_copy`, rename it to + `{original_name}_copy`. + - If a field **already** ends with `_copy`, leave it untouched (do not + double-suffix, e.g. never produce `Foo_copy_copy`). + - It is also strongly recommended (per the CLI's own help text) to + likewise suffix the `display` field with `_copy`, to avoid UI confusion + with the original marketplace integration. Apply the same + already-suffixed check before mutating it. + +4. **Write the file back to disk**, mutating the **original file the user + pointed to** — no new directory or file is created. + +#### Example transformation + +```yaml +# BEFORE +commonfields: + id: PolarSecurity +name: Polar Security +display: Polar Security + +# AFTER +commonfields: + id: PolarSecurity_copy +name: Polar Security_copy +display: Polar Security_copy +``` + +--- + +## Step 5 — Upload via `upload-custom-integration` + +Run this from the **same activated virtual environment/shell session** +established in Step 2 (`which demisto-sdk` should still resolve inside +`.venv`). If a new terminal was opened since then, re-activate first +(`source .venv/bin/activate`). + +Run: + +```bash +demisto-sdk upload-custom-integration -i +``` + +The CLI itself will re-validate the `_copy` marker and fail with a clear +error if Step 4 was somehow skipped or incomplete — treat that as a signal +to re-check Step 4, not as a reason to add `--force-id`. + +### Handling `--force-id` (only on explicit human request) + +Only if the user has **explicitly** asked, in their own words, to bypass the +`_copy` requirement: + +1. Show them the CLI's own built-in warning verbatim (do not paraphrase): + > Uploading a custom integration whose ID matches a marketplace + > integration ID will cause subsequent installations of the pack that + > contains that integration to fail with a system error. + > + > ACTION REQUIRED — before continuing, verify ALL of the following: + > 1. Your chosen ID is completely unique and does NOT match the original + > integration ID. + > 2. Your chosen ID does NOT match any other integration ID already + > present in the repository. + > 3. Your chosen ID does NOT match any integration ID published on the + > Marketplace. +2. Get explicit confirmation from the user that they've verified the above. +3. Only then run: + + ```bash + demisto-sdk upload-custom-integration -i --force-id + ``` + +--- + +## Step 6 — Post-Execution Summary Report + +On success, print a structured summary: + +``` +🚀 Custom Integration Upload Summary + +- Target Integration File: +- Custom Integration ID: +- Custom Display Name: +- Target Platform Tenant: +- Executed Command: demisto-sdk upload-custom-integration +- Status: SUCCESS ✅ + +Next Steps: +1. Log in to your Cortex Platform instance. +2. Go to Settings → Configuration → Data Sources & Integrations → Add New. +3. Locate your integration (e.g., ). It will display a + "Custom" badge next to its title. +4. Click Add to configure and use your custom duplicate instance. +``` + +On failure, surface the CLI's error output verbatim and stop — do not retry +with `--force-id` unless the human explicitly asks after seeing the error. diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index 7afe27e19d29..86c5907623f0 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -23,8 +23,6 @@ /Packs/QRadar/Integrations/QRadar_v3/* @ilappe /Packs/Slack/Integrations/* @amshamah419 @rshunim /Packs/SplunkPy/Integrations/SplunkPy/* @ilappe -/Packs/MicrosoftExchangeOnPremise/Integrations @amshamah419 -/Packs/MicrosoftExchangeOnline/Integrations @amshamah419 /Packs/ContentManagement/* @DeanArbel @kamalq97 /Packs/TAXIIServer/Integrations/TAXII2Server/* @Ni-Knight /Packs/FeedTAXII/Integrations/FeedTAXII2/* @Ni-Knight @@ -33,8 +31,8 @@ # Important Scripts /Packs/CommonScripts/Scripts/SetGridField/* @melamedbn /Packs/CommonScripts/Scripts/ParseEmailFiles/* @moishce -/Packs/ApiModules/Scripts/MicrosoftApiModule @michal-dagan @inbalapt1 @noydavidi -/Packs/ApiModules/Scripts/MicrosoftGraphMailApiModule @michal-dagan @inbalapt1 @noydavidi +/Packs/ApiModules/Scripts/MicrosoftApiModule @noydavidi +/Packs/ApiModules/Scripts/MicrosoftGraphMailApiModule @noydavidi /Packs/ApiModules/Scripts/EWSApiModule @noydavidi /Packs/ApiModules/Scripts/TAXII2ApiModule/* @Ni-Knight /Packs/CommonScripts/Scripts/StixCreator/* @Ni-Knight @@ -61,9 +59,9 @@ validation_config.toml @yedidyacohenpalo # XDR Related -/Packs/CortexXDR/Integrations/ @jlevypaloalto -/Packs/Core/Integrations/ @jlevypaloalto -/Packs/ApiModules/Scripts/CoreIRApiModule/* @jlevypaloalto +/Packs/CortexXDR/Integrations/ @jlevypaloalto @Shellyber +/Packs/Core/Integrations/ @jlevypaloalto @Shellyber +/Packs/ApiModules/Scripts/CoreIRApiModule/* @jlevypaloalto @Shellyber # Demisto Class Packs/ApiModules/Scripts/DemistoClassApiModule/DemistoClassApiModule.py @dantavori @@ -77,27 +75,6 @@ Packs/ApiModules/Scripts/DemistoClassApiModule/DemistoClassApiModule.py @dantavo /Packs/ApiModules/Scripts/TAXII2ApiModule/* @MosheEichler /Packs/ApiModules/Scripts/NGINXApiModule/* @MosheEichler -# ML scripts -/Packs/ML/ @jlevypaloalto -/Packs/PhishingURL/ @jlevypaloalto -/Packs/Phishing/Scripts/PhishingDedupPreprocessingRule @jlevypaloalto -/Packs/Phishing/Scripts/FindDuplicateEmailIncidents @jlevypaloalto -/Packs/Base/Scripts/DrawRelatedIncidentsCanvas@jlevypaloalto -/Packs/Campaign/Scripts/FindEmailCampaign/ @jlevypaloalto -/Packs/CortexXDR/Scripts/DBotGroupXDRIncidents/ @jlevypaloalto -/Packs/Base/Scripts/FindSimilarIncidentsByText/ @jlevypaloalto -/Packs/Base/Scripts/DBotTrainTextClassifierV2/ @jlevypaloalto -/Packs/Base/Scripts/DBotShowClusteringModelInfo/ @jlevypaloalto -/Packs/Base/Scripts/DBotPredictPhishingWords/ @jlevypaloalto -/Packs/Base/Scripts/DBotFindSimilarIncidents/ @jlevypaloalto -/Packs/Base/Scripts/DBotTrainClustering/ @jlevypaloalto -/Packs/Base/Scripts/DBotBuildPhishingClassifier/ @jlevypaloalto -/Packs/Base/Scripts/DBotPreprocessTextData/ @jlevypaloalto -/Packs/Base/Scripts/DBotFindSimilarIncidentsByIndicators/ @jlevypaloalto -/Packs/Base/Scripts/DBotSuggestClassifierMapping/ @jlevypaloalto -/Packs/Base/Scripts/GetMLModelEvaluation/ @jlevypaloalto -/Packs/Base/Scripts/DBotMLFetchData/ @jlevypaloalto - # Platform Automation /Packs/AzureDevOps/ @omerKarkKatz /Packs/Jira/ @omerKarkKatz @@ -105,12 +82,11 @@ Packs/ApiModules/Scripts/DemistoClassApiModule/DemistoClassApiModule.py @dantavo /Packs/MicrosoftTeams/ @omerKarkKatz /Packs/Slack/ @omerKarkKatz -/Packs/PingIdentity @Shellyber -/Packs/Okta @Shellyber +/Packs/Okta/Integrations/Okta_v2 @Shellyber /Packs/Core @Shellyber /Packs/CortexXDR @Shellyber /Packs/SentinelOne @Shellyber -/Packs/CrowdStrikeFalcon @Shellyber @talzich @JasBeilin +/Packs/CrowdStrikeFalcon @Shellyber @talzich /Packs/CrowdStrikeFalconSandbox @Shellyber /Packs/CrowdStrikeFalconStreamingV2 @Shellyber /Packs/MicrosoftExchangeOnline @Shellyber @noydavidi diff --git a/.github/content_roles.json b/.github/content_roles.json index 73f19db3a454..95157b1832de 100644 --- a/.github/content_roles.json +++ b/.github/content_roles.json @@ -12,7 +12,8 @@ ], "CONTRIBUTION_TL": "kamalq97", "CONTRIBUTION_SECURITY_REVIEWER": ["Benimanela"], - "MAPPING_REVIEWER": ["eepstain"], + "CONTRIBUTION_DOCKERFILES_REVIEWER": "shmuel44", + "MAPPING_REVIEWER": ["OBavly"], "ON_CALL_DEVS": [ "jbabazadeh", "YairGlick", diff --git a/.github/github_workflow_scripts/check_if_connectus_approved_label_exists.py b/.github/github_workflow_scripts/check_if_connectus_approved_label_exists.py new file mode 100644 index 000000000000..f517ac4ffb36 --- /dev/null +++ b/.github/github_workflow_scripts/check_if_connectus_approved_label_exists.py @@ -0,0 +1,320 @@ +""" +Verifies that PRs which add a new XSOAR-supported integration, or modify the +`configuration:` block of an existing one, carry the `connectus-approved` label +before they can be merged. The check: + + 1. Detects integration YML files touched by the PR whose owning pack is + XSOAR-supported (``"support": "xsoar"`` in ``pack_metadata.json``). + 2. Flags the PR when such a file is newly added, or when the `configuration:` + block of an existing one differs between the PR base and head. Renamed and + copied files are compared against their pre-rename path, so a pure move + that leaves `configuration:` untouched does not trigger the gate. + 3. If the PR is flagged, ensures the `connectus-approved` label is set (added + once the Connectus approval flow has signed off on the change). + +Checks that cannot be completed - unreadable pack metadata, a YML that cannot be +fetched at either ref, or a failure to list the PR's files - fail safe: the +change is treated as gated. A transient API error must never silently let an +unapproved configuration change through. + +Exit codes: + 0 - No triggering change OR the `connectus-approved` label is set. + 1 - Triggering change exists but the `connectus-approved` label is missing. +""" + +import argparse +import json +import sys +from pathlib import PurePosixPath +from typing import Any + +import urllib3 +import yaml +from github import Github +from github.PullRequest import PullRequest +from github.Repository import Repository + +from utils import timestamped_print + +urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning) +print = timestamped_print # noqa: A001 - intentional, matches existing scripts + +CONNECTUS_APPROVED_LABEL = "connectus-approved" + +PACKS_DIR = "Packs" +INTEGRATIONS_DIR = "Integrations" +PACK_METADATA_FILE = "pack_metadata.json" +XSOAR_SUPPORT = "xsoar" +CONFIGURATION_KEY = "configuration" +YML_SUFFIXES = (".yml", ".yaml") + +ADDED_STATUS = "added" +# GitHub reports a plain content edit as "modified"; "changed" is the rarer +# variant it emits for files whose mode/type changed alongside the content. +MODIFIED_STATUSES = ("modified", "changed") +# Both carry `previous_filename`, pointing at the path the content had at base. +RENAMED_STATUSES = ("renamed", "copied") + + +def arguments_handler() -> argparse.Namespace: + """Validates and parses script arguments. + + Returns: + Namespace: Parsed arguments object. + """ + parser = argparse.ArgumentParser(description=f"Check if {CONNECTUS_APPROVED_LABEL} label exists.") + parser.add_argument("-p", "--pr_number", required=True, help="The PR number to check if the label exists.") + parser.add_argument("-g", "--github_token", required=True, help="The GitHub token to authenticate the GitHub client.") + return parser.parse_args() + + +def is_integration_yml(file_path: str) -> bool: + """ + Return True if `file_path` is an integration's entity YML file, i.e. it matches + ``Packs//Integrations//.yml``. + + Matching on the full path shape (rather than "any YML under Integrations/") + deliberately excludes companion files that live in the same folder but are + not the integration definition - README.md, *_test.py, CHANGELOG.md, and + test-data YAMLs - none of which can contain a `configuration:` block. + + ``PurePosixPath`` is used because filenames come from GitHub's PR API and are + always forward-slash separated, regardless of the runner's OS. + """ + path = PurePosixPath(file_path) + parts = path.parts + + if len(parts) != 5: + return False + if parts[0] != PACKS_DIR or parts[2] != INTEGRATIONS_DIR: + return False + if path.suffix.lower() not in YML_SUFFIXES: + return False + # The YML must be named after its containing integration directory. + return path.stem == parts[3] + + +def get_pack_name(file_path: str) -> str: + """Return the pack name for a ``Packs//...`` path (empty string if not under Packs).""" + parts = PurePosixPath(file_path).parts + if len(parts) < 2 or parts[0] != PACKS_DIR: + return "" + return parts[1] + + +def get_head_repo(pr: PullRequest) -> Repository: + """ + Return the repository that hosts the PR's head commits. + + For a PR opened from a fork this is the fork, not the upstream repo - the + head SHA simply does not exist upstream, so reading head content from + ``pr.base.repo`` would 404 and silently skip the file. ``pr.head.repo`` is + None when the fork has been deleted; fall back to the base repo so we + degrade to the previous behaviour rather than raising. + """ + return pr.head.repo or pr.base.repo + + +def get_file_content_at_ref(repo: Repository, file_path: str, ref: str) -> str | None: + """ + Fetch the raw text content of `file_path` at the given git `ref` from `repo`. + + The caller must pass the repository that actually contains `ref`: the base + repo for `pr.base.sha`, and :func:`get_head_repo` for `pr.head.sha`. + + Returns None when the file does not exist at that ref (e.g. a newly added + file has no base version) or cannot be read. + """ + try: + contents = repo.get_contents(file_path, ref=ref) + except Exception as e: + print(f"Warning: Could not fetch '{file_path}' at ref {ref}: {e}") + return None + + if isinstance(contents, list): # path is a directory - not something we can parse + print(f"Warning: '{file_path}' at ref {ref} is a directory, skipping.") + return None + + try: + return contents.decoded_content.decode("utf-8") + except Exception as e: + print(f"Warning: Could not decode '{file_path}' at ref {ref}: {e}") + return None + + +def parse_yml(content: str, file_path: str) -> dict[str, Any] | None: + """Parse YAML `content` into a dict, returning None if it is malformed or not a mapping.""" + try: + parsed = yaml.safe_load(content) + except yaml.YAMLError as e: + print(f"Warning: Failed to parse YAML file {file_path}: {e}") + return None + return parsed if isinstance(parsed, dict) else None + + +def is_xsoar_supported_pack(pr: PullRequest, pack_name: str, support_cache: dict[str, bool]) -> bool: + """ + Return True if `pack_name`'s metadata declares ``"support": "xsoar"``. + + The metadata is read at the PR head SHA so that a pack newly added (or + re-leveled) by this very PR is evaluated against its proposed state. + Results are cached per pack so a PR touching several integrations in one + pack costs a single API call. + + Fail-safe: when the metadata cannot be fetched or parsed we cannot rule the + pack out, so it is treated as XSOAR-supported. Assuming "not supported" + would let a gated change through on a transient API error, whereas assuming + "supported" merely asks for a label on a pack that may not have needed one. + Do not flip this default. + """ + if pack_name in support_cache: + return support_cache[pack_name] + + supported = True + metadata_path = f"{PACKS_DIR}/{pack_name}/{PACK_METADATA_FILE}" + content = get_file_content_at_ref(get_head_repo(pr), metadata_path, pr.head.sha) + + if content is None: + print(f"Warning: Could not read {metadata_path}; treating pack '{pack_name}' as XSOAR-supported to be safe.") + else: + try: + metadata = json.loads(content) + supported = metadata.get("support") == XSOAR_SUPPORT + except (json.JSONDecodeError, AttributeError) as e: + print(f"Warning: Failed to parse {metadata_path}: {e}; treating pack '{pack_name}' as XSOAR-supported to be safe.") + + support_cache[pack_name] = supported + return supported + + +def get_configuration(parsed_yml: dict[str, Any] | None) -> Any: + """Return the `configuration` block of a parsed integration YML (empty list when absent).""" + if not parsed_yml: + return [] + return parsed_yml.get(CONFIGURATION_KEY, []) + + +def is_configuration_modified(pr: PullRequest, file_path: str, base_path: str | None = None) -> bool: + """ + Return True if the `configuration:` block of `file_path` differs between the + PR base and head. + + Comparing the parsed blocks (rather than the raw diff) means cosmetic YAML + changes elsewhere in the file - a `dockerimage` bump, a reworded + `description` - correctly do NOT trigger the gate, while any real change to + a configuration param does. + + `base_path` is the path the content had at the base ref. It differs from + `file_path` only for renamed/copied files and defaults to `file_path`. + + Fail-safe: if either side cannot be fetched the comparison is undecidable, + so True is returned - an unverifiable change still requires approval. + """ + base_path = base_path or file_path + base_content = get_file_content_at_ref(pr.base.repo, base_path, pr.base.sha) + head_content = get_file_content_at_ref(get_head_repo(pr), file_path, pr.head.sha) + + if base_content is None or head_content is None: + print(f"Warning: Could not compare '{file_path}' between base and head; requiring approval to be safe.") + return True + + base_configuration = get_configuration(parse_yml(base_content, file_path)) + head_configuration = get_configuration(parse_yml(head_content, file_path)) + + return base_configuration != head_configuration + + +def check_pr_contains_connectus_changes(pr: PullRequest) -> list[str]: + """ + Return a list of human-readable reasons why this PR requires Connectus approval + (empty when it does not). + + A reason is recorded when an integration YML in an XSOAR-supported pack is + either newly added, or has had its `configuration:` block modified. A rename + is resolved against the file's pre-rename path, so moving an integration + without touching `configuration:` is not a reason on its own. + """ + reasons: list[str] = [] + support_cache: dict[str, bool] = {} + + try: + files = list(pr.get_files()) + except Exception as e: + # Fail safe: if we cannot inspect the PR we require the label rather than + # silently letting a potentially-gated change through. + print(f"Error listing PR files: {e}. Requiring the label to be safe.") + return [f"- Could not inspect the PR's files ({e}); requiring '{CONNECTUS_APPROVED_LABEL}' to be safe."] + + for file in files: + if not is_integration_yml(file.filename): + continue + + pack_name = get_pack_name(file.filename) + if not is_xsoar_supported_pack(pr, pack_name, support_cache): + print(f"Skipping '{file.filename}': pack '{pack_name}' is not {XSOAR_SUPPORT}-supported.") + continue + + try: + if file.status == ADDED_STATUS: + reasons.append(f"- New XSOAR-supported integration added: {file.filename}") + elif file.status in RENAMED_STATUSES: + previous_filename = getattr(file, "previous_filename", None) + if not previous_filename or not is_integration_yml(previous_filename): + # Renamed from something that was not an integration definition + # (or GitHub did not tell us where from): the integration YML + # appears at this path for the first time, so treat it as new. + reasons.append(f"- New XSOAR-supported integration added: {file.filename}") + elif is_configuration_modified(pr, file.filename, base_path=previous_filename): + reasons.append( + "- Modified 'configuration' of XSOAR-supported integration: " + f"{file.filename} (renamed from {previous_filename})" + ) + elif file.status in MODIFIED_STATUSES and is_configuration_modified(pr, file.filename): + reasons.append(f"- Modified 'configuration' of XSOAR-supported integration: {file.filename}") + except Exception as e: + print(f"Warning: Error processing {file.filename}: {e}") + continue + + return reasons + + +def main() -> None: + """ + Checks that the "connectus-approved" label exists on a PR when required. If the + label exists the workflow passes; if it is missing (and required) the workflow fails. + """ + options = arguments_handler() + pr_number = int(options.pr_number) + + github_client: Github = Github(options.github_token, verify=False) + content_repo: Repository = github_client.get_repo("demisto/content") + pr: PullRequest = content_repo.get_pull(pr_number) + + pr_label_names = {label.name for label in pr.labels} + + print(f"Checking if {CONNECTUS_APPROVED_LABEL} label exist in PR {pr_number}") + if CONNECTUS_APPROVED_LABEL in pr_label_names: + print(f"SUCCESS: PR #{pr_number} has the required label: {CONNECTUS_APPROVED_LABEL}") + sys.exit(0) + + reasons = check_pr_contains_connectus_changes(pr) + if not reasons: + print( + "INFO: PR does not add a new XSOAR-supported integration or modify integration configuration params.\n" + f" The '{CONNECTUS_APPROVED_LABEL}' label is not required for this PR." + ) + sys.exit(0) + + reasons_text = "\n".join(reasons) + print( + f"ERROR: Required label '{CONNECTUS_APPROVED_LABEL}' is missing from PR #{pr_number}.\n" + " The following changes require Connectus approval:\n" + f"{reasons_text}\n" + " Please ask the Connectus team to review the changes and add the " + f"'{CONNECTUS_APPROVED_LABEL}' label once approved." + ) + sys.exit(1) + + +if __name__ == "__main__": + main() diff --git a/.github/github_workflow_scripts/create_internal_pr.py b/.github/github_workflow_scripts/create_internal_pr.py index 05d99721395c..48e18fa486c9 100755 --- a/.github/github_workflow_scripts/create_internal_pr.py +++ b/.github/github_workflow_scripts/create_internal_pr.py @@ -237,7 +237,7 @@ def remove_branch_protection(repo, branch_name, t): def prepare_body(pr: PullRequest): - body = f"## Original External PR\r\n[external pull request]({pr.html_url})\r\n\r\n" + body = f"## Related PRs\r\n**Original External PR:** {pr.html_url}\r\n\r\n" if "## Contributor" not in pr.body: merged_pr_author = pr.user.login body += f"## Contributor\r\n@{merged_pr_author}\r\n\r\n" @@ -376,6 +376,26 @@ def main(): except Exception as e: print(f"{t.red}Mapping PR failed: {e}{t.normal}") + # Cross-reference main and mapping PRs when both were created from the same external PR + if len(created) == 2: + main_pr, mapping_pr = created[0], created[1] + try: + main_pr.edit( + body=main_pr.body.replace( + f"**Original External PR:** {merged_pr.html_url}", + f"**Original External PR:** {merged_pr.html_url}\r\n**Mapping Internal PR:** {mapping_pr.html_url}", + ) + ) + mapping_pr.edit( + body=mapping_pr.body.replace( + f"**Original External PR:** {merged_pr.html_url}", + f"**Original External PR:** {merged_pr.html_url}\r\n**Main Internal PR:** {main_pr.html_url}", + ) + ) + print(f"{t.cyan}Cross-referenced main PR #{main_pr.number} and mapping PR #{mapping_pr.number}{t.normal}") + except Exception as e: + print(f"{t.red}Failed to cross-reference PRs: {e}{t.normal}") + for pr in created: remove_branch_protection(content_repo, pr.head.ref, t) diff --git a/.github/github_workflow_scripts/github_workflow_scripts_tests/check_if_connectus_approved_label_test.py b/.github/github_workflow_scripts/github_workflow_scripts_tests/check_if_connectus_approved_label_test.py new file mode 100644 index 000000000000..8e5d074a9de1 --- /dev/null +++ b/.github/github_workflow_scripts/github_workflow_scripts_tests/check_if_connectus_approved_label_test.py @@ -0,0 +1,724 @@ +""" +Unit tests for ``github_workflow_scripts/check_if_connectus_approved_label_exists.py``. + +The script talks to the GitHub API via the ``PyGithub`` library; for unit +testing we never hit the network - we hand-build lightweight mock objects that +expose only the attributes the script actually reads. +""" + +import json + +import pytest + +MODULE = "github_workflow_scripts.check_if_connectus_approved_label_exists" + +XSOAR_METADATA = json.dumps({"name": "MyPack", "support": "xsoar"}) +PARTNER_METADATA = json.dumps({"name": "MyPack", "support": "partner"}) + +INTEGRATION_YML = "Packs/MyPack/Integrations/MyIntegration/MyIntegration.yml" +OLD_INTEGRATION_YML = "Packs/MyPack/Integrations/OldIntegration/OldIntegration.yml" +PACK_METADATA = "Packs/MyPack/pack_metadata.json" + + +def integration_yml(configuration: str, dockerimage: str = "demisto/python3:3.10.1") -> str: + """Build a minimal integration YML with the supplied ``configuration`` block.""" + return ( + "commonfields:\n" + " id: MyIntegration\n" + "name: MyIntegration\n" + f"configuration:\n{configuration}" + "script:\n" + " type: python\n" + f" dockerimage: {dockerimage}\n" + ) + + +ONE_PARAM = "- display: Server URL\n name: url\n type: 0\n required: true\n" +TWO_PARAMS = ONE_PARAM + "- display: API Key\n name: apikey\n type: 4\n required: true\n" + + +# --------------------------------------------------------------------------- +# Tiny stand-in objects (no PyGithub dependency at test-collection time). +# --------------------------------------------------------------------------- + + +class _MockFile: + def __init__(self, filename: str, status: str = "modified", previous_filename: str | None = None): + self.filename = filename + self.status = status + # PyGithub only exposes `previous_filename` on renamed/copied entries, so + # the attribute is left off entirely when no rename source is supplied. + if previous_filename is not None: + self.previous_filename = previous_filename + + +class _MockLabel: + def __init__(self, name: str): + self.name = name + + +class _MockContentFile: + def __init__(self, content: str): + self.decoded_content = content.encode("utf-8") + + +class _MockRepo: + """Serves file contents keyed by ``(path, ref)``; raises when the pair is unknown.""" + + def __init__(self, contents: dict[tuple[str, str], str] | None = None): + self._contents = contents or {} + + def get_contents(self, path: str, ref: str): + try: + return _MockContentFile(self._contents[(path, ref)]) + except KeyError: + raise Exception(f"404: {path}@{ref} not found") + + +class _MockRef: + def __init__(self, sha: str, repo: _MockRepo | None = None): + self.sha = sha + self.repo = repo + + +class _MockPullRequest: + """Minimal stand-in matching the surface area used by the script under test.""" + + BASE_SHA = "base-sha" + HEAD_SHA = "head-sha" + + def __init__( + self, + number: int = 42, + files=None, + labels=None, + contents=None, + files_raise: bool = False, + head_contents=None, + head_repo_deleted: bool = False, + ): + """ + `contents` populates the base repo. `head_contents`, when given, puts the + head content in a *separate* repo reachable only via ``pr.head.repo``, + which is how GitHub models a PR opened from a fork. When it is omitted + both refs are served by the base repo (same-repo PR). + """ + self.number = number + self._files = files or [] + self._files_raise = files_raise + self.labels = labels or [] + self.base = _MockRef(self.BASE_SHA, _MockRepo(contents)) + if head_repo_deleted: + # GitHub reports head.repo as None once the source fork is deleted. + self.head = _MockRef(self.HEAD_SHA, None) + else: + head_repo = _MockRepo(head_contents) if head_contents is not None else self.base.repo + self.head = _MockRef(self.HEAD_SHA, head_repo) + + def get_files(self): + if self._files_raise: + raise Exception("GitHub API is down") + return iter(self._files) + + +def _contents(base_yml: str | None = None, head_yml: str | None = None, metadata: str = XSOAR_METADATA) -> dict: + """Build the ``(path, ref) -> content`` map for a single-integration PR.""" + contents: dict[tuple[str, str], str] = {(PACK_METADATA, _MockPullRequest.HEAD_SHA): metadata} + if base_yml is not None: + contents[(INTEGRATION_YML, _MockPullRequest.BASE_SHA)] = base_yml + if head_yml is not None: + contents[(INTEGRATION_YML, _MockPullRequest.HEAD_SHA)] = head_yml + return contents + + +def _run_main(mocker, pr: _MockPullRequest) -> int: + """Run the script's ``main()`` against ``pr`` and return its exit code.""" + from github_workflow_scripts import check_if_connectus_approved_label_exists as script + + mock_repo = mocker.MagicMock() + mock_repo.get_pull.return_value = pr + mocker.patch.object(script, "Github", return_value=mocker.MagicMock(get_repo=lambda _: mock_repo)) + mocker.patch.object( + script, + "arguments_handler", + return_value=mocker.MagicMock(pr_number=str(pr.number), github_token="token"), + ) + + with pytest.raises(SystemExit) as exit_info: + script.main() + return exit_info.value.code + + +# --------------------------------------------------------------------------- +# is_integration_yml +# --------------------------------------------------------------------------- + + +@pytest.mark.parametrize( + "file_path, expected", + [ + # The integration entity YML - the only shape we gate on. + (INTEGRATION_YML, True), + ("Packs/MyPack/Integrations/MyIntegration/MyIntegration.yaml", True), + # Companion files in the same folder are not the integration definition. + ("Packs/MyPack/Integrations/MyIntegration/MyIntegration.py", False), + ("Packs/MyPack/Integrations/MyIntegration/README.md", False), + ("Packs/MyPack/Integrations/MyIntegration/MyIntegration_test.py", False), + # A YML in the integration folder not named after it (e.g. test data). + ("Packs/MyPack/Integrations/MyIntegration/test_data/response.yml", False), + ("Packs/MyPack/Integrations/MyIntegration/command_examples.yml", False), + # Other content entities must not trigger the gate. + ("Packs/MyPack/Scripts/MyScript/MyScript.yml", False), + ("Packs/MyPack/Playbooks/MyPlaybook.yml", False), + ("Packs/MyPack/ModelingRules/MyRule/MyRule.yml", False), + ("Packs/MyPack/pack_metadata.json", False), + # Paths outside Packs/. + (".github/workflows/connectus-approved.yml", False), + ("Tests/conf.json", False), + ], +) +def test_is_integration_yml(file_path, expected): + """ + Given: + - A file path from a PR's changed-files list. + + When: + - Running is_integration_yml. + + Then: + - Only ``Packs//Integrations//.yml`` is recognised as + an integration definition. + """ + from github_workflow_scripts.check_if_connectus_approved_label_exists import is_integration_yml + + assert is_integration_yml(file_path) is expected + + +@pytest.mark.parametrize( + "file_path, expected", + [ + (INTEGRATION_YML, "MyPack"), + ("Packs/OtherPack/pack_metadata.json", "OtherPack"), + ("Tests/conf.json", ""), + ("README.md", ""), + ], +) +def test_get_pack_name(file_path, expected): + """ + Given: + - A file path. + + When: + - Running get_pack_name. + + Then: + - The owning pack name is returned, or an empty string outside Packs/. + """ + from github_workflow_scripts.check_if_connectus_approved_label_exists import get_pack_name + + assert get_pack_name(file_path) == expected + + +# --------------------------------------------------------------------------- +# The four mandated end-to-end scenarios +# --------------------------------------------------------------------------- + + +def test_new_xsoar_integration_without_label_fails(mocker, capsys): + """ + Given: + - A PR that adds a new integration YML in an XSOAR-supported pack. + - The 'connectus-approved' label is missing. + + When: + - Running the script's main function. + + Then: + - The check fails (exit code 1) and the log names the offending file. + """ + pr = _MockPullRequest( + files=[_MockFile(INTEGRATION_YML, status="added")], + contents=_contents(head_yml=integration_yml(ONE_PARAM)), + ) + + assert _run_main(mocker, pr) == 1 + + output = capsys.readouterr().out + assert "ERROR" in output + assert "connectus-approved" in output + assert "New XSOAR-supported integration added" in output + assert INTEGRATION_YML in output + + +def test_modified_configuration_without_label_fails(mocker, capsys): + """ + Given: + - A PR that modifies the 'configuration' block of an existing integration + in an XSOAR-supported pack. + - The 'connectus-approved' label is missing. + + When: + - Running the script's main function. + + Then: + - The check fails (exit code 1) and the log names the offending file. + """ + pr = _MockPullRequest( + files=[_MockFile(INTEGRATION_YML, status="modified")], + contents=_contents( + base_yml=integration_yml(ONE_PARAM), + head_yml=integration_yml(TWO_PARAMS), + ), + ) + + assert _run_main(mocker, pr) == 1 + + output = capsys.readouterr().out + assert "ERROR" in output + assert "connectus-approved" in output + assert "Modified 'configuration' of XSOAR-supported integration" in output + assert INTEGRATION_YML in output + + +def test_label_present_passes_despite_changes(mocker, capsys): + """ + Given: + - A PR that both adds a new integration and changes configuration params. + - The 'connectus-approved' label IS present. + + When: + - Running the script's main function. + + Then: + - The check passes (exit code 0) regardless of the detected changes. + """ + pr = _MockPullRequest( + files=[_MockFile(INTEGRATION_YML, status="added")], + labels=[_MockLabel("connectus-approved")], + contents=_contents(head_yml=integration_yml(TWO_PARAMS)), + ) + + assert _run_main(mocker, pr) == 0 + + output = capsys.readouterr().out + assert "SUCCESS" in output + assert "ERROR" not in output + + +def test_no_triggering_changes_passes(mocker, capsys): + """ + Given: + - A PR that touches no integration YMLs (only a script and release notes). + - The 'connectus-approved' label is missing. + + When: + - Running the script's main function. + + Then: + - The check passes (exit code 0) because the label is not required. + """ + pr = _MockPullRequest( + files=[ + _MockFile("Packs/MyPack/Scripts/MyScript/MyScript.py", status="modified"), + _MockFile("Packs/MyPack/ReleaseNotes/1_0_1.md", status="added"), + ], + contents=_contents(), + ) + + assert _run_main(mocker, pr) == 0 + + output = capsys.readouterr().out + assert "INFO" in output + assert "not required" in output + assert "ERROR" not in output + + +# --------------------------------------------------------------------------- +# check_pr_contains_connectus_changes - supporting cases +# --------------------------------------------------------------------------- + + +def test_non_xsoar_pack_is_ignored(): + """ + Given: + - A PR adding a new integration to a partner-supported pack. + + When: + - Running check_pr_contains_connectus_changes. + + Then: + - No reasons are returned: the gate only covers XSOAR-supported packs. + """ + from github_workflow_scripts.check_if_connectus_approved_label_exists import check_pr_contains_connectus_changes + + pr = _MockPullRequest( + files=[_MockFile(INTEGRATION_YML, status="added")], + contents=_contents(head_yml=integration_yml(ONE_PARAM), metadata=PARTNER_METADATA), + ) + + assert check_pr_contains_connectus_changes(pr) == [] + + +def test_unchanged_configuration_is_ignored(): + """ + Given: + - A PR modifying an integration YML where only the dockerimage changed + and the 'configuration' block is identical. + + When: + - Running check_pr_contains_connectus_changes. + + Then: + - No reasons are returned: non-configuration edits don't require approval. + """ + from github_workflow_scripts.check_if_connectus_approved_label_exists import check_pr_contains_connectus_changes + + pr = _MockPullRequest( + files=[_MockFile(INTEGRATION_YML, status="modified")], + contents=_contents( + base_yml=integration_yml(ONE_PARAM, dockerimage="demisto/python3:3.10.1"), + head_yml=integration_yml(ONE_PARAM, dockerimage="demisto/python3:3.10.2"), + ), + ) + + assert check_pr_contains_connectus_changes(pr) == [] + + +def test_undecidable_diff_requires_the_label(): + """ + Given: + - A PR whose modified integration YML cannot be fetched at the base ref, + leaving the configuration comparison undecidable. + + When: + - Running check_pr_contains_connectus_changes. + + Then: + - The change is flagged: an unverifiable configuration edit must not slip + through unapproved just because the API call failed. + """ + from github_workflow_scripts.check_if_connectus_approved_label_exists import check_pr_contains_connectus_changes + + pr = _MockPullRequest( + files=[_MockFile(INTEGRATION_YML, status="modified")], + contents=_contents(head_yml=integration_yml(TWO_PARAMS)), # no base entry + ) + + reasons = check_pr_contains_connectus_changes(pr) + assert len(reasons) == 1 + assert "Modified 'configuration'" in reasons[0] + + +def test_unparsable_yml_is_handled_gracefully(): + """ + Given: + - A PR whose modified integration YML is malformed at the head ref. + + When: + - Running check_pr_contains_connectus_changes. + + Then: + - Parsing fails softly; the malformed head yields an empty configuration, + which differs from the base, so the change is still flagged for review. + """ + from github_workflow_scripts.check_if_connectus_approved_label_exists import check_pr_contains_connectus_changes + + pr = _MockPullRequest( + files=[_MockFile(INTEGRATION_YML, status="modified")], + contents=_contents( + base_yml=integration_yml(ONE_PARAM), + head_yml="configuration: [unclosed", + ), + ) + + reasons = check_pr_contains_connectus_changes(pr) + assert len(reasons) == 1 + assert "Modified 'configuration'" in reasons[0] + + +def test_unreadable_pack_metadata_is_treated_as_xsoar(): + """ + Given: + - A PR adding an integration whose pack_metadata.json cannot be read. + + When: + - Running check_pr_contains_connectus_changes. + + Then: + - The pack is assumed XSOAR-supported and the addition is flagged: we + cannot rule the pack out, so we fail safe rather than fail open. + """ + from github_workflow_scripts.check_if_connectus_approved_label_exists import check_pr_contains_connectus_changes + + pr = _MockPullRequest( + files=[_MockFile(INTEGRATION_YML, status="added")], + contents={}, # no metadata available at any ref + ) + + reasons = check_pr_contains_connectus_changes(pr) + assert len(reasons) == 1 + assert "New XSOAR-supported integration added" in reasons[0] + + +def test_malformed_pack_metadata_is_treated_as_xsoar(): + """ + Given: + - A PR adding an integration whose pack_metadata.json is not valid JSON. + + When: + - Running check_pr_contains_connectus_changes. + + Then: + - The pack is assumed XSOAR-supported and the addition is flagged. + """ + from github_workflow_scripts.check_if_connectus_approved_label_exists import check_pr_contains_connectus_changes + + pr = _MockPullRequest( + files=[_MockFile(INTEGRATION_YML, status="added")], + contents=_contents(head_yml=integration_yml(ONE_PARAM), metadata='{"support": "xsoar"'), + ) + + reasons = check_pr_contains_connectus_changes(pr) + assert len(reasons) == 1 + assert "New XSOAR-supported integration added" in reasons[0] + + +# --------------------------------------------------------------------------- +# Renamed / copied files +# --------------------------------------------------------------------------- + + +def _rename_contents(base_yml: str, head_yml: str, base_path: str = OLD_INTEGRATION_YML) -> dict: + """Build the content map for a rename: base content lives at the pre-rename path.""" + return { + (PACK_METADATA, _MockPullRequest.HEAD_SHA): XSOAR_METADATA, + (base_path, _MockPullRequest.BASE_SHA): base_yml, + (INTEGRATION_YML, _MockPullRequest.HEAD_SHA): head_yml, + } + + +def test_pure_rename_is_not_flagged(): + """ + Given: + - A PR that renames an integration YML without touching its + 'configuration' block. + + When: + - Running check_pr_contains_connectus_changes. + + Then: + - No reason is recorded: a move is not a configuration change. + """ + from github_workflow_scripts.check_if_connectus_approved_label_exists import check_pr_contains_connectus_changes + + pr = _MockPullRequest( + files=[_MockFile(INTEGRATION_YML, status="renamed", previous_filename=OLD_INTEGRATION_YML)], + contents=_rename_contents(integration_yml(ONE_PARAM), integration_yml(ONE_PARAM)), + ) + + assert check_pr_contains_connectus_changes(pr) == [] + + +def test_rename_with_configuration_change_is_flagged(): + """ + Given: + - A PR that renames an integration YML and also adds a configuration param. + + When: + - Running check_pr_contains_connectus_changes. + + Then: + - The change is flagged and the reason names both the new and old paths. + """ + from github_workflow_scripts.check_if_connectus_approved_label_exists import check_pr_contains_connectus_changes + + pr = _MockPullRequest( + files=[_MockFile(INTEGRATION_YML, status="renamed", previous_filename=OLD_INTEGRATION_YML)], + contents=_rename_contents(integration_yml(ONE_PARAM), integration_yml(TWO_PARAMS)), + ) + + reasons = check_pr_contains_connectus_changes(pr) + assert len(reasons) == 1 + assert "Modified 'configuration'" in reasons[0] + assert INTEGRATION_YML in reasons[0] + assert OLD_INTEGRATION_YML in reasons[0] + + +def test_rename_from_non_integration_path_is_flagged_as_new(): + """ + Given: + - A PR that renames a non-integration file into the integration YML path, + so the integration definition appears there for the first time. + + When: + - Running check_pr_contains_connectus_changes. + + Then: + - It is flagged as a new integration rather than compared as a rename. + """ + from github_workflow_scripts.check_if_connectus_approved_label_exists import check_pr_contains_connectus_changes + + pr = _MockPullRequest( + files=[ + _MockFile( + INTEGRATION_YML, + status="renamed", + previous_filename="Packs/MyPack/Integrations/MyIntegration/draft.yml", + ) + ], + contents=_contents(head_yml=integration_yml(ONE_PARAM)), + ) + + reasons = check_pr_contains_connectus_changes(pr) + assert len(reasons) == 1 + assert "New XSOAR-supported integration added" in reasons[0] + + +def test_rename_without_previous_filename_is_flagged_as_new(): + """ + Given: + - A renamed integration YML for which GitHub reported no previous_filename. + + When: + - Running check_pr_contains_connectus_changes. + + Then: + - It is flagged as new rather than raising an AttributeError. + """ + from github_workflow_scripts.check_if_connectus_approved_label_exists import check_pr_contains_connectus_changes + + pr = _MockPullRequest( + files=[_MockFile(INTEGRATION_YML, status="renamed")], # no previous_filename attribute + contents=_contents(head_yml=integration_yml(ONE_PARAM)), + ) + + reasons = check_pr_contains_connectus_changes(pr) + assert len(reasons) == 1 + assert "New XSOAR-supported integration added" in reasons[0] + + +def test_changed_status_is_compared_like_modified(): + """ + Given: + - A PR whose integration YML carries GitHub's rarer 'changed' status and + whose configuration block was edited. + + When: + - Running check_pr_contains_connectus_changes. + + Then: + - It is compared exactly like a 'modified' file and flagged. + """ + from github_workflow_scripts.check_if_connectus_approved_label_exists import check_pr_contains_connectus_changes + + pr = _MockPullRequest( + files=[_MockFile(INTEGRATION_YML, status="changed")], + contents=_contents( + base_yml=integration_yml(ONE_PARAM), + head_yml=integration_yml(TWO_PARAMS), + ), + ) + + reasons = check_pr_contains_connectus_changes(pr) + assert len(reasons) == 1 + assert "Modified 'configuration'" in reasons[0] + + +def test_pack_support_lookup_is_cached(): + """ + Given: + - A PR adding two integrations that live in the same pack. + + When: + - Running check_pr_contains_connectus_changes. + + Then: + - Both are flagged, and the pack metadata is fetched only once. + """ + from github_workflow_scripts.check_if_connectus_approved_label_exists import check_pr_contains_connectus_changes + + second_yml = "Packs/MyPack/Integrations/OtherIntegration/OtherIntegration.yml" + pr = _MockPullRequest( + files=[_MockFile(INTEGRATION_YML, status="added"), _MockFile(second_yml, status="added")], + contents=_contents(head_yml=integration_yml(ONE_PARAM)), + ) + + metadata_fetches = [] + original_get_contents = pr.base.repo.get_contents + + def counting_get_contents(path, ref): + if path == PACK_METADATA: + metadata_fetches.append(ref) + return original_get_contents(path, ref) + + pr.base.repo.get_contents = counting_get_contents + + reasons = check_pr_contains_connectus_changes(pr) + assert len(reasons) == 2 + assert len(metadata_fetches) == 1 + + +def test_api_failure_requires_the_label(): + """ + Given: + - Listing the PR's files raises (e.g. a GitHub API outage). + + When: + - Running check_pr_contains_connectus_changes. + + Then: + - A reason is returned so the gate fails safe rather than letting a + potentially-gated change through unreviewed. + """ + from github_workflow_scripts.check_if_connectus_approved_label_exists import check_pr_contains_connectus_changes + + pr = _MockPullRequest(files_raise=True) + + reasons = check_pr_contains_connectus_changes(pr) + assert len(reasons) == 1 + assert "Could not inspect" in reasons[0] + + +# --------------------------------------------------------------------------- +# Small helpers +# --------------------------------------------------------------------------- + + +@pytest.mark.parametrize( + "parsed_yml, expected", + [ + ({"configuration": [{"name": "url"}]}, [{"name": "url"}]), + ({"name": "NoConfig"}, []), + (None, []), + ], +) +def test_get_configuration(parsed_yml, expected): + """ + Given: + - A parsed integration YML (or None when parsing failed). + + When: + - Running get_configuration. + + Then: + - The configuration block is returned, defaulting to an empty list. + """ + from github_workflow_scripts.check_if_connectus_approved_label_exists import get_configuration + + assert get_configuration(parsed_yml) == expected + + +def test_label_name_is_stable(): + """ + Given: + - The workflow and the branch-protection rule both reference the label by name. + + When: + - Reading CONNECTUS_APPROVED_LABEL. + + Then: + - It is exactly 'connectus-approved'; any rename is a breaking change. + """ + from github_workflow_scripts.check_if_connectus_approved_label_exists import CONNECTUS_APPROVED_LABEL + + assert CONNECTUS_APPROVED_LABEL == "connectus-approved" diff --git a/.github/github_workflow_scripts/github_workflow_scripts_tests/create_internal_pr_test.py b/.github/github_workflow_scripts/github_workflow_scripts_tests/create_internal_pr_test.py new file mode 100644 index 000000000000..f3ba7e13d4d3 --- /dev/null +++ b/.github/github_workflow_scripts/github_workflow_scripts_tests/create_internal_pr_test.py @@ -0,0 +1,248 @@ +from unittest.mock import MagicMock, patch + +from github_workflow_scripts.create_internal_pr import ( + main, + prepare_body, + replace_fixes_with_relates_in_pr_body, +) + + +def _make_mock_pr(html_url: str, number: int, body: str = "", login: str = "contributor1"): + """Create a minimal mock PullRequest.""" + pr = MagicMock() + pr.html_url = html_url + pr.number = number + pr.body = body + pr.user.login = login + return pr + + +class TestCrossReferencePRs: + """Tests for the cross-reference logic that links main and mapping PRs to each other.""" + + EXTERNAL_URL = "https://github.com/demisto/content/pull/1234" + MAIN_URL = "https://github.com/demisto/content/pull/1235" + MAPPING_URL = "https://github.com/demisto/content/pull/1236" + + def test_cross_reference_main_pr_body_contains_mapping_link(self): + """ + Given: + - A main PR body generated from an external PR. + When: + - The cross-reference replace is applied to add the mapping PR link. + Then: + - The main PR body contains the mapping PR URL right after the external PR link. + """ + ext_pr = _make_mock_pr(self.EXTERNAL_URL, 1234, body="description\n\nrelates: #999") + body = prepare_body(ext_pr) + + updated = body.replace( + f"**Original External PR:** {self.EXTERNAL_URL}", + f"**Original External PR:** {self.EXTERNAL_URL}\r\n**Mapping Internal PR:** {self.MAPPING_URL}", + ) + + assert f"**Mapping Internal PR:** {self.MAPPING_URL}" in updated + + def test_cross_reference_mapping_pr_body_contains_main_link(self): + """ + Given: + - A mapping PR body generated from an external PR. + When: + - The cross-reference replace is applied to add the main PR link. + Then: + - The mapping PR body contains the main PR URL right after the external PR link. + """ + ext_pr = _make_mock_pr(self.EXTERNAL_URL, 1234, body="description\n\nrelates: #999") + body = replace_fixes_with_relates_in_pr_body(prepare_body(ext_pr)) + + updated = body.replace( + f"**Original External PR:** {self.EXTERNAL_URL}", + f"**Original External PR:** {self.EXTERNAL_URL}\r\n**Main Internal PR:** {self.MAIN_URL}", + ) + + assert f"**Main Internal PR:** {self.MAIN_URL}" in updated + + def test_cross_reference_links_are_grouped_at_top(self): + """ + Given: + - Both a main and mapping PR body after cross-referencing. + When: + - Checking the position of the Related PRs section. + Then: + - The Related PRs section (with all links) appears before the Contributor section. + """ + ext_pr = _make_mock_pr(self.EXTERNAL_URL, 1234, body="description\n\nrelates: #999") + body = prepare_body(ext_pr) + + updated = body.replace( + f"**Original External PR:** {self.EXTERNAL_URL}", + f"**Original External PR:** {self.EXTERNAL_URL}\r\n**Mapping Internal PR:** {self.MAPPING_URL}", + ) + + assert updated.index("## Related PRs") < updated.index("## Contributor") + assert updated.index("**Mapping Internal PR:**") < updated.index("## Contributor") + + def test_cross_reference_no_duplicate_external_link(self): + """ + Given: + - A PR body after cross-referencing. + When: + - Checking for the external PR marker. + Then: + - The Original External PR line appears exactly once. + """ + ext_pr = _make_mock_pr(self.EXTERNAL_URL, 1234, body="description\n\nrelates: #999") + body = prepare_body(ext_pr) + + updated = body.replace( + f"**Original External PR:** {self.EXTERNAL_URL}", + f"**Original External PR:** {self.EXTERNAL_URL}\r\n**Mapping Internal PR:** {self.MAPPING_URL}", + ) + + assert updated.count("**Original External PR:**") == 1 + + @patch("github_workflow_scripts.create_internal_pr.get_env_var") + @patch("github_workflow_scripts.create_internal_pr.Github") + @patch("github_workflow_scripts.create_internal_pr.separate_pr_files") + @patch("github_workflow_scripts.create_internal_pr.split_branch_with_git") + @patch("github_workflow_scripts.create_internal_pr.create_pr") + @patch("github_workflow_scripts.create_internal_pr.remove_branch_protection") + @patch("github_workflow_scripts.create_internal_pr.get_content_roles", return_value=None) + @patch("github_workflow_scripts.create_internal_pr.is_organization_member", return_value=False) + def test_main_calls_edit_on_both_prs_when_both_created( + self, + mock_is_org_member, + mock_get_content_roles, + mock_remove_protection, + mock_create_pr, + mock_split_branch, + mock_separate, + mock_github_cls, + mock_get_env_var, + ): + """ + Given: + - An external PR that contains both XSOAR and XSIAM files. + When: + - main() creates both a main PR and a mapping PR. + Then: + - edit() is called on both PRs to cross-reference each other. + - The main PR body contains the mapping PR URL. + - The mapping PR body contains the main PR URL. + """ + # -- setup env vars -- + import json + + payload = {"pull_request": {"number": 1234}} + + def env_side_effect(name): + if name == "EVENT_PAYLOAD": + return json.dumps(payload) + return "fake-token" + + mock_get_env_var.side_effect = env_side_effect + + # -- setup GitHub mocks -- + ext_pr = _make_mock_pr(self.EXTERNAL_URL, 1234, body="description\n\nrelates: #999") + ext_pr.base.ref = "contrib-branch" + ext_pr.labels = [] + ext_pr.assignees = [] + ext_pr.merged_by = None + ext_pr.get_review_requests.return_value = ([], []) + ext_pr.get_files.return_value = [] + + mock_repo = MagicMock() + mock_repo.get_pull.return_value = ext_pr + mock_github_cls.return_value.get_repo.return_value = mock_repo + + # -- both file types present -- + xsoar_file = MagicMock() + xsiam_file = MagicMock() + mock_separate.return_value = ([xsoar_file], [xsiam_file]) + mock_split_branch.return_value = ("contrib-branch-main", "contrib-branch-mapping") + + # -- mock created PRs -- + main_pr = _make_mock_pr(self.MAIN_URL, 1235) + main_pr.body = prepare_body(ext_pr) + main_pr.head.ref = "contrib-branch-main" + + mapping_pr = _make_mock_pr(self.MAPPING_URL, 1236) + mapping_pr.body = replace_fixes_with_relates_in_pr_body(prepare_body(ext_pr)) + mapping_pr.head.ref = "contrib-branch-mapping" + + mock_create_pr.side_effect = [main_pr, mapping_pr] + + # -- run -- + main() + + # -- assert edit was called on both PRs -- + assert main_pr.edit.call_count == 1 + assert mapping_pr.edit.call_count == 1 + + main_updated_body = main_pr.edit.call_args.kwargs.get("body", "") + mapping_updated_body = mapping_pr.edit.call_args.kwargs.get("body", "") + + assert f"**Mapping Internal PR:** {self.MAPPING_URL}" in main_updated_body + assert f"**Main Internal PR:** {self.MAIN_URL}" in mapping_updated_body + + @patch("github_workflow_scripts.create_internal_pr.get_env_var") + @patch("github_workflow_scripts.create_internal_pr.Github") + @patch("github_workflow_scripts.create_internal_pr.separate_pr_files") + @patch("github_workflow_scripts.create_internal_pr.create_pr") + @patch("github_workflow_scripts.create_internal_pr.remove_branch_protection") + @patch("github_workflow_scripts.create_internal_pr.get_content_roles", return_value=None) + @patch("github_workflow_scripts.create_internal_pr.is_organization_member", return_value=False) + def test_no_cross_reference_when_only_one_pr_created( + self, + mock_is_org_member, + mock_get_content_roles, + mock_remove_protection, + mock_create_pr, + mock_separate, + mock_github_cls, + mock_get_env_var, + ): + """ + Given: + - An external PR that contains only XSOAR files (no XSIAM). + When: + - main() creates only a main PR. + Then: + - edit() is never called (no cross-reference needed). + """ + import json + + payload = {"pull_request": {"number": 1234}} + + def env_side_effect(name): + if name == "EVENT_PAYLOAD": + return json.dumps(payload) + return "fake-token" + + mock_get_env_var.side_effect = env_side_effect + + ext_pr = _make_mock_pr(self.EXTERNAL_URL, 1234, body="description\n\nrelates: #999") + ext_pr.base.ref = "contrib-branch" + ext_pr.labels = [] + ext_pr.assignees = [] + ext_pr.merged_by = None + ext_pr.get_review_requests.return_value = ([], []) + ext_pr.get_files.return_value = [] + + mock_repo = MagicMock() + mock_repo.get_pull.return_value = ext_pr + mock_github_cls.return_value.get_repo.return_value = mock_repo + + # Only XSOAR files + xsoar_file = MagicMock() + mock_separate.return_value = ([xsoar_file], []) + + main_pr = _make_mock_pr(self.MAIN_URL, 1235) + main_pr.body = prepare_body(ext_pr) + main_pr.head.ref = "contrib-branch" + + mock_create_pr.return_value = main_pr + + main() + + main_pr.edit.assert_not_called() diff --git a/.github/workflows/connectus-approved.yml b/.github/workflows/connectus-approved.yml new file mode 100644 index 000000000000..af77a5ac5cad --- /dev/null +++ b/.github/workflows/connectus-approved.yml @@ -0,0 +1,37 @@ +name: connectus-approved label verification +on: + pull_request: + types: [opened, synchronize, labeled, unlabeled] + + +permissions: + contents: read + +jobs: + check_connectus_approved_label_job: + runs-on: ubuntu-latest + if: github.repository == 'demisto/content' + steps: + - name: Checkout + uses: actions/checkout@v4 + - name: Set up Python + uses: actions/setup-python@v5 + with: + python-version: "3.10" # The Python version set here is the minimum supported by content, if you change it here, please change all the places containing this comment. + - name: Setup Poetry + uses: Gr1N/setup-poetry@48b0f77c8c1b1b19cb962f0f00dff7b4be8f81ec # v9 + - name: Install python dependencies + run: | + poetry install --with github-actions + - name: set pythonpath + run: | + echo "PYTHONPATH=$GITHUB_WORKSPACE" >> $GITHUB_ENV + - name: Check if connectus-approved label exists + id: check_label + env: + PR_NUMBER: ${{ github.event.pull_request.number }} + GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} + run: | + echo "Checking label connectus-approved for: $PR_NUMBER" + cd .github/github_workflow_scripts + poetry run python check_if_connectus_approved_label_exists.py --pr_number $PR_NUMBER --github_token $GITHUB_TOKEN diff --git a/.gitignore b/.gitignore index beff941e8e33..81816dab0a69 100644 --- a/.gitignore +++ b/.gitignore @@ -116,7 +116,12 @@ report.md plans/ cooc_command_examples.txt +# Local investigation workspace (customer data, logs, dumps, scratch files). Never commit. +/Investigations/ + # IDEX custom tools and rules .roomodes .roo/ .idex/ +.worktree-context/ + diff --git a/.hooks/bootstrap b/.hooks/bootstrap index 76494dec8a25..97d3af924efd 100755 --- a/.hooks/bootstrap +++ b/.hooks/bootstrap @@ -21,7 +21,7 @@ PATH=~/.local/bin:$PATH if [[ "$1" == "-h" ]] || [[ "$1" == "--help" ]]; then cat << __EOF__ Setup development environment (run with no arguments): -* install pre-commit hooks (set NO_HOOKS=1 to skip) +* install pre-commit and pre-push hooks (set NO_HOOKS=1 to skip) __EOF__ exit 0 fi @@ -39,6 +39,14 @@ GIT_HOOKS_DIR="${PWD}/.git/hooks" if [ -n "${NO_HOOKS}" ]; then echo "Skipping hooks setup as environment variable NO_HOOKS is set" else + if [ ! -e "${GIT_HOOKS_DIR}/pre-push" ]; then + echo "Installing 'pre-push' hook" + ln -s "${HOOKS_DIR}/pre-push" "${GIT_HOOKS_DIR}/pre-push" + exit_on_error $? "Failed to install pre-push hook" + else + echo "Skipping install of pre-push hook as it already exists." + fi + if [ ! -e "${GIT_HOOKS_DIR}/pre-commit" ]; then echo "Installing 'pre-commit' hooks" ln -s "${HOOKS_DIR}/pre-commit" "${GIT_HOOKS_DIR}/pre-commit" diff --git a/.hooks/pre-push b/.hooks/pre-push new file mode 100755 index 000000000000..4d4a3e993a91 --- /dev/null +++ b/.hooks/pre-push @@ -0,0 +1,29 @@ +#!/bin/sh +# Block accidental direct pushes to protected branches of the official repos. +# Pushes to forks and any other remote are left alone. +# Bypass intentionally with: git push --no-verify + +# git invokes this hook as: pre-push +# Normalize the URL so the match can be anchored to the end: without this, +# '*demisto/content*' would also catch content-private, content-docs, etc. +repo=${2%/} +repo=${repo%.git} +case "$repo" in + *demisto/content|*cortex-content/content) ;; + *) exit 0 ;; +esac + +protected="master main" + +while read -r _local_ref _local_sha remote_ref _remote_sha; do + branch=${remote_ref#refs/heads/} + for p in $protected; do + if [ "$branch" = "$p" ]; then + echo "pre-push: refusing to push directly to '$p'." >&2 + echo " Open a PR, or use 'git push --no-verify' if you really mean it." >&2 + exit 1 + fi + done +done + +exit 0 diff --git a/.secrets_white_list.json b/.secrets_white_list.json index 393242ea65cd..034dae50d031 100644 --- a/.secrets_white_list.json +++ b/.secrets_white_list.json @@ -1,31 +1,32 @@ -{ - "files": [], - "scripts": [], - "iocs": [], - "falsePositives": [ - "https://api.criminalip.io", - "api.criminalip.io", - "CriminalIP.IP.score.outbound", - "CriminalIP.IP.whois.data.org_name", - "CriminalIP.IP.whois.data.postal_code", - "CriminalIP.IP.whois.data.longitude", - "CriminalIP.IP.whois.data.confirmed_time", - "CriminalIP.IP.vpn.data.confirmed_time", - "CriminalIP.IP.anonymous_vpn.data", - "CriminalIP.IP.anonymous_vpn.data.vpn_name", - "CriminalIP.IP.anonymous_vpn.data.vpn_url", - "CriminalIP.IP.anonymous_vpn.data.confirmed_time", - "CriminalIP.IP.webcam.data.image_path", - "CriminalIP.IP.webcam.data.open_port_no", - "CriminalIP.IP.webcam.data.confirmed_time", - "CriminalIP.IP.honeypot.data.log_date", - "CriminalIP.IP.honeypot.data.dst_port", - "CriminalIP.IP.honeypot.data.user_agent", - "CriminalIP.IP.honeypot.data.confirmed_time", - "CriminalIP.IP.port.data.confirmed_time", - "CriminalIP.IP.port.data.ssl_info_raw", - "CriminalIP.IP.port.data.technologies", - "CriminalIP.IP.port.data.technologies.tech_name", - "CriminalIP.IP.port.data.technologies.tech_logo_url" - ] +{ + "files": [], + "scripts": [], + "iocs": [], + "falsePositives": [ + "https://api-your-tenant-url", + "https://api.criminalip.io", + "api.criminalip.io", + "CriminalIP.IP.score.outbound", + "CriminalIP.IP.whois.data.org_name", + "CriminalIP.IP.whois.data.postal_code", + "CriminalIP.IP.whois.data.longitude", + "CriminalIP.IP.whois.data.confirmed_time", + "CriminalIP.IP.vpn.data.confirmed_time", + "CriminalIP.IP.anonymous_vpn.data", + "CriminalIP.IP.anonymous_vpn.data.vpn_name", + "CriminalIP.IP.anonymous_vpn.data.vpn_url", + "CriminalIP.IP.anonymous_vpn.data.confirmed_time", + "CriminalIP.IP.webcam.data.image_path", + "CriminalIP.IP.webcam.data.open_port_no", + "CriminalIP.IP.webcam.data.confirmed_time", + "CriminalIP.IP.honeypot.data.log_date", + "CriminalIP.IP.honeypot.data.dst_port", + "CriminalIP.IP.honeypot.data.user_agent", + "CriminalIP.IP.honeypot.data.confirmed_time", + "CriminalIP.IP.port.data.confirmed_time", + "CriminalIP.IP.port.data.ssl_info_raw", + "CriminalIP.IP.port.data.technologies", + "CriminalIP.IP.port.data.technologies.tech_name", + "CriminalIP.IP.port.data.technologies.tech_logo_url" + ] } \ No newline at end of file diff --git a/AGENTS.md b/AGENTS.md index 1c871a5d31e6..caac2f5e0ff4 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -37,6 +37,7 @@ demisto-sdk pre-commit -a - **Error Handling**: Use `return_error("message")` for user-facing errors. Only raise exceptions for unexpected failures. - **Outputs**: Use `CommandResults` objects with `return_results()`: `return_results(CommandResults(outputs_prefix='MyPrefix', outputs=data))`. Output keys should be CamelCase. - **Logging**: Use `demisto.debug()` and `demisto.info()`. Avoid `print()`. +- **Local Investigation Files**: Put every non-content artifact (customer logs, war room exports, HAR files, API responses, scratch scripts, drafts of Jira comments or PR bodies, plans, reports) under `/Investigations/` at the repo root, one subfolder per ticket (for example `Investigations/XSUP-12345/`). The folder is gitignored. Never write these files anywhere else in the repo, and never move them into `Packs/` or other tracked paths. They may contain customer data or secrets. ## Architecture Notes diff --git a/Config/core_packs_platform_list.json b/Config/core_packs_platform_list.json index 9c999a8d4586..f4a2ab33353d 100644 --- a/Config/core_packs_platform_list.json +++ b/Config/core_packs_platform_list.json @@ -11,21 +11,6 @@ "cloud_posture" ] }, - { - "id": "AIAgents", - "supportedModules": [ - "agentix", - "asm", - "cloud", - "cloud_appsec", - "cloud_posture", - "cloud_runtime_security", - "edr", - "email_security", - "exposure_management", - "xsiam" - ] - }, { "id": "AutoFocus", "supportedModules": [ @@ -48,7 +33,8 @@ "cloud_runtime_security", "edr", "exposure_management", - "xsiam" + "xsiam", + "data_security" ] }, { @@ -104,8 +90,7 @@ "agentix", "cloud_posture", "xsiam", - "edr", - "agentix_xsiam" + "edr" ] }, { diff --git a/Packs/AIAgents/ReleaseNotes/1_0_39.md b/Packs/AIAgents/ReleaseNotes/1_0_39.md new file mode 100644 index 000000000000..918262cdd31e --- /dev/null +++ b/Packs/AIAgents/ReleaseNotes/1_0_39.md @@ -0,0 +1,3 @@ +## AI Agents + +- Removed the **AI Agents** pack from the platform core packs list. No functional changes in this release. diff --git a/Packs/AIAgents/ReleaseNotes/1_0_40.md b/Packs/AIAgents/ReleaseNotes/1_0_40.md new file mode 100644 index 000000000000..7007b3e49f26 --- /dev/null +++ b/Packs/AIAgents/ReleaseNotes/1_0_40.md @@ -0,0 +1,3 @@ +## AI Agents + +- Marked the pack as hidden. No functional changes in this release. diff --git a/Packs/AIAgents/pack_metadata.json b/Packs/AIAgents/pack_metadata.json index b87a7518ddcb..135fbaf001dc 100644 --- a/Packs/AIAgents/pack_metadata.json +++ b/Packs/AIAgents/pack_metadata.json @@ -2,7 +2,8 @@ "name": "AI Agents", "description": "AI Agents", "support": "xsoar", - "currentVersion": "1.0.38", + "hidden": true, + "currentVersion": "1.0.40", "serverMinVersion": "8.13", "author": "Cortex XSOAR", "url": "https://www.paloaltonetworks.com/cortex", diff --git a/Packs/AMP/Integrations/AMPv2/AMPv2.yml b/Packs/AMP/Integrations/AMPv2/AMPv2.yml index 3d934c99a358..816dbada23a4 100644 --- a/Packs/AMP/Integrations/AMPv2/AMPv2.yml +++ b/Packs/AMP/Integrations/AMPv2/AMPv2.yml @@ -87,6 +87,13 @@ configuration: type: 13 required: false section: Collect +- display: Incidents Fetch Interval + name: incidentFetchInterval + defaultvalue: '1' + required: false + type: 19 + section: Collect + advanced: true - display: Fetch incidents name: isFetch type: 8 diff --git a/Packs/AMP/ModelingRules/AMP/AMP.yml b/Packs/AMP/ModelingRules/AMP/AMP.yml index 2bc491899199..f463944bbcaa 100644 --- a/Packs/AMP/ModelingRules/AMP/AMP.yml +++ b/Packs/AMP/ModelingRules/AMP/AMP.yml @@ -1,4 +1,5 @@ fromversion: 8.3.0 +toversion: 8.14.0 id: AMP_ModelingRule name: AMP Modeling Rule rules: '' diff --git a/Packs/AMP/ModelingRules/AMP_2_11/AMP_2_11.xif b/Packs/AMP/ModelingRules/AMP_2_11/AMP_2_11.xif new file mode 100644 index 000000000000..53e961309a3a --- /dev/null +++ b/Packs/AMP/ModelingRules/AMP_2_11/AMP_2_11.xif @@ -0,0 +1,49 @@ +[MODEL: dataset = "cisco_secure_endpoint_raw"] +alter + external_ip = computer -> external_ip, + network_addresses_ips = arraymap(computer -> network_addresses[], "@element" -> ip), + first_ip_network_addresses = json_extract_scalar(arrayindex(computer -> network_addresses[], 0), "$.ip"), + file_description = concat(file -> file_name, " - ", file -> disposition), + cloud_ioc_description = cloud_ioc -> description, + remote_ip_network_info = network_info -> remote_ip, + protocol_network_info = network_info -> nfm.protocol +| alter + xdm.event.id = to_string(id), + xdm.event.type = event_type, + xdm.event.description = if(file_description != null and cloud_ioc_description != null, concat(cloud_ioc_description, "; ", file_description), coalesce(cloud_ioc_description, file_description )), + xdm.event.outcome = if(lowercase(event_type) contains "fail", XDM_CONST.OUTCOME_FAILED, lowercase(event_type) contains "succ", XDM_CONST.OUTCOME_SUCCESS, null), + xdm.event.outcome_reason = error -> description, + xdm.alert.original_threat_id = coalesce(detection_id, threat_hunting -> incident_hunt_guid), + xdm.alert.severity = severity, + xdm.alert.original_threat_name = detection, + xdm.alert.name = coalesce(detection, threat_hunting -> incident_title), + xdm.alert.description = threat_hunting -> incident_summary, + xdm.alert.mitre_tactics = arraymap(tactics -> [], trim("@element", "\"")), + xdm.alert.mitre_techniques = arraymap(techniques -> [], trim("@element", "\"")), + xdm.alert.risks = vulnerabilities -> [], + xdm.source.host.hostname = computer -> hostname, + xdm.source.user.username = computer -> user, + xdm.source.identity.username = computer -> user, + xdm.source.agent.identifier = connector_guid, + xdm.intermediate.ipv4 = if(external_ip ~= "\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}", external_ip, null), + xdm.intermediate.ipv6 = if(external_ip ~= "[a-fA-F0-9\:]{1,5}[a-fA-F0-9\:]{1,5}[a-fA-F0-9\:]{1,5}[a-fA-F0-9\:]{1,5}[a-fA-F0-9\:]{1,5}[a-fA-F0-9\:]{1,5}[a-fA-F0-9\:]{1,5}[a-fA-F0-9\:]{1,5}", external_ip, null), + xdm.source.ipv4 = if(first_ip_network_addresses ~= "\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}", first_ip_network_addresses, null), + xdm.source.ipv6 = if(first_ip_network_addresses ~= "[a-fA-F0-9\:]{1,5}[a-fA-F0-9\:]{1,5}[a-fA-F0-9\:]{1,5}[a-fA-F0-9\:]{1,5}[a-fA-F0-9\:]{1,5}[a-fA-F0-9\:]{1,5}[a-fA-F0-9\:]{1,5}[a-fA-F0-9\:]{1,5}", first_ip_network_addresses, null), + xdm.source.host.ipv4_addresses = arrayfilter(network_addresses_ips, "@element" ~= "\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}"), + xdm.source.host.ipv6_addresses = arrayfilter(network_addresses_ips, "@element" ~= "[a-fA-F0-9\:]{1,5}[a-fA-F0-9\:]{1,5}[a-fA-F0-9\:]{1,5}[a-fA-F0-9\:]{1,5}[a-fA-F0-9\:]{1,5}[a-fA-F0-9\:]{1,5}[a-fA-F0-9\:]{1,5}[a-fA-F0-9\:]{1,5}"), + xdm.source.host.mac_addresses = arraymap(computer -> network_addresses[], "@element" -> mac), + xdm.target.file.filename = file -> file_name, + xdm.target.file.path = file -> file_path, + xdm.target.file.md5 = file -> identity.md5, + xdm.target.file.sha256 = file -> identity.sha256, + xdm.source.process.pid = coalesce(to_integer(file -> parent.process_id), to_integer(network_info -> parent.process_id)), + xdm.source.process.command_line = command_line -> arguments, + xdm.source.process.executable.filename = coalesce(file -> parent.file_name, network_info -> parent.file_name), + xdm.source.process.executable.md5 = coalesce(file -> parent.identity.md5, network_info -> parent.identity.md5), + xdm.source.process.executable.sha256 = coalesce(file -> parent.identity.sha256, network_info -> parent.identity.sha256), + xdm.target.ipv4 = if(remote_ip_network_info ~= "\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}", remote_ip_network_info, null), + xdm.target.ipv6 = if(remote_ip_network_info ~= "[a-fA-F0-9\:]{1,5}[a-fA-F0-9\:]{1,5}[a-fA-F0-9\:]{1,5}[a-fA-F0-9\:]{1,5}[a-fA-F0-9\:]{1,5}[a-fA-F0-9\:]{1,5}[a-fA-F0-9\:]{1,5}[a-fA-F0-9\:]{1,5}", remote_ip_network_info, null), + xdm.target.port = to_integer(network_info -> remote_port), + xdm.target.url = network_info -> dirty_url, + xdm.network.http.url = network_info -> dirty_url, + xdm.network.ip_protocol = if(protocol_network_info = "HOPOPT", XDM_CONST.IP_PROTOCOL_HOPOPT, protocol_network_info = "ICMP",XDM_CONST.IP_PROTOCOL_ICMP, protocol_network_info = "IGMP",XDM_CONST.IP_PROTOCOL_IGMP, protocol_network_info = "GGP",XDM_CONST.IP_PROTOCOL_GGP, protocol_network_info = "IP",XDM_CONST.IP_PROTOCOL_IP, protocol_network_info = "ST",XDM_CONST.IP_PROTOCOL_ST, protocol_network_info = "TCP",XDM_CONST.IP_PROTOCOL_TCP, protocol_network_info = "CBT",XDM_CONST.IP_PROTOCOL_CBT, protocol_network_info = "EGP",XDM_CONST.IP_PROTOCOL_EGP, protocol_network_info = "IGP",XDM_CONST.IP_PROTOCOL_IGP, protocol_network_info = "BBN_RCC_MON",XDM_CONST.IP_PROTOCOL_BBN_RCC_MON, protocol_network_info = "NVP_II",XDM_CONST.IP_PROTOCOL_NVP_II, protocol_network_info = "PUP",XDM_CONST.IP_PROTOCOL_PUP, protocol_network_info = "ARGUS",XDM_CONST.IP_PROTOCOL_ARGUS, protocol_network_info = "EMCON",XDM_CONST.IP_PROTOCOL_EMCON, protocol_network_info = "XNET",XDM_CONST.IP_PROTOCOL_XNET, protocol_network_info = "CHAOS",XDM_CONST.IP_PROTOCOL_CHAOS, protocol_network_info = "UDP",XDM_CONST.IP_PROTOCOL_UDP, protocol_network_info = "MUX",XDM_CONST.IP_PROTOCOL_MUX, protocol_network_info = "DCN_MEAS",XDM_CONST.IP_PROTOCOL_DCN_MEAS, protocol_network_info = "HMP",XDM_CONST.IP_PROTOCOL_HMP, protocol_network_info = "PRM",XDM_CONST.IP_PROTOCOL_PRM, protocol_network_info = "XNS_IDP",XDM_CONST.IP_PROTOCOL_XNS_IDP, protocol_network_info = "TRUNK_1",XDM_CONST.IP_PROTOCOL_TRUNK_1, protocol_network_info = "TRUNK_2",XDM_CONST.IP_PROTOCOL_TRUNK_2, protocol_network_info = "LEAF_1",XDM_CONST.IP_PROTOCOL_LEAF_1, protocol_network_info = "LEAF_2",XDM_CONST.IP_PROTOCOL_LEAF_2, protocol_network_info = "RDP",XDM_CONST.IP_PROTOCOL_RDP, protocol_network_info = "IRTP",XDM_CONST.IP_PROTOCOL_IRTP, protocol_network_info = "ISO_TP4",XDM_CONST.IP_PROTOCOL_ISO_TP4, protocol_network_info = "NETBLT",XDM_CONST.IP_PROTOCOL_NETBLT, protocol_network_info = "MFE_NSP",XDM_CONST.IP_PROTOCOL_MFE_NSP, protocol_network_info = "MERIT_INP",XDM_CONST.IP_PROTOCOL_MERIT_INP, protocol_network_info = "DCCP",XDM_CONST.IP_PROTOCOL_DCCP, protocol_network_info = "3PC",XDM_CONST.IP_PROTOCOL_3PC, protocol_network_info = "IDPR",XDM_CONST.IP_PROTOCOL_IDPR, protocol_network_info = "XTP",XDM_CONST.IP_PROTOCOL_XTP, protocol_network_info = "DDP",XDM_CONST.IP_PROTOCOL_DDP, protocol_network_info = "IDPR_CMTP",XDM_CONST.IP_PROTOCOL_IDPR_CMTP, protocol_network_info = "TP",XDM_CONST.IP_PROTOCOL_TP, protocol_network_info = "IL",XDM_CONST.IP_PROTOCOL_IL, protocol_network_info = "IPV6",XDM_CONST.IP_PROTOCOL_IPV6, protocol_network_info = "SDRP",XDM_CONST.IP_PROTOCOL_SDRP, protocol_network_info = "IPV6_ROUTE",XDM_CONST.IP_PROTOCOL_IPV6_ROUTE, protocol_network_info = "IPV6_FRAG",XDM_CONST.IP_PROTOCOL_IPV6_FRAG, protocol_network_info = "IDRP",XDM_CONST.IP_PROTOCOL_IDRP, protocol_network_info = "RSVP",XDM_CONST.IP_PROTOCOL_RSVP, protocol_network_info = "GRE",XDM_CONST.IP_PROTOCOL_GRE, protocol_network_info = "DSR",XDM_CONST.IP_PROTOCOL_DSR, protocol_network_info = "BNA",XDM_CONST.IP_PROTOCOL_BNA, protocol_network_info = "ESP",XDM_CONST.IP_PROTOCOL_ESP, protocol_network_info = "AH",XDM_CONST.IP_PROTOCOL_AH, protocol_network_info = "I_NLSP",XDM_CONST.IP_PROTOCOL_I_NLSP, protocol_network_info = "SWIPE",XDM_CONST.IP_PROTOCOL_SWIPE, protocol_network_info = "NARP",XDM_CONST.IP_PROTOCOL_NARP, protocol_network_info = "MOBILE",XDM_CONST.IP_PROTOCOL_MOBILE, protocol_network_info = "TLSP",XDM_CONST.IP_PROTOCOL_TLSP, protocol_network_info = "SKIP",XDM_CONST.IP_PROTOCOL_SKIP, protocol_network_info = "IPV6_ICMP",XDM_CONST.IP_PROTOCOL_IPV6_ICMP, protocol_network_info = "IPV6_NONXT",XDM_CONST.IP_PROTOCOL_IPV6_NONXT, protocol_network_info = "IPV6_OPTS",XDM_CONST.IP_PROTOCOL_IPV6_OPTS, protocol_network_info = "CFTP",XDM_CONST.IP_PROTOCOL_CFTP, protocol_network_info = "SAT_EXPAK",XDM_CONST.IP_PROTOCOL_SAT_EXPAK, protocol_network_info = "KRYPTOLAN",XDM_CONST.IP_PROTOCOL_KRYPTOLAN, protocol_network_info = "RVD",XDM_CONST.IP_PROTOCOL_RVD, protocol_network_info = "IPPC",XDM_CONST.IP_PROTOCOL_IPPC, protocol_network_info = "SAT_MON",XDM_CONST.IP_PROTOCOL_SAT_MON, protocol_network_info = "VISA",XDM_CONST.IP_PROTOCOL_VISA, protocol_network_info = "IPCV",XDM_CONST.IP_PROTOCOL_IPCV, protocol_network_info = "CPNX",XDM_CONST.IP_PROTOCOL_CPNX, protocol_network_info = "CPHB",XDM_CONST.IP_PROTOCOL_CPHB, protocol_network_info = "WSN",XDM_CONST.IP_PROTOCOL_WSN, protocol_network_info = "PVP",XDM_CONST.IP_PROTOCOL_PVP, protocol_network_info = "BR_SAT_MON",XDM_CONST.IP_PROTOCOL_BR_SAT_MON, protocol_network_info = "SUN_ND",XDM_CONST.IP_PROTOCOL_SUN_ND, protocol_network_info = "WB_MON",XDM_CONST.IP_PROTOCOL_WB_MON, protocol_network_info = "WB_EXPAK",XDM_CONST.IP_PROTOCOL_WB_EXPAK, protocol_network_info = "ISO_IP",XDM_CONST.IP_PROTOCOL_ISO_IP, protocol_network_info = "VMTP",XDM_CONST.IP_PROTOCOL_VMTP, protocol_network_info = "SECURE_VMTP",XDM_CONST.IP_PROTOCOL_SECURE_VMTP, protocol_network_info = "VINES",XDM_CONST.IP_PROTOCOL_VINES, protocol_network_info = "TTP",XDM_CONST.IP_PROTOCOL_TTP, protocol_network_info = "NSFNET_IGP",XDM_CONST.IP_PROTOCOL_NSFNET_IGP, protocol_network_info = "DGP",XDM_CONST.IP_PROTOCOL_DGP, protocol_network_info = "TCF",XDM_CONST.IP_PROTOCOL_TCF, protocol_network_info = "EIGRP",XDM_CONST.IP_PROTOCOL_EIGRP, protocol_network_info = "OSPFIGP",XDM_CONST.IP_PROTOCOL_OSPFIGP, protocol_network_info = "SPRITE_RPC",XDM_CONST.IP_PROTOCOL_SPRITE_RPC, protocol_network_info = "LARP",XDM_CONST.IP_PROTOCOL_LARP, protocol_network_info = "MTP",XDM_CONST.IP_PROTOCOL_MTP, protocol_network_info = "AX25",XDM_CONST.IP_PROTOCOL_AX25, protocol_network_info = "IPIP",XDM_CONST.IP_PROTOCOL_IPIP, protocol_network_info = "MICP",XDM_CONST.IP_PROTOCOL_MICP, protocol_network_info = "SCC_SP",XDM_CONST.IP_PROTOCOL_SCC_SP, protocol_network_info = "ETHERIP",XDM_CONST.IP_PROTOCOL_ETHERIP, protocol_network_info = "ENCAP",XDM_CONST.IP_PROTOCOL_ENCAP, protocol_network_info = "GMTP",XDM_CONST.IP_PROTOCOL_GMTP, protocol_network_info = "IFMP",XDM_CONST.IP_PROTOCOL_IFMP, protocol_network_info = "PNNI",XDM_CONST.IP_PROTOCOL_PNNI, protocol_network_info = "PIM",XDM_CONST.IP_PROTOCOL_PIM, protocol_network_info = "ARIS",XDM_CONST.IP_PROTOCOL_ARIS, protocol_network_info = "SCPS",XDM_CONST.IP_PROTOCOL_SCPS, protocol_network_info = "QNX",XDM_CONST.IP_PROTOCOL_QNX, protocol_network_info = "AN",XDM_CONST.IP_PROTOCOL_AN, protocol_network_info = "IPCOMP",XDM_CONST.IP_PROTOCOL_IPCOMP, protocol_network_info = "SNP",XDM_CONST.IP_PROTOCOL_SNP, protocol_network_info = "COMPAQ_PEER",XDM_CONST.IP_PROTOCOL_COMPAQ_PEER, protocol_network_info = "IPX_IN_IP",XDM_CONST.IP_PROTOCOL_IPX_IN_IP, protocol_network_info = "VRRP",XDM_CONST.IP_PROTOCOL_VRRP, protocol_network_info = "PGM",XDM_CONST.IP_PROTOCOL_PGM, protocol_network_info = "L2TP",XDM_CONST.IP_PROTOCOL_L2TP, protocol_network_info = "DDX",XDM_CONST.IP_PROTOCOL_DDX, protocol_network_info = "IATP",XDM_CONST.IP_PROTOCOL_IATP, protocol_network_info = "STP",XDM_CONST.IP_PROTOCOL_STP, protocol_network_info = "SRP",XDM_CONST.IP_PROTOCOL_SRP, protocol_network_info = "UTI",XDM_CONST.IP_PROTOCOL_UTI, protocol_network_info = "SMP",XDM_CONST.IP_PROTOCOL_SMP, protocol_network_info = "SM",XDM_CONST.IP_PROTOCOL_SM, protocol_network_info = "PTP",XDM_CONST.IP_PROTOCOL_PTP, protocol_network_info = "ISIS",XDM_CONST.IP_PROTOCOL_ISIS, protocol_network_info = "FIRE",XDM_CONST.IP_PROTOCOL_FIRE, protocol_network_info = "CRTP",XDM_CONST.IP_PROTOCOL_CRTP, protocol_network_info = "CRUDP",XDM_CONST.IP_PROTOCOL_CRUDP, protocol_network_info = "SSCOPMCE",XDM_CONST.IP_PROTOCOL_SSCOPMCE, protocol_network_info = "IPLT",XDM_CONST.IP_PROTOCOL_IPLT, protocol_network_info = "SPS",XDM_CONST.IP_PROTOCOL_SPS, protocol_network_info = "PIPE",XDM_CONST.IP_PROTOCOL_PIPE, protocol_network_info = "SCTP",XDM_CONST.IP_PROTOCOL_SCTP, protocol_network_info = "FC",XDM_CONST.IP_PROTOCOL_FC, protocol_network_info = "RSVP_E2E_IGNORE",XDM_CONST.IP_PROTOCOL_RSVP_E2E_IGNORE, protocol_network_info = "MOBILITY",XDM_CONST.IP_PROTOCOL_MOBILITY, protocol_network_info = "UDPLITE",XDM_CONST.IP_PROTOCOL_UDPLITE, protocol_network_info = "MPLS_IN_IP",XDM_CONST.IP_PROTOCOL_MPLS_IN_IP, protocol_network_info = "MANET",XDM_CONST.IP_PROTOCOL_MANET, protocol_network_info = "HIP",XDM_CONST.IP_PROTOCOL_HIP, protocol_network_info = "SHIM6",XDM_CONST.IP_PROTOCOL_SHIM6, protocol_network_info = "WESP",XDM_CONST.IP_PROTOCOL_WESP, protocol_network_info = "ROHC",XDM_CONST.IP_PROTOCOL_ROHC, protocol_network_info = "RESERVED",XDM_CONST.IP_PROTOCOL_RESERVED,to_string(protocol_network_info)); \ No newline at end of file diff --git a/Packs/AMP/ModelingRules/AMP_2_11/AMP_2_11.yml b/Packs/AMP/ModelingRules/AMP_2_11/AMP_2_11.yml new file mode 100644 index 000000000000..65a27a90711f --- /dev/null +++ b/Packs/AMP/ModelingRules/AMP_2_11/AMP_2_11.yml @@ -0,0 +1,8 @@ +fromversion: 8.15.0 +id: AMP_ModelingRule +name: AMP Modeling Rule +rules: '' +schema: '' +tags: +supportedModules: +- xsiam diff --git a/Packs/AMP/ModelingRules/AMP_2_11/AMP_2_11_schema.json b/Packs/AMP/ModelingRules/AMP_2_11/AMP_2_11_schema.json new file mode 100644 index 000000000000..ab0aa9406dcb --- /dev/null +++ b/Packs/AMP/ModelingRules/AMP_2_11/AMP_2_11_schema.json @@ -0,0 +1,68 @@ +{ + "cisco_secure_endpoint_raw": { + "computer": { + "type": "string", + "is_array": false + }, + "id": { + "type": "int", + "is_array": false + }, + "event_type": { + "type": "string", + "is_array": false + }, + "file": { + "type": "string", + "is_array": false + }, + "detection_id": { + "type": "string", + "is_array": false + }, + "severity": { + "type": "string", + "is_array": false + }, + "detection": { + "type": "string", + "is_array": false + }, + "connector_guid": { + "type": "string", + "is_array": false + }, + "tactics": { + "type": "string", + "is_array": false + }, + "techniques": { + "type": "string", + "is_array": false + }, + "cloud_ioc": { + "type": "string", + "is_array": false + }, + "command_line": { + "type": "string", + "is_array": false + }, + "vulnerabilities": { + "type": "string", + "is_array": false + }, + "error": { + "type": "string", + "is_array": false + }, + "threat_hunting": { + "type": "string", + "is_array": false + }, + "network_info": { + "type": "string", + "is_array": false + } + } +} \ No newline at end of file diff --git a/Packs/AMP/ReleaseNotes/2_1_18.md b/Packs/AMP/ReleaseNotes/2_1_18.md new file mode 100644 index 000000000000..2118a24eef7a --- /dev/null +++ b/Packs/AMP/ReleaseNotes/2_1_18.md @@ -0,0 +1,7 @@ + +#### Modeling Rules + +##### New: AMP Modeling Rule + +- New: Added a new modeling rule version that maps the `xdm.*.identity.*` fields alongside the existing `xdm.*.user.*` fields. +<~XSIAM> (Available from Cortex XSIAM 2.11). diff --git a/Packs/AMP/ReleaseNotes/2_1_19.md b/Packs/AMP/ReleaseNotes/2_1_19.md new file mode 100644 index 000000000000..832d96690f20 --- /dev/null +++ b/Packs/AMP/ReleaseNotes/2_1_19.md @@ -0,0 +1,6 @@ + +#### Integrations + +##### Cisco AMP v2 + +- Updated the Fetch Incidents configuration parameters. diff --git a/Packs/AMP/pack_metadata.json b/Packs/AMP/pack_metadata.json index 684966d70c86..323835b8607d 100644 --- a/Packs/AMP/pack_metadata.json +++ b/Packs/AMP/pack_metadata.json @@ -2,7 +2,7 @@ "name": "Cisco AMP", "description": "Uses CISCO AMP Endpoint", "support": "xsoar", - "currentVersion": "2.1.17", + "currentVersion": "2.1.19", "author": "Cortex XSOAR", "url": "https://www.paloaltonetworks.com/cortex", "email": "", diff --git a/Packs/ANYRUN/Integrations/AnyRunFeed/AnyRunFeed.py b/Packs/ANYRUN/Integrations/AnyRunFeed/AnyRunFeed.py index 79f131422d70..c635b5b9568d 100644 --- a/Packs/ANYRUN/Integrations/AnyRunFeed/AnyRunFeed.py +++ b/Packs/ANYRUN/Integrations/AnyRunFeed/AnyRunFeed.py @@ -8,7 +8,7 @@ from anyrun import RunTimeException DATE_TIME_FORMAT = "%Y-%m-%d %H:%M:%S" -VERSION = "PA-XSOAR:2.3.0" +VERSION = "PA-XSOAR:2.4.0" def test_module(params: dict) -> str: # pragma: no cover @@ -77,17 +77,39 @@ def convert_indicators(indicators: list[dict]) -> list[dict]: for indicator in indicators: indicator_type, indicator_value = extract_indicator_data(indicator) + fields: dict[str, Any] = { + "firstseenbysource": indicator.get("created"), + "first_seen": indicator.get("created"), + "modified": indicator.get("modified"), + "last_seen": indicator.get("modified"), + "vendor": "ANY.RUN", + "source": "ANY.RUN TI Feed", + "tags": indicator.get("labels") or [], + "publications": [ + { + "title": ref.get("source_name") or "", + "link": ref.get("url") or "", + "source": "ANY.RUN TI Feed", + "timestamp": indicator.get("created"), + } + for ref in indicator.get("external_references") or [] + if ref.get("url") + ], + } + if indicator_type == "domain-name": + fields["communitynotes"] = [ + { + "notes": ref["url"], + "timestamp": indicator.get("created"), + } + for ref in indicator.get("external_references") or [] + if ref.get("url") + ] + indicator_payload = { "value": indicator_value, "type": {"ipv4-addr": "IP", "url": "URL", "domain-name": "Domain"}.get(indicator_type), - "fields": { - "firstseenbysource": indicator.get("created"), - "first_seen": indicator.get("created"), - "modified": indicator.get("modified"), - "last_seen": indicator.get("modified"), - "vendor": "ANY.RUN", - "source": "ANY.RUN TI Feed", - }, + "fields": fields, } converted_indicators.append(indicator_payload) diff --git a/Packs/ANYRUN/Integrations/AnyRunFeed/AnyRunFeed.yml b/Packs/ANYRUN/Integrations/AnyRunFeed/AnyRunFeed.yml index dcc8c3f852a9..c7d09c597f56 100644 --- a/Packs/ANYRUN/Integrations/AnyRunFeed/AnyRunFeed.yml +++ b/Packs/ANYRUN/Integrations/AnyRunFeed/AnyRunFeed.yml @@ -71,7 +71,6 @@ configuration: - never - interval - indicatorType - - suddenDeath - display: "" name: feedExpirationInterval type: 1 @@ -95,6 +94,14 @@ configuration: type: 0 section: Collect additionalinfo: Supports CSV values. +- additionalinfo: Whether the feed should pull only new or modified indicators that have been sent from the integration. As the determination if the indicator is new or modified happens on the 3rd-party vendor's side, and only indicators that are new or modified are sent to Cortex XSOAR, all indicators coming from these feeds are labeled new or modified. + defaultvalue: 'true' + display: Incremental Feed + hidden: true + name: feedIncremental + type: 8 + section: Collect + required: false - display: Traffic Light Protocol Color name: tlp_color options: @@ -110,7 +117,7 @@ description: Threat Intelligence Feeds provide data on the known indicators of c display: ANY.RUN TI Feed name: ANY.RUN TI Feed script: - dockerimage: demisto/anyrun-sdk:1.0.0.10120494 + dockerimage: demisto/anyrun-sdk:1.0.0.12844876 subtype: python3 runonce: false script: '-' @@ -119,3 +126,6 @@ script: tests: - No tests fromversion: 6.0.0 +supportedModules: +- agentix +- xsiam diff --git a/Packs/ANYRUN/Integrations/AnyRunFeed/AnyRunFeed_test.py b/Packs/ANYRUN/Integrations/AnyRunFeed/AnyRunFeed_test.py index 002564f345ae..35bb2e719468 100644 --- a/Packs/ANYRUN/Integrations/AnyRunFeed/AnyRunFeed_test.py +++ b/Packs/ANYRUN/Integrations/AnyRunFeed/AnyRunFeed_test.py @@ -33,6 +33,40 @@ def test_convert_indicators_returns_valid_demisto_indicators(indicators: list[di "last_seen": "2025-05-17T20:15:39.130Z", "vendor": "ANY.RUN", "source": "ANY.RUN TI Feed", + "tags": [], + "publications": [], }, } ] + + +def test_convert_indicators_populates_communitynotes_for_domain_indicators(): + indicators = [ + { + "created": "2022-02-23T07:34:19.000Z", + "modified": "2025-05-17T20:15:39.130Z", + "pattern": "[domain-name:value = 'some_domain.com']", + "labels": ["malware"], + "external_references": [ + {"source_name": "ANY.RUN", "url": "https://app.any.run/tasks/some-task-id"}, + ], + } + ] + + converted = convert_indicators(indicators) + + assert converted[0]["fields"]["tags"] == ["malware"] + assert converted[0]["fields"]["publications"] == [ + { + "title": "ANY.RUN", + "link": "https://app.any.run/tasks/some-task-id", + "source": "ANY.RUN TI Feed", + "timestamp": "2022-02-23T07:34:19.000Z", + } + ] + assert converted[0]["fields"]["communitynotes"] == [ + { + "notes": "https://app.any.run/tasks/some-task-id", + "timestamp": "2022-02-23T07:34:19.000Z", + } + ] diff --git a/Packs/ANYRUN/Integrations/AnyRunLookup/AnyRunLookup.py b/Packs/ANYRUN/Integrations/AnyRunLookup/AnyRunLookup.py index f8acb1b5d301..437cd713033e 100644 --- a/Packs/ANYRUN/Integrations/AnyRunLookup/AnyRunLookup.py +++ b/Packs/ANYRUN/Integrations/AnyRunLookup/AnyRunLookup.py @@ -5,7 +5,7 @@ from anyrun.connectors import LookupConnector from anyrun import RunTimeException -VERSION = "PA-XSOAR:2.3.0" +VERSION = "PA-XSOAR:2.4.0" DBOT_SCORE_TYPE_RESOLVER = { "destination_ip": DBotScoreType.IP, diff --git a/Packs/ANYRUN/Integrations/AnyRunLookup/AnyRunLookup.yml b/Packs/ANYRUN/Integrations/AnyRunLookup/AnyRunLookup.yml index b99bf30b103c..b8648a0b6685 100644 --- a/Packs/ANYRUN/Integrations/AnyRunLookup/AnyRunLookup.yml +++ b/Packs/ANYRUN/Integrations/AnyRunLookup/AnyRunLookup.yml @@ -942,7 +942,7 @@ script: - contextPath: DBotScore.Vendor description: The vendor used to calculate the score. type: String - dockerimage: demisto/anyrun-sdk:1.0.0.10120494 + dockerimage: demisto/anyrun-sdk:1.0.0.10440656 subtype: python3 runonce: false script: '-' diff --git a/Packs/ANYRUN/Integrations/AnyRunSandbox/AnyRunSandbox.py b/Packs/ANYRUN/Integrations/AnyRunSandbox/AnyRunSandbox.py index 4c369d34779d..b01fd5c6a9be 100644 --- a/Packs/ANYRUN/Integrations/AnyRunSandbox/AnyRunSandbox.py +++ b/Packs/ANYRUN/Integrations/AnyRunSandbox/AnyRunSandbox.py @@ -1,3 +1,4 @@ +import json import demistomock as demisto from CommonServerPython import * @@ -11,7 +12,7 @@ ) -VERSION = "PA-XSOAR:2.3.0" +VERSION = "PA-XSOAR:2.4.0" SCORE_TO_VERDICT = {0: "Unknown", 1: "Suspicious", 2: "Malicious"} @@ -22,11 +23,17 @@ "sha256": "File SHA-256", } +DEFAULT_ROOT_URL = "any.run" + def test_module(params: dict) -> str: # pragma: no cover """Performs ANY.RUN API call to verify integration is operational""" try: - with BaseSandboxConnector(get_authentication(params), trust_env=argToBoolean(params.get("proxy", False))) as connector: + with BaseSandboxConnector( + get_authentication(params), + trust_env=argToBoolean(params.get("proxy", False)), + root_url=params.get("root_url") or DEFAULT_ROOT_URL, + ) as connector: connector.check_authorization() return "ok" except RunTimeException as exception: @@ -76,6 +83,7 @@ def build_context_path(analysis_type: str, connector: WindowsConnector | LinuxCo def start_analyse( + params: dict, args: dict, analysis_type: str, connector: WindowsConnector | LinuxConnector | AndroidConnector, @@ -83,6 +91,7 @@ def start_analyse( """ Process Sandbox analysis + :param params: Demisto params :param args: Demisto args :param analysis_type: ANY.RUN Sandbox submission type :param connector: ANY.RUN connector instance @@ -94,20 +103,21 @@ def start_analyse( else: task_uuid = connector.run_url_analysis(**args) - return_results( - CommandResults( - outputs_prefix="ANYRUN.SandboxURL", - outputs=f"Link to the interactive analysis: https://app.any.run/tasks/{task_uuid}", - ignore_auto_extract=True, - ) - ) + root_url = params.get("root_url") or DEFAULT_ROOT_URL return_results( - CommandResults( - outputs_prefix=build_context_path(analysis_type, connector), - outputs=task_uuid, - ignore_auto_extract=True, - ) + [ + CommandResults( + outputs_prefix="ANYRUN.SandboxURL", + outputs=f"Link to the interactive analysis: https://app.{root_url}/tasks/{task_uuid}", + ignore_auto_extract=True, + ), + CommandResults( + outputs_prefix=build_context_path(analysis_type, connector), + outputs=task_uuid, + ignore_auto_extract=True, + ), + ] ) @@ -116,9 +126,10 @@ def detonate_entity_windows(params: dict, args: dict, analysis_type: str) -> Non get_authentication(params), integration=VERSION, trust_env=argToBoolean(params.get("proxy", False)), - verify_ssl=not params.get("insecure"), + verify_ssl=not argToBoolean(params.get("insecure", False)), + root_url=params.get("root_url") or DEFAULT_ROOT_URL, ) as connector: - start_analyse(args, analysis_type, connector) + start_analyse(params, args, analysis_type, connector) def detonate_entity_linux(params: dict, args: dict, analysis_type: str) -> None: # pragma: no cover @@ -126,9 +137,10 @@ def detonate_entity_linux(params: dict, args: dict, analysis_type: str) -> None: get_authentication(params), integration=VERSION, trust_env=argToBoolean(params.get("proxy", False)), - verify_ssl=not params.get("insecure"), + verify_ssl=not argToBoolean(params.get("insecure", False)), + root_url=params.get("root_url") or DEFAULT_ROOT_URL, ) as connector: - start_analyse(args, analysis_type, connector) + start_analyse(params, args, analysis_type, connector) def detonate_entity_android(params: dict, args: dict, analysis_type: str) -> None: # pragma: no cover @@ -136,9 +148,10 @@ def detonate_entity_android(params: dict, args: dict, analysis_type: str) -> Non get_authentication(params), integration=VERSION, trust_env=argToBoolean(params.get("proxy", False)), - verify_ssl=not params.get("insecure"), + verify_ssl=not argToBoolean(params.get("insecure", False)), + root_url=params.get("root_url") or DEFAULT_ROOT_URL, ) as connector: - start_analyse(args, analysis_type, connector) + start_analyse(params, args, analysis_type, connector) def detonate_file_widows(params: dict, args: dict) -> None: # pragma: no cover @@ -173,6 +186,7 @@ def delete_task(params: dict, args: dict) -> None: # pragma: no cover integration=VERSION, trust_env=argToBoolean(params.get("proxy", False)), verify_ssl=not params.get("insecure"), + root_url=params.get("root_url") or DEFAULT_ROOT_URL, ) as connector: connector.delete_task(task_uuid) @@ -187,10 +201,12 @@ def download_analysis_sample(params: dict, args: dict, download_type: str) -> No integration=VERSION, trust_env=argToBoolean(params.get("proxy", False)), verify_ssl=not params.get("insecure"), + root_url=params.get("root_url") or DEFAULT_ROOT_URL, ) as connector: if download_type == "pcap": return_results(fileResult(f"{task_uuid}_traffic_dump.pcap", connector.download_pcap(task_uuid))) - return_results(fileResult(f"{task_uuid}_sample.zip", connector.download_file_sample(task_uuid))) + else: + return_results(fileResult(f"{task_uuid}_sample.zip", connector.download_file_sample(task_uuid))) def get_analysis_verdict(params: dict, args: dict) -> None: # pragma: no cover @@ -201,6 +217,7 @@ def get_analysis_verdict(params: dict, args: dict) -> None: # pragma: no cover integration=VERSION, trust_env=argToBoolean(params.get("proxy", False)), verify_ssl=not params.get("insecure"), + root_url=params.get("root_url") or DEFAULT_ROOT_URL, ) as connector: for _ in connector.get_task_status(task_uuid): pass @@ -222,6 +239,7 @@ def get_user_limits(params: dict) -> None: # pragma: no cover integration=VERSION, trust_env=argToBoolean(params.get("proxy", False)), verify_ssl=not params.get("insecure"), + root_url=params.get("root_url") or DEFAULT_ROOT_URL, ) as connector: user_limits = connector.get_user_limits() @@ -240,6 +258,7 @@ def get_analysis_history(params: dict, args: dict) -> None: # pragma: no cover integration=VERSION, trust_env=argToBoolean(params.get("proxy", False)), verify_ssl=not params.get("insecure"), + root_url=params.get("root_url") or DEFAULT_ROOT_URL, ) as connector: analysis_history = connector.get_analysis_history(**args) @@ -252,7 +271,7 @@ def get_analysis_history(params: dict, args: dict) -> None: # pragma: no cover ) -def create_indicators(report: dict, task_uuid: str) -> None: # pragma: no cover +def create_indicators(report: dict, task_uuid: str, root_url: str) -> None: # pragma: no cover """ Excludes IOCs from the analysis report. Sends them to Threat Intel @@ -283,7 +302,7 @@ def create_indicators(report: dict, task_uuid: str) -> None: # pragma: no cover "fields": { "vendor": "ANY.RUN", "service": "ANY.RUN Cloud Sandbox", - "description": f"https://app.any.run/tasks/{task_uuid}", + "description": f"https://app.{root_url}/tasks/{task_uuid}", }, } ) @@ -314,27 +333,26 @@ def create_indicators(report: dict, task_uuid: str) -> None: # pragma: no cover def get_analysis_report(params: dict, args: dict) -> None: # pragma: no cover task_uuid = args.get("task_uuid", "") report_format = args.get("report_format") + root_url = params.get("root_url") or DEFAULT_ROOT_URL with SandboxConnector.windows( get_authentication(params), integration=VERSION, trust_env=argToBoolean(params.get("proxy", False)), verify_ssl=not params.get("insecure"), + root_url=root_url, ) as connector: + if report_format == "summary": + report_format = "json" + report = connector.get_analysis_report(task_uuid, report_format=report_format) if report_format == "html": return_results(fileResult(f"anyrun_report_{task_uuid}.html", report)) - elif report_format == "summary": - return_results( - CommandResults( - outputs_prefix="ANYRUN.SandboxAnalysis", - outputs=report, - ignore_auto_extract=True, - ) - ) + elif report_format == "json": + return_results(fileResult(f"anyrun_report_{task_uuid}.json", json.dumps(report))) elif report_format == "ioc" and report: - create_indicators(report, task_uuid) + create_indicators(report, task_uuid, root_url) return_results( CommandResults( @@ -377,6 +395,7 @@ def main(): # pragma: no cover elif demisto.command() == "anyrun-detonate-file-android": detonate_file_android(params, args) elif demisto.command() == "anyrun-detonate-url-android": + args.pop("obj_ext_browser", None) detonate_url_android(params, args) elif demisto.command() == "anyrun-get-analysis-report": get_analysis_report(params, args) @@ -387,6 +406,8 @@ def main(): # pragma: no cover raise NotImplementedError(f"Command {demisto.command()} is not implemented") except RunTimeException as exception: return_error(exception.description, error=str(exception.json)) + except Exception as e: + return_error(f"Failed to execute {demisto.command()} command.\nError:\n{str(e)}", error=traceback.format_exc()) if __name__ in ["__main__", "builtin", "builtins"]: diff --git a/Packs/ANYRUN/Integrations/AnyRunSandbox/AnyRunSandbox.yml b/Packs/ANYRUN/Integrations/AnyRunSandbox/AnyRunSandbox.yml index 3ee9eed39b94..1f9c0812b93e 100644 --- a/Packs/ANYRUN/Integrations/AnyRunSandbox/AnyRunSandbox.yml +++ b/Packs/ANYRUN/Integrations/AnyRunSandbox/AnyRunSandbox.yml @@ -21,6 +21,13 @@ configuration: type: 8 section: Connect required: false +- display: Root URL + name: root_url + type: 0 + required: false + section: Connect + defaultvalue: any.run + additionalinfo: Base domain for the ANY.RUN connection. Enter only the domain (e.g. any.run), without a scheme or trailing slash. description: ANY.RUN Sandbox is an online interactive sandbox for malware analysis, a tool for detection, monitoring, and research of cyber threats in real time. display: ANY.RUN Cloud Sandbox name: ANY.RUN Cloud Sandbox @@ -592,6 +599,7 @@ script: defaultValue: 'Google Chrome' required: false type: String + hidden: true predefined: - 'Google Chrome' - 'Mozilla Firefox' @@ -1450,7 +1458,7 @@ script: - contextPath: ANYRUN.SandboxAnalysisReportVerdict description: "The analysis verdict." type: String - dockerimage: demisto/anyrun-sdk:1.0.0.10120494 + dockerimage: demisto/anyrun-sdk:1.0.0.12844876 subtype: python3 runonce: false script: '-' diff --git a/Packs/ANYRUN/Integrations/AnyRunSandbox/AnyRunSandbox_description.md b/Packs/ANYRUN/Integrations/AnyRunSandbox/AnyRunSandbox_description.md index 5807f455869e..d06fe6e580e0 100644 --- a/Packs/ANYRUN/Integrations/AnyRunSandbox/AnyRunSandbox_description.md +++ b/Packs/ANYRUN/Integrations/AnyRunSandbox/AnyRunSandbox_description.md @@ -13,9 +13,10 @@ API-KEY from your ANY.RUN account is required. 3. Click **Add instance** to create and configure a new integration instance. 4. Insert ANY.RUN API-KEY into the **Password** parameter. 5. Please use "ANY.RUN" as username. -6Click **Test** to validate the URLs, token, and connection. +6. Click **Test** to validate the URLs, token, and connection. -| **Parameter** | **Description** | **Required** | -|------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------| --- | -| Password | ANY.RUN API-KEY without prefix | True | | True | +| **Parameter** | **Description** +|------------------|--------------------------------------------------------------------------| +| Password | ANY.RUN API-KEY without prefix +| Root URL | Base domain for the ANY.RUN connection. By default, the domain is any.run. diff --git a/Packs/ANYRUN/Integrations/AnyRunSandbox/README.md b/Packs/ANYRUN/Integrations/AnyRunSandbox/README.md index 6eab5c733e07..f25b4b9ea06f 100644 --- a/Packs/ANYRUN/Integrations/AnyRunSandbox/README.md +++ b/Packs/ANYRUN/Integrations/AnyRunSandbox/README.md @@ -39,6 +39,7 @@ Integration is officially supported from XSOAR 8.x | **Parameter** | **Description** | **Required** | |-----------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------| --- | | Password | ANY.RUN API-KEY without prefix. | True | +| Root URL | Base domain for the ANY.RUN connection. By default, the domain is any.run. | False | ## Commands @@ -246,7 +247,6 @@ Perform URL analysis using Android VM. | opt_network_residential_proxy_geo | Residential proxy geo location option. Example: US, AU. Default is fastest. | Optional | | opt_privacy_type | Privacy settings. Possible values are: public, bylink, owner, byteam. Default is bylink. | Optional | | opt_timeout | Timeout option. Size range: 10-660. Default is 120. | Optional | -| obj_ext_browser | Browser name. Possible values are: Google Chrome, Mozilla Firefox. Default is Google Chrome. | Optional | #### Context Output diff --git a/Packs/ANYRUN/Playbooks/ANYRUN_Detonate_URL_Android.yml b/Packs/ANYRUN/Playbooks/ANYRUN_Detonate_URL_Android.yml index 9c2a347b303d..d3d574676f2d 100644 --- a/Packs/ANYRUN/Playbooks/ANYRUN_Detonate_URL_Android.yml +++ b/Packs/ANYRUN/Playbooks/ANYRUN_Detonate_URL_Android.yml @@ -56,8 +56,6 @@ tasks: scriptarguments: env_locale: simple: ${inputs.env_locale} - obj_ext_browser: - simple: ${inputs.obj_ext_browser} obj_url: simple: ${inputs.obj_url} opt_network_connect: @@ -523,16 +521,10 @@ inputs: playbookInputQuery: - key: opt_timeout value: - simple: "60" + simple: "120" required: false description: 'Timeout option. Size range: 10-660.' playbookInputQuery: -- key: obj_ext_browser - value: - simple: Google Chrome - required: false - description: 'Browser name. Supports: Google Chrome, Mozilla Firefox' - playbookInputQuery: inputSections: - inputs: - obj_url @@ -554,10 +546,6 @@ inputSections: - opt_timeout name: Options description: "" -- inputs: - - obj_ext_browser - name: Object - description: "" outputSections: - outputs: [] name: General (Outputs group) diff --git a/Packs/ANYRUN/Playbooks/ANYRUN_Detonate_URL_Android_README.md b/Packs/ANYRUN/Playbooks/ANYRUN_Detonate_URL_Android_README.md index b49b0140dc91..ec42d1e633a1 100644 --- a/Packs/ANYRUN/Playbooks/ANYRUN_Detonate_URL_Android_README.md +++ b/Packs/ANYRUN/Playbooks/ANYRUN_Detonate_URL_Android_README.md @@ -39,8 +39,7 @@ This playbook does not use any sub-playbooks. | opt_network_residential_proxy | Residential proxy using. | False | Optional | | opt_network_residential_proxy_geo | Residential proxy geo location option. Example: US, AU. | fastest | Optional | | opt_privacy_type | Privacy settings. Supports: public, bylink, owner, byteam. | bylink | Optional | -| opt_timeout | Timeout option. Size range: 10-660. | 60 | Optional | -| obj_ext_browser | Browser name. Supports: Google Chrome, Mozilla Firefox | Google Chrome | Optional | +| opt_timeout | Timeout option. Size range: 10-660. | 120 | Optional | ## Playbook Outputs diff --git a/Packs/ANYRUN/Playbooks/ANYRUN_Detonate_URL_Linux.yml b/Packs/ANYRUN/Playbooks/ANYRUN_Detonate_URL_Linux.yml index a31095c0199f..4162cc787c91 100644 --- a/Packs/ANYRUN/Playbooks/ANYRUN_Detonate_URL_Linux.yml +++ b/Packs/ANYRUN/Playbooks/ANYRUN_Detonate_URL_Linux.yml @@ -2,8 +2,8 @@ id: ANYRUN Detonate Url Linux version: -1 name: ANYRUN Detonate Url Linux fromversion: 6.0.0 -description: This playbook submits a URL extracted from an indicator to the - ANY.RUN cloud sandbox for dynamic analysis in an Linux environment. +description: This playbook submits a URL extracted from an indicator to the + ANY.RUN cloud sandbox for dynamic analysis in an Linux environment. It automates the analysis of potentially malicious URLs on Linux OS. starttaskid: "0" tasks: @@ -228,7 +228,7 @@ tasks: - "6" scriptarguments: brandname: - simple: ANY.RUN Sandbox + simple: ANY.RUN Cloud Sandbox separatecontext: false continueonerrortype: "" view: |- diff --git a/Packs/ANYRUN/Playbooks/ANYRUN_Detonate_URL_Windows.yml b/Packs/ANYRUN/Playbooks/ANYRUN_Detonate_URL_Windows.yml index 7cd2f790d4df..1506767db6a8 100644 --- a/Packs/ANYRUN/Playbooks/ANYRUN_Detonate_URL_Windows.yml +++ b/Packs/ANYRUN/Playbooks/ANYRUN_Detonate_URL_Windows.yml @@ -550,7 +550,7 @@ inputs: playbookInputQuery: - key: opt_timeout value: - simple: "60" + simple: "120" required: false description: 'Timeout option. Size range: 10-660.' playbookInputQuery: diff --git a/Packs/ANYRUN/Playbooks/ANYRUN_Detonate_URL_Windows_README.md b/Packs/ANYRUN/Playbooks/ANYRUN_Detonate_URL_Windows_README.md index 032d016f576e..dc36459defea 100644 --- a/Packs/ANYRUN/Playbooks/ANYRUN_Detonate_URL_Windows_README.md +++ b/Packs/ANYRUN/Playbooks/ANYRUN_Detonate_URL_Windows_README.md @@ -42,7 +42,7 @@ This playbook does not use any sub-playbooks. | opt_network_residential_proxy | Residential proxy using. | False | Optional | | opt_network_residential_proxy_geo | Residential proxy geo location option. Example: US, AU. | fastest | Optional | | opt_privacy_type | Privacy settings. Supports: public, bylink, owner, byteam. | bylink | Optional | -| opt_timeout | Timeout option. Size range: 10-660. | 60 | Optional | +| opt_timeout | Timeout option. Size range: 10-660. | 120 | Optional | | obj_ext_browser | Browser name. Supports: Google Chrome, Mozilla Firefox, Internet Explorer, Microsoft Edge | Google Chrome | Optional | | obj_ext_extension | Change extension to valid | True | Optional | diff --git a/Packs/ANYRUN/ReleaseNotes/2_4_0.json b/Packs/ANYRUN/ReleaseNotes/2_4_0.json new file mode 100644 index 000000000000..1ca6c1d25e67 --- /dev/null +++ b/Packs/ANYRUN/ReleaseNotes/2_4_0.json @@ -0,0 +1,4 @@ +{ + "breakingChanges": true, + "breakingChangesNotes": "Removed the obj_ext_browser argument from the anyrun-detonate-url-android command and from the ANYRUN Detonate Url Android playbook. Android detonation does not support browser selection via the API. Custom playbooks or automations that pass this argument to anyrun-detonate-url-android or use it as an input to the ANYRUN Detonate Url Android playbook must be updated to remove it." +} diff --git a/Packs/ANYRUN/ReleaseNotes/2_4_0.md b/Packs/ANYRUN/ReleaseNotes/2_4_0.md new file mode 100644 index 000000000000..869fa1237aa2 --- /dev/null +++ b/Packs/ANYRUN/ReleaseNotes/2_4_0.md @@ -0,0 +1,36 @@ + +#### Integrations + +##### ANY.RUN Cloud Sandbox + +- New: Added the *Root URL* configuration parameter (default: *any.run*), used to build the interactive analysis link and IOC descriptions instead of a hardcoded *app.any.run* host. Falls back safely to the default when left blank. +- Updated the ***anyrun-detonate-url-android*** command by removing the *obj_ext_browser* argument. This option is not supported by ANY.RUN's Android sandbox API and caused detonation to fail. +- Updated the ***anyrun-download-analysis-sample*** command so it no longer returns both the pcap and the file sample when *download_type* is *pcap*; the branches are now mutually exclusive. +- Updated the ***anyrun-get-analysis-report*** command with *report_format=summary* to now return the report as a downloaded JSON file (*anyrun_report_\.json*) instead of a JSON context output. +- Updated the Docker image to: *demisto/anyrun-sdk:1.0.0.10440656*. +- Fixed an issue with a typo in the setup instructions. + +##### ANY.RUN TI Feed + +- New: Indicators now include *tags* (from feed labels) and a *publications* field populated from the feed's external references. Domain indicators additionally populate *communitynotes* with the same reference links. +- New: Added the *Incremental Feed* configuration parameter (enabled by default) so that indicators absent from a given delta fetch are no longer incorrectly flagged as "Removed from feed". +- Updated the Docker image to: *demisto/anyrun-sdk:1.0.0.10440656*. + +##### ANY.RUN TI Lookup + +- Updated the Docker image to: *demisto/anyrun-sdk:1.0.0.10440656*. + +#### Playbooks + +##### ANYRUN Detonate Url Android + +- Updated the playbook by removing the *obj_ext_browser* input, task argument, and input section. Android detonation does not support browser selection via the API, and passing this argument caused Android URL detonation to fail. +- Updated the playbook inputs: *opt_timeout* default changed from *60* to *120*. + +##### ANYRUN Detonate Url Linux + +- Fixed an issue with the *brandname* value used by the "Is there an active ANY.RUN instance?" condition task, from *ANY.RUN Sandbox* to *ANY.RUN Cloud Sandbox*, matching the integration's actual brand name. + +##### ANYRUN Detonate Url Windows + +- Updated the playbook inputs: *opt_timeout* default changed from *60* to *120*. diff --git a/Packs/ANYRUN/ReleaseNotes/2_4_1.md b/Packs/ANYRUN/ReleaseNotes/2_4_1.md new file mode 100644 index 000000000000..fcb38c440f0a --- /dev/null +++ b/Packs/ANYRUN/ReleaseNotes/2_4_1.md @@ -0,0 +1,11 @@ + +#### Integrations + +##### ANY.RUN TI Feed + +- Updated the Docker image to: *demisto/anyrun-sdk:1.0.0.12844876*. + +##### ANY.RUN Cloud Sandbox + +- Updated the Docker image to: *demisto/anyrun-sdk:1.0.0.12844876*. + diff --git a/Packs/ANYRUN/pack_metadata.json b/Packs/ANYRUN/pack_metadata.json index fee1316e0a22..1e216967e078 100644 --- a/Packs/ANYRUN/pack_metadata.json +++ b/Packs/ANYRUN/pack_metadata.json @@ -2,7 +2,7 @@ "name": "ANY.RUN", "description": "Empowers SOC teams with a Cloud Sandbox for real-time malware analysis, Threat Intelligence Lookup, and high-quality feeds to enhance detection and threat coverage.", "support": "partner", - "currentVersion": "2.3.5", + "currentVersion": "2.4.1", "author": "ANY.RUN", "url": "https://any.run/", "email": "techsupport@any.run", diff --git a/Packs/AWS-AccessAnalyzer/Integrations/AWS-AccessAnalyzer/AWS-AccessAnalyzer.yml b/Packs/AWS-AccessAnalyzer/Integrations/AWS-AccessAnalyzer/AWS-AccessAnalyzer.yml index 37c98dbd8fbe..0e59cd8bf2c8 100644 --- a/Packs/AWS-AccessAnalyzer/Integrations/AWS-AccessAnalyzer/AWS-AccessAnalyzer.yml +++ b/Packs/AWS-AccessAnalyzer/Integrations/AWS-AccessAnalyzer/AWS-AccessAnalyzer.yml @@ -61,6 +61,13 @@ configuration: type: 13 section: Collect required: false +- display: Incidents Fetch Interval + name: incidentFetchInterval + defaultvalue: '1' + required: false + type: 19 + section: Collect + advanced: true - display: Access Key name: credentials type: 9 @@ -330,7 +337,7 @@ script: name: roleSessionDuration description: Updates findings with the new values provided in the request. name: aws-access-analyzer-update-findings - dockerimage: demisto/boto3py3:1.0.0.10221838 + dockerimage: demisto/boto3py3:1.0.0.12887184 isfetch: true runonce: false script: '-' diff --git a/Packs/AWS-AccessAnalyzer/ReleaseNotes/1_1_45.md b/Packs/AWS-AccessAnalyzer/ReleaseNotes/1_1_45.md new file mode 100644 index 000000000000..4058c993f053 --- /dev/null +++ b/Packs/AWS-AccessAnalyzer/ReleaseNotes/1_1_45.md @@ -0,0 +1,6 @@ + +#### Integrations + +##### AWS - AccessAnalyzer + +- Updated the Fetch Incidents configuration parameters. diff --git a/Packs/AWS-AccessAnalyzer/ReleaseNotes/1_1_46.md b/Packs/AWS-AccessAnalyzer/ReleaseNotes/1_1_46.md new file mode 100644 index 000000000000..f0722f241b57 --- /dev/null +++ b/Packs/AWS-AccessAnalyzer/ReleaseNotes/1_1_46.md @@ -0,0 +1,7 @@ + +#### Integrations + +##### AWS - AccessAnalyzer + +- Updated the Docker image to: *demisto/boto3py3:1.0.0.12887184*. + diff --git a/Packs/AWS-AccessAnalyzer/pack_metadata.json b/Packs/AWS-AccessAnalyzer/pack_metadata.json index f00316f04445..3a8f4bf99e03 100644 --- a/Packs/AWS-AccessAnalyzer/pack_metadata.json +++ b/Packs/AWS-AccessAnalyzer/pack_metadata.json @@ -2,7 +2,7 @@ "name": "AWS - AccessAnalyzer", "description": "Amazon Web Services IAM Access Analyzer", "support": "xsoar", - "currentVersion": "1.1.44", + "currentVersion": "1.1.46", "author": "Cortex XSOAR", "url": "https://www.paloaltonetworks.com/cortex", "email": "", diff --git a/Packs/AWS-Athena/Integrations/AWS-Athena/AWS-Athena.yml b/Packs/AWS-Athena/Integrations/AWS-Athena/AWS-Athena.yml index 7c981120e02f..bcc3a5b6f0e8 100644 --- a/Packs/AWS-Athena/Integrations/AWS-Athena/AWS-Athena.yml +++ b/Packs/AWS-Athena/Integrations/AWS-Athena/AWS-Athena.yml @@ -431,7 +431,7 @@ script: - contextPath: AWS.Athena.QueryResults description: List of query results. type: List - dockerimage: demisto/boto3py3:1.0.0.10221838 + dockerimage: demisto/boto3py3:1.0.0.12887184 subtype: python3 tests: - No tests diff --git a/Packs/AWS-Athena/ReleaseNotes/2_0_13.md b/Packs/AWS-Athena/ReleaseNotes/2_0_13.md new file mode 100644 index 000000000000..2677693ef9c7 --- /dev/null +++ b/Packs/AWS-Athena/ReleaseNotes/2_0_13.md @@ -0,0 +1,7 @@ + +#### Integrations + +##### AWS - Athena + +- Updated the Docker image to: *demisto/boto3py3:1.0.0.12887184*. + diff --git a/Packs/AWS-Athena/pack_metadata.json b/Packs/AWS-Athena/pack_metadata.json index 531f5c061518..7526b9b4182c 100644 --- a/Packs/AWS-Athena/pack_metadata.json +++ b/Packs/AWS-Athena/pack_metadata.json @@ -2,7 +2,7 @@ "name": "AWS - Athena", "description": "Amazon Web Services Athena", "support": "xsoar", - "currentVersion": "2.0.12", + "currentVersion": "2.0.13", "author": "Cortex XSOAR", "url": "https://www.paloaltonetworks.com/cortex", "email": "", diff --git a/Packs/AWS-CloudTrail/Integrations/AWS-CloudTrail/AWS-CloudTrail.yml b/Packs/AWS-CloudTrail/Integrations/AWS-CloudTrail/AWS-CloudTrail.yml index 57af66be9671..860cfcba0f6b 100644 --- a/Packs/AWS-CloudTrail/Integrations/AWS-CloudTrail/AWS-CloudTrail.yml +++ b/Packs/AWS-CloudTrail/Integrations/AWS-CloudTrail/AWS-CloudTrail.yml @@ -432,7 +432,7 @@ script: - contextPath: AWS.CloudTrail.TrailStatus.LatestDigestDeliveryError description: Displays any Amazon S3 error that CloudTrail encountered when attempting to deliver a digest file to the designated bucket. type: string - dockerimage: demisto/boto3py3:1.0.0.10221838 + dockerimage: demisto/boto3py3:1.0.0.12887184 runonce: false script: '' subtype: python3 diff --git a/Packs/AWS-CloudTrail/ModelingRules/AWSCloudTrail/AWSCloudTrail.yml b/Packs/AWS-CloudTrail/ModelingRules/AWSCloudTrail/AWSCloudTrail.yml index d6bb68ef0043..16adf1577ebe 100644 --- a/Packs/AWS-CloudTrail/ModelingRules/AWSCloudTrail/AWSCloudTrail.yml +++ b/Packs/AWS-CloudTrail/ModelingRules/AWSCloudTrail/AWSCloudTrail.yml @@ -1,4 +1,5 @@ fromversion: 8.7.0 +toversion: 8.14.0 id: Amazon_AWS_CloudTrail_ModelingRule name: Amazon AWS CloudTrail Modeling Rule rules: '' diff --git a/Packs/AWS-CloudTrail/ModelingRules/AWSCloudTrail_2_11/AWSCloudTrail_2_11.xif b/Packs/AWS-CloudTrail/ModelingRules/AWSCloudTrail_2_11/AWSCloudTrail_2_11.xif new file mode 100644 index 000000000000..96662e813858 --- /dev/null +++ b/Packs/AWS-CloudTrail/ModelingRules/AWSCloudTrail_2_11/AWSCloudTrail_2_11.xif @@ -0,0 +1,55 @@ +[MODEL: dataset = amazon_aws_raw] +/* + CloudTrail Audit Logs + AWS event schema version: 1.11 + https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-event-reference-record-contents.html +*/ +filter _log_type = "Cloud Audit Log" +| alter // initializations + resource_arn = arraystring(arraymap(resources -> [], "@element" -> ARN), ","), + resource_account_id = arraystring(arraymap(resources -> [], "@element" -> accountId), ","), + resource_name = arraystring(arraymap(resources -> [], arrayindex(regextract("@element" -> ARN, "/([^\/]+)$"), -1)), ","), // extract last segment from ARN after last / + resource_type = arraystring(arraymap(resources -> [], "@element" -> type), ","), + user_identity_arn = userIdentity -> arn, + user_identity_type = userIdentity -> type, + source_ipv4 = if(sourceIPAddress ~= "(?:\d{1,3}\.){3}\d{1,3}", sourceIPAddress), // The IP address that the request was made from. For actions that originate from the service console, the address reported is for the underlying customer resource, not the console web server. + source_ipv6 = if(sourceIPAddress ~= "(?:[a-fA-F\d]{0,4}\:){2,7}[a-fA-F\d]{0,4}", sourceIPAddress) +| alter // XDM Mappings + xdm.event.description = object_create("requestParameters", requestParameters, "additionalEventData", additionalEventData, "responseElements", responseElements), + xdm.event.id = eventId, + xdm.event.operation_sub_type = eventName, + xdm.event.original_event_type = eventType, + xdm.event.outcome = if(errorCode = null, XDM_CONST.OUTCOME_SUCCESS, XDM_CONST.OUTCOME_FAILED), + xdm.event.outcome_reason = if(errorMessage != null and errorMessage != "", concat(errorCode, ": ", errorMessage), errorCode), + xdm.event.type = eventName, + xdm.network.session_id = requestID, + xdm.network.tls.cipher = tlsDetails -> cipherSuite, + xdm.network.tls.protocol_version = tlsDetails -> tlsVersion, + xdm.observer.content_version = eventVersion, + xdm.observer.name = eventSource, + xdm.observer.type = eventCategory, + xdm.session_context_id = sharedEventID, + xdm.source.cloud.project_id = userIdentity -> accountId, + xdm.source.cloud.provider = XDM_CONST.CLOUD_PROVIDER_AWS, + xdm.source.host.device_id = vpcEndpointId, + xdm.source.host.ipv4_public_addresses = arraymap(split(source_ipv4, ", "), if(not incidr(trim("@element"), "10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, 127.0.0.0/8, 169.254.0.0/16, 100.64.0.0/10"),trim("@element"))), + xdm.source.host.fqdn = if(source_ipv4 = null and source_ipv6 = null and sourceIPAddress !~= "AWS Internal", sourceIPAddress), // For services in AWS, only the DNS name is displayed. + xdm.source.ipv4 = source_ipv4, + xdm.source.ipv6 = source_ipv6, + xdm.source.user_agent = userAgent, + xdm.source.user.groups = if(user_identity_type in ("Role", "AssumedRole"), arraycreate(userIdentity -> sessionContext.sessionIssuer.userName)), + xdm.source.identity.groups = if(user_identity_type in ("Role", "AssumedRole"), arraycreate(userIdentity -> sessionContext.sessionIssuer.userName)), + xdm.source.user.identifier = coalesce(userIdentity -> principalId, userIdentity -> onBehalfOf.userId, user_identity_arn), + xdm.source.identity.identifier = coalesce(userIdentity -> principalId, userIdentity -> onBehalfOf.userId, user_identity_arn), + xdm.source.user.username = coalesce(userIdentity -> userName, userIdentity -> sessionContext.sessionIssuer.userName), + xdm.source.identity.username = coalesce(userIdentity -> userName, userIdentity -> sessionContext.sessionIssuer.userName), + xdm.source.user.user_type = if(user_identity_type = "Unknown", XDM_CONST.IDENTITY_TYPE_UNKNOWN, user_identity_type), + xdm.source.identity.user_type = if(user_identity_type = "Unknown", XDM_CONST.IDENTITY_TYPE_UNKNOWN, user_identity_type), + xdm.target.cloud.project_id = if(resource_account_id != null and resource_account_id != "", resource_account_id, coalesce(recipientAccountId, vpcEndpointAccountId)), + xdm.target.cloud.provider = XDM_CONST.CLOUD_PROVIDER_AWS, + xdm.target.cloud.region = awsRegion, + xdm.target.host.fqdn = coalesce(requestParameters -> Host, tlsDetails -> clientProvidedHostHeader), + xdm.target.host.hostname = coalesce(requestParameters -> Host, tlsDetails -> clientProvidedHostHeader), + xdm.target.resource.id = if(resource_arn != "", resource_arn), + xdm.target.resource.name = if(resource_name != null and resource_name != "", resource_name, resource_arn), + xdm.target.resource.type = if(resource_type != "", resource_type); \ No newline at end of file diff --git a/Packs/AWS-CloudTrail/ModelingRules/AWSCloudTrail_2_11/AWSCloudTrail_2_11.yml b/Packs/AWS-CloudTrail/ModelingRules/AWSCloudTrail_2_11/AWSCloudTrail_2_11.yml new file mode 100644 index 000000000000..6294afbf0e1e --- /dev/null +++ b/Packs/AWS-CloudTrail/ModelingRules/AWSCloudTrail_2_11/AWSCloudTrail_2_11.yml @@ -0,0 +1,8 @@ +fromversion: 8.15.0 +id: Amazon_AWS_CloudTrail_ModelingRule +name: Amazon AWS CloudTrail Modeling Rule +rules: '' +schema: '' +tags: +supportedModules: +- xsiam diff --git a/Packs/AWS-CloudTrail/ModelingRules/AWSCloudTrail_2_11/AWSCloudTrail_2_11_schema.json b/Packs/AWS-CloudTrail/ModelingRules/AWSCloudTrail_2_11/AWSCloudTrail_2_11_schema.json new file mode 100644 index 000000000000..ec814b8e9e3c --- /dev/null +++ b/Packs/AWS-CloudTrail/ModelingRules/AWSCloudTrail_2_11/AWSCloudTrail_2_11_schema.json @@ -0,0 +1,104 @@ +{ + "amazon_aws_raw": { + "_log_type": { + "type": "string", + "is_array": false + }, + "additionalEventData": { + "type": "string", + "is_array": false + }, + "awsRegion": { + "type": "string", + "is_array": false + }, + "errorCode": { + "type": "string", + "is_array": false + }, + "errorMessage": { + "type": "string", + "is_array": false + }, + "eventCategory": { + "type": "string", + "is_array": false + }, + "eventID": { + "type": "string", + "is_array": false + }, + "eventName": { + "type": "string", + "is_array": false + }, + "eventSource": { + "type": "string", + "is_array": false + }, + "eventType": { + "type": "string", + "is_array": false + }, + "eventVersion": { + "type": "string", + "is_array": false + }, + "recipientAccountId": { + "type": "string", + "is_array": false + }, + "requestID": { + "type": "string", + "is_array": false + }, + "requestParameters": { + "type": "string", + "is_array": false + }, + "responseElements": { + "type": "string", + "is_array": false + }, + "resources": { + "type": "string", + "is_array": false + }, + "sharedEventID": { + "type": "string", + "is_array": false + }, + "sourceIPAddress": { + "type": "string", + "is_array": false + }, + "tlsDetails": { + "type": "string", + "is_array": false + }, + "userAgent": { + "type": "string", + "is_array": false + }, + "userIdentity": { + "type": "string", + "is_array": false + }, + "vpcEndpointAccountId": { + "type": "string", + "is_array": false + }, + "vpcEndpointId": { + "type": "string", + "is_array": false + }, + "eventTime": { + "type": "datetime", + "is_array": false + }, + "start": { + "type": "string", + "is_array": false + } + } +} \ No newline at end of file diff --git a/Packs/AWS-CloudTrail/ReleaseNotes/1_1_17.md b/Packs/AWS-CloudTrail/ReleaseNotes/1_1_17.md new file mode 100644 index 000000000000..7b1fb360d423 --- /dev/null +++ b/Packs/AWS-CloudTrail/ReleaseNotes/1_1_17.md @@ -0,0 +1,7 @@ + +#### Modeling Rules + +##### New: AWS-CloudTrail Modeling Rule + +- New: Added a new modeling rule version that maps the `xdm.*.identity.*` fields alongside the existing `xdm.*.user.*` fields. +<~XSIAM> (Available from Cortex XSIAM 2.11). diff --git a/Packs/AWS-CloudTrail/ReleaseNotes/1_1_18.md b/Packs/AWS-CloudTrail/ReleaseNotes/1_1_18.md new file mode 100644 index 000000000000..79198b55ef14 --- /dev/null +++ b/Packs/AWS-CloudTrail/ReleaseNotes/1_1_18.md @@ -0,0 +1,4 @@ + +## AWS-CloudTrail + +- Documentation and metadata improvements. diff --git a/Packs/AWS-CloudTrail/ReleaseNotes/1_1_19.md b/Packs/AWS-CloudTrail/ReleaseNotes/1_1_19.md new file mode 100644 index 000000000000..334530766499 --- /dev/null +++ b/Packs/AWS-CloudTrail/ReleaseNotes/1_1_19.md @@ -0,0 +1,7 @@ + +#### Integrations + +##### AWS - CloudTrail + +- Updated the Docker image to: *demisto/boto3py3:1.0.0.12887184*. + diff --git a/Packs/AWS-CloudTrail/ReleaseNotes/1_1_20.md b/Packs/AWS-CloudTrail/ReleaseNotes/1_1_20.md new file mode 100644 index 000000000000..f619bfbc7153 --- /dev/null +++ b/Packs/AWS-CloudTrail/ReleaseNotes/1_1_20.md @@ -0,0 +1,7 @@ + +#### Modeling Rules + +##### New: Amazon AWS CloudTrail Modeling Rule + +- New: Added a new modeling rule version that maps the `xdm.*.identity.*` fields alongside the existing `xdm.*.user.*` fields. +<~XSIAM> (Available from Cortex XSIAM 2.11). diff --git a/Packs/AWS-CloudTrail/pack_metadata.json b/Packs/AWS-CloudTrail/pack_metadata.json index 93eb62302da6..c0fbc2c60c18 100644 --- a/Packs/AWS-CloudTrail/pack_metadata.json +++ b/Packs/AWS-CloudTrail/pack_metadata.json @@ -2,7 +2,7 @@ "name": "AWS - CloudTrail", "description": "Amazon Web Services CloudTrail.", "support": "xsoar", - "currentVersion": "1.1.16", + "currentVersion": "1.1.20", "author": "Cortex XSOAR", "url": "https://www.paloaltonetworks.com/cortex", "email": "", diff --git a/Packs/AWS-EC2/ReleaseNotes/1_4_33.md b/Packs/AWS-EC2/ReleaseNotes/1_4_33.md new file mode 100644 index 000000000000..1d9109a0b698 --- /dev/null +++ b/Packs/AWS-EC2/ReleaseNotes/1_4_33.md @@ -0,0 +1,11 @@ + +#### Scripts + +##### AwsEC2GetPublicSGRules + +- Updated the Docker image to: *demisto/python3:3.12.14.13053055*. + +##### AwsEC2SyncAccounts + +- Updated the Docker image to: *demisto/python3:3.12.14.13053055*. + diff --git a/Packs/AWS-EC2/ReleaseNotes/1_4_34.md b/Packs/AWS-EC2/ReleaseNotes/1_4_34.md new file mode 100644 index 000000000000..97148ac2fd0f --- /dev/null +++ b/Packs/AWS-EC2/ReleaseNotes/1_4_34.md @@ -0,0 +1,3 @@ +## AWS - EC2 + +- Locked dependencies of the pack to ensure stability for versioned core packs. No changes in this release. diff --git a/Packs/AWS-EC2/Scripts/AwsEC2GetPublicSGRules/AwsEC2GetPublicSGRules.yml b/Packs/AWS-EC2/Scripts/AwsEC2GetPublicSGRules/AwsEC2GetPublicSGRules.yml index ce88c41cc48f..386eb9eed41e 100644 --- a/Packs/AWS-EC2/Scripts/AwsEC2GetPublicSGRules/AwsEC2GetPublicSGRules.yml +++ b/Packs/AWS-EC2/Scripts/AwsEC2GetPublicSGRules/AwsEC2GetPublicSGRules.yml @@ -62,6 +62,6 @@ tags: timeout: '0' type: python subtype: python3 -dockerimage: demisto/python3:3.12.13.10116658 +dockerimage: demisto/python3:3.12.14.13053055 runas: DBotWeakRole fromversion: 5.0.0 diff --git a/Packs/AWS-EC2/Scripts/AwsEC2SyncAccounts/AwsEC2SyncAccounts.yml b/Packs/AWS-EC2/Scripts/AwsEC2SyncAccounts/AwsEC2SyncAccounts.yml index 6a98354da6dd..66f69f29c1de 100644 --- a/Packs/AWS-EC2/Scripts/AwsEC2SyncAccounts/AwsEC2SyncAccounts.yml +++ b/Packs/AWS-EC2/Scripts/AwsEC2SyncAccounts/AwsEC2SyncAccounts.yml @@ -23,7 +23,7 @@ tags: timeout: '0' type: python subtype: python3 -dockerimage: demisto/python3:3.12.13.10116658 +dockerimage: demisto/python3:3.12.14.13053055 runas: DBotWeakRole fromversion: 6.10.0 tests: diff --git a/Packs/AWS-EC2/pack_metadata.json b/Packs/AWS-EC2/pack_metadata.json index 5890dec01302..62628dfc3229 100644 --- a/Packs/AWS-EC2/pack_metadata.json +++ b/Packs/AWS-EC2/pack_metadata.json @@ -2,7 +2,7 @@ "name": "AWS - EC2", "description": "Amazon Web Services Elastic Compute Cloud (EC2)", "support": "xsoar", - "currentVersion": "1.4.32", + "currentVersion": "1.4.34", "author": "Cortex XSOAR", "url": "https://www.paloaltonetworks.com/cortex", "email": "", diff --git a/Packs/AWS-EKS/Integrations/AWSEKS/AWSEKS.yml b/Packs/AWS-EKS/Integrations/AWSEKS/AWSEKS.yml index 6de449ba2fc1..ae9f9eb971e5 100644 --- a/Packs/AWS-EKS/Integrations/AWSEKS/AWSEKS.yml +++ b/Packs/AWS-EKS/Integrations/AWSEKS/AWSEKS.yml @@ -539,7 +539,7 @@ script: script: '-' type: python subtype: python3 - dockerimage: demisto/boto3py3:1.0.0.10221838 + dockerimage: demisto/boto3py3:1.0.0.12887184 fromversion: 6.9.0 tests: - AWSEKS-Test diff --git a/Packs/AWS-EKS/ReleaseNotes/1_0_15.md b/Packs/AWS-EKS/ReleaseNotes/1_0_15.md new file mode 100644 index 000000000000..712bcf8c6f7d --- /dev/null +++ b/Packs/AWS-EKS/ReleaseNotes/1_0_15.md @@ -0,0 +1,7 @@ + +#### Integrations + +##### AWS-EKS + +- Updated the Docker image to: *demisto/boto3py3:1.0.0.12887184*. + diff --git a/Packs/AWS-EKS/pack_metadata.json b/Packs/AWS-EKS/pack_metadata.json index d58a128c775a..05663c978449 100644 --- a/Packs/AWS-EKS/pack_metadata.json +++ b/Packs/AWS-EKS/pack_metadata.json @@ -2,7 +2,7 @@ "name": "AWS - EKS", "description": "The AWS EKS integration allows for the management and operation of Amazon Elastic Kubernetes Service (EKS) clusters.", "support": "xsoar", - "currentVersion": "1.0.14", + "currentVersion": "1.0.15", "author": "Cortex XSOAR", "url": "https://www.paloaltonetworks.com/cortex", "email": "", diff --git a/Packs/AWS-Enrichment-Remediation/ReleaseNotes/1_1_34.md b/Packs/AWS-Enrichment-Remediation/ReleaseNotes/1_1_34.md new file mode 100644 index 000000000000..d14d93d4bf65 --- /dev/null +++ b/Packs/AWS-Enrichment-Remediation/ReleaseNotes/1_1_34.md @@ -0,0 +1,11 @@ + +#### Scripts + +##### AWSRecreateSG + +- Updated the Docker image to: *demisto/python3:3.12.14.13053055*. + +##### AWSAccountHierarchy + +- Updated the Docker image to: *demisto/python3:3.12.14.13053055*. + diff --git a/Packs/AWS-Enrichment-Remediation/ReleaseNotes/1_1_35.md b/Packs/AWS-Enrichment-Remediation/ReleaseNotes/1_1_35.md new file mode 100644 index 000000000000..ea470956433d --- /dev/null +++ b/Packs/AWS-Enrichment-Remediation/ReleaseNotes/1_1_35.md @@ -0,0 +1,3 @@ +## AWS Enrichment and Remediation + +- Locked dependencies of the pack to ensure stability for versioned core packs. No changes in this release. diff --git a/Packs/AWS-Enrichment-Remediation/Scripts/AWSAccountHierarchy/AWSAccountHierarchy.yml b/Packs/AWS-Enrichment-Remediation/Scripts/AWSAccountHierarchy/AWSAccountHierarchy.yml index 31f74028222d..3a39de16d53f 100644 --- a/Packs/AWS-Enrichment-Remediation/Scripts/AWSAccountHierarchy/AWSAccountHierarchy.yml +++ b/Packs/AWS-Enrichment-Remediation/Scripts/AWSAccountHierarchy/AWSAccountHierarchy.yml @@ -6,7 +6,7 @@ comment: Determine AWS account hierarchy by looking up parent objects until the commonfields: id: AWSAccountHierarchy version: -1 -dockerimage: demisto/python3:3.12.13.10116658 +dockerimage: demisto/python3:3.12.14.13053055 enabled: true engineinfo: {} name: AWSAccountHierarchy diff --git a/Packs/AWS-Enrichment-Remediation/Scripts/AWSRecreateSG/AWSRecreateSG.yml b/Packs/AWS-Enrichment-Remediation/Scripts/AWSRecreateSG/AWSRecreateSG.yml index 559b45a43ae9..a184cdb134bf 100644 --- a/Packs/AWS-Enrichment-Remediation/Scripts/AWSRecreateSG/AWSRecreateSG.yml +++ b/Packs/AWS-Enrichment-Remediation/Scripts/AWSRecreateSG/AWSRecreateSG.yml @@ -51,7 +51,7 @@ dependson: - AWS - EC2|||aws-ec2-authorize-security-group-egress-rule - AWS - EC2|||aws-ec2-revoke-security-group-ingress-rule - AWS - EC2|||aws-ec2-revoke-security-group-egress-rule -dockerimage: demisto/python3:3.12.13.10116658 +dockerimage: demisto/python3:3.12.14.13053055 enabled: true name: AWSRecreateSG runas: DBotWeakRole diff --git a/Packs/AWS-Enrichment-Remediation/pack_metadata.json b/Packs/AWS-Enrichment-Remediation/pack_metadata.json index cf8d3f65fded..d3bac92fba9c 100644 --- a/Packs/AWS-Enrichment-Remediation/pack_metadata.json +++ b/Packs/AWS-Enrichment-Remediation/pack_metadata.json @@ -2,7 +2,7 @@ "name": "AWS Enrichment and Remediation", "description": "Playbooks using multiple AWS content packs for enrichment and remediation purposes", "support": "xsoar", - "currentVersion": "1.1.33", + "currentVersion": "1.1.35", "author": "Cortex XSOAR", "url": "https://www.paloaltonetworks.com/cortex", "email": "", diff --git a/Packs/AWS-GuardDuty/.pack-ignore b/Packs/AWS-GuardDuty/.pack-ignore index 459381d1be37..6da161a77844 100644 --- a/Packs/AWS-GuardDuty/.pack-ignore +++ b/Packs/AWS-GuardDuty/.pack-ignore @@ -1,5 +1,5 @@ [file:AWSGuardDuty.yml] -ignore=IN126,BA108,BA109 +ignore=IN126,BA108,BA109,BA129 [known_words] gd diff --git a/Packs/AWS-GuardDuty/.secrets-ignore b/Packs/AWS-GuardDuty/.secrets-ignore index e093c0f393c6..bcfe50567e4a 100644 --- a/Packs/AWS-GuardDuty/.secrets-ignore +++ b/Packs/AWS-GuardDuty/.secrets-ignore @@ -1 +1,4 @@ -https://docs.aws.amazon.com/guardduty \ No newline at end of file +https://docs.aws.amazon.com/guardduty +https://s3.eu-central-1.amazonaws.com +https://s3.us-west-2.amazonaws.com +https://s3.amazonaws.com diff --git a/Packs/AWS-GuardDuty/Integrations/AWSGuardDuty/AWSGuardDuty.py b/Packs/AWS-GuardDuty/Integrations/AWSGuardDuty/AWSGuardDuty.py index c83ef8a8de31..20f443a5b919 100644 --- a/Packs/AWS-GuardDuty/Integrations/AWSGuardDuty/AWSGuardDuty.py +++ b/Packs/AWS-GuardDuty/Integrations/AWSGuardDuty/AWSGuardDuty.py @@ -22,6 +22,7 @@ DATE_FORMAT = "%Y-%m-%dT%H:%M:%S.%fZ" MAX_RESULTS_RESPONSE = 50 +MAX_TAGS = 50 class DatetimeEncoder(json.JSONEncoder): @@ -34,6 +35,39 @@ def default(self, obj: Any): # pylint: disable=E0202 return json.JSONEncoder.default(self, obj) +def parse_tag_field_to_dict(tags_string: str) -> dict: + """ + Parses a list representation of key and value with the form of 'key=,value=' into a + string-to-string map suitable for the GuardDuty API Tags parameter. You can specify up to 50 tags + per resource. The value may contain commas; only the ";" character separates pairs. + + Args: + tags_string: The name and value list, e.g. "key=Environment,value=prod;key=Team,value=sec". + + Returns: + A dict of the form {"": ""}, e.g. {"Environment": "prod", "Team": "sec"}. + """ + tags: dict[str, str] = {} + list_tags = argToList(tags_string, separator=";") + if len(list_tags) > MAX_TAGS: + demisto.debug( + f"[AWSGuardDuty] parse_tag_field_to_dict: The current number of tags is {len(list_tags)}. Number of tags exceeds " + f"{MAX_TAGS}, parsing only the first {MAX_TAGS} tags." + ) + list_tags = list_tags[0:MAX_TAGS] + # According to the AWS Tag restrictions docs. + regex = re.compile(r"^key=([a-zA-Z0-9\s+\-=._:/@]{1,128}),value=(.{0,256})$", flags=re.UNICODE) + for tag in list_tags: + match_tag = regex.match(tag) + if match_tag is None: + raise ValueError( + f"Could not parse field: {tag}. Please make sure you provided like so: key=abc,value=123;key=fed,value=456" + ) + tags[match_tag.group(1)] = match_tag.group(2) + demisto.debug(f"[AWSGuardDuty] parse_tag_field_to_dict: parsed {len(tags)} tag(s) with keys {list(tags.keys())}.") + return tags + + def create_detector(client: "GuardDutyClient", args: dict) -> CommandResults: """ Creates a single Amazon GuardDuty detector. @@ -356,6 +390,189 @@ def update_threat_intel_set(client: "GuardDutyClient", args: dict): raise Exception(f"Failed updating ThreatIntel set {args.get('threatIntelSetId')}. Response was: {response}") +def create_threat_entity_set(client: "GuardDutyClient", args: dict) -> CommandResults: + detector_id = args.get("detectorId") + demisto.debug( + f"[AWSGuardDuty] create_threat_entity_set: started for Detector {detector_id}, " + f"name={args.get('name')}, format={args.get('format')}, activate={args.get('activate')}, " + f"location_provided={bool(args.get('location'))}, " + f"expectedBucketOwner_provided={bool(args.get('expectedBucketOwner'))}, " + f"tags_provided={bool(args.get('tags'))}." + ) + tags = args.get("tags") + kwargs = remove_empty_elements( + { + "DetectorId": detector_id, + "Activate": arg_to_bool_or_none(args.get("activate")), + "Format": args.get("format"), + "Location": args.get("location"), + "Name": args.get("name"), + "ExpectedBucketOwner": args.get("expectedBucketOwner"), + "Tags": parse_tag_field_to_dict(tags) if tags else None, + } + ) + + demisto.debug( + f"[AWSGuardDuty] create_threat_entity_set: calling create_threat_entity_set for Detector " + f"{detector_id} with request fields: {list(kwargs.keys())}." + ) + response = client.create_threat_entity_set(**kwargs) + demisto.debug(f"[AWSGuardDuty] create_threat_entity_set: API response keys: {list(response.keys())}.") + + data = {"DetectorId": detector_id, "ThreatEntitySetId": response.get("ThreatEntitySetId")} + demisto.debug( + f"[AWSGuardDuty] create_threat_entity_set: created Threat Entity Set {data['ThreatEntitySetId']} " + f"for Detector {detector_id}. Returning CommandResults." + ) + + readable_output = tableToMarkdown("AWS GuardDuty ThreatEntity Set", data) + return CommandResults( + readable_output=readable_output, + outputs=data, + outputs_prefix="AWS.GuardDuty.ThreatEntitySet", + outputs_key_field="ThreatEntitySetId", + ) + + +def update_threat_entity_set(client: "GuardDutyClient", args: dict) -> CommandResults: + threat_entity_set_id = args.get("threatEntitySetId") + demisto.debug( + f"[AWSGuardDuty] update_threat_entity_set: started for Threat Entity Set {threat_entity_set_id} " + f"of Detector {args.get('detectorId')}, activate={args.get('activate')}, " + f"location_provided={bool(args.get('location'))}, name={args.get('name')}, " + f"expectedBucketOwner_provided={bool(args.get('expectedBucketOwner'))}." + ) + kwargs = remove_empty_elements( + { + "DetectorId": args.get("detectorId"), + "ThreatEntitySetId": threat_entity_set_id, + "Activate": arg_to_bool_or_none(args.get("activate")), + "Location": args.get("location"), + "Name": args.get("name"), + "ExpectedBucketOwner": args.get("expectedBucketOwner"), + } + ) + + demisto.debug( + f"[AWSGuardDuty] update_threat_entity_set: calling update_threat_entity_set for Threat Entity Set " + f"{threat_entity_set_id} of Detector {args.get('detectorId')} with request fields: {list(kwargs.keys())}." + ) + response = client.update_threat_entity_set(**kwargs) + demisto.debug( + f"[AWSGuardDuty] update_threat_entity_set: {threat_entity_set_id=} API response HTTPStatusCode=" + f"{response.get('ResponseMetadata', {}).get('HTTPStatusCode')}." + ) + + if response == {} or response.get("ResponseMetadata", {}).get("HTTPStatusCode") == 200: + return CommandResults(readable_output=f"Threat entity set {threat_entity_set_id} was updated successfully") + else: + status_code = response.get("ResponseMetadata", {}).get("HTTPStatusCode") + raise DemistoException(f"Failed updating Threat Entity set {threat_entity_set_id}. HTTPStatusCode was: {status_code}") + + +def get_threat_entity_set(client: "GuardDutyClient", args: dict) -> CommandResults: + detector_id = args.get("detectorId") + threat_entity_set_id = args.get("threatEntitySetId") + demisto.debug( + f"[AWSGuardDuty] get_threat_entity_set: started for Threat Entity Set {threat_entity_set_id} " + f"of Detector {detector_id}." + ) + response = client.get_threat_entity_set(DetectorId=detector_id, ThreatEntitySetId=threat_entity_set_id) + demisto.debug( + f"[AWSGuardDuty] get_threat_entity_set: API response keys: {list(response.keys())}, " f"Status={response.get('Status')}." + ) + data = remove_empty_elements( + { + "DetectorId": detector_id, + "ThreatEntitySetId": threat_entity_set_id, + "Name": response.get("Name"), + "Format": response.get("Format"), + "Location": response.get("Location"), + "Status": response.get("Status"), + "ExpectedBucketOwner": response.get("ExpectedBucketOwner"), + "CreatedAt": response.get("CreatedAt"), + "UpdatedAt": response.get("UpdatedAt"), + "Tags": response.get("Tags"), + } + ) + demisto.debug( + f"[AWSGuardDuty] get_threat_entity_set: returning {len(data)} non-empty field(s) for Threat Entity Set " + f"{threat_entity_set_id}." + ) + + return CommandResults( + readable_output=tableToMarkdown("AWS GuardDuty ThreatEntity Set", data), + outputs=data, + outputs_prefix="AWS.GuardDuty.ThreatEntitySet", + outputs_key_field="ThreatEntitySetId", + ) + + +def delete_threat_entity_set(client: "GuardDutyClient", args: dict) -> CommandResults: + threat_entity_set_id = args.get("threatEntitySetId") + detector_id = args.get("detectorId") + demisto.debug( + f"[AWSGuardDuty] delete_threat_entity_set: started for Threat Entity Set {threat_entity_set_id} " + f"from Detector {detector_id}." + ) + response = client.delete_threat_entity_set(DetectorId=detector_id, ThreatEntitySetId=threat_entity_set_id) + demisto.debug( + f"[AWSGuardDuty] delete_threat_entity_set: {threat_entity_set_id=} API response HTTPStatusCode=" + f"{response.get('ResponseMetadata', {}).get('HTTPStatusCode')}." + ) + if response == {} or response.get("ResponseMetadata", {}).get("HTTPStatusCode") == 200: + return CommandResults( + readable_output=f"Threat Entity Set {threat_entity_set_id} was deleted from Detector " f"{detector_id} successfully" + ) + else: + status_code = response.get("ResponseMetadata", {}).get("HTTPStatusCode") + raise DemistoException(f"Failed to delete Threat Entity set {threat_entity_set_id}. HTTPStatusCode was: {status_code}") + + +def list_threat_entity_sets(client: "GuardDutyClient", args: dict) -> CommandResults: + detector_id = args.get("detectorId") + limit, page_size, page = get_pagination_args(args) + demisto.debug( + f"[AWSGuardDuty] list_threat_entity_sets: started for Detector {detector_id} " + f"with limit={limit}, page_size={page_size}, page={page}." + ) + + paginator = client.get_paginator("list_threat_entity_sets") + response_iterator = paginator.paginate( + DetectorId=detector_id, + PaginationConfig={ + "MaxItems": limit, + "PageSize": page_size, + }, + ) + demisto.debug("[AWSGuardDuty] list_threat_entity_sets: paginator created, iterating pages.") + + data = [] + for i, page_response in enumerate(response_iterator): + if page is None or (page - 1) == i: + page_ids = page_response.get("ThreatEntitySetIds", []) + demisto.debug(f"[AWSGuardDuty] list_threat_entity_sets: page index {i} returned {len(page_ids)} ID(s).") + for threat_entity_set in page_ids: + data.append({"DetectorId": detector_id, "ThreatEntitySetId": threat_entity_set}) + if page: + break + + demisto.debug( + f"[AWSGuardDuty] list_threat_entity_sets: found {len(data)} Threat Entity Set(s) for Detector " + f"{detector_id}. Returning CommandResults." + ) + + readable_output = tableToMarkdown( + "AWS GuardDuty ThreatEntity Sets", data, headers=["DetectorId", "ThreatEntitySetId"], removeNull=True + ) + return CommandResults( + readable_output=readable_output, + outputs=data, + outputs_prefix="AWS.GuardDuty.ThreatEntitySet", + outputs_key_field="ThreatEntitySetId", + ) + + def severity_mapping(severity: Optional[float]) -> Optional[int]: demisto_severity = None if severity: @@ -836,6 +1053,21 @@ def main(): # pragma: no cover elif demisto.command() == "aws-gd-update-threatintel-set": result = update_threat_intel_set(client, demisto.args()) + elif demisto.command() == "aws-gd-create-threat-entity-set": + result = create_threat_entity_set(client, args) + + elif demisto.command() == "aws-gd-update-threat-entity-set": + result = update_threat_entity_set(client, args) + + elif demisto.command() == "aws-gd-get-threat-entity-set": + result = get_threat_entity_set(client, args) + + elif demisto.command() == "aws-gd-delete-threat-entity-set": + result = delete_threat_entity_set(client, args) + + elif demisto.command() == "aws-gd-list-threat-entity-sets": + result = list_threat_entity_sets(client, args) + elif demisto.command() == "aws-gd-list-findings": result = list_findings(client, demisto.args()) diff --git a/Packs/AWS-GuardDuty/Integrations/AWSGuardDuty/AWSGuardDuty.yml b/Packs/AWS-GuardDuty/Integrations/AWSGuardDuty/AWSGuardDuty.yml index 66060bbff980..04c5298714e9 100644 --- a/Packs/AWS-GuardDuty/Integrations/AWSGuardDuty/AWSGuardDuty.yml +++ b/Packs/AWS-GuardDuty/Integrations/AWSGuardDuty/AWSGuardDuty.yml @@ -621,6 +621,181 @@ script: name: roleSessionDuration description: Updates the ThreatIntelSet specified by ThreatIntelSet ID. name: aws-gd-update-threatintel-set + - name: aws-gd-create-threat-entity-set + description: Creates a new threat entity set. In a threat entity set, you can provide known malicious threat entities for your AWS environment. + execution: true + arguments: + - name: detectorId + description: The unique ID of the detector that will own the threat entity set. + required: true + - name: name + description: 'The user-friendly name of the threat entity set. Length 1-64.' + required: true + - name: format + description: The format of the entity-list file. + auto: PREDEFINED + predefined: + - TXT + - STIX + - OTX_CSV + - ALIEN_VAULT + - PROOF_POINT + - FIRE_EYE + required: true + - name: location + description: 'The S3 URI of the entity-list file. For example, https://s3.us-east-1.amazonaws.com/example-bucket/threat-entities.txt.' + required: true + - name: activate + description: Whether GuardDuty should start using the uploaded set immediately. + required: true + auto: PREDEFINED + predefined: + - 'True' + - 'False' + - name: expectedBucketOwner + description: The AWS account ID that owns the S3 bucket in the location parameter. Prevents cross-account spoofing. + - name: tags + description: 'The tags to add to the new threat entity set. Format: key=,value=. Separate multiple tags with a semicolon (;). For example, key=Environment,value=prod;key=Team,value=sec.' + - name: region + description: The AWS Region. If not specified, the default region is used. + - name: roleArn + description: The Amazon Resource Name (ARN) of the role to assume. + - name: roleSessionName + description: The identifier for the assumed role session. + - name: roleSessionDuration + description: 'The duration, in seconds, of the role session. Range: 900 seconds (15 minutes) to the maximum session duration setting for the role.' + outputs: + - contextPath: AWS.GuardDuty.ThreatEntitySet.DetectorId + description: The unique ID of the detector. + type: string + - contextPath: AWS.GuardDuty.ThreatEntitySet.ThreatEntitySetId + description: The ID of the created threat entity set resource. + type: string + - name: aws-gd-update-threat-entity-set + description: Updates the threat entity set that is specified by the threat entity set ID. + execution: true + arguments: + - name: detectorId + description: The ID of the detector that owns the set. + required: true + - name: threatEntitySetId + description: The ID of the threat entity set to update. + required: true + - name: activate + description: Whether GuardDuty should use the updated set. + auto: PREDEFINED + predefined: + - 'True' + - 'False' + - name: location + description: 'The updated S3 URI of the entity-list file. For example, https://s3.us-east-1.amazonaws.com/example-bucket/threat-entities-v2.txt.' + - name: name + description: 'The updated user-friendly name. Length 1-64.' + - name: expectedBucketOwner + description: The AWS account ID that owns the S3 bucket in the location parameter. + - name: region + description: The AWS Region. If not specified, the default region is used. + - name: roleArn + description: The Amazon Resource Name (ARN) of the role to assume. + - name: roleSessionName + description: The identifier for the assumed role session. + - name: roleSessionDuration + description: 'The duration, in seconds, of the role session. Range: 900 seconds (15 minutes) to the maximum session duration setting for the role.' + - name: aws-gd-get-threat-entity-set + description: Retrieves the threat entity set that is specified by the threat entity set ID. + arguments: + - name: detectorId + description: The unique ID of the detector that owns the threat entity set. + required: true + - name: threatEntitySetId + description: The ID of the threat entity set to retrieve. + required: true + - name: region + description: The AWS Region. If not specified, the default region is used. + - name: roleArn + description: The Amazon Resource Name (ARN) of the role to assume. + - name: roleSessionName + description: The identifier for the assumed role session. + - name: roleSessionDuration + description: 'The duration, in seconds, of the role session. Range: 900 seconds (15 minutes) to the maximum session duration setting for the role.' + outputs: + - contextPath: AWS.GuardDuty.ThreatEntitySet.DetectorId + description: The unique ID of the detector. + type: string + - contextPath: AWS.GuardDuty.ThreatEntitySet.ThreatEntitySetId + description: The unique identifier for the threat entity set. + type: string + - contextPath: AWS.GuardDuty.ThreatEntitySet.Name + description: The user-friendly name of the threat entity set. + type: string + - contextPath: AWS.GuardDuty.ThreatEntitySet.Format + description: The format of the threat entity set file. + type: string + - contextPath: AWS.GuardDuty.ThreatEntitySet.Location + description: The S3 URI of the threat entity set file. + type: string + - contextPath: AWS.GuardDuty.ThreatEntitySet.Status + description: The status of the threat entity set. + type: string + - contextPath: AWS.GuardDuty.ThreatEntitySet.ExpectedBucketOwner + description: The AWS account ID that owns the S3 bucket in the location parameter. + type: string + - contextPath: AWS.GuardDuty.ThreatEntitySet.CreatedAt + description: The timestamp when the threat entity set was created. For example, 2020-01-01T00:00:00.000Z. + type: date + - contextPath: AWS.GuardDuty.ThreatEntitySet.UpdatedAt + description: The timestamp when the threat entity set was last updated. For example, 2020-01-01T00:00:00.000Z. + type: date + - contextPath: AWS.GuardDuty.ThreatEntitySet.Tags + description: The tags associated with the threat entity set. + type: Unknown + - name: aws-gd-delete-threat-entity-set + description: Deletes the threat entity set that is specified by the threat entity set ID. + execution: true + arguments: + - name: detectorId + description: The ID of the detector that owns the set. + required: true + - name: threatEntitySetId + description: The ID of the threat entity set to delete. + required: true + - name: region + description: The AWS Region. If not specified, the default region is used. + - name: roleArn + description: The Amazon Resource Name (ARN) of the role to assume. + - name: roleSessionName + description: The identifier for the assumed role session. + - name: roleSessionDuration + description: 'The duration, in seconds, of the role session. Range: 900 seconds (15 minutes) to the maximum session duration setting for the role.' + - name: aws-gd-list-threat-entity-sets + description: Lists the threat entity sets of the GuardDuty service specified by the detector ID. + arguments: + - name: detectorId + description: The ID of the detector whose threat entity sets you want to list. + required: true + - name: limit + description: The maximum number of results across all pages. + defaultValue: '50' + - name: page_size + description: The page size for the underlying paginator. Valid range is 1-50. + defaultValue: '50' + - name: page + description: The 1-based page index. When set, only that page is returned. + - name: region + description: The AWS Region. If not specified, the default region is used. + - name: roleArn + description: The Amazon Resource Name (ARN) of the role to assume. + - name: roleSessionName + description: The identifier for the assumed role session. + - name: roleSessionDuration + description: 'The duration, in seconds, of the role session. Range: 900 seconds (15 minutes) to the maximum session duration setting for the role.' + outputs: + - contextPath: AWS.GuardDuty.ThreatEntitySet.DetectorId + description: The unique ID of the detector. + type: string + - contextPath: AWS.GuardDuty.ThreatEntitySet.ThreatEntitySetId + description: The unique identifier for a threat entity set. + type: string - arguments: - description: The ID of the detector that specifies the GuardDuty service whose findings you want to list. name: detectorId @@ -633,14 +808,14 @@ script: name: roleSessionName - description: The duration, in seconds, of the role session. The value can range from 900 seconds (15 minutes) up to the maximum session duration setting for the role. name: roleSessionDuration - - defaultValue: '50' - description: Number of total results to query. + - description: The number of total results to query. name: limit + defaultValue: '50' - description: Specific page to query. name: page - - defaultValue: '50' - description: Number of total results in each page. + - description: Number of total results in each page. name: page_size + defaultValue: '50' description: Lists Amazon GuardDuty findings for the specified detector ID. name: aws-gd-list-findings outputs: @@ -662,8 +837,8 @@ script: name: roleSessionName - description: The duration, in seconds, of the role session. The value can range from 900 seconds (15 minutes) up to the maximum session duration setting for the role. name: roleSessionDuration - - name: returnRawResponse - description: Select 'true' to save all fields from the response to the context. Otherwise, complex fields will be stored in JSON format. Default value is False. + - description: Select 'true' to save all fields from the response to the context. Otherwise, complex fields will be stored in JSON format. Default value is False. + name: returnRawResponse auto: PREDEFINED predefined: - 'true' @@ -888,7 +1063,7 @@ script: - contextPath: AWS.GuardDuty.Members.UpdatedAt description: The time a member was last updated. type: string - dockerimage: demisto/boto3py3:1.0.0.10221838 + dockerimage: demisto/boto3py3:1.0.0.12887184 isfetch: true runonce: false script: '-' diff --git a/Packs/AWS-GuardDuty/Integrations/AWSGuardDuty/AWSGuardDuty_test.py b/Packs/AWS-GuardDuty/Integrations/AWSGuardDuty/AWSGuardDuty_test.py index d9b4ceaf4d51..a5185469d494 100644 --- a/Packs/AWS-GuardDuty/Integrations/AWSGuardDuty/AWSGuardDuty_test.py +++ b/Packs/AWS-GuardDuty/Integrations/AWSGuardDuty/AWSGuardDuty_test.py @@ -4,31 +4,38 @@ import demistomock as demisto # noqa: F401 import pytest from AWSGuardDuty import ( + DemistoException, archive_findings, connection_test, create_detector, create_ip_set, create_sample_findings, + create_threat_entity_set, create_threat_intel_set, delete_detector, delete_ip_set, + delete_threat_entity_set, delete_threat_intel_set, fetch_incidents, get_detector, get_findings, get_ip_set, get_members, + get_threat_entity_set, get_threat_intel_set, list_detectors, list_findings, list_ip_sets, list_members, + list_threat_entity_sets, list_threat_intel_sets, parse_incident_from_finding, + parse_tag_field_to_dict, unarchive_findings, update_detector, update_findings_feedback, update_ip_set, + update_threat_entity_set, update_threat_intel_set, ) from test_data.api_responses_for_test import ( @@ -114,7 +121,22 @@ def list_threat_intel_sets(self, **kwargs): def update_threat_intel_set(self, **kwargs): pass - def get_paginator(self, **kwargs): + def create_threat_entity_set(self, **kwargs): + pass + + def update_threat_entity_set(self, **kwargs): + pass + + def get_threat_entity_set(self, **kwargs): + pass + + def delete_threat_entity_set(self, **kwargs): + pass + + def list_threat_entity_sets(self, **kwargs): + pass + + def get_paginator(self, operation_name=None, **kwargs): pass @@ -580,6 +602,523 @@ def test_create_threat_intel_set(mocker): assert command_results.outputs == {"DetectorId": "some_id", "ThreatIntelSetId": "threat1"} +@pytest.mark.parametrize( + "tags_str, expected", + [ + pytest.param("key=Environment,value=prod", {"Environment": "prod"}, id="single pair"), + pytest.param( + "key=Environment,value=prod;key=Team,value=sec", + {"Environment": "prod", "Team": "sec"}, + id="multiple pairs", + ), + pytest.param("key=Owner,value=team-a,team-b", {"Owner": "team-a,team-b"}, id="value with commas"), + ], +) +def test_parse_tag_field_to_dict(tags_str, expected): + """ + Given: + A tags string in the "key=,value=" format (values may contain commas) + + When: + Running parse_tag_field_to_dict + + Then: + assert the string is parsed into the expected key-to-value mapping. + """ + assert parse_tag_field_to_dict(tags_str) == expected + + +@pytest.mark.parametrize( + "tags_str", + [ + pytest.param("Environment=prod", id="missing key/value fields"), + pytest.param("key=Environment", id="missing value"), + ], +) +def test_parse_tag_field_to_dict_invalid(tags_str): + """ + Given: + A malformed tags string + + When: + Running parse_tag_field_to_dict + + Then: + assert a ValueError is raised describing the expected format. + """ + with pytest.raises(ValueError, match="Could not parse field"): + parse_tag_field_to_dict(tags_str) + + +def test_parse_tag_field_to_dict_exceeds_max_tags(mocker): + """ + Given: + A tags string with more than 50 tag pairs + + When: + Running parse_tag_field_to_dict + + Then: + assert only the first 50 tags are parsed. + """ + tags_str = ";".join(f"key=k{i},value=v{i}" for i in range(60)) + + result = parse_tag_field_to_dict(tags_str) + + assert len(result) == 50 + assert "k0" in result + assert "k49" in result + assert "k50" not in result + + +def test_create_threat_entity_set(mocker): + """ + Given: + AWSClient session + create_threat_entity_set valid response + + When: + Running create_threat_entity_set command + + Then: + assert api calls are called exactly once and as expected. + """ + mocked_client = MockedBoto3Client() + create_threat_entity_set_mock = mocker.patch.object( + MockedBoto3Client, "create_threat_entity_set", side_effect=[{"ThreatEntitySetId": "entity1"}] + ) + + command_results = create_threat_entity_set( + mocked_client, + { + "detectorId": "some_id", + "activate": "True", + "format": "some_format", + "location": "some_location", + "name": "some_name", + }, + ) + + create_threat_entity_set_mock.assert_called_with( + DetectorId="some_id", Activate=True, Format="some_format", Location="some_location", Name="some_name" + ) + assert command_results.outputs == {"DetectorId": "some_id", "ThreatEntitySetId": "entity1"} + + +def test_create_threat_entity_set_with_optional_args(mocker): + """ + Given: + AWSClient session + create_threat_entity_set valid response and the optional expectedBucketOwner and tags arguments + + When: + Running create_threat_entity_set command + + Then: + assert the optional arguments are passed to the api call as expected and tags are parsed to a map. + """ + mocked_client = MockedBoto3Client() + create_threat_entity_set_mock = mocker.patch.object( + MockedBoto3Client, "create_threat_entity_set", side_effect=[{"ThreatEntitySetId": "entity1"}] + ) + + command_results = create_threat_entity_set( + mocked_client, + { + "detectorId": "some_id", + "activate": "True", + "format": "some_format", + "location": "some_location", + "name": "some_name", + "expectedBucketOwner": "123456789012", + "tags": "key=Environment,value=prod;key=Team,value=sec", + }, + ) + + create_threat_entity_set_mock.assert_called_with( + DetectorId="some_id", + Activate=True, + Format="some_format", + Location="some_location", + Name="some_name", + ExpectedBucketOwner="123456789012", + Tags={"Environment": "prod", "Team": "sec"}, + ) + assert command_results.outputs == {"DetectorId": "some_id", "ThreatEntitySetId": "entity1"} + + +def test_create_threat_entity_set_command_results(mocker): + """ + Given: + AWSClient session + create_threat_entity_set valid response + + When: + Running create_threat_entity_set command + + Then: + assert the CommandResults outputs_prefix, outputs_key_field and readable_output are as expected. + """ + mocked_client = MockedBoto3Client() + mocker.patch.object(MockedBoto3Client, "create_threat_entity_set", side_effect=[{"ThreatEntitySetId": "entity1"}]) + + command_results = create_threat_entity_set( + mocked_client, + { + "detectorId": "some_id", + "activate": "False", + "format": "TXT", + "location": "some_location", + "name": "some_name", + }, + ) + + assert command_results.outputs_prefix == "AWS.GuardDuty.ThreatEntitySet" + assert command_results.outputs_key_field == "ThreatEntitySetId" + assert "AWS GuardDuty ThreatEntity Set" in command_results.readable_output + assert "entity1" in command_results.readable_output + + +@pytest.mark.parametrize( + "response, raises", + [ + pytest.param({}, does_not_raise(), id="Success"), + pytest.param(RESPONSE_METADATA, does_not_raise(), id="Success with Metadata"), + pytest.param({"response": "bad"}, pytest.raises(Exception), id="Failure"), + ], +) +def test_update_threat_entity_set(mocker, response, raises): + """ + Given: + AWSClient session + update_threat_entity_set various responses + + When: + Running update_threat_entity_set command + + Then: + assert exceptions are being raised according to api response + assert api calls are called exactly once and as expected. + """ + mocked_client = MockedBoto3Client() + update_threat_entity_set_mock = mocker.patch.object(MockedBoto3Client, "update_threat_entity_set", side_effect=[response]) + + with raises: + update_threat_entity_set( + mocked_client, + { + "detectorId": "some_id", + "threatEntitySetId": "ThreatEntitySetId1", + "activate": "True", + "location": "here", + "name": "some_name", + }, + ) + + update_threat_entity_set_mock.assert_called_with( + DetectorId="some_id", ThreatEntitySetId="ThreatEntitySetId1", Activate=True, Location="here", Name="some_name" + ) + + +def test_update_threat_entity_set_readable_output(mocker): + """ + Given: + AWSClient session + update_threat_entity_set successful response + + When: + Running update_threat_entity_set command + + Then: + assert the returned confirmation message contains the ThreatEntitySet ID. + """ + mocked_client = MockedBoto3Client() + mocker.patch.object(MockedBoto3Client, "update_threat_entity_set", side_effect=[{}]) + + result = update_threat_entity_set(mocked_client, {"detectorId": "some_id", "threatEntitySetId": "ThreatEntitySetId1"}) + + assert result.readable_output == "Threat entity set ThreatEntitySetId1 was updated successfully" + + +def test_update_threat_entity_set_failure_message(mocker): + """ + Given: + AWSClient session + update_threat_entity_set response indicating a failure + + When: + Running update_threat_entity_set command + + Then: + assert an exception is raised with a descriptive failure message. + """ + mocked_client = MockedBoto3Client() + mocker.patch.object(MockedBoto3Client, "update_threat_entity_set", side_effect=[{"response": "bad"}]) + + with pytest.raises(Exception, match="Failed updating Threat Entity set ThreatEntitySetId1"): + update_threat_entity_set(mocked_client, {"detectorId": "some_id", "threatEntitySetId": "ThreatEntitySetId1"}) + + +def test_get_threat_entity_set(mocker): + """ + Given: + AWSClient session + get_threat_entity_set valid response + + When: + Running get_threat_entity_set command + + Then: + assert the API is called as expected and the outputs contain all the entity set fields. + """ + mocked_client = MockedBoto3Client() + get_mock = mocker.patch.object( + MockedBoto3Client, + "get_threat_entity_set", + side_effect=[ + { + "Name": "some_name", + "Format": "TXT", + "Location": "some_location", + "Status": "ACTIVE", + "ExpectedBucketOwner": "123456789012", + "CreatedAt": "2024-01-01T00:00:00Z", + "UpdatedAt": "2024-01-02T00:00:00Z", + "Tags": {"Environment": "prod"}, + } + ], + ) + + command_results = get_threat_entity_set(mocked_client, {"detectorId": "some_id", "threatEntitySetId": "ThreatEntitySetId1"}) + + get_mock.assert_called_with(DetectorId="some_id", ThreatEntitySetId="ThreatEntitySetId1") + assert command_results.outputs == { + "DetectorId": "some_id", + "ThreatEntitySetId": "ThreatEntitySetId1", + "Name": "some_name", + "Format": "TXT", + "Location": "some_location", + "Status": "ACTIVE", + "ExpectedBucketOwner": "123456789012", + "CreatedAt": "2024-01-01T00:00:00Z", + "UpdatedAt": "2024-01-02T00:00:00Z", + "Tags": {"Environment": "prod"}, + } + + +def test_get_threat_entity_set_command_results(mocker): + """ + Given: + AWSClient session + get_threat_entity_set valid response + + When: + Running get_threat_entity_set command + + Then: + assert the CommandResults outputs_prefix, outputs_key_field and readable_output are as expected. + """ + mocked_client = MockedBoto3Client() + mocker.patch.object(MockedBoto3Client, "get_threat_entity_set", side_effect=[{"Name": "some_name", "Status": "ACTIVE"}]) + + command_results = get_threat_entity_set(mocked_client, {"detectorId": "some_id", "threatEntitySetId": "ThreatEntitySetId1"}) + + assert command_results.outputs_prefix == "AWS.GuardDuty.ThreatEntitySet" + assert command_results.outputs_key_field == "ThreatEntitySetId" + assert "AWS GuardDuty ThreatEntity Set" in command_results.readable_output + assert "some_name" in command_results.readable_output + + +def test_get_threat_entity_set_empty_fields(mocker): + """ + Given: + AWSClient session + get_threat_entity_set response with empty optional fields + + When: + Running get_threat_entity_set command + + Then: + assert empty fields are removed from the outputs and the identifying fields remain. + """ + mocked_client = MockedBoto3Client() + mocker.patch.object( + MockedBoto3Client, "get_threat_entity_set", side_effect=[{"Name": "some_name", "Tags": {}, "Location": None}] + ) + + command_results = get_threat_entity_set(mocked_client, {"detectorId": "some_id", "threatEntitySetId": "ThreatEntitySetId1"}) + + assert command_results.outputs == { + "DetectorId": "some_id", + "ThreatEntitySetId": "ThreatEntitySetId1", + "Name": "some_name", + } + + +@pytest.mark.parametrize( + "response, raises", + [ + pytest.param({}, does_not_raise(), id="Success"), + pytest.param(RESPONSE_METADATA, does_not_raise(), id="Success with Metadata"), + pytest.param({"response": "bad"}, pytest.raises(Exception), id="Failure"), + ], +) +def test_delete_threat_entity_set(mocker, response, raises): + """ + Given: + AWSClient session + delete_threat_entity_set various responses + + When: + Running delete_threat_entity_set command + + Then: + assert exceptions are being raised according to api response + assert api calls are called exactly once and as expected. + """ + mocked_client = MockedBoto3Client() + delete_threat_entity_set_mock = mocker.patch.object(MockedBoto3Client, "delete_threat_entity_set", side_effect=[response]) + + with raises: + delete_threat_entity_set(mocked_client, {"detectorId": "some_id", "threatEntitySetId": "ThreatEntitySetId1"}) + + delete_threat_entity_set_mock.assert_called_with(DetectorId="some_id", ThreatEntitySetId="ThreatEntitySetId1") + + +def test_delete_threat_entity_set_readable_output(mocker): + """ + Given: + AWSClient session + delete_threat_entity_set successful response + + When: + Running delete_threat_entity_set command + + Then: + assert the returned confirmation message contains the ThreatEntitySet ID and Detector ID. + """ + mocked_client = MockedBoto3Client() + mocker.patch.object(MockedBoto3Client, "delete_threat_entity_set", side_effect=[{}]) + + result = delete_threat_entity_set(mocked_client, {"detectorId": "some_id", "threatEntitySetId": "ThreatEntitySetId1"}) + + assert result.readable_output == "Threat Entity Set ThreatEntitySetId1 was deleted from Detector some_id successfully" + + +def test_delete_threat_entity_set_failure_message(mocker): + """ + Given: + AWSClient session + delete_threat_entity_set response indicating a failure + + When: + Running delete_threat_entity_set command + + Then: + assert an exception is raised with a descriptive failure message. + """ + mocked_client = MockedBoto3Client() + mocker.patch.object(MockedBoto3Client, "delete_threat_entity_set", side_effect=[{"response": "bad"}]) + + with pytest.raises(Exception, match="Failed to delete Threat Entity set ThreatEntitySetId1"): + delete_threat_entity_set(mocked_client, {"detectorId": "some_id", "threatEntitySetId": "ThreatEntitySetId1"}) + + +def test_list_threat_entity_sets(mocker): + """ + Given: + AWSClient session + list_threat_entity_sets valid paginated response + + When: + Running list_threat_entity_sets command + + Then: + assert the paginator is called as expected and the outputs contain the detector and entity set IDs. + """ + mocked_client = MockedBoto3Client() + get_paginator_mock = mocker.patch.object(MockedBoto3Client, "get_paginator", side_effect=[MockedPaginator()]) + paginate_mock = mocker.patch.object( + MockedPaginator, "paginate", side_effect=[[{"ThreatEntitySetIds": ["entity1", "entity2"]}]] + ) + + command_results = list_threat_entity_sets(mocked_client, {"detectorId": "some_id"}) + + get_paginator_mock.assert_called_with("list_threat_entity_sets") + paginate_mock.assert_called_with(DetectorId="some_id", PaginationConfig={"MaxItems": 50, "PageSize": 50}) + assert command_results.outputs == [ + {"DetectorId": "some_id", "ThreatEntitySetId": "entity1"}, + {"DetectorId": "some_id", "ThreatEntitySetId": "entity2"}, + ] + assert command_results.outputs_prefix == "AWS.GuardDuty.ThreatEntitySet" + + +def test_list_threat_entity_sets_specific_page(mocker): + """ + Given: + AWSClient session + list_threat_entity_sets paginated response spanning multiple pages and the page argument + + When: + Running list_threat_entity_sets command + + Then: + assert only the requested page's entity sets are returned. + """ + mocked_client = MockedBoto3Client() + mocker.patch.object(MockedBoto3Client, "get_paginator", side_effect=[MockedPaginator()]) + mocker.patch.object( + MockedPaginator, + "paginate", + side_effect=[[{"ThreatEntitySetIds": ["entity1"]}, {"ThreatEntitySetIds": ["entity2"]}]], + ) + + command_results = list_threat_entity_sets(mocked_client, {"detectorId": "some_id", "page": "2", "page_size": "1"}) + + assert command_results.outputs == [ + {"DetectorId": "some_id", "ThreatEntitySetId": "entity2"}, + ] + + +def test_list_threat_entity_sets_empty(mocker): + """ + Given: + AWSClient session + list_threat_entity_sets paginated response with no ThreatEntitySet IDs + + When: + Running list_threat_entity_sets command + + Then: + assert an empty list is returned when there are no threat entity sets. + """ + mocked_client = MockedBoto3Client() + mocker.patch.object(MockedBoto3Client, "get_paginator", side_effect=[MockedPaginator()]) + mocker.patch.object(MockedPaginator, "paginate", side_effect=[[{"ThreatEntitySetIds": []}]]) + + command_results = list_threat_entity_sets(mocked_client, {"detectorId": "some_id"}) + + assert command_results.outputs == [] + + +def test_list_threat_entity_sets_invalid_page_size(mocker): + """ + Given: + AWSClient session + list_threat_entity_sets called with a page_size above the allowed maximum + + When: + Running list_threat_entity_sets command + + Then: + assert a DemistoException is raised for the out-of-range page_size. + """ + mocked_client = MockedBoto3Client() + + with pytest.raises(DemistoException, match="page_size argument must be between 1 to 50"): + list_threat_entity_sets(mocked_client, {"detectorId": "some_id", "page_size": "51"}) + + EXPECTED_IP_SET_RESULT = { "DetectorId": "some_id", "Format": "TXT", diff --git a/Packs/AWS-GuardDuty/Integrations/AWSGuardDuty/README.md b/Packs/AWS-GuardDuty/Integrations/AWSGuardDuty/README.md index e92902c9dba1..0f7576b4e0c2 100644 --- a/Packs/AWS-GuardDuty/Integrations/AWSGuardDuty/README.md +++ b/Packs/AWS-GuardDuty/Integrations/AWSGuardDuty/README.md @@ -630,9 +630,9 @@ Action: _guardduty:ListFindings_ | roleArn | The Amazon Resource Name (ARN) of the role to assume. | Optional | | roleSessionName | An identifier for the assumed role session. | Optional | | roleSessionDuration | The duration, in seconds, of the role session. The value can range from 900 seconds (15 minutes) up to the maximum session duration setting for the role. | Optional | -| limit | Number of total results to query. Default is `50`. | Optional | +| limit | The number of total results to query. Default is 50. | Optional | | page | Specific page to query. | Optional | -| page_size | Number of total results in each page. Default is `50`. | Optional | +| page_size | Number of total results in each page. Default is 50. | Optional | #### Context Output @@ -917,6 +917,174 @@ Action: _guardduty:GetMembers_ ```!aws-gd-get-members detectorIds=4f1fc7cd7dsg26sdf4328d8dc813 accountIds=1f3fc2cd1dag26sdf4338d8aa813``` +### aws-gd-create-threat-entity-set + +*** +Creates a new threat entity set. In a threat entity set, you can provide known malicious threat entities for your AWS environment. + +#### Base Command + +`aws-gd-create-threat-entity-set` + +#### Input + +| **Argument Name** | **Description** | **Required** | +| --- | --- | --- | +| detectorId | The unique ID of the detector that will own the threat entity set. | Required | +| name | The user-friendly name of the threat entity set. Length 1-64. | Required | +| format | The format of the entity-list file. Possible values are: TXT, STIX, OTX_CSV, ALIEN_VAULT, PROOF_POINT, FIRE_EYE. | Required | +| location | The S3 URI of the entity-list file. For example, https://s3.us-east-1.amazonaws.com/example-bucket/threat-entities.txt. | Required | +| activate | Whether GuardDuty should start using the uploaded set immediately. Possible values are: True, False. | Required | +| expectedBucketOwner | The AWS account ID that owns the S3 bucket in the location parameter. Prevents cross-account spoofing. | Optional | +| tags | The tags to add to the new threat entity set. Format: key=<KEY>,value=<VALUE>. Separate multiple tags with a semicolon (;). For example, key=Environment,value=prod;key=Team,value=sec. | Optional | +| region | The AWS Region. If not specified, the default region is used. | Optional | +| roleArn | The Amazon Resource Name (ARN) of the role to assume. | Optional | +| roleSessionName | The identifier for the assumed role session. | Optional | +| roleSessionDuration | The duration, in seconds, of the role session. Range: 900 seconds (15 minutes) to the maximum session duration setting for the role. | Optional | + +#### Context Output + +| **Path** | **Type** | **Description** | +| --- | --- | --- | +| AWS.GuardDuty.ThreatEntitySet.DetectorId | string | The unique ID of the detector. | +| AWS.GuardDuty.ThreatEntitySet.ThreatEntitySetId | string | The ID of the created threat entity set resource. | + +#### Command Example + +```!aws-gd-create-threat-entity-set detectorId=12abc34d567e8fa901bc2d34e56789f0 name=malicious-entities format=TXT location=https://s3.us-east-1.amazonaws.com/example-bucket/threat-entities.txt activate=True tags=key=Environment,value=prod;key=Team,value=sec``` + +### aws-gd-update-threat-entity-set + +*** +Updates the threat entity set that is specified by the threat entity set ID. + +#### Base Command + +`aws-gd-update-threat-entity-set` + +#### Input + +| **Argument Name** | **Description** | **Required** | +| --- | --- | --- | +| detectorId | The ID of the detector that owns the set. | Required | +| threatEntitySetId | The ID of the threat entity set to update. | Required | +| activate | Whether GuardDuty should use the updated set. Possible values are: True, False. | Optional | +| location | The updated S3 URI of the entity-list file. For example, https://s3.us-east-1.amazonaws.com/example-bucket/threat-entities-v2.txt. | Optional | +| name | The updated user-friendly name. Length 1-64. | Optional | +| expectedBucketOwner | The AWS account ID that owns the S3 bucket in the location parameter. | Optional | +| region | The AWS Region. If not specified, the default region is used. | Optional | +| roleArn | The Amazon Resource Name (ARN) of the role to assume. | Optional | +| roleSessionName | The identifier for the assumed role session. | Optional | +| roleSessionDuration | The duration, in seconds, of the role session. Range: 900 seconds (15 minutes) to the maximum session duration setting for the role. | Optional | + +#### Context Output + +There is no context output for this command. + +#### Command Example + +```!aws-gd-update-threat-entity-set detectorId=12abc34d567e8fa901bc2d34e56789f0 threatEntitySetId=ab12cd34ef56gh78ij90 activate=True name=malicious-entities-v2``` + +### aws-gd-get-threat-entity-set + +*** +Retrieves the threat entity set that is specified by the threat entity set ID. + +#### Base Command + +`aws-gd-get-threat-entity-set` + +#### Input + +| **Argument Name** | **Description** | **Required** | +| --- | --- | --- | +| detectorId | The unique ID of the detector that owns the threat entity set. | Required | +| threatEntitySetId | The ID of the threat entity set to retrieve. | Required | +| region | The AWS Region. If not specified, the default region is used. | Optional | +| roleArn | The Amazon Resource Name (ARN) of the role to assume. | Optional | +| roleSessionName | The identifier for the assumed role session. | Optional | +| roleSessionDuration | The duration, in seconds, of the role session. Range: 900 seconds (15 minutes) to the maximum session duration setting for the role. | Optional | + +#### Context Output + +| **Path** | **Type** | **Description** | +| --- | --- | --- | +| AWS.GuardDuty.ThreatEntitySet.DetectorId | string | The unique ID of the detector. | +| AWS.GuardDuty.ThreatEntitySet.ThreatEntitySetId | string | The unique identifier for the threat entity set. | +| AWS.GuardDuty.ThreatEntitySet.Name | string | The user-friendly name of the threat entity set. | +| AWS.GuardDuty.ThreatEntitySet.Format | string | The format of the threat entity set file. | +| AWS.GuardDuty.ThreatEntitySet.Location | string | The S3 URI of the threat entity set file. | +| AWS.GuardDuty.ThreatEntitySet.Status | string | The status of the threat entity set. | +| AWS.GuardDuty.ThreatEntitySet.ExpectedBucketOwner | string | The AWS account ID that owns the S3 bucket in the location parameter. | +| AWS.GuardDuty.ThreatEntitySet.CreatedAt | date | The timestamp when the threat entity set was created. For example, 2020-01-01T00:00:00.000Z. | +| AWS.GuardDuty.ThreatEntitySet.UpdatedAt | date | The timestamp when the threat entity set was last updated. For example, 2020-01-01T00:00:00.000Z. | +| AWS.GuardDuty.ThreatEntitySet.Tags | Unknown | The tags associated with the threat entity set. | + +#### Command Example + +```!aws-gd-get-threat-entity-set detectorId=12abc34d567e8fa901bc2d34e56789f0 threatEntitySetId=ab12cd34ef56gh78ij90``` + +### aws-gd-delete-threat-entity-set + +*** +Deletes the threat entity set that is specified by the threat entity set ID. + +#### Base Command + +`aws-gd-delete-threat-entity-set` + +#### Input + +| **Argument Name** | **Description** | **Required** | +| --- | --- | --- | +| detectorId | The ID of the detector that owns the set. | Required | +| threatEntitySetId | The ID of the threat entity set to delete. | Required | +| region | The AWS Region. If not specified, the default region is used. | Optional | +| roleArn | The Amazon Resource Name (ARN) of the role to assume. | Optional | +| roleSessionName | The identifier for the assumed role session. | Optional | +| roleSessionDuration | The duration, in seconds, of the role session. Range: 900 seconds (15 minutes) to the maximum session duration setting for the role. | Optional | + +#### Context Output + +There is no context output for this command. + +#### Command Example + +```!aws-gd-delete-threat-entity-set detectorId=12abc34d567e8fa901bc2d34e56789f0 threatEntitySetId=ab12cd34ef56gh78ij90``` + +### aws-gd-list-threat-entity-sets + +*** +Lists the threat entity sets of the GuardDuty service specified by the detector ID. + +#### Base Command + +`aws-gd-list-threat-entity-sets` + +#### Input + +| **Argument Name** | **Description** | **Required** | +| --- | --- | --- | +| detectorId | The ID of the detector whose threat entity sets you want to list. | Required | +| limit | The maximum number of results across all pages. Default is 50. | Optional | +| page_size | The page size for the underlying paginator. Valid range is 1-50. Default is 50. | Optional | +| page | The 1-based page index. When set, only that page is returned. | Optional | +| region | The AWS Region. If not specified, the default region is used. | Optional | +| roleArn | The Amazon Resource Name (ARN) of the role to assume. | Optional | +| roleSessionName | The identifier for the assumed role session. | Optional | +| roleSessionDuration | The duration, in seconds, of the role session. Range: 900 seconds (15 minutes) to the maximum session duration setting for the role. | Optional | + +#### Context Output + +| **Path** | **Type** | **Description** | +| --- | --- | --- | +| AWS.GuardDuty.ThreatEntitySet.DetectorId | string | The unique ID of the detector. | +| AWS.GuardDuty.ThreatEntitySet.ThreatEntitySetId | string | The unique identifier for a threat entity set. | + +#### Command Example + +```!aws-gd-list-threat-entity-sets detectorId=12abc34d567e8fa901bc2d34e56789f0``` + ## Breaking changes from the previous version of this integration - AWS-GuardDuty The following sections list the changes in this version. diff --git a/Packs/AWS-GuardDuty/Integrations/AWSGuardDutyEventCollector/AWSGuardDutyEventCollector.py b/Packs/AWS-GuardDuty/Integrations/AWSGuardDutyEventCollector/AWSGuardDutyEventCollector.py index 1e9fe30a0c59..b4001ea7edc7 100644 --- a/Packs/AWS-GuardDuty/Integrations/AWSGuardDutyEventCollector/AWSGuardDutyEventCollector.py +++ b/Packs/AWS-GuardDuty/Integrations/AWSGuardDutyEventCollector/AWSGuardDutyEventCollector.py @@ -1,5 +1,5 @@ import json -from datetime import date, datetime +from datetime import date, datetime, timedelta, UTC from typing import TYPE_CHECKING import demistomock as demisto # noqa: F401 @@ -17,6 +17,12 @@ MAX_RESULTS = 50 GD_SEVERITY_DICT = {"Low": 1, "Medium": 4, "High": 7} +# A fully-deduped cursor boundary second is only escaped once it is +# at least this far in the past relative to the fetch start time. This keeps a +# still-current second pinned (so a genuine later same-second update is still +# catchable) while breaking a boundary that has clearly gone stale. +STALE_BOUNDARY_MARGIN = timedelta(minutes=2) + PRODUCT = "guardduty" VENDOR = "aws" @@ -52,7 +58,7 @@ def _normalize_last_ids_entry(value) -> set[str]: """Coerce a stored ``last_ids`` value into a set of ids. The integration historically stored ``last_ids[detector_id]`` as a single - string (the last finding id seen). To fix XSUP-67097 we now track every + string (the last finding id seen). We now track every finding id sharing the cursor's ``UpdatedAt`` second, which means the value is conceptually a set. ``demisto.setLastRun`` serializes as JSON, so the on-disk representation must be a ``list``. This helper normalizes @@ -83,7 +89,7 @@ def _build_finding_criterion(updated_at: Optional[datetime], severity: str, excl updated_at: Inclusive lower bound on ``updatedAt``. severity: Minimum severity label (Low/Medium/High). exclude_archived: When ``True``, adds ``service.archived = false`` so suppressed/archived - findings (XSUP-67097 / XSUP-71079 complaint #2) are not re-fetched. + findings are not re-fetched. Returns: The criterion dict. @@ -103,6 +109,49 @@ def _event_updated_at(event: dict) -> Any: return event.get("UpdatedAt", event.get("CreatedAt")) +def _cursor_second(value: Any) -> datetime: + """Return the whole-second (microseconds truncated), naive-UTC datetime for a timestamp. + + GuardDuty findings carry MILLISECOND-precision ``UpdatedAt`` + values, and several findings routinely share the same whole second while + differing in their milliseconds. The fetch cursor and the same-second + sibling dedup are defined at SECOND resolution, + so all comparisons and grouping must truncate sub-second precision. Doing so + keeps the cursor deterministic (independent of AWS ``get_findings`` ordering) + and lets every finding sharing the cursor second be recognized as an + already-seen sibling on the inclusive ``Gte`` re-query. + + Timezone normalization: values reach this function from two sources with + different awareness. ``first_fetch`` (via ``arg_to_datetime``) is tz-AWARE for + absolute dates, while finding timestamps go through ``parse_date_string`` + (``strptime`` with no ``%z``) and are always NAIVE. Comparing the two (dedup + equality, and the stale-boundary ``<`` check against ``datetime.utcnow()``, + which is naive) would either never match (silent re-ingestion) or raise + ``TypeError: can't compare offset-naive and offset-aware datetimes``. To make + every cursor value consistently comparable, a tz-aware input is first + CONVERTED to UTC and THEN stripped of ``tzinfo`` (order matters — dropping + tzinfo before converting would shift, e.g., a ``+05:00`` cursor by 5 hours). + + Args: + value: A timestamp string or a ``datetime`` (naive or tz-aware). Every + GuardDuty finding carries a real ``UpdatedAt``/``CreatedAt``, so a + value is always present. + + Returns: + The corresponding naive-UTC ``datetime`` truncated to whole seconds. + + Raises: + ValueError: If ``value`` is not a parseable timestamp. This is not + expected from real GuardDuty data and surfaces loudly rather than + silently corrupting the fetch cursor. + """ + dt = value if isinstance(value, datetime) else parse_date_string(value) + if dt.tzinfo is not None: + # Convert to UTC first, then drop tzinfo so the result is naive-UTC. + dt = dt.astimezone(UTC).replace(tzinfo=None) + return dt.replace(microsecond=0) + + def get_events( aws_client: "GuardDutyClient", collect_from: dict, @@ -137,7 +186,7 @@ def get_events( Each value is a ``list[str]`` (JSON-serializable for setLastRun). new_collect_from (dict): The new collect_from dict, expected to receive as collect_from input in the next run. - Note (XSUP-71079): The fetch cursor is second-resolution and the ``updatedAt`` filter is inclusive + Note: The fetch cursor is second-resolution and the ``updatedAt`` filter is inclusive (``Gte``). To avoid silently skipping findings, the cursor is NEVER advanced into a second that was only partially consumed because ``limit`` was reached. When a fetch is truncated mid-second the cursor is rolled back to the last fully-drained second (and its sibling ids are persisted) so the next run @@ -153,16 +202,31 @@ def get_events( demisto.debug(f"AWSGuardDutyEventCollector Starting get_events. {collect_from=}, {collect_from_default=}, {last_ids=}") # List all detectors + list_detectors_page = 0 while next_token: + list_detectors_page += 1 list_detectors_args: dict = {"MaxResults": detectors_num} if next_token != "starting_token": list_detectors_args.update({"NextToken": next_token}) response = aws_client.list_detectors(**list_detectors_args) - detector_ids += response.get("DetectorIds", []) - next_token = response.get("NextToken", "") + page_detector_ids = response.get("DetectorIds", []) + raw_next_token = response.get("NextToken", "") + detector_ids += page_detector_ids + # Diagnostic: log per-page enumeration so a truncated/short-circuited pagination is visible. + demisto.debug( + f"AWSGuardDutyEventCollector - list_detectors page {list_detectors_page}: " + f"MaxResults={detectors_num}, returned={len(page_detector_ids)}, " + f"running_total={len(detector_ids)}, has_next_token={bool(raw_next_token)}, " + f"next_token_len={len(raw_next_token) if raw_next_token else 0}, " + f"page_detector_ids={page_detector_ids}" + ) + next_token = raw_next_token - demisto.debug(f"AWSGuardDutyEventCollector - Found detector ids: {detector_ids}") + demisto.debug( + f"AWSGuardDutyEventCollector - Finished list_detectors after {list_detectors_page} page(s). " + f"Total detectors found: {len(detector_ids)}. Found detector ids: {detector_ids}" + ) for detector_id in detector_ids: demisto.debug( @@ -173,16 +237,18 @@ def get_events( finding_ids: list = [] detector_events: list = [] updated_at = parse_date_string(collect_from.get(detector_id)) if collect_from.get(detector_id) else collect_from_default - # XSUP-67097: dedup against ALL ids seen at the cursor second, not just one. + # Dedup against all ids seen at the cursor second. seen_ids = _normalize_last_ids_entry(last_ids.get(detector_id)) - # List all finding ids - while next_token and len(events) + len(finding_ids) < limit: + # List all finding ids. The ``limit`` budget is applied PER DETECTOR so that a busy detector + # earlier in the loop can never starve detectors processed later (each detector gets its own + # full ``limit`` per cycle instead of sharing one global budget across all detectors). + while next_token and len(set(finding_ids)) < limit: demisto.debug(f"AWSGuardDutyEventCollector - Getting more finding ids with {next_token=}, {updated_at=}") list_finding_args = { "DetectorId": detector_id, "FindingCriteria": {"Criterion": _build_finding_criterion(updated_at, severity, exclude_archived)}, "SortCriteria": {"AttributeName": "updatedAt", "OrderBy": "ASC"}, - "MaxResults": min(limit - (len(events) + len(set(finding_ids))), MAX_RESULTS), + "MaxResults": min(limit - len(set(finding_ids)), MAX_RESULTS), } if next_token != "starting_token": list_finding_args.update({"NextToken": next_token}) @@ -202,87 +268,118 @@ def get_events( findings_response = aws_client.get_findings(DetectorId=detector_id, FindingIds=chunk_of_finding_ids) detector_events += findings_response.get("Findings", []) - # Dedup already-seen findings — but only at the cursor second. - # - # XSUP-67097: drop every finding we already ingested that still shares the cursor's UpdatedAt - # second (same-second siblings re-returned by the inclusive Gte query). - # - # XSUP-72455: do NOT drop a finding whose UpdatedAt has advanced past the cursor second. GuardDuty - # findings are long-lived and update in place; when a recurring finding gets a new occurrence its - # UpdatedAt moves forward and AWS returns it again. That is a legitimate new event and must be - # ingested. The previous ID-only dedup dropped it because its id was in last_ids, which produced an - # empty result and, because the cursor only advances when events are ingested, pinned the fetch - # behind that finding indefinitely. + # Drop already-seen findings only when their UpdatedAt still shares the cursor second. + # Findings whose UpdatedAt advanced past the cursor second are new events and are kept. + # Compared at whole-second resolution to catch millisecond siblings of the same second. + cursor_second = _cursor_second(updated_at) if seen_ids: before_ids = [ev.get("Id") for ev in detector_events] detector_events = [ ev for ev in detector_events - if ev.get("Id") not in seen_ids or parse_date_string(_event_updated_at(ev)) != updated_at + if ev.get("Id") not in seen_ids or _cursor_second(_event_updated_at(ev)) != cursor_second ] after_ids = [ev.get("Id") for ev in detector_events] if before_ids != after_ids: demisto.debug( f"AWSGuardDutyEventCollector - Dedup removed already-seen same-second findings " f"for {detector_id=}. Before: {before_ids}, after: {after_ids}, removed via {seen_ids=} " - f"at cursor second {updated_at=}." + f"at cursor second {cursor_second=}." + ) + else: + demisto.debug( + f"AWSGuardDutyEventCollector - Dedup ran for {detector_id=} at cursor second " + f"{cursor_second=} but removed nothing ({len(before_ids)} findings kept, " + f"{len(seen_ids)} seen ids carried in)." ) demisto.debug(f"AWSGuardDutyEventCollector - {detector_id=} findings found ({len(detector_events)}): {detector_events}") events += detector_events demisto.debug(f"AWSGuardDutyEventCollector - Number of events is {len(events)}") - # XSUP-71079: advance the cursor safely. - # - # The cursor is second-resolution and the updatedAt query is inclusive (Gte). Two failure modes - # are guarded here: - # 1. Same-second siblings (XSUP-67097): persist EVERY finding id whose UpdatedAt equals the - # cursor second so the next run can dedup them all (not just one). - # 2. Mid-second truncation (XSUP-71079): if this fetch stopped because it hit `limit` while - # there were still un-fetched findings (next_token is truthy) AND the last second is only - # partially consumed, advancing the cursor to that last second would skip the remaining - # siblings of that second (they fall on the same inclusive boundary but AWS may order them - # after the truncation point). To guarantee no loss we roll the cursor back to the last - # FULLY-drained second and persist its sibling ids, so the next run re-queries the - # truncated second from its start and makes forward progress. - truncated_by_limit = bool(next_token) # loop exited with a pending token => stopped due to limit + # Advance the cursor. The cursor is second-resolution and the updatedAt query is inclusive (Gte), + # so persist every finding id at the cursor second for next-run dedup, and roll the cursor back to + # the last fully-drained second when a limit-truncated page leaves a partial second. + truncated_by_limit = bool(next_token) # pending token => stopped due to limit if detector_events: - last_cursor_ts = _event_updated_at(detector_events[-1]) - cursor_ts = last_cursor_ts + # Pick the cursor at whole-second resolution: latest ingested second, max UpdatedAt within it. + event_seconds = [(ev, _cursor_second(_event_updated_at(ev))) for ev in detector_events] + latest_second = max(sec for _, sec in event_seconds) + cursor_second_target = latest_second if truncated_by_limit: - # Find the latest second strictly older than the last (partial) second. - distinct_seconds = {_event_updated_at(ev) for ev in detector_events} - fully_drained = sorted((s for s in distinct_seconds if s != last_cursor_ts), key=parse_date_string) + # Find the latest whole-second strictly older than the last (partial) second. + distinct_seconds = {sec for _, sec in event_seconds} + fully_drained = sorted(s for s in distinct_seconds if s != latest_second) if fully_drained: - cursor_ts = fully_drained[-1] + cursor_second_target = fully_drained[-1] demisto.debug( f"AWSGuardDutyEventCollector - Fetch truncated by limit for {detector_id=}. " - f"Rolling cursor back from partial second {last_cursor_ts} to last fully-drained " - f"second {cursor_ts} to avoid skipping same-second siblings." + f"Rolling cursor back from partial second {latest_second} to last fully-drained " + f"second {cursor_second_target} to avoid skipping same-second siblings." ) else: # The entire page is a single second that we could not fully drain. Keep the cursor # on that second and accumulate seen ids so progress happens via dedup next run. demisto.debug( f"AWSGuardDutyEventCollector - Fetch truncated by limit for {detector_id=} within a " - f"single second {last_cursor_ts}; keeping cursor and accumulating seen ids." + f"single second {latest_second}; keeping cursor and accumulating seen ids." ) + # Same-second siblings of the target second. + cursor_second_events = [ev for ev, sec in event_seconds if sec == cursor_second_target] + cursor_ts = max(_event_updated_at(ev) for ev in cursor_second_events) new_collect_from[detector_id] = cursor_ts - cursor_sibling_ids = {ev.get("Id") for ev in detector_events if _event_updated_at(ev) == cursor_ts} + cursor_sibling_ids = {ev.get("Id") for ev in cursor_second_events} cursor_sibling_ids.discard(None) - # Carry forward previously-seen ids when the cursor second did not advance past them, - # so we never forget same-second siblings across runs. - if seen_ids and parse_date_string(cursor_ts) == updated_at: + # Carry forward prior seen ids when the cursor second did not advance past them. + carried_forward = seen_ids and cursor_second_target == cursor_second + if carried_forward: cursor_sibling_ids |= seen_ids + demisto.debug( + f"AWSGuardDutyEventCollector - Cursor advance for {detector_id=}: " + f"boundary_second={cursor_second_target}, cursor_ts={cursor_ts!r}, " + f"same_second_sibling_count={len(cursor_sibling_ids)}, {carried_forward=}." + ) # Stored as list so demisto.setLastRun can JSON-serialize it; round-trips via # _normalize_last_ids_entry on the next call. new_last_ids[detector_id] = sorted(cursor_sibling_ids) elif finding_ids: - # No detector_events but we did see ids — keep the prior seen_ids as-is so - # we don't forget about them on the next fetch. - new_last_ids[detector_id] = sorted(seen_ids) if seen_ids else [] + # Everything listed deduped away as already-seen same-second siblings. If the fetch was not + # limit-truncated and the boundary second is safely in the past, step the cursor one second + # past it and clear seen ids to avoid a permanent stall; a recent boundary is left pinned. + now = datetime.utcnow() + boundary_is_stale = cursor_second < (now - STALE_BOUNDARY_MARGIN) + if not truncated_by_limit and boundary_is_stale: + next_second = cursor_second + timedelta(seconds=1) + new_collect_from[detector_id] = next_second.isoformat() + new_last_ids[detector_id] = [] + demisto.debug( + f"AWSGuardDutyEventCollector - Fully-deduped STALE boundary for {detector_id=}: " + f"{len(finding_ids)} finding id(s) listed but all deduped, nothing newer, and the " + f"boundary second {cursor_second} is older than {STALE_BOUNDARY_MARGIN} before " + f"now ({now}). Advancing cursor to {next_second.isoformat()} and clearing seen ids " + f"to break the stall." + ) + else: + # Bounded pause: keep prior seen ids for a still-current or limit-truncated boundary. + new_last_ids[detector_id] = sorted(seen_ids) if seen_ids else [] + demisto.debug( + f"AWSGuardDutyEventCollector - No new events for {detector_id=}: {len(finding_ids)} finding " + f"id(s) were listed but all were deduped as already-seen. Cursor left unchanged " + f"({truncated_by_limit=}, {boundary_is_stale=}); {len(seen_ids)} seen id(s) carried forward." + ) + else: + # Nothing matched the query this cycle; cursor unchanged. + demisto.debug( + f"AWSGuardDutyEventCollector - No findings returned for {detector_id=} this cycle. " f"Cursor left unchanged." + ) demisto.debug(f"AWSGuardDutyEventCollector - Total number of events is {len(events)}") + # Log the outgoing cursor value/type and per-detector last_ids counts before persisting. + demisto.debug( + "AWSGuardDutyEventCollector - Persisting state. " + f"new_collect_from={ {k: f'{v!r} ({type(v).__name__})' for k, v in new_collect_from.items()} }, " + f"new_last_ids_counts={ {k: len(v) for k, v in new_last_ids.items()} }" + ) events = convert_events_with_datetime_to_str(events) return events, new_last_ids, new_collect_from diff --git a/Packs/AWS-GuardDuty/Integrations/AWSGuardDutyEventCollector/AWSGuardDutyEventCollector.yml b/Packs/AWS-GuardDuty/Integrations/AWSGuardDutyEventCollector/AWSGuardDutyEventCollector.yml index 7e46f0a062b5..ad1277ebdba2 100644 --- a/Packs/AWS-GuardDuty/Integrations/AWSGuardDutyEventCollector/AWSGuardDutyEventCollector.yml +++ b/Packs/AWS-GuardDuty/Integrations/AWSGuardDutyEventCollector/AWSGuardDutyEventCollector.yml @@ -119,6 +119,22 @@ configuration: defaultvalue: 'false' additionalinfo: When selected, findings that have been archived in GuardDuty (including those archived by suppression rules) are not fetched. section: Collect +- display: Fetch Events + name: isFetchEvents + type: 8 + required: false + section: Collect + hidden: + - xsoar +- display: Events Fetch Interval + name: eventFetchInterval + type: 19 + required: false + defaultvalue: '1' + section: Collect + advanced: true + hidden: + - xsoar description: Amazon Web Services Guard Duty Service (gd) event collector integration for Cortex XSIAM. script: script: "" @@ -148,7 +164,7 @@ script: name: limit description: Manual command used to fetch events and display them. name: aws-gd-get-events - dockerimage: demisto/boto3py3:1.0.0.10221838 + dockerimage: demisto/boto3py3:1.0.0.12887184 isfetchevents: true subtype: python3 marketplaces: diff --git a/Packs/AWS-GuardDuty/Integrations/AWSGuardDutyEventCollector/AWSGuardDutyEventCollector_test.py b/Packs/AWS-GuardDuty/Integrations/AWSGuardDutyEventCollector/AWSGuardDutyEventCollector_test.py index c24e9d27828e..7f0e089d7336 100644 --- a/Packs/AWS-GuardDuty/Integrations/AWSGuardDutyEventCollector/AWSGuardDutyEventCollector_test.py +++ b/Packs/AWS-GuardDuty/Integrations/AWSGuardDutyEventCollector/AWSGuardDutyEventCollector_test.py @@ -1,10 +1,18 @@ +import copy from contextlib import nullcontext as does_not_raise -from datetime import datetime +from datetime import datetime, timedelta, timezone from unittest.mock import call import demistomock as demisto import pytest -from AWSGuardDutyEventCollector import _event_updated_at, _normalize_last_ids_entry, get_events +from AWSGuardDutyEventCollector import ( + STALE_BOUNDARY_MARGIN, + _cursor_second, + _event_updated_at, + _normalize_last_ids_entry, + get_events, +) +from CommonServerPython import arg_to_datetime from test_data.finding_for_test import FINDING, FINDING_OUTPUT, MOST_GENERAL_FINDING, MOST_GENERAL_FINDING_STR LIST_DETECTORS_RESPONSE = {"DetectorIds": ["detector_id1"]} @@ -18,6 +26,28 @@ FINDINGS = {"Findings": [FINDING]} +@pytest.fixture(autouse=True) +def isolate_shared_findings(): + """Protect shared module-level test data from cross-test mutation bleed. + + Many tests build inputs via ``FINDING.copy()`` / ``FINDING_OUTPUT.copy()``, + which is a SHALLOW copy — nested dicts/lists are shared with the module-level + original, so an in-place mutation in one test could leak into another and + cause order-dependent flakiness. This autouse fixture snapshots the shared + mutable objects with ``copy.deepcopy`` before each test and restores their + contents afterwards, guaranteeing every test starts from a pristine copy + regardless of what a previous test mutated. + """ + shared = [FINDING, FINDING_OUTPUT, MOST_GENERAL_FINDING] + snapshots = [copy.deepcopy(obj) for obj in shared] + try: + yield + finally: + for obj, snapshot in zip(shared, snapshots): + obj.clear() + obj.update(snapshot) + + def get_expected_list_finding_args( detector_id: str, updated_at_ts: int, gd_severity: int, max_results: int | None, next_token: str | None ): @@ -139,7 +169,7 @@ def test_test_module(mocker): ), call( **get_expected_list_finding_args( - detector_id="detector_id2", updated_at_ts=1661681559000, gd_severity=1, max_results=9, next_token=None + detector_id="detector_id2", updated_at_ts=1661681559000, gd_severity=1, max_results=10, next_token=None ) ), ], @@ -460,7 +490,7 @@ def test_get_events_returns_datetime_as_str(mocker, list_detectors_res, list_fin [call(DetectorId="detector_id1", FindingIds=["finding_id1"])], [update_finding_id(FINDING_OUTPUT.copy(), "finding_id1", updated_at="2022-09-28T10:12:39.923854")], {"detector_id1": "2022-09-28T10:12:39.923854"}, - # XSUP-67097: last_ids is now stored as list[str] of all ids sharing + # last_ids is now stored as list[str] of all ids sharing # the cursor's UpdatedAt second, not just one id. {"detector_id1": ["finding_id1"]}, id="1 detector, 1 new finding", @@ -470,7 +500,7 @@ def test_get_events_returns_datetime_as_str(mocker, list_detectors_res, list_fin {"detector_id1": "finding_id0"}, [{"DetectorIds": ["detector_id1"]}], [{"FindingIds": ["finding_id0", "finding_id1"]}], - # XSUP-72455: both ids are fetched so the collector can inspect each finding's + # Both ids are fetched so the collector can inspect each finding's # UpdatedAt. finding_id0 still shares the cursor second (already ingested) and is # deduped after the fetch; finding_id1 is new and is ingested. [ @@ -495,29 +525,6 @@ def test_get_events_returns_datetime_as_str(mocker, list_detectors_res, list_fin {"detector_id1": ["finding_id1"]}, id="1 detector, 1 new finding, 1 old finding", ), - pytest.param( - {"detector_id1": "2022-08-28T10:12:39.923854"}, - {"detector_id1": "finding_id0"}, - [{"DetectorIds": ["detector_id1"]}], - [{"FindingIds": ["finding_id0"]}], - # XSUP-72455: the id is fetched so its UpdatedAt can be inspected. It still shares the - # cursor second (already ingested), so it is deduped after the fetch — no events, cursor unchanged. - [{"Findings": [update_finding_id(FINDING.copy(), "finding_id0", updated_at="2022-08-28T10:12:39.923854")]}], - [call(MaxResults=50)], - [ - call( - **get_expected_list_finding_args( - detector_id="detector_id1", updated_at_ts=1661681559000, gd_severity=1, max_results=10, next_token=None - ) - ) - ], - [call(DetectorId="detector_id1", FindingIds=["finding_id0"])], - [], - {"detector_id1": "2022-08-28T10:12:39.923854"}, - # last_ids preserves the already-seen id at the unchanged cursor second. - {"detector_id1": ["finding_id0"]}, - id="1 detector, 1 old finding", - ), pytest.param( {"detector_id1": "2022-08-28T10:12:39.923854"}, {"detector_id1": "finding_id0"}, @@ -556,7 +563,7 @@ def test_get_events_returns_datetime_as_str(mocker, list_detectors_res, list_fin ), call( **get_expected_list_finding_args( - detector_id="detector_id2", updated_at_ts=1659003099000, gd_severity=1, max_results=9, next_token=None + detector_id="detector_id2", updated_at_ts=1659003099000, gd_severity=1, max_results=10, next_token=None ) ), ], @@ -626,8 +633,53 @@ def test_fetch_events( assert new_last_ids == expected_new_last_ids +def test_fetch_events_single_recent_old_finding_stays_pinned(mocker): + """ + Given: + One detector whose cursor sits at a RECENT second, and the only finding + returned is the already-ingested one sharing that cursor second (no + NextToken). Extracted from the former parametrized "1 detector, 1 old + finding" case; the boundary is recent so it is within + STALE_BOUNDARY_MARGIN. + + When: + get_events runs. + + Then: + The already-seen finding is deduped (no events), and because the boundary + is still current the cursor stays pinned with last_ids preserved. (A stale + boundary would instead advance via the freeze-breaker.) + + Reference: + AWSGuardDutyEventCollector.get_events — same-second dedup pins a current + boundary; the stale-boundary advance is time-gated. + """ + cursor = datetime.utcnow().strftime("%Y-%m-%dT%H:%M:%S.%f") + + mocked_client, _, _, _ = create_mocked_client( + mocker=mocker, + list_detectors_res=[{"DetectorIds": ["detector_id1"]}], + list_finding_ids_res=[{"FindingIds": ["finding_id0"]}], + get_findings_res=[{"Findings": [update_finding_id(FINDING.copy(), "finding_id0", updated_at=cursor)]}], + ) + + events, new_last_ids, new_collect_from = get_events( + aws_client=mocked_client, + collect_from={"detector_id1": cursor}, + collect_from_default=datetime(2022, 7, 28, 10, 11, 39, 923854), + last_ids={"detector_id1": "finding_id0"}, + severity="Low", + limit=10, + ) + + assert events == [] + # Current boundary: cursor pinned and the already-seen id preserved. + assert new_collect_from == {"detector_id1": cursor} + assert new_last_ids == {"detector_id1": ["finding_id0"]} + + # --------------------------------------------------------------------------- -# Regression test for XSUP-67097 / XSUP-67552 — same-second sibling-loss bug. +# Regression test for the same-second sibling-loss bug. # # Scenario: three findings (A, B, C) all share the same UpdatedAt timestamp. # Run 1: AWS pagination returns [A, B] (limit=2 cuts off page 2). @@ -689,7 +741,7 @@ def test_same_second_sibling_loss_xsup_67097(mocker): run1_ids = sorted(e["Id"] for e in events_run1) assert run1_ids == ["finding_A", "finding_B"], f"Sanity check failed: run 1 should ingest A and B, got {run1_ids}" - # Cursor state after run 1: with the XSUP-67097 fix, last_ids stores ALL ids + # Cursor state after run 1: last_ids stores ALL ids # whose UpdatedAt equals the cursor (the same-second siblings), not just one. # Both A and B share the cursor second so both must be remembered. assert last_ids_after_run1 == {"det1": ["finding_A", "finding_B"]} @@ -726,7 +778,7 @@ def test_same_second_sibling_loss_xsup_67097(mocker): # once. Today's code drops C — this assertion fails as proof of the bug. all_ingested_ids = sorted(e["Id"] for e in (events_run1 + events_run2)) assert all_ingested_ids == ["finding_A", "finding_B", "finding_C"], ( - f"XSUP-67097 same-second sibling loss: expected all three findings " + f"Same-second sibling loss: expected all three findings " f"to be ingested across the two fetch cycles, but got {all_ingested_ids}. " f"Finding 'finding_C' was silently dropped because the dedup at " f"AWSGuardDutyEventCollector.py:135 slices after the single stored " @@ -789,7 +841,7 @@ def test_legacy_last_ids_str_shape_still_works(mocker): # --------------------------------------------------------------------------- -# Regression test for XSUP-71079 — mid-second cursor truncation. +# Regression test for mid-second cursor truncation. # # Scenario: a fetch is truncated by `limit` in the MIDDLE of a second that has # more siblings than were fetched. The cursor must NOT advance into that @@ -865,7 +917,7 @@ def test_exclude_archived_adds_criterion_xsup_71079(mocker): (default) the criterion does NOT include the archived filter. Reference: - AWSGuardDutyEventCollector._build_finding_criterion — XSUP-71079 #2. + AWSGuardDutyEventCollector._build_finding_criterion. """ finding = update_finding_id(FINDING.copy(), "finding_1", updated_at="2026-04-10T01:35:09.000000") @@ -910,7 +962,7 @@ def test_exclude_archived_adds_criterion_xsup_71079(mocker): # --------------------------------------------------------------------------- -# Regression test for XSUP-72455 — a recurring finding that is RE-UPDATED after +# Regression test for a recurring finding that is RE-UPDATED after # being stored in last_ids must not be suppressed forever. # # Scenario: finding_X is ingested at cursor second T1 and stored in last_ids. @@ -968,13 +1020,12 @@ def test_reupdated_finding_is_not_suppressed_xsup_72455(mocker): # The re-updated finding is a legitimate new event and must be ingested. assert [e["Id"] for e in events] == ["finding_X"], ( - "XSUP-72455: a finding whose UpdatedAt advanced past the stored cursor " + "A finding whose UpdatedAt advanced past the stored cursor " "second was dropped by ID-only dedup, so no events were ingested." ) # The cursor must advance to the finding's new second (T2), not stay pinned at T1. assert new_collect_from == {"det1": t2}, ( - "XSUP-72455: cursor stayed pinned at the old second because the " - "re-updated finding was suppressed, blocking all forward progress." + "Cursor stayed pinned at the old second because the " "re-updated finding was suppressed, blocking all forward progress." ) # last_ids now reflects the new cursor second (T2), holding finding_X so a # same-second re-query does not re-ingest it. @@ -993,13 +1044,16 @@ def test_same_id_same_second_still_deduped_xsup_72455(mocker): Then: finding_X must be deduped (not re-ingested) and the cursor stays at T1. - This guards that the XSUP-72455 fix does not regress the original - same-second dedup (XSUP-67097) behavior. + This guards that the re-update fix does not regress the original + same-second dedup behavior. Reference: AWSGuardDutyEventCollector.get_events — same-second re-reads are still deduped. """ - t1 = "2026-07-03T15:48:55.563000" + # A RECENT cursor second: the re-read is deduped and the cursor stays pinned + # (a real later same-second update could still arrive). Using a fixed past + # second would trip the stale-boundary advance instead. + t1 = datetime.utcnow().replace(microsecond=563000).strftime("%Y-%m-%dT%H:%M:%S.%f") same_second_again = update_finding_id(FINDING.copy(), "finding_X", updated_at=t1) client, _, _, _ = create_mocked_client( @@ -1100,7 +1154,7 @@ def test_multi_detector_truncation_advances_cursors_independently(mocker): Reference: AWSGuardDutyEventCollector.get_events — per-detector cursor bookkeeping - (XSUP-71079 truncation rollback applied per detector). + (truncation rollback applied per detector). """ a_t1 = "2026-04-10T01:35:08.000000" a_t2 = "2026-04-10T01:35:09.000000" @@ -1108,23 +1162,25 @@ def test_multi_detector_truncation_advances_cursors_independently(mocker): finding_a1 = update_finding_id(FINDING.copy(), "finding_A1", updated_at=a_t1) finding_a2 = update_finding_id(FINDING.copy(), "finding_A2", updated_at=a_t2) + finding_a3 = update_finding_id(FINDING.copy(), "finding_A3", updated_at=a_t2) finding_b1 = update_finding_id(FINDING.copy(), "finding_B1", updated_at=b_t1) - # limit=3: det_A returns 3 finding ids on one page (reaching the limit so its - # loop exits by limit) but one id is a duplicate, so it ingests only 2 events - # (A1, A2) and leaves budget for det_B to drain its single finding. + # limit=3 is applied PER DETECTOR. det_A returns 3 unique finding ids on one + # page, filling its own limit so its loop exits by limit with a pending + # NextToken (truncated mid-boundary at the partial second A_t2). det_B then + # gets its own fresh limit and drains its single finding. client, _, _, _ = create_mocked_client( mocker=mocker, list_detectors_res=[{"DetectorIds": ["det_A", "det_B"]}], list_finding_ids_res=[ - # det_A: 3 ids fill the limit => loop exits by limit; NextToken still - # pending => truncated mid-boundary at the partial second A_t2. - {"FindingIds": ["finding_A1", "finding_A2", "finding_A2"], "NextToken": "more"}, - # det_B: no NextToken => fully drained. + # det_A: 3 unique ids fill the per-detector limit => loop exits by + # limit; NextToken still pending => truncated mid-boundary at A_t2. + {"FindingIds": ["finding_A1", "finding_A2", "finding_A3"], "NextToken": "more"}, + # det_B: no NextToken => fully drained (its own fresh limit budget). {"FindingIds": ["finding_B1"]}, ], get_findings_res=[ - {"Findings": [finding_a1, finding_a2]}, + {"Findings": [finding_a1, finding_a2, finding_a3]}, {"Findings": [finding_b1]}, ], ) @@ -1139,7 +1195,7 @@ def test_multi_detector_truncation_advances_cursors_independently(mocker): ) # All distinct fetched findings are still returned this run (nothing dropped). - assert sorted(e["Id"] for e in events) == ["finding_A1", "finding_A2", "finding_B1"] + assert sorted(e["Id"] for e in events) == ["finding_A1", "finding_A2", "finding_A3", "finding_B1"] # detector_A: truncated mid-boundary => cursor rolls back to the last # fully-drained second (A_t1), NOT the partial A_t2. last_ids holds A_t1's @@ -1153,6 +1209,76 @@ def test_multi_detector_truncation_advances_cursors_independently(mocker): assert new_last_ids["det_B"] == ["finding_B1"] +def test_busy_detector_does_not_starve_later_detector(mocker): + """ + Given: + Two detectors processed in order [det_busy, det_late]. det_busy alone + produces exactly `limit` findings on its first page (filling the budget), + while det_late has its own findings to return. + + When: + get_events processes both detectors in the same run with limit=2. + + Then: + The `limit` budget is applied PER DETECTOR, so det_busy consuming the + whole limit must NOT prevent det_late from being queried. Both detectors' + findings are ingested and both cursors advance. This guards against a + global-budget starvation defect where a busy earlier detector left later + detectors un-queried with their cursor frozen. + + Reference: + AWSGuardDutyEventCollector.get_events — per-detector `limit` budget. + """ + busy_t = "2026-04-10T03:00:00.000000" + late_t = "2026-04-10T04:00:00.000000" + + finding_busy1 = update_finding_id(FINDING.copy(), "finding_busy1", updated_at=busy_t) + finding_busy2 = update_finding_id(FINDING.copy(), "finding_busy2", updated_at=busy_t) + finding_late1 = update_finding_id(FINDING.copy(), "finding_late1", updated_at=late_t) + finding_late2 = update_finding_id(FINDING.copy(), "finding_late2", updated_at=late_t) + + # limit=2: det_busy returns 2 ids on its first (and only) page => it alone + # fills the entire limit. Under a global budget, det_late's finding-ids loop + # guard would be False on entry, so list_findings would NEVER be called for + # det_late and it would be starved. Per-detector budget prevents that. + client, _, list_findings_mock, _ = create_mocked_client( + mocker=mocker, + list_detectors_res=[{"DetectorIds": ["det_busy", "det_late"]}], + list_finding_ids_res=[ + {"FindingIds": ["finding_busy1", "finding_busy2"]}, # det_busy: fills limit, fully drained + {"FindingIds": ["finding_late1", "finding_late2"]}, # det_late: must still be queried + ], + get_findings_res=[ + {"Findings": [finding_busy1, finding_busy2]}, + {"Findings": [finding_late1, finding_late2]}, + ], + ) + + events, new_last_ids, new_collect_from = get_events( + aws_client=client, + collect_from={}, + collect_from_default=datetime(2026, 4, 10, 2, 0, 0), + last_ids={}, + severity="Low", + limit=2, + ) + + # det_late was queried (not starved): list_findings called for BOTH detectors. + assert list_findings_mock.call_count == 2 + # Both detectors' findings are ingested — the later detector is not dropped. + assert sorted(e["Id"] for e in events) == [ + "finding_busy1", + "finding_busy2", + "finding_late1", + "finding_late2", + ] + # Both cursors advance independently to their respective seconds. + assert new_collect_from["det_busy"] == busy_t + assert new_collect_from["det_late"] == late_t + assert sorted(new_last_ids["det_busy"]) == ["finding_busy1", "finding_busy2"] + assert sorted(new_last_ids["det_late"]) == ["finding_late1", "finding_late2"] + + # --------------------------------------------------------------------------- # Single-boundary-cannot-drain — an entire truncated page shares one UpdatedAt. # @@ -1242,8 +1368,11 @@ def test_same_boundary_reupdate_does_not_freeze_fetch_forever(mocker): AWSGuardDutyEventCollector.get_events — dedup is scoped to the cursor second, so a later-boundary update always resumes forward progress. """ - t1 = "2026-07-03T15:48:55.563000" - t2 = "2026-07-04T08:04:55.843000" + # T1 is a RECENT second so the same-second re-reads stay pinned (not tripped by + # the stale-boundary advance). T2 is a strictly-later second. + now = datetime.utcnow() + t1 = now.replace(microsecond=563000).strftime("%Y-%m-%dT%H:%M:%S.%f") + t2 = (now + timedelta(seconds=17)).replace(microsecond=843000).strftime("%Y-%m-%dT%H:%M:%S.%f") # ---- Runs that keep re-reading finding_X at the SAME second T1 ---------- for run_number in range(3): @@ -1353,3 +1482,766 @@ def test_emitted_events_expose_updated_at_for_time_mapping(mocker): # Each event's _time source is the expected, distinct UpdatedAt value. assert sorted(_event_updated_at(e) for e in events) == [ts_1, ts_2] + + +# --------------------------------------------------------------------------- +# Regression test for sub-second (millisecond) cursor churn. +# +# Production GuardDuty findings carry MILLISECOND-precision UpdatedAt values +# (e.g. "...:23.311Z", "...:23.936Z"). Several findings routinely share the +# same whole SECOND while differing in their milliseconds. The fetch cursor is +# meant to be a per-second boundary, and every +# finding sharing the cursor's second must be remembered so the inclusive Gte +# re-query can dedup them. +# +# Two defects combined to re-ingest the whole second on every run (observed in +# live telemetry: the same (Id, UpdatedAt) pair ingested up to 1440x/day): +# 1. The cursor was set from detector_events[-1] (LIST order), not the max +# timestamp. AWS get_findings does NOT preserve the requested id order, so +# the cursor oscillated between the earliest/latest ms of the same second. +# 2. The remembered sibling set was scoped to the EXACT-microsecond cursor +# value, so only the one finding at that precise ms was remembered — the +# other same-second siblings were forgotten and re-ingested next run. +# +# This test reproduces the loop: three findings in one second at different ms, +# returned by get_findings in a NON-sorted order, run across two fetch cycles +# with the state round-tripped as a list (as setLastRun would persist it). +# --------------------------------------------------------------------------- + + +def test_millisecond_siblings_not_reingested_xsup_73410(mocker): + """ + Given: + Three findings (A, B, C) on one detector, all sharing the same whole + SECOND 13:18:23 but with different MILLISECONDS (.311 / .627 / .936). + get_findings returns them in a NON-sorted order — valid AWS behavior, + since get_findings does not preserve FindingIds order. + + When: + Run 1 fetches from before the second and ingests all three. + Run 2 re-queries with the persisted cursor (state round-tripped as a + list, as setLastRun serializes it). The inclusive Gte re-returns the + same three findings, again in a non-sorted order. + + Then: + Run 1 ingests A, B, C exactly once. + Run 2 ingests NOTHING — all three are recognized as the same + already-seen same-second findings and deduped. + The persisted cursor is deterministic (the max, .936) and stable across + runs (no oscillation), and last_ids remembers ALL three siblings. + + With the pre-fix code, run 2 re-ingests all three and the cursor + oscillates within the second, so these assertions fail — proving the + millisecond-churn re-ingestion bug. + + Reference: + AWSGuardDutyEventCollector.get_events — cursor advance + same-second + sibling dedup must operate at whole-second resolution. + """ + # Use a RECENT whole-second (within STALE_BOUNDARY_MARGIN of now) so the + # fully-deduped re-query in run 2 correctly stays pinned rather than tripping + # the stale-boundary freeze-breaker. A real later same-second update could + # still arrive at a current second, so pinning is the correct behavior here. + base_second = datetime.utcnow().replace(microsecond=0) + ts_a = base_second.replace(microsecond=311000).strftime("%Y-%m-%dT%H:%M:%S.%f") + ts_b = base_second.replace(microsecond=627000).strftime("%Y-%m-%dT%H:%M:%S.%f") + ts_c = base_second.replace(microsecond=936000).strftime("%Y-%m-%dT%H:%M:%S.%f") + + def make_findings(): + # Fresh copies each run so mutations never leak between cycles. + finding_a = update_finding_id(FINDING.copy(), "finding_A", updated_at=ts_a) + finding_b = update_finding_id(FINDING.copy(), "finding_B", updated_at=ts_b) + finding_c = update_finding_id(FINDING.copy(), "finding_C", updated_at=ts_c) + return finding_a, finding_b, finding_c + + # ------------------------------------------------------------------ Run 1 + fa1, fb1, fc1 = make_findings() + run1_client, _, _, _ = create_mocked_client( + mocker=mocker, + list_detectors_res=[{"DetectorIds": ["det1"]}], + list_finding_ids_res=[{"FindingIds": ["finding_A", "finding_B", "finding_C"]}], + # get_findings returns them in a DIFFERENT order than requested. + get_findings_res=[{"Findings": [fc1, fa1, fb1]}], + ) + + events_run1, last_ids_after_run1, collect_from_after_run1 = get_events( + aws_client=run1_client, + collect_from={}, + collect_from_default=datetime(2026, 8, 1, 13, 18, 0), + last_ids={}, + severity="Low", + limit=10, + ) + + assert sorted(e["Id"] for e in events_run1) == [ + "finding_A", + "finding_B", + "finding_C", + ], "Sanity: run 1 must ingest all three same-second findings." + # The cursor must be the MAX timestamp of the second — deterministic, not + # whichever finding happened to be last in get_findings order. + assert collect_from_after_run1 == { + "det1": ts_c + }, f"Cursor must advance to the max same-second timestamp, got {collect_from_after_run1}." + # Every finding sharing the cursor second must be remembered for dedup. + assert last_ids_after_run1 == { + "det1": ["finding_A", "finding_B", "finding_C"] + }, f"All same-second siblings must be remembered, got {last_ids_after_run1}." + + # ------------------------------------------------------------------ Run 2 + # The inclusive Gte re-query returns the same three findings again, in yet + # another order. State is round-tripped as a list (setLastRun serialization). + mocker.resetall() + fa2, fb2, fc2 = make_findings() + run2_client, _, _, _ = create_mocked_client( + mocker=mocker, + list_detectors_res=[{"DetectorIds": ["det1"]}], + list_finding_ids_res=[{"FindingIds": ["finding_A", "finding_B", "finding_C"]}], + get_findings_res=[{"Findings": [fb2, fc2, fa2]}], + ) + + events_run2, last_ids_after_run2, collect_from_after_run2 = get_events( + aws_client=run2_client, + collect_from=collect_from_after_run1, + collect_from_default=datetime(2026, 8, 1, 13, 18, 0), + last_ids=last_ids_after_run1, + severity="Low", + limit=10, + ) + + # ------------------------------------------------------------ Assertion + # Nothing new happened — the same second's findings must NOT be re-ingested. + assert events_run2 == [], ( + f"Millisecond re-ingest: the same same-second findings were " + f"re-ingested on run 2, got {[e['Id'] for e in events_run2]}. The inclusive " + f"Gte re-query returns them and they must all be deduped as already-seen." + ) + # The cursor must be STABLE — no oscillation within the second across runs. + assert collect_from_after_run2 == {"det1": ts_c}, ( + f"Cursor must remain stable at the max same-second timestamp across runs, " + f"got {collect_from_after_run2} (oscillation indicates the churn bug)." + ) + assert last_ids_after_run2 == {"det1": ["finding_A", "finding_B", "finding_C"]} + + +# --------------------------------------------------------------------------- +# Regression test for the FULLY-DEDUPED STALE BOUNDARY +# dead-end. +# +# The cursor is second-resolution and the updatedAt query is inclusive (Gte). +# On a quiet detector the fetch can reach a state where the ONLY finding at/after +# the cursor second is one already in last_ids. Every cycle then: +# 1. list_findings returns just that one already-seen id (nothing newer), +# 2. dedup drops it as a same-second sibling -> 0 events, +# 3. the cursor only advances when events are ingested -> it never moves. +# The fetch is wedged forever, even though the boundary second is long in the +# past. Live evidence: the cursor sat frozen at a single second for 9+ hours, +# re-querying one finding, ingesting nothing. +# +# The fix is time-gated: when a fetch is NOT truncated by limit (AWS returned +# everything at/after the cursor) and everything deduped away, AND the boundary +# second is safely in the PAST relative to collect_from_default (so no new +# same-second finding can still arrive), advance the cursor one second past the +# boundary and drop the stale seen_ids. A RECENT boundary must still stay pinned +# (that bounded pause is required by test_same_boundary_reupdate_...). +# --------------------------------------------------------------------------- + + +def test_stale_fully_deduped_boundary_advances_cursor_xsup_71079(mocker): + """ + Given: + A quiet detector whose cursor sits at a second T0 that is well in the + PAST (hours before collect_from_default/"now"). The only finding the + inclusive Gte query returns is finding_X, which is already in last_ids + (an already-ingested same-second finding). The fetch is NOT truncated by + limit — AWS returned everything at/after T0 and there is nothing newer. + + When: + get_events runs this cycle. + + Then: + Nothing is ingested (finding_X is correctly deduped), but because the + boundary second is stale and fully drained, the cursor MUST advance one + second past T0 so the next run queries strictly after the dead boundary, + and the stale seen_ids for that detector MUST be cleared. This breaks the + perpetual freeze. + + With the pre-fix code the elif-finding_ids branch leaves the cursor and + seen_ids unchanged, so the cursor stays frozen at T0 forever — these + assertions fail, proving the dead-end. + + Reference: + AWSGuardDutyEventCollector.get_events — the fully-deduped-boundary + advance guard (time-gated so a recent boundary still pins). + """ + # A fixed past boundary is stale against real wall-clock now — no mocking needed. + # collect_from_default is the first_fetch lookback FLOOR and is intentionally + # unrelated to the boundary; the guard must not depend on it. + collect_from_default_floor = datetime(2020, 1, 1, 0, 0, 0) + t0 = "2020-01-01T08:19:28.960000" # years in the past — clearly stale + t0_plus_1s = "2020-01-01T08:19:29" # cursor must step to the next second + + stale_only = update_finding_id(FINDING.copy(), "finding_X", updated_at=t0) + client, _, _, _ = create_mocked_client( + mocker=mocker, + list_detectors_res=[{"DetectorIds": ["det1"]}], + # Only the already-seen boundary finding is returned; NO NextToken -> not + # truncated by limit, so AWS has nothing newer at/after T0. + list_finding_ids_res=[{"FindingIds": ["finding_X"]}], + get_findings_res=[{"Findings": [stale_only]}], + ) + + events, new_last_ids, new_collect_from = get_events( + aws_client=client, + collect_from={"det1": t0}, + collect_from_default=collect_from_default_floor, + last_ids={"det1": ["finding_X"]}, + severity="Low", + limit=10, + ) + + # Nothing new happened this cycle — the one boundary finding is deduped. + assert events == [], "The already-seen boundary finding must not be re-ingested." + # The wedge must break: the cursor steps one whole second past the stale + # boundary so the next inclusive Gte query starts strictly after it. + assert new_collect_from == {"det1": t0_plus_1s}, ( + f"Stale fully-deduped boundary must advance the cursor past T0, got " + f"{new_collect_from} (unchanged == the permanent freeze)." + ) + # The stale seen_ids belonged to the dead boundary second; the new second has + # no known siblings, so they must be cleared. + assert new_last_ids == {"det1": []}, f"Seen ids for the abandoned boundary second must be cleared, got {new_last_ids}." + + +def test_recent_fully_deduped_boundary_stays_pinned_xsup_71079(mocker): + """ + Given: + The same fully-deduped single-finding situation as the stale-boundary + test, but the cursor second is RECENT — only moments before + collect_from_default/"now". A genuine later-same-second update could + still arrive, so the cursor must not jump past it yet. + + When: + get_events runs this cycle. + + Then: + Nothing is ingested and the cursor stays PINNED at the boundary second + with its seen_ids preserved (the bounded, correct pause). This guards the + time-gate: the freeze-breaking advance must fire ONLY for stale + boundaries, never for a still-current second. + + Reference: + AWSGuardDutyEventCollector.get_events — the fully-deduped-boundary + advance is time-gated; a recent boundary keeps the existing pin behavior. + """ + # Boundary is only a few seconds before real wall-clock now — still current, + # so it falls inside STALE_BOUNDARY_MARGIN and must stay pinned. No mocking. + now = datetime.utcnow() + t_recent = (now - timedelta(seconds=3)).strftime("%Y-%m-%dT%H:%M:%S.%f") + + recent_only = update_finding_id(FINDING.copy(), "finding_X", updated_at=t_recent) + client, _, _, _ = create_mocked_client( + mocker=mocker, + list_detectors_res=[{"DetectorIds": ["det1"]}], + list_finding_ids_res=[{"FindingIds": ["finding_X"]}], + get_findings_res=[{"Findings": [recent_only]}], + ) + + events, new_last_ids, new_collect_from = get_events( + aws_client=client, + collect_from={"det1": t_recent}, + collect_from_default=now, + last_ids={"det1": ["finding_X"]}, + severity="Low", + limit=10, + ) + + assert events == [], "The already-seen finding must not be re-ingested." + # Recent boundary: keep the bounded pause so a later same-second update is + # still catchable — cursor and seen_ids unchanged. + assert new_collect_from == {"det1": t_recent}, f"A recent fully-deduped boundary must stay pinned, got {new_collect_from}." + assert new_last_ids == { + "det1": ["finding_X"] + }, f"Seen ids for a still-current boundary must be preserved, got {new_last_ids}." + + +def test_stale_boundary_with_a_genuinely_new_finding_ingests_and_does_not_skip_xsup_71079(mocker): + """ + Given: + A stale cursor second T0. This cycle the inclusive Gte query returns the + already-seen boundary finding (finding_old, in last_ids at T0) AND a + genuinely new finding (finding_new) whose UpdatedAt is a strictly-later + second T1 — a real occurrence that must be ingested. + + When: + get_events runs. + + Then: + finding_new is ingested (it is NOT deduped — its second is past the + cursor) and the cursor advances to T1 via the NORMAL ingest path. The + stale-boundary freeze-breaker must NOT fire here (there are events), and + crucially nothing is skipped: a real update at a stale boundary is never + lost by the freeze-breaking logic. + + Reference: + AWSGuardDutyEventCollector.get_events — the stale-boundary advance lives + in the elif-finding_ids (zero-events) branch only; any real ingestion + takes the normal path and is unaffected. + """ + t0 = "2020-01-01T08:19:28.960000" # stale boundary (already seen) + t1 = "2020-01-01T08:19:45.100000" # a genuinely newer second + + finding_old = update_finding_id(FINDING.copy(), "finding_old", updated_at=t0) + finding_new = update_finding_id(FINDING.copy(), "finding_new", updated_at=t1) + client, _, _, _ = create_mocked_client( + mocker=mocker, + list_detectors_res=[{"DetectorIds": ["det1"]}], + list_finding_ids_res=[{"FindingIds": ["finding_old", "finding_new"]}], + get_findings_res=[{"Findings": [finding_old, finding_new]}], + ) + + events, new_last_ids, new_collect_from = get_events( + aws_client=client, + collect_from={"det1": t0}, + collect_from_default=datetime(2020, 1, 1, 0, 0, 0), + last_ids={"det1": ["finding_old"]}, + severity="Low", + limit=10, + ) + + # The genuinely new finding must be ingested (the old boundary one is deduped). + assert [e["Id"] for e in events] == [ + "finding_new" + ], f"A real new finding at a stale boundary must be ingested, got {[e['Id'] for e in events]}." + # Cursor advances to the new finding's second via the normal ingest path — + # NOT via the freeze-breaker (+1s), proving the guard did not fire. + assert new_collect_from == {"det1": t1}, f"Cursor must advance to the ingested finding's timestamp, got {new_collect_from}." + assert new_last_ids == { + "det1": ["finding_new"] + }, f"Only the new cursor-second sibling should be remembered, got {new_last_ids}." + + +def test_truncated_stale_fully_deduped_boundary_does_not_advance_xsup_71079(mocker): + """ + Given: + A stale cursor second T0 whose only returned finding is already-seen (so + it dedups to zero events), BUT the finding-id listing was TRUNCATED by + limit — list_findings returned a NextToken, meaning AWS still has + un-fetched findings that may sit at this same boundary second. + + When: + get_events runs. + + Then: + The cursor must NOT advance and seen_ids must be preserved. Even though + the boundary is stale, advancing past a limit-truncated second could skip + the un-fetched findings still sitting there. The freeze-breaker is gated + on `not truncated_by_limit` precisely to prevent that data loss. + + Reference: + AWSGuardDutyEventCollector.get_events — the stale-boundary advance + requires a fully-drained (non-truncated) page. + """ + t0 = "2020-01-01T08:19:28.960000" # stale boundary + + stale_seen = update_finding_id(FINDING.copy(), "finding_X", updated_at=t0) + client, _, _, _ = create_mocked_client( + mocker=mocker, + list_detectors_res=[{"DetectorIds": ["det1"]}], + # NextToken present => the id-listing loop stops on limit with more to + # fetch, so truncated_by_limit is True. + list_finding_ids_res=[{"FindingIds": ["finding_X"], "NextToken": "more"}], + get_findings_res=[{"Findings": [stale_seen]}], + ) + + events, new_last_ids, new_collect_from = get_events( + aws_client=client, + collect_from={"det1": t0}, + collect_from_default=datetime(2020, 1, 1, 0, 0, 0), + last_ids={"det1": ["finding_X"]}, + severity="Low", + limit=1, # force the listing loop to stop with the NextToken still pending + ) + + assert events == [], "Nothing new should be ingested." + # Truncated page => must NOT advance past the boundary (un-fetched siblings + # may still be there); cursor and seen_ids stay put. + assert new_collect_from == { + "det1": t0 + }, f"A limit-truncated stale boundary must NOT advance the cursor, got {new_collect_from}." + assert new_last_ids == { + "det1": ["finding_X"] + }, f"Seen ids must be preserved when the boundary page was truncated, got {new_last_ids}." + + +def test_stale_wedged_detector_advances_without_affecting_healthy_detector_xsup_71079(mocker): + """ + Given: + Two detectors on one instance: + - det_stuck: wedged at a stale fully-deduped boundary (only its + already-seen finding is returned). + - det_ok: healthy, returning a brand-new finding to ingest. + + When: + get_events runs over both detectors in one cycle. + + Then: + det_stuck's cursor is unwedged (advances one second past its stale + boundary, seen_ids cleared) while det_ok ingests its new finding and + advances normally. The freeze-breaker is per-detector and must not + disturb a healthy detector processed in the same run. + + Reference: + AWSGuardDutyEventCollector.get_events — per-detector cursor/seen-ids + handling; the stale-boundary advance is scoped to the affected detector. + """ + t_stale = "2020-01-01T08:19:28.960000" + t_stale_plus_1s = "2020-01-01T08:19:29" + t_new = "2020-01-01T17:59:10.500000" # healthy detector's new finding (ingested via normal path) + + stuck_seen = update_finding_id(FINDING.copy(), "finding_stuck", updated_at=t_stale) + ok_new = update_finding_id(FINDING.copy(), "finding_ok", updated_at=t_new) + + # Detectors are processed in listing order: det_stuck first, then det_ok. + client, _, _, _ = create_mocked_client( + mocker=mocker, + list_detectors_res=[{"DetectorIds": ["det_stuck", "det_ok"]}], + list_finding_ids_res=[ + {"FindingIds": ["finding_stuck"]}, # det_stuck listing + {"FindingIds": ["finding_ok"]}, # det_ok listing + ], + get_findings_res=[ + {"Findings": [stuck_seen]}, # det_stuck get_findings + {"Findings": [ok_new]}, # det_ok get_findings + ], + ) + + events, new_last_ids, new_collect_from = get_events( + aws_client=client, + collect_from={"det_stuck": t_stale}, # det_ok has no prior cursor + collect_from_default=datetime(2020, 1, 1, 0, 0, 0), + last_ids={"det_stuck": ["finding_stuck"]}, + severity="Low", + limit=10, + ) + + # Only the healthy detector's new finding is ingested. + assert [e["Id"] for e in events] == [ + "finding_ok" + ], f"Healthy detector's new finding must be ingested, got {[e['Id'] for e in events]}." + # Stuck detector unwedged; healthy detector advanced normally to its finding. + assert new_collect_from == {"det_stuck": t_stale_plus_1s, "det_ok": t_new}, ( + f"Stuck detector must advance past its stale boundary and healthy detector " + f"must advance to its ingested finding, got {new_collect_from}." + ) + # Stuck detector's stale seen_ids cleared; healthy detector remembers its sibling. + assert new_last_ids == {"det_stuck": [], "det_ok": ["finding_ok"]}, f"Per-detector seen ids incorrect, got {new_last_ids}." + + +def test_z_suffixed_stale_cursor_is_comparable_to_wallclock_now_xsup_71079(mocker): + """ + Given: + The cursor is a real GuardDuty timestamp string in UTC "Z" form + (e.g. "2020-01-01T08:19:28.960Z"), matching what GuardDuty emits, and the + only finding returned is already-seen (fully deduped, not truncated). + + When: + get_events evaluates the stale-boundary guard, which compares the parsed + cursor second against datetime.utcnow(). + + Then: + The comparison must not raise (parse_date_string yields a NAIVE UTC + datetime for "Z" input, and utcnow() is also naive UTC), and the stale + boundary must advance normally. This pins the naive-UTC assumption so a + future change to timestamp parsing can't silently break the comparison + (e.g. by introducing an offset-aware datetime, which would raise + "can't compare offset-naive and offset-aware datetimes"). + + Reference: + AWSGuardDutyEventCollector.get_events — staleness compares the parsed + cursor second to naive-UTC now. + """ + t0_z = "2020-01-01T08:19:28.960Z" # real GuardDuty UTC "Z" form, years past + t0_plus_1s = "2020-01-01T08:19:29" + + stale_only = update_finding_id(FINDING.copy(), "finding_X", updated_at=t0_z) + client, _, _, _ = create_mocked_client( + mocker=mocker, + list_detectors_res=[{"DetectorIds": ["det1"]}], + list_finding_ids_res=[{"FindingIds": ["finding_X"]}], + get_findings_res=[{"Findings": [stale_only]}], + ) + + with does_not_raise(): + events, new_last_ids, new_collect_from = get_events( + aws_client=client, + collect_from={"det1": t0_z}, + collect_from_default=datetime(2020, 1, 1, 0, 0, 0), + last_ids={"det1": ["finding_X"]}, + severity="Low", + limit=10, + ) + + assert events == [], "The already-seen boundary finding must not be re-ingested." + assert new_collect_from == {"det1": t0_plus_1s}, ( + f"A Z-suffixed stale boundary must compare cleanly to naive-UTC now and advance, " f"got {new_collect_from}." + ) + assert new_last_ids == {"det1": []}, f"Seen ids for the abandoned boundary second must be cleared, got {new_last_ids}." + + +# --------------------------------------------------------------------------- +# Regression tests for timezone normalization in _cursor_second. +# +# collect_from_default (the first_fetch floor) reaches get_events tz-AWARE when +# parsed by arg_to_datetime for an absolute date (e.g. "2026-08-01T00:00:00Z"), +# while finding UpdatedAt values are parsed by parse_date_string (no %z) and are +# always NAIVE. Before normalization, comparing the two either never matched +# (dedup silently failed -> every finding re-ingested) or raised +# "TypeError: can't compare offset-naive and offset-aware datetimes" against the +# naive datetime.utcnow() in the stale-boundary check. _cursor_second must render +# every value naive-UTC so all comparisons are clean. +# --------------------------------------------------------------------------- + + +def test_cursor_second_normalizes_tz_aware_to_naive_utc(): + """ + Given: + A tz-aware timestamp (both a "Z"/UTC value and a "+05:00" offset value) + and a naive value. + + When: + _cursor_second is called on each. + + Then: + The result is always naive (tzinfo is None), truncated to whole seconds, + and the offset value is converted to UTC BEFORE tzinfo is dropped (so a + +05:00 time is shifted back 5 hours, not merely stripped). + """ + # tz-aware UTC ("Z") input -> naive-UTC, seconds truncated. + aware_utc = arg_to_datetime("2026-08-01T00:00:00Z") + result_utc = _cursor_second(aware_utc) + assert result_utc.tzinfo is None + assert result_utc == datetime(2026, 8, 1, 0, 0, 0) + + # tz-aware +05:00 input -> converted to UTC (05:00 -> 00:00) THEN made naive. + aware_offset = datetime(2026, 8, 1, 5, 0, 0, tzinfo=timezone(timedelta(hours=5))) + assert _cursor_second(aware_offset) == datetime(2026, 8, 1, 0, 0, 0) + + # Naive input is unchanged apart from microsecond truncation. + assert _cursor_second(datetime(2026, 8, 1, 0, 0, 0, 123456)) == datetime(2026, 8, 1, 0, 0, 0) + + # A naive string (as produced by parse_date_string) truncates to the second. + assert _cursor_second("2026-08-01T00:00:00.936Z") == datetime(2026, 8, 1, 0, 0, 0) + + +def test_tz_aware_collect_from_default_dedups_without_typeerror(mocker): + """ + Given: + A RECENT cursor whose collect_from_default is tz-AWARE (as arg_to_datetime + returns for an absolute first_fetch), and the only finding the inclusive + Gte query returns is one already in last_ids at that same whole second + (naive UpdatedAt from parse_date_string). + + When: + get_events runs. + + Then: + The tz-aware cursor and the naive finding timestamp are both normalized to + naive-UTC, so the same-second dedup matches: no TypeError is raised and the + already-seen finding is NOT re-ingested (events == []). A recent boundary + stays pinned (not tripped by the stale-boundary advance). + + Before the normalization fix, the naive/aware mismatch made dedup never + match (re-ingesting the finding) or raised a TypeError in the stale-boundary + comparison — this test guards both. + + Reference: + AWSGuardDutyEventCollector._cursor_second — tz normalization at the single + comparison choke point. + """ + # A recent whole-second, expressed as a tz-aware "Z" string, matching how an + # absolute first_fetch is stored/parsed. The finding shares this exact second. + base_second = datetime.utcnow().replace(microsecond=0) + cursor_str = base_second.strftime("%Y-%m-%dT%H:%M:%S") + "Z" + collect_from_default_aware = arg_to_datetime(cursor_str) + finding_ts = base_second.strftime("%Y-%m-%dT%H:%M:%S.%f") + + mocked_client, _, _, _ = create_mocked_client( + mocker=mocker, + list_detectors_res=[{"DetectorIds": ["det1"]}], + list_finding_ids_res=[{"FindingIds": ["finding_seen"]}], + get_findings_res=[{"Findings": [update_finding_id(FINDING.copy(), "finding_seen", updated_at=finding_ts)]}], + ) + + with does_not_raise(): + events, new_last_ids, new_collect_from = get_events( + aws_client=mocked_client, + collect_from={"det1": cursor_str}, + collect_from_default=collect_from_default_aware, + last_ids={"det1": ["finding_seen"]}, + severity="Low", + limit=10, + ) + + assert events == [], ( + "A tz-aware cursor and a naive finding timestamp at the same second must " + "dedup cleanly; the already-seen finding must not be re-ingested." + ) + # Recent boundary: cursor stays pinned and the seen id is preserved for next run. + assert new_collect_from == {"det1": cursor_str} + assert new_last_ids == {"det1": ["finding_seen"]} + + +def test_cursor_second_raises_on_unparseable_value(): + """ + Given: + A value that is not a parseable timestamp. + + When: + _cursor_second is called on it. + + Then: + It raises (surfacing loudly rather than silently corrupting the cursor), + as promised by the docstring. + """ + with pytest.raises((ValueError, TypeError)): + _cursor_second("not-a-timestamp") + + +def test_event_updated_at_falls_back_to_created_at(): + """ + Given: + A finding with no ``UpdatedAt`` but with a ``CreatedAt``, and a finding + that has both. + + When: + _event_updated_at is called. + + Then: + It returns ``CreatedAt`` when ``UpdatedAt`` is absent, and prefers + ``UpdatedAt`` when present. (Guards the CreatedAt fallback.) + """ + assert _event_updated_at({"CreatedAt": "2026-08-01T00:00:00.000Z"}) == "2026-08-01T00:00:00.000Z" + assert ( + _event_updated_at({"UpdatedAt": "2026-08-02T00:00:00.000Z", "CreatedAt": "2026-08-01T00:00:00.000Z"}) + == "2026-08-02T00:00:00.000Z" + ) + + +def test_finding_without_id_does_not_leak_none_into_last_ids(mocker): + """ + Given: + A single NEW finding at a recent second that is missing its ``Id`` field + (defensive: real GuardDuty findings always carry an Id). + + When: + get_events ingests it and builds the same-second sibling id set. + + Then: + The event is still ingested, but ``None`` must never appear in the + persisted ``last_ids`` (guards the ``discard(None)`` in cursor bookkeeping), + and the persisted list must remain JSON-serializable/sortable. + """ + recent = datetime.utcnow().replace(microsecond=0).strftime("%Y-%m-%dT%H:%M:%S.%f") + finding_no_id = update_finding_id(FINDING.copy(), "placeholder", updated_at=recent) + finding_no_id.pop("Id", None) + + mocked_client, _, _, _ = create_mocked_client( + mocker=mocker, + list_detectors_res=[{"DetectorIds": ["det1"]}], + list_finding_ids_res=[{"FindingIds": ["some_id"]}], + get_findings_res=[{"Findings": [finding_no_id]}], + ) + + _events, new_last_ids, _new_collect_from = get_events( + aws_client=mocked_client, + collect_from={"det1": recent}, + collect_from_default=datetime(2026, 8, 1, 0, 0, 0), + last_ids={}, + severity="Low", + limit=10, + ) + + assert None not in new_last_ids.get("det1", []), "None must never be persisted as a same-second sibling id." + + +def _fully_deduped_boundary_client(mocker, cursor_str): + """Build a client whose only finding is the already-seen boundary sibling.""" + return create_mocked_client( + mocker=mocker, + list_detectors_res=[{"DetectorIds": ["det1"]}], + list_finding_ids_res=[{"FindingIds": ["finding_seen"]}], + get_findings_res=[{"Findings": [update_finding_id(FINDING.copy(), "finding_seen", updated_at=cursor_str)]}], + )[0] + + +def test_stale_boundary_well_past_margin_advances_cursor(mocker): + """ + Given: + A fully-deduped boundary whose second is one minute older than + STALE_BOUNDARY_MARGIN. The only finding returned is already in last_ids, + nothing is newer, and the page is not limit-truncated. + + When: + get_events runs. + + Then: + The stale boundary advances exactly one second past the drained second and the + now-abandoned seen ids are cleared, breaking the stall. + + Reference: + AWSGuardDutyEventCollector.get_events — the time-gated fully-deduped + stale-boundary advance (STALE_BOUNDARY_MARGIN). + """ + boundary_second = datetime.utcnow() - (STALE_BOUNDARY_MARGIN + timedelta(minutes=1)) + cursor_str = boundary_second.strftime("%Y-%m-%dT%H:%M:%S.%f") + mocked_client = _fully_deduped_boundary_client(mocker, cursor_str) + + events, new_last_ids, new_collect_from = get_events( + aws_client=mocked_client, + collect_from={"det1": cursor_str}, + collect_from_default=datetime(2026, 8, 1, 0, 0, 0), + last_ids={"det1": ["finding_seen"]}, + severity="Low", + limit=10, + ) + + expected_next = (boundary_second.replace(microsecond=0) + timedelta(seconds=1)).isoformat() + assert events == [], "The already-seen boundary finding must never be re-ingested." + assert new_collect_from == {"det1": expected_next}, "A stale boundary must advance one second past the drained second." + assert new_last_ids == {"det1": []}, "Stale-boundary advance must clear the now-abandoned seen ids." + + +def test_current_boundary_within_margin_stays_pinned(mocker): + """ + Given: + A fully-deduped boundary whose second is one minute inside + STALE_BOUNDARY_MARGIN. The only finding returned is already in last_ids, + nothing is newer, and the page is not limit-truncated. + + When: + get_events runs. + + Then: + The cursor is left pinned (a genuine later same-second update could still + arrive) and the seen ids are preserved. + + Reference: + AWSGuardDutyEventCollector.get_events — a recent boundary stays pinned. + """ + boundary_second = datetime.utcnow() - (STALE_BOUNDARY_MARGIN - timedelta(minutes=1)) + cursor_str = boundary_second.strftime("%Y-%m-%dT%H:%M:%S.%f") + mocked_client = _fully_deduped_boundary_client(mocker, cursor_str) + + events, new_last_ids, new_collect_from = get_events( + aws_client=mocked_client, + collect_from={"det1": cursor_str}, + collect_from_default=datetime(2026, 8, 1, 0, 0, 0), + last_ids={"det1": ["finding_seen"]}, + severity="Low", + limit=10, + ) + + assert events == [], "The already-seen boundary finding must never be re-ingested." + assert new_collect_from == {"det1": cursor_str}, "A current boundary must stay pinned." + assert new_last_ids == {"det1": ["finding_seen"]}, "A pinned boundary must preserve its seen ids." diff --git a/Packs/AWS-GuardDuty/Integrations/AWSGuardDutyEventCollector/test_data/finding_for_test.py b/Packs/AWS-GuardDuty/Integrations/AWSGuardDutyEventCollector/test_data/finding_for_test.py index 20b7be90d487..4d15dbeec93e 100644 --- a/Packs/AWS-GuardDuty/Integrations/AWSGuardDutyEventCollector/test_data/finding_for_test.py +++ b/Packs/AWS-GuardDuty/Integrations/AWSGuardDutyEventCollector/test_data/finding_for_test.py @@ -581,7 +581,7 @@ "Severity": 123.0, "Title": "string", "Type": "string", - "UpdatedAt": "string", + "UpdatedAt": "2015-01-01T00:00:00.000000", } MOST_GENERAL_FINDING_STR = { @@ -911,5 +911,5 @@ "Severity": 123.0, "Title": "string", "Type": "string", - "UpdatedAt": "string", + "UpdatedAt": "2015-01-01T00:00:00.000000", } diff --git a/Packs/AWS-GuardDuty/ModelingRules/AWSGuardDutyModelingRules_1_3/AWSGuardDutyModelingRules_1_3.yml b/Packs/AWS-GuardDuty/ModelingRules/AWSGuardDutyModelingRules_1_3/AWSGuardDutyModelingRules_1_3.yml index 7e10096a0fc7..b718c4071813 100644 --- a/Packs/AWS-GuardDuty/ModelingRules/AWSGuardDutyModelingRules_1_3/AWSGuardDutyModelingRules_1_3.yml +++ b/Packs/AWS-GuardDuty/ModelingRules/AWSGuardDutyModelingRules_1_3/AWSGuardDutyModelingRules_1_3.yml @@ -1,4 +1,5 @@ fromversion: 6.10.0 +toversion: 8.14.0 id: aws_guard_duty_modeling_rule name: AWSGuardDuty Modeling Rule rules: '' diff --git a/Packs/AWS-GuardDuty/ModelingRules/AWSGuardDutyModelingRules_2_11/AWSGuardDutyModelingRules_2_11.xif b/Packs/AWS-GuardDuty/ModelingRules/AWSGuardDutyModelingRules_2_11/AWSGuardDutyModelingRules_2_11.xif new file mode 100644 index 000000000000..0fa070526c9a --- /dev/null +++ b/Packs/AWS-GuardDuty/ModelingRules/AWSGuardDutyModelingRules_2_11/AWSGuardDutyModelingRules_2_11.xif @@ -0,0 +1,237 @@ +[MODEL: dataset="aws_guardduty_raw"] +alter // init aliases for root level & common objects for supporting both camelCase and PascalCase field naming convention extractions + finding_resource = coalesce(Resource, resource), + finding_service = coalesce(Service, service) +| alter + finding_process = coalesce(finding_service -> RuntimeDetails.Process{}, service -> runtimeDetails.process{}), + finding_network = coalesce(finding_service -> Action.NetworkConnectionAction{}, service -> action.networkConnectionAction{}), + finding_aws_api = coalesce(finding_service -> Action.AwsApiCallAction{}, service -> action.awsApiCallAction{}), + finding_k8s_api = coalesce(finding_service -> Action.KubernetesApiCallAction{}, service -> action.kubernetesApiCallAction{}), + finding_rds_login = coalesce(finding_service -> Action.RdsLoginAttemptAction{}, service -> action.rdsLoginAttemptAction{}) + +| alter // extract common finding fields: https://docs.aws.amazon.com/guardduty/latest/APIReference/API_Finding.html + finding_account_id = coalesce(AccountId, accountId), // The ID of the account in which the finding was generated. + finding_arn = coalesce(Arn, arn), // The ARN of the finding. + finding_description = coalesce(Description, description), // The description of the finding. + finding_id = coalesce(Id, id), // The ID of the finding. + finding_region = coalesce(Region, region), // The Region where the finding was generated. + finding_severity = to_float(coalesce(Severity, severity)), // The severity of the finding: https://docs.aws.amazon.com/guardduty/latest/ug/guardduty_findings.html#guardduty_findings-severity + finding_title = coalesce(Title, title), // The title of the finding. + finding_type = coalesce(Type, type), // The type of finding. + + // extract resource fields: https://docs.aws.amazon.com/guardduty/latest/APIReference/API_Resource.html + resource_type = coalesce(finding_resource -> ResourceType, resource -> resourceType), // The type of AWS resource. + resource_instance_id = coalesce(finding_resource -> InstanceDetails.InstanceId, resource -> instanceDetails.instanceId), // The ID of the EC2 instance. + resource_instance_type = coalesce(finding_resource -> InstanceDetails.InstanceType, resource -> instanceDetails.instanceType), // The type of the EC2 instance. + resource_username = coalesce( + finding_resource -> RdsDbUserDetails.User, resource -> rdsDbUserDetails.user, // The user name used in the anomalous login attempt. + finding_resource -> AccessKeyDetails.UserName, resource -> accessKeyDetails.userName, // The name of the user. + finding_resource -> KubernetesDetails.KubernetesUserDetails.Username, resource -> kubernetesDetails.kubernetesUserDetails.username), // The username of the user who called the Kubernetes API. + resource_user_id = coalesce( + finding_resource -> AccessKeyDetails.PrincipalId, resource -> accessKeyDetails.principalId, // The principal ID of the user. + finding_resource -> KubernetesDetails.KubernetesUserDetails.Uid, resource -> kubernetesDetails.kubernetesUserDetails.uid), // The user ID of the user who called the Kubernetes API. + resource_user_type = coalesce(finding_resource -> AccessKeyDetails.UserType, resource -> accessKeyDetails.userType), // The type of the user. + resource_user_groups = coalesce(finding_resource -> KubernetesDetails.KubernetesUserDetails.Groups[], resource -> kubernetesDetails.kubernetesUserDetails.groups[]), // The groups that include the user who called the Kubernetes API. + resource_rdsDbUserDetails_application = coalesce(finding_resource -> RdsDbUserDetails.Application, resource -> rdsDbUserDetails.application), // The application name used in the anomalous login attempt. + resource_rdsDbUserDetails_auth_method = coalesce(finding_resource -> RdsDbUserDetails.AuthMethod, resource -> rdsDbUserDetails.authMethod), // The authentication method used by the user involved in the finding. + resource_availability_zone = coalesce( // The Availability Zone of the EC2 instance. + finding_resource -> Instance.InstanceDetails.AvailabilityZone, resource -> instance.instanceDetails.availabilityZone, + finding_resource -> EcsClusterDetails.InstanceDetails.AvailabilityZone, resource -> ecsClusterDetails.instanceDetails.availabilityZone, + finding_resource -> EksClusterDetails.InstanceDetails.AvailabilityZone, resource -> eksClusterDetails.instanceDetails.availabilityZone), + resource_name = coalesce( + finding_resource -> LambdaDetails.FunctionName, resource -> lambdaDetails.functionName, // Lambda + finding_resource -> ContainerDetails.Name, resource -> containerDetails.name, // Container + finding_resource -> RdsDbUserDetails.Database, resource -> rdsDbUserDetails.database, // RDSDBInstance + finding_resource -> EcsClusterDetails.InstanceDetails.EcsClusterDetails.Name, resource -> ecsClusterDetails.instanceDetails.ecsClusterDetails.name, // ECSCluster + finding_resource -> EksClusterDetails.InstanceDetails.EksClusterDetails.Name, resource -> eksClusterDetails.instanceDetails.eksClusterDetails.name, // EKSCluster + finding_resource -> InstanceDetails.InstanceId, resource -> instanceDetails.instanceId, // Instance + finding_resource -> AccessKeyDetails.UserName, resource -> accessKeyDetails.userName, // AccessKey + arraystring(arraymap(finding_resource -> S3BucketDetails[], "@element" -> Name), ","), arraystring(arraymap(resource -> s3BucketDetails[], "@element" -> name), ",")), // S3Object + + // extract common service fields: https://docs.aws.amazon.com/guardduty/latest/APIReference/API_Service.html + service_action_type = coalesce(finding_service -> Action.ActionType, service -> action.actionType), // The GuardDuty finding activity type. + service_detector_id = coalesce(finding_service -> DetectorId, service -> detectorId), // The detector ID for the GuardDuty service. + service_evidence_threat_intelligence_details = coalesce(finding_service -> Evidence.ThreatIntelligenceDetails[], service -> evidence.threatIntelligenceDetails[]), // A list of threat intelligence details related to the evidence. + + // extract service runtime fields: https://docs.aws.amazon.com/guardduty/latest/APIReference/API_ProcessDetails.html + service_runtime_details_process_pid = to_integer(coalesce(finding_process -> Pid, finding_process -> pid)), // The ID of the process. + service_runtime_details_process_uuid = coalesce(finding_process -> Uuid, finding_process -> uuid), // The unique ID assigned to the process by GuardDuty. + service_runtime_details_process_parent_uuid = coalesce(finding_process -> ParentUuid, finding_process -> parentUuid), // The unique ID of the parent process. This ID is assigned to the parent process by GuardDuty. + service_runtime_details_process_name = coalesce(finding_process -> Name, finding_process -> name), // The name of the process. + service_runtime_details_process_executable_path = coalesce(finding_process -> ExecutablePath, finding_process -> executablePath), // The absolute path of the process executable file. + service_runtime_details_process_executable_sha256 = coalesce(finding_process -> ExecutableSha256, finding_process -> executableSha256), // The SHA256 hash of the process executable. + service_runtime_details_process_user = coalesce(finding_process -> User, finding_process -> user), // The user that executed the process. + service_runtime_details_process_user_id = coalesce(finding_process -> UserId, finding_process -> userId), // The unique ID of the user that executed the process. + + // extract service network connection action fields: https://docs.aws.amazon.com/guardduty/latest/APIReference/API_NetworkConnectionAction.html + service_action_network_connection_is_blocked = to_boolean(coalesce(finding_network -> Blocked, finding_network -> blocked)), // Indicates whether EC2 blocked the network connection to your instance. + service_action_network_connection_direction = coalesce(finding_network -> ConnectionDirection, finding_network -> connectionDirection), // The network connection direction. + service_action_network_connection_protocol = coalesce(finding_network -> Protocol, finding_network -> protocol), // The network connection protocol. + service_action_network_connection_local_ipv4 = coalesce(finding_network -> LocalIpDetails.IpAddressV4, finding_network -> localIpDetails.ipAddressV4), // The IPv4 local address of the connection. + service_action_network_connection_local_ipv6 = coalesce(finding_network -> LocalIpDetails.IpAddressV6, finding_network -> localIpDetails.ipAddressV6), // The IPv6 local address of the connection. + service_action_network_connection_local_port = coalesce(finding_network -> LocalPortDetails.Port, finding_network -> localPortDetails.port), // The port number of the local connection. + service_action_network_connection_local_port_name = coalesce(finding_network -> LocalPortDetails.PortName, finding_network -> localPortDetails.portName), // The port name of the local connection. + service_action_network_connection_remote_asn = coalesce(finding_network -> RemoteIpDetails.Organization.Asn, finding_network -> remoteIpDetails.organization.asn), //(ASN) of the internet provider of the remote IP address. + service_action_network_connection_remote_asn_org = coalesce(finding_network -> RemoteIpDetails.Organization.AsnOrg, finding_network -> remoteIpDetails.organization.asnOrg), // The organization that registered this ASN. + service_action_network_connection_remote_asn_isp = coalesce(finding_network -> RemoteIpDetails.Organization.Isp, finding_network -> remoteIpDetails.organization.isp), // The ISP information for the internet provider. + service_action_network_connection_remote_asn_isp_org = coalesce(finding_network -> RemoteIpDetails.Organization.Org, finding_network -> remoteIpDetails.organization.org), // The name of the internet provider. + service_action_network_connection_remote_ipv4 = coalesce(finding_network -> RemoteIpDetails.IpAddressV4, finding_network -> remoteIpDetails.ipAddressV4), // The IPv4 remote address of the connection. + service_action_network_connection_remote_ipv6 = coalesce(finding_network -> RemoteIpDetails.IpAddressV6, finding_network -> remoteIpDetails.ipAddressV6), // The IPv6 remote address of the connection. + service_action_network_connection_remote_port = coalesce(finding_network -> RemotePortDetails.Port, finding_network -> remotePortDetails.port), // The port number of the remote connection. + service_action_network_connection_remote_port_name = coalesce(finding_network -> RemotePortDetails.PortName, finding_network -> remotePortDetails.portName), // The port name of the remote connection. + + // extract service aws api call action (AWS_API_CALL) fields: https://docs.aws.amazon.com/guardduty/latest/APIReference/API_AwsApiCallAction.html + service_action_api_call_user_agent = coalesce( // The agent through which the API request was made. + finding_aws_api -> UserAgent, finding_aws_api -> userAgent, + to_json_string(finding_service -> AdditionalInfo.Value) -> userAgent.fullUserAgent, + to_json_string(finding_service -> additionalInfo.value) -> userAgent.fullUserAgent), + service_action_api_call_remote_ipv4 = coalesce(finding_aws_api -> RemoteIpDetails.IpAddressV4, finding_aws_api -> remoteIpDetails.ipAddressV4), // The IPv4 remote address of the connection. + service_action_api_call_remote_ipv6 = coalesce(finding_aws_api -> RemoteIpDetails.IpAddressV6, finding_aws_api -> remoteIpDetails.ipAddressV6), // The IPv6 remote address of the connection. + service_action_api_call_remote_asn = coalesce(finding_aws_api -> RemoteIpDetails.Organization.Asn, finding_aws_api -> remoteIpDetails.organization.asn), // The Autonomous System Number (ASN) of the internet provider of the remote IP address. + service_action_api_call_remote_asn_org = coalesce(finding_aws_api -> RemoteIpDetails.Organization.AsnOrg, finding_aws_api -> remoteIpDetails.organization.asnOrg), // The organization that registered this ASN. + service_action_api_call_remote_isp = coalesce(finding_aws_api -> RemoteIpDetails.Organization.Isp, finding_aws_api -> remoteIpDetails.organization.isp), // The ISP information for the internet provider. + service_action_api_call_remote_isp_org = coalesce(finding_aws_api -> RemoteIpDetails.Organization.Org, finding_aws_api -> remoteIpDetails.organization.org), // The name of the internet provider. + service_action_api_call_remote_account_id = coalesce(finding_aws_api -> RemoteAccountDetails.AccountId, finding_aws_api -> remoteAccountDetails.accountId), // The AWS account ID of the remote API caller. + service_action_api_call_error_code = coalesce(finding_aws_api -> ErrorCode, finding_aws_api -> errorCode), // The error code of the failed AWS API action. + service_action_api_call_api = coalesce(finding_aws_api -> api, finding_aws_api -> Api), + + // extract service dns request action (DNS_REQUEST) fields: https://docs.aws.amazon.com/guardduty/latest/APIReference/API_DnsRequestAction.html + service_action_dns_request_action_domain = coalesce(finding_service -> Action.DnsRequestAction.Domain, service -> action.dnsRequestAction.domain), // The domain information for the DNS query. + service_action_dns_request_action_protocol = coalesce(finding_service -> Action.DnsRequestAction.Protocol, service -> action.dnsRequestAction.protocol), // The network connection protocol observed in the activity that prompted GuardDuty to generate the finding. + service_action_dns_request_action_blocked = to_boolean(coalesce(finding_service -> Action.DnsRequestAction.Blocked, service -> action.dnsRequestAction.blocked)), // Indicates whether the targeted port is blocked. + + // extract service k8s api call action (KUBERNETES_API_CALL) fields: https://docs.aws.amazon.com/guardduty/latest/APIReference/API_KubernetesApiCallAction.html + service_action_k8s_api_call_verb = coalesce(finding_k8s_api -> Verb, finding_k8s_api -> verb), // The Kubernetes API request HTTP verb. + service_action_k8s_api_call_request_uri = coalesce(finding_k8s_api -> RequestUri, finding_k8s_api -> requestUri), // The Kubernetes API request URI. + service_action_k8s_api_call_status_code = coalesce(finding_k8s_api -> StatusCode, finding_k8s_api -> statusCode), // The resulting HTTP response code of the Kubernetes API call action. + service_action_k8s_api_call_user_agent = coalesce(finding_k8s_api -> UserAgent, finding_k8s_api -> userAgent), // The user agent of the caller of the Kubernetes API. + service_action_k8s_api_call_resource = coalesce(finding_k8s_api -> Resource, finding_k8s_api -> resource), // The resource component in the Kubernetes API call action. + service_action_k8s_api_call_resource_name = coalesce(finding_k8s_api -> ResourceName, finding_k8s_api -> resourceName), // The name of the resource in the Kubernetes API call action. + service_action_k8s_api_call_sourceIPs = coalesce(finding_k8s_api -> SourceIPs[], finding_k8s_api -> sourceIPs[]), // The IP of the Kubernetes API caller and the IPs of any proxies or load balancers between the caller and the API endpoint. + service_action_k8s_api_call_remote_ipv4 = coalesce(finding_k8s_api -> RemoteIpDetails.IpAddressV4, finding_k8s_api -> remoteIpDetails.ipAddressV4), //The IPv4 remote address of the connection. + service_action_k8s_api_call_remote_ipv6 = coalesce(finding_k8s_api -> RemoteIpDetails.IpAddressV6, finding_k8s_api -> remoteIpDetails.ipAddressV6), //The IPv6 remote address of the connection. + service_action_k8s_api_call_remote_asn = coalesce(finding_k8s_api -> RemoteIpDetails.Organization.Asn, finding_k8s_api -> remoteIpDetails.organization.asn), // The Autonomous System Number (ASN) of the internet provider of the remote IP address. + service_action_k8s_api_call_remote_asn_org = coalesce(finding_k8s_api -> RemoteIpDetails.Organization.AsnOrg, finding_k8s_api -> remoteIpDetails.organization.asnOrg), // The organization that registered this ASN. + service_action_k8s_api_call_remote_isp = coalesce(finding_k8s_api -> RemoteIpDetails.Organization.Isp, finding_k8s_api -> remoteIpDetails.organization.isp), // The ISP information for the internet provider. + service_action_k8s_api_call_remote_isp_org = coalesce(finding_k8s_api -> RemoteIpDetails.Organization.Org, finding_k8s_api -> remoteIpDetails.organization.org), // The name of the internet provider. + + // extract service port probe action (PORT_PROBE) fields: https://docs.aws.amazon.com/guardduty/latest/APIReference/API_PortProbeAction.html + service_action_port_probing_remote_ipv4_addresses = coalesce( // The IPv4 remote address of the connection. + arraymap(finding_service -> Action.PortProbeAction.PortProbeDetails[], "@element" -> RemoteIpDetails.IpAddressV4), + arraymap(service -> action.portProbeAction.portProbeDetails[], "@element" -> remoteIpDetails.ipAddressV4)), + service_action_port_probing_remote_ipv6_addresses = coalesce( // The IPv6 remote address of the connection. + arraymap(finding_service -> Action.PortProbeAction.PortProbeDetails[], "@element" -> RemoteIpDetails.IpAddressV6), + arraymap(service -> action.portProbeAction.portProbeDetails[], "@element" -> remoteIpDetails.ipAddressV6)), + + // extract service k8s role fields: https://docs.aws.amazon.com/guardduty/latest/APIReference/API_KubernetesRoleDetails.html + service_action_k8s_role_name = coalesce(finding_service -> Action.KubernetesRoleDetails.Name, service -> action.kubernetesRoleDetails.name), // The name of the Kubernetes role. + + // extract service k8s role binding fields: https://docs.aws.amazon.com/guardduty/latest/APIReference/API_KubernetesRoleBindingDetails.html + service_action_k8s_role_binding_name = coalesce(finding_service -> Action.KubernetesRoleBindingDetails.Name, service -> action.kubernetesRoleBindingDetails.name), // The name of the RoleBinding. + + // extract service rds login attempt action fields: https://docs.aws.amazon.com/guardduty/latest/APIReference/API_RdsLoginAttemptAction.html + service_action_rds_login_attempt_users = coalesce( // Indicates the user name which attempted to log in. + arraystring(arraymap(finding_rds_login -> LoginAttribute[], "@element" -> User), ","), + arraystring(arraymap(finding_rds_login -> loginAttribute[], "@element" -> user), ",")), + service_action_rds_login_attempt_applications = coalesce( // Indicates the application name used to attempt log in. + arraystring(arraymap(finding_rds_login -> LoginAttribute[], "@element" -> Application), ","), + arraystring(arraymap(finding_rds_login -> loginAttribute[], "@element" -> application), ",")), + service_action_rds_login_attempt_remote_ipv4 = coalesce(finding_rds_login -> RemoteIpDetails.IpAddressV4, finding_rds_login -> remoteIpDetails.ipAddressV4), // The IPv4 remote address of the connection. + service_action_rds_login_attempt_remote_ipv6 = coalesce(finding_rds_login -> RemoteIpDetails.IpAddressV6, finding_rds_login -> remoteIpDetails.ipAddressV6), // The IPv6 remote address of the connection. + service_action_rds_login_attempt_remote_asn = coalesce(finding_rds_login -> RemoteIpDetails.Organization.Asn, finding_rds_login -> remoteIpDetails.organization.asn), // The Autonomous System Number (ASN) of the internet provider of the remote IP address. + service_action_rds_login_attempt_remote_asn_org = coalesce(finding_rds_login -> RemoteIpDetails.Organization.AsnOrg, finding_rds_login -> remoteIpDetails.organization.asnOrg), // The organization that registered this ASN. + service_action_rds_login_attempt_remote_isp = coalesce(finding_rds_login -> RemoteIpDetails.Organization.Isp, finding_rds_login -> remoteIpDetails.organization.isp), // The ISP information for the internet provider. + service_action_rds_login_attempt_remote_isp_org = coalesce(finding_rds_login -> RemoteIpDetails.Organization.Org, finding_rds_login -> remoteIpDetails.organization.org) // The name of the internet provider. + +| alter // post extraction processing + is_connection_inbound = if(service_action_network_connection_direction != null and service_action_network_connection_direction = "INBOUND"), + is_connection_outbound = if(service_action_network_connection_direction != null and service_action_network_connection_direction = "OUTBOUND"), + http_code = to_integer(service_action_k8s_api_call_status_code), + http_verb = service_action_k8s_api_call_verb, + network_protocol = coalesce(service_action_network_connection_protocol, service_action_dns_request_action_protocol), + k8s_api_call_src_ipv4_addresses = arrayfilter(service_action_k8s_api_call_sourceIPs, "@element" ~= "(?:\d{1,3}\.){3}\d{1,3}"), + k8s_api_call_src_ipv6_addresses = arrayfilter(service_action_k8s_api_call_sourceIPs, "@element" ~= "(?:[a-fA-F\d]{0,4}\:){1,7}[a-fA-F\d]{0,4}") +| alter + network_application_protocol = if(is_connection_inbound, service_action_network_connection_local_port_name, is_connection_outbound, service_action_network_connection_remote_port_name), + source_ipv4 = if(is_connection_inbound, service_action_network_connection_remote_ipv4, is_connection_outbound, service_action_network_connection_local_ipv4, coalesce(service_action_api_call_remote_ipv4, service_action_k8s_api_call_remote_ipv4, service_action_rds_login_attempt_remote_ipv4)), + source_ipv6 = if(is_connection_inbound, service_action_network_connection_remote_ipv6, is_connection_outbound, service_action_network_connection_local_ipv6, coalesce(service_action_api_call_remote_ipv6, service_action_k8s_api_call_remote_ipv6, service_action_rds_login_attempt_remote_ipv6)), + target_ipv4 = if(is_connection_inbound, service_action_network_connection_local_ipv4, is_connection_outbound, service_action_network_connection_remote_ipv4), + target_ipv6 = if(is_connection_inbound, service_action_network_connection_local_ipv6, is_connection_outbound, service_action_network_connection_remote_ipv6) +| alter + source_ipv4_addresses = if(array_length(k8s_api_call_src_ipv4_addresses) > 0, k8s_api_call_src_ipv4_addresses, array_length(service_action_port_probing_remote_ipv4_addresses) > 0, service_action_port_probing_remote_ipv4_addresses), + source_ipv6_addresses = if(array_length(k8s_api_call_src_ipv6_addresses) > 0, k8s_api_call_src_ipv6_addresses, array_length(service_action_port_probing_remote_ipv6_addresses) > 0, service_action_port_probing_remote_ipv6_addresses) +| alter + all_source_ipv4_addresses = if(array_length(source_ipv4_addresses) > 0 and source_ipv4 != null, arrayconcat(arraycreate(source_ipv4), source_ipv4_addresses), source_ipv4 != null, arraycreate(source_ipv4), source_ipv4_addresses), + all_source_ipv6_addresses = if(array_length(source_ipv6_addresses) > 0 and source_ipv6 != null, arrayconcat(arraycreate(source_ipv6), source_ipv6_addresses), source_ipv6 != null, arraycreate(source_ipv6), source_ipv6_addresses) + +| alter // XDM mappings + xdm.alert.description = finding_description, + xdm.alert.name = finding_title, + xdm.alert.original_alert_id = finding_id, + xdm.alert.risks = if(array_length(service_evidence_threat_intelligence_details) > 0, service_evidence_threat_intelligence_details), + xdm.alert.severity = if( // https://docs.aws.amazon.com/guardduty/latest/ug/guardduty_findings.html#guardduty_findings-severity + finding_severity >= 9, "Critical", + finding_severity >= 7 and finding_severity <= 8.9 , "High", + finding_severity >= 4 and finding_severity <= 6.9 , "Medium", + finding_severity >= 1 and finding_severity <= 3.9 , "Low", + to_string(finding_severity)), + xdm.alert.subcategory = finding_type, + xdm.auth.auth_method = resource_rdsDbUserDetails_auth_method, + xdm.event.operation = service_action_type, + xdm.event.operation_sub_type = service_action_api_call_api, + xdm.event.outcome = if(service_action_network_connection_is_blocked or service_action_dns_request_action_blocked or service_action_api_call_error_code != null, XDM_CONST.OUTCOME_FAILED), + xdm.event.outcome_reason = service_action_api_call_error_code, + xdm.network.application_protocol = if(network_application_protocol != "Unknown", network_application_protocol), + xdm.network.dns.dns_question.name = service_action_dns_request_action_domain, + xdm.network.http.method = if(http_verb = null, http_verb, http_verb = "GET", XDM_CONST.HTTP_METHOD_GET, http_verb = "POST", XDM_CONST.HTTP_METHOD_POST, http_verb = "PUT", XDM_CONST.HTTP_METHOD_PUT, http_verb = "PATCH", XDM_CONST.HTTP_METHOD_PATCH, http_verb = "OPTIONS", XDM_CONST.HTTP_METHOD_OPTIONS, http_verb = "HEAD", XDM_CONST.HTTP_METHOD_HEAD, http_verb = "ACL", XDM_CONST.HTTP_METHOD_ACL, http_verb = "BASELINE_CONTROL", XDM_CONST.HTTP_METHOD_BASELINE_CONTROL, http_verb = "BIND", XDM_CONST.HTTP_METHOD_BIND, http_verb = "CHECKIN", XDM_CONST.HTTP_METHOD_CHECKIN, http_verb = "CHECKOUT", XDM_CONST.HTTP_METHOD_CHECKOUT, http_verb = "CONNECT", XDM_CONST.HTTP_METHOD_CONNECT, http_verb = "COPY", XDM_CONST.HTTP_METHOD_COPY, http_verb = "DELETE", XDM_CONST.HTTP_METHOD_DELETE, http_verb = "LABEL", XDM_CONST.HTTP_METHOD_LABEL, http_verb = "LINK", XDM_CONST.HTTP_METHOD_LINK, http_verb = "LOCK", XDM_CONST.HTTP_METHOD_LOCK, http_verb = "MERGE", XDM_CONST.HTTP_METHOD_MERGE, http_verb = "MKACTIVITY", XDM_CONST.HTTP_METHOD_MKACTIVITY, http_verb = "MKCALENDAR", XDM_CONST.HTTP_METHOD_MKCALENDAR, http_verb = "MKCOL", XDM_CONST.HTTP_METHOD_MKCOL, http_verb = "MKREDIRECTREF", XDM_CONST.HTTP_METHOD_MKREDIRECTREF, http_verb = "MKWORKSPACE", XDM_CONST.HTTP_METHOD_MKWORKSPACE, http_verb = "MOVE", XDM_CONST.HTTP_METHOD_MOVE, http_verb = "ORDERPATCH", XDM_CONST.HTTP_METHOD_ORDERPATCH, http_verb = "PRI", XDM_CONST.HTTP_METHOD_PRI, http_verb = "PROPFIND", XDM_CONST.HTTP_METHOD_PROPFIND, http_verb = "PROPPATCH", XDM_CONST.HTTP_METHOD_PROPPATCH, http_verb = "REBIND", XDM_CONST.HTTP_METHOD_REBIND, http_verb = "REPORT", XDM_CONST.HTTP_METHOD_REPORT, http_verb = "SEARCH", XDM_CONST.HTTP_METHOD_SEARCH, http_verb = "TRACE", XDM_CONST.HTTP_METHOD_TRACE, http_verb = "UNBIND", XDM_CONST.HTTP_METHOD_UNBIND, http_verb = "UNCHECKOUT", XDM_CONST.HTTP_METHOD_UNCHECKOUT, http_verb = "UNLINK", XDM_CONST.HTTP_METHOD_UNLINK, http_verb = "UNLOCK", XDM_CONST.HTTP_METHOD_UNLOCK, http_verb = "UPDATE", XDM_CONST.HTTP_METHOD_UPDATE, http_verb = "UPDATEREDIRECTREF", XDM_CONST.HTTP_METHOD_UPDATEREDIRECTREF, http_verb = "VERSION_CONTROL", XDM_CONST.HTTP_METHOD_VERSION_CONTROL, uppercase(http_verb)), + xdm.network.http.response_code = if(http_code = null, null, http_code = 200, XDM_CONST.HTTP_RSP_CODE_OK, http_code = 201, XDM_CONST.HTTP_RSP_CODE_CREATED, http_code = 302, XDM_CONST.HTTP_RSP_CODE_FOUND, http_code = 401, XDM_CONST.HTTP_RSP_CODE_UNAUTHORIZED, http_code = 403, XDM_CONST.HTTP_RSP_CODE_FORBIDDEN, http_code = 404, XDM_CONST.HTTP_RSP_CODE_NOT_FOUND, http_code = 500, XDM_CONST.HTTP_RSP_CODE_INTERNAL_SERVER_ERROR, http_code = 501, XDM_CONST.HTTP_RSP_CODE_NOT_IMPLEMENTED, http_code = 502, XDM_CONST.HTTP_RSP_CODE_BAD_GATEWAY, http_code = 503, XDM_CONST.HTTP_RSP_CODE_SERVICE_UNAVAILABLE, http_code = 504, XDM_CONST.HTTP_RSP_CODE_GATEWAY_TIMEOUT, http_code = 505, XDM_CONST.HTTP_RSP_CODE_HTTP_VERSION_NOT_SUPPORTED, http_code = 506, XDM_CONST.HTTP_RSP_CODE_VARIANT_ALSO_NEGOTIATES, http_code = 507, XDM_CONST.HTTP_RSP_CODE_INSUFFICIENT_STORAGE, http_code = 508, XDM_CONST.HTTP_RSP_CODE_LOOP_DETECTED, http_code = 511, XDM_CONST.HTTP_RSP_CODE_NETWORK_AUTHENTICATION_REQUIRED, http_code = 100, XDM_CONST.HTTP_RSP_CODE_CONTINUE, http_code = 101, XDM_CONST.HTTP_RSP_CODE_SWITCHING_PROTOCOLS, http_code = 102, XDM_CONST.HTTP_RSP_CODE_PROCESSING, http_code = 103, XDM_CONST.HTTP_RSP_CODE_EARLY_HINTS, http_code = 202, XDM_CONST.HTTP_RSP_CODE_ACCEPTED, http_code = 203, XDM_CONST.HTTP_RSP_CODE_NON__AUTHORITATIVE_INFORMATION, http_code = 204, XDM_CONST.HTTP_RSP_CODE_NO_CONTENT, http_code = 205, XDM_CONST.HTTP_RSP_CODE_RESET_CONTENT, http_code = 206, XDM_CONST.HTTP_RSP_CODE_PARTIAL_CONTENT, http_code = 207, XDM_CONST.HTTP_RSP_CODE_MULTI__STATUS, http_code = 208, XDM_CONST.HTTP_RSP_CODE_ALREADY_REPORTED, http_code = 226, XDM_CONST.HTTP_RSP_CODE_IM_USED, http_code = 300, XDM_CONST.HTTP_RSP_CODE_MULTIPLE_CHOICES, http_code = 301, XDM_CONST.HTTP_RSP_CODE_MOVED_PERMANENTLY, http_code = 303, XDM_CONST.HTTP_RSP_CODE_SEE_OTHER, http_code = 304, XDM_CONST.HTTP_RSP_CODE_NOT_MODIFIED, http_code = 305, XDM_CONST.HTTP_RSP_CODE_USE_PROXY, http_code = 307, XDM_CONST.HTTP_RSP_CODE_TEMPORARY_REDIRECT, http_code = 308, XDM_CONST.HTTP_RSP_CODE_PERMANENT_REDIRECT, http_code = 400, XDM_CONST.HTTP_RSP_CODE_BAD_REQUEST, http_code = 402, XDM_CONST.HTTP_RSP_CODE_PAYMENT_REQUIRED, http_code = 405, XDM_CONST.HTTP_RSP_CODE_METHOD_NOT_ALLOWED, http_code = 406, XDM_CONST.HTTP_RSP_CODE_NOT_ACCEPTABLE, http_code = 407, XDM_CONST.HTTP_RSP_CODE_PROXY_AUTHENTICATION_REQUIRED, http_code = 408, XDM_CONST.HTTP_RSP_CODE_REQUEST_TIMEOUT, http_code = 409, XDM_CONST.HTTP_RSP_CODE_CONFLICT, http_code = 410, XDM_CONST.HTTP_RSP_CODE_GONE, http_code = 411, XDM_CONST.HTTP_RSP_CODE_LENGTH_REQUIRED, http_code = 412, XDM_CONST.HTTP_RSP_CODE_PRECONDITION_FAILED, http_code = 413, XDM_CONST.HTTP_RSP_CODE_CONTENT_TOO_LARGE, http_code = 414, XDM_CONST.HTTP_RSP_CODE_URI_TOO_LONG, http_code = 415, XDM_CONST.HTTP_RSP_CODE_UNSUPPORTED_MEDIA_TYPE, http_code = 416, XDM_CONST.HTTP_RSP_CODE_RANGE_NOT_SATISFIABLE, http_code = 417, XDM_CONST.HTTP_RSP_CODE_EXPECTATION_FAILED, http_code = 421, XDM_CONST.HTTP_RSP_CODE_MISDIRECTED_REQUEST, http_code = 422, XDM_CONST.HTTP_RSP_CODE_UNPROCESSABLE_CONTENT, http_code = 423, XDM_CONST.HTTP_RSP_CODE_LOCKED, http_code = 424, XDM_CONST.HTTP_RSP_CODE_FAILED_DEPENDENCY, http_code = 425, XDM_CONST.HTTP_RSP_CODE_TOO_EARLY, http_code = 426, XDM_CONST.HTTP_RSP_CODE_UPGRADE_REQUIRED, http_code = 428, XDM_CONST.HTTP_RSP_CODE_PRECONDITION_REQUIRED, http_code = 429, XDM_CONST.HTTP_RSP_CODE_TOO_MANY_REQUESTS, http_code = 431, XDM_CONST.HTTP_RSP_CODE_REQUEST_HEADER_FIELDS_TOO_LARGE, http_code = 451, XDM_CONST.HTTP_RSP_CODE_UNAVAILABLE_FOR_LEGAL_REASONS, service_action_k8s_api_call_status_code), + xdm.network.http.url = service_action_k8s_api_call_request_uri, + xdm.network.ip_protocol = if(network_protocol = "UDP", XDM_CONST.IP_PROTOCOL_UDP, network_protocol = "TCP", XDM_CONST.IP_PROTOCOL_TCP), + xdm.observer.unique_identifier = service_detector_id, + xdm.source.application.name = if(resource_rdsDbUserDetails_application != null or service_action_rds_login_attempt_applications != null, arraystring(arraycreate(resource_rdsDbUserDetails_application, service_action_rds_login_attempt_applications), ",")), + xdm.source.asn.as_name = if(is_connection_inbound, service_action_network_connection_remote_asn_org, coalesce(service_action_api_call_remote_asn_org, service_action_k8s_api_call_remote_asn_org, service_action_rds_login_attempt_remote_asn_org)), + xdm.source.asn.as_number = to_integer(if(is_connection_inbound, service_action_network_connection_remote_asn, coalesce(service_action_api_call_remote_asn, service_action_k8s_api_call_remote_asn, service_action_rds_login_attempt_remote_asn))), + xdm.source.asn.isp = if(is_connection_inbound, coalesce(service_action_network_connection_remote_asn_isp, service_action_network_connection_remote_asn_isp_org), coalesce(service_action_api_call_remote_isp, service_action_api_call_remote_isp_org, service_action_k8s_api_call_remote_isp, service_action_k8s_api_call_remote_isp_org, service_action_rds_login_attempt_remote_isp, service_action_rds_login_attempt_remote_isp_org)), + xdm.source.cloud.project_id = coalesce(service_action_api_call_remote_account_id, finding_account_id), + xdm.source.cloud.provider = XDM_CONST.CLOUD_PROVIDER_AWS, + xdm.source.cloud.region = finding_region, + xdm.source.cloud.zone = resource_availability_zone, + xdm.source.host.device_model = resource_instance_type, + xdm.source.host.ipv4_addresses = if(array_length(all_source_ipv4_addresses) > 0, all_source_ipv4_addresses), + xdm.source.host.ipv6_addresses = if(array_length(all_source_ipv6_addresses) > 0, all_source_ipv6_addresses), + xdm.source.host.ipv4_public_addresses = arrayfilter(all_source_ipv4_addresses, not incidr("@element", "10.0.0.0/8") and not incidr("@element", "172.16.0.0/12") and not incidr("@element", "192.168.0.0/16") and not incidr("@element", "127.0.0.0/8") and not incidr("@element", "169.254.0.0/16") and not incidr("@element", "100.64.0.0/10")), + xdm.source.host.hostname = resource_instance_id, + xdm.source.ipv4 = coalesce(source_ipv4, arrayindex(all_source_ipv4_addresses, 0)), + xdm.source.ipv6 = coalesce(source_ipv6, arrayindex(all_source_ipv6_addresses, 0)), + xdm.source.port = to_integer(if(is_connection_inbound, service_action_network_connection_remote_port, is_connection_outbound, service_action_network_connection_local_port)), + xdm.source.process.executable.path = service_runtime_details_process_executable_path, + xdm.source.process.executable.sha256 = service_runtime_details_process_executable_sha256, + xdm.source.process.identifier = service_runtime_details_process_uuid, + xdm.source.process.name = service_runtime_details_process_name, + xdm.source.process.parent_id = service_runtime_details_process_parent_uuid, + xdm.source.user_agent = coalesce(service_action_api_call_user_agent, service_action_k8s_api_call_user_agent), + xdm.source.user.groups = if(service_action_k8s_role_name != null, arraycreate(service_action_k8s_role_name), resource_user_groups), + xdm.source.identity.groups = if(service_action_k8s_role_name != null, arraycreate(service_action_k8s_role_name), resource_user_groups), + xdm.source.user.identifier = if(service_runtime_details_process_user_id != null or resource_user_id != null, arraystring(arraycreate(resource_user_id, service_runtime_details_process_user_id), ",")), + xdm.source.identity.identifier = if(service_runtime_details_process_user_id != null or resource_user_id != null, arraystring(arraycreate(resource_user_id, service_runtime_details_process_user_id), ",")), + xdm.source.user.user_type = resource_user_type, + xdm.source.identity.user_type = resource_user_type, + xdm.source.user.username = trim(coalesce(service_action_rds_login_attempt_users, service_runtime_details_process_user, resource_username), ","), + xdm.source.identity.username = trim(coalesce(service_action_rds_login_attempt_users, service_runtime_details_process_user, resource_username), ","), + xdm.target.asn.as_name = if(is_connection_outbound, service_action_api_call_remote_asn_org), + xdm.target.asn.as_number = to_integer(if(is_connection_outbound, service_action_api_call_remote_asn)), + xdm.target.asn.isp = if(is_connection_outbound, coalesce(service_action_api_call_remote_isp, service_action_api_call_remote_isp_org)), + xdm.target.cloud.project_id = finding_account_id, + xdm.target.cloud.provider = XDM_CONST.CLOUD_PROVIDER_AWS, + xdm.target.cloud.region = finding_region, + xdm.target.cloud.zone = resource_availability_zone, + xdm.target.host.ipv4_addresses = if(target_ipv4 != null, arraycreate(target_ipv4)), + xdm.target.host.ipv6_addresses = if(target_ipv6 != null, arraycreate(target_ipv6)), + xdm.target.host.ipv4_public_addresses = if(target_ipv4 != null and not incidr(target_ipv4, "10.0.0.0/8") and not incidr(target_ipv4, "172.16.0.0/12") and not incidr(target_ipv4, "192.168.0.0/16") and not incidr(target_ipv4, "127.0.0.0/8") and not incidr(target_ipv4, "169.254.0.0/16") and not incidr(target_ipv4, "100.64.0.0/10"), arraycreate(target_ipv4)), + xdm.target.ipv4 = target_ipv4, + xdm.target.ipv6 = target_ipv6, + xdm.target.port = to_integer(if(is_connection_inbound, service_action_network_connection_local_port, is_connection_outbound, service_action_network_connection_remote_port)), + xdm.target.resource.id = finding_arn, + xdm.target.resource.name = coalesce(resource_name, service_action_k8s_api_call_resource_name, service_action_k8s_role_binding_name), + xdm.target.resource.type = resource_type, + xdm.target.resource.value = service_action_k8s_api_call_resource; \ No newline at end of file diff --git a/Packs/AWS-GuardDuty/ModelingRules/AWSGuardDutyModelingRules_2_11/AWSGuardDutyModelingRules_2_11.yml b/Packs/AWS-GuardDuty/ModelingRules/AWSGuardDutyModelingRules_2_11/AWSGuardDutyModelingRules_2_11.yml new file mode 100644 index 000000000000..05404f1055a8 --- /dev/null +++ b/Packs/AWS-GuardDuty/ModelingRules/AWSGuardDutyModelingRules_2_11/AWSGuardDutyModelingRules_2_11.yml @@ -0,0 +1,8 @@ +fromversion: 8.15.0 +id: AWSGuardDuty_ModelingRule +name: AWSGuardDuty Modeling Rule +rules: '' +schema: '' +tags: AWSGuardDuty +supportedModules: +- xsiam diff --git a/Packs/AWS-GuardDuty/ModelingRules/AWSGuardDutyModelingRules_2_11/AWSGuardDutyModelingRules_2_11_schema.json b/Packs/AWS-GuardDuty/ModelingRules/AWSGuardDutyModelingRules_2_11/AWSGuardDutyModelingRules_2_11_schema.json new file mode 100644 index 000000000000..c4cc51b7a254 --- /dev/null +++ b/Packs/AWS-GuardDuty/ModelingRules/AWSGuardDutyModelingRules_2_11/AWSGuardDutyModelingRules_2_11_schema.json @@ -0,0 +1,44 @@ +{ + "aws_guardduty_raw": { + "accountId": { + "type": "string", + "is_array": false + }, + "arn": { + "type": "string", + "is_array": false + }, + "description": { + "type": "string", + "is_array": false + }, + "id": { + "type": "string", + "is_array": false + }, + "region": { + "type": "string", + "is_array": false + }, + "resource": { + "type": "string", + "is_array": false + }, + "service": { + "type": "string", + "is_array": false + }, + "severity": { + "type": "float", + "is_array": false + }, + "title": { + "type": "string", + "is_array": false + }, + "type": { + "type": "string", + "is_array": false + } + } +} \ No newline at end of file diff --git a/Packs/AWS-GuardDuty/ReleaseNotes/1_4_1.md b/Packs/AWS-GuardDuty/ReleaseNotes/1_4_1.md new file mode 100644 index 000000000000..3ca05f6a3620 --- /dev/null +++ b/Packs/AWS-GuardDuty/ReleaseNotes/1_4_1.md @@ -0,0 +1,7 @@ + +#### Modeling Rules + +##### New: AWSGuardDuty Modeling Rule + +- New: Added a new modeling rule version that maps the `xdm.*.identity.*` fields alongside the existing `xdm.*.user.*` fields. +<~XSIAM> (Available from Cortex XSIAM 2.11). diff --git a/Packs/AWS-GuardDuty/ReleaseNotes/1_4_2.md b/Packs/AWS-GuardDuty/ReleaseNotes/1_4_2.md new file mode 100644 index 000000000000..5dccc0780898 --- /dev/null +++ b/Packs/AWS-GuardDuty/ReleaseNotes/1_4_2.md @@ -0,0 +1,8 @@ + +#### Integrations + +##### AWS - GuardDuty Event Collector + +- Fixed an issue where findings sharing the same second could be re-fetched every cycle, causing duplicate events. +- Fixed an issue where fetching could stall on a detector when the only findings returned had already been collected, preventing newer findings from being fetched. +- Added debug logging to simplify troubleshooting. diff --git a/Packs/AWS-GuardDuty/ReleaseNotes/1_5_0.md b/Packs/AWS-GuardDuty/ReleaseNotes/1_5_0.md new file mode 100644 index 000000000000..45fcfc1ecd61 --- /dev/null +++ b/Packs/AWS-GuardDuty/ReleaseNotes/1_5_0.md @@ -0,0 +1,10 @@ + +#### Integrations + +##### AWS - GuardDuty + +- Added the ***aws-gd-create-threat-entity-set*** command to create a GuardDuty Threat Entity Set. +- Added the ***aws-gd-update-threat-entity-set*** command to update a GuardDuty Threat Entity Set. +- Added the ***aws-gd-get-threat-entity-set*** command to retrieve a GuardDuty Threat Entity Set. +- Added the ***aws-gd-list-threat-entity-sets*** command to list the GuardDuty Threat Entity Sets of a detector. +- Added the ***aws-gd-delete-threat-entity-set*** command to delete a GuardDuty Threat Entity Set. diff --git a/Packs/AWS-GuardDuty/ReleaseNotes/1_5_1.md b/Packs/AWS-GuardDuty/ReleaseNotes/1_5_1.md new file mode 100644 index 000000000000..c3a53d1699a4 --- /dev/null +++ b/Packs/AWS-GuardDuty/ReleaseNotes/1_5_1.md @@ -0,0 +1,11 @@ + +#### Integrations + +##### AWS - GuardDuty + +- Updated the Docker image to: *demisto/boto3py3:1.0.0.12887184*. + +##### AWS - GuardDuty Event Collector + +- Updated the Docker image to: *demisto/boto3py3:1.0.0.12887184*. + diff --git a/Packs/AWS-GuardDuty/pack_metadata.json b/Packs/AWS-GuardDuty/pack_metadata.json index 202384ded98a..98ca8cdb9035 100644 --- a/Packs/AWS-GuardDuty/pack_metadata.json +++ b/Packs/AWS-GuardDuty/pack_metadata.json @@ -2,7 +2,7 @@ "name": "AWS - GuardDuty", "description": "Amazon Web Services Guard Duty Service (gd)", "support": "xsoar", - "currentVersion": "1.4.0", + "currentVersion": "1.5.1", "author": "Cortex XSOAR", "url": "https://www.paloaltonetworks.com/cortex", "email": "", diff --git a/Packs/AWS-IAM/Integrations/AWS-IAM/AWS-IAM.yml b/Packs/AWS-IAM/Integrations/AWS-IAM/AWS-IAM.yml index 6e4312bd710f..c5cf6226077d 100644 --- a/Packs/AWS-IAM/Integrations/AWS-IAM/AWS-IAM.yml +++ b/Packs/AWS-IAM/Integrations/AWS-IAM/AWS-IAM.yml @@ -1754,7 +1754,7 @@ script: required: false description: Deletes a virtual MFA device. name: aws-iam-delete-mfa-devices - dockerimage: demisto/boto3py3:1.0.0.10221838 + dockerimage: demisto/boto3py3:1.0.0.12887184 runonce: false script: '-' subtype: python3 diff --git a/Packs/AWS-IAM/ReleaseNotes/1_1_88.md b/Packs/AWS-IAM/ReleaseNotes/1_1_88.md new file mode 100644 index 000000000000..ec78fceca5e5 --- /dev/null +++ b/Packs/AWS-IAM/ReleaseNotes/1_1_88.md @@ -0,0 +1,7 @@ + +#### Integrations + +##### AWS - Identity and Access Management + +- Updated the Docker image to: *demisto/boto3py3:1.0.0.12887184*. + diff --git a/Packs/AWS-IAM/ReleaseNotes/1_1_89.md b/Packs/AWS-IAM/ReleaseNotes/1_1_89.md new file mode 100644 index 000000000000..2579c4c26952 --- /dev/null +++ b/Packs/AWS-IAM/ReleaseNotes/1_1_89.md @@ -0,0 +1,3 @@ +## AWS - IAM + +- Locked dependencies of the pack to ensure stability for versioned core packs. No changes in this release. diff --git a/Packs/AWS-IAM/pack_metadata.json b/Packs/AWS-IAM/pack_metadata.json index 36dfce871731..53f67e8564ef 100644 --- a/Packs/AWS-IAM/pack_metadata.json +++ b/Packs/AWS-IAM/pack_metadata.json @@ -3,7 +3,7 @@ "description": "Amazon Web Services Identity and Access Management (IAM)", "support": "xsoar", "author": "Cortex XSOAR", - "currentVersion": "1.1.87", + "currentVersion": "1.1.89", "url": "https://www.paloaltonetworks.com/cortex", "email": "", "created": "2020-04-14T00:00:00Z", diff --git a/Packs/AWS-IAMIdentityCenter/Integrations/AWSIAMIdentityCenter/AWSIAMIdentityCenter.yml b/Packs/AWS-IAMIdentityCenter/Integrations/AWSIAMIdentityCenter/AWSIAMIdentityCenter.yml index c58cd6b77ec8..837ab672e07c 100644 --- a/Packs/AWS-IAMIdentityCenter/Integrations/AWSIAMIdentityCenter/AWSIAMIdentityCenter.yml +++ b/Packs/AWS-IAMIdentityCenter/Integrations/AWSIAMIdentityCenter/AWSIAMIdentityCenter.yml @@ -1174,7 +1174,7 @@ script: required: true description: Updates an IAM Identity Center group for your AWS account. name: aws-iam-identitycenter-update-group - dockerimage: demisto/boto3py3:1.0.0.10221838 + dockerimage: demisto/boto3py3:1.0.0.12887184 runonce: false script: '' subtype: python3 diff --git a/Packs/AWS-IAMIdentityCenter/ReleaseNotes/1_0_15.md b/Packs/AWS-IAMIdentityCenter/ReleaseNotes/1_0_15.md new file mode 100644 index 000000000000..fd16c8aa6bab --- /dev/null +++ b/Packs/AWS-IAMIdentityCenter/ReleaseNotes/1_0_15.md @@ -0,0 +1,7 @@ + +#### Integrations + +##### AWS - IAM Identity Center + +- Updated the Docker image to: *demisto/boto3py3:1.0.0.12887184*. + diff --git a/Packs/AWS-IAMIdentityCenter/pack_metadata.json b/Packs/AWS-IAMIdentityCenter/pack_metadata.json index b8b52cc03358..502d8c4bbae9 100644 --- a/Packs/AWS-IAMIdentityCenter/pack_metadata.json +++ b/Packs/AWS-IAMIdentityCenter/pack_metadata.json @@ -2,7 +2,7 @@ "name": "AWS - IAM Identity Center", "description": "AWS IAM Identity Center\n\nWith AWS IAM Identity Center (successor to AWS Single Sign-On), you can manage sign-in security for your workforce identities, also known as workforce users. IAM Identity Center provides one place where you can create or connect workforce users and manage their access centrally across all their AWS accounts and applications. IAM Identity Center is the recommended approach for workforce authentication and authorization in AWS, for organizations of any size and type.", "support": "xsoar", - "currentVersion": "1.0.14", + "currentVersion": "1.0.15", "author": "Cortex XSOAR", "url": "", "email": "", diff --git a/Packs/AWS-Organizations/Integrations/AWSOrganizations/AWSOrganizations.yml b/Packs/AWS-Organizations/Integrations/AWSOrganizations/AWSOrganizations.yml index 43b319ea4914..230c6ec3acc3 100644 --- a/Packs/AWS-Organizations/Integrations/AWSOrganizations/AWSOrganizations.yml +++ b/Packs/AWS-Organizations/Integrations/AWSOrganizations/AWSOrganizations.yml @@ -691,7 +691,7 @@ script: - contextPath: AWS.Organizations.TagNextToken description: If not null, indicates that more output is available than is included in the current response. Use this value as the next_token argument in a subsequent call of the command to get the next part of the output. type: String - dockerimage: demisto/boto3py3:1.0.0.10221838 + dockerimage: demisto/boto3py3:1.0.0.12887184 feed: false longRunning: false longRunningPort: false diff --git a/Packs/AWS-Organizations/ReleaseNotes/1_0_18.md b/Packs/AWS-Organizations/ReleaseNotes/1_0_18.md new file mode 100644 index 000000000000..ac66ba50ff48 --- /dev/null +++ b/Packs/AWS-Organizations/ReleaseNotes/1_0_18.md @@ -0,0 +1,7 @@ + +#### Integrations + +##### AWS - Organizations + +- Updated the Docker image to: *demisto/boto3py3:1.0.0.12887184*. + diff --git a/Packs/AWS-Organizations/ReleaseNotes/1_0_19.md b/Packs/AWS-Organizations/ReleaseNotes/1_0_19.md new file mode 100644 index 000000000000..eaf8473b0ee6 --- /dev/null +++ b/Packs/AWS-Organizations/ReleaseNotes/1_0_19.md @@ -0,0 +1,3 @@ +## AWS Organizations + +- Locked dependencies of the pack to ensure stability for versioned core packs. No changes in this release. diff --git a/Packs/AWS-Organizations/pack_metadata.json b/Packs/AWS-Organizations/pack_metadata.json index c4b90a4444c6..a58676e337bc 100644 --- a/Packs/AWS-Organizations/pack_metadata.json +++ b/Packs/AWS-Organizations/pack_metadata.json @@ -2,7 +2,7 @@ "name": "AWS Organizations", "description": "AWS Organizations offers policy-based management for multiple AWS accounts.", "support": "xsoar", - "currentVersion": "1.0.17", + "currentVersion": "1.0.19", "author": "Cortex XSOAR", "url": "https://www.paloaltonetworks.com/cortex", "email": "", diff --git a/Packs/AWS-Route53/Integrations/AWSRoute53/AWSRoute53.yml b/Packs/AWS-Route53/Integrations/AWSRoute53/AWSRoute53.yml index 6864307179af..c9f2c9472c93 100644 --- a/Packs/AWS-Route53/Integrations/AWSRoute53/AWSRoute53.yml +++ b/Packs/AWS-Route53/Integrations/AWSRoute53/AWSRoute53.yml @@ -411,7 +411,7 @@ script: - contextPath: AWS.Route53.RecordSetsChange.Comment description: A complex type that describes change information about changes made to your hosted zone. type: string - dockerimage: demisto/boto3py3:1.0.0.10221838 + dockerimage: demisto/boto3py3:1.0.0.12887184 runonce: false script: '' subtype: python3 diff --git a/Packs/AWS-Route53/ReleaseNotes/1_1_51.md b/Packs/AWS-Route53/ReleaseNotes/1_1_51.md new file mode 100644 index 000000000000..41f4eae5b77b --- /dev/null +++ b/Packs/AWS-Route53/ReleaseNotes/1_1_51.md @@ -0,0 +1,7 @@ + +#### Integrations + +##### AWS - Route53 + +- Updated the Docker image to: *demisto/boto3py3:1.0.0.12887184*. + diff --git a/Packs/AWS-Route53/ReleaseNotes/1_1_52.md b/Packs/AWS-Route53/ReleaseNotes/1_1_52.md new file mode 100644 index 000000000000..847d3668bdca --- /dev/null +++ b/Packs/AWS-Route53/ReleaseNotes/1_1_52.md @@ -0,0 +1,3 @@ +## AWS - Route53 + +- Locked dependencies of the pack to ensure stability for versioned core packs. No changes in this release. diff --git a/Packs/AWS-Route53/pack_metadata.json b/Packs/AWS-Route53/pack_metadata.json index 689ff4eae089..9c3bbd387aac 100644 --- a/Packs/AWS-Route53/pack_metadata.json +++ b/Packs/AWS-Route53/pack_metadata.json @@ -2,7 +2,7 @@ "name": "AWS - Route53", "description": "Amazon Web Services Managed Cloud DNS Service.", "support": "xsoar", - "currentVersion": "1.1.50", + "currentVersion": "1.1.52", "author": "Cortex XSOAR", "url": "https://www.paloaltonetworks.com/cortex", "email": "", diff --git a/Packs/AWS-S3/Integrations/AWS-S3/AWS-S3.yml b/Packs/AWS-S3/Integrations/AWS-S3/AWS-S3.yml index 4bbb51d6e265..b4d4a6ea29a3 100644 --- a/Packs/AWS-S3/Integrations/AWS-S3/AWS-S3.yml +++ b/Packs/AWS-S3/Integrations/AWS-S3/AWS-S3.yml @@ -405,7 +405,7 @@ script: required: true description: Creates or modifies the PublicAccessBlock configuration for an Amazon S3 bucket. name: aws-s3-put-public-access-block - dockerimage: demisto/boto3py3:1.0.0.10221838 + dockerimage: demisto/boto3py3:1.0.0.12887184 runonce: false script: '' type: python diff --git a/Packs/AWS-S3/ReleaseNotes/1_2_43.md b/Packs/AWS-S3/ReleaseNotes/1_2_43.md new file mode 100644 index 000000000000..e518e3a7a94b --- /dev/null +++ b/Packs/AWS-S3/ReleaseNotes/1_2_43.md @@ -0,0 +1,7 @@ + +#### Integrations + +##### AWS - S3 + +- Updated the Docker image to: *demisto/boto3py3:1.0.0.12887184*. + diff --git a/Packs/AWS-S3/ReleaseNotes/1_2_44.md b/Packs/AWS-S3/ReleaseNotes/1_2_44.md new file mode 100644 index 000000000000..b530392bde3a --- /dev/null +++ b/Packs/AWS-S3/ReleaseNotes/1_2_44.md @@ -0,0 +1,3 @@ +## AWS - S3 + +- Locked dependencies of the pack to ensure stability for versioned core packs. No changes in this release. diff --git a/Packs/AWS-S3/pack_metadata.json b/Packs/AWS-S3/pack_metadata.json index 87c126964ab9..f1444e3bf3e8 100644 --- a/Packs/AWS-S3/pack_metadata.json +++ b/Packs/AWS-S3/pack_metadata.json @@ -2,7 +2,7 @@ "name": "AWS - S3", "description": "Amazon Web Services Simple Storage Service (S3)", "support": "xsoar", - "currentVersion": "1.2.42", + "currentVersion": "1.2.44", "author": "Cortex XSOAR", "url": "https://www.paloaltonetworks.com/cortex", "email": "", diff --git a/Packs/AWS-SNS-Listener/Integrations/AWSSNSListener/AWSSNSListener.yml b/Packs/AWS-SNS-Listener/Integrations/AWSSNSListener/AWSSNSListener.yml index d58a17f28821..722f7a333ceb 100644 --- a/Packs/AWS-SNS-Listener/Integrations/AWSSNSListener/AWSSNSListener.yml +++ b/Packs/AWS-SNS-Listener/Integrations/AWSSNSListener/AWSSNSListener.yml @@ -62,7 +62,7 @@ display: AWS-SNS-Listener name: AWS-SNS-Listener script: commands: [] - dockerimage: demisto/fastapi:0.125.0.10158186 + dockerimage: demisto/fastapi:0.125.0.12689358 longRunning: true longRunningPort: true script: '-' @@ -83,3 +83,4 @@ triggers: action: hidden: true required: false +excludefromtightlycoupled: true diff --git a/Packs/AWS-SNS-Listener/ReleaseNotes/1_0_17.md b/Packs/AWS-SNS-Listener/ReleaseNotes/1_0_17.md new file mode 100644 index 000000000000..a3f67d1bac18 --- /dev/null +++ b/Packs/AWS-SNS-Listener/ReleaseNotes/1_0_17.md @@ -0,0 +1,7 @@ + +#### Integrations + +##### AWS-SNS-Listener + +- Updated the Docker image to: *demisto/fastapi:0.125.0.12689358*. + diff --git a/Packs/AWS-SNS-Listener/ReleaseNotes/1_0_18.md b/Packs/AWS-SNS-Listener/ReleaseNotes/1_0_18.md new file mode 100644 index 000000000000..c304fef25965 --- /dev/null +++ b/Packs/AWS-SNS-Listener/ReleaseNotes/1_0_18.md @@ -0,0 +1,6 @@ + +#### Integrations + +##### AWS-SNS-Listener + +- Documentation and metadata improvements. diff --git a/Packs/AWS-SNS-Listener/pack_metadata.json b/Packs/AWS-SNS-Listener/pack_metadata.json index 9ab985f59132..5659c2594a69 100644 --- a/Packs/AWS-SNS-Listener/pack_metadata.json +++ b/Packs/AWS-SNS-Listener/pack_metadata.json @@ -2,7 +2,7 @@ "name": "AWS-SNS-Listener", "description": "A long running AWS SNS Listener service that can subscribe to an SNS topic and create incidents from the messages received.", "support": "xsoar", - "currentVersion": "1.0.16", + "currentVersion": "1.0.18", "author": "Cortex XSOAR", "url": "https://www.paloaltonetworks.com/cortex", "email": "", diff --git a/Packs/AWS-SNS/Integrations/AWSSNS/AWSSNS.yml b/Packs/AWS-SNS/Integrations/AWSSNS/AWSSNS.yml index ad1589cd2118..5596a710ea39 100644 --- a/Packs/AWS-SNS/Integrations/AWSSNS/AWSSNS.yml +++ b/Packs/AWS-SNS/Integrations/AWSSNS/AWSSNS.yml @@ -232,7 +232,7 @@ script: outputs: - contextPath: AWS.SNS.Subscriptions.SubscriptionArn description: The Subscription Arn. - dockerimage: demisto/boto3py3:1.0.0.10221838 + dockerimage: demisto/boto3py3:1.0.0.12887184 script: '' subtype: python3 type: python diff --git a/Packs/AWS-SNS/ReleaseNotes/1_0_27.md b/Packs/AWS-SNS/ReleaseNotes/1_0_27.md new file mode 100644 index 000000000000..98de0a4e088f --- /dev/null +++ b/Packs/AWS-SNS/ReleaseNotes/1_0_27.md @@ -0,0 +1,7 @@ + +#### Integrations + +##### AWS - SNS + +- Updated the Docker image to: *demisto/boto3py3:1.0.0.12887184*. + diff --git a/Packs/AWS-SNS/pack_metadata.json b/Packs/AWS-SNS/pack_metadata.json index d3006ab27dd2..54dfd5cff4f4 100644 --- a/Packs/AWS-SNS/pack_metadata.json +++ b/Packs/AWS-SNS/pack_metadata.json @@ -2,7 +2,7 @@ "name": "AWS - SNS", "description": "This is the integration content pack which can create or delete topic/subscription on AWS Simple Notification System and send the message via SNS as well.", "support": "xsoar", - "currentVersion": "1.0.26", + "currentVersion": "1.0.27", "author": "Cortex XSOAR", "url": "", "email": "", diff --git a/Packs/AWS-SQS/Integrations/AWS-SQS/AWS-SQS.yml b/Packs/AWS-SQS/Integrations/AWS-SQS/AWS-SQS.yml index 0b47d9192a72..d2e329f53657 100644 --- a/Packs/AWS-SQS/Integrations/AWS-SQS/AWS-SQS.yml +++ b/Packs/AWS-SQS/Integrations/AWS-SQS/AWS-SQS.yml @@ -102,6 +102,13 @@ configuration: type: 13 section: Connect required: false +- display: Incidents Fetch Interval + name: incidentFetchInterval + defaultvalue: '1' + required: false + type: 19 + section: Collect + advanced: true - display: AWS STS Regional Endpoints additionalinfo: Sets the AWS_STS_REGIONAL_ENDPOINTS environment variable to specify the AWS STS endpoint resolution logic. By default, this option is set to “legacy” in AWS. Leave empty if the environment variable is already set using server configuration. name: sts_regional_endpoint diff --git a/Packs/AWS-SQS/ReleaseNotes/1_2_37.md b/Packs/AWS-SQS/ReleaseNotes/1_2_37.md new file mode 100644 index 000000000000..aa438c7cba06 --- /dev/null +++ b/Packs/AWS-SQS/ReleaseNotes/1_2_37.md @@ -0,0 +1,6 @@ + +#### Integrations + +##### AWS - SQS + +- Updated the Fetch Incidents configuration parameters. diff --git a/Packs/AWS-SQS/pack_metadata.json b/Packs/AWS-SQS/pack_metadata.json index 9fa7738c2878..8436cad15225 100644 --- a/Packs/AWS-SQS/pack_metadata.json +++ b/Packs/AWS-SQS/pack_metadata.json @@ -2,7 +2,7 @@ "name": "AWS - SQS", "description": "Amazon Web Services Simple Queuing Service (SQS)", "support": "xsoar", - "currentVersion": "1.2.36", + "currentVersion": "1.2.37", "author": "Cortex XSOAR", "url": "https://www.paloaltonetworks.com/cortex", "email": "", diff --git a/Packs/AWS-SecurityHub/Integrations/AWSSecurityHubEventCollector/AWSSecurityHubEventCollector.py b/Packs/AWS-SecurityHub/Integrations/AWSSecurityHubEventCollector/AWSSecurityHubEventCollector.py index 6907314e0b1a..8fec86762437 100644 --- a/Packs/AWS-SecurityHub/Integrations/AWSSecurityHubEventCollector/AWSSecurityHubEventCollector.py +++ b/Packs/AWS-SecurityHub/Integrations/AWSSecurityHubEventCollector/AWSSecurityHubEventCollector.py @@ -26,26 +26,39 @@ def parse_aws_timestamp(timestamp_str: str) -> dt.datetime: """ - Parse AWS timestamp with flexible format support. - The AWS API returns timestamps in two formats: - - With milliseconds: "2023-01-01T00:00:00.000Z" - - Without milliseconds: "2023-01-01T00:00:00Z" + Parse an AWS timestamp into a naive UTC datetime. + + AWS returns 'CreatedAt' in several ISO-8601 shapes, e.g.: + - With milliseconds and 'Z': "2023-01-01T00:00:00.000Z" + - Without milliseconds and 'Z': "2023-01-01T00:00:00Z" + - With a UTC/other offset: "2026-08-08T19:55:21.905451+00:00" + + The returned value feeds strftime() when building the AWS findings filter, which + expects a naive UTC value. A timezone-aware value would shift the fetch window, + so any offset is normalized to UTC and then dropped, yielding a naive datetime. Args: - timestamp_str (str): Timestamp string from AWS API + timestamp_str (str): Timestamp string from AWS API. Returns: - datetime: Parsed datetime object + datetime: Parsed naive UTC datetime object. Raises: - ValueError: If timestamp format is not supported + ValueError: If the timestamp cannot be parsed as ISO-8601. """ - try: - # Try parsing with milliseconds first - return dt.datetime.strptime(timestamp_str, DATETIME_FORMAT) - except ValueError: - # Fallback to parsing without milliseconds - return dt.datetime.strptime(timestamp_str, DATETIME_FORMAT_NO_MS) + for fmt in (DATETIME_FORMAT, DATETIME_FORMAT_NO_MS): + try: + return dt.datetime.strptime(timestamp_str, fmt) + except ValueError: + continue + + # Fallback: accept any ISO-8601 form (e.g. an offset-style "+00:00"). + # 'Z' is normalized to "+00:00" for fromisoformat compatibility on older Python versions. + parsed = dt.datetime.fromisoformat(timestamp_str.replace("Z", "+00:00")) + if parsed.tzinfo is not None: + # Normalize to UTC, then drop tzinfo so the value stays naive for strftime(). + parsed = parsed.astimezone(dt.UTC).replace(tzinfo=None) + return parsed def generate_last_run(events: list["AwsSecurityFindingTypeDef"], previous_last_run: dict | None = None) -> dict[str, Any]: @@ -239,8 +252,9 @@ def fetch_events( """ demisto.debug(f"Fetching events with last_run: {last_run}") if last_run.get("last_update_date"): + # If the stored cursor cannot be parsed, let the ValueError propagate and fail the + # fetch loudly rather than silently resetting the fetch window. start_time = parse_aws_timestamp(last_run["last_update_date"]) - else: start_time = cast(dt.datetime, first_fetch_time) diff --git a/Packs/AWS-SecurityHub/Integrations/AWSSecurityHubEventCollector/AWSSecurityHubEventCollector.yml b/Packs/AWS-SecurityHub/Integrations/AWSSecurityHubEventCollector/AWSSecurityHubEventCollector.yml index e9f455937b3c..9d88a4210346 100644 --- a/Packs/AWS-SecurityHub/Integrations/AWSSecurityHubEventCollector/AWSSecurityHubEventCollector.yml +++ b/Packs/AWS-SecurityHub/Integrations/AWSSecurityHubEventCollector/AWSSecurityHubEventCollector.yml @@ -132,7 +132,7 @@ script: name: end_time description: Fetch events from AWS Security Hub. name: aws-securityhub-get-events - dockerimage: demisto/boto3py3:1.0.0.10221838 + dockerimage: demisto/boto3py3:1.0.0.12887184 isfetchevents: true script: '-' subtype: python3 diff --git a/Packs/AWS-SecurityHub/Integrations/AWSSecurityHubEventCollector/AWSSecurityHubEventCollector_test.py b/Packs/AWS-SecurityHub/Integrations/AWSSecurityHubEventCollector/AWSSecurityHubEventCollector_test.py index 1749a0074f12..d068c73365d8 100644 --- a/Packs/AWS-SecurityHub/Integrations/AWSSecurityHubEventCollector/AWSSecurityHubEventCollector_test.py +++ b/Packs/AWS-SecurityHub/Integrations/AWSSecurityHubEventCollector/AWSSecurityHubEventCollector_test.py @@ -128,16 +128,29 @@ class TestParseAwsTimestamp: ("2023-01-01T12:30:45.123Z", dt.datetime(2023, 1, 1, 12, 30, 45, 123000)), # Valid timestamp without milliseconds ("2023-01-01T12:30:45Z", dt.datetime(2023, 1, 1, 12, 30, 45)), + # ISO-8601 with UTC offset (+00:00) - returned verbatim by AWS CreatedAt for some sources + ("2026-08-08T19:55:21.905451+00:00", dt.datetime(2026, 8, 8, 19, 55, 21, 905451)), + # ISO-8601 with UTC offset, no microseconds + ("2026-08-08T19:55:21+00:00", dt.datetime(2026, 8, 8, 19, 55, 21)), + # ISO-8601 with a non-zero offset should be normalized to naive UTC + ("2026-08-08T21:55:21+02:00", dt.datetime(2026, 8, 8, 19, 55, 21)), + # ISO-8601 without 'Z' and without offset: fails both strptime formats and is + # parsed by fromisoformat as an already-naive datetime (no tzinfo normalization). + ("2026-08-08T19:55:21.905451", dt.datetime(2026, 8, 8, 19, 55, 21, 905451)), + ("2026-08-08T19:55:21", dt.datetime(2026, 8, 8, 19, 55, 21)), ], ) def test_parse_timestamp_valid_formats(self, timestamp_str, expected_result): """ - Given: A valid timestamp string from AWS API (with or without milliseconds). + Given: A valid timestamp string from AWS API (Z-suffixed or ISO-8601 offset form). When: Parsing the timestamp using parse_aws_timestamp. - Then: Should return the correct datetime object. + Then: Should return the correct naive UTC datetime object. """ result = parse_aws_timestamp(timestamp_str) assert result == expected_result + # The value feeds strftime() to build the AWS filter, so it must be naive (no tzinfo) + # to avoid shifting the fetch window. + assert result.tzinfo is None def test_parse_timestamp_invalid_format(self): """ @@ -145,7 +158,7 @@ def test_parse_timestamp_invalid_format(self): When: Parsing the timestamp using parse_aws_timestamp. Then: Should raise ValueError. """ - invalid_timestamp = "2023/01/01 12:30:45" + invalid_timestamp = "not-a-timestamp" with pytest.raises(ValueError): parse_aws_timestamp(invalid_timestamp) @@ -662,3 +675,33 @@ def test_fetch_events_api_error_handling(fetch_events_mocks): # Verify error logging error_calls = [call.args[0] for call in mocks["error"].call_args_list] assert any("Error while fetching events" in msg for msg in error_calls) + + +def test_fetch_events_unparseable_last_run_raises(fetch_events_mocks): + """ + Test fetch_events fails loudly when last_run holds a value that parse_aws_timestamp + cannot parse, rather than silently resetting the fetch window. + + Given: last_run["last_update_date"] cannot be parsed + When: Calling fetch_events + Then: The ValueError from parse_aws_timestamp propagates and get_events is never called. + """ + mocks = fetch_events_mocks + + # Simulate parse_aws_timestamp rejecting the stored cursor. That a truly malformed value + # actually raises here is covered by TestParseAwsTimestamp; this test isolates propagation. + parse_error = ValueError("Invalid isoformat string: 'not-a-parseable-timestamp'") + mocks["parse_timestamp"].side_effect = parse_error + + with pytest.raises(ValueError) as exc_info: + fetch_events( + client=mocks["client"], + last_run={"last_update_date": "not-a-parseable-timestamp", "last_update_date_finding_ids": ["old-1"]}, + first_fetch_time=FIRST_FETCH_TIME, + page_size=50, + limit=0, + ) + + assert exc_info.value is parse_error + # The fetch fails before any events are requested. + mocks["get_events"].assert_not_called() diff --git a/Packs/AWS-SecurityHub/Integrations/AWS_SecurityHub/AWS_SecurityHub.yml b/Packs/AWS-SecurityHub/Integrations/AWS_SecurityHub/AWS_SecurityHub.yml index d9158b623e51..a8ca11f3b3ba 100644 --- a/Packs/AWS-SecurityHub/Integrations/AWS_SecurityHub/AWS_SecurityHub.yml +++ b/Packs/AWS-SecurityHub/Integrations/AWS_SecurityHub/AWS_SecurityHub.yml @@ -122,6 +122,13 @@ configuration: type: 13 section: Connect required: false +- display: Incidents Fetch Interval + name: incidentFetchInterval + defaultvalue: '1' + required: false + type: 19 + section: Collect + advanced: true - display: Security Hub Incidents Severity Level name: sh_severity type: 15 diff --git a/Packs/AWS-SecurityHub/Integrations/AWS_SecurityHub_V2/AWS_SecurityHub_V2.yml b/Packs/AWS-SecurityHub/Integrations/AWS_SecurityHub_V2/AWS_SecurityHub_V2.yml index 4c7c06d03daa..00362209541c 100644 --- a/Packs/AWS-SecurityHub/Integrations/AWS_SecurityHub_V2/AWS_SecurityHub_V2.yml +++ b/Packs/AWS-SecurityHub/Integrations/AWS_SecurityHub_V2/AWS_SecurityHub_V2.yml @@ -1,7 +1,7 @@ name: AWS - Security Hub v2 display: AWS - Security Hub v2 category: IT Services -provider: Amazon Web Services +provider: Amazon sectionorder: - Connect - Collect @@ -320,7 +320,7 @@ script: - name: remoteId description: The remote finding metadata UID to update. required: false - dockerimage: demisto/boto3py3:1.0.0.11314142 + dockerimage: demisto/boto3py3:1.0.0.12887184 isfetch: true ismappable: true isremotesyncin: true diff --git a/Packs/AWS-SecurityHub/ReleaseNotes/2_0_1.md b/Packs/AWS-SecurityHub/ReleaseNotes/2_0_1.md new file mode 100644 index 000000000000..6d4c15bae254 --- /dev/null +++ b/Packs/AWS-SecurityHub/ReleaseNotes/2_0_1.md @@ -0,0 +1,6 @@ + +#### Integrations + +##### AWS - Security Hub v2 + +Updated the integration provider name to **Amazon**. diff --git a/Packs/AWS-SecurityHub/ReleaseNotes/2_0_2.md b/Packs/AWS-SecurityHub/ReleaseNotes/2_0_2.md new file mode 100644 index 000000000000..718ad57f1b44 --- /dev/null +++ b/Packs/AWS-SecurityHub/ReleaseNotes/2_0_2.md @@ -0,0 +1,6 @@ + +#### Integrations + +##### AWS - Security Hub + +- Updated the Fetch Incidents configuration parameters. diff --git a/Packs/AWS-SecurityHub/ReleaseNotes/2_0_3.md b/Packs/AWS-SecurityHub/ReleaseNotes/2_0_3.md new file mode 100644 index 000000000000..de592cfab333 --- /dev/null +++ b/Packs/AWS-SecurityHub/ReleaseNotes/2_0_3.md @@ -0,0 +1,11 @@ + +#### Integrations + +##### AWS Security Hub Event Collector + +- Updated the Docker image to: *demisto/boto3py3:1.0.0.12887184*. + +##### AWS - Security Hub v2 + +- Updated the Docker image to: *demisto/boto3py3:1.0.0.12887184*. + diff --git a/Packs/AWS-SecurityHub/ReleaseNotes/2_0_4.md b/Packs/AWS-SecurityHub/ReleaseNotes/2_0_4.md new file mode 100644 index 000000000000..d6bc88f31818 --- /dev/null +++ b/Packs/AWS-SecurityHub/ReleaseNotes/2_0_4.md @@ -0,0 +1,6 @@ + +#### Integrations + +##### AWS Security Hub Event Collector + +- Fixed an issue where the AWS Security Hub Event Collector fetch events failed due to an unsupported timestamp format returned by AWS. diff --git a/Packs/AWS-SecurityHub/pack_metadata.json b/Packs/AWS-SecurityHub/pack_metadata.json index f882613e2c0e..91070b2056f0 100644 --- a/Packs/AWS-SecurityHub/pack_metadata.json +++ b/Packs/AWS-SecurityHub/pack_metadata.json @@ -2,7 +2,7 @@ "name": "AWS - Security Hub", "description": "Amazon Web Services Security Hub Service.", "support": "xsoar", - "currentVersion": "2.0.0", + "currentVersion": "2.0.4", "author": "Cortex XSOAR", "url": "https://www.paloaltonetworks.com/cortex", "email": "", diff --git a/Packs/AWS-SecurityLake/Integrations/AWSSecurityLake/AWSSecurityLake.yml b/Packs/AWS-SecurityLake/Integrations/AWSSecurityLake/AWSSecurityLake.yml index fdcc669b2249..eeb9b0b8006a 100644 --- a/Packs/AWS-SecurityLake/Integrations/AWSSecurityLake/AWSSecurityLake.yml +++ b/Packs/AWS-SecurityLake/Integrations/AWSSecurityLake/AWSSecurityLake.yml @@ -1212,7 +1212,7 @@ script: script: "-" type: python subtype: python3 - dockerimage: demisto/boto3py3:1.0.0.10221838 + dockerimage: demisto/boto3py3:1.0.0.12887184 feed: false isfetch: false fromversion: 6.10.0 diff --git a/Packs/AWS-SecurityLake/ReleaseNotes/1_0_20.md b/Packs/AWS-SecurityLake/ReleaseNotes/1_0_20.md new file mode 100644 index 000000000000..e5ec28c99b1c --- /dev/null +++ b/Packs/AWS-SecurityLake/ReleaseNotes/1_0_20.md @@ -0,0 +1,7 @@ + +#### Integrations + +##### Amazon Security Lake + +- Updated the Docker image to: *demisto/boto3py3:1.0.0.12887184*. + diff --git a/Packs/AWS-SecurityLake/pack_metadata.json b/Packs/AWS-SecurityLake/pack_metadata.json index 5595f503fde6..c51a18bda7be 100644 --- a/Packs/AWS-SecurityLake/pack_metadata.json +++ b/Packs/AWS-SecurityLake/pack_metadata.json @@ -2,7 +2,7 @@ "name": "Amazon - Security Lake", "description": "Amazon Security Lake is a fully managed security data lake service.", "support": "xsoar", - "currentVersion": "1.0.19", + "currentVersion": "1.0.20", "author": "Cortex XSOAR", "url": "https://www.paloaltonetworks.com/cortex", "email": "", diff --git a/Packs/AWS/Integrations/AWS/AWS.py b/Packs/AWS/Integrations/AWS/AWS.py index 3b6d020f5a4f..f19e71ff2129 100644 --- a/Packs/AWS/Integrations/AWS/AWS.py +++ b/Packs/AWS/Integrations/AWS/AWS.py @@ -649,6 +649,8 @@ def aws_ec2_block_device_mapping_args_builder(args: Dict[str, Any]) -> List[Dict "VolumeSize": arg_to_number(args.get("ebs_volume_size")), "VolumeType": args.get("ebs_volume_type"), "Throughput": arg_to_number(args.get("ebs_throughput")), + "EbsCardIndex": arg_to_number(args.get("ebs_card_index")), + "VolumeInitializationRate": arg_to_number(args.get("ebs_initialization_rate")), }, "NoDevice": args.get("block_device_mappings_no_device"), "VirtualName": args.get("block_device_mappings_virtual_name"), @@ -1682,7 +1684,7 @@ def get_public_access_block_command(client: BotoClient, args: Dict[str, Any]) -> if response.get("ResponseMetadata", {}).get("HTTPStatusCode") == HTTPStatus.OK: return CommandResults( - outputs_prefix="AWS.S3-Buckets", + outputs_prefix="AWS.S3.Buckets", outputs_key_field="BucketName", outputs={"BucketName": bucket_name, "PublicAccessBlock": response.get("PublicAccessBlockConfiguration", {})}, readable_output=tableToMarkdown( @@ -1723,7 +1725,7 @@ def get_bucket_encryption_command(client: BotoClient, args: Dict[str, Any]) -> C "ServerSideEncryptionConfiguration": response.get("ServerSideEncryptionConfiguration", {}), } return CommandResults( - outputs_prefix="AWS.S3-Buckets", + outputs_prefix="AWS.S3.Buckets", outputs_key_field="BucketName", outputs=outputs, readable_output=tableToMarkdown( @@ -1761,7 +1763,7 @@ def get_bucket_policy_command(client: BotoClient, args: Dict[str, Any]) -> Comma json_response = json.loads(response.get("Policy", "{}")) json_statement = json_response.get("Statement", []) return CommandResults( - outputs_prefix="AWS.S3-Buckets", + outputs_prefix="AWS.S3.Buckets", outputs_key_field="BucketName", outputs={ "BucketName": bucket_name, @@ -1866,7 +1868,7 @@ def file_download_command(client: BotoClient, args: Dict[str, Any]): filename = key.rsplit("/", 1)[-1] return fileResult(filename, data) except ClientError as err: - AWSErrorHandler.handle_client_error(err) + return AWSErrorHandler.handle_client_error(err) except Exception as e: raise DemistoException(f"Error: {str(e)}") @@ -1901,7 +1903,7 @@ def file_upload_command(client: BotoClient, args: Dict[str, Any]): return CommandResults(readable_output="Failed to upload file") @staticmethod - def get_bucket_website_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults: + def get_bucket_website_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults | None: """ Retrieves the website configuration for a specified Amazon S3 bucket. The function calls the AWS S3 'get_bucket_website' API to check if the bucket @@ -1921,7 +1923,7 @@ def get_bucket_website_command(client: BotoClient, args: Dict[str, Any]) -> Comm response = client.get_bucket_website(**kwargs) if response["ResponseMetadata"]["HTTPStatusCode"] not in [HTTPStatus.OK, HTTPStatus.NO_CONTENT]: - AWSErrorHandler.handle_response_error(response, args.get("account_id")) + return AWSErrorHandler.handle_response_error(response, args.get("account_id")) response["WebsiteConfiguration"] = { "ErrorDocument": response.get("ErrorDocument"), @@ -1939,13 +1941,13 @@ def get_bucket_website_command(client: BotoClient, args: Dict[str, Any]) -> Comm ) return CommandResults( readable_output=readable_output, - outputs_prefix="AWS.S3-Buckets.BucketWebsite", + outputs_prefix="AWS.S3.Buckets.BucketWebsite", outputs=response.get("WebsiteConfiguration", {}), raw_response=response.get("WebsiteConfiguration", {}), ) @staticmethod - def get_bucket_acl_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults: + def get_bucket_acl_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults | None: """ Retrieves the Access Control List (ACL) of a specified Amazon S3 bucket. @@ -1966,7 +1968,7 @@ def get_bucket_acl_command(client: BotoClient, args: Dict[str, Any]) -> CommandR response = client.get_bucket_acl(**kwargs) if response["ResponseMetadata"]["HTTPStatusCode"] not in [HTTPStatus.OK, HTTPStatus.NO_CONTENT]: - AWSErrorHandler.handle_response_error(response, args.get("account_id")) + return AWSErrorHandler.handle_response_error(response, args.get("account_id")) response["AccessControlPolicy"] = { "Grants": response.get("Grants"), @@ -1981,13 +1983,13 @@ def get_bucket_acl_command(client: BotoClient, args: Dict[str, Any]) -> CommandR ) return CommandResults( readable_output=readable_output, - outputs_prefix="AWS.S3-Buckets.BucketAcl", + outputs_prefix="AWS.S3.Buckets.BucketAcl", outputs=response.get("AccessControlPolicy", {}), raw_response=response.get("AccessControlPolicy", {}), ) @staticmethod - def bucket_create_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults: + def bucket_create_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults | None: """ Creates a new S3 bucket. @@ -2001,6 +2003,7 @@ def bucket_create_command(client: BotoClient, args: Dict[str, Any]) -> CommandRe bucket_name = args.get("bucket_name") location = args.get("location_constraint") or args.get("region", "") kwargs = { + "ACL": args.get("acl"), "Bucket": bucket_name, "GrantFullControl": args.get("grant_full_control"), "GrantRead": args.get("grant_read"), @@ -2019,7 +2022,7 @@ def bucket_create_command(client: BotoClient, args: Dict[str, Any]) -> CommandRe demisto.debug(f"{response=}") if response["ResponseMetadata"]["HTTPStatusCode"] not in [HTTPStatus.OK, HTTPStatus.NO_CONTENT]: - AWSErrorHandler.handle_response_error(response, args.get("account_id")) + return AWSErrorHandler.handle_response_error(response, args.get("account_id")) output = {"Location": response.get("Location"), "BucketArn": response.get("BucketArn"), "BucketName": bucket_name} @@ -2032,7 +2035,7 @@ def bucket_create_command(client: BotoClient, args: Dict[str, Any]) -> CommandRe ) @staticmethod - def buckets_list_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults: + def buckets_list_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults | None: """ Returns a list of all buckets owned by the authenticated sender of the request. @@ -2052,7 +2055,7 @@ def buckets_list_command(client: BotoClient, args: Dict[str, Any]) -> CommandRes response = client.list_buckets(**kwargs) if response["ResponseMetadata"]["HTTPStatusCode"] not in [HTTPStatus.OK, HTTPStatus.NO_CONTENT]: - AWSErrorHandler.handle_response_error(response, args.get("account_id")) + return AWSErrorHandler.handle_response_error(response, args.get("account_id")) buckets = response.get("Buckets", []) for bucket in buckets: @@ -2693,7 +2696,7 @@ def modify_snapshot_permission_command(client: BotoClient, args: Dict[str, Any]) return CommandResults(readable_output=f"Snapshot {args.get('snapshot_id')} permissions were successfully updated.") @staticmethod - def describe_instances_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults: + def describe_instances_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults | None: """ Retrieves detailed information about EC2 instances including status, configuration, and metadata. @@ -2723,7 +2726,7 @@ def describe_instances_command(client: BotoClient, args: Dict[str, Any]) -> Comm remove_nulls_from_dictionary(kwargs) response = client.describe_instances(**kwargs) if response.get("ResponseMetadata", {}).get("HTTPStatusCode") != HTTPStatus.OK: - AWSErrorHandler.handle_response_error(response, args.get("account_id")) + return AWSErrorHandler.handle_response_error(response, args.get("account_id")) response = serialize_response_with_datetime_encoding(response) # Extract instances from reservations reservations = response.get("Reservations", []) @@ -2749,7 +2752,7 @@ def describe_instances_command(client: BotoClient, args: Dict[str, Any]) -> Comm ) @staticmethod - def run_instances_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults: + def run_instances_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults | None: """ Runs one or more Amazon EC2 instances. @@ -2853,7 +2856,7 @@ def run_instances_command(client: BotoClient, args: Dict[str, Any]) -> CommandRe response = client.run_instances(**kwargs) if response.get("ResponseMetadata", {}).get("HTTPStatusCode") != HTTPStatus.OK: - AWSErrorHandler.handle_response_error(response, args.get("account_id")) + return AWSErrorHandler.handle_response_error(response, args.get("account_id")) response = serialize_response_with_datetime_encoding(response) instances = response.get("Instances", []) if not instances: @@ -3260,7 +3263,7 @@ def describe_ipam_resource_discovery_associations_command(client: BotoClient, ar return command_results @staticmethod - def get_latest_ami_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults: + def get_latest_ami_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults | None: """ Retrieves information about the latest Amazon Machine Image (AMI) based on provided filters. The function calls the AWS EC2 'describe_images' API, sorts the results by @@ -3287,7 +3290,7 @@ def get_latest_ami_command(client: BotoClient, args: Dict[str, Any]) -> CommandR remove_nulls_from_dictionary(kwargs) response = client.describe_images(**kwargs) if response["ResponseMetadata"]["HTTPStatusCode"] not in [HTTPStatus.OK, HTTPStatus.NO_CONTENT]: - AWSErrorHandler.handle_response_error(response, args.get("account_id")) + return AWSErrorHandler.handle_response_error(response, args.get("account_id")) amis = response.get("Images", []) iterates = 1 @@ -3296,7 +3299,7 @@ def get_latest_ami_command(client: BotoClient, args: Dict[str, Any]) -> CommandR kwargs["NextToken"] = response.get("nextToken") response = client.describe_images(**kwargs) if response["ResponseMetadata"]["HTTPStatusCode"] not in [HTTPStatus.OK, HTTPStatus.NO_CONTENT]: - AWSErrorHandler.handle_response_error(response, args.get("account_id")) + return AWSErrorHandler.handle_response_error(response, args.get("account_id")) amis.extend(response.get("Images", [])) iterates += 1 @@ -3327,7 +3330,7 @@ def get_latest_ami_command(client: BotoClient, args: Dict[str, Any]) -> CommandR ) @staticmethod - def create_network_acl_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults: + def create_network_acl_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults | None: """ Creates a Network Access Control List (Network ACL) for the specified VPC. The function calls the AWS EC2 'create_network_acl' API. It requires the ID @@ -3356,7 +3359,7 @@ def create_network_acl_command(client: BotoClient, args: Dict[str, Any]) -> Comm response = client.create_network_acl(**kwargs) if response["ResponseMetadata"]["HTTPStatusCode"] not in [HTTPStatus.OK, HTTPStatus.NO_CONTENT]: - AWSErrorHandler.handle_response_error(response, args.get("account_id")) + return AWSErrorHandler.handle_response_error(response, args.get("account_id")) network_acl = response.get("NetworkAcl") readable_data = { @@ -3387,7 +3390,7 @@ def create_network_acl_command(client: BotoClient, args: Dict[str, Any]) -> Comm ) @staticmethod - def get_ipam_discovered_public_addresses_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults: + def get_ipam_discovered_public_addresses_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults | None: """ aws-ec2-get-ipam-discovered-public-addresses: Gets the public IP addresses that have been discovered by IPAM. @@ -3410,7 +3413,7 @@ def get_ipam_discovered_public_addresses_command(client: BotoClient, args: Dict[ remove_nulls_from_dictionary(kwargs) response = client.get_ipam_discovered_public_addresses(**kwargs) if response["ResponseMetadata"]["HTTPStatusCode"] not in [HTTPStatus.OK, HTTPStatus.NO_CONTENT]: - AWSErrorHandler.handle_response_error(response, args.get("account_id")) + return AWSErrorHandler.handle_response_error(response, args.get("account_id")) if not response.get("IpamDiscoveredPublicAddresses"): return CommandResults(readable_output="No Ipam Discovered Public Addresses were found.") @@ -3430,7 +3433,7 @@ def get_ipam_discovered_public_addresses_command(client: BotoClient, args: Dict[ ) @staticmethod - def create_tags_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults: + def create_tags_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults | None: """ Adds or overwrites one or more tags for the specified AWS resources. @@ -3454,12 +3457,12 @@ def create_tags_command(client: BotoClient, args: Dict[str, Any]) -> CommandResu response = client.create_tags(**kwargs) if response["ResponseMetadata"]["HTTPStatusCode"] not in [HTTPStatus.OK, HTTPStatus.NO_CONTENT]: - AWSErrorHandler.handle_response_error(response, args.get("account_id")) + return AWSErrorHandler.handle_response_error(response, args.get("account_id")) return CommandResults(readable_output="The resources where tagged successfully") @staticmethod - def network_interface_attribute_modify_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults: + def network_interface_attribute_modify_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults | None: """ Modifies the specified network interface attribute. Args: @@ -3475,7 +3478,7 @@ def network_interface_attribute_modify_command(client: BotoClient, args: Dict[st response = client.modify_network_interface_attribute(**kwargs) if response["ResponseMetadata"]["HTTPStatusCode"] not in [HTTPStatus.OK, HTTPStatus.NO_CONTENT]: - AWSErrorHandler.handle_response_error(response, args.get("account_id")) + return AWSErrorHandler.handle_response_error(response, args.get("account_id")) demisto.debug(f"{response=}") outputs = { @@ -3494,7 +3497,7 @@ def network_interface_attribute_modify_command(client: BotoClient, args: Dict[st ) @staticmethod - def regions_describe_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults: + def regions_describe_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults | None: """ Describes the Regions that are enabled for your account, or all Regions. @@ -3522,7 +3525,7 @@ def regions_describe_command(client: BotoClient, args: Dict[str, Any]) -> Comman demisto.debug(f"{kwargs=}") response = client.describe_regions(**kwargs) if response["ResponseMetadata"]["HTTPStatusCode"] not in [HTTPStatus.OK, HTTPStatus.NO_CONTENT]: - AWSErrorHandler.handle_response_error(response, args.get("account_id")) + return AWSErrorHandler.handle_response_error(response, args.get("account_id")) regions = response.get("Regions") readable_output = tableToMarkdown( @@ -3670,7 +3673,7 @@ def describe_security_groups_command(client: BotoClient, args: Dict[str, Any]) - return AWSErrorHandler.handle_response_error(response) @staticmethod - def describe_addresses_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults: + def describe_addresses_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults | None: """ Describes one or more Elastic IP addresses. @@ -3704,7 +3707,7 @@ def describe_addresses_command(client: BotoClient, args: Dict[str, Any]) -> Comm response = client.describe_addresses(**kwargs) if response.get("ResponseMetadata", {}).get("HTTPStatusCode") != HTTPStatus.OK: - AWSErrorHandler.handle_response_error(response, args.get("account_id")) + return AWSErrorHandler.handle_response_error(response, args.get("account_id")) addresses = response.get("Addresses", []) if not addresses: @@ -3737,7 +3740,7 @@ def describe_addresses_command(client: BotoClient, args: Dict[str, Any]) -> Comm ) @staticmethod - def allocate_address_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults: + def allocate_address_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults | None: """ Allocates an Elastic IP address to your AWS account. @@ -3767,7 +3770,7 @@ def allocate_address_command(client: BotoClient, args: Dict[str, Any]) -> Comman print_debug_logs(client, f"Allocating address with parameters: {kwargs}") response = client.allocate_address(**kwargs) if response.get("ResponseMetadata", {}).get("HTTPStatusCode") != HTTPStatus.OK: - AWSErrorHandler.handle_response_error(response, args.get("account_id")) + return AWSErrorHandler.handle_response_error(response, args.get("account_id")) # Serialize response to handle datetime objects response = serialize_response_with_datetime_encoding(response) @@ -3788,7 +3791,7 @@ def allocate_address_command(client: BotoClient, args: Dict[str, Any]) -> Comman ) @staticmethod - def associate_address_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults: + def associate_address_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults | None: """ Associates an Elastic IP address with an instance or a network interface. @@ -3817,7 +3820,7 @@ def associate_address_command(client: BotoClient, args: Dict[str, Any]) -> Comma response = client.associate_address(**kwargs) if response.get("ResponseMetadata", {}).get("HTTPStatusCode") != HTTPStatus.OK: - AWSErrorHandler.handle_response_error(response, args.get("account_id")) + return AWSErrorHandler.handle_response_error(response, args.get("account_id")) # Build output data output_data = { @@ -3840,7 +3843,7 @@ def associate_address_command(client: BotoClient, args: Dict[str, Any]) -> Comma ) @staticmethod - def disassociate_address_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults: + def disassociate_address_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults | None: """ Disassociates an Elastic IP address from the instance or network interface it's associated with. @@ -3858,7 +3861,7 @@ def disassociate_address_command(client: BotoClient, args: Dict[str, Any]) -> Co response = client.disassociate_address(**kwargs) if response.get("ResponseMetadata", {}).get("HTTPStatusCode") != HTTPStatus.OK: - AWSErrorHandler.handle_response_error(response, args.get("account_id")) + return AWSErrorHandler.handle_response_error(response, args.get("account_id")) return CommandResults( readable_output=f"Successfully disassociated Elastic IP address (Association ID: {args.get('association_id')})", @@ -3866,7 +3869,7 @@ def disassociate_address_command(client: BotoClient, args: Dict[str, Any]) -> Co ) @staticmethod - def release_address_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults: + def release_address_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults | None: """ Releases the specified Elastic IP address. @@ -3890,7 +3893,7 @@ def release_address_command(client: BotoClient, args: Dict[str, Any]) -> Command response = client.release_address(**kwargs) if response.get("ResponseMetadata", {}).get("HTTPStatusCode") != HTTPStatus.OK: - AWSErrorHandler.handle_response_error(response, args.get("account_id")) + return AWSErrorHandler.handle_response_error(response, args.get("account_id")) return CommandResults( readable_output=f"Successfully released Elastic IP address (Allocation ID: {args.get('allocation_id')})", @@ -3940,7 +3943,7 @@ def authorize_security_group_egress_command(client: BotoClient, args: Dict[str, return AWSErrorHandler.handle_response_error(response) @staticmethod - def describe_images_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults: + def describe_images_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults | None: """ Describes one or more Amazon Machine Images (AMIs) available to you. @@ -3995,7 +3998,7 @@ def describe_images_command(client: BotoClient, args: Dict[str, Any]) -> Command response = client.describe_images(**kwargs) if response.get("ResponseMetadata", {}).get("HTTPStatusCode") != HTTPStatus.OK: - AWSErrorHandler.handle_response_error(response, args.get("account_id")) + return AWSErrorHandler.handle_response_error(response, args.get("account_id")) images = response.get("Images", []) if not images: @@ -4029,7 +4032,7 @@ def describe_images_command(client: BotoClient, args: Dict[str, Any]) -> Command ) @staticmethod - def create_image_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults: + def create_image_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults | None: """ Creates an Amazon EBS-backed AMI from an Amazon EBS-backed instance. @@ -4072,7 +4075,7 @@ def create_image_command(client: BotoClient, args: Dict[str, Any]) -> CommandRes response = client.create_image(**kwargs) if response.get("ResponseMetadata", {}).get("HTTPStatusCode") != HTTPStatus.OK: - AWSErrorHandler.handle_response_error(response, args.get("account_id")) + return AWSErrorHandler.handle_response_error(response, args.get("account_id")) # Serialize response to handle datetime objects response = serialize_response_with_datetime_encoding(response) @@ -4100,7 +4103,7 @@ def create_image_command(client: BotoClient, args: Dict[str, Any]) -> CommandRes ) @staticmethod - def deregister_image_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults: + def deregister_image_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults | None: """ Deregisters the specified Amazon Machine Image (AMI). @@ -4120,7 +4123,7 @@ def deregister_image_command(client: BotoClient, args: Dict[str, Any]) -> Comman response = client.deregister_image(ImageId=image_id) if response.get("ResponseMetadata", {}).get("HTTPStatusCode") != HTTPStatus.OK: - AWSErrorHandler.handle_response_error(response, args.get("account_id")) + return AWSErrorHandler.handle_response_error(response, args.get("account_id")) return CommandResults( readable_output=f"Successfully deregistered AMI: {image_id}", @@ -4128,7 +4131,7 @@ def deregister_image_command(client: BotoClient, args: Dict[str, Any]) -> Comman ) @staticmethod - def copy_image_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults: + def copy_image_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults | None: """ Copy an Amazon Machine Image (AMI) from a source region to the current region. @@ -4170,7 +4173,7 @@ def copy_image_command(client: BotoClient, args: Dict[str, Any]) -> CommandResul response = client.copy_image(**kwargs) if response.get("ResponseMetadata", {}).get("HTTPStatusCode") != HTTPStatus.OK: - AWSErrorHandler.handle_response_error(response, args.get("account_id")) + return AWSErrorHandler.handle_response_error(response, args.get("account_id")) region = args.get("region", "") @@ -4283,7 +4286,7 @@ def monitor_instances_command(client: BotoClient, args: Dict[str, Any]) -> Comma response = client.monitor_instances(InstanceIds=instance_ids) if response.get("ResponseMetadata", {}).get("HTTPStatusCode") != HTTPStatus.OK: - AWSErrorHandler.handle_response_error(response, args.get("account_id")) + return AWSErrorHandler.handle_response_error(response, args.get("account_id")) instance_monitorings = response.get("InstanceMonitorings", []) @@ -4331,7 +4334,7 @@ def unmonitor_instances_command(client: BotoClient, args: Dict[str, Any]) -> Com response = client.unmonitor_instances(InstanceIds=instance_ids) if response.get("ResponseMetadata", {}).get("HTTPStatusCode") != HTTPStatus.OK: - AWSErrorHandler.handle_response_error(response, args.get("account_id")) + return AWSErrorHandler.handle_response_error(response, args.get("account_id")) instance_monitorings = response.get("InstanceMonitorings", []) @@ -4379,7 +4382,7 @@ def reboot_instances_command(client: BotoClient, args: Dict[str, Any]) -> Comman response = client.reboot_instances(InstanceIds=instance_ids) if response.get("ResponseMetadata", {}).get("HTTPStatusCode") != HTTPStatus.OK: - AWSErrorHandler.handle_response_error(response, args.get("account_id")) + return AWSErrorHandler.handle_response_error(response, args.get("account_id")) return CommandResults( readable_output=f"Successfully initiated reboot for instances: {', '.join(instance_ids)}", raw_response=response @@ -4567,7 +4570,7 @@ def describe_iam_instance_profile_associations_command(client: BotoClient, args: response = client.describe_iam_instance_profile_associations(**kwargs) if response.get("ResponseMetadata", {}).get("HTTPStatusCode") != HTTPStatus.OK: - AWSErrorHandler.handle_response_error(response, args.get("account_id")) + return AWSErrorHandler.handle_response_error(response, args.get("account_id")) associations = response.get("IamInstanceProfileAssociations", []) @@ -4609,7 +4612,7 @@ def get_password_data_command(client: BotoClient, args: Dict[str, Any]) -> Comma response = client.get_password_data(InstanceId=instance_id) if response.get("ResponseMetadata", {}).get("HTTPStatusCode") != HTTPStatus.OK: - AWSErrorHandler.handle_response_error(response, args.get("account_id")) + return AWSErrorHandler.handle_response_error(response, args.get("account_id")) # Serialize datetime response = serialize_response_with_datetime_encoding(response) @@ -4669,7 +4672,7 @@ def describe_reserved_instances_command(client: BotoClient, args: Dict[str, Any] response = client.describe_reserved_instances(**kwargs) if response.get("ResponseMetadata", {}).get("HTTPStatusCode") != HTTPStatus.OK: - AWSErrorHandler.handle_response_error(response, args.get("account_id")) + return AWSErrorHandler.handle_response_error(response, args.get("account_id")) # Serialize datetime objects serialized_response = serialize_response_with_datetime_encoding(response) @@ -4706,7 +4709,7 @@ def describe_reserved_instances_command(client: BotoClient, args: Dict[str, Any] ) @staticmethod - def describe_volumes_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults: + def describe_volumes_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults | None: """ Describes the specified EBS volumes or all of your EBS volumes. @@ -4736,7 +4739,7 @@ def describe_volumes_command(client: BotoClient, args: Dict[str, Any]) -> Comman response = client.describe_volumes(**kwargs) if response.get("ResponseMetadata", {}).get("HTTPStatusCode") != HTTPStatus.OK: - AWSErrorHandler.handle_response_error(response, args.get("account_id")) + return AWSErrorHandler.handle_response_error(response, args.get("account_id")) response = serialize_response_with_datetime_encoding(response) volumes = response.get("Volumes", []) @@ -4763,7 +4766,7 @@ def describe_volumes_command(client: BotoClient, args: Dict[str, Any]) -> Comman ) @staticmethod - def modify_volume_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults: + def modify_volume_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults | None: """ Modifies several parameters of an existing EBS volume, including volume size, volume type, and IOPS capacity. @@ -4788,7 +4791,7 @@ def modify_volume_command(client: BotoClient, args: Dict[str, Any]) -> CommandRe response = client.modify_volume(**kwargs) if response.get("ResponseMetadata", {}).get("HTTPStatusCode") != HTTPStatus.OK: - AWSErrorHandler.handle_response_error(response, args.get("account_id")) + return AWSErrorHandler.handle_response_error(response, args.get("account_id")) response = serialize_response_with_datetime_encoding(response) volume_modification = response.get("VolumeModification", {}) @@ -4826,7 +4829,7 @@ def modify_volume_command(client: BotoClient, args: Dict[str, Any]) -> CommandRe ) @staticmethod - def create_volume_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults: + def create_volume_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults | None: """ Creates an EBS volume that can be attached to an instance in the same Availability Zone. @@ -4859,7 +4862,7 @@ def create_volume_command(client: BotoClient, args: Dict[str, Any]) -> CommandRe response = client.create_volume(**kwargs) if response.get("ResponseMetadata", {}).get("HTTPStatusCode") != HTTPStatus.OK: - AWSErrorHandler.handle_response_error(response, args.get("account_id")) + return AWSErrorHandler.handle_response_error(response, args.get("account_id")) response = serialize_response_with_datetime_encoding(response) @@ -4880,7 +4883,7 @@ def create_volume_command(client: BotoClient, args: Dict[str, Any]) -> CommandRe ) @staticmethod - def attach_volume_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults: + def attach_volume_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults | None: """ Attaches an EBS volume to a running or stopped instance and exposes it to the instance with the specified device name. @@ -4901,7 +4904,7 @@ def attach_volume_command(client: BotoClient, args: Dict[str, Any]) -> CommandRe response = client.attach_volume(**kwargs) if response.get("ResponseMetadata", {}).get("HTTPStatusCode") != HTTPStatus.OK: - AWSErrorHandler.handle_response_error(response, args.get("account_id")) + return AWSErrorHandler.handle_response_error(response, args.get("account_id")) response = serialize_response_with_datetime_encoding(response) outputs = {k: v for k, v in response.items() if k != "ResponseMetadata"} @@ -4922,7 +4925,7 @@ def attach_volume_command(client: BotoClient, args: Dict[str, Any]) -> CommandRe ) @staticmethod - def detach_volume_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults: + def detach_volume_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults | None: """ Detaches an EBS volume from an instance. @@ -4945,7 +4948,7 @@ def detach_volume_command(client: BotoClient, args: Dict[str, Any]) -> CommandRe response = client.detach_volume(**kwargs) if response.get("ResponseMetadata", {}).get("HTTPStatusCode") != HTTPStatus.OK: - AWSErrorHandler.handle_response_error(response, args.get("account_id")) + return AWSErrorHandler.handle_response_error(response, args.get("account_id")) response = serialize_response_with_datetime_encoding(response) @@ -4967,7 +4970,7 @@ def detach_volume_command(client: BotoClient, args: Dict[str, Any]) -> CommandRe ) @staticmethod - def delete_volume_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults: + def delete_volume_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults | None: """ Deletes the specified EBS volume. The volume must be in the available state (not attached to an instance). @@ -4983,12 +4986,12 @@ def delete_volume_command(client: BotoClient, args: Dict[str, Any]) -> CommandRe response = client.delete_volume(VolumeId=volume_id) if response.get("ResponseMetadata", {}).get("HTTPStatusCode") != HTTPStatus.OK: - AWSErrorHandler.handle_response_error(response, args.get("account_id")) + return AWSErrorHandler.handle_response_error(response, args.get("account_id")) return CommandResults(readable_output=f"Successfully deleted volume {volume_id}") @staticmethod - def describe_snapshots_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults: + def describe_snapshots_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults | None: """ Describes one or more Amazon EBS snapshots available to you. @@ -5026,7 +5029,7 @@ def describe_snapshots_command(client: BotoClient, args: Dict[str, Any]) -> Comm response = client.describe_snapshots(**kwargs) if response.get("ResponseMetadata", {}).get("HTTPStatusCode") != HTTPStatus.OK: - AWSErrorHandler.handle_response_error(response, args.get("account_id")) + return AWSErrorHandler.handle_response_error(response, args.get("account_id")) response = serialize_response_with_datetime_encoding(response) snapshots = response.get("Snapshots", []) @@ -5064,7 +5067,7 @@ def describe_snapshots_command(client: BotoClient, args: Dict[str, Any]) -> Comm ) @staticmethod - def delete_snapshot_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults: + def delete_snapshot_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults | None: """ Deletes the specified Amazon EBS snapshot. @@ -5081,12 +5084,12 @@ def delete_snapshot_command(client: BotoClient, args: Dict[str, Any]) -> Command response = client.delete_snapshot(SnapshotId=snapshot_id) if response.get("ResponseMetadata", {}).get("HTTPStatusCode") != HTTPStatus.OK: - AWSErrorHandler.handle_response_error(response, args.get("account_id")) + return AWSErrorHandler.handle_response_error(response, args.get("account_id")) return CommandResults(readable_output=f"Successfully deleted snapshot {snapshot_id}", raw_response=response) @staticmethod - def copy_snapshot_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults: + def copy_snapshot_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults | None: """ Copies a point-in-time snapshot of an Amazon EBS volume and stores it in Amazon S3. @@ -5123,7 +5126,7 @@ def copy_snapshot_command(client: BotoClient, args: Dict[str, Any]) -> CommandRe response = client.copy_snapshot(**kwargs) if response.get("ResponseMetadata", {}).get("HTTPStatusCode") != HTTPStatus.OK: - AWSErrorHandler.handle_response_error(response, args.get("account_id")) + return AWSErrorHandler.handle_response_error(response, args.get("account_id")) outputs = {k: v for k, v in response.items() if k != "ResponseMetadata"} readable_output = tableToMarkdown( @@ -5192,7 +5195,7 @@ def snapshot_completed_waiter_command(client: BotoClient, args: Dict[str, Any]) raise DemistoException(f"Waiter error: {str(e)}") @staticmethod - def describe_launch_templates_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults: + def describe_launch_templates_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults | None: """ Describes one or more launch templates. @@ -5226,7 +5229,7 @@ def describe_launch_templates_command(client: BotoClient, args: Dict[str, Any]) response = client.describe_launch_templates(**kwargs) if response.get("ResponseMetadata", {}).get("HTTPStatusCode") != HTTPStatus.OK: - AWSErrorHandler.handle_response_error(response, args.get("account_id")) + return AWSErrorHandler.handle_response_error(response, args.get("account_id")) # Serialize response to handle datetime objects response = serialize_response_with_datetime_encoding(response) @@ -5259,7 +5262,7 @@ def describe_launch_templates_command(client: BotoClient, args: Dict[str, Any]) ) @staticmethod - def create_launch_template_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults: + def create_launch_template_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults | None: """ Creates a launch template. A launch template contains the parameters to launch an instance. @@ -5276,7 +5279,7 @@ def create_launch_template_command(client: BotoClient, args: Dict[str, Any]) -> response = client.create_launch_template(**kwargs) if response.get("ResponseMetadata", {}).get("HTTPStatusCode") != HTTPStatus.OK: - AWSErrorHandler.handle_response_error(response, args.get("account_id")) + return AWSErrorHandler.handle_response_error(response, args.get("account_id")) # Serialize response to handle datetime objects response = serialize_response_with_datetime_encoding(response) @@ -5304,7 +5307,7 @@ def create_launch_template_command(client: BotoClient, args: Dict[str, Any]) -> ) @staticmethod - def delete_launch_template_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults: + def delete_launch_template_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults | None: """ Deletes a launch template. Deleting a launch template deletes all of its versions. @@ -5331,7 +5334,7 @@ def delete_launch_template_command(client: BotoClient, args: Dict[str, Any]) -> response = client.delete_launch_template(**kwargs) if response.get("ResponseMetadata", {}).get("HTTPStatusCode") != HTTPStatus.OK: - AWSErrorHandler.handle_response_error(response, args.get("account_id")) + return AWSErrorHandler.handle_response_error(response, args.get("account_id")) # Serialize response to handle datetime objects response = serialize_response_with_datetime_encoding(response) @@ -5359,7 +5362,7 @@ def delete_launch_template_command(client: BotoClient, args: Dict[str, Any]) -> ) @staticmethod - def create_fleet_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults: + def create_fleet_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults | None: """ Launches an EC2 Fleet. @@ -5380,7 +5383,7 @@ def create_fleet_command(client: BotoClient, args: Dict[str, Any]) -> CommandRes response = client.create_fleet(**kwargs) if response.get("ResponseMetadata", {}).get("HTTPStatusCode") != HTTPStatus.OK: - AWSErrorHandler.handle_response_error(response, args.get("account_id")) + return AWSErrorHandler.handle_response_error(response, args.get("account_id")) response = serialize_response_with_datetime_encoding(response) outputs = {k: v for k, v in response.items() if k != "ResponseMetadata"} @@ -5400,7 +5403,7 @@ def create_fleet_command(client: BotoClient, args: Dict[str, Any]) -> CommandRes ) @staticmethod - def delete_fleet_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults: + def delete_fleet_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults | None: """ Deletes the specified EC2 Fleet. @@ -5420,7 +5423,7 @@ def delete_fleet_command(client: BotoClient, args: Dict[str, Any]) -> CommandRes response = client.delete_fleets(**kwargs) if response.get("ResponseMetadata", {}).get("HTTPStatusCode") != HTTPStatus.OK: - AWSErrorHandler.handle_response_error(response, args.get("account_id")) + return AWSErrorHandler.handle_response_error(response, args.get("account_id")) response = serialize_response_with_datetime_encoding(response) successful = response.get("SuccessfulFleetDeletions", []) @@ -5466,7 +5469,7 @@ def delete_fleet_command(client: BotoClient, args: Dict[str, Any]) -> CommandRes ) @staticmethod - def describe_fleets_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults: + def describe_fleets_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults | None: """ Describes one or more of your EC2 Fleets. @@ -5493,7 +5496,7 @@ def describe_fleets_command(client: BotoClient, args: Dict[str, Any]) -> Command response = client.describe_fleets(**kwargs) if response.get("ResponseMetadata", {}).get("HTTPStatusCode") != HTTPStatus.OK: - AWSErrorHandler.handle_response_error(response, args.get("account_id")) + return AWSErrorHandler.handle_response_error(response, args.get("account_id")) response = serialize_response_with_datetime_encoding(response) fleets = response.get("Fleets", []) @@ -5519,7 +5522,7 @@ def describe_fleets_command(client: BotoClient, args: Dict[str, Any]) -> Command ) @staticmethod - def describe_fleet_instances_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults: + def describe_fleet_instances_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults | None: """ Describes the running instances for the specified EC2 Fleet. @@ -5543,7 +5546,7 @@ def describe_fleet_instances_command(client: BotoClient, args: Dict[str, Any]) - response = client.describe_fleet_instances(**kwargs) if response.get("ResponseMetadata", {}).get("HTTPStatusCode") != HTTPStatus.OK: - AWSErrorHandler.handle_response_error(response, args.get("account_id")) + return AWSErrorHandler.handle_response_error(response, args.get("account_id")) response = serialize_response_with_datetime_encoding(response) active_instances = response.get("ActiveInstances", []) @@ -5555,6 +5558,7 @@ def describe_fleet_instances_command(client: BotoClient, args: Dict[str, Any]) - response_data = {k: v for k, v in response.items() if k != "ResponseMetadata"} return CommandResults( + outputs_prefix="AWS.EC2.Fleets", outputs=response_data, outputs_key_field="FleetId", readable_output=tableToMarkdown( @@ -5568,7 +5572,7 @@ def describe_fleet_instances_command(client: BotoClient, args: Dict[str, Any]) - ) @staticmethod - def modify_fleet_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults: + def modify_fleet_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults | None: """ Modifies the specified EC2 Fleet. @@ -5603,7 +5607,7 @@ def modify_fleet_command(client: BotoClient, args: Dict[str, Any]) -> CommandRes response = client.modify_fleet(**kwargs) if response.get("ResponseMetadata", {}).get("HTTPStatusCode") != HTTPStatus.OK: - AWSErrorHandler.handle_response_error(response, args.get("account_id")) + return AWSErrorHandler.handle_response_error(response, args.get("account_id")) return CommandResults( outputs_prefix="AWS.EC2.Fleets", @@ -5614,7 +5618,7 @@ def modify_fleet_command(client: BotoClient, args: Dict[str, Any]) -> CommandRes ) @staticmethod - def delete_vpc_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults: + def delete_vpc_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults | None: """ Deletes the specified VPC. @@ -5630,12 +5634,12 @@ def delete_vpc_command(client: BotoClient, args: Dict[str, Any]) -> CommandResul response = client.delete_vpc(VpcId=vpc_id) if response.get("ResponseMetadata", {}).get("HTTPStatusCode") != HTTPStatus.OK: - AWSErrorHandler.handle_response_error(response, args.get("account_id")) + return AWSErrorHandler.handle_response_error(response, args.get("account_id")) return CommandResults(readable_output=f"Successfully deleted VPC {vpc_id}") @staticmethod - def create_vpc_endpoint_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults: + def create_vpc_endpoint_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults | None: """ Creates a VPC endpoint for a specified service. An endpoint enables you to create a private connection between your VPC and the service. @@ -5683,7 +5687,7 @@ def create_vpc_endpoint_command(client: BotoClient, args: Dict[str, Any]) -> Com response = client.create_vpc_endpoint(**kwargs) if response.get("ResponseMetadata", {}).get("HTTPStatusCode") != HTTPStatus.OK: - AWSErrorHandler.handle_response_error(response, args.get("account_id")) + return AWSErrorHandler.handle_response_error(response, args.get("account_id")) response = serialize_response_with_datetime_encoding(response) vpc_endpoint = response.get("VpcEndpoint", {}) @@ -5703,7 +5707,7 @@ def create_vpc_endpoint_command(client: BotoClient, args: Dict[str, Any]) -> Com ) @staticmethod - def describe_internet_gateways_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults: + def describe_internet_gateways_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults | None: """ Describes one or more of your internet gateways. @@ -5727,7 +5731,7 @@ def describe_internet_gateways_command(client: BotoClient, args: Dict[str, Any]) response = client.describe_internet_gateways(**kwargs) if response.get("ResponseMetadata", {}).get("HTTPStatusCode") != HTTPStatus.OK: - AWSErrorHandler.handle_response_error(response, args.get("account_id")) + return AWSErrorHandler.handle_response_error(response, args.get("account_id")) response = serialize_response_with_datetime_encoding(response) internet_gateways = response.get("InternetGateways", []) @@ -5767,7 +5771,7 @@ def describe_internet_gateways_command(client: BotoClient, args: Dict[str, Any]) ) @staticmethod - def detach_internet_gateway_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults: + def detach_internet_gateway_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults | None: """ Detaches an internet gateway from a VPC, disabling connectivity between the internet and the VPC. @@ -5785,12 +5789,12 @@ def detach_internet_gateway_command(client: BotoClient, args: Dict[str, Any]) -> response = client.detach_internet_gateway(InternetGatewayId=internet_gateway_id, VpcId=vpc_id) if response.get("ResponseMetadata", {}).get("HTTPStatusCode") != HTTPStatus.OK: - AWSErrorHandler.handle_response_error(response, args.get("account_id")) + return AWSErrorHandler.handle_response_error(response, args.get("account_id")) return CommandResults(readable_output=f"Successfully detached internet gateway {internet_gateway_id} from VPC {vpc_id}") @staticmethod - def delete_internet_gateway_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults: + def delete_internet_gateway_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults | None: """ Deletes the specified internet gateway. @@ -5806,12 +5810,12 @@ def delete_internet_gateway_command(client: BotoClient, args: Dict[str, Any]) -> response = client.delete_internet_gateway(InternetGatewayId=internet_gateway_id) if response.get("ResponseMetadata", {}).get("HTTPStatusCode") != HTTPStatus.OK: - AWSErrorHandler.handle_response_error(response, args.get("account_id")) + return AWSErrorHandler.handle_response_error(response, args.get("account_id")) return CommandResults(readable_output=f"Successfully deleted internet gateway {internet_gateway_id}") @staticmethod - def delete_subnet_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults: + def delete_subnet_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults | None: """ Deletes the specified subnet. @@ -5827,12 +5831,12 @@ def delete_subnet_command(client: BotoClient, args: Dict[str, Any]) -> CommandRe response = client.delete_subnet(SubnetId=subnet_id) if response.get("ResponseMetadata", {}).get("HTTPStatusCode") != HTTPStatus.OK: - AWSErrorHandler.handle_response_error(response, args.get("account_id")) + return AWSErrorHandler.handle_response_error(response, args.get("account_id")) return CommandResults(readable_output=f"Successfully deleted subnet {subnet_id}") @staticmethod - def create_network_acl_entry_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults: + def create_network_acl_entry_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults | None: """ Creates an entry (a rule) in a network ACL with the specified rule number. @@ -5871,12 +5875,12 @@ def create_network_acl_entry_command(client: BotoClient, args: Dict[str, Any]) - response = client.create_network_acl_entry(**kwargs) if response.get("ResponseMetadata", {}).get("HTTPStatusCode") != HTTPStatus.OK: - AWSErrorHandler.handle_response_error(response, args.get("account_id")) + return AWSErrorHandler.handle_response_error(response, args.get("account_id")) return CommandResults(readable_output=f"Successfully created network ACL entry for {args.get('network_acl_id')}") @staticmethod - def describe_key_pairs_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults: + def describe_key_pairs_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults | None: """ Describes the specified key pairs or all of your key pairs. Args: @@ -5897,7 +5901,7 @@ def describe_key_pairs_command(client: BotoClient, args: Dict[str, Any]) -> Comm print_debug_logs(client, f"Describing key pairs with parameters: {kwargs}") response = client.describe_key_pairs(**kwargs) if response.get("ResponseMetadata", {}).get("HTTPStatusCode") != HTTPStatus.OK: - AWSErrorHandler.handle_response_error(response, args.get("account_id")) + return AWSErrorHandler.handle_response_error(response, args.get("account_id")) response = serialize_response_with_datetime_encoding(response) key_pairs = response.get("KeyPairs", []) @@ -5920,7 +5924,7 @@ def describe_key_pairs_command(client: BotoClient, args: Dict[str, Any]) -> Comm ) @staticmethod - def allocate_hosts_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults: + def allocate_hosts_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults | None: """ Allocates a Dedicated Host to your account. Args: @@ -5950,7 +5954,7 @@ def allocate_hosts_command(client: BotoClient, args: Dict[str, Any]) -> CommandR print_debug_logs(client, f"Allocating Dedicated Hosts with parameters: {kwargs}") response = client.allocate_hosts(**kwargs) if response.get("ResponseMetadata", {}).get("HTTPStatusCode") != HTTPStatus.OK: - AWSErrorHandler.handle_response_error(response, args.get("account_id")) + return AWSErrorHandler.handle_response_error(response, args.get("account_id")) host_ids = response.get("HostIds", []) return CommandResults( @@ -5961,7 +5965,7 @@ def allocate_hosts_command(client: BotoClient, args: Dict[str, Any]) -> CommandR ) @staticmethod - def release_hosts_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults: + def release_hosts_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults | None: """ Releases the specified Dedicated Hosts. Args: @@ -5975,7 +5979,7 @@ def release_hosts_command(client: BotoClient, args: Dict[str, Any]) -> CommandRe response = client.release_hosts(HostIds=host_ids) if response.get("ResponseMetadata", {}).get("HTTPStatusCode") != HTTPStatus.OK: - AWSErrorHandler.handle_response_error(response, args.get("account_id")) + return AWSErrorHandler.handle_response_error(response, args.get("account_id")) outputs = remove_empty_elements( { @@ -5998,7 +6002,7 @@ def release_hosts_command(client: BotoClient, args: Dict[str, Any]) -> CommandRe ) @staticmethod - def create_traffic_mirror_session_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults: + def create_traffic_mirror_session_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults | None: """ Creates a Traffic Mirror session. Args: @@ -6023,7 +6027,7 @@ def create_traffic_mirror_session_command(client: BotoClient, args: Dict[str, An print_debug_logs(client, f"Creating Traffic Mirror session with parameters: {kwargs}") response = client.create_traffic_mirror_session(**kwargs) if response.get("ResponseMetadata", {}).get("HTTPStatusCode") != HTTPStatus.OK: - AWSErrorHandler.handle_response_error(response, args.get("account_id")) + return AWSErrorHandler.handle_response_error(response, args.get("account_id")) response = serialize_response_with_datetime_encoding(response) session = response.get("TrafficMirrorSession", {}) @@ -6208,7 +6212,7 @@ def associate_access_policy_command(client: BotoClient, args: Dict[str, Any]) -> ) @staticmethod - def update_access_entry_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults: + def update_access_entry_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults | None: """ Updates an existing Access Entry for an Amazon EKS cluster. @@ -6240,7 +6244,7 @@ def update_access_entry_command(client: BotoClient, args: Dict[str, Any]) -> Com response = client.update_access_entry(**kwargs) if response.get("ResponseMetadata", {}).get("HTTPStatusCode") != HTTPStatus.OK: - AWSErrorHandler.handle_response_error(response, args.get("account_id")) + return AWSErrorHandler.handle_response_error(response, args.get("account_id")) access_entry = serialize_response_with_datetime_encoding(response.get("accessEntry", {})) @@ -6262,7 +6266,7 @@ def update_access_entry_command(client: BotoClient, args: Dict[str, Any]) -> Com ) @staticmethod - def create_access_entry_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults: + def create_access_entry_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults | None: """ Creates a new Access Entry for an Amazon EKS cluster. @@ -6293,7 +6297,7 @@ def create_access_entry_command(client: BotoClient, args: Dict[str, Any]) -> Com response = client.create_access_entry(**kwargs) if response.get("ResponseMetadata", {}).get("HTTPStatusCode") != HTTPStatus.OK: - AWSErrorHandler.handle_response_error(response, args.get("account_id")) + return AWSErrorHandler.handle_response_error(response, args.get("account_id")) access_entry = serialize_response_with_datetime_encoding(response.get("accessEntry", {})) @@ -6315,7 +6319,7 @@ def create_access_entry_command(client: BotoClient, args: Dict[str, Any]) -> Com ) @staticmethod - def list_clusters_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults: + def list_clusters_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults | None: """ Returns a list of EKS clusters owned by the authenticated sender of the request. @@ -6335,7 +6339,7 @@ def list_clusters_command(client: BotoClient, args: Dict[str, Any]) -> CommandRe response = client.list_clusters(**kwargs) if response.get("ResponseMetadata", {}).get("HTTPStatusCode") != HTTPStatus.OK: - AWSErrorHandler.handle_response_error(response, args.get("account_id")) + return AWSErrorHandler.handle_response_error(response, args.get("account_id")) clusters = response.get("clusters", []) next_token = response.get("nextToken") @@ -6639,7 +6643,7 @@ def modify_event_subscription_command(client: BotoClient, args: Dict[str, Any]) raise DemistoException(f"Error: {str(e)}") @staticmethod - def describe_db_instances_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults: + def describe_db_instances_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults | None: """ Describes provisioned RDS instances. @@ -6667,7 +6671,7 @@ def describe_db_instances_command(client: BotoClient, args: Dict[str, Any]) -> C response = serialize_response_with_datetime_encoding(response) if response.get("ResponseMetadata", {}).get("HTTPStatusCode") != HTTPStatus.OK: - AWSErrorHandler.handle_response_error(response, args.get("account_id")) + return AWSErrorHandler.handle_response_error(response, args.get("account_id")) db_instances = response.get("DBInstances", []) if not db_instances: @@ -6702,7 +6706,7 @@ class Redshift: service = AWSServices.Redshift @staticmethod - def modify_cluster_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults: + def modify_cluster_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults | None: """ Modifies the settings of a cluster. @@ -6747,7 +6751,7 @@ def modify_cluster_command(client: BotoClient, args: Dict[str, Any]) -> CommandR response = client.modify_cluster(**kwargs) if response.get("ResponseMetadata", {}).get("HTTPStatusCode") != HTTPStatus.OK: - AWSErrorHandler.handle_response_error(response, args.get("account_id")) + return AWSErrorHandler.handle_response_error(response, args.get("account_id")) response = serialize_response_with_datetime_encoding(response) cluster_data = response.get("Cluster", {}) @@ -7450,7 +7454,7 @@ class Lambda: service = AWSServices.LAMBDA @staticmethod - def get_function_configuration_command(client: BotoClient, args: Dict[str, Any]): + def get_function_configuration_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults: """ Retrieves the configuration information for a Lambda function. @@ -7501,7 +7505,7 @@ def get_function_configuration_command(client: BotoClient, args: Dict[str, Any]) ) @staticmethod - def get_function_url_configuration_command(client: BotoClient, args: Dict[str, Any]): + def get_function_url_configuration_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults: """ Retrieves the configuration for a Lambda function URL. @@ -7546,7 +7550,7 @@ def get_function_url_configuration_command(client: BotoClient, args: Dict[str, A ) @staticmethod - def update_function_url_configuration_command(client: BotoClient, args: Dict[str, Any]): + def update_function_url_configuration_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults: """ Updates the configuration for a Lambda function URL. @@ -7648,7 +7652,7 @@ def _parse_policy_response(data: dict[str, Any]) -> tuple[dict, list | None]: return policy_table, statements_table @staticmethod - def get_policy_command(client: BotoClient, args: Dict[str, Any]): + def get_policy_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults | None: """ Retrieves the policy for a Lambda function from AWS and parses it into a dictionary. @@ -7666,7 +7670,7 @@ def get_policy_command(client: BotoClient, args: Dict[str, Any]): response = client.get_policy(**kwargs) if response.get("ResponseMetadata", {}).get("HTTPStatusCode") != HTTPStatus.OK: - AWSErrorHandler.handle_response_error(response, args.get("account_id")) + return AWSErrorHandler.handle_response_error(response, args.get("account_id")) fixed_response = {} fixed_response["AccountId"] = args.get("account_id", "") @@ -7746,7 +7750,7 @@ def invoke_command(client: BotoClient, args: Dict[str, Any]): ) @staticmethod - def get_function_command(client: BotoClient, args: Dict[str, Any]): + def get_function_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults | None: """ Retrieves information about a Lambda function including configuration, code location, and metadata. @@ -7771,7 +7775,7 @@ def get_function_command(client: BotoClient, args: Dict[str, Any]): response = client.get_function(**kwargs) if response.get("ResponseMetadata", {}).get("HTTPStatusCode") != HTTPStatus.OK: - AWSErrorHandler.handle_response_error(response, args.get("account_id")) + return AWSErrorHandler.handle_response_error(response, args.get("account_id")) # Serialize response with datetime encoding response = serialize_response_with_datetime_encoding(response) @@ -7803,7 +7807,7 @@ def get_function_command(client: BotoClient, args: Dict[str, Any]): ) @staticmethod - def list_functions_command(client: BotoClient, args: Dict[str, Any]): + def list_functions_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults | None: """ Lists Lambda functions in the specified region. @@ -7829,7 +7833,7 @@ def list_functions_command(client: BotoClient, args: Dict[str, Any]): response = client.list_functions(**pagination_kwargs) if response.get("ResponseMetadata", {}).get("HTTPStatusCode") != HTTPStatus.OK: - AWSErrorHandler.handle_response_error(response, args.get("account_id")) + return AWSErrorHandler.handle_response_error(response, args.get("account_id")) # Serialize response with datetime encoding serialized_response = serialize_response_with_datetime_encoding(response) @@ -7864,7 +7868,7 @@ def list_functions_command(client: BotoClient, args: Dict[str, Any]): ) @staticmethod - def list_aliases_command(client: BotoClient, args: Dict[str, Any]): + def list_aliases_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults | None: """ Lists aliases for a Lambda function. @@ -7896,7 +7900,7 @@ def list_aliases_command(client: BotoClient, args: Dict[str, Any]): response = client.list_aliases(**kwargs) if response.get("ResponseMetadata", {}).get("HTTPStatusCode") != HTTPStatus.OK: - AWSErrorHandler.handle_response_error(response, args.get("account_id")) + return AWSErrorHandler.handle_response_error(response, args.get("account_id")) # Serialize response with datetime encoding serialized_response = serialize_response_with_datetime_encoding(response) @@ -7928,7 +7932,7 @@ def list_aliases_command(client: BotoClient, args: Dict[str, Any]): ) @staticmethod - def get_account_settings_command(client: BotoClient, args: Dict[str, Any]): + def get_account_settings_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults | None: """ Retrieves account settings for AWS Lambda. @@ -7946,7 +7950,7 @@ def get_account_settings_command(client: BotoClient, args: Dict[str, Any]): response = client.get_account_settings() if response.get("ResponseMetadata", {}).get("HTTPStatusCode") != HTTPStatus.OK: - AWSErrorHandler.handle_response_error(response, args.get("account_id")) + return AWSErrorHandler.handle_response_error(response, args.get("account_id")) # Serialize response with datetime encoding serialized_response = serialize_response_with_datetime_encoding(response) @@ -7993,7 +7997,7 @@ def get_account_settings_command(client: BotoClient, args: Dict[str, Any]): ) @staticmethod - def list_versions_by_function_command(client: BotoClient, args: Dict[str, Any]): + def list_versions_by_function_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults | None: """ Lists the versions of a Lambda function and returns the results. @@ -8022,7 +8026,7 @@ def list_versions_by_function_command(client: BotoClient, args: Dict[str, Any]): response = client.list_versions_by_function(**kwargs) if response.get("ResponseMetadata", {}).get("HTTPStatusCode") != HTTPStatus.OK: - AWSErrorHandler.handle_response_error(response, args.get("account_id")) + return AWSErrorHandler.handle_response_error(response, args.get("account_id")) # Serialize response with datetime encoding serialized_response = serialize_response_with_datetime_encoding(response) @@ -8090,7 +8094,7 @@ def delete_function_url_config_command(client: BotoClient, args: Dict[str, Any]) return AWSErrorHandler.handle_response_error(response) @staticmethod - def create_function_command(client: BotoClient, args: Dict[str, Any]): + def create_function_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults | None: """ Creates a Lambda function from AWS. @@ -8126,7 +8130,7 @@ def create_function_command(client: BotoClient, args: Dict[str, Any]): response = client.create_function(**kwargs) if response.get("ResponseMetadata", {}).get("HTTPStatusCode") != HTTPStatus.CREATED: - AWSErrorHandler.handle_response_error(response, args.get("account_id")) + return AWSErrorHandler.handle_response_error(response, args.get("account_id")) # Serialize response with datetime encoding response = serialize_response_with_datetime_encoding(response) @@ -8156,7 +8160,7 @@ def create_function_command(client: BotoClient, args: Dict[str, Any]): ) @staticmethod - def list_layer_versions_command(client: BotoClient, args: Dict[str, Any]): + def list_layer_versions_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults | None: """ Lists the versions of an Lambda layer. @@ -8193,7 +8197,7 @@ def list_layer_versions_command(client: BotoClient, args: Dict[str, Any]): response = client.list_layer_versions(**kwargs) if response.get("ResponseMetadata", {}).get("HTTPStatusCode") != HTTPStatus.OK: - AWSErrorHandler.handle_response_error(response, args.get("account_id")) + return AWSErrorHandler.handle_response_error(response, args.get("account_id")) # Serialize response with datetime encoding serialized_response = serialize_response_with_datetime_encoding(response) @@ -8224,7 +8228,7 @@ def list_layer_versions_command(client: BotoClient, args: Dict[str, Any]): ) @staticmethod - def delete_function_command(client: BotoClient, args: Dict[str, Any]): + def delete_function_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults | None: """ Deletes a Lambda function from AWS. @@ -8252,7 +8256,7 @@ def delete_function_command(client: BotoClient, args: Dict[str, Any]): return AWSErrorHandler.handle_response_error(response) @staticmethod - def delete_layer_version_command(client: BotoClient, args: Dict[str, Any]): + def delete_layer_version_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults | None: """ Deletes a version of a Lambda layer. @@ -8279,7 +8283,7 @@ def delete_layer_version_command(client: BotoClient, args: Dict[str, Any]): return AWSErrorHandler.handle_response_error(response) @staticmethod - def publish_layer_version_command(client: BotoClient, args: Dict[str, Any]): + def publish_layer_version_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults | None: """ Creates a Lambda layer from a ZIP archive. @@ -8334,7 +8338,7 @@ def publish_layer_version_command(client: BotoClient, args: Dict[str, Any]): response = client.publish_layer_version(**kwargs) if response.get("ResponseMetadata", {}).get("HTTPStatusCode") not in [HTTPStatus.OK, HTTPStatus.CREATED]: - AWSErrorHandler.handle_response_error(response, args.get("account_id")) + return AWSErrorHandler.handle_response_error(response, args.get("account_id")) # Serialize response with datetime encoding outputs = serialize_response_with_datetime_encoding(response) @@ -8366,7 +8370,7 @@ def publish_layer_version_command(client: BotoClient, args: Dict[str, Any]): ) @staticmethod - def update_function_configuration_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults: + def update_function_configuration_command(client: BotoClient, args: Dict[str, Any]) -> CommandResults | None: """ Updates the configuration of a Lambda function. @@ -8385,7 +8389,7 @@ def update_function_configuration_command(client: BotoClient, args: Dict[str, An response = client.update_function_configuration(**kwargs) if response.get("ResponseMetadata", {}).get("HTTPStatusCode") != HTTPStatus.OK: - AWSErrorHandler.handle_response_error(response, args.get("account_id")) + return AWSErrorHandler.handle_response_error(response, args.get("account_id")) outputs = copy.deepcopy(response) if outputs.get("ResponseMetadata", {}): @@ -9352,7 +9356,7 @@ def command_list_command(client: BotoClient, args: Dict[str, Any]) -> CommandRes response = client.list_commands(**kwargs) if response.get("ResponseMetadata", {}).get("HTTPStatusCode") != HTTPStatus.OK: - AWSErrorHandler.handle_response_error(response, args.get("account_id")) + return AWSErrorHandler.handle_response_error(response, args.get("account_id")) response = serialize_response_with_datetime_encoding(response) print_debug_logs(client, f"Retry {response=}") diff --git a/Packs/AWS/Integrations/AWS/AWS.yml b/Packs/AWS/Integrations/AWS/AWS.yml index 232538080044..2b412f2a21df 100644 --- a/Packs/AWS/Integrations/AWS/AWS.yml +++ b/Packs/AWS/Integrations/AWS/AWS.yml @@ -1041,17 +1041,17 @@ script: description: The bucket name for which to get the website configuration. required: true outputs: - - contextPath: AWS.S3-Buckets.BucketWebsite.ErrorDocument + - contextPath: AWS.S3.Buckets.BucketWebsite.ErrorDocument description: The object key name of the website error document to use for 4XX class errors. type: Object - - contextPath: AWS.S3-Buckets.BucketWebsite.IndexDocument + - contextPath: AWS.S3.Buckets.BucketWebsite.IndexDocument description: The name of the index document for the website (for example index.html). type: Object - - contextPath: AWS.S3-Buckets.BucketWebsite.RedirectAllRequestsTo - description: Specifies the redirect behavior of all requests to a website endpoint of an Amazon S3 bucket. + - contextPath: AWS.S3.Buckets.BucketWebsite.RedirectAllRequestsTo + description: The redirect behavior of all requests to a website endpoint of an Amazon S3 bucket. type: Object - - contextPath: AWS.S3-Buckets.BucketWebsite.RoutingRules - description: Rules that define when a redirect is applied and the redirect behavior. + - contextPath: AWS.S3.Buckets.BucketWebsite.RoutingRules + description: The rules that define when a redirect is applied and the redirect behavior. type: Array - name: aws-s3-bucket-acl-get @@ -1105,11 +1105,11 @@ script: description: Specifies the S3 bucket whose ACL is being requested. required: true outputs: - - contextPath: AWS.S3-Buckets.BucketAcl.Grants - description: A list of grants. + - contextPath: AWS.S3.Buckets.BucketAcl.Grants + description: The list of grants. type: Array - - contextPath: AWS.S3-Buckets.BucketAcl.Owner - description: Container for the bucket owner's display name and ID. + - contextPath: AWS.S3.Buckets.BucketAcl.Owner + description: The container for the bucket owner's display name and ID. type: Object - name: aws-iam-account-password-policy-get @@ -5167,7 +5167,9 @@ script: description: The ID of the AWS account with which the EC2 instance is associated. This key is only present when the parameter "AWS organization accounts" is provided. type: string - - arguments: + - name: aws-ec2-ipam-resource-discoveries-describe + description: "Returns details for IPAM resource discoveries. A resource discovery is an IPAM component that enables IPAM to manage and monitor resources owned by the account. Required IAM permission: ec2:DescribeIpamResourceDiscoveries." + arguments: - description: A comma-separated list of the IPAM resource discovery IDs. name: ipam_resource_discovery_ids - description: One or more filters separated by ';' (for example, name=,values=;name=,values=). See AWS documentation for details & filter options. @@ -5177,12 +5179,13 @@ script: defaultValue: 50 - description: The token for the next set of results. name: next_token - - description: The Amazon Web Services region for the IP address. - name: address_region - description: The AWS account ID. Required for Cortex Platform (which includes Cortex XSIAM version >=3.0 and Cortex Cloud). Optional for Cortex XSOAR and Cortex XSIAM version < 3.0, where it can be retrieved from the integration configuration. name: account_id required: false required:platform: true + - description: The Amazon Web Services region for the IP address. + name: address_region + hidden: true - description: The AWS region. Required for Cortex Platform (which includes Cortex XSIAM version >=3.0 and Cortex Cloud). Optional for Cortex XSOAR and Cortex XSIAM version < 3.0, where it can be retrieved from the integration configuration. name: region required: false @@ -5236,10 +5239,10 @@ script: - contextPath: AWS.EC2.IpamResourceDiscoveries.AccountId description: The ID of the AWS account associated with the EC2 instance. This key is only present when the parameter "AWS organization accounts" is provided. type: string - description: "Returns details for IPAM resource discoveries. A resource discovery is an IPAM component that enables IPAM to manage and monitor resources owned by the account. Required IAM permission: ec2:DescribeIpamResourceDiscoveries." - name: aws-ec2-ipam-resource-discoveries-describe - - arguments: + - name: aws-ec2-ipam-resource-discovery-associations-describe + description: "Returns details for a resource discovery associated with an Amazon VPC IPAM. Required IAM permission: ec2:DescribeIpamResourceDiscoveryAssociations." + arguments: - description: A comma-separated list of the resource discovery association IDs. name: ipam_resource_discovery_association_ids - description: One or more filters separated by ';' (for example, name=,values=;name=,values=). See AWS documentation for details & filter options. @@ -5249,12 +5252,13 @@ script: defaultValue: 50 - description: The token for the next set of results. name: next_token - - description: The Amazon Web Services region for the IP address. - name: address_region - description: The AWS account ID. Required for Cortex Platform (which includes Cortex XSIAM version >=3.0 and Cortex Cloud). Optional for Cortex XSOAR and Cortex XSIAM version < 3.0, where it can be retrieved from the integration configuration. name: account_id required: false required:platform: true + - description: The Amazon Web Services region for the IP address. + name: address_region + hidden: true - description: The AWS region. Required for Cortex Platform (which includes Cortex XSIAM version >=3.0 and Cortex Cloud). Optional for Cortex XSOAR and Cortex XSIAM version < 3.0, where it can be retrieved from the integration configuration. name: region required: false @@ -5308,8 +5312,6 @@ script: - contextPath: AWS.EC2.IpamResourceDiscoveryAssociations.AccountId description: The ID of the AWS account associated with the EC2 instance. This key is only present when the parameter "AWS organization accounts" is provided. type: string - description: "Returns details for a resource discovery associated with an Amazon VPC IPAM. Required IAM permission: ec2:DescribeIpamResourceDiscoveryAssociations." - name: aws-ec2-ipam-resource-discovery-associations-describe - name: aws-cloudtrail-trail-update description: 'Updates trail settings for event logging and log file handling. Designates an existing bucket for log delivery without requiring a service restart. Note: This must be executed from the region where the trail was created. Required IAM permission: cloudtrail:UpdateTrail.' @@ -7090,16 +7092,16 @@ script: - contextPath: AWS.CloudTrail.Trails.CloudWatchLogsRoleArn description: The role assumed by CloudTrail to write logs to the CloudWatch log group. type: string - - contextPath: AWS.CloudTrail.KmsKeyId + - contextPath: AWS.CloudTrail.Trails.KmsKeyId description: The KMS key ID that encrypts logs delivered by CloudTrail. type: string - - contextPath: AWS.CloudTrail.HasCustomEventSelectors + - contextPath: AWS.CloudTrail.Trails.HasCustomEventSelectors description: Specifies if the trail has custom event selectors. type: boolean - - contextPath: AWS.CloudTrail.HasInsightSelectors + - contextPath: AWS.CloudTrail.Trails.HasInsightSelectors description: Specifies whether a trail has insight types specified in an InsightSelector list. type: boolean - - contextPath: AWS.CloudTrail.IsOrganizationTrail + - contextPath: AWS.CloudTrail.Trails.IsOrganizationTrail description: Specifies whether the trail is an organization trail. type: boolean - name: aws-s3-bucket-policy-delete @@ -7207,19 +7209,19 @@ script: - description: The account ID of the expected bucket owner. If the account ID that you provide does not match the actual owner of the bucket, the request fails. name: expected_bucket_owner outputs: - - contextPath: AWS.S3-Buckets.BucketName - description: Name of the S3 bucket. + - contextPath: AWS.S3.Buckets.BucketName + description: The name of the S3 bucket. type: string - - contextPath: AWS.S3-Buckets.PublicAccessBlock.BlockPublicAcls + - contextPath: AWS.S3.Buckets.PublicAccessBlock.BlockPublicAcls description: Whether Amazon S3 blocks public access control lists (ACLs) for this bucket and objects in this bucket. type: boolean - - contextPath: AWS.S3-Buckets.PublicAccessBlock.IgnorePublicAcls + - contextPath: AWS.S3.Buckets.PublicAccessBlock.IgnorePublicAcls description: Whether Amazon S3 ignores public ACLs for this bucket and objects in this bucket. type: boolean - - contextPath: AWS.S3-Buckets.PublicAccessBlock.BlockPublicPolicy + - contextPath: AWS.S3.Buckets.PublicAccessBlock.BlockPublicPolicy description: Whether Amazon S3 blocks public bucket policies for this bucket. type: boolean - - contextPath: AWS.S3-Buckets.PublicAccessBlock.RestrictPublicBuckets + - contextPath: AWS.S3.Buckets.PublicAccessBlock.RestrictPublicBuckets description: Whether Amazon S3 restricts public bucket policies for this bucket. type: boolean @@ -8103,11 +8105,11 @@ script: name: bucket required: true outputs: - - contextPath: AWS.S3-Buckets.BucketName - description: Name of the S3 bucket. + - contextPath: AWS.S3.Buckets.BucketName + description: The name of the S3 bucket. type: string - - contextPath: AWS.S3-Buckets.ServerSideEncryptionConfiguration.Rules - description: Container for information about a particular server-side encryption configuration rule. + - contextPath: AWS.S3.Buckets.ServerSideEncryptionConfiguration.Rules + description: The container for information about a particular server-side encryption configuration rule. type: array - name: aws-s3-bucket-policy-get @@ -8163,29 +8165,32 @@ script: - description: The account ID of the expected bucket owner. If the account ID that you provide does not match the actual owner of the bucket, the request fails. name: expected_bucket_owner outputs: - - contextPath: AWS.S3-Buckets.BucketName - description: Name of the S3 bucket. + - contextPath: AWS.S3.Buckets.BucketName + description: The name of the S3 bucket. type: string - - contextPath: AWS.S3-Buckets.Policy.Version + - contextPath: AWS.S3.Buckets.Policy.Version description: The version of the policy. type: string - - contextPath: AWS.S3-Buckets.Policy.Id + - contextPath: AWS.S3.Buckets.Policy.Id description: The id of the policy. type: string - - contextPath: AWS.S3-Buckets.Policy.Statement.Sid - description: Identifier of the policy statement. + - contextPath: AWS.S3.Buckets.Policy.Statement.Sid + description: The identifier of the policy statement. type: string - - contextPath: AWS.S3-Buckets.Policy.Statement.Effect - description: Specifies whether the statement results in an allow or an explicit deny. + - contextPath: AWS.S3.Buckets.Policy.Statement.Effect + description: The specification of whether the statement results in an allow or an explicit deny. type: string - - contextPath: AWS.S3-Buckets.Policy.Statement.Principal - description: Specify the principal that is allowed or denied access to a resource. - - contextPath: AWS.S3-Buckets.Policy.Statement.Action - description: Describes the specific action or actions that will be allowed or denied. - - contextPath: AWS.S3-Buckets.Policy.Statement.Resource - description: Defines the object or objects that the statement applies to. - - contextPath: AWS.S3-Buckets.Policy.Statement.Condition - description: Specify conditions for when a policy is in effect. + - contextPath: AWS.S3.Buckets.Policy.Statement.Principal + description: The principal that is allowed or denied access to a resource. + type: Unknown + - contextPath: AWS.S3.Buckets.Policy.Statement.Action + description: The specific action or actions that will be allowed or denied. + type: Unknown + - contextPath: AWS.S3.Buckets.Policy.Statement.Resource + description: The object or objects that the statement applies to. + type: Unknown + - contextPath: AWS.S3.Buckets.Policy.Statement.Condition + description: The conditions for when a policy is in effect. type: string - name: aws-lambda-function-configuration-get @@ -21114,7 +21119,7 @@ script: script: '-' type: python subtype: python3 - dockerimage: demisto/boto3py3:1.0.0.10221838 + dockerimage: demisto/boto3py3:1.0.0.12887184 supportsquickactions: true fromversion: 5.5.0 source: 'cooc' diff --git a/Packs/AWS/Integrations/AWS/AWS_test.py b/Packs/AWS/Integrations/AWS/AWS_test.py index dbab17933940..e9c95340e51f 100644 --- a/Packs/AWS/Integrations/AWS/AWS_test.py +++ b/Packs/AWS/Integrations/AWS/AWS_test.py @@ -6,6 +6,38 @@ import pytest from CommonServerPython import CommandResults, DemistoException +import ast +import os +import re + +QUICK_ACTION_SUFFIX = "-quick-action" +_INTEGRATION_DIR = os.path.dirname(os.path.abspath(__file__)) +_YML_PATH = os.path.join(_INTEGRATION_DIR, "AWS.yml") +_PY_PATH = os.path.join(_INTEGRATION_DIR, "AWS.py") + +# Platform-standard arguments that are resolved centrally rather than read with +# args.get(...) inside each command handler: +# - region / account_id -> consumed by get_service_client() when building the +# boto3 client and by the multi-account fan-out in execute_aws_command(). +# - limit / next_token -> consumed by the shared build_pagination_kwargs(). +# - polling_timeout / interval_in_seconds / hide_polling_output -> consumed by +# the @polling_function decorator, not by the handler body. +PLATFORM_STANDARD_ARGS = { + "region", + "account_id", + "limit", + "next_token", + "polling_timeout", + "interval_in_seconds", + "hide_polling_output", +} + +# Output roots produced by platform helpers rather than by an explicit +# outputs_prefix in the handler. `File.*` entries come from fileResult(). +PLATFORM_STANDARD_OUTPUT_ROOTS = ("File.",) + +# An ``ast`` node that declares a function (sync or async). +FunctionNode = ast.FunctionDef | ast.AsyncFunctionDef def test_parse_resource_ids_with_valid_input(): @@ -2119,7 +2151,7 @@ def test_s3_get_bucket_policy_command_success(mocker): result = S3.get_bucket_policy_command(mock_client, args) assert isinstance(result, CommandResults) - assert result.outputs_prefix == "AWS.S3-Buckets" + assert result.outputs_prefix == "AWS.S3.Buckets" assert result.outputs_key_field == "BucketName" assert result.outputs["BucketName"] == "test-bucket" assert result.outputs["Policy"] == policy_document @@ -2319,7 +2351,7 @@ def test_s3_get_bucket_encryption_command_success(mocker): result = S3.get_bucket_encryption_command(mock_client, args) assert isinstance(result, CommandResults) - assert result.outputs_prefix == "AWS.S3-Buckets" + assert result.outputs_prefix == "AWS.S3.Buckets" assert result.outputs_key_field == "BucketName" assert result.outputs["BucketName"] == "test-bucket" assert "ServerSideEncryptionConfiguration" in result.outputs @@ -2484,7 +2516,7 @@ def test_s3_get_public_access_block_command_success(mocker): result = S3.get_public_access_block_command(mock_client, args) assert isinstance(result, CommandResults) - assert result.outputs_prefix == "AWS.S3-Buckets" + assert result.outputs_prefix == "AWS.S3.Buckets" assert result.outputs_key_field == "BucketName" assert result.outputs["BucketName"] == "test-bucket" assert result.outputs["PublicAccessBlock"] == public_access_block_config @@ -6198,6 +6230,46 @@ def test_get_bucket_website_command_failure(mocker): mock_error_handler.assert_called_once() +def test_get_bucket_website_command_context_output(mocker): + """ + Given: + - A mocked boto3 S3 client returning a full website configuration + (IndexDocument, ErrorDocument, RedirectAllRequestsTo, RoutingRules). + - A valid bucket name. + When: + - get_bucket_website_command is called. + Then: + - The CommandResults context output prefix is "AWS.S3.Buckets.BucketWebsite". + - The outputs match the expected website configuration exactly. + """ + from AWS import S3 + + mock_client = mocker.Mock() + index_document = {"Suffix": "index.html"} + error_document = {"Key": "error.html"} + redirect_all_requests_to = {"HostName": "example.com", "Protocol": "https"} + routing_rules = [{"Redirect": {"ReplaceKeyPrefixWith": "documents/"}}] + mock_client.get_bucket_website.return_value = { + "ResponseMetadata": {"HTTPStatusCode": HTTPStatus.OK}, + "IndexDocument": index_document, + "ErrorDocument": error_document, + "RedirectAllRequestsTo": redirect_all_requests_to, + "RoutingRules": routing_rules, + } + args = {"bucket": "mock_bucket_name"} + + result = S3.get_bucket_website_command(mock_client, args) + + assert isinstance(result, CommandResults) + assert result.outputs_prefix == "AWS.S3.Buckets.BucketWebsite" + assert result.outputs == { + "ErrorDocument": error_document, + "IndexDocument": index_document, + "RedirectAllRequestsTo": redirect_all_requests_to, + "RoutingRules": routing_rules, + } + + def test_get_bucket_acl_command_success(mocker): """ Given: A mocked boto3 S3 client and a valid bucket name. @@ -6230,6 +6302,41 @@ def test_get_bucket_acl_command_failure(mocker): mock_error_handler.assert_called_once() +def test_get_bucket_acl_command_context_output(mocker): + """ + Given: + - A mocked boto3 S3 client returning a full access control policy (Grants and Owner). + - A valid bucket name. + When: + - get_bucket_acl_command is called. + Then: + - The CommandResults context output prefix is "AWS.S3.Buckets.BucketAcl". + - The outputs match the expected access control policy exactly. + """ + from AWS import S3 + + mock_client = mocker.Mock() + owner = {"DisplayName": "owner-display-name", "ID": "owner-id"} + grants = [ + { + "Grantee": {"Type": "CanonicalUser", "DisplayName": "owner-display-name", "ID": "owner-id"}, + "Permission": "FULL_CONTROL", + } + ] + mock_client.get_bucket_acl.return_value = { + "ResponseMetadata": {"HTTPStatusCode": HTTPStatus.OK}, + "Owner": owner, + "Grants": grants, + } + args = {"bucket": "mock_bucket_name"} + + result = S3.get_bucket_acl_command(mock_client, args) + + assert isinstance(result, CommandResults) + assert result.outputs_prefix == "AWS.S3.Buckets.BucketAcl" + assert result.outputs == {"Grants": grants, "Owner": owner} + + def test_create_network_acl_command_success(mocker): """ Given: A mocked boto3 EC2 client and a valid VPC ID. @@ -22155,3 +22262,438 @@ def test_update_logging_configuration_command_api_error(mocker): NetworkFirewall.update_logging_configuration_command(mock_client, args) mock_error_handler.assert_called_once() + + +# --------------------------------------------------------------------------- +# YML <-> PY wiring assertion tests +# +# These tests read the integration's .yml and .py from disk and assert that +# every command, its arguments, and its output prefixes declared in the YML are +# actually wired up in the Python code. This proves the command, its args, and +# its outputs are implemented as declared. +# +# The comparison is PER-COMMAND: each YML command is resolved to its handler +# function via COMMANDS_MAPPING in AWS.py. Unlike the GCP integration (whose +# handlers are plain module-level functions), AWS handlers are `@staticmethod`s +# on service classes (e.g. ``S3.buckets_list_command``), and they frequently +# delegate argument parsing to shared builders (e.g. +# ``aws_ec2_fleet_create_args_builder(args)``). The analysis is therefore done +# with the ``ast`` module and follows those delegations transitively, so an +# argument read inside a helper still counts as wired. +# +# Quick-action commands (names ending in "-quick-action") and the built-in +# "test-module" command are intentionally excluded. +# +# NOTE: these tests are pure static analysis - they only parse the .yml/.py +# text. They perform no network/API calls, read no environment variables, and +# are independent of execution order and of the system clock. +# --------------------------------------------------------------------------- + + +def _is_included_command(command_name: str) -> bool: + """Return whether the command should be checked by the YML <-> PY assertions. + + Args: + command_name: The command name as declared in the YML / COMMANDS_MAPPING. + + Returns: + True if the command is neither ``test-module`` nor a quick action, False otherwise. + """ + return command_name != "test-module" and not command_name.endswith(QUICK_ACTION_SUFFIX) + + +def _load_yml_spec() -> dict: + """Parse the integration YML and return its command specifications. + + Arguments marked ``hidden: true`` in the YML are excluded: they are not + user-facing (they carry internal polling state or a fixed defaultValue for a + quick action) and are frequently consumed by shared/platform code rather + than read directly in the command handler. + + Returns: + A dict mapping command_name -> {"args": [arg_names], "outputs": [contextPaths]} + for every non-quick-action, non-test-module command. + """ + import yaml + + with open(_YML_PATH, encoding="utf-8") as f: + yml = yaml.safe_load(f) + + spec: dict[str, dict[str, list[str]]] = {} + for command in yml.get("script", {}).get("commands", []): + name = command.get("name", "") + if not _is_included_command(name): + continue + arg_names = [arg["name"] for arg in (command.get("arguments") or []) if arg.get("name") and not arg.get("hidden")] + context_paths = [out["contextPath"] for out in (command.get("outputs") or []) if out.get("contextPath")] + spec[name] = {"args": arg_names, "outputs": context_paths} + return spec + + +def _load_py_tree() -> ast.Module: + """Parse the integration .py file into an AST. + + Returns: + The parsed AWS.py module as an ``ast.Module``. + """ + with open(_PY_PATH, encoding="utf-8") as f: + return ast.parse(f.read()) + + +@pytest.fixture(scope="module") +def yml_spec() -> dict: + """Provide the AWS.yml command specifications, parsed once for the whole module. + + Returns: + The value of :func:`_load_yml_spec`, shared by every test in this module. + """ + return _load_yml_spec() + + +@pytest.fixture(scope="module") +def py_tree() -> ast.Module: + """Provide the AWS.py AST, parsed once for the whole module. + + Returns: + The value of :func:`_load_py_tree`, shared by every test in this module. + """ + return _load_py_tree() + + +def _parse_commands_mapping(tree: ast.Module) -> dict[str, str]: + """Parse ``COMMANDS_MAPPING`` in AWS.py, mapping command name -> handler name. + + ``COMMANDS_MAPPING`` is an annotated assignment (``: dict[str, Callable]``), + so both ``Assign`` and ``AnnAssign`` nodes are considered. + + Args: + tree: The parsed AWS.py module. + + Returns: + A dict mapping command_name -> dotted handler name (e.g. "S3.buckets_list_command"), + excluding test-module and quick-action commands. + """ + mapping: dict[str, str] = {} + for node in ast.walk(tree): + if isinstance(node, ast.Assign): + targets = [t.id for t in node.targets if isinstance(t, ast.Name)] + elif isinstance(node, ast.AnnAssign) and isinstance(node.target, ast.Name): + targets = [node.target.id] + else: + continue + if "COMMANDS_MAPPING" not in targets or not isinstance(node.value, ast.Dict): + continue + for key, value in zip(node.value.keys, node.value.values): + if isinstance(key, ast.Constant) and isinstance(key.value, str) and _is_included_command(key.value): + mapping[key.value] = ast.unparse(value) + return mapping + + +def _index_functions(tree: ast.Module) -> tuple: + """Index every function in the module by qualified name and by short name. + + Args: + tree: The parsed AWS.py module. + + Returns: + A ``(qualified_index, short_index)`` tuple of dicts mapping a function name to its + definition node, where qualified names look like "S3.buckets_list_command" and short + names look like "buckets_list_command" (first definition wins on a short-name clash). + """ + qualified: dict[str, FunctionNode] = {} + short: dict[str, FunctionNode] = {} + + def walk(node: ast.AST, prefix: str = "") -> None: + """Recursively index the functions declared under ``node``. + + Args: + node: The AST node whose ``body`` is scanned (module, class, or function). + prefix: The dotted name prefix accumulated from the enclosing scopes. + + Returns: + None. ``qualified`` and ``short`` are populated in place. + """ + for child in node.body: + if isinstance(child, ast.ClassDef): + walk(child, f"{prefix}{child.name}.") + elif isinstance(child, ast.FunctionDef | ast.AsyncFunctionDef): + name = f"{prefix}{child.name}" + qualified[name] = child + short.setdefault(child.name, child) + walk(child, f"{name}.") + + walk(tree) + return qualified, short + + +def _resolve(name: str, qualified: dict, short: dict) -> FunctionNode | None: + """Resolve a (possibly dotted) callee name to its function definition node. + + Args: + name: The callee name as written in the source (e.g. "S3.buckets_list_command"). + qualified: Index of functions by qualified (dotted) name. + short: Index of functions by bare function name. + + Returns: + The matching function definition node, or None if the name is not defined in AWS.py. + """ + return qualified.get(name) or short.get(name.split(".")[-1]) + + +def _camel_to_snake(value: str) -> str: + """Convert a CamelCase string to its snake_case form. + + Mirrors ``CommonServerPython.camel_case_to_underscore``, which AWS.py uses at + runtime to derive argument names from AWS API keys, so consecutive uppercase + letters (acronyms) are kept together instead of being split letter by letter. + + Args: + value: The string to convert (e.g. "KMSKeyId"). + + Returns: + The snake_case form of the string (e.g. "kms_key_id"). + """ + partially_split = re.sub(r"(.)([A-Z][a-z]+)", r"\1_\2", value) + return re.sub(r"([a-z0-9])([A-Z])", r"\1_\2", partially_split).lower() + + +def _string_literals(node: ast.AST) -> set[str]: + """Collect all string literals in a function, plus their snake_case forms. + + Some handlers derive the argument name from a CamelCase AWS API key, e.g. + ``camel_case_to_underscore("Runtime")`` -> ``"runtime"``, so the snake_case + variants are included to avoid false positives. + + Args: + node: The AST node (typically a function definition) to scan. + + Returns: + A set of every string literal found under the node, unioned with the snake_case + form of each of those literals. + """ + literals = {n.value for n in ast.walk(node) if isinstance(n, ast.Constant) and isinstance(n.value, str)} + return literals | {_camel_to_snake(v) for v in literals} + + +def _callees_receiving_args(node: ast.AST) -> list[str]: + """Find the functions a handler calls while forwarding its ``args`` dict. + + Args: + node: The AST node (typically a handler's function definition) to scan. + + Returns: + A list of callee names (as written in the source) that receive ``args`` either + positionally or as a keyword argument. + """ + callees: list[str] = [] + for child in ast.walk(node): + if not isinstance(child, ast.Call): + continue + forwards_args = any(isinstance(a, ast.Name) and a.id == "args" for a in child.args) or any( + isinstance(kw.value, ast.Name) and kw.value.id == "args" for kw in child.keywords + ) + if forwards_args: + callees.append(ast.unparse(child.func)) + return callees + + +def _collect_arg_names( + node: ast.AST, + qualified: dict, + short: dict, + depth: int = 6, + seen: set[int] | None = None, +) -> set[str]: + """Collect argument names read by a handler, following ``args`` delegation. + + Args: + node: The handler function definition to scan. + qualified: Index of functions by qualified (dotted) name. + short: Index of functions by bare function name. + depth: Maximum delegation depth to follow into callees that receive ``args``. + seen: Ids of already-visited function nodes, used to avoid infinite recursion. + + Returns: + A set of candidate argument names (string literals, plus their snake_case forms) + read by the handler and by the helpers it forwards ``args`` to. + """ + seen = seen if seen is not None else set() + found = _string_literals(node) + if depth <= 0: + return found + for callee in _callees_receiving_args(node): + target = _resolve(callee, qualified, short) + if target is not None and id(target) not in seen: + seen.add(id(target)) + found |= _collect_arg_names(target, qualified, short, depth - 1, seen) + return found + + +def _collect_output_prefixes( + node: ast.AST, + qualified: dict, + short: dict, + depth: int = 4, + seen: set[int] | None = None, +) -> set[str]: + """Collect context-path-like literals produced by a handler and its helpers. + + Args: + node: The handler function definition to scan. + qualified: Index of functions by qualified (dotted) name. + short: Index of functions by bare function name. + depth: Maximum call depth to follow into helper functions. + seen: Ids of already-visited function nodes, used to avoid infinite recursion. + + Returns: + A set of dotted string literals (with any trailing ``(...)`` DT filter stripped) + that look like context output prefixes. + """ + seen = seen if seen is not None else set() + found = {re.sub(r"\(.*\)$", "", literal) for literal in _string_literals(node) if "." in literal} + if depth <= 0: + return found + for child in ast.walk(node): + if not isinstance(child, ast.Call): + continue + target = _resolve(ast.unparse(child.func), qualified, short) + if target is not None and id(target) not in seen: + seen.add(id(target)) + found |= _collect_output_prefixes(target, qualified, short, depth - 1, seen) + return found + + +@pytest.mark.parametrize( + "value, expected", + [ + ("Runtime", "runtime"), + ("RuntimeVersion", "runtime_version"), + ("bucketName", "bucket_name"), + ("KMSKeyId", "kms_key_id"), + ("S3Bucket", "s3_bucket"), + ("already_snake", "already_snake"), + ], +) +def test_camel_to_snake(value: str, expected: str) -> None: + """ + Given: A CamelCase string, either a plain one or one containing an acronym. + When: Converting it with _camel_to_snake. + Then: The result matches what CommonServerPython.camel_case_to_underscore produces + at runtime, keeping consecutive uppercase letters together instead of + splitting them letter by letter. + + Args: + value: The string to convert. + expected: The expected snake_case form. + """ + assert _camel_to_snake(value) == expected + + +def test_yml_commands_are_wired_in_py(yml_spec: dict, py_tree: ast.Module) -> None: + """ + Given: The integration YML declaring command names. + When: Comparing against the COMMANDS_MAPPING wired in AWS.py. + Then: Every non-quick-action YML command must be wired in the .py, and every + wired handler must resolve to a real function. + + Args: + yml_spec: The parsed AWS.yml command specifications (module-scoped fixture). + py_tree: The parsed AWS.py AST (module-scoped fixture). + """ + # When + command_map = _parse_commands_mapping(py_tree) + qualified, short = _index_functions(py_tree) + + # Then + missing = sorted(name for name in yml_spec if name not in command_map) + assert not missing, ( + "The following commands are declared in AWS.yml but are NOT wired in " f"COMMANDS_MAPPING in AWS.py: {missing}" + ) + + unresolved = sorted( + f"{command} -> {handler}" for command, handler in command_map.items() if _resolve(handler, qualified, short) is None + ) + assert not unresolved, ( + "The following COMMANDS_MAPPING handlers do not resolve to a function " f"defined in AWS.py: {unresolved}" + ) + + +def test_yml_args_match_py_handler_verbatim(yml_spec: dict, py_tree: ast.Module) -> None: + """ + Given: The arguments declared per command in the integration YML. + When: Comparing (verbatim) against the argument names read in that command's + resolved handler in AWS.py, following delegation into shared builders. + Then: Each YML argument name must appear exactly as-is in the handler. + Any naming difference (snake_case vs camelCase, casing) fails, except + for platform-standard args. + + Args: + yml_spec: The parsed AWS.yml command specifications (module-scoped fixture). + py_tree: The parsed AWS.py AST (module-scoped fixture). + """ + # Given + command_map = _parse_commands_mapping(py_tree) + qualified, short = _index_functions(py_tree) + + # When + mismatches: list[str] = [] + for command_name in sorted(yml_spec): + handler = command_map.get(command_name) + node = _resolve(handler, qualified, short) if handler else None + if node is None: + # Missing wiring is reported by test_yml_commands_are_wired_in_py. + continue + handler_args = _collect_arg_names(node, qualified, short) + for arg_name in yml_spec[command_name]["args"]: + if arg_name in PLATFORM_STANDARD_ARGS: + continue + if arg_name not in handler_args: + mismatches.append(f'{command_name} (handler {handler}) -> args.get("{arg_name}")') + + # Then + assert not mismatches, ( + "The following YML arguments are NOT read verbatim from args in their " + "command's handler in AWS.py (a naming difference such as snake_case vs " + "camelCase means the YML and PY are out of sync):\n" + "\n".join(mismatches) + ) + + +def test_yml_output_prefixes_match_py_handler(yml_spec: dict[str, dict[str, list[str]]], py_tree: ast.Module) -> None: + """ + Given: The output contextPaths declared per command in the integration YML. + When: Comparing against the output prefixes declared in that command's + resolved handler in AWS.py. + Then: Every YML output contextPath must be covered by an output prefix + declared in the handler (the prefix must be a leading segment of the + contextPath), except for platform-produced outputs. + + Args: + yml_spec: The parsed AWS.yml command specifications (module-scoped fixture). + py_tree: The parsed AWS.py AST (module-scoped fixture). + """ + # Given + command_map = _parse_commands_mapping(py_tree) + qualified, short = _index_functions(py_tree) + + def _is_covered(context_path: str, prefixes: set) -> bool: + return any(context_path == prefix or context_path.startswith(prefix + ".") for prefix in prefixes) + + # When + uncovered: list = [] + for command_name in sorted(yml_spec): + handler = command_map.get(command_name) + node = _resolve(handler, qualified, short) if handler else None + if node is None: + continue + handler_prefixes = _collect_output_prefixes(node, qualified, short) + for context_path in yml_spec[command_name]["outputs"]: + if context_path.startswith(PLATFORM_STANDARD_OUTPUT_ROOTS): + continue + if not _is_covered(context_path, handler_prefixes): + uncovered.append(f"{command_name} (handler {handler}) -> {context_path}") + + # Then + assert not uncovered, ( + "The following YML output contextPaths are NOT covered by any output " + "prefix declared in their command's handler in AWS.py:\n" + "\n".join(uncovered) + ) diff --git a/Packs/AWS/Integrations/AWS/README.md b/Packs/AWS/Integrations/AWS/README.md index 2811e2b1f933..48b479652f70 100644 --- a/Packs/AWS/Integrations/AWS/README.md +++ b/Packs/AWS/Integrations/AWS/README.md @@ -297,10 +297,10 @@ Retrieves the website configuration for a bucket. Required IAM permission: s3:Ge | **Path** | **Type** | **Description** | | --- | --- | --- | -| AWS.S3-Buckets.BucketWebsite.ErrorDocument | Object | The object key name of the website error document to use for 4XX class errors. | -| AWS.S3-Buckets.BucketWebsite.IndexDocument | Object | The name of the index document for the website \(for example index.html\). | -| AWS.S3-Buckets.BucketWebsite.RedirectAllRequestsTo | Object | Specifies the redirect behavior of all requests to a website endpoint of an Amazon S3 bucket. | -| AWS.S3-Buckets.BucketWebsite.RoutingRules | Array | Rules that define when a redirect is applied and the redirect behavior. | +| AWS.S3.Buckets.BucketWebsite.ErrorDocument | Object | The object key name of the website error document to use for 4XX class errors. | +| AWS.S3.Buckets.BucketWebsite.IndexDocument | Object | The name of the index document for the website \(for example index.html\). | +| AWS.S3.Buckets.BucketWebsite.RedirectAllRequestsTo | Object | The redirect behavior of all requests to a website endpoint of an Amazon S3 bucket. | +| AWS.S3.Buckets.BucketWebsite.RoutingRules | Array | The rules that define when a redirect is applied and the redirect behavior. | ### aws-s3-bucket-acl-get @@ -323,8 +323,8 @@ Retrieves the access control list (ACL) of a bucket. Required IAM permission: s3 | **Path** | **Type** | **Description** | | --- | --- | --- | -| AWS.S3-Buckets.BucketAcl.Grants | Array | A list of grants. | -| AWS.S3-Buckets.BucketAcl.Owner | Object | Container for the bucket owner's display name and ID. | +| AWS.S3.Buckets.BucketAcl.Grants | Array | The list of grants. | +| AWS.S3.Buckets.BucketAcl.Owner | Object | The container for the bucket owner's display name and ID. | ### aws-iam-account-password-policy-get @@ -2225,10 +2225,10 @@ Retrieves settings for a specific trail or returns information about all trails | AWS.CloudTrail.Trails.LogFileValidationEnabled | boolean | Whether log file validation is enabled. | | AWS.CloudTrail.Trails.CloudWatchLogsLogGroupArn | string | The ARN of the CloudWatch log group to which CloudTrail logs are delivered. | | AWS.CloudTrail.Trails.CloudWatchLogsRoleArn | string | The role assumed by CloudTrail to write logs to the CloudWatch log group. | -| AWS.CloudTrail.KmsKeyId | string | The KMS key ID that encrypts logs delivered by CloudTrail. | -| AWS.CloudTrail.HasCustomEventSelectors | boolean | Specifies if the trail has custom event selectors. | -| AWS.CloudTrail.HasInsightSelectors | boolean | Specifies whether a trail has insight types specified in an InsightSelector list. | -| AWS.CloudTrail.IsOrganizationTrail | boolean | Specifies whether the trail is an organization trail. | +| AWS.CloudTrail.Trails.KmsKeyId | string | The KMS key ID that encrypts logs delivered by CloudTrail. | +| AWS.CloudTrail.Trails.HasCustomEventSelectors | boolean | Specifies if the trail has custom event selectors. | +| AWS.CloudTrail.Trails.HasInsightSelectors | boolean | Specifies whether a trail has insight types specified in an InsightSelector list. | +| AWS.CloudTrail.Trails.IsOrganizationTrail | boolean | Specifies whether the trail is an organization trail. | ### aws-s3-bucket-policy-delete @@ -2273,11 +2273,11 @@ Retrieves the public access block configuration for an Amazon S3 bucket. Shows t | **Path** | **Type** | **Description** | | --- | --- | --- | -| AWS.S3-Buckets.BucketName | string | Name of the S3 bucket. | -| AWS.S3-Buckets.PublicAccessBlock.BlockPublicAcls | boolean | Whether Amazon S3 blocks public access control lists \(ACLs\) for this bucket and objects in this bucket. | -| AWS.S3-Buckets.PublicAccessBlock.IgnorePublicAcls | boolean | Whether Amazon S3 ignores public ACLs for this bucket and objects in this bucket. | -| AWS.S3-Buckets.PublicAccessBlock.BlockPublicPolicy | boolean | Whether Amazon S3 blocks public bucket policies for this bucket. | -| AWS.S3-Buckets.PublicAccessBlock.RestrictPublicBuckets | boolean | Whether Amazon S3 restricts public bucket policies for this bucket. | +| AWS.S3.Buckets.BucketName | string | The name of the S3 bucket. | +| AWS.S3.Buckets.PublicAccessBlock.BlockPublicAcls | boolean | Whether Amazon S3 blocks public access control lists \(ACLs\) for this bucket and objects in this bucket. | +| AWS.S3.Buckets.PublicAccessBlock.IgnorePublicAcls | boolean | Whether Amazon S3 ignores public ACLs for this bucket and objects in this bucket. | +| AWS.S3.Buckets.PublicAccessBlock.BlockPublicPolicy | boolean | Whether Amazon S3 blocks public bucket policies for this bucket. | +| AWS.S3.Buckets.PublicAccessBlock.RestrictPublicBuckets | boolean | Whether Amazon S3 restricts public bucket policies for this bucket. | ### aws-ec2-images-describe @@ -2608,8 +2608,8 @@ Retrieves the default encryption configuration for an Amazon S3 bucket. Shows th | **Path** | **Type** | **Description** | | --- | --- | --- | -| AWS.S3-Buckets.BucketName | string | Name of the S3 bucket. | -| AWS.S3-Buckets.ServerSideEncryptionConfiguration.Rules | array | Container for information about a particular server-side encryption configuration rule. | +| AWS.S3.Buckets.BucketName | string | The name of the S3 bucket. | +| AWS.S3.Buckets.ServerSideEncryptionConfiguration.Rules | array | The container for information about a particular server-side encryption configuration rule. | ### aws-s3-bucket-policy-get @@ -2633,15 +2633,15 @@ Retrieves the bucket policy for an Amazon S3 bucket. Returns the policy document | **Path** | **Type** | **Description** | | --- | --- | --- | -| AWS.S3-Buckets.BucketName | string | Name of the S3 bucket. | -| AWS.S3-Buckets.Policy.Version | string | The version of the policy. | -| AWS.S3-Buckets.Policy.Id | string | The id of the policy. | -| AWS.S3-Buckets.Policy.Statement.Sid | string | Identifier of the policy statement. | -| AWS.S3-Buckets.Policy.Statement.Effect | string | Specifies whether the statement results in an allow or an explicit deny. | -| AWS.S3-Buckets.Policy.Statement.Principal | unknown | Specify the principal that is allowed or denied access to a resource. | -| AWS.S3-Buckets.Policy.Statement.Action | unknown | Describes the specific action or actions that will be allowed or denied. | -| AWS.S3-Buckets.Policy.Statement.Resource | unknown | Defines the object or objects that the statement applies to. | -| AWS.S3-Buckets.Policy.Statement.Condition | string | Specify conditions for when a policy is in effect. | +| AWS.S3.Buckets.BucketName | string | The name of the S3 bucket. | +| AWS.S3.Buckets.Policy.Version | string | The version of the policy. | +| AWS.S3.Buckets.Policy.Id | string | The id of the policy. | +| AWS.S3.Buckets.Policy.Statement.Sid | string | The identifier of the policy statement. | +| AWS.S3.Buckets.Policy.Statement.Effect | string | The specification of whether the statement results in an allow or an explicit deny. | +| AWS.S3.Buckets.Policy.Statement.Principal | Unknown | The principal that is allowed or denied access to a resource. | +| AWS.S3.Buckets.Policy.Statement.Action | Unknown | The specific action or actions that will be allowed or denied. | +| AWS.S3.Buckets.Policy.Statement.Resource | Unknown | The object or objects that the statement applies to. | +| AWS.S3.Buckets.Policy.Statement.Condition | string | The conditions for when a policy is in effect. | ### aws-lambda-function-configuration-get diff --git a/Packs/AWS/ReleaseNotes/2_10_0.md b/Packs/AWS/ReleaseNotes/2_10_0.md new file mode 100644 index 000000000000..2cb5875afbdb --- /dev/null +++ b/Packs/AWS/ReleaseNotes/2_10_0.md @@ -0,0 +1,3 @@ +## AWS + +- Documentation and metadata improvements. diff --git a/Packs/AWS/ReleaseNotes/2_7_0.json b/Packs/AWS/ReleaseNotes/2_7_0.json new file mode 100644 index 000000000000..214438515893 --- /dev/null +++ b/Packs/AWS/ReleaseNotes/2_7_0.json @@ -0,0 +1,4 @@ +{ + "breakingChanges": true, + "breakingChangesNotes": "The context outputs paths of the following commands were updated: aws-s3-bucket-website-get, aws-s3-bucket-acl-get, aws-s3-public-access-block-get, aws-s3-bucket-encryption-get, aws-s3-bucket-policy-get." +} \ No newline at end of file diff --git a/Packs/AWS/ReleaseNotes/2_7_0.md b/Packs/AWS/ReleaseNotes/2_7_0.md new file mode 100644 index 000000000000..4de8758527ee --- /dev/null +++ b/Packs/AWS/ReleaseNotes/2_7_0.md @@ -0,0 +1,8 @@ +#### Integrations + +##### Amazon Web Services +- Updated the context output path of the command ***aws-s3-bucket-website-get*** to start with the prefix *AWS.S3.Buckets*. +- Updated the context output path of the command ***aws-s3-bucket-acl-get*** to start with the prefix *AWS.S3.Buckets*. +- Updated the context output path of the command ***aws-s3-public-access-block-get*** to start with the prefix *AWS.S3.Buckets*. +- Updated the context output path of the command ***aws-s3-bucket-encryption-get*** to start with the prefix *AWS.S3.Buckets*. +- Updated the context output path of the command ***aws-s3-bucket-policy-get*** to start with the prefix *AWS.S3.Buckets*. \ No newline at end of file diff --git a/Packs/AWS/ReleaseNotes/2_8_0.json b/Packs/AWS/ReleaseNotes/2_8_0.json new file mode 100644 index 000000000000..1ce22454000e --- /dev/null +++ b/Packs/AWS/ReleaseNotes/2_8_0.json @@ -0,0 +1,4 @@ +{ + "breakingChanges": true, + "breakingChangesNotes": "The context outputs paths of the following command was updated: aws-cloudtrail-trails-describe. Now all of the command outputs have the same prefix AWS.CloudTrail.Trails." +} \ No newline at end of file diff --git a/Packs/AWS/ReleaseNotes/2_8_0.md b/Packs/AWS/ReleaseNotes/2_8_0.md new file mode 100644 index 000000000000..298c41732976 --- /dev/null +++ b/Packs/AWS/ReleaseNotes/2_8_0.md @@ -0,0 +1,9 @@ +#### Integrations + +##### Amazon Web Services +- Fixed an issue where the arguments **ebs_card_index** and **ebs_initialization_rate** in the command ***aws-ec2-launch-template-create*** weren't forwarded to AWS. +- Fixed an issue where the argument *acl* in the command ***aws-s3-bucket-create*** wasn't forwarded to AWS. +- Fixed an issue where the command ***aws-ec2-fleet-instances-describe*** was lacking an output prefix. +- Updated the ***aws-ec2-ipam-resource-discoveries-describe*** command by removing the irrelevant argument *address_region*. +- Updated the ***aws-ec2-ipam-resource-discovery-associations-describe*** command by removing the irrelevant argument *address_region*. +- Updated the ***aws-cloudtrail-trails-describe*** command context outputs so all of them will have the prefix *AWS.CloudTrail.Trails*. diff --git a/Packs/AWS/ReleaseNotes/2_9_0.md b/Packs/AWS/ReleaseNotes/2_9_0.md new file mode 100644 index 000000000000..f4ef641567c9 --- /dev/null +++ b/Packs/AWS/ReleaseNotes/2_9_0.md @@ -0,0 +1,6 @@ +<~PLATFORM> +#### Integrations + +##### Amazon Web Services +Fixed an issue where the *account_id* was processed as an integer instead of a string. + \ No newline at end of file diff --git a/Packs/AWS/ReleaseNotes/2_9_1.md b/Packs/AWS/ReleaseNotes/2_9_1.md new file mode 100644 index 000000000000..7182246e49fb --- /dev/null +++ b/Packs/AWS/ReleaseNotes/2_9_1.md @@ -0,0 +1,7 @@ + +#### Integrations + +##### Amazon Web Services + +- Updated the Docker image to: *demisto/boto3py3:1.0.0.12887184*. + diff --git a/Packs/AWS/pack_metadata.json b/Packs/AWS/pack_metadata.json index 970e60b67c00..9e695358605a 100644 --- a/Packs/AWS/pack_metadata.json +++ b/Packs/AWS/pack_metadata.json @@ -2,7 +2,7 @@ "name": "AWS", "description": "Amazon Web Services", "support": "xsoar", - "currentVersion": "2.6.0", + "currentVersion": "2.10.0", "author": "Cortex", "url": "https://www.paloaltonetworks.com/cortex", "email": "", @@ -25,6 +25,5 @@ "marketplacev2", "platform" ], - "source": "cooc", - "managed": true + "derivedSource": "cooc" } \ No newline at end of file diff --git a/Packs/AWS_SystemManager/Integrations/AWSSystemManager/AWSSystemManager.yml b/Packs/AWS_SystemManager/Integrations/AWSSystemManager/AWSSystemManager.yml index 9b4d5bec2685..11715b6f7952 100644 --- a/Packs/AWS_SystemManager/Integrations/AWSSystemManager/AWSSystemManager.yml +++ b/Packs/AWS_SystemManager/Integrations/AWSSystemManager/AWSSystemManager.yml @@ -2398,7 +2398,7 @@ script: script: '-' type: python subtype: python3 - dockerimage: demisto/boto3py3:1.0.0.10221838 + dockerimage: demisto/boto3py3:1.0.0.12887184 fromversion: 6.9.0 marketplaces: - xsoar diff --git a/Packs/AWS_SystemManager/ReleaseNotes/1_0_25.md b/Packs/AWS_SystemManager/ReleaseNotes/1_0_25.md new file mode 100644 index 000000000000..935e22b3f6e9 --- /dev/null +++ b/Packs/AWS_SystemManager/ReleaseNotes/1_0_25.md @@ -0,0 +1,7 @@ + +#### Integrations + +##### AWS - System Manager + +- Updated the Docker image to: *demisto/boto3py3:1.0.0.12887184*. + diff --git a/Packs/AWS_SystemManager/ReleaseNotes/1_0_26.md b/Packs/AWS_SystemManager/ReleaseNotes/1_0_26.md new file mode 100644 index 000000000000..dfdf78025600 --- /dev/null +++ b/Packs/AWS_SystemManager/ReleaseNotes/1_0_26.md @@ -0,0 +1,7 @@ + +#### Scripts + +##### AWSPackageUpgrade + +- Updated the Docker image to: *demisto/python3:3.12.14.13053055*. + diff --git a/Packs/AWS_SystemManager/ReleaseNotes/1_0_27.md b/Packs/AWS_SystemManager/ReleaseNotes/1_0_27.md new file mode 100644 index 000000000000..80e6e8b4ba88 --- /dev/null +++ b/Packs/AWS_SystemManager/ReleaseNotes/1_0_27.md @@ -0,0 +1,3 @@ +## AWS Systems Manager + +- Locked dependencies of the pack to ensure stability for versioned core packs. No changes in this release. diff --git a/Packs/AWS_SystemManager/Scripts/AWSPackageUpgrade/AWSPackageUpgrade.yml b/Packs/AWS_SystemManager/Scripts/AWSPackageUpgrade/AWSPackageUpgrade.yml index 704bcdab664a..7c7a0b29908f 100644 --- a/Packs/AWS_SystemManager/Scripts/AWSPackageUpgrade/AWSPackageUpgrade.yml +++ b/Packs/AWS_SystemManager/Scripts/AWSPackageUpgrade/AWSPackageUpgrade.yml @@ -36,7 +36,7 @@ comment: This is an AWS script that upgrades a package on the AWS EC2 instance u commonfields: id: AWSPackageUpgrade version: -1 -dockerimage: demisto/python3:3.12.13.10116658 +dockerimage: demisto/python3:3.12.14.13053055 enabled: true engineinfo: {} name: AWSPackageUpgrade diff --git a/Packs/AWS_SystemManager/pack_metadata.json b/Packs/AWS_SystemManager/pack_metadata.json index d788fd6fab58..1bcbc15ec7ee 100644 --- a/Packs/AWS_SystemManager/pack_metadata.json +++ b/Packs/AWS_SystemManager/pack_metadata.json @@ -2,7 +2,7 @@ "name": "AWS Systems Manager", "description": "AWS Systems Manager is the operations hub for your AWS applications and resources and a secure end-to-end management solution for hybrid cloud environments that enables safe and secure operations at scale.", "support": "xsoar", - "currentVersion": "1.0.24", + "currentVersion": "1.0.27", "author": "Cortex XSOAR", "url": "https://www.paloaltonetworks.com/cortex", "email": "", diff --git a/Packs/AWS_WAF/Integrations/AWSWAF/AWSWAF.py b/Packs/AWS_WAF/Integrations/AWSWAF/AWSWAF.py index b1dc2a807c48..84db00b44c11 100644 --- a/Packs/AWS_WAF/Integrations/AWSWAF/AWSWAF.py +++ b/Packs/AWS_WAF/Integrations/AWSWAF/AWSWAF.py @@ -1009,8 +1009,8 @@ def main() -> None: # pragma: no cover aws_role_arn = params.get("roleArn") aws_role_session_name = params.get("roleSessionName") aws_role_session_duration = params.get("sessionDuration") - aws_access_key_id = params.get("access_key", {}).get("password") or params.get("access_key") - aws_secret_access_key = params.get("secret_key", {}).get("password") or params.get("secret_key") + aws_access_key_id = params.get("access_key", {}).get("password") + aws_secret_access_key = params.get("secret_key", {}).get("password") verify_certificate = not params.get("insecure", True) timeout = params.get("timeout") or 1 retries = params.get("retries") or 5 diff --git a/Packs/AWS_WAF/Integrations/AWSWAF/AWSWAF.yml b/Packs/AWS_WAF/Integrations/AWSWAF/AWSWAF.yml index af944676b42c..8cfbe7dabca8 100644 --- a/Packs/AWS_WAF/Integrations/AWSWAF/AWSWAF.yml +++ b/Packs/AWS_WAF/Integrations/AWSWAF/AWSWAF.yml @@ -1746,7 +1746,7 @@ script: script: '-' type: python subtype: python3 - dockerimage: demisto/boto3py3:1.0.0.10221838 + dockerimage: demisto/boto3py3:1.0.0.12887184 fromversion: 6.5.0 tests: - No tests (auto formatted) diff --git a/Packs/AWS_WAF/Integrations/AWSWAF/AWSWAF_test.py b/Packs/AWS_WAF/Integrations/AWSWAF/AWSWAF_test.py index b3015e0653f6..6c4f5cb8ba92 100644 --- a/Packs/AWS_WAF/Integrations/AWSWAF/AWSWAF_test.py +++ b/Packs/AWS_WAF/Integrations/AWSWAF/AWSWAF_test.py @@ -1,6 +1,7 @@ import json import pytest +import AWSWAF from AWSWAF import BYTE_MATCH_STATEMENT, OPERATOR_TO_STATEMENT_OPERATOR, REGEX_MATCH_STATEMENT from CommonServerPython import * @@ -648,3 +649,80 @@ def test_delete_rule_group_command(mocker): delete_rule_group_mock = mocker.patch.object(client, "delete_rule_group") delete_rule_group_command(client=client, args=delete_rule_group_args) delete_rule_group_mock.assert_called_with(Name="name", Id="id", Scope="REGIONAL", LockToken="lockToken") + + +def build_credentials_param(password: str) -> dict: + """Builds a type-9 credentials parameter as sent by the platform.""" + return { + "credential": "", + "credentials": { + "cacheVersn": 0, + "id": "", + "locked": False, + "modified": "0001-01-01T00:00:00Z", + "name": "", + "password": "", + "sizeInBytes": 0, + "sshkey": "", + "sshkeyPass": "", + "user": "", + "vaultInstanceId": "", + "version": 0, + "workgroup": "", + }, + "identifier": "", + "password": password, + "passwordChanged": False, + } + + +@pytest.mark.parametrize( + "access_key_password, secret_key_password", + [ + pytest.param("", "", id="empty credentials"), + pytest.param("test_access_key", "test_secret_key", id="populated credentials"), + ], +) +def test_main_credentials_params_retrieval(mocker, access_key_password, secret_key_password): + """ + Given: + - Integration params where access_key and secret_key are credentials objects, + either left empty or populated with a password. + When: + - Running main() with the test-module command. + Then: + - The access key and secret key passed to validate_params and AWSClient are the + password strings, and never the raw credentials dict. + """ + + # Given + params = { + "defaultRegion": "us-east-1", + "roleArn": "arn:aws:iam::123456789012:role/test", + "roleSessionName": "test_session", + "access_key": build_credentials_param(access_key_password), + "secret_key": build_credentials_param(secret_key_password), + } + mocker.patch.object(demisto, "params", return_value=params) + mocker.patch.object(demisto, "args", return_value={}) + mocker.patch.object(demisto, "command", return_value="test-module") + validate_params_mock = mocker.patch.object(AWSWAF, "validate_params") + aws_client_mock = mocker.patch.object(AWSWAF, "AWSClient") + mocker.patch.object(AWSWAF, "connection_test", return_value="ok") + return_results_mock = mocker.patch.object(AWSWAF, "return_results") + return_error_mock = mocker.patch.object(AWSWAF, "return_error") + + # When + AWSWAF.main() + + # Then + return_error_mock.assert_not_called() + return_results_mock.assert_any_call("ok") + validate_params_mock.assert_called_once_with( + "us-east-1", "arn:aws:iam::123456789012:role/test", "test_session", access_key_password, secret_key_password + ) + client_kwargs = aws_client_mock.call_args.kwargs + assert client_kwargs["aws_access_key_id"] == access_key_password + assert client_kwargs["aws_secret_access_key"] == secret_key_password + assert not isinstance(client_kwargs["aws_access_key_id"], dict) + assert not isinstance(client_kwargs["aws_secret_access_key"], dict) diff --git a/Packs/AWS_WAF/ReleaseNotes/1_0_23.md b/Packs/AWS_WAF/ReleaseNotes/1_0_23.md new file mode 100644 index 000000000000..0248a8b1f83f --- /dev/null +++ b/Packs/AWS_WAF/ReleaseNotes/1_0_23.md @@ -0,0 +1,7 @@ + +#### Integrations + +##### AWS-WAF + +- Updated the Docker image to: *demisto/boto3py3:1.0.0.12887184*. + diff --git a/Packs/AWS_WAF/ReleaseNotes/1_0_24.md b/Packs/AWS_WAF/ReleaseNotes/1_0_24.md new file mode 100644 index 000000000000..683e750c93c8 --- /dev/null +++ b/Packs/AWS_WAF/ReleaseNotes/1_0_24.md @@ -0,0 +1,5 @@ + +#### Integrations + +##### AWS-WAF +Fixed an issue where the *Access Key* and *Secret Key* were incorrectly populated when left empty. \ No newline at end of file diff --git a/Packs/AWS_WAF/pack_metadata.json b/Packs/AWS_WAF/pack_metadata.json index 1d5a531775b5..d041a9d74940 100644 --- a/Packs/AWS_WAF/pack_metadata.json +++ b/Packs/AWS_WAF/pack_metadata.json @@ -2,7 +2,7 @@ "name": "AWS WAF", "description": "Amazon Web Services Web Application Firewall (WAF)", "support": "xsoar", - "currentVersion": "1.0.22", + "currentVersion": "1.0.24", "author": "Cortex XSOAR", "url": "https://www.paloaltonetworks.com/cortex", "email": "", @@ -15,7 +15,7 @@ "useCases": [ "Network Security" ], - "keywords": [], + "keywords": ["AWS"], "marketplaces": [ "xsoar", "marketplacev2", diff --git a/Packs/Absolute/Integrations/Absolute/Absolute.py b/Packs/Absolute/Integrations/Absolute/Absolute.py index 64f12a89c063..a36969c3a57d 100644 --- a/Packs/Absolute/Integrations/Absolute/Absolute.py +++ b/Packs/Absolute/Integrations/Absolute/Absolute.py @@ -350,7 +350,7 @@ def fetch_events_between_dates(self, fetch_limit: int, start_date: datetime, end return all_events def prepare_query_string_for_fetch_events( - self, start_date: datetime, end_date: datetime, page_size: int = None, next_page: str = None + self, start_date: datetime, end_date: datetime, page_size: Optional[int] = None, next_page: Optional[str] = None ) -> str: """ Prepares the query string for fetching events based on the provided parameters. @@ -1142,6 +1142,261 @@ def get_device_location_command(args, client) -> CommandResults: return CommandResults(readable_output=f"No device locations found in {INTEGRATION} for the given filters: {args}") +def prepare_wipe_request_query(args: dict, status: str, next_page: Optional[str] = None, page_size: Optional[int] = None) -> str: + created_from = args.get("created_from_date_time_utc") + created_to = args.get("created_to_date_time_utc") + request_status = args.get("request_status") or args.get("action_status") + + query_params = [] + if created_from: + query_params.append(f"createdFromDateTimeUtc={created_from}") + if created_to: + query_params.append(f"createdToDateTimeUtc={created_to}") + if request_status: + query_params.append(f"{status}={request_status}") + if next_page: + query_params.append(f"nextPage={next_page}") + if page_size: + query_params.append(f"pageSize={page_size}") + + return "&".join(query_params) + + +def wipe_actions_list_command(args, client) -> CommandResults: + """ + Get the status of actions by devices or by request. + + Args: + args (Dict[str, Any]): Command arguments. + client (ClientV3): Absolute Client. + + Returns: + CommandResults: The command results. + """ + prefix = "[absolute-wipe-actions-list]" + demisto.debug(f"{prefix} {args=}") + request_uid = args.get("request_uid") + device_uids = argToList(args.get("device_uids")) + next_page = args.get("page") + limit = arg_to_number(args.get("limit", DEFAULT_LIMIT)) + + payload: dict[str, Any] = {} + payload["deviceUids"] = device_uids + + query_string = prepare_wipe_request_query(args, "actionStatus", next_page=next_page, page_size=limit) + if request_uid: + demisto.debug(f"{prefix} request_uid provided: {request_uid}. Getting actions for the request.") + demisto.debug(f"{prefix} query_string: {query_string}.\nPayload: {payload}.") + res = client.api_request_absolute( + "POST", + f"/v3/actions/wipe/get-actions/{request_uid}", + query_string=query_string, + body=payload, + ) + else: + demisto.debug(f"{prefix} request_uid not provided. Getting actions for the devices.") + demisto.debug(f"{prefix} query_string: {query_string}.\nPayload: {payload}.") + res = client.api_request_absolute( + "POST", + "/v3/actions/wipe/get-actions", + query_string=query_string, + body=payload, + ) + + if res: + outputs = res + human_readable = tableToMarkdown( + f"{INTEGRATION} wipe actions list:", + outputs, + headers=["actionUid", "requestUid", "deviceUid", "deviceName", "esn", "actionStatus"], + headerTransform=string_to_table_header, + removeNull=True, + ) + return CommandResults( + outputs_prefix="Absolute.WipeAction", + outputs=outputs, + outputs_key_field="actionUid", + readable_output=human_readable, + raw_response=res, + ) + else: + return CommandResults(readable_output=f"No wipe actions found in {INTEGRATION} for the given filters: {args}") + + +def wipe_request_cancel_command(args, client) -> CommandResults: + """ + Cancel a wipe request based on provided arguments. + + Args: + args (Dict[str, Any]): Command arguments. + client (ClientV3): Absolute Client. + + Returns: + CommandResults: The command results. + """ + prefix = "[absolute-wipe-request-cancel]" + demisto.debug(f"{prefix} {args=}") + request_uid = args.get("request_uid") + action_uids = argToList(args.get("action_uids")) + cancel_all_actions = args.get("cancel_all_actions", False) + + payload: dict[str, Any] = {} + if action_uids: + payload["actionUids"] = action_uids + payload["cancelAllActions"] = argToBoolean(cancel_all_actions) + + demisto.debug(f"{prefix} Canceling wipe actions for the request {request_uid}.\nPayload: {payload}.") + res = client.api_request_absolute( + "POST", + f"/v3/actions/wipe/cancel-actions/{request_uid}", + body=payload, + success_status_code=[202], + resp_type="response", + ) + + return CommandResults( + readable_output=f"Wipe actions for the request {request_uid} have been successfully canceled.", + raw_response=res, + ) + + +def wipe_request_create_command(args, client) -> CommandResults: + """ + Create a Wipe request for the devices based on provided arguments. + + Args: + args (Dict[str, Any]): Command arguments. + client (ClientV3): Absolute Client. + + Returns: + CommandResults: The command results. + """ + prefix = "[absolute-wipe-request-create]" + demisto.debug(f"{prefix} {args=}") + device_uids = argToList(args.get("device_uids")) + mac_user_name = args.get("mac_user_name") + mac_pwd = args.get("mac_pwd") + name = args.get("name") + description = args.get("description") + secure_erase_count = arg_to_number(args.get("secure_erase_count"), required=False) + + payload: dict[str, Any] = { + "deviceUids": device_uids, + "unenrollDevicesAndFreeLicenses": argToBoolean(args.get("unenroll_devices_and_free_licenses", False)), + } + if name: + payload["name"] = name + if description: + payload["description"] = description + if mac_user_name: + payload["macUsername"] = mac_user_name + if mac_pwd: + payload["macPwd"] = mac_pwd + + firmware_wipe_requested = args.get("firmware_wipe_requested") + if firmware_wipe_requested is not None: + payload["firmwareWipeRequested"] = argToBoolean(firmware_wipe_requested) + + crypto_wipe_requested = args.get("crypto_wipe_requested") + if crypto_wipe_requested is not None: + payload["cryptoWipeRequested"] = argToBoolean(crypto_wipe_requested) + + wipe_used_space_only = args.get("wipe_used_space_only") + if wipe_used_space_only is not None: + payload["wipeUsedSpaceOnly"] = argToBoolean(wipe_used_space_only) + + delete_all_files_requested = args.get("delete_all_files_requested") + if delete_all_files_requested is not None: + payload["deleteAllFilesRequested"] = argToBoolean(delete_all_files_requested) + + disable_window_os = args.get("disable_window_os") + if disable_window_os is not None: + payload["disableWindowOS"] = argToBoolean(disable_window_os) + + if secure_erase_count is not None: + if secure_erase_count not in (1, 3, 7): + raise_demisto_exception("secure_erase_count must be one of: 1, 3, 7.") + payload["secureEraseCount"] = secure_erase_count + + demisto.debug(f"{prefix} Creating wipe request with payload: {payload}") + res = client.api_request_absolute( + "POST", + "/v3/actions/requests/wipe", + body=payload, + ) + if not res: + raise DemistoException(f"{INTEGRATION} error: Failed to create wipe request - received empty response from API.") + + return CommandResults( + readable_output=f"Wipe request {res.get('requestUid')} has been successfully created.", + raw_response=res, + ) + + +def wipe_request_list_command(args, client) -> CommandResults: + """ + Retrieves wipe requests based on provided arguments. + + Args: + args (Dict[str, Any]): Command arguments. + client (ClientV3): Absolute Client. + + Returns: + CommandResults: The command results. + """ + prefix = "[absolute-wipe-request-list]" + demisto.debug(f"{prefix} {args=}") + request_uid = args.get("request_uid") + next_page = args.get("page") + limit = arg_to_number(args.get("limit", DEFAULT_LIMIT)) + + hr_headers = [ + "requestId", + "requestUid", + "requestStatus", + "totalDevices", + "pending", + "processing", + "completed", + "canceled", + "failed", + ] + + if request_uid: + demisto.debug(f"{prefix} request_uid provided, fetching details for request_uid: {request_uid}") + response = client.send_request_to_api("GET", f"/v3/actions/requests/wipe/{request_uid}", "", ok_codes=(200,)) + res = response.get("data") if isinstance(response, dict) else response + else: + demisto.debug(f"{prefix} no request_uid provided, fetching wipe requests list with filters: {args}") + query_string = prepare_wipe_request_query(args, "requestStatus", next_page=next_page, page_size=limit) + query_string = client.prepare_query_string_for_canonical_request(query_string) + demisto.debug(f"{prefix} query_string for wipe requests list: {query_string}") + response = client.send_request_to_api("GET", "/v3/actions/requests/wipe", query_string, ok_codes=(200,)) + res = response.get("data") if isinstance(response, dict) else response + + if res: + human_readable = tableToMarkdown( + f"{INTEGRATION} wipe request details:" if request_uid else f"{INTEGRATION} wipe requests list:", + res, + headers=hr_headers, + headerTransform=string_to_table_header, + removeNull=True, + ) + return CommandResults( + outputs_prefix="Absolute.WipeRequest", + outputs=res, + outputs_key_field="requestUid", + readable_output=human_readable, + raw_response=res, + ) + else: + return CommandResults( + readable_output=f"No wipe request found in {INTEGRATION} for the given request_uid: {request_uid}" + if request_uid + else f"No wipe requests found in {INTEGRATION} for the given filters: {args}" + ) + + """ EVENT COLLECTOR """ @@ -1313,6 +1568,18 @@ def main() -> None: # pragma: no cover send_events_to_xsiam(events=events, vendor=VENDOR, product=PRODUCT) return_results(command_result) + elif demisto.command() == "absolute-wipe-request-list": + return_results(wipe_request_list_command(args=args, client=client_v3)) + + elif demisto.command() == "absolute-wipe-actions-list": + return_results(wipe_actions_list_command(args=args, client=client_v3)) + + elif demisto.command() == "absolute-wipe-request-create": + return_results(wipe_request_create_command(args=args, client=client_v3)) + + elif demisto.command() == "absolute-wipe-request-cancel": + return_results(wipe_request_cancel_command(args=args, client=client_v3)) + else: raise NotImplementedError(f"{demisto.command()} is not an existing {INTEGRATION} command.") diff --git a/Packs/Absolute/Integrations/Absolute/Absolute.yml b/Packs/Absolute/Integrations/Absolute/Absolute.yml index b9d6815076ba..348f0b383fc5 100644 --- a/Packs/Absolute/Integrations/Absolute/Absolute.yml +++ b/Packs/Absolute/Integrations/Absolute/Absolute.yml @@ -53,6 +53,7 @@ configuration: supportedModules: - xsiam - defaultvalue: 'false' + display: Fetch Events name: isFetchEvents type: 8 required: false @@ -813,6 +814,210 @@ script: description: The maximum number of records to return per page. Note, this may be restricted by fixed system limits. name: absolute-device-get-events description: Retrieves a list of events from the Absolute instance. + - arguments: + - name: request_uid + description: The system-defined unique identifier of the Device Action request. + - name: request_status + description: Filters results to include responses where requestStatus equals the given value. + auto: PREDEFINED + predefined: + - Init + - Canceled + - Completed + - InProgress + - Failed + - Scheduled + - PendingApproval + - name: created_from_date_time_utc + description: Filters results to include responses where createdDateTimeUtc is from the given date and time (inclusive). Format . Example 2011-12-03T10:15:30.000Z. + - name: created_to_date_time_utc + description: Filters results to include responses where createdDateTimeUtc is to the given date and time (inclusive). Format . Example 2011-12-03T10:15:30.000Z. + - name: page + description: The pagination token for the next page of results. + - name: limit + description: Pagination limit - maximum number of events to return. + defaultValue: '50' + name: absolute-wipe-request-list + description: Get information about the Wipe requests. + outputs: + - contextPath: Absolute.WipeRequest.requestId + description: The ID of the wipe request. + type: String + - contextPath: Absolute.WipeRequest.requestUid + description: The unique identifier of the wipe request. + type: String + - contextPath: Absolute.WipeRequest.requestStatus + description: The status of the wipe request. + type: String + - contextPath: Absolute.WipeRequest.totalDevices + description: The total number of devices in the wipe request. + type: Number + - contextPath: Absolute.WipeRequest.pending + description: The number of devices pending wipe. + type: Number + - contextPath: Absolute.WipeRequest.processing + description: The number of devices currently processing the wipe. + type: Number + - contextPath: Absolute.WipeRequest.completed + description: The number of devices that have completed the wipe. + type: Number + - contextPath: Absolute.WipeRequest.canceled + description: The number of devices that have canceled the wipe. + type: Number + - contextPath: Absolute.WipeRequest.failed + description: The number of devices that have failed the wipe. + type: Number + - contextPath: Absolute.WipeRequest.macUsername + description: The username of the user with administrative rights (applies to Mac devices only). + type: String + - contextPath: Absolute.WipeRequest.macPwd + description: The password of the user with administrative rights (applies to Mac devices only). + type: String + - contextPath: Absolute.WipeRequest.unenrollDevicesAndFreeLicenses + description: Indicates whether a Windows device should be unenrolled after the Wipe is complete. + type: Boolean + - contextPath: Absolute.WipeRequest.createdDateTimeUtc + description: The date and time when the wipe request was created. + type: Date + - contextPath: Absolute.WipeRequest.updatedDateTimeUtc + description: The date and time when the wipe request was last updated. + type: Date + - contextPath: Absolute.WipeRequest.requester + description: The requester of the wipe request. + type: String + - arguments: + - name: request_uid + description: The system-defined unique identifier of the Device Action request. + - name: device_uids + description: The system-defined unique identifiers of the devices included in the request. Passing an empty list will return results for all applicable devices. + isArray: true + required: true + - name: action_status + description: Filters results to include responses where action_status equals the given value. + auto: PREDEFINED + predefined: + - Failed + - Pending + - Canceled + - Processing + - Completed + - name: created_from_date_time_utc + description: Filters results to include responses where createdDateTimeUtc is from the given date and time (inclusive). Format . Example 2011-12-03T10:15:30.000Z. + - name: created_to_date_time_utc + description: Filters results to include responses where createdDateTimeUtc is to the given date and time (inclusive). Format . Example 2011-12-03T10:15:30.000Z. + - name: page + description: The pagination token for the next page of results. + - name: limit + description: Pagination limit - maximum number of events to return. + defaultValue: '50' + name: absolute-wipe-actions-list + description: Get the status of actions by devices or by request. + outputs: + - contextPath: Absolute.WipeAction.actionUid + description: The unique identifier of the wipe action. + type: String + - contextPath: Absolute.WipeAction.requestUid + description: The unique identifier of the wipe request. + type: String + - contextPath: Absolute.WipeAction.deviceUid + description: The unique identifier of the device. + type: String + - contextPath: Absolute.WipeAction.deviceName + description: The name of the device. + type: String + - contextPath: Absolute.WipeAction.esn + description: The Electronic Serial Number of the device. + type: String + - contextPath: Absolute.WipeAction.actionStatus + description: The status of the wipe action. + type: String + - contextPath: Absolute.WipeAction.updatedDateTimeUtc + description: The date and time when the wipe action was last updated. + type: Date + - contextPath: Absolute.WipeAction.createdDateTimeUtc + description: The date and time when the wipe action was created. + type: Date + - contextPath: Absolute.WipeAction.certificateUid + description: The unique identifier of the certificate. + type: String + - arguments: + - name: device_uids + description: The system-defined unique identifiers of the devices included in the request. Passing an empty list will return results for all applicable devices. + isArray: true + required: true + - name: name + description: The name of the Wipe request. + - name: description + description: The user-defined description of the Wipe request. + - name: firmware_wipe_requested + description: Indicates whether a Firmware Wipe is requested. + auto: PREDEFINED + predefined: + - 'true' + - 'false' + - name: crypto_wipe_requested + description: Indicates whether a Cryptographic Wipe is requested. Default is true if no other Wipe options are specified. + auto: PREDEFINED + predefined: + - 'true' + - 'false' + - name: wipe_used_space_only + description: Indicates whether only the device's used disk space is wiped. Applies to Cryptographic Wipe requests only. + auto: PREDEFINED + predefined: + - 'true' + - 'false' + - name: delete_all_files_requested + description: Indicates whether a Delete All Files wipe is requested. + auto: PREDEFINED + predefined: + - 'true' + - 'false' + - name: disable_window_os + description: Indicates whether all non-OS and user profile data is deleted and the Windows operating system is disabled after the Wipe is complete. Applies only to Delete All Files wipe requests on Windows devices. + auto: PREDEFINED + predefined: + - 'true' + - 'false' + - name: secure_erase_count + description: Indicates the number of times the file content is overwritten with non-sensitive data (series of zeros and ones). Applies to Delete All Files wipe requests only. + auto: PREDEFINED + predefined: + - '1' + - '3' + - '7' + - name: mac_user_name + description: The username of the user with administrative rights (applies to Mac devices only). + - name: mac_pwd + description: The password of the user with administrative rights (applies to Mac devices only). + secret: true + - name: unenroll_devices_and_free_licenses + description: Indicates whether a Windows device should be unenrolled after the Wipe is complete. + auto: PREDEFINED + predefined: + - 'true' + - 'false' + defaultValue: 'false' + name: absolute-wipe-request-create + description: Create a Wipe request for the devices. + outputs: [] + - arguments: + - name: request_uid + description: The system-defined unique identifier of the Device Action request. + required: true + - name: action_uids + description: The system-defined unique identifiers of the actions included in the request. + isArray: true + - name: cancel_all_actions + description: Indicates whether all actions in the request should be canceled. + auto: PREDEFINED + predefined: + - 'true' + - 'false' + defaultValue: 'false' + name: absolute-wipe-request-cancel + description: Attempt to cancel all Wipe actions with an actionStatus of Pending for the request specified by the requestUid path parameter. + outputs: [] runonce: false script: '-' type: python diff --git a/Packs/Absolute/Integrations/Absolute/Absolute_test.py b/Packs/Absolute/Integrations/Absolute/Absolute_test.py index acd3650dcce7..5da5d8117c7c 100644 --- a/Packs/Absolute/Integrations/Absolute/Absolute_test.py +++ b/Packs/Absolute/Integrations/Absolute/Absolute_test.py @@ -931,3 +931,276 @@ def test_last_run_hold_created_time_and_events_sorted_by_created_time_in_fetch_e assert last_run_object.get("latest_events_time") == events[-1].get("createdDateTimeUtc") for event in events: assert event.get("_time") == event.get("eventDateTimeUtc") + + +def test_wipe_request_list_command(mocker, absolute_client_v3): + """ + Given: + - All relevant arguments for the command that is executed + + When: + - wipe_request_list_command is executed + + Then: + - The http request is called with the right arguments + """ + from Absolute import wipe_request_list_command + + response = util_load_json("test_data/wipe_requests.json") + mocker.patch.object(absolute_client_v3, "send_request_to_api", return_value={"data": response}) + command_results = wipe_request_list_command( + args={"created_from_date_time_utc": "2018-1-03T10:15:30.000Z", "created_to_date_time_utc": "2020-1-03T10:15:30.000Z"}, + client=absolute_client_v3, + ) + assert isinstance(command_results.outputs, list) + assert command_results.outputs[0].get("requestUid") == "23dbd460-0547-466d-a655-99d9340ff598" + + +def test_wipe_request_list_command_with_request_uid(mocker, absolute_client_v3): + """ + Given: + - A request_uid argument is provided + + When: + - wipe_request_list_command is executed + + Then: + - The http request is called with the right endpoint and the single request data is returned + """ + from Absolute import wipe_request_list_command + + response = util_load_json("test_data/wipe_requests.json") + mock_api = mocker.patch.object(absolute_client_v3, "send_request_to_api", return_value={"data": response[0]}) + command_results = wipe_request_list_command( + args={"request_uid": "23dbd460-0547-466d-a655-99d9340ff598"}, + client=absolute_client_v3, + ) + assert command_results.outputs.get("requestUid") == "23dbd460-0547-466d-a655-99d9340ff598" + call_args = mock_api.call_args + assert "/v3/actions/requests/wipe/23dbd460-0547-466d-a655-99d9340ff598" in call_args[0][1] + + +def test_wipe_actions_list_command(mocker, absolute_client_v3): + """ + Given: + - All relevant arguments for the command that is executed + + When: + - wipe_actions_list_command is executed + + Then: + - The http request is called with the right arguments + """ + from Absolute import wipe_actions_list_command + + response = util_load_json("test_data/wipe_actions.json") + mocker.patch.object(absolute_client_v3, "api_request_absolute", return_value=response) + command_results = wipe_actions_list_command( + args={ + "request_uid": "1", + "device_uids": "2", + "created_from_date_time_utc": "2018-1-03T10:15:30.000Z", + "created_to_date_time_utc": "2020-1-03T10:15:30.000Z", + }, + client=absolute_client_v3, + ) + assert isinstance(command_results.outputs, list) + assert command_results.outputs[0].get("requestUid") == "1" + assert command_results.outputs[0].get("actionUid") == "7f0741c5-fbe9-4144-8c54-498db6559304" + + +def test_wipe_request_create_command(mocker, absolute_client_v3): + """ + Given: + - All relevant arguments for the command that is executed + + When: + - wipe_request_create_command is executed + + Then: + - The http request is called with the right arguments + """ + from Absolute import wipe_request_create_command + + response = {"requestUid": "23dbd460-0547-466d-a655-99d9340ff598"} + mocker.patch.object(absolute_client_v3, "api_request_absolute", return_value=response) + command_results = wipe_request_create_command( + args={"device_uids": "497f6eca-6276-4993-bfeb-53cbbbba6f08"}, client=absolute_client_v3 + ) + assert command_results.readable_output == "Wipe request 23dbd460-0547-466d-a655-99d9340ff598 has been successfully created." + + +def test_wipe_request_create_command_with_all_args(mocker, absolute_client_v3): + """ + Given: + - All optional arguments for the wipe request create command + + When: + - wipe_request_create_command is executed with all arguments + + Then: + - The http request is called with the right payload including all optional fields + """ + from Absolute import wipe_request_create_command + + response = {"requestUid": "23dbd460-0547-466d-a655-99d9340ff598"} + mock_api = mocker.patch.object(absolute_client_v3, "api_request_absolute", return_value=response) + command_results = wipe_request_create_command( + args={ + "device_uids": "497f6eca-6276-4993-bfeb-53cbbbba6f08", + "name": "Test Wipe", + "description": "Test wipe description", + "firmware_wipe_requested": "true", + "crypto_wipe_requested": "false", + "wipe_used_space_only": "true", + "delete_all_files_requested": "true", + "disable_window_os": "false", + "secure_erase_count": "3", + "mac_user_name": "admin", + "mac_pwd": "password123", + "unenroll_devices_and_free_licenses": "true", + }, + client=absolute_client_v3, + ) + assert command_results.readable_output == "Wipe request 23dbd460-0547-466d-a655-99d9340ff598 has been successfully created." + + call_kwargs = mock_api.call_args + payload = call_kwargs.kwargs.get("body") or call_kwargs[1].get("body") + assert payload["deviceUids"] == ["497f6eca-6276-4993-bfeb-53cbbbba6f08"] + assert payload["name"] == "Test Wipe" + assert payload["description"] == "Test wipe description" + assert payload["firmwareWipeRequested"] is True + assert payload["cryptoWipeRequested"] is False + assert payload["wipeUsedSpaceOnly"] is True + assert payload["deleteAllFilesRequested"] is True + assert payload["disableWindowOS"] is False + assert payload["secureEraseCount"] == 3 + assert payload["macUsername"] == "admin" + assert payload["macPwd"] == "password123" + assert payload["unenrollDevicesAndFreeLicenses"] is True + + +def test_wipe_request_create_command_invalid_secure_erase_count(mocker, absolute_client_v3): + """ + Given: + - An invalid secure_erase_count value (not 1, 3, or 7) + + When: + - wipe_request_create_command is executed + + Then: + - A DemistoException is raised with the appropriate error message + """ + from Absolute import wipe_request_create_command + + with pytest.raises(DemistoException, match="secure_erase_count must be one of: 1, 3, 7."): + wipe_request_create_command( + args={ + "device_uids": "497f6eca-6276-4993-bfeb-53cbbbba6f08", + "secure_erase_count": "5", + }, + client=absolute_client_v3, + ) + + +def test_wipe_request_cancel_command(mocker, absolute_client_v3): + """ + Given: + - A request_uid and default arguments + - The Absolute API returns HTTP 202 with an empty body (as documented for cancel-actions) + + When: + - wipe_request_cancel_command is executed + + Then: + - The command completes successfully without a JSON parse error (regression test for + the bug where resp_type defaulted to "json" and failed to parse the empty 202 body) + - api_request_absolute is called with resp_type="response" + - The readable_output confirms cancellation + """ + from Absolute import wipe_request_cancel_command + from requests.models import Response + + # Simulate a real 202 empty-body response coming back from the Absolute API. + empty_202 = Response() + empty_202.status_code = 202 + empty_202._content = b"" + http_mock = mocker.patch.object(absolute_client_v3, "_http_request", return_value=empty_202) + api_spy = mocker.spy(absolute_client_v3, "api_request_absolute") + + command_results = wipe_request_cancel_command(args={"request_uid": "1"}, client=absolute_client_v3) + + assert command_results.readable_output == "Wipe actions for the request 1 have been successfully canceled." + # Regression guard: resp_type MUST be "response" so BaseClient does not try to JSON-parse b''. + assert api_spy.call_args.kwargs.get("resp_type") == "response" + # Sanity: BaseClient was called with the JWS validation URL (real path exercised end-to-end). + assert http_mock.call_count == 1 + + +def test_wipe_request_cancel_command_cancel_all(mocker, absolute_client_v3): + """ + Given: + - cancel_all_actions is set to true and no action_uids are provided + - The Absolute API returns HTTP 202 with an empty body + + When: + - wipe_request_cancel_command is executed + + Then: + - The command completes successfully (no JSON parse error on empty 202) + - The payload includes cancelAllActions=true and does not include actionUids + - api_request_absolute is called with resp_type="response" + """ + from Absolute import wipe_request_cancel_command + from requests.models import Response + + empty_202 = Response() + empty_202.status_code = 202 + empty_202._content = b"" + mocker.patch.object(absolute_client_v3, "_http_request", return_value=empty_202) + api_spy = mocker.spy(absolute_client_v3, "api_request_absolute") + + command_results = wipe_request_cancel_command( + args={"request_uid": "1", "cancel_all_actions": "true"}, + client=absolute_client_v3, + ) + assert command_results.readable_output == "Wipe actions for the request 1 have been successfully canceled." + + payload = api_spy.call_args.kwargs.get("body") + assert payload["cancelAllActions"] is True + assert "actionUids" not in payload + assert api_spy.call_args.kwargs.get("resp_type") == "response" + + +def test_wipe_request_cancel_command_cancel_all_false(mocker, absolute_client_v3): + """ + Given: + - cancel_all_actions is explicitly set to false with an action_uid provided + - The Absolute API returns HTTP 202 with an empty body + + When: + - wipe_request_cancel_command is executed + + Then: + - The command completes successfully (no JSON parse error on empty 202) + - The payload includes cancelAllActions=false (not silently dropped) and actionUids + - api_request_absolute is called with resp_type="response" + """ + from Absolute import wipe_request_cancel_command + from requests.models import Response + + empty_202 = Response() + empty_202.status_code = 202 + empty_202._content = b"" + mocker.patch.object(absolute_client_v3, "_http_request", return_value=empty_202) + api_spy = mocker.spy(absolute_client_v3, "api_request_absolute") + + wipe_request_cancel_command( + args={"request_uid": "1", "action_uids": "uid1", "cancel_all_actions": "false"}, + client=absolute_client_v3, + ) + + payload = api_spy.call_args.kwargs.get("body") + assert payload["cancelAllActions"] is False + assert payload["actionUids"] == ["uid1"] + assert api_spy.call_args.kwargs.get("resp_type") == "response" diff --git a/Packs/Absolute/Integrations/Absolute/README.md b/Packs/Absolute/Integrations/Absolute/README.md index 07b7de727a18..62ccf30050ff 100644 --- a/Packs/Absolute/Integrations/Absolute/README.md +++ b/Packs/Absolute/Integrations/Absolute/README.md @@ -1100,3 +1100,139 @@ In the ***absolute-device-application-list*** command, the following argument wa - *Absolute.DeviceUnenroll.Serial* - *Absolute.DeviceUnenroll.SystemName* - *Absolute.DeviceUnenroll.Username* + +### absolute-wipe-request-list + +*** +Get information about the Wipe requests. + +#### Base Command + +`absolute-wipe-request-list` + +#### Input + +| **Argument Name** | **Description** | **Required** | +| --- | --- | --- | +| request_uid | The system-defined unique identifier of the Device Action request. | Optional | +| request_status | Filters results to include responses where requestStatus equals the given value. Possible values are: Init, Canceled, Completed, InProgress, Failed, Scheduled, PendingApproval. | Optional | +| created_from_date_time_utc | Filters results to include responses where createdDateTimeUtc is from the given date and time (inclusive). Format <yyyy-MM-dd'T'HH:mm:ss.SSS'Z'>. Example 2011-12-03T10:15:30.000Z. | Optional | +| created_to_date_time_utc | Filters results to include responses where createdDateTimeUtc is to the given date and time (inclusive). Format <yyyy-MM-dd'T'HH:mm:ss.SSS'Z'>. Example 2011-12-03T10:15:30.000Z. | Optional | +| page | The pagination token for the next page of results. | Optional | +| limit | Pagination limit - maximum number of events to return. Default is 50. | Optional | + +#### Context Output + +| **Path** | **Type** | **Description** | +| --- | --- | --- | +| Absolute.WipeRequest.requestId | String | The ID of the wipe request. | +| Absolute.WipeRequest.requestUid | String | The unique identifier of the wipe request. | +| Absolute.WipeRequest.requestStatus | String | The status of the wipe request. | +| Absolute.WipeRequest.totalDevices | Number | The total number of devices in the wipe request. | +| Absolute.WipeRequest.pending | Number | The number of devices pending wipe. | +| Absolute.WipeRequest.processing | Number | The number of devices currently processing the wipe. | +| Absolute.WipeRequest.completed | Number | The number of devices that have completed the wipe. | +| Absolute.WipeRequest.canceled | Number | The number of devices that have canceled the wipe. | +| Absolute.WipeRequest.failed | Number | The number of devices that have failed the wipe. | +| Absolute.WipeRequest.macUsername | String | The username of the user with administrative rights \(applies to Mac devices only\). | +| Absolute.WipeRequest.macPwd | String | The password of the user with administrative rights \(applies to Mac devices only\). | +| Absolute.WipeRequest.unenrollDevicesAndFreeLicenses | Boolean | Indicates whether a Windows device should be unenrolled after the Wipe is complete. | +| Absolute.WipeRequest.createdDateTimeUtc | Date | The date and time when the wipe request was created. | +| Absolute.WipeRequest.updatedDateTimeUtc | Date | The date and time when the wipe request was last updated. | +| Absolute.WipeRequest.requester | String | The requester of the wipe request. | + +### absolute-wipe-actions-list + +*** +Get the status of actions by devices or by request. + +#### Base Command + +`absolute-wipe-actions-list` + +#### Input + +| **Argument Name** | **Description** | **Required** | +| --- | --- | --- | +| request_uid | The system-defined unique identifier of the Device Action request. | Optional | +| device_uids | The system-defined unique identifiers of the devices included in the request. Passing an empty list will return results for all applicable devices. | Required | +| action_status | Filters results to include responses where action_status equals the given value. Possible values are: Failed, Pending, Canceled, Processing, Completed. | Optional | +| created_from_date_time_utc | Filters results to include responses where createdDateTimeUtc is from the given date and time (inclusive). Format <yyyy-MM-dd'T'HH:mm:ss.SSS'Z'>. Example 2011-12-03T10:15:30.000Z. | Optional | +| created_to_date_time_utc | Filters results to include responses where createdDateTimeUtc is to the given date and time (inclusive). Format <yyyy-MM-dd'T'HH:mm:ss.SSS'Z'>. Example 2011-12-03T10:15:30.000Z. | Optional | +| page | The pagination token for the next page of results. | Optional | +| limit | Pagination limit - maximum number of events to return. Default is 50. | Optional | + +#### Context Output + +| **Path** | **Type** | **Description** | +| --- | --- | --- | +| Absolute.WipeAction.actionUid | String | The unique identifier of the wipe action. | +| Absolute.WipeAction.requestUid | String | The unique identifier of the wipe request. | +| Absolute.WipeAction.deviceUid | String | The unique identifier of the device. | +| Absolute.WipeAction.deviceName | String | The name of the device. | +| Absolute.WipeAction.esn | String | The Electronic Serial Number of the device. | +| Absolute.WipeAction.actionStatus | String | The status of the wipe action. | +| Absolute.WipeAction.updatedDateTimeUtc | Date | The date and time when the wipe action was last updated. | +| Absolute.WipeAction.createdDateTimeUtc | Date | The date and time when the wipe action was created. | +| Absolute.WipeAction.certificateUid | String | The unique identifier of the certificate. | + +### absolute-wipe-request-create + +*** +Create a Wipe request for the devices. + +#### Base Command + +`absolute-wipe-request-create` + +#### Input + +| **Argument Name** | **Description** | **Required** | +| --- | --- | --- | +| device_uids | The system-defined unique identifiers of the devices included in the request. Passing an empty list will return results for all applicable devices. | Required | +| name | The name of the Wipe request. | Optional | +| description | The user-defined description of the Wipe request. | Optional | +| firmware_wipe_requested | Indicates whether a Firmware Wipe is requested. Possible values are: true, false. | Optional | +| crypto_wipe_requested | Indicates whether a Cryptographic Wipe is requested. Default is true if no other Wipe options are specified. Possible values are: true, false. | Optional | +| wipe_used_space_only | Indicates whether only the device's used disk space is wiped. Applies to Cryptographic Wipe requests only. Possible values are: true, false. | Optional | +| delete_all_files_requested | Indicates whether a Delete All Files wipe is requested. Possible values are: true, false. | Optional | +| disable_window_os | Indicates whether all non-OS and user profile data is deleted and the Windows operating system is disabled after the Wipe is complete. Applies only to Delete All Files wipe requests on Windows devices. Possible values are: true, false. | Optional | +| secure_erase_count | Indicates the number of times the file content is overwritten with non-sensitive data (series of zeros and ones). Applies to Delete All Files wipe requests only. Possible values are: 1, 3, 7. | Optional | +| mac_user_name | The username of the user with administrative rights (applies to Mac devices only). | Optional | +| mac_pwd | The password of the user with administrative rights (applies to Mac devices only). | Optional | +| unenroll_devices_and_free_licenses | Indicates whether a Windows device should be unenrolled after the Wipe is complete. Possible values are: true, false. Default is false. | Optional | + +#### Context Output + +There is no context output for this command. + +### absolute-wipe-request-cancel + +*** +Attempt to cancel all Wipe actions with an actionStatus of Pending for the request specified by the requestUid path parameter. + +#### Base Command + +`absolute-wipe-request-cancel` + +#### Input + +| **Argument Name** | **Description** | **Required** | +| --- | --- | --- | +| request_uid | The system-defined unique identifier of the Device Action request. | Required | +| action_uids | The system-defined unique identifiers of the actions included in the request. | Optional | +| cancel_all_actions | Indicates whether all actions in the request should be canceled. Possible values are: true, false. Default is false. | Optional | + +#### Context Output + +There is no context output for this command. + +<~PLATFORM> + +## License Requirements + +The following configuration parameters require the **Cortex XSIAM** license: + +- Fetch Events + + diff --git a/Packs/Absolute/Integrations/Absolute/test_data/cancel_wipe_request.json b/Packs/Absolute/Integrations/Absolute/test_data/cancel_wipe_request.json new file mode 100644 index 000000000000..2de14b2469f1 --- /dev/null +++ b/Packs/Absolute/Integrations/Absolute/test_data/cancel_wipe_request.json @@ -0,0 +1,11 @@ +{ + "code": "INVALID_REQUEST", + "message": "invalid request", + "details": [ + { + "message": "invalid json", + "code": "INVALID_JSON", + "field": "deviceUids" + } + ] +} \ No newline at end of file diff --git a/Packs/Absolute/Integrations/Absolute/test_data/create_wipe_request.json b/Packs/Absolute/Integrations/Absolute/test_data/create_wipe_request.json new file mode 100644 index 000000000000..891892503958 --- /dev/null +++ b/Packs/Absolute/Integrations/Absolute/test_data/create_wipe_request.json @@ -0,0 +1,10 @@ +{ + "data": { + "requestUid": "23dbd460-0547-466d-a655-99d9340ff598" + }, + "metadata": { + "pagination": { + "nextPage": "string" + } + } +} \ No newline at end of file diff --git a/Packs/Absolute/Integrations/Absolute/test_data/wipe_actions.json b/Packs/Absolute/Integrations/Absolute/test_data/wipe_actions.json new file mode 100644 index 000000000000..35491472a37e --- /dev/null +++ b/Packs/Absolute/Integrations/Absolute/test_data/wipe_actions.json @@ -0,0 +1,13 @@ +[ + { + "actionUid": "7f0741c5-fbe9-4144-8c54-498db6559304", + "requestUid": "1", + "deviceUid": "2", + "deviceName": "string", + "esn": "string", + "updatedDateTimeUtc": "2019-08-24T14:15:22Z", + "createdDateTimeUtc": "2019-08-24T14:15:22Z", + "actionStatus": "Pending", + "certificateUid": "string" + } +] \ No newline at end of file diff --git a/Packs/Absolute/Integrations/Absolute/test_data/wipe_requests.json b/Packs/Absolute/Integrations/Absolute/test_data/wipe_requests.json new file mode 100644 index 000000000000..1efc3fcef0c6 --- /dev/null +++ b/Packs/Absolute/Integrations/Absolute/test_data/wipe_requests.json @@ -0,0 +1,20 @@ + +[ + { + "totalDevices": 0, + "pending": 0, + "processing": 0, + "completed": 0, + "canceled": 0, + "failed": 0, + "requestId": "string", + "requestUid": "23dbd460-0547-466d-a655-99d9340ff598", + "requestStatus": "string", + "createdDateTimeUtc": "2019-08-24T14:15:22Z", + "updatedDateTimeUtc": "2019-08-24T14:15:22Z", + "requester": "string", + "macUsername": "string", + "macPwd": "string", + "unenrollDevicesAndFreeLicenses": true + } + ] diff --git a/Packs/Absolute/ModelingRules/Absolute/Absolute.yml b/Packs/Absolute/ModelingRules/Absolute/Absolute.yml index 9ae7613c700c..e85fef9ec9ef 100644 --- a/Packs/Absolute/ModelingRules/Absolute/Absolute.yml +++ b/Packs/Absolute/ModelingRules/Absolute/Absolute.yml @@ -1,4 +1,5 @@ fromversion: 8.4.0 +toversion: 8.14.0 id: Absolute_ModelingRule name: Absolute Modeling Rule rules: '' diff --git a/Packs/Absolute/ModelingRules/Absolute_2_11/Absolute_2_11.xif b/Packs/Absolute/ModelingRules/Absolute_2_11/Absolute_2_11.xif new file mode 100644 index 000000000000..cb45235510c8 --- /dev/null +++ b/Packs/Absolute/ModelingRules/Absolute_2_11/Absolute_2_11.xif @@ -0,0 +1,52 @@ +[MODEL: dataset = absolute_secure_endpoint_raw] +alter + get_IpAddress_ipv4 = if(parsed_fields_NewValue -> IpAddress ~= "\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}", parsed_fields_NewValue -> IpAddress), + get_IpAddress_ipv6 = if(parsed_fields_NewValue -> IpAddress ~= "\:", parsed_fields_NewValue -> IpAddress), + get_Longitude = if(objectDisplayName = "Device Location", parsed_fields_OldValue -> Longitude), + get_Latitude = if(objectDisplayName = "Device Location", parsed_fields_OldValue -> Latitude) +| alter +// General Fields + xdm.event.id = id, + xdm.event.type = eventType, + xdm.event.operation_sub_type = verb, + xdm.source.host.device_id = if(actorObjectType = "Device", actorDisplayId), + xdm.source.user.identifier = if(actorObjectType = "User", actorDisplayId), + xdm.source.identity.identifier = if(actorObjectType = "User", actorDisplayId), + xdm.source.application.version = if(actorObjectType = "System", actorDisplayId), + xdm.source.user.identity_type = if( + actorObjectType = "Device", XDM_CONST.IDENTITY_TYPE_MACHINE, + actorObjectType = "User", XDM_CONST.IDENTITY_TYPE_USER, + actorObjectType = "System", XDM_CONST.IDENTITY_TYPE_BUILTIN, + actorObjectType = "NA", XDM_CONST.IDENTITY_TYPE_UNKNOWN, + actorObjectType = null, null, to_string(actorObjectType)), + xdm.source.identity.identity_type = if( + actorObjectType = "Device", XDM_CONST.IDENTITY_TYPE_MACHINE, + actorObjectType = "User", XDM_CONST.IDENTITY_TYPE_USER, + actorObjectType = "System", XDM_CONST.IDENTITY_TYPE_BUILTIN, + actorObjectType = "NA", XDM_CONST.IDENTITY_TYPE_UNKNOWN, + actorObjectType = null, null, to_string(actorObjectType)), + xdm.source.host.hostname = if(actorObjectType = "Device", actorDisplayName), + xdm.source.user.username = if(actorObjectType = "User", actorDisplayName), + xdm.source.identity.username = if(actorObjectType = "User", actorDisplayName), + xdm.source.application.name = if(actorObjectType = "System", actorDisplayName), + xdm.target.resource.id = objectDisplayId, + xdm.target.resource.type = objectObjectType, + xdm.target.resource.name = objectDisplayName, + xdm.target.resource.sub_type = to_string(object_create( + "secondaryObjectDisplayId", secondaryObjectDisplayId, + "secondaryObjectDisplayName", secondaryObjectDisplayName, + "secondaryObjectType",secondaryObjectType)), +// Fields extracted from objectProperties + xdm.target.host.ipv6_addresses = arraycreate(parsed_fields_NewValue -> DeviceSystemInformationUpdated_LocalIpV6Addr), + xdm.target.host.ipv6_public_addresses = arraycreate(parsed_fields_NewValue -> DeviceSystemInformationUpdated_PublicIpV6Addr), + xdm.target.host.ipv4_public_addresses = arraycreate(parsed_fields_NewValue -> DeviceSystemInformationUpdated_PublicIpAddr), + xdm.target.host.ipv4_addresses = arraycreate(parsed_fields_NewValue -> DeviceSystemInformationUpdated_LocalIpAddr), + xdm.source.ipv4 = get_IpAddress_ipv4, + xdm.source.ipv6 = get_IpAddress_ipv6, + xdm.source.user_agent = parsed_fields_NewValue -> BrowserAgent, + xdm.source.location.longitude = to_float(get_Longitude), + xdm.source.location.latitude = to_float(get_Latitude), + xdm.source.location.city = if(objectDisplayName = "Device Location", parsed_fields_OldValue -> City), + xdm.source.location.country = if(objectDisplayName = "Device Location", parsed_fields_OldValue -> CountryName), + xdm.target.resource_before.value = to_string(parsed_fields_OldValue), + xdm.target.resource.value = to_string(parsed_fields_NewValue); \ No newline at end of file diff --git a/Packs/Absolute/ModelingRules/Absolute_2_11/Absolute_2_11.yml b/Packs/Absolute/ModelingRules/Absolute_2_11/Absolute_2_11.yml new file mode 100644 index 000000000000..99f6197b95a4 --- /dev/null +++ b/Packs/Absolute/ModelingRules/Absolute_2_11/Absolute_2_11.yml @@ -0,0 +1,8 @@ +fromversion: 8.15.0 +id: Absolute_ModelingRule +name: Absolute Modeling Rule +rules: '' +schema: '' +tags: '' +supportedModules: +- xsiam diff --git a/Packs/Absolute/ModelingRules/Absolute_2_11/Absolute_2_11_schema.json b/Packs/Absolute/ModelingRules/Absolute_2_11/Absolute_2_11_schema.json new file mode 100644 index 000000000000..4f55d8d92cf0 --- /dev/null +++ b/Packs/Absolute/ModelingRules/Absolute_2_11/Absolute_2_11_schema.json @@ -0,0 +1,64 @@ +{ + "absolute_secure_endpoint_raw": { + "parsed_fields_NewValue": { + "type": "string", + "is_array": false + }, + "parsed_fields_OldValue": { + "type": "string", + "is_array": false + }, + "id": { + "type": "string", + "is_array": false + }, + "verb": { + "type": "string", + "is_array": false + }, + "eventType": { + "type": "string", + "is_array": false + }, + "actorDisplayId": { + "type": "string", + "is_array": false + }, + "actorObjectType": { + "type": "string", + "is_array": false + }, + "objectDisplayId": { + "type": "string", + "is_array": false + }, + "actorDisplayName": { + "type": "string", + "is_array": false + }, + "objectObjectType": { + "type": "string", + "is_array": false + }, + "objectDisplayName": { + "type": "string", + "is_array": false + }, + "secondaryObjectType": { + "type": "string", + "is_array": false + }, + "secondaryObjectDisplayName": { + "type": "string", + "is_array": false + }, + "secondaryObjectDisplayId": { + "type": "string", + "is_array": false + }, + "objectProperties": { + "type": "string", + "is_array": false + } + } +} diff --git a/Packs/Absolute/ReleaseNotes/2_0_12.md b/Packs/Absolute/ReleaseNotes/2_0_12.md new file mode 100644 index 000000000000..c0a6628fa084 --- /dev/null +++ b/Packs/Absolute/ReleaseNotes/2_0_12.md @@ -0,0 +1,7 @@ + +#### Modeling Rules + +##### New: Absolute Modeling Rule + +- New: Added a new modeling rule version that maps the `xdm.*.identity.*` fields alongside the existing `xdm.*.user.*` fields. +<~XSIAM> (Available from Cortex XSIAM 2.11). diff --git a/Packs/Absolute/ReleaseNotes/2_0_13.md b/Packs/Absolute/ReleaseNotes/2_0_13.md new file mode 100644 index 000000000000..42540da35bfc --- /dev/null +++ b/Packs/Absolute/ReleaseNotes/2_0_13.md @@ -0,0 +1,9 @@ + +#### Integrations + +##### Absolute + +- Added support for the ***absolute-wipe-request-list*** command that get information about the wipe requests. +- Added support for the ***absolute-wipe-actions-list*** command that get the status of actions by devices or by request. +- Added support for the ***absolute-wipe-request-create*** command that create a wipe request for the devices. +- Added support for the ***absolute-wipe-request-cancel*** command that attempts to cancel all wipe actions with an action status of pending for the request specified by the *request_uid* path parameter. diff --git a/Packs/Absolute/pack_metadata.json b/Packs/Absolute/pack_metadata.json index cb0c5f08b2fd..e260722a62fe 100644 --- a/Packs/Absolute/pack_metadata.json +++ b/Packs/Absolute/pack_metadata.json @@ -2,7 +2,7 @@ "name": "Absolute", "description": "Absolute is an adaptive endpoint security solution that delivers device security, data security and asset management of endpoints", "support": "xsoar", - "currentVersion": "2.0.11", + "currentVersion": "2.0.13", "author": "Cortex XSOAR", "url": "https://www.paloaltonetworks.com/cortex", "email": "", diff --git a/Packs/AbuseDB/ReleaseNotes/1_1_7.md b/Packs/AbuseDB/ReleaseNotes/1_1_7.md new file mode 100644 index 000000000000..0d133b538f05 --- /dev/null +++ b/Packs/AbuseDB/ReleaseNotes/1_1_7.md @@ -0,0 +1,7 @@ + +#### Scripts + +##### AbuseIPDBPopulateIndicators + +- Updated the Docker image to: *demisto/python3:3.12.14.13053055*. + diff --git a/Packs/AbuseDB/ReleaseNotes/1_1_8.md b/Packs/AbuseDB/ReleaseNotes/1_1_8.md new file mode 100644 index 000000000000..7f06baea36f3 --- /dev/null +++ b/Packs/AbuseDB/ReleaseNotes/1_1_8.md @@ -0,0 +1,7 @@ + +#### Scripts + +##### AbuseIPDBPopulateIndicators + +- Updated the Docker image to: *demisto/python3:3.12.14.13495286*. + diff --git a/Packs/AbuseDB/Scripts/AbuseIPDBPopulateIndicators/AbuseIPDBPopulateIndicators.yml b/Packs/AbuseDB/Scripts/AbuseIPDBPopulateIndicators/AbuseIPDBPopulateIndicators.yml index 9cdc78436a99..3df275eefc5f 100644 --- a/Packs/AbuseDB/Scripts/AbuseIPDBPopulateIndicators/AbuseIPDBPopulateIndicators.yml +++ b/Packs/AbuseDB/Scripts/AbuseIPDBPopulateIndicators/AbuseIPDBPopulateIndicators.yml @@ -28,4 +28,4 @@ runas: DBotWeakRole tests: - AbuseIPDB PopulateIndicators Test fromversion: 5.0.0 -dockerimage: demisto/python3:3.12.13.10404775 +dockerimage: demisto/python3:3.12.14.13495286 diff --git a/Packs/AbuseDB/pack_metadata.json b/Packs/AbuseDB/pack_metadata.json index 23dcdbd6e439..369cf74eb29a 100644 --- a/Packs/AbuseDB/pack_metadata.json +++ b/Packs/AbuseDB/pack_metadata.json @@ -2,7 +2,7 @@ "name": "AbuseIPDB", "description": "Central repository to report and identify IP addresses that have been associated with malicious activity online. Check the Detailed Information section for more information on how to configure the integration.", "support": "xsoar", - "currentVersion": "1.1.6", + "currentVersion": "1.1.8", "author": "Cortex XSOAR", "url": "https://www.paloaltonetworks.com/cortex", "email": "", diff --git a/Packs/AccessInvestigation/ReleaseNotes/1_2_20.md b/Packs/AccessInvestigation/ReleaseNotes/1_2_20.md new file mode 100644 index 000000000000..4a0c5ba98e57 --- /dev/null +++ b/Packs/AccessInvestigation/ReleaseNotes/1_2_20.md @@ -0,0 +1,3 @@ +## Access Investigation + +- Locked dependencies of the pack to ensure stability for versioned core packs. No changes in this release. diff --git a/Packs/AccessInvestigation/pack_metadata.json b/Packs/AccessInvestigation/pack_metadata.json index 5b581313d239..fa57e09e5a59 100644 --- a/Packs/AccessInvestigation/pack_metadata.json +++ b/Packs/AccessInvestigation/pack_metadata.json @@ -2,7 +2,7 @@ "name": "Access Investigation", "description": "This Content Pack automates response to unauthorised access incidents and contains customer access incident views and layouts to aid investigation.", "support": "xsoar", - "currentVersion": "1.2.19", + "currentVersion": "1.2.20", "author": "Cortex XSOAR", "url": "https://www.paloaltonetworks.com/cortex", "email": "", diff --git a/Packs/ActiveMQ/Integrations/ActiveMQ/ActiveMQ.yml b/Packs/ActiveMQ/Integrations/ActiveMQ/ActiveMQ.yml index 7cf1d713395a..2bb185069797 100644 --- a/Packs/ActiveMQ/Integrations/ActiveMQ/ActiveMQ.yml +++ b/Packs/ActiveMQ/Integrations/ActiveMQ/ActiveMQ.yml @@ -60,6 +60,13 @@ configuration: type: 13 required: false section: Collect +- display: Incidents Fetch Interval + name: incidentFetchInterval + defaultvalue: '1' + required: false + type: 19 + section: Collect + advanced: true - display: Topic Name (for subscription) name: topic-name type: 0 diff --git a/Packs/ActiveMQ/ReleaseNotes/1_1_21.md b/Packs/ActiveMQ/ReleaseNotes/1_1_21.md new file mode 100644 index 000000000000..d44db4d3b155 --- /dev/null +++ b/Packs/ActiveMQ/ReleaseNotes/1_1_21.md @@ -0,0 +1,6 @@ + +#### Integrations + +##### ActiveMQ + +- Updated the Fetch Incidents configuration parameters. diff --git a/Packs/ActiveMQ/pack_metadata.json b/Packs/ActiveMQ/pack_metadata.json index 5955e10c9135..6206929ab9b8 100644 --- a/Packs/ActiveMQ/pack_metadata.json +++ b/Packs/ActiveMQ/pack_metadata.json @@ -2,7 +2,7 @@ "name": "ActiveMQ", "description": "Uses Durable Topic Subscribers to fetch messages and ingest them as incidents in Demisto.", "support": "xsoar", - "currentVersion": "1.1.20", + "currentVersion": "1.1.21", "author": "Cortex XSOAR", "url": "https://www.paloaltonetworks.com/cortex", "email": "", diff --git a/Packs/Active_Directory_Query/.secrets-ignore b/Packs/Active_Directory_Query/.secrets-ignore index c4ddce660f8d..c2d9c0ba9ba8 100644 --- a/Packs/Active_Directory_Query/.secrets-ignore +++ b/Packs/Active_Directory_Query/.secrets-ignore @@ -14,3 +14,4 @@ test@demisto.int user@test.local manager@test.local eric mann +local admin diff --git a/Packs/Active_Directory_Query/ReleaseNotes/1_8_1.md b/Packs/Active_Directory_Query/ReleaseNotes/1_8_1.md new file mode 100644 index 000000000000..3c0f4ad40bb5 --- /dev/null +++ b/Packs/Active_Directory_Query/ReleaseNotes/1_8_1.md @@ -0,0 +1,7 @@ + +#### Scripts + +##### IAMInitADUser + +- Updated the Docker image to: *demisto/py3-tools:1.0.0.11597437*. + diff --git a/Packs/Active_Directory_Query/ReleaseNotes/1_8_2.md b/Packs/Active_Directory_Query/ReleaseNotes/1_8_2.md new file mode 100644 index 000000000000..d78dbdae3139 --- /dev/null +++ b/Packs/Active_Directory_Query/ReleaseNotes/1_8_2.md @@ -0,0 +1,11 @@ + +#### Scripts + +##### IAMInitADUser + +- Updated the Docker image to: *demisto/py3-tools:1.0.0.13176850*. + +##### SendEmailToManager + +- Updated the Docker image to: *demisto/python3:3.12.14.13053055*. + diff --git a/Packs/Active_Directory_Query/ReleaseNotes/1_8_3.md b/Packs/Active_Directory_Query/ReleaseNotes/1_8_3.md new file mode 100644 index 000000000000..bfeb2caaf231 --- /dev/null +++ b/Packs/Active_Directory_Query/ReleaseNotes/1_8_3.md @@ -0,0 +1,3 @@ +## Active Directory Query + +- Locked dependencies of the pack to ensure stability for versioned core packs. No changes in this release. diff --git a/Packs/Active_Directory_Query/Scripts/IAMInitADUser/IAMInitADUser.yml b/Packs/Active_Directory_Query/Scripts/IAMInitADUser/IAMInitADUser.yml index b66729f29728..e7554ebce83c 100644 --- a/Packs/Active_Directory_Query/Scripts/IAMInitADUser/IAMInitADUser.yml +++ b/Packs/Active_Directory_Query/Scripts/IAMInitADUser/IAMInitADUser.yml @@ -80,7 +80,7 @@ tags: - active directory - Utility type: python -dockerimage: demisto/py3-tools:1.0.0.10120494 +dockerimage: demisto/py3-tools:1.0.0.13176850 runas: DBotWeakRole tests: - Active Directory Test diff --git a/Packs/Active_Directory_Query/Scripts/SendEmailToManager/SendEmailToManager.yml b/Packs/Active_Directory_Query/Scripts/SendEmailToManager/SendEmailToManager.yml index 1d47d2c81a3e..f2dc6439de39 100644 --- a/Packs/Active_Directory_Query/Scripts/SendEmailToManager/SendEmailToManager.yml +++ b/Packs/Active_Directory_Query/Scripts/SendEmailToManager/SendEmailToManager.yml @@ -42,7 +42,7 @@ dependson: timeout: 0s tests: - No test -dockerimage: demisto/python3:3.12.13.10116658 +dockerimage: demisto/python3:3.12.14.13053055 supportedModules: - agentix - xsiam diff --git a/Packs/Active_Directory_Query/pack_metadata.json b/Packs/Active_Directory_Query/pack_metadata.json index e733cbcbbbb0..36648de9846c 100644 --- a/Packs/Active_Directory_Query/pack_metadata.json +++ b/Packs/Active_Directory_Query/pack_metadata.json @@ -2,7 +2,7 @@ "name": "Active Directory Query", "description": "Active Directory Query integration enables you to access and manage Active Directory objects (users, contacts, and computers).", "support": "xsoar", - "currentVersion": "1.8.0", + "currentVersion": "1.8.3", "author": "Cortex XSOAR", "url": "https://www.paloaltonetworks.com/cortex", "email": "", diff --git a/Packs/AdminByRequest/.secrets-ignore b/Packs/AdminByRequest/.secrets-ignore index e69de29bb2d1..f954b9cc4d6d 100644 --- a/Packs/AdminByRequest/.secrets-ignore +++ b/Packs/AdminByRequest/.secrets-ignore @@ -0,0 +1 @@ +local admin \ No newline at end of file diff --git a/Packs/AdminByRequest/ModelingRules/AdminByRequest/AdminByRequest.yml b/Packs/AdminByRequest/ModelingRules/AdminByRequest/AdminByRequest.yml index 73dd6ccc1ade..3351cd5f0444 100644 --- a/Packs/AdminByRequest/ModelingRules/AdminByRequest/AdminByRequest.yml +++ b/Packs/AdminByRequest/ModelingRules/AdminByRequest/AdminByRequest.yml @@ -1,4 +1,5 @@ fromversion: 6.10.0 +toversion: 8.14.0 id: Admin_By_Request_ModelingRule name: Admin By Request Modeling Rule rules: '' diff --git a/Packs/AdminByRequest/ModelingRules/AdminByRequest_2_11/AdminByRequest_2_11.xif b/Packs/AdminByRequest/ModelingRules/AdminByRequest_2_11/AdminByRequest_2_11.xif new file mode 100644 index 000000000000..d8f908be1993 --- /dev/null +++ b/Packs/AdminByRequest/ModelingRules/AdminByRequest_2_11/AdminByRequest_2_11.xif @@ -0,0 +1,47 @@ +[MODEL: dataset=admin_by_request_raw] +filter source_log_type = "auditlog" +| alter additional_data = object_create("Application_Name", application -> name, "Application_Vendor", application -> vendor, "Application_Threat" , application -> threat, "Application_VT_Link", application -> virustotalLink, "Request_Reason", reason, "Approved_By", approvedBy , "Approved_By_Email", approvedByEmail , "Denied_By", deniedBy , "Denied_By_Email", deniedByEmail , "Denied_Reason", deniedReason,"Audit_Log_URL", auditLogURL) +| alter + xdm.target.file.filename = application -> file, + xdm.target.file.path = application -> path, + xdm.target.file.sha256 = application -> ["sha256"], + xdm.target.host.os = computer -> platform, + xdm.event.outcome_reason = application -> scanResult, + xdm.event.description = additional_data, + xdm.target.host.hostname = computer -> name, + xdm.event.type = source_log_type, + xdm.event.operation = type, + xdm.source.user.username = user -> account, + xdm.source.identity.username = user -> account, + xdm.source.user.upn = user -> email, + xdm.source.identity.upn = user -> email; +filter source_log_type = "events" +| alter + event_log_level = to_string(eventLevel), + additional_data = object_create("Application_Name", application -> name, "Application_Vendor", application -> vendor, "Additional_Data", additionalData) +| alter + xdm.target.file.filename = application -> file, + xdm.target.file.path = application -> path, + xdm.target.host.hostname = computerName, + xdm.event.log_level = if(event_log_level = "0", XDM_CONST.LOG_LEVEL_INFORMATIONAL , event_log_level = "1", XDM_CONST.LOG_LEVEL_WARNING , event_log_level = "2", XDM_CONST.LOG_LEVEL_ALERT), + xdm.event.type = source_log_type, + xdm.event.description = additional_data, + xdm.source.user.username = coalesce(userAccount,userName), + xdm.source.identity.username = coalesce(userAccount,userName), + xdm.event.operation = eventText; +filter source_log_type = "request" +| alter additional_data = object_create("Application_Name", application -> name, "Application_Vendor", application -> vendor, "Application_Threat" , application -> threat, "Application_VT_Link", application -> virustotalLink, "Request_Reason", reason, "Approved_By" ,approvedBy , "Approved_By_Email", approvedByEmail , "Denied_By", deniedBy , "Denied_By_Email", deniedByEmail , "Denied_Reason", deniedReason, "Audit_Log_Link", auditlogLink) +| alter + xdm.target.file.filename = application -> file, + xdm.target.file.path = application -> path, + xdm.target.file.sha256 = application -> ["sha256"], + xdm.event.outcome_reason = application -> scanResult, + xdm.target.host.os = computer -> platform, + xdm.event.description = additional_data, + xdm.target.host.hostname = computer -> name, + xdm.event.type = source_log_type, + xdm.source.user.username = user -> account, + xdm.source.identity.username = user -> account, + xdm.source.user.upn = user -> email, + xdm.source.identity.upn = user -> email, + xdm.event.operation = type; \ No newline at end of file diff --git a/Packs/AdminByRequest/ModelingRules/AdminByRequest_2_11/AdminByRequest_2_11.yml b/Packs/AdminByRequest/ModelingRules/AdminByRequest_2_11/AdminByRequest_2_11.yml new file mode 100644 index 000000000000..01da725efb5f --- /dev/null +++ b/Packs/AdminByRequest/ModelingRules/AdminByRequest_2_11/AdminByRequest_2_11.yml @@ -0,0 +1,8 @@ +fromversion: 8.15.0 +id: Admin_By_Request_ModelingRule +name: Admin By Request Modeling Rule +rules: '' +schema: '' +tags: 'Admin By Request' +supportedModules: +- xsiam diff --git a/Packs/AdminByRequest/ModelingRules/AdminByRequest_2_11/AdminByRequest_2_11_schema.json b/Packs/AdminByRequest/ModelingRules/AdminByRequest_2_11/AdminByRequest_2_11_schema.json new file mode 100644 index 000000000000..a45d3754e59a --- /dev/null +++ b/Packs/AdminByRequest/ModelingRules/AdminByRequest_2_11/AdminByRequest_2_11_schema.json @@ -0,0 +1,80 @@ +{ + "admin_by_request_raw": { + "application": { + "type": "string", + "is_array": false + }, + "computer": { + "type": "string", + "is_array": false + }, + "auditLogURL": { + "type": "string", + "is_array": false + }, + "source_log_type": { + "type": "string", + "is_array": false + }, + "type": { + "type": "string", + "is_array": false + }, + "user": { + "type": "string", + "is_array": false + }, + "computerName": { + "type": "string", + "is_array": false + }, + "eventLevel": { + "type": "string", + "is_array": false + }, + "additionalData": { + "type": "string", + "is_array": false + }, + "userAccount": { + "type": "string", + "is_array": false + }, + "userName": { + "type": "string", + "is_array": false + }, + "reason": { + "type": "string", + "is_array": false + }, + "eventText": { + "type": "string", + "is_array": false + }, + "auditlogLink": { + "type": "string", + "is_array": false + }, + "approvedBy": { + "type": "string", + "is_array": false + }, + "approvedByEmail": { + "type": "string", + "is_array": false + }, + "deniedBy": { + "type": "string", + "is_array": false + }, + "deniedByEmail": { + "type": "string", + "is_array": false + }, + "deniedReason": { + "type": "string", + "is_array": false + } + } +} \ No newline at end of file diff --git a/Packs/AdminByRequest/ReleaseNotes/1_0_10.md b/Packs/AdminByRequest/ReleaseNotes/1_0_10.md new file mode 100644 index 000000000000..6ad1bbba4a99 --- /dev/null +++ b/Packs/AdminByRequest/ReleaseNotes/1_0_10.md @@ -0,0 +1,7 @@ + +#### Modeling Rules + +##### New: AdminByRequest Modeling Rule + +- New: Added a new modeling rule version that maps the `xdm.*.identity.*` fields alongside the existing `xdm.*.user.*` fields. +<~XSIAM> (Available from Cortex XSIAM 2.11). diff --git a/Packs/AdminByRequest/ReleaseNotes/1_0_11.md b/Packs/AdminByRequest/ReleaseNotes/1_0_11.md new file mode 100644 index 000000000000..ac2edc379bf8 --- /dev/null +++ b/Packs/AdminByRequest/ReleaseNotes/1_0_11.md @@ -0,0 +1,4 @@ + +## AdminByRequest + +- Documentation and metadata improvements. diff --git a/Packs/AdminByRequest/ReleaseNotes/1_0_12.md b/Packs/AdminByRequest/ReleaseNotes/1_0_12.md new file mode 100644 index 000000000000..88eff6ff3db5 --- /dev/null +++ b/Packs/AdminByRequest/ReleaseNotes/1_0_12.md @@ -0,0 +1,7 @@ + +#### Modeling Rules + +##### New: Admin By Request Modeling Rule + +- New: Added a new modeling rule version that maps the `xdm.*.identity.*` fields alongside the existing `xdm.*.user.*` fields. +<~XSIAM> (Available from Cortex XSIAM 2.11). diff --git a/Packs/AdminByRequest/pack_metadata.json b/Packs/AdminByRequest/pack_metadata.json index 4563d4f9d815..d6ce0c628687 100644 --- a/Packs/AdminByRequest/pack_metadata.json +++ b/Packs/AdminByRequest/pack_metadata.json @@ -2,7 +2,7 @@ "name": "Admin By Request", "description": "AdminByRequest is a Privileged Access Management (PAM) solution that enables secure, temporary elevation to local admin rights.", "support": "xsoar", - "currentVersion": "1.0.9", + "currentVersion": "1.0.12", "author": "Cortex XSOAR", "url": "https://www.paloaltonetworks.com/cortex", "email": "", @@ -24,4 +24,4 @@ "agentix", "xsiam" ] -} \ No newline at end of file +} diff --git a/Packs/Aella_StarLight/Integrations/AellaStarLight/AellaStarLight.yml b/Packs/Aella_StarLight/Integrations/AellaStarLight/AellaStarLight.yml index 86ea047bb9c8..7c785962dae4 100644 --- a/Packs/Aella_StarLight/Integrations/AellaStarLight/AellaStarLight.yml +++ b/Packs/Aella_StarLight/Integrations/AellaStarLight/AellaStarLight.yml @@ -32,6 +32,13 @@ configuration: type: 13 required: false section: Collect +- display: Incidents Fetch Interval + name: incidentFetchInterval + defaultvalue: '1' + required: false + type: 19 + section: Collect + advanced: true - display: Fetching interval in minutes (default is 15, minimum is 15 ) name: fetch_interval section: Collect diff --git a/Packs/Aella_StarLight/ReleaseNotes/1_0_21.md b/Packs/Aella_StarLight/ReleaseNotes/1_0_21.md new file mode 100644 index 000000000000..bff89269999d --- /dev/null +++ b/Packs/Aella_StarLight/ReleaseNotes/1_0_21.md @@ -0,0 +1,6 @@ + +#### Integrations + +##### Aella Star Light + +- Updated the Fetch Incidents configuration parameters. diff --git a/Packs/Aella_StarLight/pack_metadata.json b/Packs/Aella_StarLight/pack_metadata.json index f676baa5a343..9703214dab12 100644 --- a/Packs/Aella_StarLight/pack_metadata.json +++ b/Packs/Aella_StarLight/pack_metadata.json @@ -2,7 +2,7 @@ "name": "Aella Star Light", "description": "Aella Star Light Integration", "support": "community", - "currentVersion": "1.0.20", + "currentVersion": "1.0.21", "author": "Aella Star Light", "url": "", "email": "", diff --git a/Packs/AgariPhishingDefense/Integrations/AgariPhishingDefense/AgariPhishingDefense.yml b/Packs/AgariPhishingDefense/Integrations/AgariPhishingDefense/AgariPhishingDefense.yml index 9288790db2fd..3bac221e592a 100644 --- a/Packs/AgariPhishingDefense/Integrations/AgariPhishingDefense/AgariPhishingDefense.yml +++ b/Packs/AgariPhishingDefense/Integrations/AgariPhishingDefense/AgariPhishingDefense.yml @@ -3,33 +3,44 @@ provider: Fortra commonfields: id: Agari Phishing Defense version: -1 +sectionorder: +- Connect +- Collect configuration: - defaultvalue: https://api.agari.com/v1/ep display: URL to connect to Agari name: url required: true type: 0 + section: Connect - additionalinfo: API Client ID created in the User management UI. display: API Key name: apikey required: true type: 4 + section: Connect - additionalinfo: API Client Secret created in the User management UI. display: Secret Key name: apisecret required: true type: 4 + section: Connect - defaultvalue: '50' display: Maximum number of incidents to fetch every time name: max_fetch required: true type: 0 + section: Collect + supportedModules: + - agentix + - xsiam - additionalinfo: The time range to consider for the initial data fetch. ( , e.g., 2 minutes, 2 hours, 2 days, 2 months, 2 years). Default is 3 days. defaultvalue: 3 days display: First fetch time interval name: first_fetch type: 0 required: false + section: Collect - additionalinfo: Apply the filter to fetch policy events having actions such as "deliver", "mark-spam", "move", "inbox", "delete", "none". Fetches all types of actions by default. display: Policy action taken name: fetch_policy_actions @@ -42,6 +53,7 @@ configuration: - none type: 15 required: false + section: Collect - additionalinfo: Exclude alerts while fetching policy events of type 'Message Alert' or 'System Alert'. defaultvalue: System Alert display: Exclude alerts @@ -51,28 +63,50 @@ configuration: - Message Alert type: 15 required: false + section: Collect - additionalinfo: "Syntax: field.operator(operand) {and/or field.operator(operand)} \n\nExample: created_at.after(2020-11-11T09:16:44Z) or id.eq(537557694)" display: Fetches policy events as per the filter to limit the amount of data. Can be applied to specific fields. name: policy_filter type: 12 required: false + section: Collect - defaultvalue: Agari Phishing Defense Policy Event display: Incident type name: incidentType type: 13 required: false + section: Collect + supportedModules: + - agentix + - xsiam +- display: Incidents Fetch Interval + name: incidentFetchInterval + defaultvalue: '1' + required: false + type: 19 + section: Collect + advanced: true + supportedModules: + - agentix + - xsiam - display: Fetch incidents name: isFetch type: 8 required: false + section: Collect + supportedModules: + - agentix + - xsiam - display: Trust any certificate (not secure) name: insecure type: 8 required: false + section: Connect - display: Use system proxy settings name: proxy type: 8 required: false + section: Connect description: Agari Phishing Defense stops phishing, BEC, and other identity deception attacks that trick employees into harming your business. display: Agari Phishing Defense name: Agari Phishing Defense diff --git a/Packs/AgariPhishingDefense/ReleaseNotes/1_1_28.md b/Packs/AgariPhishingDefense/ReleaseNotes/1_1_28.md new file mode 100644 index 000000000000..78398a3f8696 --- /dev/null +++ b/Packs/AgariPhishingDefense/ReleaseNotes/1_1_28.md @@ -0,0 +1,6 @@ + +#### Integrations + +##### Agari Phishing Defense + +- Documentation and metadata improvements. diff --git a/Packs/AgariPhishingDefense/pack_metadata.json b/Packs/AgariPhishingDefense/pack_metadata.json index 0483ad9840f7..c43c3c1c4718 100644 --- a/Packs/AgariPhishingDefense/pack_metadata.json +++ b/Packs/AgariPhishingDefense/pack_metadata.json @@ -2,7 +2,7 @@ "name": "Agari Phishing Defense", "description": "Use the Agari Phishing Defense integration to retrieve Policy Events as Incidents, retrieve messages and remediate suspected messages.", "support": "partner", - "currentVersion": "1.1.27", + "currentVersion": "1.1.28", "author": "Agari", "url": "https://www.agari.com/support/", "email": "support@agari.com", diff --git a/Packs/AggregatedScripts/.pack-ignore b/Packs/AggregatedScripts/.pack-ignore index 7b5bf9e92e79..0760cc5343e5 100644 --- a/Packs/AggregatedScripts/.pack-ignore +++ b/Packs/AggregatedScripts/.pack-ignore @@ -15,3 +15,6 @@ ignore=BA127 [file:IndicatorEnrichment.yml] ignore=BA127,SC101 + +[file:BlockDomain.yml] +ignore=BA129 diff --git a/Packs/AggregatedScripts/ReleaseNotes/1_3_53.md b/Packs/AggregatedScripts/ReleaseNotes/1_3_53.md new file mode 100644 index 000000000000..0f6f6b308ef8 --- /dev/null +++ b/Packs/AggregatedScripts/ReleaseNotes/1_3_53.md @@ -0,0 +1,7 @@ +<~PLATFORM> +#### Scripts + +##### quarantine-file + +- Updated the script to run the Cortex Core quarantine file action as a Cortex platform built-in command. On the platform, the executing brand is now reported as "Cortex Builtin". + diff --git a/Packs/AggregatedScripts/ReleaseNotes/1_4_0.md b/Packs/AggregatedScripts/ReleaseNotes/1_4_0.md new file mode 100644 index 000000000000..1a4e772ffc65 --- /dev/null +++ b/Packs/AggregatedScripts/ReleaseNotes/1_4_0.md @@ -0,0 +1,6 @@ + +#### Scripts + +##### New: block-domain + +- Added the **block-domain** script, which blocks one or more domains across your configured security products. Currently only supports Palo Alto Networks Panorama. diff --git a/Packs/AggregatedScripts/ReleaseNotes/1_4_1.md b/Packs/AggregatedScripts/ReleaseNotes/1_4_1.md new file mode 100644 index 000000000000..c288ef99d31f --- /dev/null +++ b/Packs/AggregatedScripts/ReleaseNotes/1_4_1.md @@ -0,0 +1,6 @@ + +#### Scripts + +##### block-url + +New: Added a script that blocks a list of URLs in supported integrations. diff --git a/Packs/AggregatedScripts/ReleaseNotes/1_4_2.md b/Packs/AggregatedScripts/ReleaseNotes/1_4_2.md new file mode 100644 index 000000000000..3a23f01957e7 --- /dev/null +++ b/Packs/AggregatedScripts/ReleaseNotes/1_4_2.md @@ -0,0 +1,3 @@ +## Aggregated Scripts + +- Locked dependencies of the pack to ensure stability for versioned core packs. No changes in this release. diff --git a/Packs/AggregatedScripts/ReleaseNotes/1_4_3.md b/Packs/AggregatedScripts/ReleaseNotes/1_4_3.md new file mode 100644 index 000000000000..abb525bdf688 --- /dev/null +++ b/Packs/AggregatedScripts/ReleaseNotes/1_4_3.md @@ -0,0 +1,63 @@ + +#### Scripts + +##### expire-password + +- Updated the Docker image to: *demisto/python3:3.12.14.13053055*. + +##### url-enrichment + +- Updated the Docker image to: *demisto/python3:3.12.14.13053055*. + +##### quarantine-file + +- Updated the Docker image to: *demisto/python3:3.12.14.13053055*. + +##### clear-user-session + +- Updated the Docker image to: *demisto/python3:3.12.14.13053055*. + +##### file-enrichment + +- Updated the Docker image to: *demisto/python3:3.12.14.13053055*. + +##### get-user-data + +- Updated the Docker image to: *demisto/python3:3.12.14.13053055*. + +##### domain-enrichment + +- Updated the Docker image to: *demisto/python3:3.12.14.13053055*. + +##### ip-enrichment + +- Updated the Docker image to: *demisto/python3:3.12.14.13053055*. + +##### isolate-endpoint + +- Updated the Docker image to: *demisto/python3:3.12.14.13053055*. + +##### cve-enrichment + +- Updated the Docker image to: *demisto/python3:3.12.14.13053055*. + +##### indicator-enrichment + +- Updated the Docker image to: *demisto/python3:3.12.14.13053055*. + +##### disable-user + +- Updated the Docker image to: *demisto/python3:3.12.14.13053055*. + +##### block-domain + +- Updated the Docker image to: *demisto/python3:3.12.14.13053055*. + +##### get-endpoint-data + +- Updated the Docker image to: *demisto/python3:3.12.14.13053055*. + +##### block-external-ip + +- Updated the Docker image to: *demisto/python3:3.12.14.13053055*. + diff --git a/Packs/AggregatedScripts/ReleaseNotes/1_4_4.md b/Packs/AggregatedScripts/ReleaseNotes/1_4_4.md new file mode 100644 index 000000000000..e801215325b7 --- /dev/null +++ b/Packs/AggregatedScripts/ReleaseNotes/1_4_4.md @@ -0,0 +1,53 @@ + +#### Scripts + +##### isolate-endpoint + +- Updated the Docker image to: *demisto/python3:3.12.14.13495286*. + +##### url-enrichment + +- Updated the Docker image to: *demisto/python3:3.12.14.13495286*. + +##### quarantine-file + +- Updated the Docker image to: *demisto/python3:3.12.14.13495286*. + +##### block-url + +- Updated the Docker image to: *demisto/python3:3.12.14.13495286*. + +##### file-enrichment + +- Updated the Docker image to: *demisto/python3:3.12.14.13495286*. + +##### get-user-data + +- Updated the Docker image to: *demisto/python3:3.12.14.13495286*. + + +##### cve-enrichment + +- Updated the Docker image to: *demisto/python3:3.12.14.13495286*. + +##### indicator-enrichment + +- Updated the Docker image to: *demisto/python3:3.12.14.13495286*. + +##### block-domain + +- Updated the Docker image to: *demisto/python3:3.12.14.13495286*. + + +##### clear-user-session + +- Updated the Docker image to: *demisto/python3:3.12.14.13495286*. + +##### domain-enrichment + +- Updated the Docker image to: *demisto/python3:3.12.14.13495286*. + +##### block-external-ip + +- Updated the Docker image to: *demisto/python3:3.12.14.13495286*. + diff --git a/Packs/AggregatedScripts/Scripts/BlockDomain/BlockDomain.py b/Packs/AggregatedScripts/Scripts/BlockDomain/BlockDomain.py new file mode 100644 index 000000000000..b58259d9cdf9 --- /dev/null +++ b/Packs/AggregatedScripts/Scripts/BlockDomain/BlockDomain.py @@ -0,0 +1,1152 @@ +import hashlib +import re +from typing import Any + +import demistomock as demisto # noqa: F401 +from CommonServerPython import * # noqa: F401 +from CommonServerUserPython import * # noqa: F401 + +""" CONSTANTS """ + +SUPPORTED_BRANDS = ["Panorama"] + +OBJECT_NAME_PREFIX = "Cortex-" +# PAN-OS object names are capped at 63 chars; reserve room for the prefix and hash suffix. +MAX_OBJECT_NAME_LENGTH = 63 +HASH_SUFFIX_LENGTH = 8 +OBJECT_NAME_SANITIZE_REGEX = re.compile(r"[^A-Za-z0-9.\-]") + +PRE_POST = "pre-rulebase" + +FQDN_REGEX = re.compile(r"^(?=.{1,253}$)(?!-)[A-Za-z0-9-]{1,63}(? bool: + """Check whether a domain is a wildcard (unsupported). + + Args: + domain (str): The domain to check. + Returns: + True if the domain contains an asterisk, False otherwise. + """ + return "*" in domain + + +def is_valid_fqdn(domain: str) -> bool: + """Check whether a value is a valid, non-wildcard FQDN. + + Args: + domain (str): The domain to validate. + Returns: + True if the value looks like a valid FQDN, False otherwise. + """ + return bool(FQDN_REGEX.match(domain)) + + +def derive_object_name(domain: str) -> str: + """Derive a deterministic PAN-OS address-object name from a domain. + + The name is a pure function of the domain so re-runs are idempotent. On overflow of the PAN-OS + max object-name length, the sanitised body is truncated and a short deterministic hash suffix is + appended to keep the name unique. + + Args: + domain (str): The domain to derive the object name from. + Returns: + The derived object name (for example, 'Cortex-evil.example.com'). + """ + sanitised = OBJECT_NAME_SANITIZE_REGEX.sub("-", domain).strip("-") + candidate = f"{OBJECT_NAME_PREFIX}{sanitised}" + if len(candidate) <= MAX_OBJECT_NAME_LENGTH: + return candidate + + digest = hashlib.sha256(domain.encode("utf-8")).hexdigest()[:HASH_SUFFIX_LENGTH] + keep = MAX_OBJECT_NAME_LENGTH - len(OBJECT_NAME_PREFIX) - 1 - HASH_SUFFIX_LENGTH # 1 for the '-' separator. + truncated = sanitised[:keep].strip("-") + return f"{OBJECT_NAME_PREFIX}{truncated}-{digest}" + + +def normalize_tags(raw_tags: Any) -> list[str]: + """Normalize a PAN-OS tags value into a clean list of tag names. + + PAN-OS may return an entity's tags as a single string, a list of strings, or None/absent. + This flattens all of those into a list (empty when there are no tags). + + Args: + raw_tags (Any): The raw 'Tags' value from a PAN-OS context entry. + Returns: + A list of tag-name strings (possibly empty). + """ + if not raw_tags: + return [] + if isinstance(raw_tags, list): + return [str(tag) for tag in raw_tags if tag] + return [str(raw_tags)] + + +def build_result_row( + domain: str, + brand: str | None, + status: str, + result: str, + message: str, + rule_name: str | None = None, +) -> dict: + """Assemble a single BlockDomain row. + + Any field with no meaningful value is stored as ``None`` (real JSON null) rather than an empty + string, so downstream context never carries empty-string placeholders (e.g. a failed-validation + row has ``Brand: null`` and ``RuleName: null`` instead of ``""``). + + Note: there is intentionally no ``Action`` field - the security team confirmed that ``Result`` + plus ``Message`` convey everything they need, so per-row Created/Modified/Unchanged is not + surfaced. Change-detection is still performed internally to gate the commit/push. + + Args: + domain (str): The processed domain. + brand (str | None): The brand used, or None when not applicable (e.g. failed validation). + status (str): The lifecycle status. + result (str): Success or Failed. + message (str): A human-readable message. + rule_name (str | None): The rule name used, or None when no rule was involved. + Returns: + A dict representing a single result row, with empty values normalised to None. + """ + return { + "Domain": domain or None, + "Brand": brand or None, + "Status": status or None, + "Result": result or None, + "RuleName": rule_name or None, + "Message": message or None, + } + + +def validate_domains(domain_list: list) -> tuple[list, list]: + """Split the input into valid domains and failed-validation rows. + + Wildcard and invalid entries fail validation and never reach a vendor; they produce a per-row + Failed result while the rest of the list continues. + + Args: + domain_list (list): The list of domains to validate. + Returns: + A tuple of (valid_domains, failed_rows). + """ + valid_domains: list = [] + failed_rows: list = [] + for domain in domain_list: + if is_wildcard(domain): + failed_rows.append( + build_result_row( + domain=domain, + brand=None, + status=STATUS_FAILED, + result=RESULT_FAILED, + message=f"Wildcard domain '{domain}' is not supported by this script; skipped.", + ) + ) + elif not is_valid_fqdn(domain): + failed_rows.append( + build_result_row( + domain=domain, + brand=None, + status=STATUS_FAILED, + result=RESULT_FAILED, + message=f"Invalid FQDN '{domain}' - skipped.", + ) + ) + else: + valid_domains.append(domain) + return valid_domains, failed_rows + + +def get_enabled_brands() -> set: + """Return the set of brands that have at least one active instance. + + Returns: + A set of enabled brand names. + """ + modules = demisto.getModules() + return {module.get("brand") for module in modules.values() if module.get("state") == "active"} + + +""" EXECUTE-COMMAND / CONTEXT HELPERS """ + + +def run_execute_command(command_name: str, args: dict[str, Any]) -> list[dict]: + """Execute a command and return its raw entries. + + Args: + command_name (str): The command to execute. + args (dict): The command arguments. + Returns: + The raw list of command entries. + Raises: + DemistoException: If the command returned no entries (callers index res[0] downstream). + """ + res = demisto.executeCommand(command_name, args) + if not res: + raise DemistoException(f"Command {command_name!r} returned an empty response.") + return res + + +def is_polling_in_progress(poll_response: Any) -> bool: + """Return whether a polling function's result indicates it should keep polling. + + The @polling_function decorator attaches a ScheduledCommand to the returned CommandResults only + while polling should continue; when the job finishes it returns the plain response with none. + + Args: + poll_response (Any): The value returned by a @polling_function-decorated call. + Returns: + True if polling should continue, False when the job has finished. + """ + return getattr(poll_response, "scheduled_command", None) is not None + + +def get_relevant_context(original_context: dict[str, Any], key: str) -> dict | list: + """Get the relevant context object from the execute_command response, tolerating suffixed keys. + + Args: + original_context (dict): The 'EntryContext' from the command response. + key (str): The key to extract. + Returns: + A dict or list that is the relevant command context. + """ + if not original_context: + return {} + if relevant_context := original_context.get(key, {}): + return relevant_context + for k in original_context: + if k.startswith(key): + return original_context.get(k, {}) + return {} + + +""" HUMAN-READABLE / FINAL-RESULT AGGREGATION """ + + +def build_verbose_human_readable(responses: list) -> str: + """Concatenate the per-command human-readable outputs into a single, blank-line-separated string. + + Args: + responses (list): The accumulated command responses (each a list of entries). + Returns: + A single markdown string with each command's human-readable output separated by a blank line, + or an empty string if no command produced human-readable output. + """ + human_readables: list = [] + for res in responses or []: + for entry in res or []: + command_hr = entry.get("HumanReadable") + if command_hr and command_hr != str(None): + human_readables.append(command_hr) + # A leading "" yields a blank line separating the summary table from the first verbose entry. + return "\n\n".join(["", *human_readables]) if human_readables else "" + + +def build_final_command_results(rows: list, verbose: bool, responses: list) -> CommandResults: + """Build the single final CommandResults for the run. + + The CommandResults carries the aggregated BlockDomain context and a markdown summary table. + When verbose is True, the per-command human-readable outputs are appended to the same readable + output (blank-line separated), mirroring the ExpirePassword aggregated script. + + When there is at least one row and every row failed, the whole run is reported as an error. + + Args: + rows (list): The aggregated BlockDomain rows. + verbose (bool): Whether to append per-command human-readable output. + responses (list): The accumulated command responses (used only when verbose). + Returns: + A single CommandResults to return from the script. + """ + all_failed = bool(rows) and all(row.get("Result") == RESULT_FAILED for row in rows) + table_title = "Block Domain: All runs failed." if all_failed else "Block Domain" + readable_output = tableToMarkdown( + table_title, + rows, + headers=["Domain", "Brand", "Status", "Result", "RuleName", "Message"], + removeNull=True, + ) + if all_failed: + readable_output += "\n\n**All runs failed.** Review the table above for the specific error messages." + if verbose: + readable_output += build_verbose_human_readable(responses) + command_results = CommandResults( + outputs_prefix="BlockDomain", + outputs_key_field=["Domain", "Brand"], + outputs=rows, + readable_output=readable_output, + ) + if all_failed: + # Surface the whole run as an error entry so the war room / playbook can branch on failure. + command_results.entry_type = EntryType.ERROR + return command_results + + +""" PAN-OS FLOW """ + + +class DynamicGroupError(Exception): + """Raised when the target address-group exists and is dynamic (customer-managed).""" + + +class PanOs: + """Implements the PAN-OS static-address-group domain-blocking flow. + + The address-group and the deny rule are singletons (their names are constant), so they are + ensured once per run. Each valid domain then gets an FQDN address-object that is added to the + group. Commit + optional push happen once after all domains are processed. + """ + + def __init__(self, args: dict) -> None: + """Initialize the PanOs flow. + + Args: + args (dict): The flow arguments (domains, rule_name, address_group, tag, etc.). + """ + self.args = args + self.brand = "Panorama" + self.rule_name = args["rule_name"] + self.address_group = args["address_group"] + self.tag = args.get("tag", "") + self.log_forwarding_name = args.get("log_forwarding_name", "") + # Key MUST match the YAML arg name; args_for_next_run re-passes it during polling re-entry. + self.domains: list = args.get("domain_list", []) + self.responses: list = [] + # True once the rule was created or edited this run + self._rule_changed: bool = False + # True once any address object was created, or added to the group, this run. + self._address_made_changes: bool = False + # The pre-existing rule's current tags (captured once per run by rule_destinations), used to + # merge in the configured tag without duplicating it. + self._rule_tags: list[str] = [] + # The pre-existing rule's current log-forwarding profile (single value, captured once per + # run by rule_destinations). + self._rule_log_forwarding: str = "" + + # ---- tag helper ---------------------------------------------------- + + def ensure_tag(self) -> None: + """Create the configured tag up front so the group and rule can reference it. + + Creating the tag here decouples it from address-object creation, which is important when + every object already exists and the create-address auto-create path is skipped. An error + from an already-existing tag is expected and tolerated (logged, not raised). + + Returns: + None. + """ + if not self.tag: + return + command_name = "pan-os-create-tag" + res = run_execute_command(command_name, {"name": self.tag}) + self.responses.append(res) + if is_error(res): + # An already-existing tag is the common, benign case; log and continue. + demisto.debug( + f"{LOG_TAG} pan-os-create-tag for '{self.tag}' returned an error " f"(likely already exists): {get_error(res)}" + ) + + def merged_tags_if_missing(self, existing_tags: list[str]) -> list[str] | None: + """Return the tag list to write when the configured tag is not already present. + + Args: + existing_tags (list[str]): The entity's current tags. + Returns: + The merged tag list to write, or None when nothing needs to change. + """ + if not self.tag or self.tag in existing_tags: + return None + return [*existing_tags, self.tag] + + # ---- execution helper ---------------------------------------------- + + def execute_or_raise(self, command_name: str, command_args: dict, error_prefix: str) -> list[dict]: + """Run a command, record its response, and raise on error. + + Args: + command_name (str): The command to execute. + command_args (dict): The command arguments. + error_prefix (str): A prefix for the raised error message. + Returns: + The raw command entries. + """ + demisto.debug(f"{LOG_TAG} executing {command_name!r} with args={command_args}") + res = run_execute_command(command_name, command_args) + self.responses.append(res) + if is_error(res): + raise DemistoException(f"{error_prefix}: {get_error(res)}") + return res + + # ---- context probes ------------------------------------------------- + + def address_object_exists(self, object_name: str) -> tuple[bool, list[str]]: + """Check whether an address-object already exists, and return its current tags. + + Args: + object_name (str): The address-object name to probe. + Returns: + A tuple of (exists, current_tags). current_tags is empty when the object is absent or + has no tags. + """ + command_name = "pan-os-get-address" + res = run_execute_command(command_name, {"name": object_name}) + # pan-os-get-address raises when the object is absent; treat that as 'does not exist'. + if is_error(res): + return False, [] + self.responses.append(res) + context = get_relevant_context(res[0].get("EntryContext", {}), "Panorama.Addresses") + items = context if isinstance(context, list) else [context] + for item in items: + if item.get("Name") == object_name: + return True, normalize_tags(item.get("Tags")) + return False, [] + + def get_address_group(self) -> dict | None: + """Return the target address-group context dict, or None if it does not exist. + + Returns: + The address-group context dict, or None. + """ + res = self.execute_or_raise("pan-os-list-address-groups", {}, "Failed to list address groups") + context = get_relevant_context(res[0].get("EntryContext", {}), "Panorama.AddressGroups") + items = context if isinstance(context, list) else [context] + for item in items: + if item.get("Name") == self.address_group: + return item + return None + + def rule_destinations(self) -> tuple[bool, list, list[str]]: + """Return whether the rule exists, its current destination list, and its current tags. + + Also records the rule's current log-forwarding profile on ``self._rule_log_forwarding`` + (single-value profile) so ensure_rule can add it only when the configured profile differs. + + Returns: + A tuple of (rule_exists, destination_list, current_tags). + """ + res = self.execute_or_raise("pan-os-list-rules", {"pre_post": PRE_POST}, "Failed to list rules") + context = get_relevant_context(res[0].get("EntryContext", {}), "Panorama.SecurityRule") + items = context if isinstance(context, list) else [context] + for item in items: + if item.get("Name") == self.rule_name: + destination = item.get("Destination") + destination_list = destination if isinstance(destination, list) else [destination] if destination else [] + self._rule_log_forwarding = item.get("LogForwardingProfile") or "" + return True, destination_list, normalize_tags(item.get("Tags")) + return False, [], [] + + # ---- single-run writes (group + rule are singletons) ---------------- + + def ensure_group(self, group_context: dict | None, object_names: list[str]) -> None: + """Create the static address-group with all members, or add this run's members to it. + + A static group references its members by name, so every address object must already exist on + the device before the group is created or edited (otherwise PAN-OS rejects the reference). + Callers must create all objects first, then call this once with the full list of names. + + pan-os-edit-address-group requires exactly one of element_to_add/element_to_remove for a + static group and applies tags only alongside it, so the configured tag is merged into the + same element_to_add call. Re-adding an existing member is a no-op (the command de-duplicates). + + Args: + group_context (dict | None): The existing group's context, or None if it does not exist. + object_names (list[str]): Every address-object name for this run (the full member list). + Returns: + None. + Raises: + DynamicGroupError: If the group exists but is a customer-managed dynamic group. + """ + if group_context is None: + create_args: dict = {"name": self.address_group, "type": "static", "addresses": object_names} + if self.tag: + create_args["tags"] = self.tag + self.execute_or_raise( + "pan-os-create-address-group", create_args, f"Failed to create address-group '{self.address_group}'" + ) + self._address_made_changes = True + return + + group_type = (group_context.get("Type") or "").lower() + if group_type == "dynamic": + raise DynamicGroupError( + f"Address-group '{self.address_group}' already exists as dynamic; " f"will not modify a managed dynamic group." + ) + + existing_members = normalize_tags(group_context.get("Addresses")) + members_to_add = [name for name in object_names if name not in existing_members] + merged_tags = self.merged_tags_if_missing(normalize_tags(group_context.get("Tags"))) + if not members_to_add and merged_tags is None: + return + # element_to_add is required by the edit; when only the tag changed, re-add existing members. + edit_args: dict = {"name": self.address_group, "type": "static", "element_to_add": members_to_add or object_names} + if merged_tags is not None: + edit_args["tags"] = merged_tags + self.execute_or_raise( + "pan-os-edit-address-group", + edit_args, + f"Failed to add object(s) to address-group '{self.address_group}'", + ) + self._address_made_changes = True + + def ensure_rule(self, rule_present: bool, rule_destinations: list) -> None: + """Ensure the deny rule exists, points at the group, carries the tag, and sits at the top. + + Args: + rule_present (bool): Whether the rule already exists. + rule_destinations (list): The rule's current destination list. + Returns: + None. + """ + if not rule_present: + create_rule_args: dict = { + "rulename": self.rule_name, + "action": "deny", + "source": "any", + "destination": self.address_group, + "application": "any", + "service": "any", + "pre_post": PRE_POST, + "where": "top", + } + if self.tag: + create_rule_args["tags"] = self.tag + if self.log_forwarding_name: + create_rule_args["log_forwarding"] = self.log_forwarding_name + self.execute_or_raise("pan-os-create-rule", create_rule_args, f"Failed to create rule '{self.rule_name}'") + self._rule_changed = True + else: + if self.address_group not in rule_destinations: + # Rule exists but doesn't reference our group - add without replacing existing destinations. + self.execute_or_raise( + "pan-os-edit-rule", + { + "rulename": self.rule_name, + "element_to_change": "destination", + "element_value": self.address_group, + "behaviour": "add", + "pre_post": PRE_POST, + }, + f"Failed to add group to rule '{self.rule_name}'", + ) + self._rule_changed = True + # Add the configured tag to the pre-existing rule if it isn't already present. Uses + # behaviour=add so it never clobbers or duplicates the rule's existing tags. + if self.tag and self.tag not in self._rule_tags: + self.execute_or_raise( + "pan-os-edit-rule", + { + "rulename": self.rule_name, + "element_to_change": "tag", + "element_value": self.tag, + "behaviour": "add", + "pre_post": PRE_POST, + }, + f"Failed to add tag to rule '{self.rule_name}'", + ) + self._rule_changed = True + # Add the configured log-forwarding profile to the pre-existing rule if it differs. + if self.log_forwarding_name and self.log_forwarding_name != self._rule_log_forwarding: + self.execute_or_raise( + "pan-os-edit-rule", + { + "rulename": self.rule_name, + "element_to_change": "log-forwarding", + "element_value": self.log_forwarding_name, + "pre_post": PRE_POST, + }, + f"Failed to set log-forwarding profile on rule '{self.rule_name}'", + ) + self._rule_changed = True + # Enforce top placement only when we created/changed the rule; a pre-existing, unchanged + # rule is left where it is to avoid a needless move (and candidate-config churn). + if self._rule_changed or not rule_present: + self.execute_or_raise( + "pan-os-move-rule", + {"rulename": self.rule_name, "where": "top", "pre_post": PRE_POST}, + f"Failed to move rule '{self.rule_name}' to top", + ) + + def ensure_address_object(self, domain: str) -> tuple[str, bool, str]: + """Ensure a single domain's FQDN address-object exists and carries the configured tag. + + Object-only: it does not touch the group or the rule. The group is built afterwards from the + full set of object names, so every member reference resolves at group-creation time. + + Args: + domain (str): The domain to block. + Returns: + A tuple of (object_name, changed, message). 'changed' is True when the object was created + or its tag was added. + """ + object_name = derive_object_name(domain) + object_exists, object_tags = self.address_object_exists(object_name) + if object_exists: + # Merge the configured tag onto the existing object only when it is missing. + merged = self.merged_tags_if_missing(object_tags) + if merged is not None: + self.execute_or_raise( + "pan-os-edit-address", + {"name": object_name, "element_to_change": "tag", "element_value": merged}, + f"Failed to add tag to address-object '{object_name}'", + ) + return object_name, True, f"Address-object '{object_name}' already existed; tag '{self.tag}' added." + return object_name, False, f"Address-object '{object_name}' already exists." + + create_args: dict = {"name": object_name, "fqdn": domain} + if self.tag: + # create_tag=true auto-creates the tag; pan-os-create-address fails otherwise. + create_args["tag"] = self.tag + create_args["create_tag"] = "true" + self.execute_or_raise("pan-os-create-address", create_args, f"Failed to create address-object '{object_name}'") + return object_name, True, f"Address-object '{object_name}' created for '{domain}'." + + # ---- orchestration -------------------------------------------------- + + def process_domains(self) -> list: + """Block all domains in three phases: objects, then the static group, then the rule. + + The phased order is required because a static address-group references its members by name, + so every address object must exist before the group is created or edited. + + Returns: + The list of BlockDomain rows for the processed domains. + """ + rows: list = [] + try: + # Create the tag up front so group/rule creation can reference it even when every + # address object already exists (and the create-address auto-create path is skipped). + self.ensure_tag() + + # Phase 1: create/verify EVERY address object first (each carrying the tag). + object_names: list[str] = [] + per_domain_messages: list[tuple[str, str]] = [] + for domain in self.domains: + object_name, changed, message = self.ensure_address_object(domain) + object_names.append(object_name) + per_domain_messages.append((domain, message)) + self._address_made_changes = self._address_made_changes or changed + + # Phase 2: create/edit the static group ONCE with the full member list (all objects now + # exist, so every reference resolves). Tag is merged in the same call. + group_context = self.get_address_group() + self.ensure_group(group_context, object_names) + + # Phase 3: ensure the rule once (group now exists, so its destination reference resolves). + rule_present, rule_destinations, self._rule_tags = self.rule_destinations() + self.ensure_rule(rule_present, rule_destinations) + + for domain, message in per_domain_messages: + rows.append( + build_result_row( + domain=domain, + brand=self.brand, + status=STATUS_DONE, + result=RESULT_SUCCESS, + rule_name=self.rule_name, + message=f"{message} Added to '{self.address_group}'. Rule '{self.rule_name}' enforced at top.", + ) + ) + except DynamicGroupError as dyn_err: + # Abort the whole brand for this run: the target group is dynamic, so we can't add static + # members. Every requested domain failed to be blocked, so report them as failures. + demisto.error(f"{LOG_TAG} process_domains aborted for domains {self.domains}: {dyn_err}") + for domain in self.domains: + rows.append( + build_result_row( + domain=domain, + brand=self.brand, + status=STATUS_FAILED, + result=RESULT_FAILED, + rule_name="", + message=str(dyn_err), + ) + ) + except Exception as ex: + demisto.error(f"{LOG_TAG} process_domains failed for domains {self.domains}: {traceback.format_exc()}") + for domain in self.domains: + rows.append( + build_result_row( + domain=domain, + brand=self.brand, + status=STATUS_FAILED, + result=RESULT_FAILED, + rule_name=self.rule_name, + message=f"Failed to block '{domain}' on Panorama: {ex!s}", + ) + ) + return rows + + def pan_os_is_panorama(self) -> bool: + """Check whether the instance is a Panorama (vs a single firewall). + + Returns: + True if the instance model is 'Panorama', False otherwise. + """ + command_name = "pan-os" + res = run_execute_command(command_name, {"cmd": "", "type": "op"}) + self.responses.append(res) + context = get_relevant_context(res[0].get("EntryContext", {}), "Panorama.Command") + model = context.get("response", {}).get("result", {}).get("system", {}).get("model", "") # type: ignore + return model == "Panorama" + + def reduce_responses(self) -> list: + """Reduce the accumulated responses to the parts needed across polling cycles. + + Returns: + A list of reduced response entries suitable for serialization to context. + """ + reduced = [] + for res in self.responses: + reduced.append( + [ + { + "HumanReadable": entry.get("HumanReadable"), + "Contents": entry.get("Contents"), + "Type": entry.get("Type"), + "Metadata": entry.get("Metadata"), + } + for entry in res + ] + ) + return reduced + + def restore_responses(self) -> None: + """Restore the accumulated responses that were serialized to context in a previous cycle.""" + stored = demisto.context().get("panorama_responses", "") or "" + self.responses = json.loads(stored) if stored else [] + + def save_responses(self) -> None: + """Serialize the accumulated responses to context for the next polling cycle.""" + demisto.setContext("panorama_responses", json.dumps(self.reduce_responses())) + + def manage_pan_os_flow(self) -> Any: # pragma: no cover + """Dispatch the PAN-OS flow to the correct state. + + On re-entry (a push or commit job is in flight) the flow jumps straight to the relevant + status poller. Otherwise it starts the object/group/rule flow. + + Returns: + A PollResult when a job is in flight, or the list of result rows when finished. + """ + # Polling re-entry rules: + # - commit_job_id is carried in self.args by args_for_next_run. + # - push_job_id is only written to demisto.context() by pan_os_push_to_device. + # - A fresh manual invocation has neither in args; any leftover context is stale and + # must be scrubbed so it can't hijack the fresh run. + incident_context = demisto.context() + commit_job_id = self.args.get("commit_job_id") + context_push_job_id = demisto.get(incident_context, "push_job_id") + context_commit_job_id = demisto.get(incident_context, "commit_job_id") + + is_polling_reentry = bool(commit_job_id) + if not is_polling_reentry and (context_commit_job_id or context_push_job_id): + demisto.debug( + f"{LOG_TAG} Stale polling context on fresh invocation " + f"(commit={context_commit_job_id!r}, push={context_push_job_id!r}); clearing." + ) + demisto.setContext("commit_job_id", "") + demisto.setContext("push_job_id", "") + demisto.setContext("panorama_responses", "") + demisto.setContext("block_domain_rows", "") + context_push_job_id = None + + push_job_id = context_push_job_id if is_polling_reentry else None + + demisto.debug(f"{LOG_TAG} dispatch: {commit_job_id=}, {push_job_id=}") + if push_job_id: + return self.handle_push_in_flight(push_job_id) + if commit_job_id: + return self.handle_commit_in_flight(commit_job_id) + return self.start_flow() + + def handle_push_in_flight(self, push_job_id: str) -> Any: # pragma: no cover + """Poll the status of an in-flight push-to-device-group job. + + Args: + push_job_id (str): The push job ID to poll. + Returns: + A PollResult while the push is running, or the final result rows when it finishes. + """ + self.restore_responses() + self.args["push_job_id"] = push_job_id + res_push_status = pan_os_push_status(self.args, self.responses) + if not is_polling_in_progress(res_push_status): + demisto.debug(f"{LOG_TAG} Push job {push_job_id} finished.") + return self.finish() + self.save_responses() + return res_push_status + + def handle_commit_in_flight(self, commit_job_id: str) -> Any: # pragma: no cover + """Poll the status of an in-flight commit job, then start the push if needed. + + Args: + commit_job_id (str): The commit job ID to poll. + Returns: + A PollResult while commit/push is running, or the final result rows when finished. + """ + self.args["commit_job_id"] = commit_job_id + self.restore_responses() + poll_commit_status = pan_os_commit_status(self.args, self.responses) + if is_polling_in_progress(poll_commit_status): + self.save_responses() + return poll_commit_status + demisto.debug(f"{LOG_TAG} Commit job {commit_job_id} finished.") + # Commit finished - push to the device group if this is Panorama. + if self.pan_os_is_panorama(): + poll_push = pan_os_push_to_device(self.args, self.responses) + if not is_polling_in_progress(poll_push): + return self.finish() + self.save_responses() + return poll_push + return self.finish() + + def start_flow(self) -> Any: # pragma: no cover + """Run the object/group/rule flow, then start the commit if there were changes. + + Returns: + A PollResult while the commit is running, or the final result rows when finished. + """ + rows = self.process_domains() + demisto.setContext("block_domain_rows", json.dumps(rows)) + # Commit/push only when the run mutated Panorama state, to avoid a needless multi-minute + # push poll on a pure "nothing changed" run. + made_changes = self._address_made_changes or self._rule_changed + auto_commit = argToBoolean(self.args.get("auto_commit", True)) + demisto.debug(f"{LOG_TAG} start_flow: {made_changes=}, {auto_commit=}, {len(rows)} row(s)") + if made_changes and auto_commit: + poll_commit = pan_os_commit(self.args, self.responses) + if not is_polling_in_progress(poll_commit): + return self.finish() + self.save_responses() + return poll_commit + # Nothing changed on the device: return the rows and skip the commit/push. + demisto.debug(f"{LOG_TAG} start_flow: no changes detected; skipping commit/push.") + return rows + + def finish(self) -> list: # pragma: no cover + """Clean up polling context and return the final result rows. + + The accumulated responses (restored from context across polling cycles) are kept on the + instance so the caller can build verbose output before they are cleared from context. + + Returns: + The list of BlockDomain rows accumulated for the run. + """ + rows_raw = demisto.context().get("block_domain_rows", "") or "" + # Preserve responses on the instance for verbose output before clearing context. + stored = demisto.context().get("panorama_responses", "") or "" + if stored: + try: + self.responses = json.loads(stored) + except (ValueError, TypeError) as err: + demisto.debug(f"{LOG_TAG} Could not parse stored responses from context; ignoring. Error: {err}") + demisto.setContext("commit_job_id", "") + demisto.setContext("push_job_id", "") + demisto.setContext("panorama_responses", "") + demisto.setContext("block_domain_rows", "") + try: + return json.loads(rows_raw) if rows_raw else [] + except (ValueError, TypeError): + return [] + + +""" POLLING FUNCTIONS (commit / push) """ + + +@polling_function(name="block-domain", interval=60, timeout=1200) +def pan_os_commit(args: dict, responses: list) -> PollResult: + """Execute pan-os-commit. + + Args: + args (dict): The arguments of the function. + responses (list): The responses of the commands executed so far. + Returns: + The PollResult object. + """ + command_name = "pan-os-commit" + res_commit = run_execute_command(command_name, {"polling": True}) + responses.append(res_commit) + # Surface a failed commit instead of silently treating it as "no changes to commit". + if is_error(res_commit): + failure_output = {"Status": "Failure"} + return PollResult( + response=CommandResults( + outputs=failure_output, + readable_output=tableToMarkdown("Commit Status:", failure_output, removeNull=True), + ), + args_for_next_run=args, + continue_to_poll=False, + ) + polling_args = res_commit[0].get("Metadata", {}).get("pollingArgs", {}) + job_id = polling_args.get("commit_job_id") + if job_id: + context_output = {"JobID": job_id, "Status": "Pending"} + continue_to_poll = True + commit_output: Any = CommandResults( + outputs=context_output, readable_output=tableToMarkdown("Commit Status:", context_output, removeNull=True) + ) + demisto.setContext("commit_job_id", job_id) + else: + commit_output = res_commit[0].get("Contents") or "There are no changes to commit." + continue_to_poll = False + + args_for_next_run = args | { + "commit_job_id": job_id, + "interval_in_seconds": arg_to_number(args.get("interval_in_seconds", 60)), + "timeout": arg_to_number(args.get("timeout", 1200)), + "polling": True, + } + return PollResult( + response=commit_output, + continue_to_poll=continue_to_poll, + args_for_next_run=args_for_next_run, + partial_result=CommandResults(readable_output=f"Waiting for commit job ID {job_id} to finish..."), + ) + + +@polling_function(name="block-domain", interval=60, timeout=1200) +def pan_os_commit_status(args: dict, responses: list) -> PollResult: + """Check the status of the commit job in pan-os. + + Args: + args (dict): The arguments of the function. + responses (list): The responses of the previous command. + Returns: + The PollResult object. + """ + commit_job_id = args["commit_job_id"] + command_name = "pan-os-commit-status" + res_commit_status = run_execute_command(command_name, {"job_id": commit_job_id}) + responses.append(res_commit_status) + # When pan-os-commit-status errors, Contents is a plain string instead of the nested dict. + # Treat as a terminal failure to avoid a `.get()` crash on a string. + raw_contents = res_commit_status[0].get("Contents", {}) if res_commit_status else {} + if not isinstance(raw_contents, dict): + commit_output = {"JobID": commit_job_id, "Status": "Failure"} + return PollResult( + response=CommandResults( + outputs=commit_output, + outputs_key_field="JobID", + readable_output=tableToMarkdown("Commit Status:", commit_output, removeNull=True), + ), + args_for_next_run=args, + continue_to_poll=False, + ) + result_commit_status = raw_contents.get("response", {}).get("result", {}).get("job", {}) + job_result = result_commit_status.get("result") + continue_to_poll = result_commit_status.get("status") != "FIN" + if continue_to_poll: + commit_output = {"JobID": commit_job_id, "Status": "Pending"} + else: + commit_output = {"JobID": commit_job_id, "Status": "Success" if job_result == "OK" else "Failure"} + return PollResult( + response=CommandResults( + outputs=commit_output, + outputs_key_field="JobID", + readable_output=tableToMarkdown("Commit Status:", commit_output, removeNull=True), + ), + args_for_next_run=args, + continue_to_poll=continue_to_poll, + ) + + +@polling_function(name="block-domain", interval=60, timeout=1200) +def pan_os_push_to_device(args: dict, responses: list) -> PollResult: + """Execute pan-os-push-to-device-group. + + Args: + args (dict): The arguments of the function. + responses (list): The responses of the previous command. + Returns: + The PollResult object. + """ + command_name = "pan-os-push-to-device-group" + res_push_to_device = run_execute_command(command_name, {"polling": True}) + responses.append(res_push_to_device) + # Surface a failed push instead of silently treating it as "no changes to push". + if is_error(res_push_to_device): + push_output = {"Status": "Failure"} + return PollResult( + response=CommandResults( + outputs=push_output, + readable_output=tableToMarkdown("Push to Device Group:", push_output, removeNull=True), + ), + args_for_next_run=args, + continue_to_poll=False, + ) + polling_args = res_push_to_device[0].get("Metadata", {}).get("pollingArgs", {}) + job_id = polling_args.get("push_job_id") + device_group = polling_args.get("device-group") + if job_id: + context_output = {"DeviceGroup": device_group, "JobID": job_id, "Status": "Pending"} + continue_to_poll = True + push_cr = CommandResults( + outputs_key_field="JobID", + outputs=context_output, + readable_output=tableToMarkdown("Push to Device Group:", context_output, removeNull=True), + ) + demisto.setContext("push_job_id", job_id) + else: + push_cr = CommandResults(readable_output=res_push_to_device[0].get("Contents") or "There are no changes to push.") + continue_to_poll = False + return PollResult( + response=push_cr, + continue_to_poll=continue_to_poll, + partial_result=CommandResults(readable_output=f"Waiting for Job-ID {job_id} to finish pushing the changes..."), + ) + + +@polling_function(name="block-domain", interval=60, timeout=1200) +def pan_os_push_status(args: dict, responses: list) -> PollResult: + """Check the status of the push job in pan-os. + + Args: + args (dict): The arguments of the function. + responses (list): The responses of the previous command. + Returns: + The PollResult object. + """ + push_job_id = args["push_job_id"] + command_name = "pan-os-push-status" + res_push_status = run_execute_command(command_name, {"job_id": push_job_id}) + responses.append(res_push_status) + # When pan-os-push-status errors, Contents is a plain string instead of the nested dict. + # Treat as a terminal failure to avoid a `.get()` crash on a string (mirrors pan_os_commit_status). + raw_contents = res_push_status[0].get("Contents", {}) if res_push_status else {} + if is_error(res_push_status) or not isinstance(raw_contents, dict): + push_output = {"JobID": push_job_id, "Status": "Failure"} + return PollResult( + response=CommandResults( + outputs=push_output, + outputs_key_field="JobID", + readable_output=tableToMarkdown("Push to Device Group:", push_output, ["JobID", "Status"], removeNull=True), + ), + args_for_next_run=args, + continue_to_poll=False, + ) + push_status = raw_contents.get("response", {}).get("result", {}).get("job", {}).get("status", "") + continue_to_poll = bool(push_status and push_status != "FIN") + context_output = {"Status": push_status, "JobID": push_job_id} + push_cr = CommandResults( + outputs_key_field="JobID", + outputs=context_output, + readable_output=tableToMarkdown("Push to Device Group:", context_output, ["JobID", "Status"], removeNull=True), + ) + return PollResult( + response=push_cr, + continue_to_poll=continue_to_poll, + partial_result=CommandResults(readable_output=f"Waiting for Job-ID {push_job_id} to finish pushing the changes..."), + ) + + +""" MAIN FUNCTION """ + + +def main() -> None: # pragma: no cover + try: + args = demisto.args() + demisto.debug(f"{LOG_TAG} block-domain invoked with {args=}") + + domain_list = argToList(args.get("domain_list", [])) + rule_name = args.get("rule_name", "Cortex - Block Domain") + log_forwarding_name = args.get("log_forwarding_name", "") + address_group = args.get("address_group", "Blocked Domains - Cortex") + tag = args.get("tag", "cortex-blocked-domains") + auto_commit = argToBoolean(args.get("auto_commit", True)) + verbose = argToBoolean(args.get("verbose", False)) + brands_to_run = argToList(args.get("brands", ",".join(SUPPORTED_BRANDS))) + + if not domain_list: + raise DemistoException("domain_list argument is required.") + + valid_domains, failed_rows = validate_domains(domain_list) + enabled_brands = get_enabled_brands() + brands_to_run = brands_to_run or list(SUPPORTED_BRANDS) + + runnable_brands = [b for b in brands_to_run if b in SUPPORTED_BRANDS and b in enabled_brands] + if not runnable_brands: + raise DemistoException( + f"No integrations were found for the brands {brands_to_run}. " + f"Please verify the brand instances' setup. Supported brands: {SUPPORTED_BRANDS}." + ) + + results: list = list(failed_rows) + command_responses: list = [] # accumulated per-command responses, used for verbose output. + + for brand in brands_to_run: + if brand not in SUPPORTED_BRANDS: + results.append( + build_result_row( + domain="", + brand=brand, + status=STATUS_FAILED, + result=RESULT_FAILED, + message=f"The brand {brand} is not supported by 'block-domain'. Supported: {SUPPORTED_BRANDS}.", + ) + ) + elif brand not in enabled_brands: + results.append( + build_result_row( + domain="", + brand=brand, + status=STATUS_FAILED, + result=RESULT_FAILED, + message=f"The brand {brand} isn't enabled.", + ) + ) + elif brand == "Panorama" and valid_domains: + pan_os = PanOs( + { + # Key MUST match the YAML arg name; polling re-invocation validates it. + "domain_list": valid_domains, + "rule_name": rule_name, + "log_forwarding_name": log_forwarding_name, + "address_group": address_group, + "tag": tag, + "auto_commit": auto_commit, + "verbose": verbose, + "commit_job_id": args.get("commit_job_id"), + "push_job_id": args.get("push_job_id"), + "polling": True, + } + ) + pan_os_result = pan_os.manage_pan_os_flow() + # A list means the run finished. Anything else (PollResult / bare CommandResults + # from a freshly-started poll) means a job is in flight. + if not isinstance(pan_os_result, list): + return_results(pan_os_result) + return + results.extend(pan_os_result) + command_responses.extend(pan_os.responses) + + return_results(build_final_command_results(results, verbose, command_responses)) + + except Exception as ex: + demisto.error(f"{LOG_TAG} block-domain failed: {traceback.format_exc()}") + return_error(f"Failed to execute block-domain. Error: {ex!s}") + + +""" ENTRY POINT """ + +if __name__ in ("__main__", "__builtin__", "builtins"): + main() diff --git a/Packs/AggregatedScripts/Scripts/BlockDomain/BlockDomain.yml b/Packs/AggregatedScripts/Scripts/BlockDomain/BlockDomain.yml new file mode 100644 index 000000000000..87e9c3981c2a --- /dev/null +++ b/Packs/AggregatedScripts/Scripts/BlockDomain/BlockDomain.yml @@ -0,0 +1,93 @@ +args: +- description: A comma-separated list of domain FQDNs to block. Wildcard entries (e.g. *.evil.com) are not supported and are skipped. + isArray: true + name: domain_list + required: true +- defaultValue: 'Cortex - Block Domain' + description: The name of the rule which will be created in the relevant products. + isArray: false + name: rule_name + required: false +- description: The Panorama log forwarding object name that specifies the log Forwarding setting to apply to the PAN-OS custom rules. + isArray: false + name: log_forwarding_name + required: false +- description: The name of the PAN-OS Panorama or Firewall address group used to hold the blocked domain FQDN objects. + isArray: false + name: address_group + required: false + defaultValue: 'Blocked Domains - Cortex' +- description: Whether to commit the new rule and push to the device group at the end of the run. + isArray: false + name: auto_commit + required: false + defaultValue: 'true' + auto: PREDEFINED + predefined: + - 'true' + - 'false' +- description: The designated tag name for the domain FQDN object. Applied to every object the script creates. + isArray: false + name: tag + required: false + defaultValue: 'cortex-blocked-domains' +- description: |- + A comma-separated list of integration brands to run the command for. If not provided, the command runs for all available integrations. + isArray: true + name: brands + required: false + auto: PREDEFINED + predefined: + - 'Panorama' +- description: Whether to retrieve a human-readable entry for every command or only the final result. True retrieves a human-readable entry for every command. False retrieves a human-readable entry only for the final result. + name: verbose + defaultValue: 'false' + auto: PREDEFINED + predefined: + - 'true' + - 'false' +- description: The commit job ID to use in polling commands. Automatically filled by polling. + name: commit_job_id + hidden: true +- description: The push job ID to use in polling commands. Automatically filled by polling. + name: push_job_id + hidden: true +comment: Blocks a list of domain FQDNs across the configured security products. +commonfields: + id: block-domain + version: -1 +enabled: false +name: block-domain +outputs: +- contextPath: BlockDomain.Domain + description: The domain FQDN that was processed. + type: String +- contextPath: BlockDomain.Brand + description: The brand (integration) used to block the domain. + type: String +- contextPath: BlockDomain.Status + description: "The lifecycle status of the action. One of 'Done', 'Pending', 'Skipped', 'Failed'." + type: String +- contextPath: BlockDomain.Result + description: "The result of the action. 'Success' or 'Failed'." + type: String +- contextPath: BlockDomain.RuleName + description: The name of the rule used for this integration. Empty if no rule was used. + type: String +- contextPath: BlockDomain.Message + description: A message concerning the result of the action. + type: String +script: '-' +system: false +timeout: 20m0s +type: python +subtype: python3 +dockerimage: demisto/python3:3.12.14.13495286 +fromversion: 6.10.0 +marketplaces: +- xsoar +- marketplacev2 +- platform +polling: true +tests: +- No tests (auto formatted) diff --git a/Packs/AggregatedScripts/Scripts/BlockDomain/BlockDomain_test.py b/Packs/AggregatedScripts/Scripts/BlockDomain/BlockDomain_test.py new file mode 100644 index 000000000000..9c201335b0fa --- /dev/null +++ b/Packs/AggregatedScripts/Scripts/BlockDomain/BlockDomain_test.py @@ -0,0 +1,1662 @@ +import json + +import pytest + +import BlockDomain +from BlockDomain import ( + OBJECT_NAME_PREFIX, + MAX_OBJECT_NAME_LENGTH, + RESULT_FAILED, + RESULT_SUCCESS, + STATUS_DONE, + STATUS_FAILED, + PanOs, + build_final_command_results, + build_verbose_human_readable, + derive_object_name, + is_valid_fqdn, + is_wildcard, + normalize_tags, + pan_os_commit, + pan_os_commit_status, + pan_os_push_status, + pan_os_push_to_device, + run_execute_command, + validate_domains, +) + + +def ok_entry(entry_context=None, contents="ok", instance=None, brand="Panorama"): + """Build a minimal successful execute_command entry. + + When ``instance`` is provided, includes a ``Metadata`` block matching what the platform + actually returns (``Metadata.instance`` / ``Metadata.brand``) so tests can assert that the + aggregate script correctly captures the serving-instance name from response entries. + """ + entry: dict = {"Type": 1, "Contents": contents, "HumanReadable": "", "EntryContext": entry_context or {}} + if instance is not None: + entry["Metadata"] = {"instance": instance, "brand": brand} + return entry + + +def err_entry(contents="error"): + """Build a minimal error execute_command entry (Type 4 == entryTypes['error']).""" + return {"Type": 4, "Contents": contents, "HumanReadable": "", "EntryContext": {}} + + +def test_run_execute_command_returns_entries(monkeypatch): + """ + Given: + - executeCommand returns a non-empty list of entries. + When: + - Calling run_execute_command. + Then: + - The entries are returned unchanged. + """ + entries = [ok_entry()] + monkeypatch.setattr(BlockDomain.demisto, "executeCommand", lambda *a, **k: entries) + + assert run_execute_command("pan-os-get-address", {"name": "x"}) == entries + + +@pytest.mark.parametrize("empty_response", [[], None]) +def test_run_execute_command_raises_on_empty_response(monkeypatch, empty_response): + """ + Given: + - executeCommand returns an empty response (empty list or None). + When: + - Calling run_execute_command. + Then: + - A DemistoException is raised instead of letting downstream res[0] indexing crash. + """ + monkeypatch.setattr(BlockDomain.demisto, "executeCommand", lambda *a, **k: empty_response) + + with pytest.raises(BlockDomain.DemistoException, match="returned an empty response"): + run_execute_command("pan-os-get-address", {"name": "x"}) + + +@pytest.mark.parametrize( + "domain, expected", + [ + ("*.evil.com", True), + ("evil.*.com", True), + ("evil.example.com", False), + ("sub.domain.co.uk", False), + ], +) +def test_is_wildcard(domain, expected): + """ + Given: + - A domain string that may or may not contain a wildcard character. + When: + - Calling is_wildcard to detect wildcard patterns. + Then: + - Returns True for domains containing '*', False otherwise. + """ + assert is_wildcard(domain) is expected + + +@pytest.mark.parametrize( + "domain, expected", + [ + ("evil.example.com", True), + ("sub.domain.co.uk", True), + ("a.b", True), + ("no-dot", False), + ("*.evil.com", False), + ("-leading.example.com", False), + ("trailing-.example.com", False), + ("space in.example.com", False), + ("", False), + ], +) +def test_is_valid_fqdn(domain, expected): + """ + Given: + - A domain string that may or may not be a syntactically valid FQDN. + When: + - Calling is_valid_fqdn to validate the domain. + Then: + - Returns True for well-formed FQDNs (has dot, valid labels, no wildcard/illegal chars), + and False for anything else including empty strings and wildcards. + """ + assert is_valid_fqdn(domain) is expected + + +def test_derive_object_name_is_deterministic(): + """ + Given: + - The same FQDN passed to derive_object_name twice. + When: + - Comparing the two returned object names. + Then: + - Both invocations return the exact same string (deterministic mapping). + """ + assert derive_object_name("evil.example.com") == derive_object_name("evil.example.com") + + +def test_derive_object_name_illegal_chars(): + """ + Given: + - A domain containing an underscore, which is not a legal PAN-OS object-name character. + When: + - Calling derive_object_name. + Then: + - Underscores are normalized to hyphens so the resulting name is accepted by PAN-OS. + """ + assert derive_object_name("bad_domain.example.com") == "Cortex-bad-domain.example.com" + + +def test_derive_object_name_overflow_truncates_and_hashes(): + """ + Given: + - A domain long enough that the naive prefixed name would exceed the PAN-OS + MAX_OBJECT_NAME_LENGTH limit. + When: + - Calling derive_object_name. + Then: + - The returned name fits within MAX_OBJECT_NAME_LENGTH, still starts with the + "Cortex-" prefix, and remains deterministic across calls. + """ + long_domain = ("a" * 80) + ".example.com" + name = derive_object_name(long_domain) + assert len(name) <= MAX_OBJECT_NAME_LENGTH + assert name.startswith(OBJECT_NAME_PREFIX) + + +def test_derive_object_name_overflow_distinct_domains_are_unique(): + """ + Given: + - Two DIFFERENT over-length domains. + When: + - Deriving each object name (both take the truncate+hash overflow path). + Then: + - The two names are different, proving the hash suffix disambiguates over-length names + (the digest is a function of the full domain, not the truncated body). + """ + name_a = derive_object_name(("a" * 80) + ".example.com") + name_b = derive_object_name(("b" * 80) + ".example.com") + assert name_a != name_b + + +def test_derive_object_name_overflow_same_prefix_different_tail_are_unique(): + """ + Given: + - Two over-length domains that share an identical long leading segment (so their + truncated bodies are the same) but differ only in a tail beyond the truncation point. + When: + - Deriving each object name. + Then: + - The names are STILL unique - the digest is computed over the full original domain, so + differences past the truncation boundary are preserved in the hash suffix. This is the + key anti-collision guarantee for the prefix-hash naming scheme. + """ + shared_head = "a" * 80 + name_a = derive_object_name(f"{shared_head}.first-tail.example.com") + name_b = derive_object_name(f"{shared_head}.second-tail.example.com") + assert name_a != name_b + # Both are truncated to the same length budget, so equal length; only the digest differs. + assert len(name_a) == len(name_b) + + +def test_validate_domains_splits_valid_and_failed(): + """ + Given: + - A mixed list of domains containing valid FQDNs, a wildcard, and an invalid FQDN. + When: + - Calling validate_domains to partition the input. + Then: + - Valid FQDNs are returned in the first list; the wildcard and invalid entries are + returned as failed rows with STATUS_FAILED / RESULT_FAILED and appropriate messages. + """ + valid, failed = validate_domains(["evil.example.com", "*.evil.com", "no-dot", "phish.attacker.net"]) + + assert valid == ["evil.example.com", "phish.attacker.net"] + assert len(failed) == 2 + + wildcard_row = next(row for row in failed if row["Domain"] == "*.evil.com") + assert wildcard_row["Status"] == STATUS_FAILED + assert wildcard_row["Result"] == RESULT_FAILED + assert "Wildcard" in wildcard_row["Message"] + # No brand/rule applies to a validation-failed row - these must be real null, not "". + assert wildcard_row["Brand"] is None + assert wildcard_row["RuleName"] is None + + invalid_row = next(row for row in failed if row["Domain"] == "no-dot") + assert invalid_row["Status"] == STATUS_FAILED + assert invalid_row["Result"] == RESULT_FAILED + assert "Invalid FQDN" in invalid_row["Message"] + + +def _pan_os(domains): + return PanOs( + { + "domain_list": domains, + "rule_name": "Cortex - Block Domain", + "address_group": "Blocked Domains - Cortex", + "tag": "cortex-blocked-domains", + "log_forwarding_name": "", + "auto_commit": True, + } + ) + + +def _mock_execute(monkeypatch, responses_by_command): + """Patch BlockDomain.demisto.executeCommand to answer per command name (order-independent). + + ``responses_by_command`` maps a command name to the entry list it should return. Probe commands + (pan-os-get-address / -list-address-groups / -list-rules) supply the tenant state; write commands + default to a bare success entry when not specified. ``pan-os-create-tag`` is always auto-answered. + + Args: + monkeypatch: The pytest monkeypatch fixture. + responses_by_command (dict): Mapping of command name -> entry list to return. + Returns: + The ``calls`` list of (command_name, args) tuples recorded during the run. + """ + calls: list = [] + + def _dispatch(command_name, args=None, *a, **k): + calls.append((command_name, args or {})) + if command_name == "pan-os-create-tag": + return [ok_entry()] + return responses_by_command.get(command_name, [ok_entry()]) + + monkeypatch.setattr(BlockDomain.demisto, "executeCommand", _dispatch) + return calls + + +def test_process_domains_create_everything(monkeypatch): + """ + Given: + - A tenant where neither the address group nor the security rule exist yet, and the + address object for the domain also does not exist. + When: + - Calling process_domains for a single domain. + Then: + - The address is created first, then the group is seeded with that object, then the + rule is created against the now-existing group, then the rule is moved to the top. + The resulting row is Done / Success / Created and carries the expected rule name. + """ + _mock_execute( + monkeypatch, + { + "pan-os-get-address": [err_entry("not found")], + "pan-os-list-address-groups": [ok_entry({"Panorama.AddressGroups": []})], + "pan-os-list-rules": [ok_entry({"Panorama.SecurityRule": []})], + }, + ) + pan = _pan_os(["evil.example.com"]) + rows = pan.process_domains() + assert len(rows) == 1 + assert rows[0]["Status"] == STATUS_DONE + assert rows[0]["Result"] == RESULT_SUCCESS + assert rows[0]["RuleName"] == "Cortex - Block Domain" + # Action is no longer surfaced; change-detection is tracked internally to gate the commit. + assert pan._address_made_changes is True + assert "Action" not in rows[0] + + +def test_ensure_tag_creates_tag_up_front(monkeypatch): + """ + Given: + - A PanOs flow configured with a tag. + When: + - Calling ensure_tag. + Then: + - pan-os-create-tag is invoked exactly once with the configured tag name, so the tag exists + before any group/rule references it (independent of whether an address object is created). + """ + calls: list = [] + + def _capture(name, args): + calls.append((name, args)) + return [ok_entry()] + + monkeypatch.setattr(BlockDomain.demisto, "executeCommand", _capture) + + _pan_os(["evil.example.com"]).ensure_tag() + + assert calls == [("pan-os-create-tag", {"name": "cortex-blocked-domains"})] + + +def test_ensure_tag_tolerates_already_exists_error(monkeypatch): + """ + Given: + - pan-os-create-tag returns an error (the tag already exists on the device). + When: + - Calling ensure_tag. + Then: + - ensure_tag does NOT raise - an already-existing tag is an expected, benign outcome. + """ + + def _err(name, args): + return [err_entry("tag already exists")] + + monkeypatch.setattr(BlockDomain.demisto, "executeCommand", _err) + + # Must not raise. + _pan_os(["evil.example.com"]).ensure_tag() + + +def test_ensure_tag_noop_when_no_tag(monkeypatch): + """ + Given: + - A PanOs flow with an empty tag. + When: + - Calling ensure_tag. + Then: + - No pan-os-create-tag call is made (nothing to create). + """ + calls: list = [] + + def _capture(name, args): + calls.append(name) + return [ok_entry()] + + monkeypatch.setattr(BlockDomain.demisto, "executeCommand", _capture) + + pan = _pan_os(["evil.example.com"]) + pan.tag = "" + pan.ensure_tag() + + assert calls == [] + + +# ---- tag helpers (pure) ------------------------------------------------ + + +@pytest.mark.parametrize( + "raw, expected", + [ + (None, []), + ("", []), + ("solo", ["solo"]), + (["a", "b"], ["a", "b"]), + (["a", "", None, "b"], ["a", "b"]), + ], +) +def test_normalize_tags(raw, expected): + """ + Given: + - A raw PAN-OS 'Tags' value that may be None, an empty string, a single string, or a list + possibly containing falsy entries. + When: + - Calling normalize_tags. + Then: + - It flattens the value into a clean list of tag-name strings, dropping falsy entries. + """ + assert normalize_tags(raw) == expected + + +@pytest.mark.parametrize( + "existing, expected", + [ + ([], ["cortex-blocked-domains"]), + (["other"], ["other", "cortex-blocked-domains"]), + (["cortex-blocked-domains"], None), + (["other", "cortex-blocked-domains"], None), + ], +) +def test_merged_tags_if_missing(existing, expected): + """ + Given: + - An entity's current tag list and a PanOs flow configured with 'cortex-blocked-domains'. + When: + - Calling merged_tags_if_missing. + Then: + - Returns the de-duplicated union (existing + configured tag) when the tag is absent, and + None (no edit needed) when the tag is already present. The existing tag is never duplicated. + """ + assert _pan_os(["evil.example.com"]).merged_tags_if_missing(existing) == expected + + +def test_start_flow_skips_commit_when_all_actions_unchanged(monkeypatch): + """ + Given: + - A tenant already fully configured for the requested domain (group exists, rule exists, + address object already a member) so every row is Unchanged. + When: + - Calling start_flow. + Then: + - pan_os_commit is never invoked; start_flow returns the Unchanged rows directly + (skipping the multi-minute commit + push polling cycle on Panorama). + """ + calls: list = [] + + def _capture(name, args): + calls.append((name, args)) + seq = { + "pan-os-list-address-groups": [ + ok_entry( + { + "Panorama.AddressGroups": [ + { + "Name": "Blocked Domains - Cortex", + "Type": "static", + "Addresses": ["Cortex-evil.example.com"], + "Tags": ["cortex-blocked-domains"], + } + ] + } + ) + ], + "pan-os-list-rules": [ + ok_entry( + { + "Panorama.SecurityRule": [ + { + "Name": "Cortex - Block Domain", + "Destination": ["Blocked Domains - Cortex"], + "Tags": ["cortex-blocked-domains"], + } + ] + } + ) + ], + "pan-os-get-address": [ + ok_entry({"Panorama.Addresses": {"Name": "Cortex-evil.example.com", "Tags": ["cortex-blocked-domains"]}}) + ], + "pan-os-move-rule": [ok_entry()], + } + return seq.get(name, [ok_entry()]) + + monkeypatch.setattr(BlockDomain.demisto, "executeCommand", _capture) + monkeypatch.setattr( + BlockDomain, "pan_os_commit", lambda *a, **k: pytest.fail("pan_os_commit must not be called when all rows are Unchanged") + ) + monkeypatch.setattr(BlockDomain.demisto, "setContext", lambda *a, **k: None) + + result = _pan_os(["evil.example.com"]).start_flow() + + assert isinstance(result, list) + assert len(result) == 1 + assert "Action" not in result[0] + assert "pan-os-commit" not in [c[0] for c in calls] + + +def test_start_flow_commits_when_at_least_one_row_modified(monkeypatch): + """ + Given: + - A tenant where the group and rule already exist but the address object for the + requested domain is missing, so at least one row will end up Created. + When: + - Calling start_flow. + Then: + - pan_os_commit is invoked (candidate config was modified and must be pushed). + """ + commit_called: list = [] + + def _fake_commit(args, responses): + commit_called.append(True) + return BlockDomain.CommandResults(readable_output="fake commit ok") + + def _capture(name, args): + seq = { + "pan-os-list-address-groups": [ + ok_entry({"Panorama.AddressGroups": [{"Name": "Blocked Domains - Cortex", "Type": "static", "Addresses": []}]}) + ], + "pan-os-list-rules": [ + ok_entry( + {"Panorama.SecurityRule": [{"Name": "Cortex - Block Domain", "Destination": ["Blocked Domains - Cortex"]}]} + ) + ], + "pan-os-get-address": [err_entry("not found")], + "pan-os-create-address": [ok_entry()], + "pan-os-edit-address-group": [ok_entry()], + "pan-os-move-rule": [ok_entry()], + } + return seq.get(name, [ok_entry()]) + + monkeypatch.setattr(BlockDomain.demisto, "executeCommand", _capture) + monkeypatch.setattr(BlockDomain.demisto, "setContext", lambda *a, **k: None) + monkeypatch.setattr(BlockDomain, "pan_os_commit", _fake_commit) + monkeypatch.setattr(BlockDomain.demisto, "context", lambda: {"block_domain_rows": "[]"}) + + _pan_os(["evil.example.com"]).start_flow() + + assert commit_called, "pan_os_commit must be called when at least one row is Created/Modified" + + +def test_process_domains_rule_created_when_object_unchanged_sets_rule_changed(monkeypatch): + """ + Given: + - The address object and group already exist and the object is already a member (so the + per-domain Action is Unchanged), but the requested rule_name does NOT exist yet, so a + new rule must be created. + When: + - Calling process_domains. + Then: + - The row Action is Unchanged (object/membership did not change), but the instance's + _rule_changed flag is set True because pan-os-create-rule ran. This is what lets + start_flow still commit a rule-only change. + """ + obj = "Cortex-evil.example.com" + + def _capture(name, args): + seq = { + # Group exists, already contains the object, and already carries the tag -> Unchanged. + "pan-os-list-address-groups": [ + ok_entry( + { + "Panorama.AddressGroups": [ + { + "Name": "Blocked Domains - Cortex", + "Type": "static", + "Addresses": [obj], + "Tags": ["cortex-blocked-domains"], + } + ] + } + ) + ], + # No rule with the requested name exists -> ensure_rule will create it. + "pan-os-list-rules": [ok_entry({"Panorama.SecurityRule": []})], + # Object exists and already carries the tag -> no object tag edit. + "pan-os-get-address": [ok_entry({"Panorama.Addresses": [{"Name": obj, "Tags": ["cortex-blocked-domains"]}]})], + "pan-os-create-rule": [ok_entry()], + "pan-os-move-rule": [ok_entry()], + } + return seq.get(name, [ok_entry()]) + + monkeypatch.setattr(BlockDomain.demisto, "executeCommand", _capture) + + pan_os = _pan_os(["evil.example.com"]) + rows = pan_os.process_domains() + + # Object + membership did not change (so _made_changes stays False), but the rule was created, + # so _rule_changed is True - that alone must still trigger a commit in start_flow. + assert "Action" not in rows[0] + assert pan_os._address_made_changes is False + assert pan_os._rule_changed is True + + +def test_start_flow_commits_when_only_the_rule_changed(monkeypatch): + """ + Given: + - Every address object is already present and a member (all rows Unchanged), but a new + rule had to be created this run (rule-only change). + When: + - Calling start_flow. + Then: + - pan_os_commit is still invoked, so the newly created rule is actually committed/pushed + instead of silently sitting in the candidate config. + """ + obj = "Cortex-evil.example.com" + commit_called: list = [] + + def _fake_commit(args, responses): + commit_called.append(True) + return BlockDomain.CommandResults(readable_output="fake commit ok") + + def _capture(name, args): + seq = { + "pan-os-list-address-groups": [ + ok_entry({"Panorama.AddressGroups": [{"Name": "Blocked Domains - Cortex", "Type": "static", "Addresses": [obj]}]}) + ], + "pan-os-list-rules": [ok_entry({"Panorama.SecurityRule": []})], + "pan-os-get-address": [ok_entry({"Panorama.Addresses": [{"Name": obj}]})], + "pan-os-create-rule": [ok_entry()], + "pan-os-move-rule": [ok_entry()], + } + return seq.get(name, [ok_entry()]) + + monkeypatch.setattr(BlockDomain.demisto, "executeCommand", _capture) + monkeypatch.setattr(BlockDomain.demisto, "setContext", lambda *a, **k: None) + monkeypatch.setattr(BlockDomain, "pan_os_commit", _fake_commit) + monkeypatch.setattr(BlockDomain.demisto, "context", lambda: {"block_domain_rows": "[]"}) + + _pan_os(["evil.example.com"]).start_flow() + + assert commit_called, "pan_os_commit must be called when only the rule changed (all objects Unchanged)" + + +def test_process_domains_missing_group_created_lazily_with_first_object(monkeypatch): + """ + Given: + - A tenant with neither group nor rule pre-configured, and no address object for the + requested domain. PAN-OS refuses to create a static group with no members, and + refuses to create a rule whose destination does not resolve to an existing object. + When: + - Calling process_domains for a single domain. + Then: + - The address is created first; then the group is created lazily seeded with that + first object (never as an empty static group); then the rule is created against the + now-existing group. No edit-address-group call is emitted for the seed domain. + """ + calls: list = [] + + def _capture(name, args): + calls.append((name, args)) + seq = { + "pan-os-list-address-groups": [ok_entry({"Panorama.AddressGroups": []})], + "pan-os-list-rules": [ok_entry({"Panorama.SecurityRule": []})], + "pan-os-create-rule": [ok_entry()], + "pan-os-move-rule": [ok_entry()], + "pan-os-get-address": [err_entry("not found")], + "pan-os-create-address": [ok_entry()], + "pan-os-create-address-group": [ok_entry()], + } + return seq.get(name, [ok_entry()]) + + monkeypatch.setattr(BlockDomain.demisto, "executeCommand", _capture) + + _pan_os(["evil.example.com"]).process_domains() + + names = [c[0] for c in calls] + assert names.index("pan-os-create-address") < names.index("pan-os-create-address-group") + assert names.index("pan-os-create-address-group") < names.index("pan-os-create-rule") + group_create_args = next(args for name, args in calls if name == "pan-os-create-address-group") + assert group_create_args["type"] == "static" + # The group is created once with the full member list (all objects created first). + assert group_create_args["addresses"] == ["Cortex-evil.example.com"] + rule_create_args = next(args for name, args in calls if name == "pan-os-create-rule") + assert rule_create_args["destination"] == "Blocked Domains - Cortex" + assert "pan-os-edit-address-group" not in names + + +def test_process_domains_multi_domain_creates_all_objects_before_group(monkeypatch): + """ + Given: + - A tenant with no group, no rule, and neither address object existing, for TWO domains. + When: + - Calling process_domains for both domains. + Then: + - Both address objects are created BEFORE the single group-create, and the group is created + once with the full member list (this is the ordering that avoids the reference error). + """ + calls = _mock_execute( + monkeypatch, + { + "pan-os-get-address": [err_entry("not found")], + "pan-os-list-address-groups": [ok_entry({"Panorama.AddressGroups": []})], + "pan-os-list-rules": [ok_entry({"Panorama.SecurityRule": []})], + }, + ) + _pan_os(["evil.example.com", "phish.attacker.net"]).process_domains() + + names = [name for name, _ in calls] + create_addresses = [i for i, name in enumerate(names) if name == "pan-os-create-address"] + group_create_index = names.index("pan-os-create-address-group") + # Both objects are created before the single group-create. + assert len(create_addresses) == 2 + assert max(create_addresses) < group_create_index + assert names.count("pan-os-create-address-group") == 1 + group_args = next(args for name, args in calls if name == "pan-os-create-address-group") + assert group_args["addresses"] == ["Cortex-evil.example.com", "Cortex-phish.attacker.net"] + + +def test_process_domains_all_unchanged(monkeypatch): + """ + Given: + - A tenant where the group, rule, and address object all already exist and the + address object is already a member of the group. + When: + - Calling process_domains. + Then: + - The resulting row is Done / Unchanged (no create/modify happened). + """ + obj = "Cortex-evil.example.com" + calls = _mock_execute( + monkeypatch, + { + "pan-os-get-address": [ok_entry({"Panorama.Addresses": [{"Name": obj, "Tags": ["cortex-blocked-domains"]}]})], + "pan-os-list-address-groups": [ + ok_entry( + { + "Panorama.AddressGroups": [ + { + "Name": "Blocked Domains - Cortex", + "Type": "static", + "Addresses": [obj], + "Tags": ["cortex-blocked-domains"], + } + ] + } + ) + ], + "pan-os-list-rules": [ + ok_entry( + { + "Panorama.SecurityRule": [ + { + "Name": "Cortex - Block Domain", + "Destination": ["Blocked Domains - Cortex"], + "Tags": ["cortex-blocked-domains"], + } + ] + } + ) + ], + }, + ) + pan = _pan_os(["evil.example.com"]) + rows = pan.process_domains() + assert rows[0]["Status"] == STATUS_DONE + # Everything already in place (incl. the tag) -> no change -> commit will be skipped. + assert pan._address_made_changes is False + assert pan._rule_changed is False + assert "Action" not in rows[0] + # A pre-existing, unchanged rule must NOT be moved (no needless candidate-config churn). + assert not any(name == "pan-os-move-rule" for name, _ in calls) + + +def test_process_domains_modified_when_added_to_existing_group(monkeypatch): + """ + Given: + - A tenant where the group and rule already exist but the group has no members yet + and the address object for the requested domain does not exist. + When: + - Calling process_domains. + Then: + - The address is created and added to the existing group; the resulting row is + Done / Created (most-significant action of the create-address step). + """ + _mock_execute( + monkeypatch, + { + "pan-os-get-address": [err_entry("not found")], + "pan-os-list-address-groups": [ + ok_entry( + { + "Panorama.AddressGroups": [ + { + "Name": "Blocked Domains - Cortex", + "Type": "static", + "Addresses": [], + "Tags": ["cortex-blocked-domains"], + } + ] + } + ) + ], + "pan-os-list-rules": [ + ok_entry( + { + "Panorama.SecurityRule": [ + { + "Name": "Cortex - Block Domain", + "Destination": ["Blocked Domains - Cortex"], + "Tags": ["cortex-blocked-domains"], + } + ] + } + ) + ], + }, + ) + pan = _pan_os(["evil.example.com"]) + rows = pan.process_domains() + assert rows[0]["Status"] == STATUS_DONE + # A new object was created and added to the group -> a change occurred -> commit will run. + assert pan._address_made_changes is True + assert "Action" not in rows[0] + + +def test_process_domains_existing_rule_missing_group_is_edited(monkeypatch): + """ + Given: + - A tenant where the rule exists but its destination does not yet reference our + address group; the group and the address object already exist. + When: + - Calling process_domains. + Then: + - The rule is edited to include the group in its destination; the resulting row is + Done / Unchanged because the address object and its group membership were unchanged + (rule-level fix is a group-scope, not per-domain, mutation). + """ + obj = "Cortex-evil.example.com" + _mock_execute( + monkeypatch, + { + "pan-os-get-address": [ok_entry({"Panorama.Addresses": [{"Name": obj, "Tags": ["cortex-blocked-domains"]}]})], + "pan-os-list-address-groups": [ + ok_entry( + { + "Panorama.AddressGroups": [ + { + "Name": "Blocked Domains - Cortex", + "Type": "static", + "Addresses": [obj], + "Tags": ["cortex-blocked-domains"], + } + ] + } + ) + ], + "pan-os-list-rules": [ + ok_entry( + { + "Panorama.SecurityRule": [ + { + "Name": "Cortex - Block Domain", + "Destination": ["something-else"], + "Tags": ["cortex-blocked-domains"], + } + ] + } + ) + ], + }, + ) + pan = _pan_os(["evil.example.com"]) + rows = pan.process_domains() + assert rows[0]["Status"] == STATUS_DONE + # Object + membership unchanged, but the rule was edited to add the group -> _rule_changed. + assert pan._address_made_changes is False + assert pan._rule_changed is True + assert "Action" not in rows[0] + + +def _tag_capture(monkeypatch): + """Patch executeCommand with a dict-based capture that tolerates call ordering. + + Returns the ``calls`` list of (command_name, args) tuples. Read/probe commands return + fixtures that model an already-existing object/group/rule WITHOUT the configured tag, so the + merge/edit paths are exercised. Every other command returns a bare success entry. + """ + obj = "Cortex-evil.example.com" + calls: list = [] + fixtures = { + "pan-os-list-address-groups": [ + ok_entry( + { + "Panorama.AddressGroups": [ + {"Name": "Blocked Domains - Cortex", "Type": "static", "Addresses": [obj], "Tags": ["other"]} + ] + } + ) + ], + "pan-os-list-rules": [ + ok_entry( + { + "Panorama.SecurityRule": [ + {"Name": "Cortex - Block Domain", "Destination": ["Blocked Domains - Cortex"], "Tags": ["other"]} + ] + } + ) + ], + "pan-os-get-address": [ok_entry({"Panorama.Addresses": [{"Name": obj, "Tags": ["other"]}]})], + } + + def _capture(name, args): + calls.append((name, args)) + return fixtures.get(name, [ok_entry()]) + + monkeypatch.setattr(BlockDomain.demisto, "executeCommand", _capture) + return calls + + +def test_process_domains_merges_tag_on_existing_object_group_and_rule(monkeypatch): + """ + Given: + - A tenant where the address object, address group, and rule all already exist, each + carrying a different pre-existing tag ('other') but NOT the configured tag. + When: + - Calling process_domains with tag='cortex-blocked-domains'. + Then: + - The configured tag is merged (never replacing) onto all three entities: + * pan-os-edit-address with element_to_change=tag and the merged list ['other', tag] + * pan-os-edit-address-group with tags=['other', tag] + * pan-os-edit-rule with element_to_change=tag, behaviour=add, element_value=tag + """ + calls = _tag_capture(monkeypatch) + pan = _pan_os(["evil.example.com"]) + rows = pan.process_domains() + + assert rows[0]["Status"] == STATUS_DONE + + edit_address = [args for name, args in calls if name == "pan-os-edit-address"] + assert edit_address == [ + { + "name": "Cortex-evil.example.com", + "element_to_change": "tag", + "element_value": ["other", "cortex-blocked-domains"], + } + ] + + # The group edit carries element_to_add (required by the static-group guard) together with the + # merged tags in one call. The object is already a member, so the full list is re-added (no-op). + edit_group = [args for name, args in calls if name == "pan-os-edit-address-group" and "tags" in args] + assert edit_group == [ + { + "name": "Blocked Domains - Cortex", + "type": "static", + "element_to_add": ["Cortex-evil.example.com"], + "tags": ["other", "cortex-blocked-domains"], + } + ] + + edit_rule_tag = [args for name, args in calls if name == "pan-os-edit-rule" and args.get("element_to_change") == "tag"] + assert edit_rule_tag == [ + { + "rulename": "Cortex - Block Domain", + "element_to_change": "tag", + "element_value": "cortex-blocked-domains", + "behaviour": "add", + "pre_post": "pre-rulebase", + } + ] + + +def test_process_domains_does_not_re_tag_when_tag_already_present(monkeypatch): + """ + Given: + - The demo scenario: an existing FQDN, an existing group, and an existing rule that ALL + already carry the configured tag; the user re-runs the command. + When: + - Calling process_domains with the same tag. + Then: + - No tag edit is issued on the object, group, or rule (the tag is never duplicated), and + nothing changes so the commit is skipped. + """ + obj = "Cortex-evil.example.com" + tag = "cortex-blocked-domains" + calls: list = [] + fixtures = { + "pan-os-list-address-groups": [ + ok_entry( + { + "Panorama.AddressGroups": [ + {"Name": "Blocked Domains - Cortex", "Type": "static", "Addresses": [obj], "Tags": [tag]} + ] + } + ) + ], + "pan-os-list-rules": [ + ok_entry( + { + "Panorama.SecurityRule": [ + {"Name": "Cortex - Block Domain", "Destination": ["Blocked Domains - Cortex"], "Tags": [tag]} + ] + } + ) + ], + "pan-os-get-address": [ok_entry({"Panorama.Addresses": [{"Name": obj, "Tags": [tag]}]})], + } + + def _capture(name, args): + calls.append((name, args)) + return fixtures.get(name, [ok_entry()]) + + monkeypatch.setattr(BlockDomain.demisto, "executeCommand", _capture) + + pan = _pan_os(["evil.example.com"]) + pan.process_domains() + + assert [args for name, args in calls if name == "pan-os-edit-address"] == [] + assert [args for name, args in calls if name == "pan-os-edit-address-group" and "tags" in args] == [] + assert [args for name, args in calls if name == "pan-os-edit-rule" and args.get("element_to_change") == "tag"] == [] + assert pan._address_made_changes is False + assert pan._rule_changed is False + + +def test_process_domains_new_rule_name_does_not_re_tag_existing_object_and_group(monkeypatch): + """ + Given: + - An existing FQDN and existing group that ALREADY carry the configured tag, but a brand-new + rule name that does not exist yet (the "just add a rule" case). + When: + - Calling process_domains. + Then: + - The object and group are NOT re-tagged (no duplication); only the new rule is created + (which carries the tag at creation time via its own create args). + """ + obj = "Cortex-evil.example.com" + tag = "cortex-blocked-domains" + calls: list = [] + fixtures = { + "pan-os-list-address-groups": [ + ok_entry( + { + "Panorama.AddressGroups": [ + {"Name": "Blocked Domains - Cortex", "Type": "static", "Addresses": [obj], "Tags": [tag]} + ] + } + ) + ], + # Rule does not exist yet -> it will be created (with the tag in its create args). + "pan-os-list-rules": [ok_entry({"Panorama.SecurityRule": []})], + "pan-os-get-address": [ok_entry({"Panorama.Addresses": [{"Name": obj, "Tags": [tag]}]})], + } + + def _capture(name, args): + calls.append((name, args)) + return fixtures.get(name, [ok_entry()]) + + monkeypatch.setattr(BlockDomain.demisto, "executeCommand", _capture) + + pan = _pan_os(["evil.example.com"]) + pan.process_domains() + + # Object and group already carry the tag -> no re-tag edits. + assert [args for name, args in calls if name == "pan-os-edit-address"] == [] + assert [args for name, args in calls if name == "pan-os-edit-address-group" and "tags" in args] == [] + # The new rule is created and carries the tag at creation. + create_rule = [args for name, args in calls if name == "pan-os-create-rule"] + assert len(create_rule) == 1 + assert create_rule[0]["tags"] == tag + assert pan._rule_changed is True + + +def _pan_os_with_log_forwarding(domains, profile): + """Build a PanOs flow configured with a log-forwarding profile name (rule-only).""" + pan = _pan_os(domains) + pan.log_forwarding_name = profile + return pan + + +def _lf_capture(monkeypatch, existing_profile): + """Patch executeCommand modelling an existing object/group/rule (everything already in place). + + The existing rule carries ``existing_profile`` as its LogForwardingProfile. Returns the + ``calls`` list of (command_name, args) tuples. Tags are pre-present so no tag edits fire. + """ + obj = "Cortex-evil.example.com" + tag = "cortex-blocked-domains" + calls: list = [] + fixtures = { + "pan-os-list-address-groups": [ + ok_entry( + { + "Panorama.AddressGroups": [ + {"Name": "Blocked Domains - Cortex", "Type": "static", "Addresses": [obj], "Tags": [tag]} + ] + } + ) + ], + "pan-os-list-rules": [ + ok_entry( + { + "Panorama.SecurityRule": [ + { + "Name": "Cortex - Block Domain", + "Destination": ["Blocked Domains - Cortex"], + "Tags": [tag], + "LogForwardingProfile": existing_profile, + } + ] + } + ) + ], + "pan-os-get-address": [ok_entry({"Panorama.Addresses": [{"Name": obj, "Tags": [tag]}]})], + } + + def _capture(name, args): + calls.append((name, args)) + return fixtures.get(name, [ok_entry()]) + + monkeypatch.setattr(BlockDomain.demisto, "executeCommand", _capture) + return calls + + +def test_process_domains_sets_log_forwarding_on_existing_rule_when_absent(monkeypatch): + """ + Given: + - An existing rule with NO log-forwarding profile, and a flow configured with one. + When: + - Calling process_domains. + Then: + - pan-os-edit-rule is issued with element_to_change=log-forwarding and the configured + profile, and _rule_changed is set. + """ + calls = _lf_capture(monkeypatch, existing_profile="") + pan = _pan_os_with_log_forwarding(["evil.example.com"], "cortex-lfp") + pan.process_domains() + + edit_lf = [args for name, args in calls if name == "pan-os-edit-rule" and args.get("element_to_change") == "log-forwarding"] + assert edit_lf == [ + { + "rulename": "Cortex - Block Domain", + "element_to_change": "log-forwarding", + "element_value": "cortex-lfp", + "pre_post": "pre-rulebase", + } + ] + assert pan._rule_changed is True + + +def test_process_domains_sets_log_forwarding_on_existing_rule_when_different(monkeypatch): + """ + Given: + - An existing rule whose log-forwarding profile differs from the configured one. + When: + - Calling process_domains. + Then: + - pan-os-edit-rule replaces it with the configured profile (single-value, no dedup concept). + """ + calls = _lf_capture(monkeypatch, existing_profile="old-lfp") + pan = _pan_os_with_log_forwarding(["evil.example.com"], "cortex-lfp") + pan.process_domains() + + edit_lf = [args for name, args in calls if name == "pan-os-edit-rule" and args.get("element_to_change") == "log-forwarding"] + assert len(edit_lf) == 1 + assert edit_lf[0]["element_value"] == "cortex-lfp" + assert pan._rule_changed is True + + +def test_process_domains_does_not_set_log_forwarding_when_already_matches(monkeypatch): + """ + Given: + - An existing rule that ALREADY carries the configured log-forwarding profile. + When: + - Calling process_domains. + Then: + - No log-forwarding edit is issued (no unnecessary edit), and since nothing else changed + the rule is left untouched. + """ + calls = _lf_capture(monkeypatch, existing_profile="cortex-lfp") + pan = _pan_os_with_log_forwarding(["evil.example.com"], "cortex-lfp") + pan.process_domains() + + edit_lf = [args for name, args in calls if name == "pan-os-edit-rule" and args.get("element_to_change") == "log-forwarding"] + assert edit_lf == [] + assert pan._rule_changed is False + + +def test_process_domains_dynamic_group_is_failed(monkeypatch): + """ + Given: + - A tenant where the target group already exists but as a *dynamic* address group + (which cannot accept manually added static members). + When: + - Calling process_domains. + Then: + - The domain could not be blocked, so the row is marked Failed / Failed and the message + explains the group is dynamic. + """ + _mock_execute( + monkeypatch, + { + "pan-os-get-address": [err_entry("not found")], + "pan-os-list-address-groups": [ + ok_entry({"Panorama.AddressGroups": [{"Name": "Blocked Domains - Cortex", "Type": "dynamic", "Match": "x"}]}) + ], + }, + ) + # Capture the abort log so it does not leak to stdout (conftest fails on any stdout). + errors: list = [] + monkeypatch.setattr(BlockDomain.demisto, "error", lambda msg: errors.append(msg)) + + rows = _pan_os(["evil.example.com"]).process_domains() + assert rows[0]["Status"] == STATUS_FAILED + assert rows[0]["Result"] == RESULT_FAILED + assert "dynamic" in rows[0]["Message"] + # The aborted domains are logged for troubleshooting. + assert any("evil.example.com" in msg for msg in errors) + + +def test_process_domains_moves_rule_to_top_when_rule_created(monkeypatch): + """ + Given: + - A tenant where neither the rule nor the address object exist yet, so the rule is created. + When: + - Calling process_domains. + Then: + - pan-os-move-rule is issued to enforce top placement for the newly created rule. + """ + calls = _mock_execute( + monkeypatch, + { + "pan-os-get-address": [err_entry("not found")], + "pan-os-list-address-groups": [ + ok_entry({"Panorama.AddressGroups": [{"Name": "Blocked Domains - Cortex", "Type": "static", "Addresses": []}]}) + ], + "pan-os-list-rules": [ok_entry({"Panorama.SecurityRule": []})], + }, + ) + pan = _pan_os(["evil.example.com"]) + pan.process_domains() + assert pan._rule_changed is True + assert any(name == "pan-os-move-rule" for name, _ in calls) + + +def test_process_domains_failure_marks_row_failed(monkeypatch): + """ + Given: + - A tenant where the group and rule do not exist yet, and the pan-os-create-address + command fails (e.g. permission denied) before the group and rule are touched. + When: + - Calling process_domains. + Then: + - The resulting row is Failed / Failed and the error message from PAN-OS is surfaced, + and the full traceback is logged via demisto.error. + """ + _mock_execute( + monkeypatch, + { + "pan-os-get-address": [err_entry("not found")], + "pan-os-create-address": [err_entry("permission denied")], + "pan-os-list-address-groups": [ok_entry({"Panorama.AddressGroups": []})], + "pan-os-list-rules": [ok_entry({"Panorama.SecurityRule": []})], + }, + ) + # Capture the traceback log so it does not leak to stdout (conftest fails on any stdout). + errors: list = [] + monkeypatch.setattr(BlockDomain.demisto, "error", lambda msg: errors.append(msg)) + + rows = _pan_os(["evil.example.com"]).process_domains() + assert rows[0]["Status"] == STATUS_FAILED + assert rows[0]["Result"] == RESULT_FAILED + assert "permission denied" in rows[0]["Message"] + assert any("process_domains failed" in msg for msg in errors) + + +def test_build_verbose_human_readable_joins_with_blank_lines(): + """ + Given: + - A list of response entries where some carry a HumanReadable string and some do not. + When: + - Calling build_verbose_human_readable. + Then: + - Entries without HumanReadable are skipped; the rest are joined with a blank line + separator, prefixed by a leading blank line so the block detaches from the summary + table above it. + """ + responses = [ + [ok_entry(contents="c1")], + [{"Type": 1, "Contents": "c2", "HumanReadable": "HR-two", "EntryContext": {}}], + [{"Type": 1, "Contents": "c3", "HumanReadable": "HR-three", "EntryContext": {}}], + ] + verbose_hr = build_verbose_human_readable(responses) + assert verbose_hr == "\n\nHR-two\n\nHR-three" + + +def test_build_verbose_human_readable_empty_when_no_hr(): + """ + Given: + - A response list where no entry has a HumanReadable string, or an empty list. + When: + - Calling build_verbose_human_readable. + Then: + - Returns an empty string (nothing to append to the summary table). + """ + assert build_verbose_human_readable([[ok_entry(contents="c1")]]) == "" + assert build_verbose_human_readable([]) == "" + + +def test_build_final_command_results_non_verbose_is_table_only(): + """ + Given: + - A rows list and a set of responses that include HumanReadable content, with + verbose=False. + When: + - Calling build_final_command_results. + Then: + - The returned CommandResults uses the BlockDomain context prefix, exposes the + rows unchanged as outputs, renders the summary table containing the domain, and does + NOT append any of the per-command verbose HR blocks. + """ + rows = [ + { + "Domain": "a.com", + "Brand": "Panorama", + "Status": STATUS_DONE, + "Result": RESULT_SUCCESS, + "RuleName": "Cortex - Block Domain", + "Message": "ok", + } + ] + responses = [[{"Type": 1, "Contents": "c", "HumanReadable": "HR", "EntryContext": {}}]] + + result = build_final_command_results(rows, verbose=False, responses=responses) + assert result.outputs_prefix == "BlockDomain" + assert result.outputs == rows + assert "a.com" in result.readable_output + assert "HR" not in result.readable_output + + +def test_build_final_command_results_verbose_appends_command_hr(): + """ + Given: + - A rows list and responses with a HumanReadable block, with verbose=True. + When: + - Calling build_final_command_results. + Then: + - The returned CommandResults exposes the rows as outputs, renders the summary table + containing the domain, and appends the per-command HR block at the end of the + readable_output (so users can see exactly what each downstream call produced). + """ + rows = [ + { + "Domain": "a.com", + "Brand": "Panorama", + "Status": STATUS_DONE, + "Result": RESULT_SUCCESS, + "RuleName": "Cortex - Block Domain", + "Message": "ok", + } + ] + responses = [[{"Type": 1, "Contents": "c", "HumanReadable": "HR-one", "EntryContext": {}}]] + + result = build_final_command_results(rows, verbose=True, responses=responses) + assert result.outputs == rows + assert "a.com" in result.readable_output + assert result.readable_output.endswith("HR-one") + + +def test_build_final_command_results_all_failed_is_error_entry(): + """ + Given: + - A rows list where every row has Result == Failed. + When: + - Calling build_final_command_results. + Then: + - The returned CommandResults is marked as an ERROR entry, still exposes the rows as + outputs, and its readable_output carries the "All runs failed" header so the war room + and any downstream playbook can branch on the failure. + """ + rows = [ + { + "Domain": "a.com", + "Brand": "Panorama", + "Status": STATUS_FAILED, + "Result": RESULT_FAILED, + "RuleName": "", + "Message": "boom", + }, + { + "Domain": "b.com", + "Brand": "Panorama", + "Status": STATUS_FAILED, + "Result": RESULT_FAILED, + "RuleName": "", + "Message": "boom", + }, + ] + + result = build_final_command_results(rows, verbose=False, responses=[]) + assert result.entry_type == BlockDomain.EntryType.ERROR + assert result.outputs == rows + assert "All runs failed" in result.readable_output + + +def test_build_final_command_results_partial_failure_is_not_error_entry(): + """ + Given: + - A rows list with at least one successful row alongside a failed row. + When: + - Calling build_final_command_results. + Then: + - The returned CommandResults is NOT an ERROR entry (entry_type is unset / not ERROR), + so a partial success is still reported as a normal result. + """ + rows = [ + { + "Domain": "a.com", + "Brand": "Panorama", + "Status": STATUS_DONE, + "Result": RESULT_SUCCESS, + "RuleName": "Cortex - Block Domain", + "Message": "ok", + }, + { + "Domain": "b.com", + "Brand": "Panorama", + "Status": STATUS_FAILED, + "Result": RESULT_FAILED, + "RuleName": "", + "Message": "boom", + }, + ] + + result = build_final_command_results(rows, verbose=False, responses=[]) + assert result.entry_type != BlockDomain.EntryType.ERROR + assert "All runs failed" not in result.readable_output + + +def test_build_final_command_results_empty_rows_is_not_error_entry(): + """ + Given: + - An empty rows list (e.g. no domains reached a vendor). + When: + - Calling build_final_command_results. + Then: + - The run is NOT reported as an error (an empty run is not an all-failed run). + """ + result = build_final_command_results([], verbose=False, responses=[]) + assert result.entry_type != BlockDomain.EntryType.ERROR + assert "All runs failed" not in result.readable_output + + +def test_pan_os_commit_status_reports_pending_while_job_active(monkeypatch): + """ + Given: + - pan-os-commit-status returns a job with result 'OK' but status 'ACT' (still running). + When: + - Calling pan_os_commit_status. + Then: + - The war-room output must NOT show 'Success' while the job is still active; it reports + 'Pending' (job.result 'OK' and job.status 'ACT' are independent PAN-OS fields, and success + is only reported once the job status is 'FIN'). + """ + active_entry = { + "Type": 1, + "Contents": {"response": {"result": {"job": {"result": "OK", "status": "ACT"}}}}, + "HumanReadable": "", + "EntryContext": {}, + } + monkeypatch.setattr(BlockDomain.demisto, "executeCommand", lambda *a, **k: [active_entry]) + + result = pan_os_commit_status({"commit_job_id": "123"}, []) + + assert result.outputs == {"JobID": "123", "Status": "Pending"} # type: ignore[attr-defined] + + +def test_pan_os_commit_status_reports_success_only_when_finished(monkeypatch): + """ + Given: + - pan-os-commit-status returns a finished job (status 'FIN') with result 'OK'. + When: + - Calling pan_os_commit_status. + Then: + - The output reports 'Success' and polling stops. + """ + fin_entry = { + "Type": 1, + "Contents": {"response": {"result": {"job": {"result": "OK", "status": "FIN"}}}}, + "HumanReadable": "", + "EntryContext": {}, + } + monkeypatch.setattr(BlockDomain.demisto, "executeCommand", lambda *a, **k: [fin_entry]) + + result = pan_os_commit_status({"commit_job_id": "123"}, []) + + assert result.outputs == {"JobID": "123", "Status": "Success"} # type: ignore[attr-defined] + assert not BlockDomain.is_polling_in_progress(result) + + +def test_pan_os_commit_reports_failure_on_error(monkeypatch): + """ + Given: + - pan-os-commit returns an error entry. + When: + - Calling pan_os_commit. + Then: + - The failure is surfaced (Status 'Failure') and polling stops, instead of being silently + treated as 'no changes to commit'. + """ + monkeypatch.setattr(BlockDomain.demisto, "executeCommand", lambda *a, **k: [err_entry("commit failed")]) + monkeypatch.setattr(BlockDomain.demisto, "setContext", lambda *a, **k: None) + + result = pan_os_commit({}, []) + + assert result.outputs == {"Status": "Failure"} # type: ignore[attr-defined] + assert not BlockDomain.is_polling_in_progress(result) + + +def test_pan_os_push_to_device_reports_failure_on_error(monkeypatch): + """ + Given: + - pan-os-push-to-device-group returns an error entry. + When: + - Calling pan_os_push_to_device. + Then: + - The failure is surfaced (Status 'Failure') and polling stops, instead of being silently + treated as 'no changes to push'. + """ + monkeypatch.setattr(BlockDomain.demisto, "executeCommand", lambda *a, **k: [err_entry("push failed")]) + monkeypatch.setattr(BlockDomain.demisto, "setContext", lambda *a, **k: None) + + result = pan_os_push_to_device({}, []) + + assert result.outputs == {"Status": "Failure"} # type: ignore[attr-defined] + assert not BlockDomain.is_polling_in_progress(result) + + +def test_pan_os_push_status_error_contents_is_terminal_failure(monkeypatch): + """ + Given: + - pan-os-push-status returns an error entry whose Contents is a plain string + (PAN-OS surfaces errors as a bare string instead of the nested status dict). + When: + - Calling pan_os_push_status. + Then: + - The function does NOT crash with 'str object has no attribute get'; it stops polling + (no ScheduledCommand attached) and reports a Failure status for the job. + """ + monkeypatch.setattr( + BlockDomain.demisto, + "executeCommand", + lambda *a, **k: [err_entry("Failed to execute pan-os-push-status. Error: job not found")], + ) + + # The @polling_function decorator unwraps the PollResult and returns its CommandResults response + # at runtime, though the declared return type is still PollResult (hence the type: ignore below). + result = pan_os_push_status({"push_job_id": "123"}, []) + + assert not BlockDomain.is_polling_in_progress(result) + assert result.outputs == {"JobID": "123", "Status": "Failure"} # type: ignore[attr-defined] + + +def test_pan_os_push_status_fin_stops_polling(monkeypatch): + """ + Given: + - pan-os-push-status returns a well-formed nested dict whose job status is 'FIN'. + When: + - Calling pan_os_push_status. + Then: + - Polling stops (no ScheduledCommand attached) and the reported job status is 'FIN'. + """ + fin_entry = { + "Type": 1, + "Contents": {"response": {"result": {"job": {"status": "FIN"}}}}, + "HumanReadable": "", + "EntryContext": {}, + } + monkeypatch.setattr(BlockDomain.demisto, "executeCommand", lambda *a, **k: [fin_entry]) + + result = pan_os_push_status({"push_job_id": "456"}, []) + + assert not BlockDomain.is_polling_in_progress(result) + assert result.outputs == {"Status": "FIN", "JobID": "456"} # type: ignore[attr-defined] + + +def _install_fake_context(monkeypatch): + """Back demisto.context()/setContext() with an in-memory dict, mirroring platform behavior. + + Returns the backing store so tests can inspect exactly what was serialized to context. + """ + store: dict = {} + monkeypatch.setattr(BlockDomain.demisto, "context", lambda: dict(store)) + monkeypatch.setattr(BlockDomain.demisto, "setContext", lambda key, value: store.__setitem__(key, value)) + return store + + +def test_save_and_restore_responses_json_round_trip(monkeypatch): + """ + Given: + - A PanOs instance whose accumulated responses contain the realistic PAN-OS entry shape + (nested Contents dict, Metadata, None HumanReadable) written to context as JSON. + When: + - Calling save_responses on one polling cycle and restore_responses on the next. + Then: + - The context value is valid JSON (not a Python repr), and the responses survive the + json.dumps -> json.loads round-trip byte-for-byte equal to the reduced form. + """ + store = _install_fake_context(monkeypatch) + + pan_os = _pan_os(["evil.example.com"]) + pan_os.responses = [ + [ + { + "Type": 1, + "Contents": {"response": {"result": {"job": {"status": "FIN", "id": "42"}}}}, + "HumanReadable": None, + "Metadata": {"instance": "Panorama_QA", "brand": "Panorama"}, + "EntryContext": {"dropped": "not serialized"}, + } + ] + ] + expected_reduced = pan_os.reduce_responses() + + pan_os.save_responses() + + # Stored value must be real JSON that json.loads can parse (would fail on a Python repr). + stored = store["panorama_responses"] + assert json.loads(stored) == expected_reduced + + # A fresh instance restoring from the same context recovers the reduced responses exactly. + fresh = _pan_os(["evil.example.com"]) + fresh.restore_responses() + assert fresh.responses == expected_reduced + + +def test_finish_reads_rows_and_responses_as_json_then_clears_context(monkeypatch): + """ + Given: + - Context holds block_domain_rows and panorama_responses that were written as JSON by a + previous polling cycle. + When: + - Calling finish(). + Then: + - The rows are parsed back from JSON and returned; the accumulated responses are restored + onto the instance for verbose output; and all polling context keys are cleared. + """ + store = _install_fake_context(monkeypatch) + + rows = [ + { + "Domain": "evil.example.com", + "Brand": "Panorama", + "Status": STATUS_DONE, + "Result": RESULT_SUCCESS, + "RuleName": "Cortex - Block Domain", + "Message": "ok", + } + ] + responses = [[{"HumanReadable": "HR", "Contents": "ok", "Type": 1, "Metadata": None}]] + store["block_domain_rows"] = json.dumps(rows) + store["panorama_responses"] = json.dumps(responses) + store["commit_job_id"] = "999" + + pan_os = _pan_os(["evil.example.com"]) + result = pan_os.finish() + + assert result == rows + assert pan_os.responses == responses + # All polling context keys are cleared on finish. + assert store["commit_job_id"] == "" + assert store["push_job_id"] == "" + assert store["panorama_responses"] == "" + assert store["block_domain_rows"] == "" + + +def test_finish_tolerates_corrupt_context_data(monkeypatch): + """ + Given: + - Context holds a non-JSON (corrupt) block_domain_rows value. + When: + - Calling finish(). + Then: + - finish() does not raise; it degrades to an empty list and still clears context. + """ + store = _install_fake_context(monkeypatch) + store["block_domain_rows"] = "{not valid json" + store["panorama_responses"] = "also not json" + + pan_os = _pan_os(["evil.example.com"]) + result = pan_os.finish() + + assert result == [] + assert store["block_domain_rows"] == "" diff --git a/Packs/AggregatedScripts/Scripts/BlockDomain/README.md b/Packs/AggregatedScripts/Scripts/BlockDomain/README.md new file mode 100644 index 000000000000..584dc2fb802a --- /dev/null +++ b/Packs/AggregatedScripts/Scripts/BlockDomain/README.md @@ -0,0 +1,38 @@ +The script blocks a list of domain FQDNs in supported integrations. Currently only supports Palo Alto Networks Panorama. + +## Script Data + +--- + +| **Name** | **Description** | +| --- | --- | +| Script Type | python3 | +| Cortex XSOAR Version | 6.10.0 | + +## Inputs + +--- + +| **Argument Name** | **Description** | +| --- | --- | +| domain_list | A comma-separated list of domain FQDNs to block. Wildcard entries \(e.g. \*.evil.com\) are not supported and are skipped. | +| rule_name | The name of the rule which will be created in the relevant products. Default: `Cortex - Block Domain`. | +| log_forwarding_name | The Panorama log forwarding object name that specifies the Log Forwarding setting to apply to the PAN-OS custom rules. | +| address_group | The name of the PAN-OS Panorama or Firewall address group used to hold the blocked domain FQDN objects. Default: `Blocked Domains - Cortex`. | +| auto_commit | Whether to commit the new rule and push to the device group at the end of the run. Default: `true`. | +| tag | The designated tag name for the domain FQDN object. Applied to every object the script creates. Default: `cortex-blocked-domains`. | +| brands | A comma-separated list of integration brands to run the command for. If not provided, the command runs for all available integrations. | +| verbose | Whether to retrieve a human-readable entry for every command or only the final result. True retrieves a human-readable entry for every command. False retrieves a human-readable entry only for the final result. Default: `false`. | + +## Outputs + +--- + +| **Path** | **Description** | **Type** | +| --- | --- | --- | +| BlockDomain.Domain | The domain FQDN that was processed. | String | +| BlockDomain.Brand | The brand \(integration\) used to block the domain. | String | +| BlockDomain.Status | The lifecycle status of the action. One of Done, Pending, Skipped, Failed. | String | +| BlockDomain.Result | The result of the action. Success or Failed. | String | +| BlockDomain.RuleName | The name of the rule used for this integration. Empty if no rule was used. | String | +| BlockDomain.Message | A message concerning the result of the action. | String | diff --git a/Packs/AggregatedScripts/Scripts/BlockExternalIp/BlockExternalIp.yml b/Packs/AggregatedScripts/Scripts/BlockExternalIp/BlockExternalIp.yml index 07a6db1de48a..49d3d6146d65 100644 --- a/Packs/AggregatedScripts/Scripts/BlockExternalIp/BlockExternalIp.yml +++ b/Packs/AggregatedScripts/Scripts/BlockExternalIp/BlockExternalIp.yml @@ -84,7 +84,7 @@ type: python subtype: python3 compliantpolicies: - IP Blockage -dockerimage: demisto/python3:3.12.13.10116658 +dockerimage: demisto/python3:3.12.14.13495286 fromversion: 6.1.0 marketplaces: - xsoar diff --git a/Packs/AggregatedScripts/Scripts/BlockUrl/BlockUrl.py b/Packs/AggregatedScripts/Scripts/BlockUrl/BlockUrl.py new file mode 100644 index 000000000000..193f089c9543 --- /dev/null +++ b/Packs/AggregatedScripts/Scripts/BlockUrl/BlockUrl.py @@ -0,0 +1,1300 @@ +import ast +import json +import re + +import demistomock as demisto +from CommonServerPython import * + + +POLLING = False +# Set by the commit and the push status functions when PAN-OS reported a terminal failure. Read by +# manage_pan_os_flow, which must then skip the push and report the failure instead of a success. +JOB_FAILURE_MESSAGE = "" + +SUPPORTED_BRANDS = ["Panorama"] +PAN_OS_BRAND = "Panorama" + +# PAN-OS returns this substring when an object does not exist. The generic "Request Failed." prefix +# is shared with real failures, so it must not be used for the not-found decision. +OBJECT_NOT_FOUND = "Object not present" +# Raised by pan-os-edit-custom-url-category / pan-os-edit-url-filter when the target object still +# carries uncommitted changes (@dirtyId). +DIRTY_OBJECT_ERROR = "Please commit the instance prior to editing" + +URL_LIST_TYPE = "URL List" +MAX_URL_LENGTH = 255 + +XML_SPECIAL_CHARS = {"<", ">", "&", '"', "'"} + +# The description stamped on every object this script creates, so an operator can tell them apart +# from user-managed objects. +OBJECT_DESCRIPTION = "Created by the Cortex block-url script." + +POLLING_INTERVAL = 30 +POLLING_TIMEOUT = 1200 + +# A commit job that finished with any result other than "OK" failed. Warnings are not a failure. +COMMIT_JOB_FINISHED_STATUS = "FIN" +COMMIT_JOB_SUCCESS_RESULT = "OK" +# The terminal statuses of a push job. Anything else, including an empty status, means "keep polling". +PUSH_COMPLETED_STATUS = "Completed" +PUSH_FAILURE_STATUSES = ("FAIL", "Failed") + + +class BlockUrlError(Exception): + """Raised when the PAN-OS flow cannot continue and the failure must be reported to the user.""" + + +""" HELPER FUNCTIONS """ + + +def _text(value: Any) -> str: + """Normalize a PAN-OS context value that may be wrapped in a '#text' dictionary. + + PAN-OS returns plain strings for committed objects, but dictionaries carrying '#text' and + '@dirtyId' keys while the object has uncommitted changes. + Args: + value (Any): The raw context value. + Returns: + The plain string value. + """ + if isinstance(value, dict): + return str(value.get("#text", "")) + if value is None: + return "" + return str(value) + + +def _text_list(value: Any) -> list[str]: + """Normalize a PAN-OS context value that holds zero, one or many possibly wrapped members. + + Args: + value (Any): The raw context value, a single member or a list of members. + Returns: + A list of plain string values, without empty entries. + """ + if value is None: + return [] + members = value if isinstance(value, list) else [value] + return [text for text in (_text(member) for member in members) if text] + + +def normalize_url(raw_url: str) -> tuple[str, str]: + """Normalize a URL into the form PAN-OS custom URL categories expect. + + PAN-OS validates almost nothing: it stores a scheme, a wildcard or a mixed-case host verbatim + and then silently fails to match them. Every rule below therefore prevents a silent non-block. + Args: + raw_url (str): The URL as supplied by the user. + Returns: + A tuple of (submitted_url, rejection_message). On rejection the submitted URL is empty. + """ + url = (raw_url or "").strip() + if not url: + return "", "URL is empty." + if re.search(r"\s", url): + return "", "URL contains whitespace, which is not supported by PAN-OS custom URL categories." + if "," in url: + return "", "URL contains a comma, which would split it into two entries when submitted to PAN-OS." + url = re.sub(r"^https?://", "", url, flags=re.IGNORECASE) + if "://" in url: + return "", "URL scheme is not supported. Only the http and https schemes are supported." + if "*" in url: + return "", "Wildcards are not supported and will not be passed to PAN-OS." + if xml_special := set(url) & XML_SPECIAL_CHARS: + return "", ( + f"URL contains the character(s) {', '.join(sorted(xml_special))}, which are not supported by " + "PAN-OS custom URL categories." + ) + url = url.rstrip("/") + if not url: + return "", "URL is empty." + if len(url) > MAX_URL_LENGTH: + return "", f"URL exceeds {MAX_URL_LENGTH} chars (PAN-OS limit)." + return lowercase_host(url), "" + + +def lowercase_host(url: str) -> str: + """Lowercase only the host part of a URL. PAN-OS does not fold case, and paths are case sensitive. + + Args: + url (str): A scheme-less URL. + Returns: + The URL with a lowercased host. + """ + host, separator, path = url.partition("/") + return f"{host.lower()}{separator}{path}" + + +def partition_urls(url_list: list[str]) -> tuple[list[dict], list[dict]]: + """Split the requested URLs into the ones that can be submitted and the ones that were rejected. + + Rejected URLs never reach any brand, and are reported with an empty SubmittedURL. + Args: + url_list (list[str]): The raw URLs supplied by the user. + Returns: + A tuple of (accepted entries, rejected entries). + """ + accepted: list[dict] = [] + rejected: list[dict] = [] + for raw_url in url_list: + submitted_url, rejection_message = normalize_url(raw_url) + if rejection_message: + rejected.append({"URL": raw_url, "SubmittedURL": "", "Result": "Failed", "Message": rejection_message}) + else: + accepted.append({"URL": raw_url, "SubmittedURL": submitted_url}) + demisto.debug(f"BU: partitioned the URLs into {len(accepted)} accepted and {len(rejected)} rejected.") + return accepted, rejected + + +def get_error_message(res: list[dict]) -> str: + """Collect the contents of every error entry in a command response. + + Args: + res (list[dict]): The response of demisto.executeCommand. + Returns: + The joined error messages, or an empty string when there was no error. + """ + messages = [str(entry.get("Contents", "")) for entry in res or [] if is_error(entry)] + return ", ".join(messages) + + +def raise_on_command_error(res: list[dict], object_description: str) -> None: + """Raise a BlockUrlError with an actionable message when a command response holds an error. + + Args: + res (list[dict]): The response of demisto.executeCommand. + object_description (str): A human readable description of the PAN-OS object being handled. + Raises: + BlockUrlError: When the response contains an error entry. + """ + message = get_error_message(res) + if not message: + return + if DIRTY_OBJECT_ERROR in message: + raise BlockUrlError( + f"The {object_description} has uncommitted changes on PAN-OS, so it cannot be edited. " + f"Commit the pending changes on PAN-OS and re-run the script. PAN-OS error: {message}" + ) + raise BlockUrlError(message) + + +def get_relevant_context(original_context: dict[str, Any], key: str) -> dict | list: + """Get the relevant context object from the execute_command response. + + Args: + original_context (dict[str, Any]): The original context ('EntryContext') of the response. + key (str): The key to extract from the original_context. + Returns: + A dict or a list that is the relevant command context. + """ + if not original_context: + return {} + if relevant_context := original_context.get(key, {}): + demisto.debug(f"BU: the {key=} was found in the first search.") + return relevant_context + for context_key in original_context: + if context_key.startswith(key): + demisto.debug(f"BU: the {key=} was found in the context as {context_key}") + return original_context.get(context_key, {}) + return {} + + +def get_context_entry(res: list[dict], key: str) -> dict: + """Get a single context entry of a given key from a command response. + + Args: + res (list[dict]): The response of demisto.executeCommand. + key (str): The context key to extract. + Returns: + The context entry as a dict, or an empty dict. + """ + if not res: + return {} + context = get_relevant_context(res[0].get("EntryContext", {}), key) + if isinstance(context, list): + return context[0] if context else {} + return context + + +def check_value_exist_in_context(value: str, context: list | dict, key: str) -> bool: + """Verify if a specific value of a specific key is present in the context. + + Args: + value (str): The value whose existence is checked. + context (list | dict): The command context. + key (str): The key to extract from the context. + Returns: + Whether the value is present. + """ + items = [context] if isinstance(context, dict) else context + for item in items or []: + if _text(item.get(key)) == value: + demisto.debug(f"BU: the {value=} was found.") + return True + demisto.debug(f"BU: the {value=} isn't in the context with the {key=}") + return False + + +def is_polling_resume() -> bool: + """Whether the current run is a polling round that resumes an earlier run. + + Returns: + True when a commit or a push job is already in flight. + """ + incident_context = demisto.context() + return bool(demisto.get(incident_context, "commit_job_id") or demisto.get(incident_context, "push_job_id")) + + +def update_brands_to_run(brands_to_run: list) -> tuple[list, set]: + """Delete the brands that were already executed from the list of brands to run. + + Used when at least one brand finished its execution before the polling of another brand is over. + Args: + brands_to_run (list): The list of brands that should be executed. + Returns: + The list of brands executed in previous runs, and the set of brands to execute now. + """ + if PAN_OS_BRAND not in brands_to_run: + return [], set(brands_to_run) + incident_context = demisto.context() + executed_brands = (incident_context.get("executed_brands", "[]")).replace("'", '"') + try: + executed_brands = json.loads(executed_brands) + except json.JSONDecodeError: + demisto.debug("BU: there was a failure in the json.loads for the executed_brands.") + executed_brands = [] + updated_brands_to_run = {brand for brand in brands_to_run if brand not in executed_brands} + demisto.debug(f"BU: removed {executed_brands=} from {brands_to_run=}") + return executed_brands, updated_brands_to_run + + +def run_execute_command(command_name: str, args: dict[str, Any]) -> list[dict]: + """Execute a command and return its raw results. + + Args: + command_name (str): The name of the command to execute. + args (dict[str, Any]): The arguments to pass to the command. + Returns: + A list of the command result entries. + """ + res = demisto.executeCommand(command_name, args) + return cast(list[dict], res) + + +""" CLIENT CLASS """ + + +class PanOs: + def __init__(self, args: dict) -> None: + self.args = args + self.responses: list = [] + self.is_panorama: bool = False + self.device_group: str = "" + self.already_present_urls: list[str] = [] + self.failure_message: str = "" + + """ TOPOLOGY """ + + def detect_topology(self) -> None: + """Detect whether the configured instance is a Panorama or a plain firewall. + + Panorama requires pre_post on every rule command and scopes objects by device group. + """ + res_pan_os = run_execute_command("pan-os", {"cmd": "", "type": "op"}) + self.responses.append(res_pan_os) + raise_on_command_error(res_pan_os, "PAN-OS system information") + context = get_relevant_context(res_pan_os[0].get("EntryContext", {}), "Panorama.Command") + model = context.get("response", {}).get("result", {}).get("system", {}).get("model", "") # type: ignore[union-attr] + self.is_panorama = model == PAN_OS_BRAND + demisto.debug(f"BU: detected {model=}, {self.is_panorama=}") + + def rule_scope_args(self) -> dict[str, Any]: + """The scoping arguments every rule command needs on Panorama and must not get on a firewall. + + Returns: + The extra arguments for the rule commands. + """ + return {"pre_post": "pre-rulebase"} if self.is_panorama else {} + + def capture_device_group(self, context: dict) -> None: + """Remember the device group PAN-OS echoed back, for the commit, the push and the xpath. + + Args: + context (dict): A PAN-OS command context entry. + """ + device_group = _text(context.get("DeviceGroup")) if isinstance(context, dict) else "" + if device_group: + self.device_group = device_group + demisto.debug(f"BU: captured {self.device_group=}") + + """ CUSTOM URL CATEGORY """ + + def get_custom_url_category(self) -> dict: + """Read the custom URL category, treating "Object not present" as missing rather than failed. + + Returns: + The category context entry, or an empty dict when the category does not exist. + Raises: + BlockUrlError: When the command failed for any other reason. + """ + category = self.args["url_category"] + res = run_execute_command("pan-os-get-custom-url-category", {"name": category}) + if error_message := get_error_message(res): + if OBJECT_NOT_FOUND in error_message: + demisto.debug(f"BU: the custom URL category {category=} does not exist yet.") + return {} + raise BlockUrlError(error_message) + self.responses.append(res) + context = get_context_entry(res, "Panorama.CustomURLCategory") + self.capture_device_group(context) + return context + + def ensure_url_category(self, existing: dict, urls: list[str]) -> list[str]: + """Create the custom URL category, or append only the URLs it does not hold yet. + + Args: + existing (dict): The existing category context, or an empty dict. + urls (list[str]): The normalized URLs to block. + Returns: + The URLs that were already present in the category. + Raises: + BlockUrlError: When the category exists with a type other than "URL List", or on failure. + """ + category = self.args["url_category"] + if not existing: + return self.create_url_category(urls) + category_type = _text(existing.get("Type")) + if category_type != URL_LIST_TYPE: + # PAN-OS silently converts a foreign-type category to a URL List and merges its members + # into the site list, destroying the object. There is no server-side protection. + raise BlockUrlError( + f"Won't modify a user-managed category '{category}' of type '{category_type}'. " + f"Only categories of type '{URL_LIST_TYPE}' are managed by this script. " + f"Use the url_category argument to point to a different category." + ) + existing_sites = _text_list(existing.get("Sites")) + missing_urls = [url for url in urls if url not in existing_sites] + already_present = [url for url in urls if url in existing_sites] + if not missing_urls: + demisto.debug(f"BU: every URL is already present in the category {category=}, skipping the edit.") + return already_present + res = run_execute_command("pan-os-edit-custom-url-category", {"name": category, "action": "add", "sites": missing_urls}) + self.responses.append(res) + raise_on_command_error(res, f"custom URL category '{category}'") + return already_present + + def create_url_category(self, urls: list[str]) -> list[str]: + """Create the custom URL category with the requested URLs. + + Args: + urls (list[str]): The normalized URLs to block. + Returns: + An empty list, since a new category holds none of the URLs yet. + Raises: + BlockUrlError: When the creation failed. + """ + category = self.args["url_category"] + res = run_execute_command( + "pan-os-create-custom-url-category", + {"name": category, "type": URL_LIST_TYPE, "sites": urls, "description": OBJECT_DESCRIPTION}, + ) + self.responses.append(res) + raise_on_command_error(res, f"custom URL category '{category}'") + self.capture_device_group(get_context_entry(res, "Panorama.CustomURLCategory")) + return [] + + """ URL FILTERING PROFILE """ + + def ensure_url_filtering_profile(self) -> None: + """Create the URL filtering profile, or attach the category to the existing one. + + Raises: + BlockUrlError: When the profile could not be read, created or edited. + """ + profile = self.args["url_filtering_profile"] + res = run_execute_command("pan-os-get-url-filter", {"name": profile}) + error_message = get_error_message(res) + if error_message and OBJECT_NOT_FOUND not in error_message: + raise BlockUrlError(error_message) + if error_message: + demisto.debug(f"BU: the URL filtering profile {profile=} does not exist yet.") + self.create_url_filtering_profile() + return + self.responses.append(res) + context = get_context_entry(res, "Panorama.URLFilter") + self.capture_device_group(context) + self.attach_category_to_profile(context) + + def create_url_filtering_profile(self) -> None: + """Create the URL filtering profile with the category blocked for Site Access. + + Raises: + BlockUrlError: When the creation failed. + """ + profile = self.args["url_filtering_profile"] + res = run_execute_command( + "pan-os-create-url-filter", + { + "name": profile, + "url_category": self.args["url_category"], + "action": "block", + "description": OBJECT_DESCRIPTION, + }, + ) + self.responses.append(res) + raise_on_command_error(res, f"URL filtering profile '{profile}'") + self.capture_device_group(get_context_entry(res, "Panorama.URLFilter")) + + def attach_category_to_profile(self, context: dict) -> None: + """Add the category to the profile block list when it is not attached yet. + + Args: + context (dict): The existing profile context. + Raises: + BlockUrlError: When the edit failed. + """ + profile = self.args["url_filtering_profile"] + category = self.args["url_category"] + if self.is_category_attached(context, category): + demisto.debug(f"BU: the category {category=} is already blocked by the profile {profile=}.") + return + res = run_execute_command( + "pan-os-edit-url-filter", + { + "name": profile, + "element_to_change": "block_categories", + "element_value": category, + "add_remove_element": "add", + }, + ) + self.responses.append(res) + raise_on_command_error(res, f"URL filtering profile '{profile}'") + + @staticmethod + def is_category_attached(context: dict, category: str) -> bool: + """Whether the category is already in the block list of the profile. + + Args: + context (dict): The profile context. + category (str): The custom URL category name. + Returns: + Whether the category is attached. + """ + if category in _text_list(context.get("block_categories")): + return True + categories = context.get("Category") or [] + if isinstance(categories, dict): + categories = [categories] + return any(_text(item.get("Name")) == category for item in categories) + + """ TAG AND SECURITY RULE """ + + def ensure_tag(self) -> None: + """Create the tag object before any rule references it. + + pan-os-create-rule fails with "tag '' is not a valid reference" when the tag object + does not exist. Re-creating an existing tag succeeds, so this call is idempotent. A failure + here is tolerated rather than aborting the whole flow. + """ + tag = self.args.get("tag", "") + if not tag: + return + res = run_execute_command("pan-os-create-tag", {"name": tag}) + if error_message := get_error_message(res): + demisto.debug(f"BU: could not create the tag {tag=}, continuing without it. {error_message=}") + return + self.responses.append(res) + + def ensure_security_rule(self) -> None: + """Create the security rule when it is missing, then attach the profile and move it to the top. + + Raises: + BlockUrlError: When the rulebase could not be read or the rule could not be created. + """ + rule_name = self.args["rule_name"] + res_list_rules = run_execute_command("pan-os-list-rules", self.rule_scope_args()) + self.responses.append(res_list_rules) + raise_on_command_error(res_list_rules, f"security rule '{rule_name}'") + context = get_relevant_context(res_list_rules[0].get("EntryContext", {}), "Panorama.SecurityRule") + if check_value_exist_in_context(rule_name, context, "Name"): + demisto.debug(f"BU: reusing the existing security rule {rule_name=}.") + self.add_tag_to_rule() + else: + self.create_security_rule() + self.apply_profile_to_rule() + self.move_rule_to_top() + + def add_tag_to_rule(self) -> None: + """Add the configured tag to an existing rule unconditionally. + + Uses behaviour=add so the rule's existing tags are preserved rather than replaced. The edit + is issued every run because the rule may have been changed outside this script (for example + through the PAN-OS UI), so the fetched tag list can be stale. Adding an already-present tag + is a no-op on the device, so this stays idempotent. + Raises: + BlockUrlError: When the edit failed. + """ + tag = self.args.get("tag", "") + if not tag: + return + rule_name = self.args["rule_name"] + edit_args: dict[str, Any] = { + "rulename": rule_name, + "element_to_change": "tag", + "element_value": tag, + "behaviour": "add", + } + edit_args |= self.rule_scope_args() + res = run_execute_command("pan-os-edit-rule", edit_args) + self.responses.append(res) + raise_on_command_error(res, f"security rule '{rule_name}'") + + def create_security_rule(self) -> None: + """Create the security rule that carries the URL filtering profile. + + Raises: + BlockUrlError: When the creation failed. + """ + rule_name = self.args["rule_name"] + create_rule_args: dict[str, Any] = { + "rulename": rule_name, + # The action is ALLOW by design. The URL filtering profile attached to this rule performs + # the blocking, and PAN-OS only inspects traffic with a profile when the rule allows it. + # A "deny" rule would drop the traffic before the profile runs, so do not change this. + "action": "allow", + "source": "any", + "destination": "any", + "application": "any", + "description": OBJECT_DESCRIPTION, + } + if tag := self.args.get("tag", ""): + create_rule_args["tags"] = tag + if log_forwarding_name := self.args.get("log_forwarding_name", ""): + create_rule_args["log_forwarding"] = log_forwarding_name + create_rule_args |= self.rule_scope_args() + res = run_execute_command("pan-os-create-rule", create_rule_args) + self.responses.append(res) + raise_on_command_error(res, f"security rule '{rule_name}'") + + def apply_profile_to_rule(self) -> None: + """Attach the URL filtering profile to the security rule. + + Raises: + BlockUrlError: When the command failed. + """ + rule_name = self.args["rule_name"] + apply_args: dict[str, Any] = { + "profile_type": "url-filtering", + "rule_name": rule_name, + "profile_name": self.args["url_filtering_profile"], + } + apply_args |= self.rule_scope_args() + res = run_execute_command("pan-os-apply-security-profile", apply_args) + self.responses.append(res) + raise_on_command_error(res, f"security rule '{rule_name}'") + + def move_rule_to_top(self) -> None: + """Move the security rule to the top of the rulebase. + + pan-os-create-rule appends the rule at the bottom, where an earlier allow rule would defeat + the block, so the move is mandatory. It is idempotent ("Rule already at the top"), therefore + it runs on every execution. + Raises: + BlockUrlError: When the command failed. + """ + rule_name = self.args["rule_name"] + move_args: dict[str, Any] = {"rulename": rule_name, "where": "top"} + move_args |= self.rule_scope_args() + res = run_execute_command("pan-os-move-rule", move_args) + self.responses.append(res) + raise_on_command_error(res, f"security rule '{rule_name}'") + + """ FLOW """ + + def reduce_pan_os_responses(self) -> list[list[dict]]: + """Reduce the stored responses to the parts the later polling rounds need. + + Returns: + A list containing the relevant parts of the command responses. + """ + demisto.debug("BU: updating the responses in reduce_pan_os_responses.") + reduced_responses = [] + for res in self.responses: + current_new_res = [] + for entry in res: + current_new_res.append( + { + "HumanReadable": entry.get("HumanReadable"), + "Contents": entry.get("Contents"), + "Type": entry.get("Type"), + "Metadata": entry.get("Metadata"), + } + ) + reduced_responses.append(current_new_res) + demisto.debug(f"BU: {len(reduced_responses)=}, {len(self.responses)=}") + return reduced_responses + + def save_state_to_context(self) -> None: + """Persist the state the next polling round needs.""" + demisto.setContext("panorama_responses", str(self.reduce_pan_os_responses())) + demisto.setContext("pan_os_device_group", self.device_group) + + def restore_state_from_context(self, incident_context: dict) -> None: + """Restore the state saved by the previous polling round. + + Args: + incident_context (dict): The incident context. + """ + self.responses = ast.literal_eval(incident_context.get("panorama_responses", "") or "[]") + self.device_group = demisto.get(incident_context, "pan_os_device_group") or "" + + def start_pan_os_flow(self) -> tuple[list, bool]: + """Run the configuration part of the PAN-OS flow. + + Returns: + A tuple of the command results, and whether the changes should be committed. The results + are populated only when the flow failed and there is nothing left to commit. + """ + try: + self.detect_topology() + existing_category = self.get_custom_url_category() + self.already_present_urls = self.ensure_url_category(existing_category, self.submitted_urls()) + self.ensure_url_filtering_profile() + self.ensure_tag() + self.ensure_security_rule() + except BlockUrlError as error: + demisto.debug(f"BU: the PAN-OS flow failed. {error=}") + self.failure_message = str(error) + return self.pan_os_finish(), False + return [], bool(self.args.get("auto_commit", True)) + + def submitted_urls(self) -> list[str]: + """The normalized URLs that are submitted to PAN-OS. + + Returns: + The list of submitted URLs. + """ + return [entry["SubmittedURL"] for entry in self.args["url_entries"]] + + def adopt_job_failure(self) -> bool: + """Adopt a commit or push job failure reported by the polling functions. + + The polling functions run outside the class and communicate a terminal PAN-OS job failure + through the JOB_FAILURE_MESSAGE global, since a failed job comes back as a type-1 success + entry that the is_error sweep in prepare_context_and_hr_multiple_executions cannot detect. + Returns: + Whether a job failure was adopted. + """ + global JOB_FAILURE_MESSAGE + if not JOB_FAILURE_MESSAGE: + return False + demisto.debug(f"BU: adopting the job failure {JOB_FAILURE_MESSAGE=}") + self.failure_message = JOB_FAILURE_MESSAGE + JOB_FAILURE_MESSAGE = "" + return True + + def pan_os_finish(self) -> list[CommandResults]: + """Clear the polling state from the context and build the final results. + + Returns: + The list of Command Results. + """ + demisto.setContext("push_job_id", "") + demisto.setContext("commit_job_id", "") + demisto.setContext("panorama_responses", "") + demisto.setContext("pan_os_device_group", "") + details = { + "brand": PAN_OS_BRAND, + "rule_name": self.args.get("rule_name", ""), + "url_category": self.args.get("url_category", ""), + "job_id": self.args.get("commit_job_id") or "", + "already_present": self.already_present_urls, + "failure_message": self.failure_message, + } + return prepare_context_and_hr_multiple_executions( + self.responses, bool(self.args.get("verbose", False)), self.args["url_entries"], details + ) + + def manage_pan_os_flow(self) -> CommandResults | list[CommandResults] | PollResult: + """Manage the different states of the PAN-OS flow. + + 1. The flow start: create or reuse the category, the profile, the tag and the rule. + 2. If auto_commit is true and there were changes, execute pan-os-commit. + 3. There is a commit job ID, check the status of the commit. + 4. The commit finished, on Panorama push the changes to the device group. + 5. There is a push job ID, check the status of the push. + 6. The push finished, build the final results. + Returns: + A PollResult while polling, otherwise a Command Result or a list of Command Results. + """ + incident_context = demisto.context() + auto_commit = self.args["auto_commit"] + commit_job_id = self.args.get("commit_job_id") + context_push_job_id = demisto.get(incident_context, "push_job_id") + context_commit_job_id = demisto.get(incident_context, "commit_job_id") + is_polling_reentry = bool(commit_job_id) + if not is_polling_reentry and (context_commit_job_id or context_push_job_id): + demisto.debug( + f"BU: stale polling context on a fresh invocation " + f"(commit={context_commit_job_id!r}, push={context_push_job_id!r}); clearing." + ) + demisto.setContext("commit_job_id", "") + demisto.setContext("push_job_id", "") + demisto.setContext("panorama_responses", "") + demisto.setContext("pan_os_device_group", "") + context_push_job_id = None + push_job_id = context_push_job_id if is_polling_reentry else None + # state 5 + if push_job_id: + demisto.debug(f"BU: has a {push_job_id=}") + self.restore_state_from_context(incident_context) + self.args["push_job_id"] = push_job_id + res_push_status = pan_os_push_status(self.args, self.responses) + # state 6 + if not POLLING: + demisto.debug("BU: finished polling, finishing the flow.") + self.adopt_job_failure() + return self.pan_os_finish() + self.save_state_to_context() + return res_push_status + # state 3 + if commit_job_id: + demisto.debug(f"BU: has a {commit_job_id=}") + self.args["commit_job_id"] = commit_job_id + self.restore_state_from_context(incident_context) + poll_commit_status = pan_os_commit_status(self.args, self.responses) + if POLLING: + self.save_state_to_context() + return poll_commit_status + # A commit that PAN-OS finished with a failure must not be pushed: the pushed candidate + # config never passed validation, and reporting success here would tell the user the URLs + # are blocked when they are not. + if self.adopt_job_failure(): + demisto.debug("BU: the commit job failed, skipping the push.") + return self.pan_os_finish() + # state 4 + self.detect_topology() + if not self.is_panorama: + demisto.debug("BU: not a Panorama instance, not pushing to the device group.") + return self.pan_os_finish() + self.args["device_group"] = self.device_group + poll_push_to_device = pan_os_push_to_device(self.args, self.responses) + if not POLLING: + demisto.debug("BU: nothing to push, finishing the flow.") + self.adopt_job_failure() + return self.pan_os_finish() + self.save_state_to_context() + return poll_push_to_device + + # state 1 + results, should_commit = self.start_pan_os_flow() + if not should_commit: + if results: + return results + not_committed = CommandResults( + readable_output=f"Not committing the changes in PAN-OS, since {auto_commit=}. " + f"Please do so manually for the changes to take effect." + ) + final_results = self.pan_os_finish() + final_results.append(not_committed) + return final_results + # state 2 + self.args["device_group"] = self.device_group + self.args["is_panorama"] = self.is_panorama + poll_result = pan_os_commit(self.args, self.responses) + if not POLLING: + return self.pan_os_finish() + self.save_state_to_context() + return poll_result + + +""" STANDALONE FUNCTION """ + + +def job_details(job: dict) -> str: + """Extract the human readable reason a PAN-OS job reports for its outcome. + + PAN-OS returns the reason either as a plain string or as a {"line": [...]} structure, and the + lines may themselves be nested lists. + Args: + job (dict): The job element of a pan-os-commit-status response. + Returns: + The joined details, or a placeholder when PAN-OS reported none. + """ + details = job.get("details") + if isinstance(details, dict): + details = details.get("line") + if isinstance(details, list): + flattened = [str(item) for line in details for item in (line if isinstance(line, list) else [line])] + return "; ".join(flattened) or "none reported" + return str(details) if details else "none reported" + + +def build_commit_args(args: dict) -> dict[str, Any]: + """Build the arguments of pan-os-commit, scoped as narrowly as PAN-OS allows. + + PAN-OS has no per-object commit, so the device group is the finest available granularity. + Args: + args (dict): The flow arguments. + Returns: + The arguments for pan-os-commit. + """ + url_count = len(args.get("url_entries", [])) + commit_args: dict[str, Any] = { + "polling": True, + "description": f"Block URL - {url_count} URL(s) - {args.get('incident_id', '')}", + "exclude_device_network_configuration": True, + } + device_group = args.get("device_group", "") + if args.get("is_panorama") and device_group: + commit_args["device-group"] = device_group + # Only safe when the objects live in the device group and not in /config/shared. + commit_args["exclude_shared_objects"] = True + return commit_args + + +def build_push_args(args: dict) -> dict[str, Any]: + """Build the arguments of pan-os-push-to-device-group. + + Args: + args (dict): The flow arguments. + Returns: + The arguments for pan-os-push-to-device-group. + """ + push_args: dict[str, Any] = { + "polling": True, + "description": f"Block URL - {args.get('incident_id', '')}", + } + if device_group := args.get("device_group", ""): + push_args["device-group"] = device_group + return push_args + + +def create_final_human_readable(failure_message: str, context: list[dict]) -> str: + """Create the human readable summary of the script. + + Args: + failure_message (str): A failure message if relevant. + context (list[dict]): The final context records. + Returns: + The human readable summary. + """ + headers = ["URL", "SubmittedURL", "Result", "Brand", "RuleName", "URLCategory", "JobID", "Message"] + name = "Failed to block the URL/s" if failure_message else "URL/s blocking summary" + demisto.debug(f"BU: creating the final human readable for {failure_message=}") + return tableToMarkdown(name=name, t=context, headers=headers, removeNull=True) + + +def create_final_context(used_integration: str, url_entries: list[dict], details: dict) -> list[dict]: + """Create the context records of the script, one per URL. + + Args: + used_integration (str): The integration that was used. + url_entries (list[dict]): The accepted URL entries, holding URL and SubmittedURL. + details (dict): The flow details, holding the rule name, the category, the job ID, the URLs + that were already present and the failure message. + Returns: + The list of context records. + """ + failure_message = details.get("failure_message", "") + already_present = details.get("already_present", []) + context = [] + for entry in url_entries: + submitted_url = entry.get("SubmittedURL", "") + if failure_message: + result, message = "Failed", failure_message + elif submitted_url in already_present: + result, message = "Skipped", "URL already present in the category." + else: + result, message = "Success", "URL was blocked successfully." + context.append( + { + "URL": entry.get("URL") or None, + "SubmittedURL": submitted_url or None, + "Brand": used_integration or None, + "Result": result or None, + "Message": message or None, + "RuleName": details.get("rule_name") or None, + "URLCategory": details.get("url_category") or None, + "JobID": details.get("job_id") or None, + } + ) + return context + + +def create_rejected_results(rejected: list[dict]) -> CommandResults: + """Create the results of the URLs that were rejected before any brand was contacted. + + Args: + rejected (list[dict]): The rejected URL entries. + Returns: + A Command Results holding the rejected records. + """ + context = [ + { + "URL": entry.get("URL") or None, + "SubmittedURL": None, + "Brand": None, + "Result": "Failed", + "Message": entry.get("Message") or None, + "RuleName": None, + "URLCategory": None, + "JobID": None, + } + for entry in rejected + ] + headers = ["URL", "SubmittedURL", "Result", "Message"] + return CommandResults( + readable_output=tableToMarkdown(name="URL/s that were not submitted", t=context, headers=headers, removeNull=True), + outputs_prefix="BlockURLResults", + outputs=context, + raw_response=context, + ) + + +def prepare_context_and_hr_multiple_executions( + responses: list[list[dict]], verbose: bool, url_entries: list[dict], details: dict +) -> list[CommandResults]: + """Create the context and the human readable of a flow made of multiple command executions. + + Args: + responses (list[list[dict]]): The responses returned from the command executions. + verbose (bool): Whether to return a human readable entry per command or only the summary. + url_entries (list[dict]): The accepted URL entries. + details (dict): The flow details. + Returns: + A list containing the relevant Command Results. + """ + demisto.debug(f"BU: in prepare_context_and_hr_multiple_executions, {len(responses)=}") + results = [] + failed_messages = [] + used_integration = details.get("brand", "") + if responses and responses[0]: + used_integration = responses[0][0].get("Metadata", {}).get("brand") or used_integration + + for res in responses: + for entry in res: + command_hr = entry.get("HumanReadable") + message = f"{used_integration}: {entry.get('Contents', '')}" + if is_error(entry): + demisto.debug(f"BU: a failure was found {message=}") + failed_messages.append(message) + elif command_hr and command_hr != str(None): + results.append(CommandResults(readable_output=f"{used_integration}:\n{command_hr}")) + elif message and isinstance(entry.get("Contents"), str): + results.append(CommandResults(readable_output=message)) + if explicit_failure := details.get("failure_message", ""): + failed_messages.append(explicit_failure) + details = details | {"failure_message": ", ".join(failed_messages)} + + final_context = create_final_context(used_integration, url_entries, details) + final_cr = CommandResults( + readable_output=create_final_human_readable(details["failure_message"], final_context), + outputs_prefix="BlockURLResults", + outputs=final_context, + raw_response=final_context, + ) + if verbose: + results.append(final_cr) + else: + results = [final_cr] + return results + + +""" COMMAND FUNCTION """ + + +@polling_function( + name="block-url", + interval=POLLING_INTERVAL, + timeout=POLLING_TIMEOUT, +) +def pan_os_commit(args: dict, responses: list) -> PollResult: + """Execute pan-os-commit. + + Args: + args (dict): The arguments of the flow. + responses (list): The responses of the command executions so far. + Returns: + A PollResult object. + """ + res_commit = run_execute_command("pan-os-commit", build_commit_args(args)) + responses.append(res_commit) + polling_args = res_commit[0].get("Metadata", {}).get("pollingArgs", {}) + job_id = polling_args.get("commit_job_id") + if job_id: + context_output = {"JobID": job_id, "Status": "Pending"} + continue_to_poll = True + commit_output = CommandResults( + outputs=context_output, readable_output=tableToMarkdown("Commit Status:", context_output, removeNull=True) + ) + demisto.debug(f"BU: initiated a commit execution {job_id=}") + demisto.setContext("commit_job_id", job_id) + else: # nothing to commit in PAN-OS, no reason to poll. + commit_output = res_commit[0].get("Contents") or "There are no changes to commit." # type: ignore[assignment] + demisto.debug(f"BU: no job_id, {commit_output}") + continue_to_poll = False + global POLLING + POLLING = continue_to_poll + + args_for_next_run = args | { + "commit_job_id": job_id, + "interval_in_seconds": arg_to_number(args.get("interval_in_seconds", POLLING_INTERVAL)), + "timeout": arg_to_number(args.get("timeout", POLLING_TIMEOUT)), + "polling": True, + } + return PollResult( + response=commit_output, + continue_to_poll=continue_to_poll, + args_for_next_run=args_for_next_run, + partial_result=CommandResults(readable_output=f"Waiting for commit job ID {job_id} to finish..."), + ) + + +@polling_function( + name="block-url", + interval=POLLING_INTERVAL, + timeout=POLLING_TIMEOUT, +) +def pan_os_commit_status(args: dict, responses: list) -> PollResult: + """Check the status of the commit in PAN-OS. + + A commit that finishes with warnings is a success: the lab returns warnings on every commit. A + commit that finishes with any result other than "OK" is a failure, and PAN-OS reports it as a + successful command entry carrying a failed job, so the failure is published through the + JOB_FAILURE_MESSAGE global for manage_pan_os_flow to skip the push and report it. + Args: + args (dict): The arguments of the flow. + responses (list): The responses of the command executions so far. + Returns: + A PollResult object. + """ + commit_job_id = args["commit_job_id"] + res_commit_status = run_execute_command("pan-os-commit-status", {"job_id": commit_job_id}) + responses.append(res_commit_status) + global POLLING, JOB_FAILURE_MESSAGE + raw_contents = res_commit_status[0].get("Contents") if res_commit_status else None + if is_error(res_commit_status) or not isinstance(raw_contents, dict): + POLLING = False + error_reason = get_error_message(res_commit_status) or str(raw_contents) + JOB_FAILURE_MESSAGE = ( + f"The PAN-OS commit job {commit_job_id} status could not be read, so the changes were not " + f"pushed and the URLs are not blocked. PAN-OS reason: {error_reason}" + ) + demisto.debug(f"BU: the commit status returned an error entry. {JOB_FAILURE_MESSAGE=}") + commit_output = {"JobID": commit_job_id, "Status": "Failure"} + return PollResult( + response=CommandResults( + outputs=commit_output, + outputs_key_field="JobID", + readable_output=tableToMarkdown("Commit Status:", commit_output, removeNull=True), + ), + args_for_next_run=args, + continue_to_poll=False, + ) + job = raw_contents.get("response", {}).get("result", {}).get("job", {}) + job_result = job.get("result") + continue_to_poll = job.get("status") != COMMIT_JOB_FINISHED_STATUS + job_failed = not continue_to_poll and job_result != COMMIT_JOB_SUCCESS_RESULT + commit_output = {"JobID": commit_job_id, "Status": "Failure" if job_failed else "Success"} + POLLING = continue_to_poll + if job_failed: + JOB_FAILURE_MESSAGE = ( + f"The PAN-OS commit job {commit_job_id} failed with the result '{job_result}', so the changes were " + f"not pushed and the URLs are not blocked. PAN-OS details: {job_details(job)}" + ) + demisto.debug(f"BU: the commit job failed. {JOB_FAILURE_MESSAGE=}") + demisto.debug(f"BU: after pan-os-commit-status {continue_to_poll=} {commit_job_id=} {job_result=}") + return PollResult( + response=CommandResults( + outputs=commit_output, + outputs_key_field="JobID", + readable_output=tableToMarkdown("Commit Status:", commit_output, removeNull=True), + ), + args_for_next_run=args, + continue_to_poll=continue_to_poll, + ) + + +@polling_function( + name="block-url", + interval=POLLING_INTERVAL, + timeout=POLLING_TIMEOUT, +) +def pan_os_push_to_device(args: dict, responses: list) -> PollResult: + """Execute pan-os-push-to-device-group. + + Args: + args (dict): The arguments of the flow. + responses (list): The responses of the command executions so far. + Returns: + A PollResult object. + """ + res_push_to_device = run_execute_command("pan-os-push-to-device-group", build_push_args(args)) + responses.append(res_push_to_device) + polling_args = res_push_to_device[0].get("Metadata", {}).get("pollingArgs", {}) + job_id = polling_args.get("push_job_id") + device_group = polling_args.get("device-group") + demisto.debug(f"BU: the polling args are {job_id=} {device_group=}") + if job_id: + context_output = {"DeviceGroup": device_group, "JobID": job_id, "Status": "Pending"} + continue_to_poll = True + push_cr = CommandResults( + outputs_key_field="JobID", + outputs=context_output, + readable_output=tableToMarkdown("Push to Device Group:", context_output, removeNull=True), + ) + demisto.setContext("push_job_id", job_id) + else: + push_cr = CommandResults(readable_output=res_push_to_device[0].get("Contents") or "There are no changes to push.") + continue_to_poll = False + global POLLING + POLLING = continue_to_poll + return PollResult( + response=push_cr, + continue_to_poll=continue_to_poll, + partial_result=CommandResults(readable_output=f"Waiting for Job-ID {job_id} to finish pushing the changes..."), + ) + + +@polling_function( + name="block-url", + interval=POLLING_INTERVAL, + timeout=POLLING_TIMEOUT, +) +def pan_os_push_status(args: dict, responses: list) -> PollResult: + """Check the status of the push in PAN-OS. + + The status is read from the context, since the human readable output of pan-os-push-status is + nearly empty while the job is still settling. + Args: + args (dict): The arguments of the flow. + responses (list): The responses of the command executions so far. + Returns: + A PollResult object. + """ + push_job_id = args["push_job_id"] + res_push_status = run_execute_command("pan-os-push-status", {"job_id": push_job_id}) + responses.append(res_push_status) + global POLLING, JOB_FAILURE_MESSAGE + if is_error(res_push_status): + POLLING = False + error_reason = get_error_message(res_push_status) + JOB_FAILURE_MESSAGE = ( + f"The PAN-OS push job {push_job_id} status could not be read, so the committed changes were " + f"not confirmed as applied and the URLs may not be blocked. PAN-OS reason: {error_reason}" + ) + demisto.debug(f"BU: the push status returned an error entry. {JOB_FAILURE_MESSAGE=}") + push_output = {"JobID": push_job_id, "Status": "Failure"} + return PollResult( + response=CommandResults( + outputs=push_output, + outputs_key_field="JobID", + readable_output=tableToMarkdown("Push to Device Group:", push_output, ["JobID", "Status"], removeNull=True), + ), + continue_to_poll=False, + ) + push_context = get_context_entry(res_push_status, "Panorama.Push") + push_status = _text(push_context.get("Status")) + push_failed = push_status in PUSH_FAILURE_STATUSES + # An empty status means the job is still settling, not that it succeeded. Treating it as done + # would report a full success for a push that never finished. The polling decorator timeout is + # the safety net for a status that never resolves. + continue_to_poll = not push_failed and push_status != PUSH_COMPLETED_STATUS + demisto.debug(f"BU: after pan-os-push-status {push_status=} {continue_to_poll=} {push_failed=}") + context_output = {"Status": push_status, "JobID": push_job_id} + push_cr = CommandResults( + outputs_key_field="JobID", + outputs=context_output, + readable_output=tableToMarkdown("Push to Device Group:", context_output, ["JobID", "Status"], removeNull=True), + ) + POLLING = continue_to_poll + if push_failed: + details = _text(push_context.get("Details")) or _text(push_context.get("Errors")) + JOB_FAILURE_MESSAGE = ( + f"The PAN-OS push job {push_job_id} failed with the status '{push_status}', so the committed changes " + f"were not applied to the managed firewalls and the URLs are not blocked. PAN-OS details: " + f"{details or 'none reported'}" + ) + demisto.debug(f"BU: the push job failed. {JOB_FAILURE_MESSAGE=}") + return PollResult( + response=push_cr, + continue_to_poll=continue_to_poll, + partial_result=CommandResults(readable_output=f"Waiting for Job-ID {push_job_id} to finish pushing the changes..."), + ) + + +""" MAIN FUNCTION """ + + +def main(): # pragma: no cover + try: + args = demisto.args() + demisto.debug(f"BU: the script block-url was called with the arguments {args=}") + url_list = argToList(args.get("url_list", [])) + brands_to_run = argToList(args.get("brands", ",".join(SUPPORTED_BRANDS))) + verbose = argToBoolean(args.get("verbose", False)) + modules = demisto.getModules() + enabled_brands = {module.get("brand") for module in modules.values() if module.get("state") == "active"} + demisto.debug(f"BU: the enabled modules are: {enabled_brands=}, {brands_to_run=}") + + executed_brands, updated_brands_to_run = update_brands_to_run(brands_to_run) + accepted_urls, rejected_urls = partition_urls(url_list) + + results: list = [] + if rejected_urls and not is_polling_resume(): + results.append(create_rejected_results(rejected_urls)) + + for brand in updated_brands_to_run: + demisto.debug(f"BU: the current brand is {brand}") + if brand not in enabled_brands: + results.append(CommandResults(readable_output=f"The brand {brand} isn't enabled.")) + executed_brands.append(brand) + continue + if brand != PAN_OS_BRAND: + return_error( + f"The brand {brand} isn't a part of the supported integrations for 'block-url'. " + f"The supported integrations are: {', '.join(SUPPORTED_BRANDS)}." + ) + if not accepted_urls: + results.append(CommandResults(readable_output="There are no valid URLs to block.")) + executed_brands.append(brand) + continue + brand_args = { + "url_list": [entry["SubmittedURL"] for entry in accepted_urls], + "url_entries": accepted_urls, + "rule_name": args.get("rule_name", "Cortex - Block URLs"), + "url_category": args.get("url_category", "Blocked URLs - Cortex"), + "url_filtering_profile": args.get("url_filtering_profile", "Cortex - Block URL profile"), + "log_forwarding_name": args.get("log_forwarding_name", ""), + "tag": args.get("tag", "cortex-blocked-urls"), + "auto_commit": argToBoolean(args.get("auto_commit", True)), + "verbose": verbose, + "brands": brands_to_run, + "commit_job_id": args.get("commit_job_id"), + "incident_id": demisto.incident().get("id", ""), + "polling": True, + } + results.append(PanOs(brand_args).manage_pan_os_flow()) + if not POLLING: + demisto.debug("BU: not in a polling mode, adding Panorama to the executed_brands.") + executed_brands.append(brand) + + if POLLING: + demisto.debug(f"BU: updating the executed_brands {executed_brands=}") + demisto.setContext("executed_brands", str(executed_brands)) + elif PAN_OS_BRAND in brands_to_run: + demisto.debug("BU: not in a polling mode, initializing the executed_brands.") + demisto.setContext("executed_brands", "") + return_results(results) + + except Exception as ex: + return_error(f"Failed to execute block-url. Error: {str(ex)}") + + +""" ENTRY POINT """ + + +if __name__ in ("__main__", "__builtin__", "builtins"): # pragma: no cover + main() diff --git a/Packs/AggregatedScripts/Scripts/BlockUrl/BlockUrl.yml b/Packs/AggregatedScripts/Scripts/BlockUrl/BlockUrl.yml new file mode 100644 index 000000000000..c306ab622775 --- /dev/null +++ b/Packs/AggregatedScripts/Scripts/BlockUrl/BlockUrl.yml @@ -0,0 +1,91 @@ +args: +- description: A comma-separated list of URLs to block. The scheme is stripped before the URL is submitted to the firewall. + isArray: true + name: url_list + required: true +- description: |- + Which integration brands to run the command for. If not provided, the command will run for all available integrations. + For multi-select provide a comma-separated list. + isArray: true + name: brands + auto: PREDEFINED + predefined: + - 'Panorama' +- defaultValue: 'Cortex - Block URLs' + description: The name of the security rule which will be created in the relevant products. + name: rule_name +- defaultValue: 'Blocked URLs - Cortex' + description: The name of the PAN-OS custom URL category which holds the blocked URLs. + name: url_category +- defaultValue: 'Cortex - Block URL profile' + description: The name of the PAN-OS URL Filtering security profile which blocks the custom URL category. + name: url_filtering_profile +- description: The Panorama log forwarding object name. Indicates what type of Log Forwarding setting will be specified in the PAN-OS custom rules. + name: log_forwarding_name +- defaultValue: 'cortex-blocked-urls' + description: The designated tag name for the objects the script creates. + name: tag +- description: Whether to commit the new rule. + name: auto_commit + defaultValue: 'true' + auto: PREDEFINED + predefined: + - 'true' + - 'false' +- description: Whether to retrieve a human-readable entry for every command or only the final result. True retrieves a human-readable entry for every command. False retrieves a human-readable entry only for the final result. + name: verbose + defaultValue: 'false' + auto: PREDEFINED + predefined: + - 'true' + - 'false' +- description: The commit job ID to use in polling commands (automatically filled by polling). + name: commit_job_id + hidden: true +- description: The push job ID to use in polling commands (automatically filled by polling). + name: push_job_id + hidden: true +comment: Blocks a list of URLs in supported integrations. +commonfields: + id: block-url + version: -1 +name: block-url +outputs: +- contextPath: BlockURLResults.URL + description: The URL that was requested to be blocked, as it was supplied. + type: String +- contextPath: BlockURLResults.SubmittedURL + description: The normalized URL that was submitted to the brand. Empty when the URL was rejected before submission. + type: String +- contextPath: BlockURLResults.Brand + description: The name of the brand that was executed. + type: String +- contextPath: BlockURLResults.Result + description: The result of the action, Success, Failed or Skipped. + type: String +- contextPath: BlockURLResults.Message + description: A message concerning the result of the action. + type: String +- contextPath: BlockURLResults.RuleName + description: The name of the security rule that blocks the URL. + type: String +- contextPath: BlockURLResults.URLCategory + description: The name of the custom URL category that holds the URL. + type: String +- contextPath: BlockURLResults.JobID + description: The ID of the PAN-OS commit job. + type: String +script: '-' +system: false +timeout: 20m0s +type: python +subtype: python3 +dockerimage: demisto/python3:3.12.14.13495286 +fromversion: 6.10.0 +marketplaces: +- xsoar +- marketplacev2 +- platform +polling: true +tests: +- No tests (auto formatted) diff --git a/Packs/AggregatedScripts/Scripts/BlockUrl/BlockUrl_test.py b/Packs/AggregatedScripts/Scripts/BlockUrl/BlockUrl_test.py new file mode 100644 index 000000000000..9654d10568ba --- /dev/null +++ b/Packs/AggregatedScripts/Scripts/BlockUrl/BlockUrl_test.py @@ -0,0 +1,1375 @@ +import json +import os + +import pytest + +import BlockUrl +import demistomock as demisto +from BlockUrl import BlockUrlError, PanOs + +TEST_DATA_DIR = os.path.join(os.path.dirname(os.path.abspath(__file__)), "test_data") + + +def util_load_json(path): + with open(os.path.join(TEST_DATA_DIR, path), encoding="utf-8") as f: + return json.loads(f.read()) + + +RESPONSES = util_load_json("pan_os_responses.json") + + +def build_pan_os(**overrides) -> PanOs: + """Build a PanOs instance with the script defaults, overridden as the test needs.""" + args = { + "url_entries": [{"URL": "http://new.example.com", "SubmittedURL": "new.example.com"}], + "rule_name": "Cortex - Block URLs", + "url_category": "Blocked URLs - Cortex", + "url_filtering_profile": "Cortex - Block URL profile", + "log_forwarding_name": "", + "tag": "cortex-blocked-urls", + "auto_commit": True, + "verbose": False, + "incident_id": "1", + } + args.update(overrides) + pan_os = PanOs(args) + pan_os.is_panorama = overrides.pop("is_panorama", True) + pan_os.device_group = overrides.pop("device_group", "Test-Device-Group") + return pan_os + + +def executed_commands(mock) -> list[tuple[str, dict]]: + """The (command name, args) pairs that were executed, in order.""" + return [(call.args[0], call.args[1]) for call in mock.call_args_list] + + +def successful_flow_responses() -> list: + """The responses of a full start_pan_os_flow run on a Panorama holding none of the objects yet.""" + return [ + RESPONSES["system_info_panorama"], + RESPONSES["object_not_found"], + RESPONSES["category_created"], + RESPONSES["object_not_found"], + RESPONSES["url_filter_created"], + RESPONSES["tag_created"], + RESPONSES["list_rules_empty"], + RESPONSES["rule_created"], + RESPONSES["profile_applied"], + RESPONSES["rule_moved"], + ] + + +""" URL NORMALIZATION """ + + +@pytest.mark.parametrize( + "raw_url, expected", + [ + ("http://example.com", "example.com"), + ("https://example.com", "example.com"), + ("HTTPS://Example.COM", "example.com"), + ("example.com/", "example.com"), + ("https://example.com/", "example.com"), + ("UPPER.example.com/Some/Path", "upper.example.com/Some/Path"), + (" example.com ", "example.com"), + ("example.com/some/path", "example.com/some/path"), + ("a" * 240 + ".example.com", "a" * 240 + ".example.com"), + ], +) +def test_normalize_url_accepts(raw_url, expected): + """ + Given: + - A URL with a scheme, a trailing slash, mixed case or surrounding whitespace. + When: + - Normalizing it before submitting it to PAN-OS. + Then: + - The scheme and the trailing slash are stripped, the host is lowercased, the path keeps its + case, and no rejection message is returned. + """ + from BlockUrl import normalize_url + + submitted_url, rejection_message = normalize_url(raw_url) + assert submitted_url == expected + assert rejection_message == "" + + +@pytest.mark.parametrize( + "raw_url, expected_message_part", + [ + ("", "empty"), + (" ", "empty"), + ("*.example.com", "Wildcards are not supported"), + ("https://*.example.com", "Wildcards are not supported"), + ("a" * 256 + ".example.com", "exceeds 255 chars"), + ("example.com,other.example.com", "comma"), + ("example .com", "whitespace"), + ("ftp://example.com", "scheme is not supported"), + ("example.com", "not supported"), + ("exa&mple.com", "not supported"), + ('exa"mple.com', "not supported"), + ("exa'mple.com", "not supported"), + ("example.com/