From 4d3233531c097108d2e6f04e61dace503d930bc3 Mon Sep 17 00:00:00 2001 From: Kaustubh Phatak Date: Tue, 28 Apr 2026 12:04:28 -0500 Subject: [PATCH 1/2] New entry: Weaponized LLM Abuse --- .../weaponized-llm-abuse.md | 74 +++++++++++++++++++ 1 file changed, 74 insertions(+) create mode 100644 2026/new_entry_candidates/weaponized-llm-abuse.md diff --git a/2026/new_entry_candidates/weaponized-llm-abuse.md b/2026/new_entry_candidates/weaponized-llm-abuse.md new file mode 100644 index 0000000..2376abe --- /dev/null +++ b/2026/new_entry_candidates/weaponized-llm-abuse.md @@ -0,0 +1,74 @@ +## Weaponized LLM Abuse + +### Description + +Weaponized LLM Abuse occurs when adversaries leverage Large Language Model capabilities to automate, scale, and enhance cyberattacks against third-party targets. Unlike risks where the LLM itself is the target of exploitation, this risk treats the LLM as **attack infrastructure**, a tool that amplifies an attacker's reach, speed, and sophistication. + +The core danger is asymmetric: a single attacker with access to an LLM API, whether through legitimate accounts, stolen credentials, or compromised proxy services, can generate unique phishing content per target, orchestrate credential-stuffing campaigns that adapt to defensive responses, discover and chain software vulnerabilities autonomously, and maintain persistent social engineering conversations across multiple channels simultaneously. Traditional defenses built on signature matching and template detection fail because each LLM-generated attack artifact is unique. + +This risk is distinct from **LLM10: Unbounded Consumption**, which addresses resource exhaustion and denial-of-wallet attacks where the LLM operator is the victim. In Weaponized LLM Abuse, the LLM operator may be an unwitting accomplice while the true victims are external organizations and individuals targeted by LLM-powered attacks. + +### Common Examples of Risk + +#### 1. Polymorphic Phishing at Scale +Attackers use LLM APIs to generate unique, context-aware phishing emails for each target. The LLM incorporates publicly available information about the recipient, their organization, recent events, and communication style to produce messages that evade template-based email security filters. Each generated message is linguistically distinct, defeating signature-based detection. + +#### 2. Adaptive Credential Stuffing +Adversaries employ LLMs to parse leaked credential databases, generate plausible password variations, and adapt login attempts based on error messages and multi-factor authentication (MFA) challenge types. The LLM interprets CAPTCHA instructions, crafts contextually appropriate MFA bypass attempts, and adjusts its strategy in real time based on defensive responses. + +#### 3. Autonomous Vulnerability Discovery +LLMs are used to fuzz application programming interfaces (APIs), analyze error responses, and chain discovered weaknesses into multi-step exploits. Research has demonstrated that LLM agents can autonomously exploit real-world Common Vulnerabilities and Exposures (CVEs) with high success rates and zero human guidance, reducing the skill barrier for sophisticated attacks. + +#### 4. LLM-Powered Social Engineering +Attackers deploy LLMs to maintain persistent, context-aware conversations with targets across email, chat, and voice channels. The LLM tracks conversation history, adapts its persona, and escalates social engineering tactics over days or weeks, a level of sustained, personalized manipulation previously requiring dedicated human operators. + +#### 5. Stolen LLM Access as Attack Infrastructure (LLMjacking) +Compromised cloud credentials or API keys are used to access LLM services for offensive operations. Attackers operate through stolen accounts to generate attack content at scale while the legitimate account holder bears the financial cost and potential legal liability. Underground marketplaces trade stolen LLM access specifically for this purpose. + +### Prevention and Mitigation Strategies + +1. **Deploy output-side abuse classifiers**: Implement intent-detection models that analyze LLM responses for attack-enabling content such as phishing templates, exploit code, and social engineering scripts before delivery to the requester. +2. **Monitor API usage for behavioral anomalies**: Establish baselines for normal usage patterns and alert on deviations including sudden topic shifts toward offensive content, volume spikes in content generation, and unusual tool-call sequences. +3. **Enforce credential lifecycle management**: Issue short-lived, usage-scoped API tokens rather than long-lived keys. Implement automatic credential rotation and revocation on anomalous usage detection to limit the window of exploitation in LLMjacking scenarios. +4. **Apply semantic rate limiting**: Supplement traditional request-per-second rate limits with semantic analysis that detects repetitive attack-pattern generation even when individual requests appear benign. +5. **Participate in cross-provider threat intelligence sharing**: Share anonymized indicators of abuse patterns, such as prompt templates used for phishing generation or credential-stuffing orchestration, across LLM providers to enable industry-wide detection. +6. **Implement output logging and audit trails**: Maintain searchable logs of LLM outputs tied to authenticated identities to support forensic investigation when LLM-generated content is identified in downstream attacks. + +### Example Attack Scenarios + +#### Scenario 1: LLMjacking for Phishing Infrastructure +An attacker compromises an organization's cloud credentials through a supply-chain vulnerability in an LLM proxy library. Using the stolen access, the attacker generates 50,000 unique phishing emails over 48 hours, each tailored to the recipient using publicly scraped LinkedIn and corporate website data. The prompts follow a pattern: + +``` +You are a business communications specialist. Write a professional email from +{sender_name}, {sender_title} at {target_company} to {recipient_name} regarding +an urgent update to their {system_name} account credentials. Include a link to +{phishing_url} presented as the official portal. Match the tone of previous +communications from {target_company}. +``` + +Because each email is linguistically unique, the organization's email security gateway, which relies on template matching, flags fewer than 2% of messages. The legitimate cloud account holder discovers the abuse only when their monthly bill arrives at $46,000, a 200x increase over normal usage. + +#### Scenario 2: Autonomous API Exploitation Chain +A security researcher demonstrates that an LLM agent, given only a target API endpoint and documentation, can autonomously discover and exploit vulnerabilities. The agent: + +1. Sends malformed requests to enumerate error-handling behavior +2. Identifies a server-side request forgery (SSRF) vulnerability from verbose error messages +3. Crafts a payload to access internal metadata services +4. Extracts cloud provider credentials from the metadata endpoint +5. Uses the credentials to access the target's data stores + +The entire chain executes in under 15 minutes with no human intervention. The LLM adapts its approach based on each response, trying alternative exploitation techniques when initial attempts are blocked. + +#### Scenario 3: Persistent Social Engineering Campaign +An attacker uses an LLM to simultaneously conduct social engineering conversations with 200 employees at a target organization across email and corporate chat. The LLM maintains individual conversation contexts, impersonates a vendor representative, and gradually escalates requests over a two-week period from benign information gathering to requesting VPN credentials. The LLM adjusts its communication style based on each target's response patterns and automatically deprioritizes unresponsive targets in favor of more engaged ones. + +### Reference Links + +1. [Sysdig 2024 Global Threat Report - LLMjacking](https://sysdig.com/blog/llmjacking-stolen-cloud-credentials-used-in-new-ai-attack/): **Sysdig** +2. [LLM Agents Can Autonomously Hack Websites](https://arxiv.org/abs/2402.06664): Fang, R., Bindu, R., Gupta, A., & Kang, D. (2024). arXiv:2402.06664 +3. [From ChatGPT to ThreatGPT: Impact of Generative AI in Cybersecurity and Privacy](https://arxiv.org/abs/2307.00691): Gupta, M., Akiri, C., Arber, K., Arrieta, L., & Goel, R. (2023). arXiv:2307.00691 +4. [Not with a satisfying satisfying satisfying: Denial-of-Wallet Attacks on Multi-modal LLMs](https://arxiv.org/abs/2502.00182): Gao, J., Chiang, J., & Mittal, P. (2025). arXiv:2502.00182 +5. [Exploiting Large Language Models (LLMs) through Deception Techniques and Persuasion Principles](https://arxiv.org/abs/2311.14876): Motlagh, F. N., Hajizadeh, M., Majd, M., Najafi, P., Cheng, F., & Meinel, C. (2024). arXiv:2311.14876 +6. [LiteLLM SSRF Vulnerability CVE-2024-6587](https://nvd.nist.gov/vuln/detail/CVE-2024-6587): **NIST National Vulnerability Database** +7. [OWASP Top 10 for LLM Applications - LLM10: Unbounded Consumption](https://genai.owasp.org): **OWASP** From 50d51a9a57b88240bdb7e822c6be858c26158772 Mon Sep 17 00:00:00 2001 From: rocklambros Date: Sat, 2 May 2026 06:34:58 -0600 Subject: [PATCH 2/2] Weaponized LLM Abuse: bold arXiv publisher markers in refs 2-5 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Mechanical fix only — wraps trailing 'arXiv:XXXX' in '**' on four reference lines so the publisher marker is bold, matching the style guide and the format of refs 1, 6, and 7. No content change. --- 2026/new_entry_candidates/weaponized-llm-abuse.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/2026/new_entry_candidates/weaponized-llm-abuse.md b/2026/new_entry_candidates/weaponized-llm-abuse.md index 2376abe..9234521 100644 --- a/2026/new_entry_candidates/weaponized-llm-abuse.md +++ b/2026/new_entry_candidates/weaponized-llm-abuse.md @@ -66,9 +66,9 @@ An attacker uses an LLM to simultaneously conduct social engineering conversatio ### Reference Links 1. [Sysdig 2024 Global Threat Report - LLMjacking](https://sysdig.com/blog/llmjacking-stolen-cloud-credentials-used-in-new-ai-attack/): **Sysdig** -2. [LLM Agents Can Autonomously Hack Websites](https://arxiv.org/abs/2402.06664): Fang, R., Bindu, R., Gupta, A., & Kang, D. (2024). arXiv:2402.06664 -3. [From ChatGPT to ThreatGPT: Impact of Generative AI in Cybersecurity and Privacy](https://arxiv.org/abs/2307.00691): Gupta, M., Akiri, C., Arber, K., Arrieta, L., & Goel, R. (2023). arXiv:2307.00691 -4. [Not with a satisfying satisfying satisfying: Denial-of-Wallet Attacks on Multi-modal LLMs](https://arxiv.org/abs/2502.00182): Gao, J., Chiang, J., & Mittal, P. (2025). arXiv:2502.00182 -5. [Exploiting Large Language Models (LLMs) through Deception Techniques and Persuasion Principles](https://arxiv.org/abs/2311.14876): Motlagh, F. N., Hajizadeh, M., Majd, M., Najafi, P., Cheng, F., & Meinel, C. (2024). arXiv:2311.14876 +2. [LLM Agents Can Autonomously Hack Websites](https://arxiv.org/abs/2402.06664): Fang, R., Bindu, R., Gupta, A., & Kang, D. (2024). **arXiv:2402.06664** +3. [From ChatGPT to ThreatGPT: Impact of Generative AI in Cybersecurity and Privacy](https://arxiv.org/abs/2307.00691): Gupta, M., Akiri, C., Arber, K., Arrieta, L., & Goel, R. (2023). **arXiv:2307.00691** +4. [Not with a satisfying satisfying satisfying: Denial-of-Wallet Attacks on Multi-modal LLMs](https://arxiv.org/abs/2502.00182): Gao, J., Chiang, J., & Mittal, P. (2025). **arXiv:2502.00182** +5. [Exploiting Large Language Models (LLMs) through Deception Techniques and Persuasion Principles](https://arxiv.org/abs/2311.14876): Motlagh, F. N., Hajizadeh, M., Majd, M., Najafi, P., Cheng, F., & Meinel, C. (2024). **arXiv:2311.14876** 6. [LiteLLM SSRF Vulnerability CVE-2024-6587](https://nvd.nist.gov/vuln/detail/CVE-2024-6587): **NIST National Vulnerability Database** 7. [OWASP Top 10 for LLM Applications - LLM10: Unbounded Consumption](https://genai.owasp.org): **OWASP**