Smart Log Monitor 🚨

AI-Assisted Cloud Log Monitoring & Threat Triage Pipeline

---

📖 Overview

Smart Log Monitor is a hybrid security pipeline that ingests AWS CloudWatch logs, reduces noise using semantic deduplication, and performs AI-assisted threat triage.

It is designed to simulate a Security Operations Center (SOC) workflow with focus on:

• Log ingestion
• Noise reduction
• Threat detection
• Automated triage

---

🧠 System Architecture

This project consists of two integrated layers:

☁️ Cloud Detection Layer

• CloudTrail → EventBridge → Lambda
• Detects suspicious AWS API activity
• Enriches logs with OSINT data

🖥️ Local Processing Layer

• Fetches logs from CloudWatch

• Deduplicates using embeddings

• Routes anomalies to AI agents

  AWS CloudWatch ───> Python Bridge (Boto3) ───> Semantic Cache (Local) (fetcher.py) (Deduplication) │ ▼ Alerting/Output <─── CrewAI Agents (LLMs) <─── Anomaly Router (Security Triage) (agent_router.py)

See architecture.md for details.

---

⚙️ Core Components

🔹 fetcher.py

• Pulls logs from CloudWatch
• Maintains checkpoint state

🔹 agent_router.py

• Routes anomalies to AI agents
• Uses CrewAI + LLM APIs

🔹 db_manager.py

• Stores processed results
• Enables audit/log persistence

🔹 state/

• Stores runtime state (checkpoint + semantic cache)
• Example files provided for reference

---

📂 Project Structure

smart-log-monitor/
├── src/
│   ├── main.py
│   ├── fetcher.py
│   ├── agent_router.py
│   ├── db_manager.py
│   └── state/
│       ├── checkpoint.example.txt
│       └── semantic_cache.example.json
│
├── cloud/
│   └── lambda_function.py
│
├── screenshots/
├── sample_logs/
├── requirements.txt
├── architecture.md
├── README.md

---

🚀 Quick Start

1. Clone

git clone https://github.com/yourusername/smart-log-monitor.git
cd smart-log-monitor

---

2. Setup Environment

python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt

---

3. Configure

Create .env from .env.example:

AWS_ACCESS_KEY_ID=
AWS_SECRET_ACCESS_KEY=
AWS_REGION=ap-south-1
LOG_GROUP_NAME=

GEMINI_API_KEY=

---

4. Run

python src/main.py

---

📊 Expected Output

[SYSTEM] Checkpoint loaded. Last sync: 2026-04-22 21:00:00
[BRIDGE] 150 new log events ingested from CloudWatch.
[CACHE]  Semantic check: 142 logs match existing patterns. Suppressing noise...
[ROUTER] 8 unique anomalies detected. Routing to AI Agents...

[AI TRIAGE REPORT]
---------------------------------------------------------
ALERT: Potential Unauthorized IAM Modification
SEVERITY: High
REASONING: Logs show 'UpdateAssumeRolePolicy' followed by a 'CreateAccessKey' 
           from an unrecognized IP. This matches a backdoor creation pattern.
ACTION: Quarantine IAM User and Rotate Keys.
---------------------------------------------------------

---

👤 Author

Bala Focus: Cloud Security | SOC Engineering | Threat Detection

---