runzel · runzel · Dec 8, 2025
diff --git a/KEYWORD_RESEARCH_FEATURES.md b/KEYWORD_RESEARCH_FEATURES.md
@@ -0,0 +1,209 @@
+# Keyword Research & Secondary Keyword Analysis Features
+
+This document describes the new keyword research and secondary keyword analysis features integrated into ContentSwift, inspired by ALwrity's SEO capabilities.
+
+## Overview
+
+ContentSwift now includes comprehensive keyword research tools to help you:
+- Discover LSI (Latent Semantic Indexing) keywords
+- Find co-occurring keywords
+- Identify question-based keywords
+- Analyze keyword difficulty
+- Get related keyword suggestions
+- (Optional) Access keyword metrics like search volume and CPC
+
+## Features
+
+### 1. LSI Keywords
+**What**: Semantically related terms found in competitor content using TF-IDF analysis.
+**Why**: Using LSI keywords helps search engines understand your content better and can improve rankings.
+**How**: Automatically extracted from competitor articles, showing relevance scores.
+
+### 2. Co-occurring Keywords
+**What**: Terms that frequently appear near your primary keyword in competitor content.
+**Why**: These keywords provide context and natural language patterns that improve content quality.
+**How**: Analyzes a 50-word window around your primary keyword in all competitor content.
+
+### 3. Question Keywords
+**What**: Question-based search queries extracted from Google autocomplete.
+**Why**: Answers user questions and captures featured snippet opportunities.
+**How**: Automatically categorizes questions by intent (informational, instructional, analytical, etc.).
+
+### 4. Related Keywords
+**What**: Keywords from Google's "Related Searches" and "People Also Ask" sections.
+**Why**: Expands topical coverage and targets additional search queries.
+**How**: Pulled directly from SerpAPI search results.
+
+### 5. Keyword Difficulty Analysis
+**What**: Competitive analysis showing how frequently competitors use your target keyword.
+**Why**: Helps you understand the optimization level needed to compete.
+**How**: Calculates average keyword count, density, and provides an Easy/Medium/Hard rating.
+
+### 6. Keyword Metrics (Optional)
+**What**: Search volume, competition level, and CPC data from DataForSEO.
+**Why**: Make data-driven decisions about which keywords to target.
+**How**: Requires DataForSEO API credentials (see Configuration below).
+
+## How to Use
+
+### In the UI
+
+1. **Search for a keyword** on the home page
+2. **Select competitor articles** to analyze
+3. **Wait for scraping** to complete (Terms and Outline tabs)
+4. **Click the "Keywords" tab** in the sidebar
+5. **View comprehensive keyword analysis** including:
+   - Primary keyword difficulty rating
+   - LSI keywords with relevance scores
+   - Co-occurring keywords with frequencies
+   - Question-based keywords organized by intent
+   - Related keywords from Google
+
+### Via API
+
+#### Get Keyword Research
+```bash
+GET /keyword-research/{post_id}
+```
+
+Returns comprehensive keyword analysis for a post.
+
+**Example Response:**
+```json
+{
+  "status": "success",
+  "data": {
+    "primary_keyword": "content marketing",
+    "lsi_keywords": [
+      {
+        "keyword": "digital marketing",
+        "relevance_score": 0.8234,
+        "type": "lsi"
+      }
+    ],
+    "co_occurring_keywords": [
+      {
+        "keyword": "strategy",
+        "frequency": 45,
+        "type": "co_occurring"
+      }
+    ],
+    "question_keywords": [
+      {
+        "keyword": "what is content marketing",
+        "type": "question",
+        "intent": "informational"
+      }
+    ],
+    "related_keywords": [
+      {
+        "keyword": "content marketing strategy",
+        "type": "related_search"
+      }
+    ],
+    "keyword_difficulty": {
+      "keyword": "content marketing",
+      "difficulty": "Medium",
+      "avg_keyword_count": 5.2,
+      "avg_keyword_density": 1.3
+    }
+  }
+}
+```
+
+#### Analyze Specific Keywords
+```bash
+POST /keyword-analyze/
+Content-Type: application/json
+
+{
+  "keywords": ["content marketing", "SEO", "digital marketing"],
+  "location": "United States"
+}
+```
+
+Gets keyword metrics (requires DataForSEO API credentials).
+
+## Configuration
+
+### Required
+- `SERPAPI_KEY` - Already configured for basic Google search features
+
+### Optional (for keyword metrics)
+Add to your `.env` file:
+```
+DATAFORSEO_LOGIN=your_login
+DATAFORSEO_PASSWORD=your_password
+```
+
+**Note**: Keyword metrics from DataForSEO are optional. All other features work without these credentials.
+
+## Technical Implementation
+
+### Backend
+- **Service Module**: `backend-crt/src/services/keyword_research.py`
+- **API Endpoints**: Added to `backend-crt/src/main.py`
+- **Database**: New `keyword_research` JSON field in Post model
+- **Dependencies**: Added `scikit-learn` for TF-IDF analysis
+
+### Frontend
+- **Component**: `frontend-crt/src/components/KeywordResearch.tsx`
+- **Integration**: New "Keywords" tab in Sidebar component
+- **Styling**: Uses existing TailwindCSS theme
+
+### Algorithms Used
+
+1. **TF-IDF (Term Frequency-Inverse Document Frequency)**
+   - Identifies semantically important terms
+   - Scores keywords by relevance across competitor content
+
+2. **N-gram Analysis**
+   - Extracts unigrams, bigrams, and trigrams
+   - Captures multi-word keyword phrases
+
+3. **Co-occurrence Analysis**
+   - Window-based keyword proximity detection
+   - Frequency-weighted relevance scoring
+
+4. **Intent Classification**
+   - Categorizes question keywords by user intent
+   - Helps prioritize content structure
+
+## Benefits
+
+### For Content Writers
+- Discover related topics naturally
+- Answer user questions comprehensively
+- Write more semantically rich content
+
+### For SEO Specialists
+- Understand keyword difficulty before writing
+- Target featured snippet opportunities
+- Expand topical authority systematically
+
+### For Content Strategists
+- Identify content gaps
+- Prioritize high-value keywords
+- Plan content clusters effectively
+
+## Future Enhancements
+
+Potential improvements for future versions:
+- Keyword trend analysis over time
+- Competitor keyword gap analysis
+- Automated content brief generation
+- Keyword clustering and grouping
+- Search intent classification
+- Seasonal keyword patterns
+
+## Support
+
+For issues or questions about keyword research features:
+1. Check that competitor content has been scraped first
+2. Verify your SERPAPI_KEY is configured
+3. Review the browser console for any API errors
+4. Check backend logs for detailed error messages
+
+## Credits
+
+These features were inspired by ALwrity's comprehensive SEO and keyword research capabilities, adapted for ContentSwift's content optimization workflow.
diff --git a/README.md b/README.md
@@ -7,6 +7,19 @@ Watch [Demo Video at Youtube](https://www.youtube.com/watch?v=HL9sPYXd1Ws&t=115s
 
 Using this tool, you'll get relevant information regarding specific keyword searches and hints on what other top-ranking results did with their article/page.
 
+## ✨ New: Advanced Keyword Research Features
+
+ContentSwift now includes comprehensive keyword research and secondary keyword analysis capabilities:
+
+- **LSI Keywords** - Discover semantically related terms using TF-IDF analysis
+- **Co-occurring Keywords** - Find terms that frequently appear near your primary keyword
+- **Question Keywords** - Identify question-based search queries from Google autocomplete
+- **Related Keywords** - Get suggestions from "Related Searches" and "People Also Ask"
+- **Keyword Difficulty Analysis** - Understand competitive landscape before writing
+- **Keyword Metrics** (optional) - Access search volume and CPC data with DataForSEO integration
+
+See [KEYWORD_RESEARCH_FEATURES.md](./KEYWORD_RESEARCH_FEATURES.md) for detailed documentation.
+
 ![sample content optimization tool - ContentSwift](/assets/sample-content-optimization-tool.webp)
 
 > Right now we're focusing on Google SERP only

diff --git a/backend-crt/.env.example b/backend-crt/.env.example
@@ -0,0 +1,16 @@
+# ContentSwift Environment Variables
+
+# Required: SerpAPI for Google Search results
+SERPAPI_KEY=your_serpapi_key_here
+
+# Optional: DataForSEO for keyword metrics (search volume, CPC, difficulty)
+# Get credentials at: https://dataforseo.com/
+DATAFORSEO_LOGIN=your_dataforseo_login
+DATAFORSEO_PASSWORD=your_dataforseo_password
+
+# Database (configured in docker-compose.yml)
+# DB_HOST=db
+# DB_PORT=5432
+# DB_NAME=crtool
+# DB_USER=postgres
+# DB_PASSWORD=password
diff --git a/backend-crt/src/db/migrations/add_keyword_research.py b/backend-crt/src/db/migrations/add_keyword_research.py
@@ -0,0 +1,47 @@
+"""
+Database migration script to add keyword_research column to posts table
+Run this script to update existing database schema
+"""
+
+import sys
+import os
+
+# Add parent directory to path to import models
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.dirname(__file__))))
+
+from sqlalchemy import text
+from db.models import engine
+
+def migrate():
+    """Add keyword_research column to posts table if it doesn't exist"""
+
+    print("Starting migration: add_keyword_research")
+
+    with engine.connect() as conn:
+        # Check if column already exists
+        result = conn.execute(text("""
+            SELECT column_name
+            FROM information_schema.columns
+            WHERE table_name='post' AND column_name='keyword_research';
+        """))
+
+        if result.fetchone():
+            print("Column 'keyword_research' already exists. Skipping migration.")
+            return
+
+        # Add the column
+        print("Adding 'keyword_research' column to 'post' table...")
+        conn.execute(text("""
+            ALTER TABLE post
+            ADD COLUMN keyword_research JSON DEFAULT NULL;
+        """))
+        conn.commit()
+
+        print("Migration completed successfully!")
+
+if __name__ == "__main__":
+    try:
+        migrate()
+    except Exception as e:
+        print(f"Migration failed: {e}")
+        sys.exit(1)
diff --git a/backend-crt/src/db/models.py b/backend-crt/src/db/models.py
@@ -12,6 +12,7 @@ class Post(SQLModel, table=True):
     search_result: dict = Field(sa_column=Column(JSON))
     choosen_links: dict = Field(sa_column=Column(JSON))
     links_scrape_result: dict = Field(sa_column=Column(JSON))
+    keyword_research: dict = Field(default=None, sa_column=Column(JSON))
     createdAt: datetime = Field(default_factory=datetime.utcnow)
     updatedAt: datetime = Field(default_factory=datetime.utcnow)