FindDuplicate

FindDuplicate is a multi-language command-line tool designed to scan software projects and identify duplicate files and duplicated code blocks across multi-file codebases.

---

Overview & Architecture

FindDuplicate performs dual-level code analysis:

1. Exact File Matching: Cryptographic hashing (MD5 & SHA-256) to detect identical files across the directory tree.
2. Code Clone Detection: Tokenization and Abstract Syntax Tree (AST) token stream matching with rolling hash algorithms to detect duplicated code blocks.

Supported Languages

• Python (.py)
• JavaScript / TypeScript / React (.js, .jsx, .ts, .tsx)
• Dart / Flutter (.dart)

---

Project Structure

FindDuplicate/
├── find_duplicate/             # Core Package Source Code
│   ├── config/                 # Configuration & Ignore Pattern Managers
│   │   ├── default_config.py
│   │   └── settings.py
│   ├── core/                   # File Crawler & Cryptographic Hashing
│   │   ├── crawler.py
│   │   └── hasher.py
│   ├── detectors/              # Duplicate Detection Engines
│   │   ├── base.py
│   │   ├── exact_file_detector.py
│   │   └── code_clone_detector.py
│   ├── models/                 # Data Schemas & Report Structures
│   │   ├── file_info.py
│   │   └── report_data.py
│   ├── parsers/                # Language-Specific AST/Token Parsers
│   │   ├── base_parser.py
│   │   ├── python_parser.py
│   │   ├── js_ts_parser.py
│   │   └── dart_parser.py
│   └── reporters/              # Multi-Format Report Exporters
│       ├── cli_reporter.py     # Terminal Rich Output
│       ├── json_reporter.py    # Machine-Readable JSON Exporter
│       └── html_reporter.py    # Interactive Web HTML Exporter
├── output/                     # Automated Report Storage Directory
│   ├── duplicate_report.html
│   └── duplicate_report.json
├── templates/                  # Jinja2 HTML Report Templates
│   └── report_template.html
├── tests/                      # Unit & Integration Test Suites
├── .duplicateignore            # Ignore Rules Specification File
├── .gitignore                  # Git Ignore Patterns File
├── pyproject.toml              # Project Build & Entry Point Configuration
├── requirements.txt            # Python Dependencies Specification
└── README.md                   # Documentation File

---

Key Features

• Automated Ignore Respect: Automatically honors .gitignore and .duplicateignore to exclude third-party folders (such as node_modules, venv, dist, .git).
• Flexible Report Formats: Supports terminal table output, structured JSON export, and interactive side-by-side HTML diff views.
• Automated Directory Creation: Output reports are automatically organized into an output/ folder.

---

Installation & Setup

1. Clone Repository

git clone https://github.com/abinjoyal/FindDuplicate.git
cd FindDuplicate

2. Install Dependencies

pip install -r requirements.txt

(Optional) Install in editable mode for global CLI command availability:

pip install -e .

---

Usage Commands

1. Terminal CLI Scan (Default View)

Scan the current directory and display results directly in the terminal:

python -m find_duplicate --path .

Scan a specific project path:

python -m find_duplicate --path "C:\path\to\your\project"

2. Generate Interactive HTML Report

python -m find_duplicate --path . --format html

Outputs file to: output/duplicate_report.html

3. Generate Machine-Readable JSON Report

python -m find_duplicate --path . --format json

Outputs file to: output/duplicate_report.json

4. Custom Output Path

python -m find_duplicate --path . --format html --output "output/custom_report.html"

5. Minimum Duplicate Lines Threshold

Specify minimum duplicate lines to match (e.g., minimum 10 lines):

python -m find_duplicate --path . --min-lines 10

---

Viewing Reports

Viewing HTML Report in Browser

Method 1: Local HTTP Web Server (Recommended)

python -m http.server 8000

Open your browser at: http://localhost:8000/output/duplicate_report.html

Method 2: Open File Directly (Windows PowerShell)

Start-Process output/duplicate_report.html

---

Report Formats Explanation

Report Format	Output Location	Description
CLI Table	Terminal Screen	Rich formatted table providing immediate scan overview and line matches.
HTML Report	output/duplicate_report.html	Interactive web dashboard featuring side-by-side diff comparisons, search filtering, and code metrics.
JSON Report	output/duplicate_report.json	Complete structured JSON dataset ideal for CI/CD automation and code quality tools.

---

Testing

Run unit tests using pytest:

pytest

---

License

MIT License