Pap Smear Processing Application

A comprehensive AI-powered system for automated cervical cytology image analysis and pap smear classification with explainable AI capabilities.

🎯 Overview

This application provides a complete pipeline for:

Automated Pap Smear Classification: Multi-class cell classification using deep learning
Explainable AI: Grad-CAM heatmaps for model interpretability
Multiple Instance Learning: Slide-level analysis from patch-level predictions
REST API: Production-ready FastAPI service with comprehensive endpoints
Real-time Inference: Fast prediction with confidence scoring and uncertainty estimation

🏥 Medical Classes

The system classifies cervical cytology images into 8 categories:

Superficial Squamous - Normal superficial cells
Intermediate Squamous - Normal intermediate cells
Columnar Epithelial - Normal columnar cells
Mild Squamous Non-keratinizing Dysplasia - Low-grade abnormality
Moderate Squamous Non-keratinizing Dysplasia - Moderate abnormality
Severe Squamous Non-keratinizing Dysplasia - High-grade abnormality
Squamous Cell Carcinoma Keratinizing Type - Malignant keratinizing
Squamous Cell Carcinoma Non-keratinizing Type - Malignant non-keratinizing

🚀 Quick Start

Prerequisites

Python 3.10+
CUDA-capable GPU (optional, for training and faster inference)
Docker and Docker Compose (for containerized deployment)
8GB+ RAM recommended

Installation

Clone the repository

git clone https://github.com/yourusername/pap-smear-processing.git
cd pap-smear-processing

Create virtual environment

python -m venv venv

# On Windows
venv\Scripts\activate

# On macOS/Linux
source venv/bin/activate

Install dependencies

pip install -r requirements.txt

Download SIPaKMeD dataset (only needed for training new models)

python -m data.download

Note: End users who just want to analyze images do NOT need to download the dataset. The dataset is only required if you want to:

Train new models from scratch
Retrain existing models
Experiment with different architectures

For image analysis using pre-trained models, skip this step.

🐳 Docker Deployment (Recommended)

CPU-only deployment:

docker-compose up pap-smear-api

GPU-enabled deployment:

docker-compose --profile gpu up pap-smear-api-gpu

Development mode:

docker-compose --profile development up pap-smear-dev

Full production stack with monitoring:

docker-compose --profile production --profile monitoring up

🔧 Local Development

Option 1: Desktop GUI Application (Easiest for Non-Technical Users)

# Launch the user-friendly GUI
python launch_gui.py

This opens a desktop application with:

Simple drag-and-drop image upload
Real-time processing with progress indicators
Clear visualization of results
No technical knowledge required

Option 2: Web Interface (Recommended for Developers)

python main.py serve

Then open your browser to http://localhost:8000 for the interactive web interface.

Option 3: API Server

python -m uvicorn api.app:app --reload --host 0.0.0.0 --port 8000

Access the API:

API Documentation: http://localhost:8000/docs
Health Check: http://localhost:8000/api/v1/health
Model Info: http://localhost:8000/api/v1/model/info

📚 Usage Examples

Python API Client

import requests
from pathlib import Path

# Single image prediction
with open('path/to/image.jpg', 'rb') as f:
    response = requests.post(
        'http://localhost:8000/api/v1/predict',
        files={'file': f},
        json={
            'return_probabilities': True,
            'return_heatmap': True,
            'confidence_threshold': 0.7
        }
    )

result = response.json()
print(f"Prediction: {result['prediction']['class_name']}")
print(f"Confidence: {result['prediction']['confidence']:.3f}")

cURL Examples

Health check:

curl -X GET "http://localhost:8000/api/v1/health"

Single prediction:

curl -X POST "http://localhost:8000/api/v1/predict" \
  -H "Content-Type: multipart/form-data" \
  -F "file=@image.jpg" \
  -F 'request={"return_probabilities": true, "return_heatmap": true}'

Batch prediction:

curl -X POST "http://localhost:8000/api/v1/predict/batch" \
  -H "Content-Type: multipart/form-data" \
  -F "files=@image1.jpg" \
  -F "files=@image2.jpg" \
  -F 'request={"prediction_params": {"return_probabilities": true}}'

🏗️ Architecture

Project Structure

Pap Smeet Processing app/
├── api/                    # FastAPI application
│   ├── app.py             # Main FastAPI app
│   ├── routes.py          # API endpoints
│   ├── models.py          # Pydantic models
│   └── utils.py           # API utilities
├── data/                   # Data processing
│   ├── download.py        # Dataset downloader
│   ├── preprocessing.py   # Image preprocessing
│   └── dataset.py         # PyTorch datasets
├── models/                 # Model architectures
│   ├── classifier.py      # Patch classifier
│   ├── mil.py            # MIL models
│   ├── trainer.py        # Training utilities
│   └── utils.py          # Model utilities
├── inference/             # Inference and explainability
│   ├── predictor.py      # Prediction classes
│   ├── explainability.py # Grad-CAM implementation
│   └── utils.py          # Inference utilities
├── utils/                 # Shared utilities
│   ├── logging.py        # Logging setup
│   ├── metrics.py        # Evaluation metrics
│   ├── visualization.py  # Plotting utilities
│   └── io.py             # File I/O utilities
├── config.py              # Configuration
├── requirements.txt       # Dependencies
├── Dockerfile            # Docker configuration
└── docker-compose.yml    # Docker Compose setup

Key Components

Data Pipeline: Automated SIPaKMeD dataset download, preprocessing, and augmentation
Model Training: Transfer learning with ResNet/EfficientNet backbones
Inference Engine: Fast prediction with batch processing support
Explainability: Grad-CAM heatmaps for model interpretability
MIL Pipeline: Multiple Instance Learning for slide-level classification
REST API: Production-ready FastAPI service
Monitoring: Comprehensive logging and experiment tracking

🔬 Training Your Own Model

Dataset Preparation

Download SIPaKMeD dataset:

python -m data.download --force

Create data splits:

python -c "
from data.dataset import create_data_splits
from config import data_config
create_data_splits(
    data_config.raw_data_path / 'sipakmed',
    data_config.processed_data_path / 'splits'
)"

Model Training

from models.trainer import ModelTrainer
from models.classifier import create_model
from data.dataset import create_dataloaders

# Create model
model = create_model("patch_classifier")

# Create data loaders
dataloaders = create_dataloaders(
    data_config.processed_data_path / 'splits',
    batch_size=32
)

# Train model
trainer = ModelTrainer(
    model=model,
    train_loader=dataloaders['train'],
    val_loader=dataloaders['val']
)

history = trainer.train()

Training with Docker

# GPU training
docker-compose --profile training up pap-smear-trainer

📊 API Endpoints

Core Endpoints

Endpoint	Method	Description
`/api/v1/health`	GET	Health check and system status
`/api/v1/predict`	POST	Single image prediction
`/api/v1/predict/batch`	POST	Batch image prediction
`/api/v1/model/info`	GET	Model information
`/api/v1/statistics`	GET	API usage statistics
`/api/v1/feedback`	POST	Submit prediction feedback

Admin Endpoints

Endpoint	Method	Description
`/api/v1/admin/load_model`	POST	Load new model
`/api/v1/admin/models`	GET	List loaded models

Response Format

{
  "success": true,
  "prediction": {
    "class_index": 0,
    "class_name": "superficial_squamous",
    "probability": 0.85,
    "confidence": 0.85
  },
  "probabilities": {
    "superficial_squamous": 0.85,
    "intermediate_squamous": 0.10,
    "columnar_epithelial": 0.05
  },
  "metrics": {
    "confidence": 0.85,
    "uncertainty": 0.15,
    "confidence_level": "high",
    "reliable_prediction": true
  },
  "processing_info": {
    "inference_time_ms": 150.5,
    "image_size": [512, 512]
  }
}

🔧 Configuration

The application uses a centralized configuration system in config.py:

# Model configuration
model_config.backbone = "resnet50"
model_config.num_classes = 8
model_config.batch_size = 32
model_config.learning_rate = 1e-4

# API configuration
api_config.host = "0.0.0.0"
api_config.port = 8000
api_config.max_file_size = 50 * 1024 * 1024  # 50MB

# Logging configuration
logging_config.use_wandb = True
logging_config.wandb_project = "pap-smear-classification"

Environment Variables

# API Configuration
API_PORT=8000
API_HOST=0.0.0.0

# Weights & Biases
WANDB_PROJECT=pap-smear-classification
WANDB_ENTITY=your-username

# Logging
LOG_LEVEL=INFO

📈 Monitoring and Logging

Weights & Biases Integration

The application integrates with Weights & Biases for experiment tracking:

from utils.logging import create_experiment_logger

# Create experiment logger
exp_logger = create_experiment_logger(
    "training_experiment",
    config={"learning_rate": 0.001, "batch_size": 32}
)

# Log metrics
exp_logger.log_metrics({"accuracy": 0.95, "loss": 0.05}, step=1)

# Log artifacts
exp_logger.log_artifact(model_path, "model")

Prometheus Metrics

When using the monitoring profile, Prometheus metrics are available at:

Prometheus: http://localhost:9090
Grafana: http://localhost:3000 (admin/admin123)

🧪 Testing

Unit Tests

pytest tests/ -v --cov=.

API Testing

# Test health endpoint
curl http://localhost:8000/api/v1/health

# Test prediction with sample image
python scripts/test_api.py

Load Testing

# Install locust
pip install locust

# Run load test
locust -f tests/load_test.py --host=http://localhost:8000

🚀 Deployment

Production Deployment

Build production image:

docker build --target production -t pap-smear-api:latest .

Deploy with Docker Compose:

docker-compose --profile production up -d

Deploy to Kubernetes:

kubectl apply -f k8s/

Performance Optimization

Model Optimization: Convert to ONNX or TorchScript for faster inference
Caching: Redis integration for response caching
Load Balancing: Nginx reverse proxy with multiple API instances
GPU Acceleration: CUDA support for faster inference

🔒 Security Considerations

Input Validation: Comprehensive file type and size validation
Rate Limiting: Built-in request rate limiting
Authentication: JWT token support (configurable)
HTTPS: SSL/TLS support via Nginx
Container Security: Non-root user in production containers

📋 Requirements

System Requirements

CPU: 4+ cores recommended
RAM: 8GB minimum, 16GB recommended
GPU: CUDA-capable GPU for training (optional for inference)
Storage: 10GB+ for datasets and models

Python Dependencies

See requirements.txt for complete list. Key dependencies:

PyTorch: Deep learning framework
FastAPI: Web framework
OpenCV: Image processing
Albumentations: Data augmentation
Weights & Biases: Experiment tracking
Grad-CAM: Model explainability

🤝 Contributing

Fork the repository
Create a feature branch (git checkout -b feature/amazing-feature)
Commit your changes (git commit -m 'Add amazing feature')
Push to the branch (git push origin feature/amazing-feature)
Open a Pull Request

Development Setup

# Install development dependencies
pip install -r requirements-dev.txt

# Install pre-commit hooks
pre-commit install

# Run tests
pytest

# Run linting
flake8 .
black .
isort .

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

SIPaKMeD Dataset: Plissiti, M.E., Dimitrakopoulos, P., Sfikas, G., Nikou, C., Krikoni, O., Charchanti, A.: SIPAKMED: A New Database for Identification of Classes and Segmentation of Nuclei in Pap Smear Images. Computational and Mathematical Methods in Medicine 2018, 8797906:1-8797906:13 (2018)
PyTorch Team: For the excellent deep learning framework
FastAPI Team: For the modern web framework
Grad-CAM Authors: For explainable AI techniques

📞 Support

For questions, issues, or contributions:

Issues: GitHub Issues
Discussions: GitHub Discussions
Email: support@papsmear-ai.com

⚠️ Disclaimer

This software is for research and educational purposes only. It should not be used for clinical diagnosis without proper validation and expert oversight. Always consult qualified medical professionals for medical decisions.

Built with ❤️ for advancing medical AI research

Name		Name	Last commit message	Last commit date
Latest commit History 10 Commits
api		api
data		data
inference		inference
models		models
output		output
utils		utils
.dockerignore		.dockerignore
.gitignore		.gitignore
COLAB_TRAINING_GUIDE.md		COLAB_TRAINING_GUIDE.md
Dockerfile		Dockerfile
Dockerfile.railway		Dockerfile.railway
LICENSE		LICENSE
README.md		README.md
__init__.py		__init__.py
config.py		config.py
docker-compose.yml		docker-compose.yml
gui_app.py		gui_app.py
launch_gui.py		launch_gui.py
main.py		main.py
railway.toml		railway.toml
requirements.railway.txt		requirements.railway.txt
requirements.txt		requirements.txt

Folders and files

Latest commit

History

Repository files navigation