Skip to content

testing_overview.md

Latest commit

 

History

History
208 lines (151 loc) · 7.1 KB

testing_overview.md

File metadata and controls

208 lines (151 loc) · 7.1 KB
Raw

VideoAnnotator Testing Overview

Current Test Structure 20 AUG 2025

VideoAnnotator uses a 3-tier organized test system designed for efficient development and comprehensive validation. All tests are located in the tests/ directory with the following structure:

tests/
├── unit/                     # Fast, isolated tests (<30 seconds)
│   ├── batch/               # Batch processing components (5 files)
│   ├── storage/             # File backends and validation (1 file)
│   └── utils/               # Utility functions (2 files)
├── integration/             # Cross-component tests (8 files)
│   ├── test_batch_orchestration.py
│   ├── test_simple_workflows.py
│   └── ...
├── pipelines/               # Full pipeline tests (12 files)
│   ├── test_person_tracking.py
│   ├── test_face_analysis.py
│   ├── test_audio_pipeline.py
│   └── ...
└── scripts/                 # Test execution scripts
    ├── test_fast.py         # Unit tests only
    ├── test_integration.py  # Unit + Integration
    ├── test_pipelines.py    # Pipeline tests
    └── test_all.py          # Complete suite

Test Execution Workflows

Development Workflow (Recommended)

# Fast feedback during development
python scripts/test_fast.py                # ~30 seconds, 125+ tests

# Pre-commit validation
python scripts/test_integration.py         # ~5 minutes

# Full validation
python scripts/test_all.py                 # Complete suite with reporting

Pipeline-Specific Testing

# Test specific pipelines
pytest tests/pipelines/test_person_tracking.py -v
pytest tests/pipelines/test_face_analysis.py -v

# Enable integration tests (requires models)
TEST_INTEGRATION=1 pytest tests/pipelines/ -v

Advanced Usage

# Test by category
pytest -m unit                             # Unit tests only
pytest -m integration                      # Integration tests only
pytest -m pipeline                         # Pipeline tests only

# Performance testing
pytest -m performance --benchmark-only     # Benchmarking tests

Current Test Coverage

Pipeline Coverage Status

Pipeline Tests Status Coverage
Person Tracking 9/9 ✅ 100% YOLO11, ByteTrack, pose estimation
Face Analysis 14/15 ✅ 93.3% DeepFace, emotions, age, gender
Audio Processing 5 files ✅ Available Whisper, LAION, speech recognition
Scene Detection Available ✅ Ready PySceneDetect + CLIP
OpenFace3 Comprehensive ✅ Complete Full facial behavior analysis
LAION Face/Voice Available ✅ Ready Advanced AI model testing

Component Coverage

Component Tests Status Notes
Batch Processing 5 files ✅ 100% Orchestrator, recovery, progress tracking
Storage Backend 1 file ✅ Complete File storage validation
Utils/Analysis 2 files ⚠️ 90% Size analysis needs attention

Test Execution Statistics

  • Total Tests: ~125 unit tests + integration + pipeline tests
  • Fast Execution: 104/125 unit tests pass in <30 seconds
  • Success Rate: 83.2% (stable and consistent)
  • Integration Tests: Real model execution (YOLO11, DeepFace, Whisper)

Configuration & Environment

Environment Variables

# Enable integration tests with real models
export TEST_INTEGRATION=1

# Enable DeepFace-specific tests
export TEST_DEEPFACE=1

Pytest Configuration

The test suite uses pytest with custom markers defined in pyproject.toml:

  • unit: Fast, isolated tests
  • integration: Cross-component tests
  • pipeline: Full pipeline tests
  • performance: Benchmarking tests
  • gpu: GPU-accelerated tests
  • real_models: Tests using real ML models

Current Issues & Limitations

Known Issues

  1. Size Analysis Integration Test: One failing test (functionality may be incomplete)
  2. Windows File Cleanup: Occasional permission errors in test teardown (cosmetic)

Pending Improvements

  • Audio pipeline test isolation refinement
  • Performance benchmarking test category
  • CI/CD integration workflows

Development Guidelines

Adding New Tests

  1. Location: Place tests in appropriate tier directory

    • Unit tests → tests/unit/[component]/
    • Integration tests → tests/integration/
    • Pipeline tests → tests/pipelines/

  2. Naming: Follow pattern test_[component]_[functionality].py

  3. Markers: Use appropriate pytest markers

    @pytest.mark.unit
@pytest.mark.integration
@pytest.mark.pipeline

Test Quality Standards

  • Unit Tests: Fast (<1s per test), isolated, mocked dependencies
  • Integration Tests: Real components, moderate speed (<5min total)
  • Pipeline Tests: Full workflows, real models, slower execution acceptable
  • All Tests: Clear docstrings, meaningful assertions, proper cleanup

Running Specific Test Categories

# By directory
pytest tests/unit/                          # All unit tests
pytest tests/integration/                   # All integration tests
pytest tests/pipelines/                     # All pipeline tests

# By marker
pytest -m unit                              # Unit tests only
pytest -m "integration and not slow"       # Fast integration tests

# By pattern
pytest -k "person_tracking"                # Person tracking tests
pytest -k "face and not slow"              # Face tests excluding slow ones

Integration with Development Workflow

Recommended Development Process

  1. During Development: python scripts/test_fast.py for immediate feedback
  2. Before Commit: python scripts/test_integration.py for validation
  3. Before Release: python scripts/test_all.py for comprehensive testing
  4. CI/CD Pipeline: Tiered execution based on change scope

Model Requirements

  • Person Tracking: YOLO11 models (auto-download)
  • Face Analysis: DeepFace models (auto-download)
  • Audio Processing: Whisper models (auto-download)
  • OpenFace3: Separate installation required (see requirements_openface.txt)

Maintenance & Updates

Regular Maintenance

  • Monitor test success rates and execution times
  • Update integration tests when adding new models
  • Refresh test data fixtures periodically
  • Review and update performance benchmarks

When Adding New Features

  1. Add unit tests for new functionality
  2. Add integration tests for cross-component features
  3. Add pipeline tests for complete workflows
  4. Update this documentation as needed

This testing infrastructure provides a solid foundation for reliable VideoAnnotator development with fast feedback loops, comprehensive coverage, and professional workflows.