README: KPD-3D benchmark - the benchmark for 3D keypoint detection algorithms

Project Description

This project is a comprehensive benchmark for evaluating the quality of feature point detectors in 3D volumes. The system automates the process of testing various keypoint detection algorithms (SIFT, LoG, DoG) on a set of 3D objects, applying affine transformations and assessing repeatability. The benchmark is submitted for publication to ICMV2026, and the link to paper will be available after a publication

Key features:

• Test data generation: automatic creation of transformed versions of 3D volumes with specified parameters (rotation, scale, noise)
• Multiple detection algorithms: support for SIFT3D_SCUT, SIFT3D_Rister, Scikit-LoG, and Scikit-DoG
• Parallel processing: efficient use of multithreading and multiprocessing
• Quality assessment: calculation of repeatability and multi-repeatability across various threshold values
• Result visualization: automatic plot generation and statistics export to CSV

---

Requirements

System requirements:

• Python 3.8+
• Windows/Linux (taking file paths in config files into account)
• Sufficient RAM to load 3D volumes (16GB+ recommended)
• For SIFT3D_SCUT: CUDA support and the 3DSIFT.exe executable

Dependencies:

See required_libraries.txt

Key libraries:

• numpy, scipy
• scikit-image
• opencv-python
• pandas, matplotlib
• vedo (3D визуализация)
• torch, torchio
• joblib, psutil
• SimpleITK

Installing:

pip install -r required_libraries.txt

External executables:

3DSIFT.exe – for the SIFT3D_SCUT algorithm (path specified in the config)

regSift3D и kpSift3D - SIFT3D_Rister

Configuration

Main configuration file: ICMV2026_main.json

{
    "path_to_data": "path/to/data/",
    "path_to_source_code": "path/to/kpd3d",
    "path_to_results": "path/to/experiments/",
    "delete_temp_files": false,
    "n_jobs_auxilary": -2,
    "n_jobs_detectors": 4
}

algorithms section:

Specifies which algorithms to test and the paths to their configurations.

augmentation_generation section:

selected_volume_list - list of objects to be tested

selected_augmentation_parameter_list - list of transformation parameter sets

augmentation_parameters_set - parameters for each set:

• rotation_0x_deg
• rotation_0y_deg
• rotation_0z_deg
• scale
• noise_level

Running

python ICMV2026_wrapper.py ICMV2026_main.json

Execution workflow:

Step 1: Configuration generation

A folder is created in path_to_results for each object from selected_volume_list Configurations are generated for the generator (gen.json) and for each algorithm ({alg}_main_config.json)

parser_config.json is created for subsequent analysis

Step 2: Transformed volume generation

generator.py is executed for each object

Transformed versions are created using the specified parameters Transformation matrices (tr_gen.json) are generated

Step 3: Keypoint detection

main_pipeline_ICMV2026.py is executed for each algorithm and object. Keypoint detection is performed on the original (reference) and transformed volumes. Results are saved to stitching_results/{algorithm}/ folders.

Step 4: Repeatability calculation

sift_repeatability_ICMV2026.py is executed

Repeatability is calculated for various threshold values. CSV files and plots are saved.

Step 5: Results analysis (optional)

ICMV2026_results_analysis.py is executed. Algorithm comparison plots are generated.

Output Data

Results Structure:

experiments/
├── {object_name}/
│   ├── gen.json
│   ├── {alg}_main_config.json
│   ├── transformed_volumes/
│   │   └── gr{N}_d{0|1|2}/
│   │       ├── rotation_0x_deg_{j}/
│   │       │   ├── rotation_0y_deg_{k}/
│   │       │   │   ├── 0000.tif, 0001.tif, ...
│   │       │   │   └── tr_gen.json
│   │       └── ...
│   └── stitching_results/
│       └── {alg}/
│           └── rotation_0x_deg_{j}/
│               └── rotation_0y_deg_{k}/
│                   └── metrics/
│                       └── {repeat}/
│                           ├── markup_keypoints.csv #or any other custom format
│                           ├── test_keypoints.csv
│                           ├── time_mem.json
│                           └── matrices.json
├── {alg}_repeatability_all_thresholds.csv
├── {alg}_multi_repeatability_all_thresholds.csv
├── {alg}_mean_repeatability_by_object.csv
├── {alg}_global_mean_repeatability.csv
├── {alg}_raw_data.pkl
└── {alg}_repeatability_plot.png

Adding a new detection algorithm

1. Create a detector Python script that accepts arguments:

markup_volume_path test_volume_path processing_folder_path \
metrics_folder_path initial_transform_matrix_path alg_params_json

2. Add to ICMV2026_main.json:

• To selected_algorithm_name_list
• To algorithm_help
• To algorithm_execution_parameters_paths
• To algorithm_paths
• To alg_interpreter_paths
• To algorithm_parsing_format
• To algorithm_output_format

3. Create a configuration file for the algorithm

Example of setting up a test object

To add a new 3D object for testing: Place the object data in data/markups/{object_name}/ as TIFF layers Add object_name to selected_volume_list Create an .off file for the CAD model if necessary