Image Inpainting using GMCNN | Deep learning-based image restoration system that fills missing or damaged regions in images with high realism and structural consistency. Built with PyTorch.
-
Mokesh Prathaban
Model architecture design and implementation. -
Rushitha Alva
Dataset preprocessing, augmentation, training, and hyperparameter tuning. -
Vijay Kumar Reddy Marripati
Performance evaluation and benchmarking.
Traditional inpainting techniques, such as patch-based methods, struggle with large missing areas and complex textures. Deep learning-based approaches, including GANs (Generative Adversarial Networks) and Diffusion Models, have shown superior performance in generating realistic image completions.
Inability to handle large missing regions – Results in artifacts and unrealistic blending. Lack of structural consistency – Fails to preserve edges and object shapes. High computational cost – Training large generative models requires significant resources.
To overcome these challenges, our approach integrates: Multi-Column Encoder: Captures both global and local features for better reconstruction. PatchGAN Discriminator: Ensures realistic local texture generation. Advanced Loss Functions: Combines L1 loss, adversarial loss, and perceptual loss for improved quality.
- Multi-column encoder for capturing fine-grained and global features.
- PatchGAN and GMCNN discriminator for improving local texture realism.
- Loss functions: L1 Loss, Adversarial Loss, and Perceptual Loss (LPIPS).
- Evaluation metrics: PSNR (Peak Signal-to-Noise Ratio) and SSIM (Structural Similarity Index).
- Trained on the Places2 dataset (10,000 images, 256x256 resolution).
- Optimized for GPU training with Automatic Mixed Precision (AMP).
Deep Learning Frameworks: PyTorch, TensorFlow
Datasets: Places2
GAN-based Models: GMCNN
Training Environment: Google Colab Pro (NVIDIA A100 GPU)
1️) Clone the Repository https://github.com/ShivKavi/Image-InPainting-Using-DeepLearning.git
2️) Install Dependencies pip install -r requirements.txt
3️) Run the Jupyter Notebook jupyter notebook InPaintFP.ipynb
- Upload your custom masked image
- It will automatically load the GMCNN MODEL with epoch 100 and produce output.
To assess the performance of the inpainting model, we use the following metrics:
- Peak Signal-to-Noise Ratio (PSNR): Measures image reconstruction quality.
- Structural Similarity Index (SSIM): Evaluates structural fidelity.
- LPIPS (Learned Perceptual Image Patch Similarity) Measures perceptual similarity
To evaluate the robustness and trustworthiness of the inpainting model, we use an attack-and-analysis approach. Specifically, we test how the model behaves when minor perturbations are introduced to the masked inputs.
-
FGSM (Fast Gradient Sign Method)
Adds a small, one-step perturbation to the input in the direction that maximizes the model’s loss.
Evaluates the model’s sensitivity to small, high-impact noise. -
PGD (Projected Gradient Descent)
A stronger, multi-step iterative attack that simulates more effective adversarial corruption.
Useful for measuring worst-case degradation in inpainting quality.
- Apply FGSM and PGD attacks to masked inputs.
- Generate inpainted outputs under both clean and adversarial conditions.
- Measure degradation in PSNR, SSIM, and LPIPS between clean and attacked outputs.
- Helps identify how sensitive the model is to input manipulation.
- Reveals if the model maintains semantic coherence under uncertainty.
- Supports development of robust, trustworthy image completion systems.
Kindly check the Results folder for the screenshots of the results
- Pathak, D., et al. (2016). "Context Encoders: Feature Learning by Inpainting." CVPR.
- Yu, J., et al. (2019). "Free-Form Image Inpainting with Gated Convolution." ICCV.
- Iizuka, S., et al. (2017). "Globally and Locally Consistent Image Completion." SIGGRAPH.
- Liu, G., et al. (2018). "Image Inpainting for Irregular Holes Using Partial Convolutions." ECCV