Scene Completion using Local Context Matching

A naive Python implementation of Local Context Matching as shown in Scene Completion Using Millions of Photographs (Hays & Efros, 2007), which was completed as part of Georgia Tech's Computational Photography Course. More information on the approach can be found here.

Given a photograph and a mask (white = region to remove), the pipeline:

1. crops a local context window around the hole,
2. finds the best matching window inside a candidate photograph with a masked SSD search,
3. cuts around the hole along minimal difference seams (Dijkstra / graph cut),
4. blends the match into the original with OpenCV seamless cloning,
5. re-synthesises the paste seams with LaMa inpainting so the final image shows no transition artefacts.

Example results

For each sample: the input photograph, its mask (white = region to fill), the best-scoring candidate photo (ranked automatically by masked context SSD), and the completed output after the LaMa seam cleanup pass.

Input	Mask	Best match	Completed output

Intermediate stages for input3

Between matching and blending, a graph-cut seam is traced around the hole and the enclosed region is replaced:

Seam mask (white = replaced)	Pasted composite	Final output (after LaMa cleanup)

Finally, the LaMa pass re-synthesises the paste boundaries — shown below as raw composite → seam band handed to the network → cleaned output:

Getting started

The project is managed with uv.

# install uv if you don't have it
curl -LsSf https://astral.sh/uv/install.sh | sh

uv sync --group dev        # creates .venv and installs all dependencies

Dependencies include opencv-python-headless (OpenCV >= 5), NumPy, scikit-image and matplotlib.

Run scene completion

# pick the best candidate from a directory automatically
uv run python local_context_matching.py \
    --image sample_images/images/input3.jpg \
    --mask sample_images/images/input3_mask.jpg \
    --candidates-dir sample_images/images/input3 \
    --save-dir output

# ...or specify the candidate image yourself
uv run python local_context_matching.py \
    --image sample_images/images/input3.jpg \
    --mask sample_images/images/input3_mask.jpg \
    --match sample_images/images/input3/result_img001.jpg \
    --save-dir output

Every intermediate stage (context windows, best match, seam mask, composites, final output) is written to --save-dir. The last stage written, output_lama.jpg, is the final output after the LaMa seam cleanup pass. Pass --no-lama to skip it entirely, or --lama-band N to change the width of the seam ring repainted by LaMa (default 12 px).

LaMa seam inpainting

The composited result inevitably shows a visible transition where the matched content is pasted over the original photo. lama_inpaint.py removes these artefacts with LaMa running through OpenCV's DNN module:

1. a thin seam ring is derived from the seam cut — band pixels straddling every paste boundary (the hole rim plus each graph-cut seam), so LaMa re-synthesises only the visible transition and leaves the matched fill itself untouched,
2. segments running along the image borders are dropped — pasted content that runs off the edge has no transition to hide there — and a degenerated seam cut (flood fill swallowed the whole context) falls back to the hole outline alone,
3. a padded window around the ring is resampled to the network's fixed 512x512 input, inpainted in one pass, and resampled back at full resolution — much sharper than squashing the whole photograph to 512x512,
4. the result is feather-blended strictly inside the ring, so pixels outside it remain bit-identical to the raw composite.

The weights live in models/lama.onnx and are tracked with Git LFS — install it before cloning, otherwise you will only fetch a pointer file instead of the ~90 MB model.

Tests

uv run pytest              # fast unit tests
uv run pytest -m slow      # + full pipeline integration test

Notebooks

uv run jupyter lab

• GistDescriptor/gist_descriptor_colour.ipynb — colour GIST descriptor: computed per BGR channel and concatenated (16 x 32 x 3 = 1536 values).

How it works

local_context_matching.py contains the whole pipeline:

Function	Purpose
read_images	load photograph, Otsu-thresholded mask and candidate image
get_masked_scene	crop the local context window around the hole and black out the hole
find_scene	masked SSD search over every alignment (cv2.matchTemplate); brute force reference kept in find_scene_bruteforce
pick_best_candidate	rank a folder of candidates by their best masked context SSD
create_seam_cut	four minimum-cost seams around the hole via skimage.graph.MCP, closed by flood fill
composite_scene	merge match into original via paste, feathered alphablend, or Poisson seamlessclone
composite	paste the completed context back into the full size image
build_seam_band_mask	thin ring straddling the boundary of the pasted (match) region
_replaced_region	replaced-area map; degenerated seam cuts fall back to the hole
_strip_image_edges	drops mask pixels along the image borders
inpaint_seams_lama	re-synthesise the paste seams with LaMa and feather-blend them back
local_context_match / scene_completion_pipeline	run everything end to end

lama_inpaint.py wraps the ONNX model itself: lazy single load of the network, windowed 512x512 inference at arbitrary resolutions (lama_inpaint), plus a standalone CLI (uv run python lama_inpaint.py [image] [mask]).

Other notes and work

Additional outputs are located in the following albums:

• http://imgur.com/a/I0q0n
• http://imgur.com/a/ZdWUS
• http://imgur.com/a/jEeXF

Whilst my (failed) attempt at the GIST descriptor is based on the hints provided on Quora. The GIST descriptor is used to perform similar image matching within the Scene Completion paper. The colour variant looks at all the colours rather than the grayscale image only.

The sample images are based on the Scene Completion work assignment which was implemented in Matlab.