Repository navigation
Create large-image tiles in a job instead of on every upload request - #2008
Merged
Merged
Conversation
girder_large_image's auto-set runs createImageItem synchronously when each file is saved, so every uploaded or imported image -- including JPG/PNG frames of plain image sequences, which never use tiles -- was probed for a tile source inside its own upload request. On a multicam flight with thousands of frames per camera, uploads queued behind that work until requests hit the gateway timeout. The DIVE plugin now defaults large_image.auto_set to off (an explicit admin setting still wins). postprocess queues a create_large_image_tiles job when a folder holds large-image files (TIFF, NITF, ...) that have no tiles yet; the job creates them one item at a time with progress and marks the folder as a dataset when done. Uploads and filesystem/S3 imports both reach postprocess, so both are covered. The existing per-item tile check is shared with convert_large_images. The multicam upload's wait for camera processing now restarts its 10-minute clock whenever a job reports progress, so a long tiling job that keeps moving is not cut off.
Keep deferred large-image tile creation from this branch alongside main's RLE postprocess job and non-blocking multicam finalize. Tag tile jobs with job_dataset_id so parent multicam folders stay in Processing until tiling finishes. Co-authored-by: Cursor <cursoragent@cursor.com>
BryonLewis
self-requested a review
October 6, 2026 15:57
BryonLewis
approved these changes
Oct 6, 2026
BryonLewis
left a comment
BryonLewis
left a comment
Collaborator
There was a problem hiding this comment.
Small minor changes to make it fit with the postprocess job system that I have for other tasks like COCO RLE mask creation.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
girder_large_image's auto-set runs createImageItem synchronously when each file is saved, so every uploaded or imported image -- including JPG/PNG frames of plain image sequences, which never use tiles -- was probed for a tile source inside its own upload request. On a multicam flight with thousands of frames per camera, uploads queued behind that work until requests hit the gateway timeout.
The DIVE plugin now defaults large_image.auto_set to off (an explicit admin setting still wins). postprocess queues a create_large_image_tiles job when a folder holds large-image files (TIFF, NITF, ...) that have no tiles yet; the job creates them one item at a time with progress and marks the folder as a dataset when done. Uploads and filesystem/S3 imports both reach postprocess, so both are covered. The existing per-item tile check is shared with convert_large_images.
The multicam upload's wait for camera processing now restarts its 10-minute clock whenever a job reports progress, so a long tiling job that keeps moving is not cut off.