Skip to content

Add strip endpoint backed by an ExifTool process pool - #1

Merged
Meldiron merged 3 commits into
appwrite:mainfrom
jaysomani:feat/strip-endpoint
Oct 16, 2026
Merged

Meldiron merged 3 commits into
appwrite:mainfrom
jaysomani:feat/strip-endpoint

Conversation

@jaysomani

@jaysomani jaysomani commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

What does this PR do?

Adds POST /strip, which removes metadata such as location, camera, and capture time from an image and returns the stripped file. The ICC color profile and orientation are kept, and pixel data is never re-encoded. This is the service Appwrite will call for buckets with stripMetadata enabled (appwrite/appwrite#14351), replacing the per-worker ExifTool processes that used ~2 GB per container.

  • internal/exiftool: a fixed pool of ExifTool processes in -stay_open mode, warmed up at startup. Each command uses a numbered -execute{N} / {readyN} marker, so a stuck command's output can never leak into the next request. A process that times out, is cancelled, or exits is killed and started again on its next use.
  • /strip accepts a raw body or a multipart file field. The format is detected from the content, so a renamed file is still stripped. Responses: 200 stripped file, 422 file ExifTool can't rewrite, 413/400 bad input, 504 timeout, 503 shutting down. Callers should keep the original on anything but 200.
  • New settings: MAX_CONCURRENT_STRIPS (default: CPUs), STRIP_TIMEOUT (30s), MAX_REQUEST_SIZE (25MiB, Appwrite strips files up to 20 MB). Buffered uploads are bounded separately from running strips, following autogravity.
  • Strip metrics: stage durations, outcomes, active commands, cancellations.

The scaffold is unchanged apart from passing the pool into newApplication and adding internal to the gofmt check.

Test Plan

  • Unit tests (fake stripper): raw and multipart uploads, request limits, every error status, deadlines, client cancellation, a slow upload hitting the server read deadline (real server), bounded concurrency and buffered uploads, temp file cleanup, shutdown, metrics.
  • make test-integration against the pinned ExifTool 13.55: GPS, camera, date, and XMP creator removed from JPEG and PNG; orientation kept; ICC profile kept byte for byte; text file left unchanged (ErrNotStripped); recovery after a timeout and after a killed process; 12 concurrent strips on 2 processes; Close wakes a waiting strip; a failed startup leaves no processes behind.
  • Docker image built from this branch, MAX_CONCURRENT_STRIPS=4:
Check Result
Startup (4 processes warmed) 0.27 s
150 KB / 2.7 MB JPEG 61 ms / 74 ms; tags removed, orientation kept, pixels identical
400 requests, 40 concurrent all 200, p50 14 ms, p95 34 ms
Container memory ~136 MiB idle, ~140 MiB after load
SIGTERM exit 0, no temp files left

Related PRs and Issues

@hansi-codes

hansi-codes Bot commented Oct 16, 2026 •

Copy link
Copy Markdown

🟢 Tier S · Ready to merge

The reviewed changes passed unit and real-ExifTool integration tests, and I found no actionable defects.

Adds a /strip endpoint backed by a warmed ExifTool process pool, with bounded uploads, request limits, deadlines, and strip telemetry. The service strips metadata while preserving orientation and ICC profiles, and includes unit and real-ExifTool coverage for the supported image formats. The newest changes stream uploads and responses through temporary files and expand format coverage and pool-size metrics.

Latest changes: The new commits move upload and response handling to disk-backed streaming, add the pool size to build-info metrics, use ExifTool status output for rewrite results, and expand integration coverage and documentation to TIFF, GIF, WebP, AVIF, and HEIC.

Verdict New comments Fixed Still open
💬 Commented 0 0 0

Note

@jaysomani does not have write access to this repository, so Hansi does not approve automatically. A maintainer can review and approve.

📂 Walkthrough · 11
File Change
.mise.toml Extends formatting checks to include the internal Go package.
CONTRIBUTING.md Documents unit and integration coverage, including the expanded image-format fixtures.
README.md Documents the strip endpoint, configuration, metrics, and operational resource characteristics.
cmd/exiftool-server/main.go Implements strip request handling, limits, deadlines, lifecycle, and disk-backed upload and response streaming.
cmd/exiftool-server/main_test.go Covers endpoint behavior, limits, concurrency, cancellation, cleanup, configuration, and telemetry.
cmd/exiftool-server/integration_test.go Exercises raw and multipart stripping with real ExifTool.
cmd/exiftool-server/telemetry.go Adds HTTP and strip-stage metrics, cancellation and outcome counters, and build information.
internal/exiftool/exiftool.go Implements the warmed process pool, metadata-removal commands, process recovery, and shutdown signaling.
internal/exiftool/exiftool_test.go Tests metadata stripping, preservation, recovery, concurrency, shutdown, and expanded image-format coverage.
internal/exiftool/testdata/photo.avif, internal/exiftool/testdata/photo.gif, internal/exiftool/testdata/photo.heic, internal/exiftool/testdata/photo.tiff, internal/exiftool/testdata/photo.webp Adds image fixtures used by the ExifTool integration tests.
internal/exiftool/warmup.jpg Embeds the JPEG used to warm each ExifTool process at startup.

Reviewed the commits since 34cd1c5 · Details · Comment @hansi-codes review to re-run, or mention @hansi-codes with a question.

@hansi-codes hansi-codes Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Tier A · See the inline comments. Summary

Comment thread cmd/exiftool-server/main.go
Comment thread internal/exiftool/exiftool.go

@hansi-codes hansi-codes Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Tier S · Looks good to merge. Summary

@Meldiron
Meldiron merged commit 7ee762e into appwrite:main Oct 16, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants