Skip to content

feat/stream Support seekable streams in modern IMAP APPEND - #59

Merged
ralflang merged 3 commits into
horde:FRAMEWORK_6_0from
neuhaus:feat/modern-append-streams
Nov 2, 2026
Merged

ralflang merged 3 commits into
horde:FRAMEWORK_6_0from
neuhaus:feat/modern-append-streams

Conversation

@neuhaus

@neuhaus neuhaus commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

The modern ImapClient::append() currently accepts message data as strings. This requires callers to load large MIME messages into memory before uploading.

This change adds support for readable, seekable PHP stream resources:

$message = fopen('/private/message.eml', 'rb');
try {
    $uids = $client->append('Objects', [['data' => $message]]);
} finally {
    fclose($message);
}

Behavior

  • Sends bytes from the stream’s current position through EOF without modifying them.
  • Restores the original position and leaves the resource open.
  • Determines the literal length and scans for NUL using bounded buffers.
  • Writes payloads in chunks of at most 64 KiB (configurable via constant ImapWireStream::CHUNK_SIZE), without materializing the entire message or making a temporary copy.
  • Uses literal8 for NUL-containing streams, requiring BINARY support.
  • A binary: true entry forces literal8 for strings and streams without NUL, also requiring BINARY. Servers may normalize line endings of ordinary literals (Dovecot stores bare LF as CRLF), so this keeps bodies that must be stored byte for byte intact.
  • Supports mixed string/stream MULTIAPPEND batches and replay after TRYCREATE.
  • Closes the connection if a source read or payload write fails mid-literal, preventing subsequent commands from being interpreted as payload data.

Callers must provide complete, correctly formed MIME messages. Non-seekable sources must be spooled by the caller first. Existing string and CATENATE inputs and the legacy API remain unchanged.

The change also preserves response codes on rejected literal announcements, allowing TRYCREATE handling before payload transmission, and releases pending commands when transmission fails.

Validation

  • 32 new regression tests cover wire framing, byte preservation, bounded writes, stream position and ownership, batching, retries, forced literal8, and failure handling.
  • Full unit suite: 1,653 tests, no failures or errors; 55 skipped, two incomplete, and one unrelated PHP 8.5 deprecation.
  • Live modern APPEND uploads of 1 MiB and 512 MiB were independently verified using the legacy partial BINARY FETCH reader, with exact length and SHA-256 matches.
  • The 512 MiB round trip completed under a 64 MiB PHP memory limit, with 10 MiB peak PHP memory.
  • Live: text bodies with bare LF, CRLF and bare CR appended with binary: true are returned byte for byte by Dovecot.

This branch is based directly on FRAMEWORK_6_0.

@what-the-diff

what-the-diff Bot commented Oct 24, 2026 •

Copy link
Copy Markdown

PR Summary

  • Enables APPEND from seekable streams, so applications can upload large messages without first loading the entire content into memory. The stream’s original position is preserved, and the stream remains available for retries.
  • Improves safety and compatibility for streamed uploads by checking inputs before sending and handling binary content, UTF-8 requirements, server capabilities, and message creation retries.
  • Makes failed uploads safer to recover from by sending content in bounded chunks and closing the connection if a write fails or is cut short. Failed commands are also removed from the pending queue.
  • Adds documentation and test coverage for stream behavior, validation, retries, and failure handling.

neuhaus and others added 2 commits October 24, 2026 22:31
A `binary: true` entry is sent as literal8 even when it contains no
NUL, for strings and streams alike, and requires BINARY. Servers may
normalize line endings of ordinary literals (Dovecot stores bare LF as
CRLF), which corrupts bodies that must be stored byte for byte, such
as Content-Transfer-Encoding: binary.
@ralflang
ralflang merged commit 8cee30d into horde:FRAMEWORK_6_0 Nov 2, 2026
1 check passed
@neuhaus
neuhaus deleted the feat/modern-append-streams branch November 2, 2026 07:01
ralflang added a commit that referenced this pull request Nov 3, 2026
Release version 3.2.0

test: Cover validation of range handling
fix: regression in SORT
fix: Issue #62 handle sequence range expansion in a memory safe way
docs(append): Add upgrading instructions for binary and catenate handling
test: Add Binary Catenate correctness tests
feat(append): Handle the binary: true & catenate scenario according to spec
Merge pull request #59 from neuhaus/feat/modern-append-streams
Merge branch 'FRAMEWORK_6_0' into feat/modern-append-streams
Read complete IMAP literals across short socket reads
Add decoded partial BINARY FETCH to the modern IMAP API
Polish binary APPEND documentation
Use local assembly for binary CATENATE inputs
Add opt-in byte-preserving binary APPEND to legacy socket
Allow forcing literal8 in modern IMAP APPEND
Support seekable streams in modern IMAP APPEND
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants