Build a comprehensive adversarial rendering fixture corpus #412

Outcome

Create a reusable, Apache Tika-inspired corpus that exercises renderflow across valid documents, malformed inputs, ambiguous formats, hostile structure, boundary values, missing capabilities, and output-validation failures. Use it to prove graceful behavior across the standalone rendering pipeline and the exhaustive-artifact roadmap before the final repository audit.

This is the deepest raw-file corpus in the Flow suite because renderflow owns static document/artifact ingestion, normalization, rendering, and validation.

Motivation

Apache Tika-style parser corpora are valuable because they turn rare format behavior, malformed inputs, and regressions into durable tests. This repository needs the equivalent for its own product boundary.

The corpus must be versioned, deterministic, redistribution-safe, inspectable, and generator-first. It must not become an unreviewed dump of large or copyrighted binaries, and its coverage manifest must never claim that finite fixtures prove support for every possible file.

Cross-suite rules

• Keep fixture generation and assertions in the owning repository.
• Use public CLI/API and versioned artifacts for cross-tool tests; never import sibling source trees.
• Prefer small source recipes plus deterministic generators over committed large binaries.
• Record seed, generator/tool version, parameters, expected digest, media type, validity class, license/provenance, and expected behavior.
• Partition fixtures into valid, boundary, malformed, unsupported, and resource-stress classes.
• Malformed fixtures must have intentional corruption recipes and named expected diagnostics.
• Resource-stress cases must be safely bounded in normal CI; expensive/nightly tiers must be explicit.
• Never include secrets, personal data, proprietary samples, malware, weaponized polyglots, or uncontrolled decompression bombs.
• Fuzz-found regressions must be minimized, assigned a stable ID, documented, and promoted into the deterministic corpus.
• Tests must validate typed outcomes, artifacts, digests, cleanup, and source immutability—not merely process exit or file existence.
• Keep generated files reproducible and add a drift check; CI must never rewrite canonical fixtures.

Coverage inventory

Create a machine-readable coverage manifest and generator recipes for the formats and capabilities renderflow actually declares. Reconcile the exact matrix against live contracts before implementation. It should include, where applicable:

Documents and publications

• PDF: minimal, multipage, unusual page boxes/rotations, mixed page sizes, embedded/subset/missing fonts, transparency, annotations/forms, attachments, outlines, invalid xref tables, incremental updates, encrypted/password-required input, truncated streams, trailing garbage, conflicting metadata, huge declared dimensions, zero-page/blank-page cases, and structurally valid but unrenderable objects.
• Office/OpenDocument/e-book/text sources supported by current providers: empty and minimal documents, nested tables/lists, tracked changes/comments, hidden content, broken relationships, missing embedded resources, malformed ZIP containers, unexpected encodings, macro-bearing declarations treated according to policy, extreme headings/links, and unsupported feature declarations.
• Markdown/HTML/CSS/SVG/XML: encoding/BOM/newline variants, invalid nesting, broken references, deep nesting, cyclic includes where possible, inaccessible contrast/structure, external-resource refusal, data URLs, oversized canvas/viewBox, malformed XML, entity-expansion refusal, and script/active-content policy.
• Images: tiny/zero-dimension declarations, extreme aspect ratios, orientation and color-profile variants, alpha/transparency, animation where applicable, truncated payloads, incorrect extensions, mismatched magic bytes/MIME declarations, malformed metadata, oversized dimensions, and unsupported codecs.
• Fonts and layout: missing fonts, corrupt fonts, fallback chains, RTL/bidirectional text, combining marks, emoji, CJK, ligatures, vertical text, non-UTF-8 intake where supported, long unbreakable tokens, overflow, clipping, and deterministic pagination.
• Layered/localized sources: missing layers, duplicate identifiers, invalid locale tags, absent glyph coverage, textless/layered variants, overflow after translation, and divergent locale page counts where contracts constrain them.

Artifact and provider failures

• Provider unavailable, wrong version, timeout, cancellation, nonzero exit, signal termination, stdout/stderr limit, partial output, invalid output, wrong format, wrong page count/dimensions, stale output, symlinked output, path escape attempt, duplicate output names, and nondeterministic output.
• Corrupt or incompatible manifests, missing provenance, digest mismatch, schema-version mismatch, incomplete artifact sets, interrupted publication, and rerun/resume cases.
• Golden visual cases for deterministic renderers, with reviewed normalization/tolerance rules and no silent baseline updates.

Test tiers

1. Pull request: small deterministic smoke and regression set.
2. Scheduled/nightly: broader format/provider matrix and bounded resource cases.
3. Manual/release: licensed or heavyweight providers that cannot run hermetically, with explicit evidence and skips.
4. Fuzzing: parsers, manifest readers, and normalization boundaries; minimized failures graduate into the corpus.

Deliverables

• A documented fixture taxonomy and stable fixture IDs.
• Deterministic generators and pinned dependencies.
• A machine-readable manifest mapping each fixture to expected outcomes, diagnostics, artifacts, and invariants.
• Golden semantic/output metadata plus narrowly justified visual baselines.
• A capability-to-fixture coverage report that flags declared capabilities with no evidence.
• CI selection/sharding and time/size/memory budgets.
• A safe contribution workflow for adding a fixture from a bug report or fuzz result.
• At least one fixture for every currently declared input family and every shared failure class.
• Roadmap reconciliation under renderflow Roadmap: universal artifact forest, maximal Renderflow profile, and Flow orchestration readiness #367 and linkage from the Flow-suite epic.

Acceptance criteria

• Every fixture is reproducible or has reviewed redistribution provenance and a pinned digest.
• Valid, boundary, malformed, unsupported, and resource-stress classes are represented.
• Format/content sniffing is tested independently from filename extensions.
• Malformed inputs produce stable typed diagnostics without panics, hangs, unbounded allocation, or unsafe partial publication.
• Active/external content follows explicit offline and security policy.
• Successful provider exit alone cannot satisfy a case.
• Output structure, format, page/frame count, dimensions, digests, provenance, accessibility evidence, and artifact-set completeness are validated where applicable.
• Cancellation, timeouts, cleanup, retry, and deterministic rerun behavior are covered.
• Source fixtures remain byte-for-byte unchanged.
• Golden baseline changes require explicit review and explain normalization/tolerance.
• The PR/nightly/release tiers have enforced resource budgets.
• CI reports coverage gaps honestly and does not claim universal format completeness.
• Repository-required stable/MSRV, contract, documentation, package, smoke, fuzz, Identity, and site validation remains green.

Dependencies and scheduling

• Coordinate with renderflow Roadmap: universal artifact forest, maximal Renderflow profile, and Flow orchestration readiness #367 instead of creating a competing exhaustive-artifact epic.
• Consume provider contracts already accepted on live main.
• Implement before the final post-roadmap repository audit so the audit can use the corpus as evidence.
• Break implementation into bounded child checkpoints if the live matrix is too large for one reviewable PR.

Non-goals

• Shipping every known file format in one PR.
• Bundling arbitrary internet samples or large copyrighted documents.
• Treating snapshots as the only assertion.
• Weakening parser/provider safety to accept malformed inputs.
• Cross-tool orchestration; flow owns it.
• Temporal processing owned by aniflow.
• File optimization owned by optiflow.

Audit-to-roadmap reconciliation — 2026-10-21

Audit: https://github.com/egohygiene/renderflow/blob/audit/flow-suite-20261021T025401Z/audits/flow-suite-20261021T025401Z.md · Complete roadmap: #367

Extend existing #366 generator-first corpus in bounded tiers rather than a second harness.

Bounded preparation: Map present capability rows to deterministic positive/negative fixture cases and assign finite slices.

Remaining acceptance (common qualification deferral; no dependency star): Deferred execution and accepted provider contracts for each claimed native route; broad corpus is not a prerequisite to implement exporters.

The maintainer's implementation-sprint qualification deferral remains in force. This audit ran no Renderflow tests, compiler/build/Cargo resolution, product lint/format/schema qualification, native/CLI/provider/visual execution, installers or new CI. Reading existing exact-source provider records is not a qualification rerun. Source implementation and authored tests are not passing-test, native-fidelity, release or downstream-consumer evidence.

This is an additive planning note. The full original issue, acceptance criteria, scope, parent references and historical checkpoints remain authoritative and unchanged. No acceptance box or issue is closed by this audit. Recommended ledger parent: #367.