Skip to content

feat(trino): append and upsert - #78

Merged
carolsimone merged 7 commits into
mainfrom
feat/write-modes-trino
Oct 11, 2026
Merged

carolsimone merged 7 commits into
mainfrom
feat/write-modes-trino

Conversation

@carolsimone

Copy link
Copy Markdown
Owner

Fifth runtime PR of the write-modes series (spec R5). Depends on the engine-port PR (#75, contract 0.8.0); stacked on #77 (DuckDB) locally — base is feat/write-modes-duckdb; retarget to main as the stack merges.

  • Risks first (spec section 8): a probe pins Trino 483 MERGE on Iceberg — update and insert in one statement from a stage table, insert-only MERGE, failure on format_version 1 (target unchanged), failure when two stage rows match one target row. All four held as written; no fallback was needed.
  • Refactor (D20): _create_stage / _insert_batches / _drop_stage shared by load, append, _upsert; load unchanged in behavior.
  • TrinoAdapter(WarehouseAdapter, AppendCapable, UpsertCapable): table_columns (information_schema, lowercased lookup, LookupError when absent); append = stage + one INSERT ... SELECT; _upsert = stage + one MERGE (no WHEN MATCHED when every column is in the key). One Iceberg commit per write; the stage drop afterwards is best-effort and only logged; zero rows create no stage.
  • ensure_table(..., primary_key=...): no constraint DDL; a key with format_version below 2 is rejected before any DDL.
  • The engine-contract conformance suite runs in the trino integration job, plus checks that a failed append or upsert leaves no stage table.

🤖 Generated with Claude Code

https://claude.ai/code/session_0137E9XmyUXhwPTRrtNfivDw

@carolsimone
carolsimone force-pushed the feat/write-modes-duckdb branch from 52344d8 to 0497fc4 Compare October 10, 2026 21:27
Base automatically changed from feat/write-modes-duckdb to main October 11, 2026 11:26
carolsimone and others added 7 commits October 11, 2026 13:31
Pins, on the Trino 483 plus Iceberg REST stack: MERGE from a stage table
updates and inserts in one statement; an insert-only MERGE inserts only new
keys; MERGE on a format_version 1 table fails and changes nothing; two stage
rows matching one target row fail.

Signed-off-by: Simone Carolini <simonecarolini.sc@gmail.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0137E9XmyUXhwPTRrtNfivDw
_create_stage and _drop_stage come out of load, beside _insert_batches: the
stage is created LIKE the target at a fresh sibling location and dropped
best-effort. load keeps its staged swap and recovery unchanged.

Signed-off-by: Simone Carolini <simonecarolini.sc@gmail.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0137E9XmyUXhwPTRrtNfivDw
TrinoAdapter declares TableIntrospection, looking up the lowercased schema and
table Trino stores and raising LookupError for a missing table. The
conformance suite runs against the Trino stack from here on.

Signed-off-by: Simone Carolini <simonecarolini.sc@gmail.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0137E9XmyUXhwPTRrtNfivDw
TrinoAdapter declares AppendCapable. Rows are batch-inserted into a stage
shaped like the target, then one INSERT SELECT writes them as a single Iceberg
commit; the stage is dropped best-effort and a failed drop is only logged.
Zero rows create no stage. Passes the append conformance cases.

Signed-off-by: Simone Carolini <simonecarolini.sc@gmail.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0137E9XmyUXhwPTRrtNfivDw
The key declares nothing (Trino has no constraint syntax) but a key with
config format_version below 2 is rejected before any DDL, because Trino's
MERGE needs Iceberg format version 2.

Signed-off-by: Simone Carolini <simonecarolini.sc@gmail.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0137E9XmyUXhwPTRrtNfivDw
TrinoAdapter declares UpsertCapable: the same stage as append, then one
MERGE INTO target USING stage that overwrites the non-key columns of matched
rows and inserts the rest, with no WHEN MATCHED branch when every column is
in the key. Passes the upsert conformance cases on the Trino stack.

Signed-off-by: Simone Carolini <simonecarolini.sc@gmail.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0137E9XmyUXhwPTRrtNfivDw
The engine-contract conformance suite moved to
continuo_engine_contract.testing.conformance in #75; the trino
integration test still imported the pre-move path and failed collection
once #75 merged to main.

Signed-off-by: Simone Carolini <simonecarolini.sc@gmail.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0137E9XmyUXhwPTRrtNfivDw
@carolsimone
carolsimone force-pushed the feat/write-modes-trino branch from 8d7772b to e335444 Compare October 11, 2026 11:31

@carolsimone carolsimone left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated code review (OCR delegation mode) over the full branch diff (26 changed files, +3013/−131): no critical/high/medium findings. One low-severity observation inline, plus one low-severity note on contract/continuo_engine_contract/keys.py (not part of this PR's diff): its ensure_unique_key docstring promises repeated keys are named "in order of first appearance", but that ordering comes from pyarrow's group_by, whose key order is not contractual — consider softening the docstring or sorting the repeated keys. Coverage: 26/26 files reviewed.

Comment on lines +597 to +602
format_version = (config or {}).get("format_version")
if primary_key and isinstance(format_version, int) and format_version < 2:
raise ValueError(
f"config 'format_version' {format_version} cannot hold an upsert table: "
"Trino's MERGE needs Iceberg format version 2 or later"
)

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Low (boundary condition): This guard only fires when the config explicitly sets format_version. If the target table already exists at Iceberg format v1 (e.g. created before the node became an upsert node) and the config omits format_version, the upsert fails later at the MERGE with a raw Trino error instead of this clear ValueError. This mirrors the postgres adapter's deliberate fail-at-ON CONFLICT stance, so it may be intentional — but Trino could cheaply detect it by reading format_version from {table}$properties when the table already exists and a key is set.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified — intentional, and I'm leaving it. The guard is deliberately config-only (documented at adapter.py:590-594): it rejects an explicit format_version < 2 before any DDL, and otherwise lets the engine reject a live-state mismatch at MERGE. That mirrors the postgres adapter's deliberate fail-at-ON CONFLICT stance for a missing key constraint, so pre-introspecting the live table would make trino stricter than postgres for the same class of problem.

The gap you describe is also narrow: modern Trino defaults new Iceberg tables to format v2, so a MERGE only hits v1 on a replace→upsert migration of a table that was explicitly created at v1. A cheap live check is possible here — the adapter already reads "schema"."table$properties" in _table_location — but the exact $properties key for the format version isn't something I can verify without the live Trino+Iceberg stack, and shipping a wrong key name would degrade the error (a RuntimeError from the probe) rather than improve it. If you'd like the clearer error for the migration case, I'll add it as a follow-up with an integration-test case that pins the property key against the real stack.

@carolsimone

Copy link
Copy Markdown
Owner Author

On the review-body note about keys.py ensure_unique_key: agreed the docstring over-promises — "in order of first appearance" leans on pyarrow group_by's key order, which isn't contractual. It's in already-merged code (contract, from #75), outside this PR's diff, so I'm not touching keys.py here. Batching it with the other merged-code cleanups into a small PR off main (softening the docstring; the repeated-key list is diagnostic only, so no need to sort it). The main branch review on this PR reported no critical/high/medium findings.

@carolsimone
carolsimone merged commit 27dd4a1 into main Oct 11, 2026
28 checks passed
@carolsimone
carolsimone deleted the feat/write-modes-trino branch October 11, 2026 12:28
carolsimone added a commit that referenced this pull request Oct 11, 2026
The three items noted in #77/#78 review threads as merged-code cleanups:
- loader.py: quote mode_type and kind (repr) in the "not allowed for kind"
  error, matching the sibling "must be one of" error; test updated.
- postgres adapter.py: ensure_table docstring port-signature ref 0.6.0 -> 0.8.0.
- keys.py: ensure_unique_key docstring no longer claims first-appearance
  order for the repeated-key message; pyarrow grouping order is not contractual.




Claude-Session: https://claude.ai/code/session_0137E9XmyUXhwPTRrtNfivDw

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant