Repository navigation
feat(trino): append and upsert - #78
Conversation
52344d8 to
0497fc4
Compare
Pins, on the Trino 483 plus Iceberg REST stack: MERGE from a stage table updates and inserts in one statement; an insert-only MERGE inserts only new keys; MERGE on a format_version 1 table fails and changes nothing; two stage rows matching one target row fail. Signed-off-by: Simone Carolini <simonecarolini.sc@gmail.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0137E9XmyUXhwPTRrtNfivDw
_create_stage and _drop_stage come out of load, beside _insert_batches: the stage is created LIKE the target at a fresh sibling location and dropped best-effort. load keeps its staged swap and recovery unchanged. Signed-off-by: Simone Carolini <simonecarolini.sc@gmail.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0137E9XmyUXhwPTRrtNfivDw
TrinoAdapter declares TableIntrospection, looking up the lowercased schema and table Trino stores and raising LookupError for a missing table. The conformance suite runs against the Trino stack from here on. Signed-off-by: Simone Carolini <simonecarolini.sc@gmail.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0137E9XmyUXhwPTRrtNfivDw
TrinoAdapter declares AppendCapable. Rows are batch-inserted into a stage shaped like the target, then one INSERT SELECT writes them as a single Iceberg commit; the stage is dropped best-effort and a failed drop is only logged. Zero rows create no stage. Passes the append conformance cases. Signed-off-by: Simone Carolini <simonecarolini.sc@gmail.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0137E9XmyUXhwPTRrtNfivDw
The key declares nothing (Trino has no constraint syntax) but a key with config format_version below 2 is rejected before any DDL, because Trino's MERGE needs Iceberg format version 2. Signed-off-by: Simone Carolini <simonecarolini.sc@gmail.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0137E9XmyUXhwPTRrtNfivDw
TrinoAdapter declares UpsertCapable: the same stage as append, then one MERGE INTO target USING stage that overwrites the non-key columns of matched rows and inserts the rest, with no WHEN MATCHED branch when every column is in the key. Passes the upsert conformance cases on the Trino stack. Signed-off-by: Simone Carolini <simonecarolini.sc@gmail.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0137E9XmyUXhwPTRrtNfivDw
The engine-contract conformance suite moved to continuo_engine_contract.testing.conformance in #75; the trino integration test still imported the pre-move path and failed collection once #75 merged to main. Signed-off-by: Simone Carolini <simonecarolini.sc@gmail.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0137E9XmyUXhwPTRrtNfivDw
8d7772b to
e335444
Compare
carolsimone
left a comment
There was a problem hiding this comment.
Automated code review (OCR delegation mode) over the full branch diff (26 changed files, +3013/−131): no critical/high/medium findings. One low-severity observation inline, plus one low-severity note on contract/continuo_engine_contract/keys.py (not part of this PR's diff): its ensure_unique_key docstring promises repeated keys are named "in order of first appearance", but that ordering comes from pyarrow's group_by, whose key order is not contractual — consider softening the docstring or sorting the repeated keys. Coverage: 26/26 files reviewed.
| format_version = (config or {}).get("format_version") | ||
| if primary_key and isinstance(format_version, int) and format_version < 2: | ||
| raise ValueError( | ||
| f"config 'format_version' {format_version} cannot hold an upsert table: " | ||
| "Trino's MERGE needs Iceberg format version 2 or later" | ||
| ) |
There was a problem hiding this comment.
Low (boundary condition): This guard only fires when the config explicitly sets format_version. If the target table already exists at Iceberg format v1 (e.g. created before the node became an upsert node) and the config omits format_version, the upsert fails later at the MERGE with a raw Trino error instead of this clear ValueError. This mirrors the postgres adapter's deliberate fail-at-ON CONFLICT stance, so it may be intentional — but Trino could cheaply detect it by reading format_version from {table}$properties when the table already exists and a key is set.
There was a problem hiding this comment.
Verified — intentional, and I'm leaving it. The guard is deliberately config-only (documented at adapter.py:590-594): it rejects an explicit format_version < 2 before any DDL, and otherwise lets the engine reject a live-state mismatch at MERGE. That mirrors the postgres adapter's deliberate fail-at-ON CONFLICT stance for a missing key constraint, so pre-introspecting the live table would make trino stricter than postgres for the same class of problem.
The gap you describe is also narrow: modern Trino defaults new Iceberg tables to format v2, so a MERGE only hits v1 on a replace→upsert migration of a table that was explicitly created at v1. A cheap live check is possible here — the adapter already reads "schema"."table$properties" in _table_location — but the exact $properties key for the format version isn't something I can verify without the live Trino+Iceberg stack, and shipping a wrong key name would degrade the error (a RuntimeError from the probe) rather than improve it. If you'd like the clearer error for the migration case, I'll add it as a follow-up with an integration-test case that pins the property key against the real stack.
|
On the review-body note about |
The three items noted in #77/#78 review threads as merged-code cleanups: - loader.py: quote mode_type and kind (repr) in the "not allowed for kind" error, matching the sibling "must be one of" error; test updated. - postgres adapter.py: ensure_table docstring port-signature ref 0.6.0 -> 0.8.0. - keys.py: ensure_unique_key docstring no longer claims first-appearance order for the repeated-key message; pyarrow grouping order is not contractual. Claude-Session: https://claude.ai/code/session_0137E9XmyUXhwPTRrtNfivDw Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Fifth runtime PR of the write-modes series (spec R5). Depends on the engine-port PR (#75, contract 0.8.0); stacked on #77 (DuckDB) locally — base is
feat/write-modes-duckdb; retarget tomainas the stack merges._create_stage/_insert_batches/_drop_stageshared byload,append,_upsert;loadunchanged in behavior.TrinoAdapter(WarehouseAdapter, AppendCapable, UpsertCapable):table_columns(information_schema, lowercased lookup,LookupErrorwhen absent);append= stage + oneINSERT ... SELECT;_upsert= stage + oneMERGE(noWHEN MATCHEDwhen every column is in the key). One Iceberg commit per write; the stage drop afterwards is best-effort and only logged; zero rows create no stage.ensure_table(..., primary_key=...): no constraint DDL; a key withformat_versionbelow 2 is rejected before any DDL.🤖 Generated with Claude Code
https://claude.ai/code/session_0137E9XmyUXhwPTRrtNfivDw