Repository navigation
feat(engine-contract): write capabilities, ensure_unique_key, conformance suite (contract 0.8.0) - #75
Conversation
carolsimone
left a comment
There was a problem hiding this comment.
High · bug: append/upsert still execute as replace (inherited from #74)
Location: continuo_python_runtime/contract/loader.py:290–299. This is inherited from the prerequisite branch, so it is posted here as a review note rather than attached to this PR's diff.
The loader accepts append and upsert, but harness.run_node() still unconditionally calls ensure_table() followed by load(), whose contract is to replace the table's contents. In reproductions using validated nodes and an in-memory adapter initially holding id=99, both modes returned success and left only id=1. Adapters without either opt-in capability also reach this path.
Until the capability-specific write dispatch is implemented, reject these modes in the harness before running the script or writing anything. Otherwise an accepted append/upsert contract can silently delete existing rows.
| counts = ( | ||
| data.select(key) | ||
| .rename_columns(positional) | ||
| .group_by(positional, use_threads=False) | ||
| .aggregate([([], "count_all")]) | ||
| ) | ||
| if counts.num_rows == data.num_rows: | ||
| return |
There was a problem hiding this comment.
High · bug: signed-zero keys bypass the uniqueness check
Arrow groups 0.0 and -0.0 separately. Consequently ensure_unique_key(pa.table({'id': [0.0, -0.0]}), ['id']) returns successfully even though the warehouse compares these keys equal.
This input also passes conform() for a non-nullable DOUBLE PRECISION key. In an in-memory DuckDB reproduction, a MERGE into an empty table inserted both rows, and SQL GROUP BY id reported one key with count 2. The template method therefore allows duplicate warehouse keys to reach _upsert, violating the invariant this guard is meant to enforce.
Normalize floating key equality (including signed zero) before grouping, or reject unsupported floating key types, and add a regression test.
There was a problem hiding this comment.
Fixed in b47e4da. ensure_unique_key now rejects floating-point key columns (halffloat/float/double) with a ValueError before grouping. Normalizing signed zero would need pyarrow.compute, but the contract package keeps pyarrow out of its runtime imports (sqlglot-only), so rejection is the right option — and ensure_unique_key is the single enforcement point the UpsertCapable.upsert template runs for every engine. Regression tests cover 0.0/-0.0, float32, a composite key with one float column, and that decimal128 keys still pass.
f7f8e8d to
b47e4da
Compare
|
The append/upsert-as-replace risk is now guarded: c587b88 (on the R1 base branch |
b47e4da to
856dd3f
Compare
Groups an Arrow table by its primary key and raises DuplicateKeyError (a ValueError) naming the first repeated keys and their counts. Upsert calls it before any engine sees the rows, because no engine handles repeated keys in one write consistently. Signed-off-by: Simone Carolini <simonecarolini.sc@gmail.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0137E9XmyUXhwPTRrtNfivDw
…mary_key TableIntrospection, AppendCapable and UpsertCapable are abstract interfaces an adapter declares by inheritance; LiveColumn is what table_columns reports. UpsertCapable.upsert is a template method that checks key uniqueness before the engine-specific _upsert. ensure_table gains an optional keyword-only primary_key that the harness passes only for upsert nodes. Signed-off-by: Simone Carolini <simonecarolini.sc@gmail.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0137E9XmyUXhwPTRrtNfivDw
IntrospectionConformance, AppendConformance and UpsertConformance hold the spec's cases (accumulation, zero rows, atomicity across statements, overwrite, idempotent rerun, repeated keys, every column in the key) plus quoting, column order, first-run upsert and the FX key types. A reference in-memory adapter runs them here, with two broken adapters as negative controls. Shipped in the wheel; pytest and pyarrow come with the testing extra. Signed-off-by: Simone Carolini <simonecarolini.sc@gmail.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0137E9XmyUXhwPTRrtNfivDw
The write capabilities are a new contract surface the adapters will subclass, so the contract moves to 0.8.0 and every exact pin (runtime, three adapters, three image requirement files) moves with it. Signed-off-by: Simone Carolini <simonecarolini.sc@gmail.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0137E9XmyUXhwPTRrtNfivDw
…ique_key Arrow groups 0.0 and -0.0 as distinct, so the uniqueness check passed rows a warehouse would treat as one key, letting a duplicate reach _upsert. The check cannot import pyarrow to normalize, so it now rejects floating-point key columns outright, with a regression test; decimal keys are unaffected. Signed-off-by: Simone Carolini <simonecarolini.sc@gmail.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0137E9XmyUXhwPTRrtNfivDw
1161ec3 to
28b6402
Compare
| def test_table_columns_reports_names_and_nullability(self, adapter, schema): | ||
| table = self.new_table(adapter, schema, COLUMNS) | ||
| reported = sorted(adapter.table_columns(schema, table), key=lambda column: column.name) | ||
| expected = [LiveColumn("id", False), LiveColumn("name", False), LiveColumn("score", True)] | ||
| if reported != expected: | ||
| raise AssertionError(f"table_columns reported {reported}, expected {expected}") | ||
|
|
||
| def test_table_columns_of_a_missing_table_raises_lookup_error(self, adapter, schema): | ||
| self.new_table(adapter, schema, COLUMNS) # the schema exists; only the table is missing | ||
| with pytest.raises(LookupError): | ||
| adapter.table_columns(schema, f"conf_missing_{uuid.uuid4().hex[:8]}") |
There was a problem hiding this comment.
Why do we have tests within this module? This is the implementaion module.
Signed-off-by: Simone Carolini <simonecarolini.sc@gmail.com>
The engine-contract conformance suite moved to continuo_engine_contract.testing.conformance in #75; the postgres integration test still imported the pre-move path and failed collection once #75 merged to main. Signed-off-by: Simone Carolini <simonecarolini.sc@gmail.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0137E9XmyUXhwPTRrtNfivDw
* refactor(postgres): one insert helper and one transaction helper load now writes through _insert_rows and _transaction, which append and upsert will share: batched execute_values inserts naming the data's columns, with the existing percent escaping, inside an explicit BEGIN and COMMIT that rolls back on error. No behavior change. Signed-off-by: Simone Carolini <simonecarolini.sc@gmail.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0137E9XmyUXhwPTRrtNfivDw * feat(postgres): table_columns from information_schema PostgresAdapter declares TableIntrospection and reports each column's name and nullability, raising LookupError for a missing table. The engine-contract conformance suite runs against the postgres stack from here on. Signed-off-by: Simone Carolini <simonecarolini.sc@gmail.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0137E9XmyUXhwPTRrtNfivDw * feat(postgres): append PostgresAdapter declares AppendCapable: one explicit transaction of batched inserts through the shared insert helper; zero rows write nothing. Passes the append conformance cases, including atomicity across insert pages, quoted names and a different column order. Signed-off-by: Simone Carolini <simonecarolini.sc@gmail.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0137E9XmyUXhwPTRrtNfivDw * feat(postgres): ensure_table declares an upsert key as PRIMARY KEY A primary_key passed to ensure_table becomes the created table's enforced, indexed PRIMARY KEY; a key on an undeclared column is rejected before any DDL. Without a key the CREATE TABLE is byte-identical to before. Signed-off-by: Simone Carolini <simonecarolini.sc@gmail.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0137E9XmyUXhwPTRrtNfivDw * feat(postgres): upsert through INSERT ON CONFLICT PostgresAdapter declares UpsertCapable: one transaction of batched inserts with ON CONFLICT (key) DO UPDATE SET each non-key column from EXCLUDED, or DO NOTHING when every column is in the key. A table without the key constraint fails instead of collecting duplicates. Passes the upsert conformance cases. Signed-off-by: Simone Carolini <simonecarolini.sc@gmail.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0137E9XmyUXhwPTRrtNfivDw * fix(postgres): import conformance from the testing subpackage The engine-contract conformance suite moved to continuo_engine_contract.testing.conformance in #75; the postgres integration test still imported the pre-move path and failed collection once #75 merged to main. Signed-off-by: Simone Carolini <simonecarolini.sc@gmail.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0137E9XmyUXhwPTRrtNfivDw --------- Signed-off-by: Simone Carolini <simonecarolini.sc@gmail.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
The engine-contract conformance suite moved to continuo_engine_contract.testing.conformance in #75; the duckdb integration test still imported the pre-move path and failed collection once #75 merged to main. Signed-off-by: Simone Carolini <simonecarolini.sc@gmail.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0137E9XmyUXhwPTRrtNfivDw
* test(duckdb): pin that DuckLake reports NOT NULL through introspection The write-modes shape check compares nullability; duckdb_columns() and information_schema both report the NOT NULL that ensure_table declared on the stack's DuckLake (postgres catalog, MinIO data). Signed-off-by: Simone Carolini <simonecarolini.sc@gmail.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0137E9XmyUXhwPTRrtNfivDw * feat(duckdb): table_columns through LakeGateway.columns LakeWarehouse declares TableIntrospection. The new LakeGateway.columns reads duckdb_columns() with the same pre-filter-then-confirm identifier matching as table_exists, and table_columns raises LookupError for a missing table. The conformance suite runs against the DuckLake stack from here on. Signed-off-by: Simone Carolini <simonecarolini.sc@gmail.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0137E9XmyUXhwPTRrtNfivDw * feat(duckdb): append LakeWarehouse declares AppendCapable: insert_arrow inside one gateway transaction, nothing for zero rows. Passes the append conformance cases on the DuckLake stack. Signed-off-by: Simone Carolini <simonecarolini.sc@gmail.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0137E9XmyUXhwPTRrtNfivDw * feat(duckdb): MERGE from a registered Arrow view DdlRenderer.merge_from renders MERGE INTO the target on the key, overwriting the other columns of a matched row and inserting the rest, with no WHEN MATCHED branch when every column is in the key. LakeGateway.merge_arrow runs it from a registered view that is always unregistered; errors stay redacted. Signed-off-by: Simone Carolini <simonecarolini.sc@gmail.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0137E9XmyUXhwPTRrtNfivDw * feat(duckdb): upsert through MERGE LakeWarehouse declares UpsertCapable: a MERGE from the registered Arrow data inside one gateway transaction, nothing for zero rows. ensure_table accepts the key and ignores it because DuckLake has no constraints. Passes the upsert conformance cases on the DuckLake stack. Signed-off-by: Simone Carolini <simonecarolini.sc@gmail.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0137E9XmyUXhwPTRrtNfivDw * fix(duckdb): import conformance from the testing subpackage The engine-contract conformance suite moved to continuo_engine_contract.testing.conformance in #75; the duckdb integration test still imported the pre-move path and failed collection once #75 merged to main. Signed-off-by: Simone Carolini <simonecarolini.sc@gmail.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0137E9XmyUXhwPTRrtNfivDw * fix(duckdb): guard merge_from against an empty key An empty key renders `ON ()`, invalid SQL. UpsertCapable.upsert already rejects an empty key through ensure_unique_key before any engine is called, so this is unreachable in the port flow; the guard makes the renderer self-defending if it is ever called directly. Signed-off-by: Simone Carolini <simonecarolini.sc@gmail.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0137E9XmyUXhwPTRrtNfivDw * refactor(duckdb): absolute intra-package imports The duckdb adapter was the only package in the repo using relative intra-package imports (27 of them). The runtime, the engine contract and the postgres/trino adapters all use absolute imports. Standardise the duckdb adapter to match: every `from .` / `from ..` becomes `from continuo_duckdb_adapter...`. No behaviour change; imports resolve identically. Signed-off-by: Simone Carolini <simonecarolini.sc@gmail.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0137E9XmyUXhwPTRrtNfivDw --------- Signed-off-by: Simone Carolini <simonecarolini.sc@gmail.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
The engine-contract conformance suite moved to continuo_engine_contract.testing.conformance in #75; the trino integration test still imported the pre-move path and failed collection once #75 merged to main. Signed-off-by: Simone Carolini <simonecarolini.sc@gmail.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0137E9XmyUXhwPTRrtNfivDw
* test(trino): probe MERGE on Iceberg before the upsert relies on it Pins, on the Trino 483 plus Iceberg REST stack: MERGE from a stage table updates and inserts in one statement; an insert-only MERGE inserts only new keys; MERGE on a format_version 1 table fails and changes nothing; two stage rows matching one target row fail. Signed-off-by: Simone Carolini <simonecarolini.sc@gmail.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0137E9XmyUXhwPTRrtNfivDw * refactor(trino): stage helpers shared by every write path _create_stage and _drop_stage come out of load, beside _insert_batches: the stage is created LIKE the target at a fresh sibling location and dropped best-effort. load keeps its staged swap and recovery unchanged. Signed-off-by: Simone Carolini <simonecarolini.sc@gmail.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0137E9XmyUXhwPTRrtNfivDw * feat(trino): table_columns from information_schema TrinoAdapter declares TableIntrospection, looking up the lowercased schema and table Trino stores and raising LookupError for a missing table. The conformance suite runs against the Trino stack from here on. Signed-off-by: Simone Carolini <simonecarolini.sc@gmail.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0137E9XmyUXhwPTRrtNfivDw * feat(trino): append through a stage table TrinoAdapter declares AppendCapable. Rows are batch-inserted into a stage shaped like the target, then one INSERT SELECT writes them as a single Iceberg commit; the stage is dropped best-effort and a failed drop is only logged. Zero rows create no stage. Passes the append conformance cases. Signed-off-by: Simone Carolini <simonecarolini.sc@gmail.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0137E9XmyUXhwPTRrtNfivDw * feat(trino): ensure_table takes an upsert key and guards format_version The key declares nothing (Trino has no constraint syntax) but a key with config format_version below 2 is rejected before any DDL, because Trino's MERGE needs Iceberg format version 2. Signed-off-by: Simone Carolini <simonecarolini.sc@gmail.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0137E9XmyUXhwPTRrtNfivDw * feat(trino): upsert through one MERGE from a stage table TrinoAdapter declares UpsertCapable: the same stage as append, then one MERGE INTO target USING stage that overwrites the non-key columns of matched rows and inserts the rest, with no WHEN MATCHED branch when every column is in the key. Passes the upsert conformance cases on the Trino stack. Signed-off-by: Simone Carolini <simonecarolini.sc@gmail.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0137E9XmyUXhwPTRrtNfivDw * fix(trino): import conformance from the testing subpackage The engine-contract conformance suite moved to continuo_engine_contract.testing.conformance in #75; the trino integration test still imported the pre-move path and failed collection once #75 merged to main. Signed-off-by: Simone Carolini <simonecarolini.sc@gmail.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0137E9XmyUXhwPTRrtNfivDw --------- Signed-off-by: Simone Carolini <simonecarolini.sc@gmail.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Adds opt-in engine write capabilities and a reusable conformance suite so warehouse adapters share the same append and upsert requirements.
TableIntrospection,AppendCapable,UpsertCapable, andLiveColumnlive incontinuo_engine_contract.port. Declaring a capability requires implementing its abstract methods.UpsertCapable.upsertchecks key uniqueness before calling the engine's_upsert.continuo_engine_contract.keysprovidesensure_unique_keyandDuplicateKeyError. Floating-point primary-key columns are rejected because Arrow grouping does not match warehouse equality for signed zero.WarehouseAdapter.ensure_tableaccepts an optional keyword-onlyprimary_key.continuo_engine_contract.testing.conformanceships reusable pytest classes for adapter integration suites, withadapterandschemafixtures. Pytest and PyArrow are optional dependencies under thetestingextra. The contract suite exercises these classes against an in-memory reference adapter and broken-adapter controls; the packaging test checks the wheel includes the testing package.CLAUDE.mdrequires real-engine conformance checks in CI for declared capabilities and shared coverage for new engine-independent contract requirements. Engine-specific behavior stays in adapter tests.Validation: 163 contract tests passed in Docker; Ruff and mypy passed. The targeted Continuo Python release and execution end-to-end tests passed against the existing local stack, including five execution subtests. Independent review found no actionable issues.