Skip to content

feat(engine-contract): write capabilities, ensure_unique_key, conformance suite (contract 0.8.0) - #75

Merged
carolsimone merged 6 commits into
mainfrom
feat/write-modes-engine-port
Oct 10, 2026
Merged

carolsimone merged 6 commits into
mainfrom
feat/write-modes-engine-port

Conversation

@carolsimone

@carolsimone carolsimone commented Oct 9, 2026 •

Copy link
Copy Markdown
Owner

Adds opt-in engine write capabilities and a reusable conformance suite so warehouse adapters share the same append and upsert requirements.

  • TableIntrospection, AppendCapable, UpsertCapable, and LiveColumn live in continuo_engine_contract.port. Declaring a capability requires implementing its abstract methods. UpsertCapable.upsert checks key uniqueness before calling the engine's _upsert.
  • continuo_engine_contract.keys provides ensure_unique_key and DuplicateKeyError. Floating-point primary-key columns are rejected because Arrow grouping does not match warehouse equality for signed zero.
  • WarehouseAdapter.ensure_table accepts an optional keyword-only primary_key.
  • continuo_engine_contract.testing.conformance ships reusable pytest classes for adapter integration suites, with adapter and schema fixtures. Pytest and PyArrow are optional dependencies under the testing extra. The contract suite exercises these classes against an in-memory reference adapter and broken-adapter controls; the packaging test checks the wheel includes the testing package.
  • CLAUDE.md requires real-engine conformance checks in CI for declared capabilities and shared coverage for new engine-independent contract requirements. Engine-specific behavior stays in adapter tests.
  • The contract version is 0.8.0, with workspace and image requirement pins aligned.

Validation: 163 contract tests passed in Docker; Ruff and mypy passed. The targeted Continuo Python release and execution end-to-end tests passed against the existing local stack, including five execution subtests. Independent review found no actionable issues.

@carolsimone carolsimone self-assigned this Oct 9, 2026

@carolsimone carolsimone left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

High · bug: append/upsert still execute as replace (inherited from #74)

Location: continuo_python_runtime/contract/loader.py:290–299. This is inherited from the prerequisite branch, so it is posted here as a review note rather than attached to this PR's diff.

The loader accepts append and upsert, but harness.run_node() still unconditionally calls ensure_table() followed by load(), whose contract is to replace the table's contents. In reproductions using validated nodes and an in-memory adapter initially holding id=99, both modes returned success and left only id=1. Adapters without either opt-in capability also reach this path.

Until the capability-specific write dispatch is implemented, reject these modes in the harness before running the script or writing anything. Otherwise an accepted append/upsert contract can silently delete existing rows.

Comment on lines +53 to +60
counts = (
data.select(key)
.rename_columns(positional)
.group_by(positional, use_threads=False)
.aggregate([([], "count_all")])
)
if counts.num_rows == data.num_rows:
return

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

High · bug: signed-zero keys bypass the uniqueness check

Arrow groups 0.0 and -0.0 separately. Consequently ensure_unique_key(pa.table({'id': [0.0, -0.0]}), ['id']) returns successfully even though the warehouse compares these keys equal.

This input also passes conform() for a non-nullable DOUBLE PRECISION key. In an in-memory DuckDB reproduction, a MERGE into an empty table inserted both rows, and SQL GROUP BY id reported one key with count 2. The template method therefore allows duplicate warehouse keys to reach _upsert, violating the invariant this guard is meant to enforce.

Normalize floating key equality (including signed zero) before grouping, or reject unsupported floating key types, and add a regression test.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in b47e4da. ensure_unique_key now rejects floating-point key columns (halffloat/float/double) with a ValueError before grouping. Normalizing signed zero would need pyarrow.compute, but the contract package keeps pyarrow out of its runtime imports (sqlglot-only), so rejection is the right option — and ensure_unique_key is the single enforcement point the UpsertCapable.upsert template runs for every engine. Regression tests cover 0.0/-0.0, float32, a composite key with one float column, and that decimal128 keys still pass.

@carolsimone
carolsimone force-pushed the feat/write-modes-engine-port branch from f7f8e8d to b47e4da Compare October 9, 2026 15:16
@carolsimone

Copy link
Copy Markdown
Owner Author

The append/upsert-as-replace risk is now guarded: c587b88 (on the R1 base branch feat/write-modes-contract-fields) rejects any non-replace mode_type in run_node before the script runs or anything is written, with a test proving no write happens for either mode. It is an interim guard that R6 removes when it lands the per-mode _WRITERS dispatch + capability check and the row-preservation end-to-end tests, all before the single v0.9.0 tag. The signed-zero uniqueness hole is fixed in b47e4da.

Base automatically changed from feat/write-modes-contract-fields to main October 10, 2026 11:05
@carolsimone
carolsimone force-pushed the feat/write-modes-engine-port branch from b47e4da to 856dd3f Compare October 10, 2026 11:08
carolsimone and others added 5 commits October 10, 2026 14:42
Groups an Arrow table by its primary key and raises DuplicateKeyError (a
ValueError) naming the first repeated keys and their counts. Upsert calls it
before any engine sees the rows, because no engine handles repeated keys in
one write consistently.

Signed-off-by: Simone Carolini <simonecarolini.sc@gmail.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0137E9XmyUXhwPTRrtNfivDw
…mary_key

TableIntrospection, AppendCapable and UpsertCapable are abstract interfaces an
adapter declares by inheritance; LiveColumn is what table_columns reports.
UpsertCapable.upsert is a template method that checks key uniqueness before the
engine-specific _upsert. ensure_table gains an optional keyword-only
primary_key that the harness passes only for upsert nodes.

Signed-off-by: Simone Carolini <simonecarolini.sc@gmail.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0137E9XmyUXhwPTRrtNfivDw
IntrospectionConformance, AppendConformance and UpsertConformance hold the
spec's cases (accumulation, zero rows, atomicity across statements, overwrite,
idempotent rerun, repeated keys, every column in the key) plus quoting, column
order, first-run upsert and the FX key types. A reference in-memory adapter
runs them here, with two broken adapters as negative controls. Shipped in the
wheel; pytest and pyarrow come with the testing extra.

Signed-off-by: Simone Carolini <simonecarolini.sc@gmail.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0137E9XmyUXhwPTRrtNfivDw
The write capabilities are a new contract surface the adapters will subclass,
so the contract moves to 0.8.0 and every exact pin (runtime, three adapters,
three image requirement files) moves with it.

Signed-off-by: Simone Carolini <simonecarolini.sc@gmail.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0137E9XmyUXhwPTRrtNfivDw
…ique_key

Arrow groups 0.0 and -0.0 as distinct, so the uniqueness check passed rows a
warehouse would treat as one key, letting a duplicate reach _upsert. The check
cannot import pyarrow to normalize, so it now rejects floating-point key
columns outright, with a regression test; decimal keys are unaffected.

Signed-off-by: Simone Carolini <simonecarolini.sc@gmail.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0137E9XmyUXhwPTRrtNfivDw
@carolsimone
carolsimone force-pushed the feat/write-modes-engine-port branch from 1161ec3 to 28b6402 Compare October 10, 2026 12:43
Comment on lines +149 to +159
def test_table_columns_reports_names_and_nullability(self, adapter, schema):
table = self.new_table(adapter, schema, COLUMNS)
reported = sorted(adapter.table_columns(schema, table), key=lambda column: column.name)
expected = [LiveColumn("id", False), LiveColumn("name", False), LiveColumn("score", True)]
if reported != expected:
raise AssertionError(f"table_columns reported {reported}, expected {expected}")

def test_table_columns_of_a_missing_table_raises_lookup_error(self, adapter, schema):
self.new_table(adapter, schema, COLUMNS) # the schema exists; only the table is missing
with pytest.raises(LookupError):
adapter.table_columns(schema, f"conf_missing_{uuid.uuid4().hex[:8]}")

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we have tests within this module? This is the implementaion module.

Signed-off-by: Simone Carolini <simonecarolini.sc@gmail.com>
@carolsimone
carolsimone merged commit c48ac67 into main Oct 10, 2026
28 checks passed
@carolsimone
carolsimone deleted the feat/write-modes-engine-port branch October 10, 2026 14:24
carolsimone added a commit that referenced this pull request Oct 10, 2026
The engine-contract conformance suite moved to
continuo_engine_contract.testing.conformance in #75; the postgres
integration test still imported the pre-move path and failed collection
once #75 merged to main.

Signed-off-by: Simone Carolini <simonecarolini.sc@gmail.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0137E9XmyUXhwPTRrtNfivDw
carolsimone added a commit that referenced this pull request Oct 10, 2026
* refactor(postgres): one insert helper and one transaction helper

load now writes through _insert_rows and _transaction, which append and
upsert will share: batched execute_values inserts naming the data's columns,
with the existing percent escaping, inside an explicit BEGIN and COMMIT that
rolls back on error. No behavior change.

Signed-off-by: Simone Carolini <simonecarolini.sc@gmail.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0137E9XmyUXhwPTRrtNfivDw

* feat(postgres): table_columns from information_schema

PostgresAdapter declares TableIntrospection and reports each column's name
and nullability, raising LookupError for a missing table. The engine-contract
conformance suite runs against the postgres stack from here on.

Signed-off-by: Simone Carolini <simonecarolini.sc@gmail.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0137E9XmyUXhwPTRrtNfivDw

* feat(postgres): append

PostgresAdapter declares AppendCapable: one explicit transaction of batched
inserts through the shared insert helper; zero rows write nothing. Passes the
append conformance cases, including atomicity across insert pages, quoted
names and a different column order.

Signed-off-by: Simone Carolini <simonecarolini.sc@gmail.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0137E9XmyUXhwPTRrtNfivDw

* feat(postgres): ensure_table declares an upsert key as PRIMARY KEY

A primary_key passed to ensure_table becomes the created table's enforced,
indexed PRIMARY KEY; a key on an undeclared column is rejected before any DDL.
Without a key the CREATE TABLE is byte-identical to before.

Signed-off-by: Simone Carolini <simonecarolini.sc@gmail.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0137E9XmyUXhwPTRrtNfivDw

* feat(postgres): upsert through INSERT ON CONFLICT

PostgresAdapter declares UpsertCapable: one transaction of batched inserts
with ON CONFLICT (key) DO UPDATE SET each non-key column from EXCLUDED, or DO
NOTHING when every column is in the key. A table without the key constraint
fails instead of collecting duplicates. Passes the upsert conformance cases.

Signed-off-by: Simone Carolini <simonecarolini.sc@gmail.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0137E9XmyUXhwPTRrtNfivDw

* fix(postgres): import conformance from the testing subpackage

The engine-contract conformance suite moved to
continuo_engine_contract.testing.conformance in #75; the postgres
integration test still imported the pre-move path and failed collection
once #75 merged to main.

Signed-off-by: Simone Carolini <simonecarolini.sc@gmail.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0137E9XmyUXhwPTRrtNfivDw

---------

Signed-off-by: Simone Carolini <simonecarolini.sc@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
carolsimone added a commit that referenced this pull request Oct 10, 2026
The engine-contract conformance suite moved to
continuo_engine_contract.testing.conformance in #75; the duckdb
integration test still imported the pre-move path and failed collection
once #75 merged to main.

Signed-off-by: Simone Carolini <simonecarolini.sc@gmail.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0137E9XmyUXhwPTRrtNfivDw
carolsimone added a commit that referenced this pull request Oct 11, 2026
* test(duckdb): pin that DuckLake reports NOT NULL through introspection

The write-modes shape check compares nullability; duckdb_columns() and
information_schema both report the NOT NULL that ensure_table declared on the
stack's DuckLake (postgres catalog, MinIO data).

Signed-off-by: Simone Carolini <simonecarolini.sc@gmail.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0137E9XmyUXhwPTRrtNfivDw

* feat(duckdb): table_columns through LakeGateway.columns

LakeWarehouse declares TableIntrospection. The new LakeGateway.columns reads
duckdb_columns() with the same pre-filter-then-confirm identifier matching as
table_exists, and table_columns raises LookupError for a missing table. The
conformance suite runs against the DuckLake stack from here on.

Signed-off-by: Simone Carolini <simonecarolini.sc@gmail.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0137E9XmyUXhwPTRrtNfivDw

* feat(duckdb): append

LakeWarehouse declares AppendCapable: insert_arrow inside one gateway
transaction, nothing for zero rows. Passes the append conformance cases on
the DuckLake stack.

Signed-off-by: Simone Carolini <simonecarolini.sc@gmail.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0137E9XmyUXhwPTRrtNfivDw

* feat(duckdb): MERGE from a registered Arrow view

DdlRenderer.merge_from renders MERGE INTO the target on the key, overwriting
the other columns of a matched row and inserting the rest, with no WHEN
MATCHED branch when every column is in the key. LakeGateway.merge_arrow runs
it from a registered view that is always unregistered; errors stay redacted.

Signed-off-by: Simone Carolini <simonecarolini.sc@gmail.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0137E9XmyUXhwPTRrtNfivDw

* feat(duckdb): upsert through MERGE

LakeWarehouse declares UpsertCapable: a MERGE from the registered Arrow data
inside one gateway transaction, nothing for zero rows. ensure_table accepts
the key and ignores it because DuckLake has no constraints. Passes the upsert
conformance cases on the DuckLake stack.

Signed-off-by: Simone Carolini <simonecarolini.sc@gmail.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0137E9XmyUXhwPTRrtNfivDw

* fix(duckdb): import conformance from the testing subpackage

The engine-contract conformance suite moved to
continuo_engine_contract.testing.conformance in #75; the duckdb
integration test still imported the pre-move path and failed collection
once #75 merged to main.

Signed-off-by: Simone Carolini <simonecarolini.sc@gmail.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0137E9XmyUXhwPTRrtNfivDw

* fix(duckdb): guard merge_from against an empty key

An empty key renders `ON ()`, invalid SQL. UpsertCapable.upsert already
rejects an empty key through ensure_unique_key before any engine is
called, so this is unreachable in the port flow; the guard makes the
renderer self-defending if it is ever called directly.

Signed-off-by: Simone Carolini <simonecarolini.sc@gmail.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0137E9XmyUXhwPTRrtNfivDw

* refactor(duckdb): absolute intra-package imports

The duckdb adapter was the only package in the repo using relative
intra-package imports (27 of them). The runtime, the engine contract and
the postgres/trino adapters all use absolute imports. Standardise the
duckdb adapter to match: every `from .` / `from ..` becomes
`from continuo_duckdb_adapter...`. No behaviour change; imports resolve
identically.

Signed-off-by: Simone Carolini <simonecarolini.sc@gmail.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0137E9XmyUXhwPTRrtNfivDw

---------

Signed-off-by: Simone Carolini <simonecarolini.sc@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
carolsimone added a commit that referenced this pull request Oct 11, 2026
The engine-contract conformance suite moved to
continuo_engine_contract.testing.conformance in #75; the trino
integration test still imported the pre-move path and failed collection
once #75 merged to main.

Signed-off-by: Simone Carolini <simonecarolini.sc@gmail.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0137E9XmyUXhwPTRrtNfivDw
carolsimone added a commit that referenced this pull request Oct 11, 2026
* test(trino): probe MERGE on Iceberg before the upsert relies on it

Pins, on the Trino 483 plus Iceberg REST stack: MERGE from a stage table
updates and inserts in one statement; an insert-only MERGE inserts only new
keys; MERGE on a format_version 1 table fails and changes nothing; two stage
rows matching one target row fail.

Signed-off-by: Simone Carolini <simonecarolini.sc@gmail.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0137E9XmyUXhwPTRrtNfivDw

* refactor(trino): stage helpers shared by every write path

_create_stage and _drop_stage come out of load, beside _insert_batches: the
stage is created LIKE the target at a fresh sibling location and dropped
best-effort. load keeps its staged swap and recovery unchanged.

Signed-off-by: Simone Carolini <simonecarolini.sc@gmail.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0137E9XmyUXhwPTRrtNfivDw

* feat(trino): table_columns from information_schema

TrinoAdapter declares TableIntrospection, looking up the lowercased schema and
table Trino stores and raising LookupError for a missing table. The
conformance suite runs against the Trino stack from here on.

Signed-off-by: Simone Carolini <simonecarolini.sc@gmail.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0137E9XmyUXhwPTRrtNfivDw

* feat(trino): append through a stage table

TrinoAdapter declares AppendCapable. Rows are batch-inserted into a stage
shaped like the target, then one INSERT SELECT writes them as a single Iceberg
commit; the stage is dropped best-effort and a failed drop is only logged.
Zero rows create no stage. Passes the append conformance cases.

Signed-off-by: Simone Carolini <simonecarolini.sc@gmail.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0137E9XmyUXhwPTRrtNfivDw

* feat(trino): ensure_table takes an upsert key and guards format_version

The key declares nothing (Trino has no constraint syntax) but a key with
config format_version below 2 is rejected before any DDL, because Trino's
MERGE needs Iceberg format version 2.

Signed-off-by: Simone Carolini <simonecarolini.sc@gmail.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0137E9XmyUXhwPTRrtNfivDw

* feat(trino): upsert through one MERGE from a stage table

TrinoAdapter declares UpsertCapable: the same stage as append, then one
MERGE INTO target USING stage that overwrites the non-key columns of matched
rows and inserts the rest, with no WHEN MATCHED branch when every column is
in the key. Passes the upsert conformance cases on the Trino stack.

Signed-off-by: Simone Carolini <simonecarolini.sc@gmail.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0137E9XmyUXhwPTRrtNfivDw

* fix(trino): import conformance from the testing subpackage

The engine-contract conformance suite moved to
continuo_engine_contract.testing.conformance in #75; the trino
integration test still imported the pre-move path and failed collection
once #75 merged to main.

Signed-off-by: Simone Carolini <simonecarolini.sc@gmail.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0137E9XmyUXhwPTRrtNfivDw

---------

Signed-off-by: Simone Carolini <simonecarolini.sc@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant