Skip to content

feat: database abstraction layer for Oracle 23ai support#947

Open
DK09876 wants to merge 35 commits intomainfrom
feature/database-abstraction
Open

feat: database abstraction layer for Oracle 23ai support#947
DK09876 wants to merge 35 commits intomainfrom
feature/database-abstraction

Conversation

@DK09876
Copy link
Copy Markdown
Contributor

@DK09876 DK09876 commented Apr 9, 2026 •
edited
Loading

Summary

  • Introduces DatabaseBackend / DatabaseConnection ABCs and SQLDialect ABC to decouple the engine from asyncpg/PostgreSQL, enabling Oracle 23ai as a second database backend
  • Full Oracle 23ai support: migrations, HNSW vector search, Oracle Text BM25, entity resolution, graph retrieval
  • All retrieval SQL moved into dialect abstraction — no more inline _is_pg checks
  • 42 files changed, 7665 insertions, 793 deletions across 6 commits

What works on Oracle

  • Retain (single-item and batch sync) — full fact extraction, entity resolution, chunking, embeddings
  • Recall — all 4 retrieval strategies: semantic (HNSW), BM25 (Oracle Text CONTAINS/SCORE), graph (link expansion), temporal
  • Reflect — disposition-aware reasoning using recalled memories
  • Bank/memory/document CRUD — all endpoints
  • Mental model creation — works (refresh requires async worker)
  • MCP — all tools functional
  • Entities — extraction, linking, cooccurrences all stored correctly

Benchmark results (2,000 items, cross-encoder reranker, 10 iterations)

Metric PostgreSQL (pg0 native) Oracle 23ai (Docker) Ratio
Mean 0.311s 0.408s 1.31x
p50 0.254s 0.356s 1.40x
p95 0.614s 0.743s 1.21x

Note: PG runs against pg0 (native/embedded) while Oracle runs in Docker, so the gap includes Docker virtualization overhead. CPU-bound phases (reranking, embedding) are identical across both backends. An apples-to-apples comparison (both in Docker or both native) would narrow the difference.

Remaining work (follow-up PRs)

Decouple worker poller from asyncpg

The worker poller (worker/poller.py) currently uses raw asyncpg pool methods (pool.fetch, pool.execute, conn.transaction()) instead of going through the DatabaseBackend abstraction. This needs to be refactored to use the backend interface — python-oracledb has full async support (create_pool_async()) and Oracle supports SELECT ... FOR UPDATE SKIP LOCKED for the same atomic claiming semantics. Until then, Oracle must run with HINDSIGHT_API_WORKER_ENABLED=false. Blocks: async/batch retain, background consolidation, async mental model refresh. Estimate: 2-3 days.

Port entity resolver fuzzy matching to dialect layer

_resolve_entities_batch_oracle_fuzzy still uses PG-only unnest($2::text[]) AS q(query_text) syntax. Oracle equivalents exist (JSON_TABLE or TABLE() for unnest, UTL_MATCH.JARO_WINKLER_SIMILARITY for fuzzy scoring). Currently falls back to full entity scan (correct but slower for large banks). Estimate: 1-2 days.

Refactor source_memory_ids array queries

10+ queries across memory_engine.py and link_expansion_retrieval.py use PG array operators (ANY(), &&, unnest()) on source_memory_ids. Two approaches: (A) switch Oracle column from CLOB to native JSON type and use JSON_EXISTS/JSON_TABLE, or (B) normalize into a junction table (fact_source_memories) which would simplify both backends and eliminate the dialect-specific rewrites entirely. Blocks include_source_facts and include_chunks in recall. Estimate: 3-5 days.

Per-bank vector index isolation

Oracle doesn't support partial indexes on the VECTOR type, but the idiomatic Oracle approach is list partitioning by bank_id with a local HNSW index per partition — this gives equivalent tenant isolation plus partition pruning at query time. Currently uses a single global HNSW index (functional, not optimal). Estimate: 2-3 days including benchmarking.

Extend bulk insert to last remaining site

2 of 3 insert sites already have Oracle bulk paths via executemany in db/ops_oracle.py. fact_storage.py still uses row-by-row fallback. Straightforward to extend the existing pattern. Estimate: < 1 day.

Key design decisions

  • Escape hatch: self._pool = backend.get_pool() keeps all existing PG consumers working unchanged during transition
  • No behavior change for PG: Pure refactor for PostgreSQL users — all existing SQL untouched
  • Lazy oracledb import: Oracle backend only imports python-oracledb when selected, no hard dependency
  • Dialect pattern: build_semantic_arm() / build_bm25_arm() / prepare_bm25_text() build complete UNION ALL subquery arms per database, eliminating all inline backend checks from retrieval.py
  • RewriteResult NamedTuple: Replaced bare tuple returns in oracle.py per project coding standards

Test plan

  • 52 new unit tests for ResultRow, factories, dialect methods, config integration
  • 12 new tests for dialect arm builders (PostgreSQL and Oracle)
  • Oracle integration test stubs (856 lines) + HTTP integration stubs (329 lines)
  • pg_trgm auto-detection tests fixed (MagicMock backend_type issue)
  • Migration idempotency fixes verified (h3c4d5e6f7g8, o0j1k2l3m4n5)
  • Full test suite: 1479 passed, 15 flaky (pre-existing, not related to this PR)
  • Manual E2E: retain → recall → reflect on Oracle 23ai Docker container
  • Direct Oracle DB verification: all tables populated correctly (11 facts, 15 entities, 15 links, 9 cooccurrences, 4 chunks, 2 documents)
  • Lint passes (ruff check)

🤖 Generated with Claude Code

@DK09876 DK09876 force-pushed the feature/database-abstraction branch from ba92cd7 to 8cd903d Compare April 14, 2026 14:32
nicoloboschi
nicoloboschi reviewed Apr 14, 2026
View reviewed changes
Comment thread hindsight-api-slim/hindsight_api/engine/search/retrieval.py Outdated
f" ORDER BY embedding <=> $1::vector"
f" LIMIT {hnsw_fetch})"
)
if not _is_pg:
Copy link
Copy Markdown
Collaborator

@nicoloboschi nicoloboschi Apr 14, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we should move this query code to a function of the abstraction and every impl has his own syntax.

all this if not _is_pg: are going to be a mess to maintain

@DK09876 DK09876 marked this pull request as ready for review April 15, 2026 20:29
DK09876 and others added 6 commits April 17, 2026 06:50
@DK09876 @claude
feat: add database abstraction layer for multi-backend support
cdc36e9 
Introduce DatabaseBackend ABC and SQLDialect ABC to decouple the engine
from asyncpg/PostgreSQL, enabling Oracle 23ai as a second database platform.

- engine/db/: DatabaseBackend (pool lifecycle), DatabaseConnection (query execution), ResultRow (uniform row access)
- engine/db/postgresql.py: asyncpg implementation
- engine/db/oracle.py: python-oracledb implementation (thin mode)
- engine/sql/: SQLDialect ABC with 20+ dialect methods (params, vector ops, JSON, upsert, FTS, etc.)
- engine/sql/postgresql.py: PG dialect ($N, <=>, @>, unnest, ON CONFLICT, etc.)
- engine/sql/oracle.py: Oracle dialect (:N, MERGE INTO, VECTOR_DISTANCE, JSON_MERGEPATCH, etc.)
- config.py: HINDSIGHT_API_DATABASE_BACKEND env var (default: postgresql)
- memory_engine.py: uses DatabaseBackend.initialize() instead of raw asyncpg.create_pool(); exposes get_pool() for backward compat
- 52 unit tests covering all abstractions

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@DK09876 @claude
test: add Oracle backend integration stubs against real Oracle 23ai
e03244f 
22 tests exercising OracleBackend + OracleDialect abstractions:
- Pool lifecycle (initialize, acquire, shutdown)
- Transaction commit/rollback semantics
- All DatabaseConnection methods (execute, executemany, fetch, fetchrow, fetchval)
- ResultRow dict-like access on real Oracle rows
- Vector insert + VECTOR_DISTANCE cosine search
- JSON insert/extract/MERGEPATCH
- MERGE INTO upsert via dialect
- ILIKE via UPPER() dialect
- UTL_MATCH fuzzy similarity via dialect
- FOR UPDATE SKIP LOCKED
- OFFSET/FETCH FIRST pagination via dialect
- Concurrent pool operations
- Dialect SQL fragments accepted by real Oracle (SYS_GUID, SYSTIMESTAMP, GREATEST)

All skip cleanly when ORACLE_TEST_DSN is not set.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@DK09876 @claude
feat: migrate all consumers to DatabaseBackend abstraction
c4cbfcd 
Replace raw asyncpg pool usage across all SQL-executing consumers with
the DatabaseBackend interface, enabling Oracle 23ai support without
changing business logic.

Key changes:
- memory_engine.py passes self._backend instead of self._pool to all
  consumers (orchestrator, entity_resolver, bank_utils, config_resolver,
  webhooks, file storage, task backend, audit logger)
- BudgetedPool updated to handle both DatabaseBackend and raw pool,
  with _wraps_backend marker for acquire_with_retry dispatch
- acquire_with_retry recognizes BudgetedPool via _wraps_backend attr
- WebhookManager uses conn-based queries instead of pool convenience methods
- consolidator, storage, task_backend, audit use acquire_with_retry
  instead of direct pool.acquire()/pool.execute()
- Oracle backend: transparent query rewriting ($N->:N, strip ::casts,
  NOW()->SYSTIMESTAMP), transaction support via savepoints, cursor fixes
- DatabaseConnection.copy_records_to_table with asyncpg native COPY
  override and executemany INSERT fallback for Oracle
- Oracle query rewriter tests and backend integration test fixes

1344 tests pass; all failures are Groq API rate limiting, not migration.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@DK09876 @claude
feat: separate Oracle and PostgreSQL code paths completely
cbe24be 
Move all Oracle-specific logic behind conn.backend_type dispatch so the
PostgreSQL path is entirely untouched when running on PG.

Key changes:
- engine/schema.py: centralized fq_table/fq_table_explicit with Oracle
  awareness (Oracle skips schema prefix since it uses ALTER SESSION)
- engine/db/base.py: backend_type property, bulk_insert_from_arrays
  method, capability flags (supports_partial_indexes, supports_bm25,
  supports_unnest, supports_pg_trgm) on DatabaseConnection/Backend
- engine/db/oracle.py: massive expansion (+782) with transparent
  PG→Oracle query rewriting, RETURNING INTO handling, UUID RAW(16)
  conversion, ANY() expansion, array-contains rewriting, executemany
  bulk insert
- entity_resolver, link_utils, fact_storage, chunk_storage, bank_utils:
  Oracle fallback paths using conn.backend_type dispatch
- search/retrieval, link_expansion_retrieval: Oracle-aware search CTE
  dispatch (no BM25, different vector distance syntax)
- task_backend, storage/postgresql, webhooks/manager, worker/poller,
  admin/cli: delegate to schema.py for table qualification
- api/http.py: use backend instead of raw pool throughout
- migrations_oracle.py: stub for Oracle DDL migrations
- tests/test_db_abstraction.py: updated for tuple return from
  _rewrite_pg_to_oracle

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@DK09876 @claude
feat: complete Oracle 23ai integration — migrations, graph retrieval,…
a9eb064 
… tests

Implement full Oracle DDL migrations (13 tables, 30+ indexes, HNSW vector
index, Oracle Text index), Oracle-specific graph retrieval using JSON_TABLE
for observation expansion, and query rewriter fixes for JSONB boolean
patterns and quoted identifiers.

Key changes:
- migrations_oracle.py: full idempotent DDL replacing the stub
- link_expansion_retrieval.py: Oracle graph retrieval via JSON_TABLE/ROW_NUMBER
- oracle.py: JSONB boolean rewrite, CTE MATERIALIZED strip, COALESCE fix
- retrieval.py: subquery alias fix for Oracle derived tables
- 49 Oracle integration tests + 10 HTTP integration tests (all passing)
- conftest.py: Oracle test fixtures, fixed import ordering

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@DK09876 @claude
refactor: move retrieval SQL into dialect abstraction and fix migrations
4b57ee7 
Per PR review feedback, moved backend-specific retrieval query SQL from
inline if/else blocks in retrieval.py into build_semantic_arm(),
build_bm25_arm(), and prepare_bm25_text() methods on the SQLDialect ABC.
Each database (PostgreSQL, Oracle) now owns its own query arm construction,
eliminating all _is_pg checks from retrieval.py.

Also fixes two migration bugs:
- h3c4d5e6f7g8: CREATE TABLE IF NOT EXISTS was a no-op when mental_models
  already existed from the reflections rename chain, so v4 columns (subtype,
  description, etc.) were never added. Added idempotent ALTER TABLE ADD
  COLUMN IF NOT EXISTS for each column.
- o0j1k2l3m4n5: CHECK constraint restricted subtype to 'directive' only,
  but memory_engine.py creates mental models with subtype='pinned'. Updated
  constraint to allow both.

Other fixes:
- Added session-scoped test cleanup to drop accumulated per-bank vector
  indexes and truncate stale data from pg0 between test runs
- Fixed pg_trgm tests: MagicMock conn needs explicit backend_type="postgresql"
  to prevent Oracle dispatch path from triggering
- Added RewriteResult NamedTuple to replace bare tuple return from
  _rewrite_pg_to_oracle (per project coding standards)
- Added type hints to _resolve_entities_batch_oracle_fuzzy

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@DK09876 DK09876 force-pushed the feature/database-abstraction branch from 0535fcc to 4b57ee7 Compare April 17, 2026 13:53
DK09876 and others added 2 commits April 17, 2026 16:31
@DK09876 @claude
refactor: centralize DataAccessOps via backend.ops pattern and fix te…
9987fa2 
…st flakiness

Thread ops through the retain pipeline (orchestrator → link_creation →
link_utils) via DatabaseBackend.ops property, eliminating scattered
create_data_access_ops() calls. Fix 14 test failures caused by LLM
non-determinism (mock deterministic facts), stale pg0 state (stamp
migrations), missing backend.ops access (use DatabaseBackend instead
of raw Pool), and incorrect default assertions.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@DK09876 @claude
ci: add Oracle 23ai test job gated behind PR label
6d10690 
Add test-api-oracle job that mirrors test-api but runs against an
Oracle 23ai Free container service. Only runs when the PR has the
"oracle-tests" label, so it's off by default and doesn't affect
normal CI. Uses pytest -m oracle marker to run Oracle-specific tests.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@DK09876 DK09876 added the oracle-tests Enable Oracle 23ai integration tests in CI label Apr 17, 2026
DK09876 and others added 17 commits April 17, 2026 20:51
@DK09876 @claude
fix(migrations): backfill mental_models.subtype for existing databases
390dc0f 
Migration h3c4d5e6f7g8 used CREATE TABLE IF NOT EXISTS which was a
no-op on databases where mental_models already existed from the
reflections rename chain. The fix added to h3c4d5e6f7g8 (Step 4b)
only helps fresh databases — existing ones already have the migration
stamped as applied. This new migration adds the missing columns
idempotently for databases in that state.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@DK09876 @claude
chore: regenerate openapi.json for v0.5.3 version bump
0d01289 
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@DK09876 @claude
fix: backfill mental_models.subtype migration and clean up Oracle CI
f8043a2 
- Add migration d5y6z7a8b9c0 to idempotently add missing columns to
  mental_models table for databases where h3c4d5e6f7g8 was already
  stamped as applied before the fix
- Remove redundant "Wait for Oracle" step in CI — services health
  check already ensures Oracle is ready before steps run

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@DK09876 @claude
fix: add oracledb to optional dependencies for Oracle CI job
800acf7 
The test-api-oracle CI job was failing because oracledb wasn't installed.
Added it as an optional 'oracle' dependency group and regenerated the
lockfile so --all-extras picks it up.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@DK09876 @claude
fix: create ASSM tablespace for Oracle CI to support VECTOR type
01efccd 
Oracle's VECTOR type requires automatic segment space management (ASSM).
The SYSTEM tablespace uses manual SSM, causing ORA-43853 on table creation.
Fix: create a dedicated tablespace with ASSM and a test user before running
Oracle tests. Also fix DSN format to use path-based service name.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@DK09876 @claude
fix: add missing Oracle DDL columns and tighten test assertions
d8a7d12 
The mental_models table in Oracle migrations was missing
structured_content and last_refreshed_source_query columns that
memory_engine.py actively reads/writes. This caused mental model
create/refresh to fail on Oracle.

The Oracle HTTP test for mental models was asserting
`status in (200, 500)` which silently accepted this failure.
Tightened to assert 200 so missing columns can't hide behind
lenient assertions again.

Also documented known PG-only code paths (task_backend, poller,
webhooks, audit, config_resolver) in the Oracle backend docstring.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@DK09876 @claude
fix: disable worker/poller on Oracle backend to prevent startup crash
de3cc81 
WorkerPoller and BrokerTaskBackend still use raw asyncpg pool APIs and
PG-specific SQL (FOR UPDATE SKIP LOCKED, ::jsonb, NOW(), etc). Starting
them on an Oracle backend would crash immediately.

Guard both the embedded worker (API startup) and standalone worker
(hindsight-worker command) to skip/exit on Oracle. Operations that
normally go through the async worker will run synchronously instead.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@DK09876 @claude
refactor: clean up abstraction layer code quality
c8744e7 
- Move _get_mu_table() from PG/Oracle ops duplicates into DataAccessOps
  base class (DRY)
- Remove unnecessary getattr() fallback for conn.backend_type in
  retrieval.py — it's a guaranteed property on DatabaseConnection
- Promote _oracle_special set to _ORACLE_TEXT_SPECIAL frozenset class
  constant on OracleDialect (avoid recreation per call)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@DK09876 @claude
fix: use SyncTaskBackend on Oracle so async operations execute inline
9f4ccec 
BrokerTaskBackend queues tasks into async_operations for the WorkerPoller
to pick up, but the poller is disabled on Oracle (it uses raw asyncpg).
This meant mental model refresh, consolidation, and async retain would
be queued as "pending" but never executed.

Fix: use SyncTaskBackend on Oracle, which executes tasks inline in the
request. This matches what the test fixtures already do and ensures all
operations complete on Oracle, just synchronously rather than in the
background.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@DK09876 @claude
fix: resolve PG regression, harden Oracle rewriter, add e2e lifecycle…
00823e8 
… test

- Restore getattr(conn, "backend_type", "postgresql") in retrieval.py
  so raw asyncpg connections (used in test_hnsw_indexes) don't crash
- Guard RETURNING→INTO rewrite to only apply on INSERT/UPDATE/DELETE,
  preventing false matches on SELECT queries
- Add _safe_cleanup helpers in Oracle tests to suppress ORA-00060
  deadlock errors during test teardown
- Add TestOracleEndToEnd::test_full_lifecycle that verifies the complete
  retain→recall→reflect→mental model refresh flow, checking final state
  (not just HTTP 200) to catch issues like the SyncTaskBackend bug

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@DK09876 @claude
fix: bind JSON-serialized lists/dicts as CLOB for Oracle
3c78f53 
Oracle's thin driver defaults short strings like '[]' to VARCHAR2,
which fails with ORA-00932 when the target column is CLOB (e.g. tags,
metadata). Detect JSON-shaped strings in _make_bind_params and use
setinputsizes to explicitly type them as DB_TYPE_CLOB.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@DK09876 @claude
fix: apply CLOB input sizes after param expansion to avoid DPY-4008
d521572 
setinputsizes must run after _expand_any_lists — otherwise it registers
types for params that get removed during expansion, causing DPY-4008
"no bind placeholder" errors. Move CLOB typing to a separate method
called after expansion in each DML path.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@DK09876 @claude
fix: Oracle CLOB literals, Text reserved words, JSON parsing, FOR SHARE
1dcffc6 
- Rewrite COALESCE(col, '[]'::jsonb) || :N::jsonb to use
  JSON_MERGEPATCH with TO_CLOB for CLOB column compatibility
- Escape Oracle Text reserved words (about, near, not, etc.) with
  curly braces in CONTAINS queries to prevent DRG-50901 parse errors
- Guard json.loads() calls for result_metadata/task_payload to handle
  Oracle's pre-parsed JSON dicts alongside PG strings
- Rewrite FOR SHARE → FOR UPDATE (Oracle doesn't support FOR SHARE)
- Fix e2e test: replace nonexistent DELETE memory endpoint with list
  assertion, use substantive content for reliable fact extraction

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@DK09876 @claude
style: apply ruff formatting to oracle.py and memory_engine.py
d9e86af 
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@DK09876 @claude
fix: Oracle test deadlocks and trigger column quoting
bb3b3e4 
- Run Oracle CI tests sequentially (-n0) to prevent ORA-00060 deadlocks
  from concurrent transactions against the same Oracle Free container.
- Fix trigger column quoting: previous guard skipped quoting bare
  `trigger` in SELECT when `"trigger"` already appeared in JSON_VALUE.
  Use negative lookbehind/lookahead to quote only unquoted occurrences.
- Use direct delete_bank (not _safe_cleanup) in test_retain_and_delete_cycle
  so mid-test bank deletion failures aren't silently swallowed.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@DK09876 @claude
refactor: eliminate if-oracle/else-pg branches from business logic
98333df 
Replace 6 explicit backend-type string checks in business logic with
polymorphic dispatch through DatabaseBackend methods:

- api/http.py: use backend.supports_worker_poller instead of config check
- worker/main.py: use backend.supports_worker_poller instead of config check
- memory_engine.py: use backend.run_migrations() for migration dispatch
- memory_engine.py: use backend.create_task_backend() for task backend selection
- memory_engine.py: use conn.parse_json() for JSON column normalization (6 sites)
- config.py: type database_backend as Literal["postgresql", "oracle"]
- db/__init__.py: refactor factories to use helper functions
- db/base.py: add supports_worker_poller, run_migrations(), create_task_backend(),
  parse_json() to DatabaseBackend/DatabaseConnection ABCs

No backend-type string checks remain in business logic files. They only
exist in the db/ and sql/ abstraction layer factories where they belong.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@DK09876 @claude
Merge origin/main into feature/database-abstraction
cde955f 
Resolves conflicts in:
- memory_engine.py: integrated statement_timeout init callback with
  backend.initialize() abstraction
- retrieval.py: added extra_where param to dialect build_semantic_arm()
  and build_bm25_arm() methods to support created_after/created_before
  time range filtering from main
- openapi.json: took main's version (0.5.4)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
DK09876 and others added 4 commits April 22, 2026 13:27
@DK09876 @claude
fix: resolve merge issues — Oracle upsert regex, alembic heads, schem…
71045c3 
…a safety

1. Oracle rewriter: fix _UPSERT_RE regex to handle nested function calls
   in VALUES clause (e.g. COALESCE($7, NOW())). The previous regex used
   [^)]+ which broke on nested parentheses. Also added
   _split_respecting_parens() to correctly split values with nested calls.

2. Alembic: add merge migration e6f7g8h9i0j1 to unify two heads
   (8c6fa6f7230b from main, d5y6z7a8b9c0 from oracle branch).

3. task_backend.py: rephrase docstring to avoid false positive from
   test_sql_schema_safety (matched "INSERTed into async_operations").

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@DK09876 @claude
fix: Oracle MERGE type mismatch and result_metadata JSON parsing
9c1d6e3 
1. Oracle COALESCE + SYSTIMESTAMP type mismatch: when a NULL datetime
   bind param appears in COALESCE(:N, SYSTIMESTAMP), Oracle's thin driver
   defaults the NULL to VARCHAR2, causing ORA-00932. Fixed by detecting
   this pattern in _apply_clob_input_sizes and hinting the param as
   DB_TYPE_TIMESTAMP_TZ.

2. result_metadata JSON parsing: main added json.loads(row["result_metadata"])
   in list_operations which fails on Oracle where JSON columns return
   pre-parsed dicts. Replaced with conn.parse_json() for backend-agnostic
   handling.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@DK09876
Merge remote-tracking branch 'origin/main' into feature/database-abst…
d18d313 
…raction
@DK09876 @claude
feat(oracle): batch insert_facts_batch with executemany and native JS…
54d0e0d 
…ON_TABLE entity resolution

- Refactor OracleOps.insert_facts_batch from N row-by-row fetchval calls
  to single executemany with client-side UUID generation (single network
  round-trip)
- Replace PG-only unnest($2::text[]) in Oracle fuzzy entity resolution
  with native JSON_TABLE to expand entity texts into rows
- Add unit tests verifying column mapping correctness, SQL structure,
  data transformation, fallback behavior, and candidate grouping

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@DK09876 DK09876 closed this Apr 24, 2026
@DK09876 DK09876 reopened this Apr 24, 2026
@DK09876 DK09876 force-pushed the feature/database-abstraction branch from b476798 to 339dc97 Compare April 24, 2026 19:25
@DK09876 @claude
feat(oracle): batch insert_facts_batch with executemany and native JS…
f47acd9 
…ON_TABLE entity resolution

- Refactor OracleOps.insert_facts_batch from N row-by-row fetchval calls
  to single executemany with client-side UUID generation (single network
  round-trip)
- Replace PG-only unnest($2::text[]) in Oracle fuzzy entity resolution
  with native JSON_TABLE to expand entity texts into rows
- Add unit tests verifying column mapping correctness, SQL structure,
  data transformation, fallback behavior, and candidate grouping

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@DK09876 DK09876 force-pushed the feature/database-abstraction branch from 339dc97 to f47acd9 Compare April 24, 2026 19:30
DK09876 and others added 5 commits April 24, 2026 12:32
@DK09876
chore: trigger CI
a9cc282
@DK09876 @claude
merge: incorporate latest main (statement_timeout, cancel_operation, …
0b42130 
…document stats)

Merge origin/main into feature/database-abstraction. Resolves conflicts
by keeping DatabaseBackend abstraction while incorporating main's new
features: statement_timeout init, per-fact-type document counts, and
cancel_operation status guard.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@DK09876 @claude
fix: merge alembic heads, fix Oracle CLOB GROUP BY, fix ty lint
96af5fd 
- Merge two alembic heads (oracle merge + cancelled status) created by
  merging main into the database-abstraction branch.
- Fix get_document query that used GROUP BY on CLOB columns which Oracle
  cannot handle. Rewrote with subquery for counts and portable CASE WHEN
  syntax instead of PG-specific FILTER (WHERE ...).
- Add ty: ignore for Windows-only subprocess attrs in daemon_embed_manager.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@DK09876 @claude
style: auto-format openai_compatible_llm.py
143a942 
Ruff reformatted a multi-line conditional to single line.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@DK09876 @claude
feat(oracle): resolve all Known Limitations — full Oracle feature parity
17fe031 
Phase 1: Refactor WorkerPoller from raw asyncpg pool to DatabaseBackend
abstraction. Oracle now uses BrokerTaskBackend (async worker/poller)
instead of SyncTaskBackend (inline). Portable SQL patterns replace
PG-specific COUNT(*) FILTER and pg_stat_activity.

Phase 2: Add observation_sources junction table replacing
source_memory_ids array/CLOB queries. Both PG and Oracle now use
identical standard SQL joins instead of dialect-specific unnest/&&
(PG) or JSON_TABLE (Oracle). Dual-write maintains backward compat.

Phase 3: Add automatic list partitioning on memory_units(bank_id)
for Oracle. Enables partition pruning on bank-scoped queries. HNSW
vector index uses ORGANIZATION NEIGHBOR PARTITIONS. Text index
(CTXSYS.CONTEXT) remains global — LOCAL not supported on LIST
partitioned tables.

All 5 Known Limitations from PR #947 review are now resolved:
1. Worker poller decoupled from asyncpg
2. Entity resolver (already done — JSON_TABLE + UTL_MATCH)
3. source_memory_ids array queries (junction table)
4. Per-bank vector index isolation (automatic partitioning)
5. Bulk insert optimization (already done — executemany)

Tests: 1805 PG passed, 60 Oracle passed, lint clean.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@nicoloboschi nicoloboschi nicoloboschi left review comments

Assignees

No one assigned

Labels

oracle-tests Enable Oracle 23ai integration tests in CI

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@DK09876 @nicoloboschi