Implement a database abstraction layer that will enable the use of different DB backends.#282

Open

Davidyz wants to merge 96 commits intomainfrom

Commits on Sep 22, 2025

WIP(cli): database abstraction layer

Zhe Yu
authored and
Davidyz
committed
WIP(cli): Add wip chromadb connector

Zhe Yu
authored and
Davidyz
committed
feat(db): Improve database abstraction and vectorisation process

Zhe Yu
authored and
Davidyz
committed
feat(db): Implement delete and drop methods for database connectors

Zhe Yu
authored and
Davidyz
committed
refactor(cli): `drop` now use the DB adapter layer.

Zhe Yu
authored and
Davidyz
committed
fix(cli): default db_url for chroma0

Zhe Yu
authored and
Davidyz
committed
fix(cli): minor fixes.

Zhe Yu
authored and
Davidyz
committed
feat(cli): implement database builder with lazy import

Zhe Yu
authored and
Davidyz
committed
refactor(cli): `ls` in CLI mode now use the DB adapter layer.

Zhe Yu
authored and
Davidyz
committed
docs about database connectors.

Zhe Yu
authored and
Davidyz
committed
feat(cli): support excluding files in queries.

Zhe Yu
authored and
Davidyz
committed
fix query filter

Zhe Yu
authored and
Davidyz
committed
feat(db): Implement config update and replace methods for database connectors

Zhe Yu
authored and
Davidyz
committed
remove some parameters.

Zhe Yu
authored and
Davidyz
committed
update `chroma0` connector to adopt the new API definition.

Zhe Yu
authored and
Davidyz
committed
fix(cli): fix default value of `excluded_files`

Zhe Yu
authored and
Davidyz
committed
fix(cli): improve assertions for `metadatas`, `documents` and `ids`

Zhe Yu
authored and
Davidyz
committed
refactor(cli): Use DB adapter layer for `ls` and `rm` commands

Zhe Yu
authored and
Davidyz
committed
docs(db): Improve docstrings for database connectors

Zhe Yu
authored and
Davidyz
committed
docs(db): Add documentation for database connectors

Zhe Yu
authored and
Davidyz
committed
feat(db): Implement orphan removal functionality in the database connector

Zhe Yu
authored and
Davidyz
committed
feat(db): Refactor database connectors to use embeddings and improve listing

Zhe Yu
authored and
Davidyz
committed
refactor(db): Implement `get_chunks` method in the database connectors

Zhe Yu
authored and
Davidyz
committed
wip(db): port `vectorise` command to the new db layer

Zhe Yu
authored and
Davidyz
committed
fix(db): Refactor `list_collection_content` to accept keyword arguments

Zhe Yu
authored and
Davidyz
committed
feat(cli): Refactor vectorise command to use DB adapter layer

Zhe Yu
authored and
Davidyz
committed
feat(cli): Refactor `update` command to use DB adapter layer

Zhe Yu
authored and
Davidyz
committed
refactor(cli): Refactor query command to use DB adapter layer

Zhe Yu
authored and
Davidyz
committed
feat(db): Enforce Chroma v0 client version

Zhe Yu
authored and
Davidyz
committed
feat(deps): Update chromadb dependency to latest and add `chromadb==0.6.3` optional dependency

Zhe Yu
authored and
Davidyz
committed
fix(cli): Fix query command by removing deprecated code and chunking queries

Zhe Yu
authored and
Davidyz
committed
refactor(cli): Refactor mcp_main to use DB adapter layer

Zhe Yu
authored and
Davidyz
committed
refactor(cli): Refactor query command to use `preprocess_query_keywords` function

Zhe Yu
authored and
Davidyz
committed
fix(cli): Remove all references to chromadb-related APIs.

Zhe Yu
authored and
Davidyz
committed
fix(chroma0): fix import origin for `get_uuid`

Zhe Yu
authored and
Davidyz
committed
refactor(lsp): Refactor LSP server to use DB adapter layer

Zhe Yu
authored and
Davidyz
committed
refactor(cli): Refactor `clean` command to use DB adapter layer

Zhe Yu
authored and
Davidyz
committed
refactor(cli)!: remove `common.py`

Zhe Yu
authored and
Davidyz
committed
docs(database): Clarify terminology in database README

Zhe Yu
authored and
Davidyz
committed
feat(chroma0): add graceful shutdown for bundled chroma server

Zhe Yu
authored and
Davidyz
committed
test(chroma0): Add unittests.

Zhe Yu
authored and
Davidyz
committed
remove unnecessary test

Zhe Yu
authored and
Davidyz
committed
tests(mcp): fix tests.

Zhe Yu
authored and
Davidyz
committed
fix(mcp): Improve error handling and test coverage

Zhe Yu
authored and
Davidyz
committed
fix(lsp): use correct file lists

Zhe Yu
authored and
Davidyz
committed
tests(chroma0): make sure database is mocked.

Zhe Yu
authored and
Davidyz
committed
fix(lsp): remove collection-related code and improved test coverage.

Zhe Yu
authored and
Davidyz
committed
tests(cli): Improve test coverage for `ls` subcommand and handle `CollectionNotFoundError`

Zhe Yu
authored and
Davidyz
committed
tests(cli): refactor tests and coverage for `files rm`

Zhe Yu
authored and
Davidyz
committed
tests(cli): Refactor query subcommand and add tests

Zhe Yu
authored and
Davidyz
committed
fix(lsp): Validate include parameters in query command

Zhe Yu
authored and
Davidyz
committed
tests(cli): Refactor drop subcommand and add exception handling

Zhe Yu
authored and
Davidyz
committed
feat(cli): Improve test coverage for `vectorise`

Zhe Yu
authored and
Davidyz
committed
tests(cli): Improve test coverage and refactor update subcommand

Zhe Yu
authored and
Davidyz
committed
tests(cli): Refactor tests and improve coverage for `main`

Zhe Yu
authored and
Davidyz
committed
tests: Refactor database types and improve test coverage

Zhe Yu
authored and
Davidyz
committed
tests(cli): Refactor config and cleanup path handling

Zhe Yu
authored and
Davidyz
committed
chore(cli): mark some code as nocover

Zhe Yu
authored and
Davidyz
committed
pin to chroma 0.6.3 for now.

Zhe Yu
authored and
Davidyz
committed
fix: use `os.path.samefile` for accurate directory comparison

Zhe Yu
authored and
Davidyz
committed
fix error message

Zhe Yu
authored and
Davidyz
committed
docs(database): Document the database connector API

Zhe Yu
authored and
Davidyz
committed
tests(chroma0): Add more tests to chroma0

Zhe Yu
authored and
Davidyz
committed
tests(cli): Improve test coverage for SpecResolver

Zhe Yu
authored and
Davidyz
committed
tests: Add tests for database types

Zhe Yu
authored and
Davidyz
committed
tests(chroma0): improve test coverages.

Zhe Yu
authored and
Davidyz
committed
docs(database): update database configuration documentation

Zhe Yu
authored and
Davidyz
committed
Auto generate docs
Davidyz
committed
docs(database): Document helper methods and error handling in the database API

Zhe Yu
authored and
Davidyz
committed
docs: Document database configuration and connector development

Zhe Yu
authored and
Davidyz
committed
Auto generate docs
Davidyz
committed
docs: clarify default chroma version

Zhe Yu
authored and
Davidyz
committed
Auto generate docs
Davidyz
committed

Commits on Sep 25, 2025

feat(cli): Report skipped files in vectorise stats
Zhe Yu
committed

Commits on Oct 3, 2025

build(cli): add `chroma0` dep group.
Zhe Yu
committed
Auto generate docs

Davidyz
authored and
github-actions[bot]
committed
chore(cli): Use chroma0 for CI in test workflow
Zhe Yu
committed
chore(cli): Document how to install extra dependencies
Zhe Yu
committed
docs(cli): reflect packaging changes.
Zhe Yu
committed
Auto generate docs

Davidyz
authored and
github-actions[bot]
committed
tests(db): Add database connector initialization test
Zhe Yu
committed

Commits on Oct 4, 2025

tests(db): skip tests when dependency's not met
Zhe Yu
committed
chore: extra coverage args via env var.
Zhe Yu
committed
feat(db): Check ChromaDB version on startup
Zhe Yu
committed

Commits on Oct 5, 2025

refactor(cli): remove obsolete opts
Zhe Yu
committed
fix(cli): Fixes db_url config retrieval
Zhe Yu
committed
refactor(chroma0): extract some stuff that can be reused by chromadb (1.x)
Zhe Yu
committed
fix(db): Upgrade chromadb version check and include enums
Zhe Yu
committed
feat(chroma): a WIP chromadb connector for chroma 1.x
Zhe Yu
committed

Commits on Oct 6, 2025

Commits on Oct 8, 2025

feat(db): Add inter-process and inter-thread locks for ChromaDB connector
Davidyz
committed

Commits on Oct 10, 2025

Commits on Nov 13, 2025

Merge branch 'main' into feat/db_layer
Davidyz
committed