Skip to content

feat(storage): add SQLite search backend - #853

Draft
Fhatu12 wants to merge 1 commit into
KusionStack:mainfrom
Fhatu12:feat/230-sqlite-storage
Draft

feat(storage): add SQLite search backend#853
Fhatu12 wants to merge 1 commit into
KusionStack:mainfrom
Fhatu12:feat/230-sqlite-storage

Conversation

@Fhatu12

@Fhatu12 Fhatu12 commented Sep 8, 2026

Copy link
Copy Markdown

Summary

  • Adds a pure-Go SQLite search storage backend as the first additional local backend.
  • Wires sqlite through the existing search storage provider/factory/options/server config path.
  • Keeps the Elasticsearch storage path intact while moving common not-found sentinels to the shared storage package.

Architecture

  • New pkg/infra/search/storage/sqlite package implements SearchStorage, ResourceStorage, ResourceGroupRuleStorage, and GeneralStorage through one Storage type.
  • SQLite schema is initialized idempotently on open and stores resources plus resource-group rules in local tables.
  • Resource fields used by Karpor queries are indexed as columns; labels/annotations and rule fields are stored as JSON text.

SQLite Driver Rationale / CGO=0 Compatibility

  • Uses modernc.org/sqlite v1.36.1, a pure-Go database/sql SQLite driver.
  • Chose v1.36.1 because Karpor declares Go 1.22 compatibility and newer modernc releases currently require newer Go toolchains.
  • Rejected github.com/mattn/go-sqlite3 because it requires CGO_ENABLED=1 and a C compiler, while Karpor defaults release builds to CGO_ENABLED=0.
  • No paid service, network service, or external runtime is required at operation time.

Search Safety Model

  • Incoming SQL text is parsed with the existing github.com/xwb1989/sqlparser dependency and compiled to SQLite SQL.
  • The original user query string is never executed directly.
  • Values, pagination, and JSON paths are parameterized; identifiers are resolved through a fixed field allowlist.
  • Non-SELECT statements, multiple statements, unsupported tables, joins, subqueries, aliases, unsafe functions, and unsupported expressions are rejected.

Supported Query Semantics

  • Supports SQL and NL paths that resolve to SQL over SELECT ... FROM resources.
  • Supports WHERE, AND, OR, parentheses, =, !=, >, <, >=, <=, LIKE, NOT LIKE, IN, NOT IN, BETWEEN, NOT BETWEEN, contains(field, 'value'), ordering, and pagination.
  • Supports scalar fields plus labels.<key> and annotations.<key> using safe JSON extraction.
  • Adds implicit deleted=false behavior when a query does not explicitly reference deleted.
  • Supports SearchByTerms and AggregateByTerms for existing Karpor manager/topology paths.

Configuration

  • --search-storage-type=sqlite
  • --sqlite-dsn=<dsn>; defaults to file:karpor-search.db?mode=rwc when omitted.
  • Existing Elasticsearch flags and selection behavior are unchanged.

Test Plan / Results

  • go test ./pkg/infra/search/storage/sqlite ./pkg/kubernetes/registry/search ./cmd/karpor/app/options ./pkg/core/manager/resourcegroup ./pkg/core/handler/resourcegroup ./pkg/core/handler/resourcegrouprule
  • go test ./pkg/infra/search/storage/... ./pkg/kubernetes/registry/search ./cmd/karpor/app/options
  • go test ./cmd/karpor ./pkg/core/handler/search ./pkg/core/manager/resourcegroup ./pkg/util/sql2es
  • CGO_ENABLED=0 go test ./pkg/infra/search/storage/sqlite
  • CGO_ENABLED=0 go build ./cmd/karpor
  • go test -race ./pkg/infra/search/storage/sqlite
  • make test
  • PATH=/home/fhatu/go/bin:$PATH make lint with golangci-lint v1.62.0
  • SKIP_UI_BUILD=true make build
  • make cover total coverage: 64.8%; SQLite package coverage: 71.4%
  • git diff --check

Known Limitations

  • SQLite does not support DSLPatternType in this pass; it returns a clear unsupported-pattern error.
  • ZincSearch and other backends are out of scope.
  • The standalone karpor syncer command still constructs Elasticsearch storage directly and remains a follow-up design surface.

AI Assistance Disclosure

  • Implemented with AI assistance from OpenAI Codex under contributor direction.

Fixes #230

@github-actions

github-actions Bot commented Sep 8, 2026

Copy link
Copy Markdown

CLA Assistant Lite bot All contributors have signed the CLA ✍️ ✅

@Fhatu12

Fhatu12 commented Sep 8, 2026

Copy link
Copy Markdown
Author

I have read the CLA Document and I hereby sign the CLA

@Fhatu12

Fhatu12 commented Sep 8, 2026

Copy link
Copy Markdown
Author

@elliotxx Could you please help with the FOSSA License Compliance check for PR #853?

It currently reports “4 issues found”, but the public GitHub status does not expose the individual findings. Could you please share the four issues from FOSSA, ideally including the affected component, detected licence, issue/policy type, and affected path?

PR head: baa2b863b2613b6916c69e47cb247b14a31aed2a

I want to make the smallest correct attribution, metadata, or dependency fix rather than guessing or suppressing the policy.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Feat: support more storage backend, such as ZincSearch

1 participant