Skip to content

test(metadata): Cover metadata exclusion rules in split_metadata() #245

Description

@shivamm-gupta

Description

In src/dynavec/metadata.py, split_metadata() partitions a document's metadata between S3 Vectors (small filterable subset) and DynamoDB (full rich metadata).

DynavecConfig provides non_filterable_keys to explicitly exclude certain fields from being sent to S3 Vectors even if they are scalar values. While split_metadata() implements this logic:

for key in candidate_keys:
    if key in config.non_filterable_keys:
        continue
    if key in metadata and _is_s3_filterable_value(metadata[key]):
        s3_meta[key] = metadata[key]

tests/test_metadata.py currently only tests the filterable_keys allowlist (test_split_respects_filterable_keys_allowlist) and does not test non_filterable_keys or its precedence rules.

Proposed Scope & Solution

Extend tests/test_metadata.py with comprehensive unit tests for non_filterable_keys:

  1. Direct exclusion: An excluded key present in non_filterable_keys stays in the DynamoDB metadata copy, but is absent from the S3 Vectors filter metadata.
  2. Precedence over allowlist: When a key is present in both filterable_keys (allowlist) and non_filterable_keys (denylist), exclusion in non_filterable_keys takes precedence.
  3. Namespace isolation tag: Confirm that non_filterable_keys cannot inadvertently strip the mandatory namespace tag (__ns).
  4. Immutability of input: Verify that the caller's input metadata dictionary is not mutated during the split.

Acceptance Criteria

  • Unit tests added to tests/test_metadata.py covering all the scenarios above.
  • Tests run 100% offline without live AWS credentials.
  • Ruff / lint passes clean.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions