Description
In src/dynavec/metadata.py, split_metadata() partitions a document's metadata between S3 Vectors (small filterable subset) and DynamoDB (full rich metadata).
DynavecConfig provides non_filterable_keys to explicitly exclude certain fields from being sent to S3 Vectors even if they are scalar values. While split_metadata() implements this logic:
for key in candidate_keys:
if key in config.non_filterable_keys:
continue
if key in metadata and _is_s3_filterable_value(metadata[key]):
s3_meta[key] = metadata[key]
tests/test_metadata.py currently only tests the filterable_keys allowlist (test_split_respects_filterable_keys_allowlist) and does not test non_filterable_keys or its precedence rules.
Proposed Scope & Solution
Extend tests/test_metadata.py with comprehensive unit tests for non_filterable_keys:
- Direct exclusion: An excluded key present in
non_filterable_keys stays in the DynamoDB metadata copy, but is absent from the S3 Vectors filter metadata.
- Precedence over allowlist: When a key is present in both
filterable_keys (allowlist) and non_filterable_keys (denylist), exclusion in non_filterable_keys takes precedence.
- Namespace isolation tag: Confirm that
non_filterable_keys cannot inadvertently strip the mandatory namespace tag (__ns).
- Immutability of input: Verify that the caller's input metadata dictionary is not mutated during the split.
Acceptance Criteria
Description
In
src/dynavec/metadata.py,split_metadata()partitions a document's metadata between S3 Vectors (small filterable subset) and DynamoDB (full rich metadata).DynavecConfigprovidesnon_filterable_keysto explicitly exclude certain fields from being sent to S3 Vectors even if they are scalar values. Whilesplit_metadata()implements this logic:tests/test_metadata.pycurrently only tests thefilterable_keysallowlist (test_split_respects_filterable_keys_allowlist) and does not testnon_filterable_keysor its precedence rules.Proposed Scope & Solution
Extend
tests/test_metadata.pywith comprehensive unit tests fornon_filterable_keys:non_filterable_keysstays in the DynamoDB metadata copy, but is absent from the S3 Vectors filter metadata.filterable_keys(allowlist) andnon_filterable_keys(denylist), exclusion innon_filterable_keystakes precedence.non_filterable_keyscannot inadvertently strip the mandatory namespace tag (__ns).Acceptance Criteria
tests/test_metadata.pycovering all the scenarios above.