fix(infra): exclude vector path from DiskANN standard index (article 1) - #100
Open
diberry wants to merge 2 commits into
Open
fix(infra): exclude vector path from DiskANN standard index (article 1)#100diberry wants to merge 2 commits into
diberry wants to merge 2 commits into
Conversation
The article-1 vector-search DiskANN container included all paths (/*) in the standard index but excluded only /_etag/?, so the /DescriptionVector embedding field was indexed both in the standard index and in the dedicated DiskANN vector index. This wastes write RUs and storage. Adds a /DescriptionVector/* exclusion to the DiskANN container's standard index, matching the sibling QuantizedFlat container which already excludes it. This is a general Cosmos DB indexing principle: when a container uses a broad /* included path, the vector field path should be excluded from the standard index because every vector index type (DiskANN, QuantizedFlat, Flat) maintains its own vector index. The principle applies regardless of how the container/index is created (Bicep, ARM, SDK at runtime, portal, or CLI). Regenerates infra/main.json and infra/database.json from the updated Bicep. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 219997ec-391e-4043-a21d-1275783b6ea2
Contributor
There was a problem hiding this comment.
Pull request overview
Updates the article-1 (vector-search) infrastructure templates so the DiskANN container excludes the vector field path from the standard index, aligning it with the existing QuantizedFlat container behavior and common Cosmos DB indexing guidance.
Changes:
- Added
/DescriptionVector/*toexcludedPathsfor thehotels_diskanncontainer indexing policy ininfra/database.bicep. - Regenerated the compiled ARM templates (
infra/database.jsonandinfra/main.json) from the updated Bicep.
Reviewed changes
Copilot reviewed 2 out of 3 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| infra/database.bicep | Adds the missing standard-index exclusion for /DescriptionVector/* on the DiskANN container. |
| infra/database.json | Regenerated compiled template reflecting the updated DiskANN indexing policy exclusion. |
| infra/main.json | Regenerated top-level deployment template to include the updated database module output. |
Deployment runs from Bicep (main.bicep + main.bicepparam via azd); the compiled ARM templates (database.json, main.json) are build artifacts that were never tracked on main. Remove them and add infra/*.json to .gitignore so bicep build output is not committed again. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 4da54cd1-6cdc-442c-af04-a13d33c96b5c
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The article-1 vector-search DiskANN container omits the vector-path exclusion from its standard index, while the sibling QuantizedFlat container already excludes
/DescriptionVector/*. This is a pre-existing issue onmainand was not introduced by PR #95.Research and rationale
This is a general Cosmos DB indexing principle: when a container uses a broad
/*included path, the vector field path should be excluded from the standard index because every vector index type (DiskANN, QuantizedFlat, Flat) maintains its own vector index. The principle applies regardless of how the container/index is created (Bicep, ARM, SDK at runtime, portal, or CLI).Article 1 and article 2 use different schemas and provisioning paths:
vector-search) uses field/DescriptionVectorwith partition key/HotelId, and its containers are provisioned through Bicep/ARM.create-index) uses field/embeddingwith partition key/Region, and its containers are provisioned at runtime from sample control-plane code. The article-2 fix belongs in PR Article 2 index creation #95: Article 2 index creation #95This fix was split out of PR #95 because PR #95 is scoped to article 2. The article-1 infra correction is independent and already exists on
main.Exact change
{ path: '/DescriptionVector/*' }to thehotels_diskanncontainer'sexcludedPathsininfra/database.bicep.infra/main.jsonandinfra/database.jsonfrom the updated Bicep.