Article 2 index creation - #95
Conversation
- Introduced a new `database.json` file to define the Azure Cosmos DB account and its associated resources, including vector search containers and indexing policies. - Updated `main.bicep` to include parameters for creating an index database and its containers. - Added outputs for the index database and its containers to facilitate further integration. - Adjusted file paths for data files to ensure correct referencing.
…tion management - Introduced `generate-appsettings.sh` script to create appsettings.json from azd environment values. - Created `Config.cs` to manage application configuration and validation. - Implemented `ControlPlane.cs` for Azure Cosmos DB control-plane operations, including container creation and cleanup. - Developed `DataPlane.cs` for data-plane operations, including document ingestion and querying with vector distance functions. - Added `HotelDocument.cs` to define the structure of hotel documents. - Implemented `Program.cs` as the entry point for the application, orchestrating setup, ingestion, and querying. - Created comprehensive tests in `test_vectordistance_fixes` to validate functionality and ensure correctness of vector distance operations. - Added project file for the test project to manage dependencies and build settings.
- Add comprehensive query pattern explanation to .NET quickstart - Document belt-and-suspenders pattern (WHERE clause + SDK partition key) - Explain benefits: ~3 RUs vs ~12 RUs, fewer docs searched - TypeScript: Add partitionKeyValue config, convert to single-partition queries Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
VectorDistance options require quoted keys in JSON-like syntax.
Changed {distanceFunction: '...'} to {'distanceFunction': '...'}
to match .NET, Python, and TypeScript implementations.
Also renamed Go result field from 'score' to 'SimilarityScore' for consistency.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The query returns 'SimilarityScore' but the struct was mapping to 'score'. Changed json tag from 'score' to 'SimilarityScore' to match the query alias. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
Looks great! However, the DiskANN container should also exclude the vector path from the standard index. Right now QuantizedFlat excludes ${vectorPath}/* but DiskANN only excludes /_etag/?, so the raw vector gets swept into the regular index via the /* included path. Since DiskANN keeps its own separate vector index, that copy is pure overhead (extra storage and write RU per insert with no query benefit), and our documented DiskANN sample excludes the path for this reason, so I'd add { path: '${vectorPath}/*' } to the DiskANN branch. Also, the ~79k-line vector JSON is committed seven times (root data/ plus one per language folder), could those share one copy or move to Git LFS? |
DiskANN maintains its own vector index, so including the raw vector in
the standard index only adds storage and write-RU overhead. Mirror the
QuantizedFlat container by excluding ${vectorPath}/*. Regenerated the
compiled ARM (main.json, database.json). Addresses PR Azure-Samples#95 review feedback.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 219997ec-391e-4043-a21d-1275783b6ea2
The ~2.4 MB vector JSON (and companion data files) were committed once per language sample in addition to the canonical repo-root /data copy. Remove the duplicates from each sample's data/ folder and leave a data/README.md instructing users to copy the required files from the repo-root /data directory. Addresses PR Azure-Samples#95 review feedback. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 219997ec-391e-4043-a21d-1275783b6ea2
Reverts the standard-index vector-path exclusions that were added to infra in this PR. Those changes were incorrect for this PR's scope: the exclusion added to the article-2 createIndexContainers block used the wrong embedding field (/DescriptionVector) — article 2 uses /embedding and provisions its container at runtime from sample code, not Bicep. Deletes infra/cosmos-db/nosql/vector-containers.bicep, an unused module with no references anywhere in infra. The legitimate DiskANN standard-index exclusion for article 1 is a pre-existing issue on main and is handled in a separate, dedicated PR. The correct article-2 fix (excluding the embedding path from the standard index) is added to the sample control-plane code in a follow-up commit. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 219997ec-391e-4043-a21d-1275783b6ea2
…thon, .NET)
The create-index samples included all paths (/*) in the standard index but only excluded /_etag/?, leaving the vector embedding field indexed twice — once in the standard index and once in the dedicated vector index. This wastes write RUs and storage.
Adds an {embeddingPath}/* exclusion to the standard index in the TypeScript, Python, and .NET samples, matching the behavior already present in the Go and Java samples.
Applies to both DiskANN and QuantizedFlat containers because they share the same index-policy builder. Addresses reviewer feedback on PR Azure-Samples#95.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 219997ec-391e-4043-a21d-1275783b6ea2
|
@TheovanKraay thanks for the review. I pushed 3 commits that address both points. Here's what each does and why the changes land only in the article-2 (create-index) files, not article 1. What changed
- excludedPaths: [{ path: "/_etag/?" }],
+ excludedPaths: [{ path: "/_etag/?" }, { path: `${embeddingPath}/*` }],Go and Java already excluded the path (
Why it only touches article 2, not article 1The two articles use different schemas and different provisioning paths:
Because the article-2 create-index samples build their container + vector index at runtime in the sample code, the correct place for the exclusion fix is that sample code — which is exactly what this PR is scoped to. Article 1's containers are provisioned through Bicep, so the equivalent DiskANN exclusion there is an infra change on |
2 participants
Summary
This PR is scoped to the article-2 create-index sample set. It adds the create-index feature and keeps the article-2 container provisioning behavior in the sample control-plane code.
Changes
/datafolder instead of committing the same large vector data repeatedly.{embeddingPath}/*from the standard index, matching the behavior already present in the Go and Java samples. This addresses the DiskANN/vector-path exclusion feedback for article 2.Scope note
Article 2 uses
/embeddingwith partition key/Regionand creates its containers at runtime from the sample control-plane code, so the article-2 indexing fix belongs in this PR's sample code.The pre-existing article-1 DiskANN standard-index exclusion issue is separate: article 1 uses
/DescriptionVectorwith partition key/HotelIdand provisions containers through Bicep. That independentmain-branch infra correction is tracked in draft PR #100: #100