feat: add Harness File Store support#197
Conversation
Wrap Harness Test Intelligence APIs behind the existing list flow so agents can retrieve failed test methods for a pipeline build with a compact structured report. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Retryimprovedlogic
Co-authored-by: Cursor <cursoragent@cursor.com>
Add selective pipeline stage retry
|
|
There was a problem hiding this comment.
Stale comment
- Medium —
src/registry/toolsets/file-store.ts,src/tools/harness-create.ts,src/tools/harness-update.ts,src/tools/harness-delete.ts,src/tools/harness-describe.ts:file_store_itemnow advertisessupportedScopes: ["account", "org", "project"]on a write-capable resource, but the public write tools still do not exposeresource_scopeand do not opt into URL-derived scope. That meansharness_describetells agents to setresource_scope, yetharness_create/harness_update/harness_deletecan only hit account/org File Store writes through undocumentedparams.resource_scope; with configured defaults they otherwise fall back to project scope. This is a structured discoverability/runtime contract mismatch.- Low —
src/utils/url-parser.ts,src/registry/toolsets/file-store.ts,tests/utils/url-parser.test.ts: the PR adds File Store URL parsing, but the resource definition still has nodeepLinkTemplate, so File Store responses cannot emitopenInHarnesslinks even though pasted URLs now parse. There is also no matchingurl-parserregression coverage for the newfile-storesegment / URL-default merge path, so this new discoverability surface is one-way and unverified.- Low —
src/registry/index.ts,README.md: the branch is currently red onpnpm docs:checkbecause adding the new toolset changed the generated resource/toolset counts without regeneratingREADME.md. CI’spnpm testandpnpm typechecksteps passed on the PR head; docs sync is the remaining failure.Overall, the File Store endpoint wiring itself is mostly aligned with the repo standards: declarative toolset registration, clean GET vs multipart-body separation, no body leakage in the update preflight, appropriate risk policies, and guarded output shaping for file downloads. The gaps are in shared-tool discoverability / round-trip UX rather than the endpoint mappings themselves.
Sent by Cursor Automation: Sunil On Demand Architecture Review
| description: "Harness File Store file or folder metadata. Supports listing, metadata lookup, create, and update.", | ||
| toolset: "file-store", | ||
| scope: "project", | ||
| supportedScopes: ["account", "org", "project"], |
There was a problem hiding this comment.
file_store_item is now a write-capable resource that advertises account/org/project scope, but the public write tools still do not publish a top-level resource_scope input and they intentionally ignore URL-derived scope on create/update/delete. That means harness_describe(resource_type='file_store_item') now tells agents they can set resource_scope, while harness_create / harness_update / harness_delete can only reach those paths through undocumented params.resource_scope and otherwise fall back to configured project scope. I think this should either be made schema-visible on the write tools or kept out of the advertised multi-scope contract.
There was a problem hiding this comment.
Stale comment
Findings:
- Medium:
file_store_itemover-advertises multi-scope writes. The new resource exposessupportedScopesthroughharness_describe, but the shared write tools still do not acceptresource_scopeand intentionally do not derive it from write URLs. That means account/org File Store writes are described as supported, but agents cannot actually invoke them throughharness_create/harness_update/harness_deletetoday.- Medium: the branch is not merge-ready because generated docs were not refreshed. Both CI jobs fail
pnpm docs:check, and I reproduced the same locally after a clean build:README.md is stale. Run \pnpm docs:generate` to refresh.`- Low: File Store URL support is currently one-way. The parser knows
/file-store/..., but there is no matching regression coverage for that shared parser path, and the resource definition does not add adeepLinkTemplate, soopenInHarnesscannot round-trip back to the same UI surface.Assumption:
I’m assuming File Store is intended to follow the same registry contract thatharness_describeexposes. If writes are intentionally project-only, the resource metadata should say that explicitly instead of advertising account/org/project support.Verification:
pnpm test-> 54 files, 1313 tests passedpnpm typecheck-> passedpnpm build && pnpm docs:check-> failed (README.md is stale. Run \pnpm docs:generate` to refresh.`)Sent by Cursor Automation: Sunil On Demand Architecture Review
| description: "Harness File Store file or folder metadata. Supports listing, metadata lookup, create, and update.", | ||
| toolset: "file-store", | ||
| scope: "project", | ||
| supportedScopes: ["account", "org", "project"], |
There was a problem hiding this comment.
supportedScopes is surfaced through harness_describe, but the shared write tools still don't accept resource_scope and intentionally do not infer it from write URLs. That means this resource now advertises account/org/project support for create / update / delete, while agents can only invoke the default/project scoping path today. Either plumb resource_scope through the write tools or narrow the File Store scope contract to the read operations so the metadata stays honest.
There was a problem hiding this comment.
Stale comment
Found 2 issues against Sunil's architecture standards.
- Medium -
README.md's public contract is still incomplete. The PR refreshes the numeric counts, but the visible catalog still does not list the newfile-storetoolset or thefile_store_item/file_store_contentresource types, so users and agents cannot actually discover the feature from the docs.- Low - The new File Store content safety logic is only partially verified. The added tests cover the happy-path UTF-8 download, but not the large/binary guard branches that prevent oversized or non-text payloads from being inlined.
Assumption:
README.mdis still intended to be a discoverable public surface, since the repo keeps checking it viascripts/generate-docs.js --check.Verification run:
pnpm test tests/registry/file-store.test.ts tests/client/harness-client.test.tspnpm typecheckpnpm build- Simulated
scripts/generate-docs.js --checkagainstorigin/pr-197'sREADME.md: it passes even though that README currently has zerofile-storementions.Sent by Cursor Automation: Sunil On Demand Architecture Review
| expect(getCall.params).not.toHaveProperty("mime_type"); | ||
| }); | ||
|
|
||
| it("reads UTF-8 file content from the download endpoint", async () => { |
There was a problem hiding this comment.
This only exercises the small UTF-8 branch of fileStoreContentExtract(). The risky part of the new feature is the guard logic for oversized text and binary payloads (>1MB refusal, base64 fallback for small binary, refusal for large binary), and none of those branches are covered yet. Given the repo's prove it works standard, I'd add focused assertions for those paths before shipping.
1577dd7 to
69fa549
Compare
There was a problem hiding this comment.
Stale comment
Found 3 remaining architecture-contract issues on the current PR head (
69fa549). The follow-up commits fixed the earlier README / docs-check drift and added File Store URL-parser coverage, and I re-verified the branch with:
pnpm test tests/registry/file-store.test.ts tests/client/harness-client.test.ts tests/registry/structural-validation.test.ts tests/utils/url-parser.test.tspnpm typecheckpnpm buildpnpm docs:checkAll of those passed, but the File Store metadata layer still overstates or misroutes a few capabilities in ways that will mislead agents using the shared registry surfaces.
Sent by Cursor Automation: Sunil On Demand Architecture Review
| description: "Harness File Store file or folder metadata. Supports listing, metadata lookup, create, and update.", | ||
| toolset: "file-store", | ||
| scope: "project", | ||
| supportedScopes: ["account", "org", "project"], |
There was a problem hiding this comment.
supportedScopes is resource-wide metadata surfaced through harness_describe, but the public write tools still do not expose a top-level resource_scope input and applyUrlDefaults() only opts read tools into URL-derived scope. That means this resource now advertises account/org/project support for create / update / delete, while agents can only reach those write paths through undocumented params.resource_scope plumbing. I think this should either be surfaced on harness_create / harness_update / harness_delete or narrowed so the advertised scope contract stays honest.
| supportedScopes: ["account", "org", "project"], | ||
| identifierFields: ["file_id"], | ||
| deepLinkTemplate: | ||
| "/ng/account/{accountId}/all/orgs/{orgIdentifier}/projects/{projectIdentifier}/setup/resources/file-store", |
There was a problem hiding this comment.
The new deep link doesn't round-trip the File Store routes this PR now teaches the URL parser. tests/utils/url-parser.test.ts recognizes both project URLs like /all/orgs/.../projects/.../setup/resources/file-store/<id> and account URLs like /all/settings/file-store/<id>, but this template always builds the project-scoped collection root and omits the selected file_id. I verified with the repo helper that account scope currently collapses to https://app.harness.io/ng/account/<id>/all/setup/resources/file-store, which doesn't match the account-level route. So openInHarness will be wrong for account-scoped reads and can never deep-link to the specific file that was fetched.
|
|
||
| const fileStoreUpdateSchema: BodySchema = { | ||
| description: "File Store file or folder update. Missing name/type/parent_identifier are hydrated from existing metadata when possible.", | ||
| fields: fileStoreCreateSchema.fields, |
There was a problem hiding this comment.
fileStoreUpdateSchema reuses the create fields verbatim, so the structured contract still marks name and type as required even though hydrateFileStoreUpdate() fills them from a metadata lookup and tests/registry/file-store.test.ts intentionally exercises update with only content / mime_type. Because harness_describe exposes bodySchema.fields to agents, this makes the update contract stricter in metadata than it is at runtime.
Co-authored-by: Cursor <cursoragent@cursor.com>
Map failed matrix stages to retry parent
Add File Store resources for metadata, uploads, and content reads through the existing registry-backed MCP tools. Co-authored-by: Cursor <cursoragent@cursor.com> AI-Session-Id: 5fee2ab6-dab8-4e3f-b487-d2c5eb3ff91c AI-Tool: claude-code AI-Model: unknown
- Extract shared isFormDataBody into src/utils/type-guards.ts, removing
duplicate definitions from harness-client.ts and registry/index.ts
- Refactor registry body-validation ternary to if/else for readability
- Fix create/update risk levels to match convention (low_write)
- Add missing delete operation (DELETE /ng/api/file-store/{identifier})
- Fix hydrateFileStoreUpdate leaking body/content fields into the GET
request by filtering input to just identifier and scope fields
- Add diagnosticHint and relatedResources for agent discoverability
- Add delete and preflight-leak regression tests
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
AI-Session-Id: 5fee2ab6-dab8-4e3f-b487-d2c5eb3ff91c
AI-Tool: claude-code
AI-Model: unknown
AI-Session-Id: 5fee2ab6-dab8-4e3f-b487-d2c5eb3ff91c AI-Tool: claude-code AI-Model: unknown
Adds deepLinkTemplate so File Store responses can emit openInHarness links. Adds 5 url-parser tests covering project-level, account-level, and applyUrlDefaults paths for the file-store URL segment. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> AI-Session-Id: 5fee2ab6-dab8-4e3f-b487-d2c5eb3ff91c AI-Tool: claude-code AI-Model: unknown
69fa549 to
12458de
Compare
There was a problem hiding this comment.
Stale comment
Important — In
src/registry/toolsets/file-store.ts, lines 132-133, thefile_store_itemdeep-link template is not round-trip honest for the resource contract. It never includes the item identifier, so everyfile_store_itemresult gets a list-page link instead of an item link. It also only models the project-scoped route, while the same resource declaressupportedScopesof account/org/project. With the currentbuildDeepLink()logic insrc/utils/deep-links.ts, account-scoped links render as/ng/account/{id}/all/setup/resources/file-store, which does not match the account-level File Store URL shape that the parser/tests accept insrc/utils/url-parser.tsandtests/utils/url-parser.test.ts(/all/settings/file-store/{id}). That breaks the pasted-URL workflow and makes the registry contract misleading. The added tests only cover URL parsing and request wiring; they never assert generatedopenInHarnessoutput, so this regression is currently untested.Important — In
README.md, line 957 and the toolset table starting at lines 1439-1475, the docs were updated to claim 171 resource types / 32 toolsets, but the generated content at this head still contains noFile StoreorTest Intelligencesections, and the toolset-name table omits bothfile-storeandtest-intelligence. A search of the README at the PR head returns no matches forfile_store_item,file_store_content,test_failure,File Store, orTest Intelligence. That leaves the public docs less discoverable than the actual registry surface.Assumptions / open questions:
- I assumed File Store deep links are expected to round-trip to the item-specific UI route because the parser/tests explicitly support
.../file-store/{identifier}URLs. If list-page-only deep links are intentional, that contract should be made explicit and the parser/tests should be adjusted to match.- I did not find a canonical org-scope File Store UI route in-repo. The account-scope mismatch is definite; the exact org-scope route likely needs confirmation from product/UI conventions.
Verified:
- Reviewed the exact current PR head
12458deb4c5f3795d24d12fbe2cb96115c99f7f8fetched viarefs/pull/197/head, againstmainat3e3ec347f91bfe72e36ca5f833fd2bc1a2bcd350.- Inspected the changed registry/toolset, URL parsing, deep-link, README, and test files directly at that SHA.
- Reproduced the current File Store deep-link output from the shipped builder logic.
- I could not execute Vitest in this workspace because dependencies are absent (
pnpm testfails withvitest: not found/ nonode_modules).Sent by Cursor Automation: Sunil On Demand Architecture Review
| supportedScopes: ["account", "org", "project"], | ||
| identifierFields: ["file_id"], | ||
| deepLinkTemplate: | ||
| "/ng/account/{accountId}/all/orgs/{orgIdentifier}/projects/{projectIdentifier}/setup/resources/file-store", |
There was a problem hiding this comment.
This template only points to the File Store list page.
Because it never includes the item identifier, every file_store_item result gets the same openInHarness URL instead of linking to the specific file/folder. It also only models the project-scoped route: with the current buildDeepLink() cleanup rules, resource_scope='account' produces /ng/account/{id}/all/setup/resources/file-store, which does not match the account-level File Store URL shape the parser/tests accept (/all/settings/file-store/{id}).
Can we make this template preserve both scope and identifier so openInHarness round-trips back into the url shortcut flow reliably?
| ## Resource Types | ||
|
|
||
| 169 resource types organized across 31 toolsets. Each resource type supports a subset of CRUD operations and optional execute actions. | ||
| 171 resource types organized across 32 toolsets. Each resource type supports a subset of CRUD operations and optional execute actions. |
There was a problem hiding this comment.
The counts were updated here, but the generated README content below still doesn't document the new surfaces.
At this head there are no README matches for file_store_item, file_store_content, test_failure, File Store, or Test Intelligence, and the toolset-name table omits both file-store and test-intelligence. That leaves docs discoverability out of sync with the actual registry contract even though the summary numbers now claim the extra toolsets/resources.
There was a problem hiding this comment.
Stale comment
Findings:
- High: the new multi-request handler abstraction is not wired into
Registry.dispatch(), so bothtest_failureandpipeline.retry_stagesare non-functional on the current head. I reproduced this locally: the focused suite fails 10 tests because each path stops after a single raw request instead of invoking the custom orchestration.- Medium:
file_store_itemstill over-advertises account/org/project write support.harness_describesurfacessupportedScopes, but the public write tools still omitresource_scopeand do not opt into URL-derived scope, so agents cannot discover account/org writes through the documented schema.- Medium: File Store
openInHarnesslinks still do not round-trip the routes the URL parser accepts. The current template drops the selected file identifier and builds the wrong account-scope route.- Medium: the PR is still red on docs sync. On the exact PR head,
pnpm typecheckandpnpm buildpassed, butpnpm docs:checkfailed withREADME.md is stale. Run \pnpm docs:generate` to refresh.The visible README catalog also still has nofile-store,test-intelligence, orretry_stages` entries.- Low:
fileStoreUpdateSchemastill reuses the create fields, soharness_describereportsname/typeas required on update even though the preflight hydrates them.Open question / assumption:
- I’m assuming File Store writes are meant to be discoverable through the same public
resource_scopecontract thatharness_describeadvertises. If writes are intentionally project-only, the metadata should be narrowed instead.Verification:
pnpm test tests/client/harness-client.test.ts tests/registry/file-store.test.ts tests/utils/url-parser.test.ts tests/registry/registry.test.ts tests/tools/tool-handlers.test.ts tests/integration/mock-harness-api.test.ts-> 10 failing tests / 281 totalpnpm typecheck-> passpnpm build-> passpnpm docs:check-> fail (README.md is stale. Run \pnpm docs:generate` to refresh.`)Sent by Cursor Automation: Sunil On Demand Architecture Review
| /** Static headers to merge into the request (e.g. Content-Type override) */ | ||
| headers?: Record<string, string>; | ||
| /** Optional custom handler for multi-request read operations. */ | ||
| handler?: (ctx: EndpointHandlerContext) => Promise<unknown>; |
There was a problem hiding this comment.
This adds EndpointSpec.handler, but the current PR never teaches Registry.dispatch() to consult it before building the default single-request path. On the exact PR head, that leaves both new handler-backed features broken at runtime: tests/registry/registry.test.ts and tests/integration/mock-harness-api.test.ts only observe a single raw request for test_failure, and the new retry_stages cases in tests/tools/tool-handlers.test.ts fail for the same reason. Until dispatch branches on spec.handler, this abstraction is only type-level metadata.
| description: "Harness File Store file or folder metadata. Supports listing, metadata lookup, create, and update.", | ||
| toolset: "file-store", | ||
| scope: "project", | ||
| supportedScopes: ["account", "org", "project"], |
There was a problem hiding this comment.
supportedScopes is surfaced through harness_describe, but the public write tools still do not expose a top-level resource_scope input and they do not opt into URL-derived scope on create/update/delete. That means file_store_item now advertises account/org/project writes that agents can only reach through undocumented params.resource_scope plumbing. I think this should either be surfaced on harness_create / harness_update / harness_delete or narrowed so the advertised contract stays honest.
| supportedScopes: ["account", "org", "project"], | ||
| identifierFields: ["file_id"], | ||
| deepLinkTemplate: | ||
| "/ng/account/{accountId}/all/orgs/{orgIdentifier}/projects/{projectIdentifier}/setup/resources/file-store", |
There was a problem hiding this comment.
This deep link still does not round-trip the File Store URLs the PR now teaches the parser to accept. It omits the selected file_id, so project-scoped reads only land on the collection root, and for account-scoped reads buildDeepLink() collapses this to /ng/account/{id}/all/setup/resources/file-store instead of the tested /ng/account/{id}/all/settings/file-store/{id} route. So openInHarness will still misroute the fetched item.
| ## Resource Types | ||
|
|
||
| 169 resource types organized across 31 toolsets. Each resource type supports a subset of CRUD operations and optional execute actions. | ||
| 171 resource types organized across 32 toolsets. Each resource type supports a subset of CRUD operations and optional execute actions. |
There was a problem hiding this comment.
I re-ran pnpm build && pnpm docs:check on the exact PR head and this branch is still red: README.md is stale. Run \pnpm docs:generate` to refresh.Also, even after generating docs in an isolated worktree, the visible README catalog still has zero mentions offile-store, test-intelligence, test_failure, file_store_item, file_store_content, or retry_stages`, so the public contract is still incomplete even beyond the count drift.
|
|
||
| const fileStoreUpdateSchema: BodySchema = { | ||
| description: "File Store file or folder update. Missing name/type/parent_identifier are hydrated from existing metadata when possible.", | ||
| fields: fileStoreCreateSchema.fields, |
There was a problem hiding this comment.
fileStoreUpdateSchema still reuses the create fields verbatim, so harness_describe reports name and type as required even though hydrateFileStoreUpdate() backfills them and the new tests intentionally update with only content / mime_type. That makes the published update contract stricter than the runtime behavior.
There was a problem hiding this comment.
Findings:
- Critical:
EndpointSpec.handleris added but never executed byRegistry.executeSpec(). On the live PR head, both new multi-request features (test_failure.listandpipeline.retry_stages) fall back to the generic single-request path and fail their own tests. I reproduced this withpnpm test tests/registry/registry.test.ts tests/integration/mock-harness-api.test.ts tests/tools/tool-handlers.test.ts; the new cases expect 3-5 client calls and only observe 1. - Important:
README.mdis still stale on the live head.pnpm build && pnpm docs:checkfails, the README still claims171resource types /32toolsets and32 of 33defaults, while the built registry loads172/33by default (190/34withHARNESS_TOOLSETS=+ai-evals). The catalog also still has nofile-storeortest-intelligenceentries. - Important: File Store still over-advertises multi-scope writes.
supportedScopestells agents thatfile_store_itemsupports account/org/project scope, but the public write tools still do not exposeresource_scopeand still do not opt into URL-derived scope. That makes theharness_describecontract looser than the actualharness_create/harness_update/harness_deletesurface. - Medium: File Store
openInHarnesslinks still do not round-trip the new parser paths. The template omitsfile_idand only models the project collection route, so account-scoped reads and item-specific links both collapse to a generic list page.
Assumption:
- I’m treating
harness_describeandREADME.mdas agent/public contracts, consistent with the repo’s architecture rules.
Verification:
pnpm install --frozen-lockfilepnpm typecheckpnpm buildpnpm docs:checkpnpm test tests/registry/registry.test.ts tests/integration/mock-harness-api.test.ts tests/tools/tool-handlers.test.tspnpm test tests/registry/file-store.test.ts tests/utils/url-parser.test.ts
Sent by Cursor Automation: Sunil On Demand Architecture Review
| /** Static headers to merge into the request (e.g. Content-Type override) */ | ||
| headers?: Record<string, string>; | ||
| /** Optional custom handler for multi-request read operations. */ | ||
| handler?: (ctx: EndpointHandlerContext) => Promise<unknown>; |
There was a problem hiding this comment.
EndpointSpec.handler is only useful if the dispatcher actually consumes it. On the current PR head, Registry.executeSpec() never checks spec.handler, so both new call sites (test_failure.list and pipeline.retry_stages) still fall through to the generic single-request path. I reproduced that with pnpm test tests/registry/registry.test.ts tests/integration/mock-harness-api.test.ts tests/tools/tool-handlers.test.ts: the new flows expect 3-5 client calls and only see 1.
| ## Resource Types | ||
|
|
||
| 169 resource types organized across 31 toolsets. Each resource type supports a subset of CRUD operations and optional execute actions. | ||
| 171 resource types organized across 32 toolsets. Each resource type supports a subset of CRUD operations and optional execute actions. |
There was a problem hiding this comment.
This README refresh is still stale on the live PR head. After pnpm build, pnpm docs:check fails here, the registry loads 172 resource types / 33 toolsets by default (190 / 34 with HARNESS_TOOLSETS=+ai-evals), but the README still says 171 / 32 and 32 of 33. It also still has no file-store, test-intelligence, or test_failure entries, so the public docs contract is behind the actual registry surface.
| description: "Harness File Store file or folder metadata. Supports listing, metadata lookup, create, and update.", | ||
| toolset: "file-store", | ||
| scope: "project", | ||
| supportedScopes: ["account", "org", "project"], |
There was a problem hiding this comment.
supportedScopes is resource-wide metadata surfaced through harness_describe, so this now advertises account/org/project support for create/update/delete as well. The problem is that the public write tools still do not publish a top-level resource_scope input and still call applyUrlDefaults() without includeResourceScope, so agents cannot actually express those account/org File Store writes through the documented tool schemas. Either plumb resource_scope through harness_create / harness_update / harness_delete or narrow the advertised scope contract.
| supportedScopes: ["account", "org", "project"], | ||
| identifierFields: ["file_id"], | ||
| deepLinkTemplate: | ||
| "/ng/account/{accountId}/all/orgs/{orgIdentifier}/projects/{projectIdentifier}/setup/resources/file-store", |
There was a problem hiding this comment.
This template still cannot round-trip the new File Store URL-parser paths. It omits {file_id}, so every item links to the same collection page, and it only models the project route even though tests/utils/url-parser.test.ts now accepts account URLs like /all/settings/file-store/{id}. That leaves openInHarness one-way for account-scoped reads and non-specific even for project-scoped items.


Add File Store resources for metadata, uploads, and content reads through the existing registry-backed MCP tools.
Description
add file store support.
you can now