[DT-3826] Formal access contract

[otchet-broad]

Addresses

DT-3826 — Ticket A-2, formal Elasticsearch
access contract.

Security risk: low — no change to authorization behavior. The contract is a written decision
record; the only production-path code change is how the capability report resolves the REST client's
version string. The admin-only capability endpoint itself lands in the base PR
(DT-3825).

Stacked on otchet-dt-3825-confirm-es-in-env — review that PR first, or read this one's diff against
that branch rather than develop.

Summary

Writes the formal access contract for dataset search, records the production A-1 measurement, and
adjusts the downstream plan to match what both turned up.

The contract — docs/plans/es-access-contract.md — is the document B-1, C-1, D-3, and E-2/E-3 get
implemented against. It enumerates every access dimension, its source of truth, what it does today,
and what the contract commits to, in mechanism-neutral terms so it holds identically under native
DLS/FLS (Epic D) and the application-layer fallback (Epic E). It is stated as Complete and
unblocked: no OPEN item stands between it and B-1/B-3/C-1.

Three decisions the contract settles

1. Restricted documents are invisible, not redacted — a caller unauthorized for a dataset gets
   no document and no hit-count contribution. Redaction leaks by counting, can't be expressed in
   native FLS, and would loosen today's DatasetService.verifyPublicVisibilityAccess behavior by
   accident.
2. Search serves one field bundle to every caller — including admins. Elasticsearch binds an FLS
   grant to the index privilege, not the document, so the planned per-document
   fieldAccessProfile marker has nothing to bind to. The obvious repair (a per-caller "privileged"
   bundle) is also unsound: DLS is a disjunction, so a custodian of one restricted dataset would get
   privileged fields projected out of every unrelated public document in the result set.
3. Preserve today's authorization; every expansion is explicit — the default for each dimension
   is PRESERVE. DAC membership, DAC chair, dacApproval, institution, policy tags, and principal
   allowlists grant no read access today and are DEFERred, so a security refactor can't become a
   silent authorization change.

Findings that change other tickets

• B-2 is cancelled — it specifies a mechanism Elasticsearch does not have.
• B-1 must drop fieldAccessProfile and hold back the four speculative dimensions rather than
  shipping unpopulated fields that read as enforcement.
• Dataset creator and study creator are distinct columns and distinct privileged paths; both
  need indexing, and the filter compares the caller's one ID against both.
• Custodian matching is case-sensitive and trims only the stored side — Alice@x.org fails
  against a stored alice@x.org (OPEN-6).
• A-3's "no new storage needed" for dataCustodianEmail is corrected — it lives in a JSON
  property bag with no referential integrity, no normalization, and no index. Workable for search
  once B-3 denormalizes it; still a gap for the non-search endpoints, and it makes
  reindex-on-custodian-change a correctness requirement.
• publicVisibility = NULL was recorded as an open policy question and is not one — the column
  is NOT NULL; the null in summary rows is the LEFT JOIN. Resolved by the schema, filter fails
  closed.

Seven OPEN items remain in §E, each a proposed change (expand, tighten, or fix a defect) with a
preserve-today default, so none blocks Phase 1.

A-1: production measured

es-security-capability-record.md records the production run (2026-08-05, write probes): Elastic
Cloud, enterprise license, all five capabilities observed — DLS and FLS enforced, not merely
accepted; keys created, used, and torn down cleanly. Four findings carried forward: Epic D's key
minting is not privilege-blocked in production (contradicting the prior assumption, in the
permissive direction — the breadth of that credential is worth its own review); the run_as
evidence is self-impersonation only; cluster-side audit logging must not be assumed; and Elastic
Cloud reserves some settings to Elastic's operators, making any future infra ask a support request.

Environment specifics stay on the ticket rather than in this public repo; the record keeps only what
the Epic D / Epic E decision rule consumes. The A-1 decision stays pending on dev and staging —
production being the environment most likely to carry the strongest license is exactly why the other
two can't be assumed to match it.

Code changes

• rest_client_compatibility came back indeterminate in production and would have in every
  deployed environment: the shade plugin strips dependency manifests, so
  RestClient.class.getPackage().getImplementationVersion() is null in the uber jar while working
  fine locally. Fixed by falling back to elasticsearch.rest.client.version in the build-generated
  mvn.properties (the mechanism SwaggerResource already uses), with the dependency version
  promoted to a pom property so it lands there. The package lookup is still tried first. Verified
  against the built consent.jar.
• track_total_hits=true on the DLS enforcement probe — the default caps hits.total.value at
  10 000, so on a larger index the evidence string would report "10000 of N" rather than the real
  count. The verdict was unaffected; the quoted evidence wasn't.
• ConsentModule javadoc documents the ordering dependency in the shared RestClient provider:
  the shutdown hook is registered at provision time, which is safe only while every consumer is
  reached from ConsentApplication.run.

Docs correction

config/ is git-ignored and rendered per developer by firecloud-develop, so nothing in this
repository sets any Elasticsearch default. DEVNOTES and the capability record previously implied a
committed compose file; both now say the durable change belongs in the firecloud-develop template
(outside this repo, still needs an owner) and that each developer applies the settings to their own
rendered copy meanwhile.

Testing

Three unit tests added to ElasticSearchCapabilityServiceTest: one guarding the pom wiring itself
(inlining the dependency version again would silently return the report to "Could not determine" in
exactly the environments it's used to decide in, and nothing else in the suite would notice), one
for an absent build property, and one confirming a matching major version names the resolved client
version. Existing search-probe assertions updated for the new query string.

No UI or workflow change, so no PO sign-off needed. Post-deploy verification is a GET /api/elasticSearch/capabilities against dev, which should now name the client version rather than
declining to.

Have you read CONTRIBUTING.md lately? If not, do that first.

• Label PR with a Jira ticket number and include a link to the ticket
• Label PR with a security risk modifier [no, low, medium, high]
• PR describes scope of changes
• Get a minimum of one thumbs worth of review, preferably two if enough team members are available
• Get PO sign-off for all non-trivial UI or workflow changes
• Verify all tests go green
• Test this change deployed correctly and works on dev environment after deployment

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
59.3% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[Copilot]

Pull request overview

This PR formalizes and records the Elasticsearch dataset-search access contract (mechanism-neutral across native DLS/FLS vs app-layer fallback), and adjusts the Elasticsearch capability report to reliably resolve the REST client version in deployed (shaded) builds.

Changes:

• Add a detailed access-contract document defining document visibility + field exposure rules and how downstream tickets must align.
• Fix capability reporting by falling back to mvn.properties for elasticsearch-rest-client version, and add track_total_hits=true to the DLS enforcement probe for accurate evidence strings.
• Update/build-wire tests and docs to reflect the contract decisions and corrected local-config expectations.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
src/test/resources/mvn.properties	Adds the REST client version property to the test-classpath mvn.properties shadow file.
src/test/java/org/broadinstitute/consent/http/service/ElasticSearchCapabilityServiceTest.java	Expands/updates capability-service tests for new probe query, build-property fallback, and additional response-shape cases.
src/main/java/org/broadinstitute/consent/http/service/ElasticSearchCapabilityService.java	Implements REST client version fallback via mvn.properties and uses track_total_hits=true in enforcement probe.
src/main/java/org/broadinstitute/consent/http/ConsentModule.java	Adds provider-level Javadoc clarifying Dropwizard lifecycle ordering expectations for RestClient shutdown hook registration.
pom.xml	Promotes the REST client version to a Maven property so it’s emitted into mvn.properties by properties-maven-plugin.
docs/plans/es-security-capability-record.md	Records production measurement details and clarifies local config realities (rendered/ignored config/).
docs/plans/es-access-contract.md	Adds the formal access contract document and enumerates decided rules + OPEN policy items.
docs/plans/elasticsearch-service-duos-ui-usage.md	Updates the plan to align with the access contract decisions (e.g., cancel per-document FLS marker, single allowlist).
DEVNOTES.md	Clarifies that config/ isn’t version-controlled and updates local security enablement guidance accordingly.

[kevinmarete]

Here are some comments:

• [P1] Native DLS does not guarantee the stated count/aggregation invisibility.
  Decision 1 says restricted documents cannot contribute to counts, facets, or aggregations and that DLS filters before scoring/counting. Elastic explicitly documents that DLS still permits search requests that infer terms and counts from inaccessible documents through aggregate information. The native path therefore needs query mediation/restrictions too, or the contract must weaken this guarantee. See Elastic’s DLS limitations.

• [P1] The fallback allowlist only filters _source, leaving other extraction channels open.
  E-3 filters only hits.hits[*]._source, while E-1 preserves caller-supplied aggregations, sorting, and highlighting. A caller could aggregate accessPolicy.custodianEmails, highlight a dynamic-map field, or receive an INTERNAL field through hit sort values. The design should validate every field reference against an allowed query-field list and filter all response channels—not just _source.

• [P1] The supposedly exhaustive field audit marks several live UI dependencies INTERNAL.
  The classification says these have no consumers, but current duos-ui/develop uses:

  • requestLocation in dataset details and the library’s “Request Path”
  • deletable for the submissions delete action
  • createUserId and study.dataSubmitterId to construct “My Data Submissions”
  • submitter.displayName in the DAC datasets table

  Generating FLS/E-3 allowlists from this table would break those flows. The audit should cover all search callers and distinguish response-visible fields from fields that must remain queryable.

[rushtong]

Looks good, thank you 👍🏽

[rushtong]

I can see this happening. For example, we removed approvedUserIds which had the potential for exposure of partially controlled info. The need for something similar could resurface. I think the risk is low, but non-zero and there are likely other workarounds we can come up with.

[otchet-broad]

Agreed on both fronts. The bigger issue I'm coming back to is aggregation across hidden documents from @kevinmarete's comments. I had forgotten about that leaky behavior in ES and need to think more about the consequences for this.

[fboulnois]

Thanks for verifying the behavior 👍