Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -283,6 +283,15 @@ While impact assessment is still pending, `next` and `explain` select that
action before deriving repository-wide Contract health. Unrelated Result and
Evidence history is not loaded for this first step.

When Contract health is required, ADF indexes Evidence and verification Results
once, then validates and hashes only records that can affect a Contract clause.
It does not repeatedly scan every Result for every clause.

New Results store shared input and freshness references once at the Result
level. An outcome carries its own references only when they differ from those
shared values. Existing Results remain readable and are not rewritten, so their
identities and downstream freshness checks remain stable.

Each action also carries advisory execution guidance. Impact assessment
normally recommends an economy model, while challenge recommends a
high-accuracy model. The listed escalation conditions tell an orchestrator when
Expand Down
4 changes: 4 additions & 0 deletions docs/concepts.ja.md
Original file line number Diff line number Diff line change
Expand Up @@ -97,6 +97,10 @@ Contractまたは条項の`evidence_mode`で、検証に掛ける費用を選べ

影響評価が済んでいない間、`adf next`と`adf explain`は、リポジトリ全体のContract検証状態を計算する前に影響評価を次の作業として選びます。この最初の段階では、無関係なChangeのResultとEvidenceを読み込みません。

Contract検証状態が必要な場合は、Evidenceと検証Resultを一度索引化し、Contract条項へ影響する記録だけを検証してハッシュを計算します。条項ごとに全Resultを繰り返し検索しません。

新しいResultでは、すべての判定結果に共通する入力参照と鮮度参照をResult全体へ一度だけ保存します。判定結果ごとの参照が共通値と異なる場合だけ、その判定結果にも保存します。既存Resultは引き続き読み込めます。識別子と後続Resultの鮮度判定を保つため、既存ファイルは自動で書き換えません。

各作業には、実行環境へ向けたモデルの推奨も含まれます。影響評価には通常、軽量なモデルを推奨します。ただし、影響なしと結論付ける場合、根拠が矛盾する場合、セキュリティ、プライバシー、決済、元に戻せないデータ変更の可能性がある場合は、精度の高いモデルへの切り替えを勧めます。ADF自体はモデルを選ばず、LLMも実行しません。

実行環境がすでに把握している処理時間、モデル名、入出力Token数、ツール呼び出し数、再試行回数は、`adf_submit`で任意に記録できます。外部Runnerは`adf_begin_execution`と`adf_complete_execution`を使い、Result提出後に確定したToken数や、失敗・中断した実行も追記できます。外部実行では、キャッシュ作成Token、キャッシュ読取Token、推論Token、実行環境が報告した米ドル費用も記録できます。この実行RecordはADFの状態、Result ID、鮮度、Evidence検証には影響しません。同じResultに提出時の計測値とRunnerの完了Recordがある場合は、Runnerの値だけを集計します。
Expand Down
4 changes: 4 additions & 0 deletions docs/implementation.md
Original file line number Diff line number Diff line change
Expand Up @@ -122,6 +122,10 @@ Explainでは、確認前を`applicability-pending`、支持された後を`not-

`contract-health`は、全ChangeのResult・Evidenceと現在のRepository観測から、Contract条項ごとの実装準拠状態を再生成します。

計算時には、条項ごとのEvidenceと、Evidenceを参照する検証Resultを一度索引化します。Schema検証と現在値のハッシュ計算は、Contract Healthへ影響するContract、Evidence、検証Result、参照先だけに限定します。Result全件を条項ごとに走査せず、無関係なResultの内容全体も検証しません。ただし、Filesystem StoreはすべてのResultファイルを列挙してJSONとして読み込むため、壊れたJSONは引き続きエラーになります。

Result提出時は、判定結果の`input_refs`と`freshness_refs`がResult全体の値と同じなら省略します。判定結果だけが追加または異なる参照を持つ場合は、その値を判定結果へ保存します。Kernelは判定結果に値がなければResult全体の値を使うため、既存形式と軽量形式を同じ意味で扱えます。Schema上で両項目はもともと省略可能なので、Schema versionは変更しません。既存Resultを変換するとResult IDと、それを参照する後続Resultの鮮度が変わるため、自動移行は行いません。

- `verified`: 成功Evidenceを参照するBuilder outcomeの入力digestがすべて現在値と一致する
- `stale`: 検証履歴はあるが、Contract・コード・設定・Evidence等の入力が変わった、または失われた
- `unverified`: 検証履歴がない
Expand Down
264 changes: 205 additions & 59 deletions src/contract_health.rs
Original file line number Diff line number Diff line change
Expand Up @@ -88,29 +88,18 @@ pub fn build_contract_health_report(
let project = project
.as_object()
.ok_or_else(|| health_error("project must be an object"))?;
for (collection, kind) in [
("changes", "change"),
("contracts", "contract"),
("decisions", "decision"),
("results", "result"),
("evidence", "evidence"),
] {
for collection in ["changes", "contracts", "decisions", "results", "evidence"] {
let records = record_array(project.get(collection), collection)?;
validate_unique_ids(records, collection)?;
for record in records {
schema_registry
.validate(kind, record)
.map_err(|error| health_error(format!("invalid {kind} record: {error}")))?;
}
}

let current_digests = current_artifact_digests(project)?;
let results = record_array(project.get("results"), "results")?;
let evidence = record_array(project.get("evidence"), "evidence")?;
let evidence_by_id = evidence
.iter()
.filter_map(|record| record["id"].as_str().map(|id| (id.to_owned(), record)))
.collect::<BTreeMap<_, _>>();
let verification_outcomes_by_evidence = verification_outcomes_by_evidence(results);
let required_current_refs = verification_freshness_refs(results);
validate_health_records(project, schema_registry, &required_current_refs)?;
let current_digests = current_artifact_digests(project, &required_current_refs)?;
let evidence_by_clause = evidence_by_clause(evidence);

let mut clauses = Vec::new();
let mut seen_clause_refs = BTreeSet::new();
Expand All @@ -137,8 +126,11 @@ pub fn build_contract_health_report(
clause,
&clause_applies_to,
evidence_mode,
results,
&evidence_by_id,
evidence_by_clause
.get(clause_ref.as_str())
.map(Vec::as_slice)
.unwrap_or(&[]),
&verification_outcomes_by_evidence,
&current_digests,
));
}
Expand All @@ -164,6 +156,117 @@ pub fn build_contract_health_report(
})
}

fn validate_health_records(
project: &serde_json::Map<String, Value>,
schema_registry: &SchemaRegistry,
required_refs: &BTreeSet<&str>,
) -> Result<(), ContractHealthError> {
for (collection, kind) in [
("changes", "change"),
("contracts", "contract"),
("decisions", "decision"),
("results", "result"),
("evidence", "evidence"),
] {
for record in record_array(project.get(collection), collection)? {
let record_id = required_string(record, "id", kind)?;
let required_for_health = matches!(collection, "contracts" | "evidence")
|| required_refs.contains(record_id)
|| (collection == "results"
&& record["result_schema"].as_str() == Some("result.evidence")
&& record["role"].as_str() == Some("Builder"));
if required_for_health {
schema_registry
.validate(kind, record)
.map_err(|error| health_error(format!("invalid {kind} record: {error}")))?;
}
}
}
Ok(())
}

#[derive(Clone, Copy)]
struct VerificationOutcome<'a> {
result: &'a Value,
outcome: &'a Value,
}

fn evidence_by_clause(evidence: &[Value]) -> BTreeMap<&str, Vec<(&str, &Value)>> {
let mut by_clause = BTreeMap::new();
for record in evidence {
let Some(evidence_id) = record["id"].as_str() else {
continue;
};
for clause_ref in record["contract_clause_refs"]
.as_array()
.into_iter()
.flatten()
.filter_map(Value::as_str)
{
by_clause
.entry(clause_ref)
.or_insert_with(Vec::new)
.push((evidence_id, record));
}
}
by_clause
}

fn verification_outcomes_by_evidence(
results: &[Value],
) -> BTreeMap<&str, Vec<VerificationOutcome<'_>>> {
let mut by_evidence = BTreeMap::new();
for result in results.iter().filter(|result| {
result["result_schema"].as_str() == Some("result.evidence")
&& result["role"].as_str() == Some("Builder")
}) {
for outcome in result["payload"]["outcomes"]
.as_array()
.into_iter()
.flatten()
{
for evidence_ref in outcome["basis_refs"]
.as_array()
.into_iter()
.flatten()
.filter_map(Value::as_str)
{
by_evidence
.entry(evidence_ref)
.or_insert_with(Vec::new)
.push(VerificationOutcome { result, outcome });
}
}
}
by_evidence
}

fn verification_freshness_refs(results: &[Value]) -> BTreeSet<&str> {
let mut references = BTreeSet::new();
for result in results.iter().filter(|result| {
result["result_schema"].as_str() == Some("result.evidence")
&& result["role"].as_str() == Some("Builder")
}) {
for outcome in result["payload"]["outcomes"]
.as_array()
.into_iter()
.flatten()
{
let manifest = outcome
.get("freshness_refs")
.or_else(|| result.get("freshness_refs"))
.or_else(|| result.get("input_refs"));
references.extend(
manifest
.and_then(Value::as_object)
.into_iter()
.flat_map(|manifest| manifest.keys().map(String::as_str)),
);
}
}
references
}

#[allow(clippy::too_many_arguments)]
fn clause_health(
contract_id: &str,
Expand All @@ -172,8 +275,8 @@ fn clause_health(
clause: &Value,
applies_to: &[String],
evidence_mode: ClauseEvidenceMode,
results: &[Value],
evidence_by_id: &BTreeMap<String, &Value>,
evidence_for_clause: &[(&str, &Value)],
verification_outcomes_by_evidence: &BTreeMap<&str, Vec<VerificationOutcome<'_>>>,
current_digests: &BTreeMap<String, String>,
) -> ClauseHealth {
let mut evidence_refs = BTreeSet::new();
Expand All @@ -183,47 +286,42 @@ fn clause_health(
let mut current_failure = false;
let mut historical_verification = false;

for (evidence_id, evidence) in evidence_by_id {
for (evidence_id, evidence) in evidence_for_clause {
if !evidence_covers_clause(evidence, clause_ref, evidence_mode) {
continue;
}
evidence_refs.insert(evidence_id.clone());
for result in results.iter().filter(|result| {
result["result_schema"].as_str() == Some("result.evidence")
&& result["role"].as_str() == Some("Builder")
&& result["change_id"] == evidence["change_id"]
}) {
for outcome in result["payload"]["outcomes"]
.as_array()
.map(Vec::as_slice)
.unwrap_or(&[])
.iter()
.filter(|outcome| string_array(&outcome["basis_refs"]).contains(evidence_id))
{
let Some(result_id) = result["id"].as_str() else {
continue;
};
verification_result_ids.insert(result_id.to_owned());
let mismatches = stale_refs_for_outcome(
outcome,
result,
current_digests,
&[contract_id, evidence_id],
);
if mismatches.is_empty() {
match (evidence["outcome"].as_str(), outcome["status"].as_str()) {
(Some("passed"), Some("satisfied"))
if evidence_verifies_outcome(evidence, outcome) =>
{
current_success = true;
}
(Some("failed" | "inconclusive"), _) => current_failure = true,
_ => {}
evidence_refs.insert((*evidence_id).to_owned());
for verification in verification_outcomes_by_evidence
.get(evidence_id)
.into_iter()
.flatten()
.filter(|verification| verification.result["change_id"] == evidence["change_id"])
{
let result = verification.result;
let outcome = verification.outcome;
let Some(result_id) = result["id"].as_str() else {
continue;
};
verification_result_ids.insert(result_id.to_owned());
let mismatches = stale_refs_for_outcome(
outcome,
result,
current_digests,
&[contract_id, evidence_id],
);
if mismatches.is_empty() {
match (evidence["outcome"].as_str(), outcome["status"].as_str()) {
(Some("passed"), Some("satisfied"))
if evidence_verifies_outcome(evidence, outcome) =>
{
current_success = true;
}
} else {
historical_verification = true;
stale_refs.extend(mismatches);
(Some("failed" | "inconclusive"), _) => current_failure = true,
_ => {}
}
} else {
historical_verification = true;
stale_refs.extend(mismatches);
}
}
}
Expand Down Expand Up @@ -310,20 +408,26 @@ fn is_sha256_digest(value: &str) -> bool {

fn current_artifact_digests(
project: &serde_json::Map<String, Value>,
required_refs: &BTreeSet<&str>,
) -> Result<BTreeMap<String, String>, ContractHealthError> {
let mut digests = BTreeMap::new();
for collection in ["changes", "contracts", "decisions", "results", "evidence"] {
for record in record_array(project.get(collection), collection)? {
let id = required_string(record, "id", "Project record")?;
digests.insert(id.to_owned(), digest_value(record)?);
if required_refs.contains(id) {
digests.insert(id.to_owned(), digest_value(record)?);
}
if collection == "contracts" {
for clause in record["clauses"]
.as_array()
.map(Vec::as_slice)
.unwrap_or(&[])
{
let clause_id = required_string(clause, "id", "Contract clause")?;
digests.insert(format!("{id}#{clause_id}"), digest_value(clause)?);
let clause_ref = format!("{id}#{clause_id}");
if required_refs.contains(clause_ref.as_str()) {
digests.insert(clause_ref, digest_value(clause)?);
}
}
}
}
Expand All @@ -335,6 +439,9 @@ fn current_artifact_digests(
.unwrap_or(&[])
{
let reference = required_string(artifact, "ref", "Repository artifact")?;
if !required_refs.contains(reference) {
continue;
}
let digest = artifact["digest"]
.as_str()
.filter(|value| !value.is_empty())
Expand Down Expand Up @@ -713,4 +820,43 @@ mod tests {
["result.retry", "result.test"]
);
}

#[test]
fn current_digests_skip_records_unrelated_to_contract_verification() {
let project = json!({
"changes": [],
"contracts": [],
"decisions": [],
"results": [{"id": "result.unrelated", "unsupported": 1.5}],
"evidence": [],
"repository": {"artifacts": []}
});
let project = project.as_object().unwrap();

assert!(current_artifact_digests(project, &BTreeSet::new()).is_ok());
let required = BTreeSet::from(["result.unrelated"]);
assert!(current_artifact_digests(project, &required).is_err());
}

#[test]
fn report_validates_only_records_that_can_affect_contract_health() {
let project = json!({
"changes": [],
"contracts": [],
"decisions": [],
"results": [{
"id": "result.unrelated",
"result_schema": "result.challenge"
}],
"evidence": [],
"repository": {"artifacts": []}
});
let report = build_contract_health_report(&project, &registry()).unwrap();
assert_eq!(report.summary.total, 0);

let mut relevant = project;
relevant["results"][0]["result_schema"] = json!("result.evidence");
relevant["results"][0]["role"] = json!("Builder");
assert!(build_contract_health_report(&relevant, &registry()).is_err());
}
}
2 changes: 1 addition & 1 deletion src/explain.rs
Original file line number Diff line number Diff line change
Expand Up @@ -384,7 +384,7 @@ fn requirement_trace(
&& role_matches
&& stale_refs.is_empty()
&& (instance.assurance == Assurance::Attestation
|| outcome_has_current_evidence(instance, outcome, snapshot));
|| outcome_has_current_evidence(instance, result, outcome, snapshot));
result_checks.push(ResultCheck {
result_id: string_field(result, "id").unwrap_or("").to_owned(),
outcome_status,
Expand Down
Loading
Loading