fix(evaluation): include grounding metadata in hallucinations judge context - #7159
Open
chelsealong wants to merge 1 commit into
Open
chelsealong wants to merge 1 commit into
chelsealong wants to merge 1 commit into
Conversation
…ontext HallucinationsV1Evaluator._create_context_for_step never included InvocationEvent.grounding_metadata in the context passed to the sentence validator, unlike the sibling rubric_based_final_response_quality_v1 metric which already calls get_grounding_metadata_as_json_str. Agents that rely on model-internal grounding tools (e.g. VertexAiSearchTool, google_search) only ever surface search results via event.grounding_metadata, never as a function_response part, so genuinely grounded sentences had no supporting evidence in the judge's context and were scored unsupported. Closes: google#7157
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Please ensure you have read the contribution guide before creating a pull request.
Link to Issue or Description of Change
1. Link to an existing issue (if applicable):
Problem:
HallucinationsV1Evaluator._create_context_for_step(src/google/adk/evaluation/hallucinations_v1.py) never included model-supplied grounding metadata (InvocationEvent.grounding_metadata) in the context string it builds for the sentence-validator judge — it only assembled developer instructions, user prompt, tool declarations, and per-eventfunction_call/function_response/text parts. For agents that use built-in/model-internal grounding tools (e.g.VertexAiSearchTool,google_search), search results only ever arrive asgroundingMetadataon the event, never as afunction_responsepart. As a result, sentences that were genuinely grounded by search had no supporting evidence in the judge's context, and the judge labeled themunsupported, inflating the hallucination rate for any search-grounded agent. The sibling metricrubric_based_final_response_quality_v1.pyalready handles this correctly viaget_grounding_metadata_as_json_str(added in #5834 for #5831), buthallucinations_v1.pywas not updated at the time.Solution:
Apply the same pattern used by
rubric_based_final_response_quality_v1.py: callget_grounding_metadata_as_json_str(fromllm_as_judge_utils.py) on the accumulated events for the current step and inject the result as aGrounding metadata:block in the context string, alongside the existingTool definitions:block. Also added a short instruction to the sentence-validator prompt clarifying thatGrounding metadatais trusted evidence from model-internal tools whose results may not otherwise appear intool_outputs.Testing Plan
Unit Tests:
Added
TestCreateContext::test_create_context_for_step_includes_grounding_metadata, which builds anInvocationEventwithgrounding_metadataset (and nocontent) and asserts the resulting context contains aGrounding metadata:block with the serializedweb_search_queries. Confirmed this test (and 6 pre-existing ones whose expected context strings needed the new block) fails without the fix:Also ran the broader evaluation test directory (excluding files that error on collection due to optional
gcp/docsextras —pandas,vertexai, etc. — not installed in this sandbox; unrelated to this change): 567 passed.Lint (
ruff,isort,pyink --check) clean on both changed files.Manual End-to-End (E2E) Tests:
Not run — this is a prompt/context-construction change in an eval metric with no UI surface; covered by unit tests above.
Checklist
Additional context
This PR was prepared with AI assistance (Claude Code), per this repo's AI-Assisted Development guidance. All changes were reviewed and verified by running the test suite and lint locally before pushing.