Skip to content

fix(evaluation): include grounding metadata in hallucinations judge context - #7159

Open
chelsealong wants to merge 1 commit into
google:mainfrom
chelsealong:fix-7157-hallucinations-grounding-metadata
Open

chelsealong wants to merge 1 commit into
google:mainfrom
chelsealong:fix-7157-hallucinations-grounding-metadata

Conversation

@chelsealong

Copy link
Copy Markdown
Contributor

Please ensure you have read the contribution guide before creating a pull request.

Link to Issue or Description of Change

1. Link to an existing issue (if applicable):

Problem:
HallucinationsV1Evaluator._create_context_for_step (src/google/adk/evaluation/hallucinations_v1.py) never included model-supplied grounding metadata (InvocationEvent.grounding_metadata) in the context string it builds for the sentence-validator judge — it only assembled developer instructions, user prompt, tool declarations, and per-event function_call/function_response/text parts. For agents that use built-in/model-internal grounding tools (e.g. VertexAiSearchTool, google_search), search results only ever arrive as groundingMetadata on the event, never as a function_response part. As a result, sentences that were genuinely grounded by search had no supporting evidence in the judge's context, and the judge labeled them unsupported, inflating the hallucination rate for any search-grounded agent. The sibling metric rubric_based_final_response_quality_v1.py already handles this correctly via get_grounding_metadata_as_json_str (added in #5834 for #5831), but hallucinations_v1.py was not updated at the time.

Solution:
Apply the same pattern used by rubric_based_final_response_quality_v1.py: call get_grounding_metadata_as_json_str (from llm_as_judge_utils.py) on the accumulated events for the current step and inject the result as a Grounding metadata: block in the context string, alongside the existing Tool definitions: block. Also added a short instruction to the sentence-validator prompt clarifying that Grounding metadata is trusted evidence from model-internal tools whose results may not otherwise appear in tool_outputs.

Testing Plan

Unit Tests:

  • I have added or updated unit tests for my change.
  • All unit tests pass locally.
$ python -m pytest tests/unittests/evaluation/test_hallucinations_v1.py tests/unittests/evaluation/test_llm_as_judge_utils.py tests/unittests/evaluation/test_rubric_based_final_response_quality_v1.py -q
........................................................                 [100%]
56 passed in 1.12s

Added TestCreateContext::test_create_context_for_step_includes_grounding_metadata, which builds an InvocationEvent with grounding_metadata set (and no content) and asserts the resulting context contains a Grounding metadata: block with the serialized web_search_queries. Confirmed this test (and 6 pre-existing ones whose expected context strings needed the new block) fails without the fix:

$ git checkout HEAD~1 -- src/google/adk/evaluation/hallucinations_v1.py
$ python -m pytest tests/unittests/evaluation/test_hallucinations_v1.py -q
...
FAILED tests/unittests/evaluation/test_hallucinations_v1.py::TestCreateContext::test_create_context_for_intermediate_step
FAILED tests/unittests/evaluation/test_hallucinations_v1.py::TestCreateContext::test_create_context_for_final_step
FAILED tests/unittests/evaluation/test_hallucinations_v1.py::TestCreateContext::test_create_context_for_step_includes_grounding_metadata - IndexError: list index out of range
FAILED tests/unittests/evaluation/test_hallucinations_v1.py::TestEvaluateInvocationsAgentTree::test_evaluate_invocations_multi_agents
FAILED tests/unittests/evaluation/test_hallucinations_v1.py::TestEvaluateInvocationsAgentTree::test_evaluate_invocations_agent_tree_skip_intermediate
FAILED tests/unittests/evaluation/test_hallucinations_v1.py::TestEvaluateInvocationsTimeWeather::test_evaluate_invocations_time_weather
FAILED tests/unittests/evaluation/test_hallucinations_v1.py::TestEvaluateInvocationsTimeWeather::test_evaluate_invocations_time_weather_skip_intermediate
7 failed, 14 passed in 1.30s
$ git checkout HEAD -- src/google/adk/evaluation/hallucinations_v1.py   # fix restored

Also ran the broader evaluation test directory (excluding files that error on collection due to optional gcp/docs extras — pandas, vertexai, etc. — not installed in this sandbox; unrelated to this change): 567 passed.

Lint (ruff, isort, pyink --check) clean on both changed files.

Manual End-to-End (E2E) Tests:

Not run — this is a prompt/context-construction change in an eval metric with no UI surface; covered by unit tests above.

Checklist

  • I have read the CONTRIBUTING.md document.
  • I have performed a self-review of my own code.
  • I have commented my code, particularly in hard-to-understand areas.
  • I have added tests that prove my fix is effective or that my feature works.
  • New and existing unit tests pass locally with my changes.
  • I have manually tested my changes end-to-end.
  • Any dependent changes have been merged and published in downstream modules.

Additional context

This PR was prepared with AI assistance (Claude Code), per this repo's AI-Assisted Development guidance. All changes were reviewed and verified by running the test suite and lint locally before pushing.

…ontext

HallucinationsV1Evaluator._create_context_for_step never included
InvocationEvent.grounding_metadata in the context passed to the sentence
validator, unlike the sibling rubric_based_final_response_quality_v1
metric which already calls get_grounding_metadata_as_json_str. Agents
that rely on model-internal grounding tools (e.g. VertexAiSearchTool,
google_search) only ever surface search results via
event.grounding_metadata, never as a function_response part, so
genuinely grounded sentences had no supporting evidence in the judge's
context and were scored unsupported.

Closes: google#7157
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

hallucinations_v1: sentence validator ignores grounding_metadata

2 participants