fix(evaluation): disable AFC in LlmAsJudge's default judge request config - #7147
Open
chelsealong wants to merge 1 commit into
Open
chelsealong wants to merge 1 commit into
chelsealong wants to merge 1 commit into
Conversation
…nfig The judge's default GenerateContentConfig() leaves automatic_function_calling unset, so google-genai takes the AFC branch in AsyncModels.generate_content and logs a spurious warning on every eval run, even though the judge sends no tools and never needs AFC. Explicitly disabling it removes the warning without changing the request sent to the model (the field is client-side only).
|
Verified offline against google-adk 2.9.1 and google-genai 2.23.0, through
#7147 makes that exact config the default fallback, so this confirms it removes the warning for criteria that set no |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #7146
Problem
Every rubric-based eval run logs this google-genai warning once per process:
It comes from the LLM-as-judge request, not the agent under evaluation.
LlmAsJudge.evaluate_invocationsbuilds the judge request with:The default
GenerateContentConfig()leavesautomatic_function_callingunset. google-genai'sAsyncModels.generate_contentthen takes its AFC branch and logs the warning, because there are no tools to make the direct_generate_contentpath get selected instead. The judge never sends tools and never needs AFC.Fix
Explicitly disable automatic function calling in the default fallback config:
This field is client-side only, so it does not change the request sent to the model. Users who pass their own
judge_model_configare unaffected.Testing plan
Added
test_evaluate_invocations_default_config_disables_afcintests/unittests/evaluation/test_llm_as_judge.py, which builds a judge with no explicitjudge_model_config, captures theLlmRequestpassed to the judge model, and assertsconfig.automatic_function_calling.disable is True.Verified the new test fails without the fix (reverted the source file only, keeping the test):
With the fix applied:
Full unit test suite:
The single failure (
test_local_eval_service.py::test_eval_injects_session_input_state_into_instruction) is pre-existing and unrelated to this change — it fails identically on unmodifiedupstream/main.Also ran
isortandpyinkon the changed files; no formatting changes needed.AI disclosure
This change was prepared with the assistance of an AI coding agent (Claude), including drafting the fix, test, and this PR description. The diff was reviewed and verified (tests run, fix confirmed to be caught by the new test) before submission.