Skip to content

fix(evaluation): disable AFC in LlmAsJudge's default judge request config - #7147

Open
chelsealong wants to merge 1 commit into
google:mainfrom
chelsealong:fix-7146-llm-judge-afc-warning
Open

chelsealong wants to merge 1 commit into
google:mainfrom
chelsealong:fix-7146-llm-judge-afc-warning

Conversation

@chelsealong

Copy link
Copy Markdown
Contributor

Fixes #7146

Problem

Every rubric-based eval run logs this google-genai warning once per process:

WARNING google_genai.models: Direct use of automatic function calling (AFC) in AsyncModels.generate_content is not recommended. ...

It comes from the LLM-as-judge request, not the agent under evaluation. LlmAsJudge.evaluate_invocations builds the judge request with:

config=self._judge_model_options.judge_model_config or genai_types.GenerateContentConfig(),

The default GenerateContentConfig() leaves automatic_function_calling unset. google-genai's AsyncModels.generate_content then takes its AFC branch and logs the warning, because there are no tools to make the direct _generate_content path get selected instead. The judge never sends tools and never needs AFC.

Fix

Explicitly disable automatic function calling in the default fallback config:

config=self._judge_model_options.judge_model_config
or genai_types.GenerateContentConfig(
    automatic_function_calling=genai_types.AutomaticFunctionCallingConfig(
        disable=True
    )
),

This field is client-side only, so it does not change the request sent to the model. Users who pass their own judge_model_config are unaffected.

Testing plan

Added test_evaluate_invocations_default_config_disables_afc in tests/unittests/evaluation/test_llm_as_judge.py, which builds a judge with no explicit judge_model_config, captures the LlmRequest passed to the judge model, and asserts config.automatic_function_calling.disable is True.

Verified the new test fails without the fix (reverted the source file only, keeping the test):

$ git checkout HEAD~1 -- src/google/adk/evaluation/llm_as_judge.py
$ python -m pytest tests/unittests/evaluation/test_llm_as_judge.py -k default_config_disables_afc -v
...
FAILED tests/unittests/evaluation/test_llm_as_judge.py::test_evaluate_invocations_default_config_disables_afc - assert None is not None
1 failed, 10 deselected in 1.16s
$ git checkout HEAD -- src/google/adk/evaluation/llm_as_judge.py

With the fix applied:

$ python -m pytest tests/unittests/evaluation/test_llm_as_judge.py -v
...
11 passed in 2.24s

Full unit test suite:

$ python -m pytest tests/unittests -q
1 failed, 15053 passed, 85 skipped, 27 xfailed, 2 xpassed, 2078 warnings, 28 subtests passed in 501.61s (0:08:21)

The single failure (test_local_eval_service.py::test_eval_injects_session_input_state_into_instruction) is pre-existing and unrelated to this change — it fails identically on unmodified upstream/main.

Also ran isort and pyink on the changed files; no formatting changes needed.

AI disclosure

This change was prepared with the assistance of an AI coding agent (Claude), including drafting the fix, test, and this PR description. The diff was reviewed and verified (tests run, fix confirmed to be caught by the new test) before submission.

…nfig

The judge's default GenerateContentConfig() leaves
automatic_function_calling unset, so google-genai takes the AFC branch
in AsyncModels.generate_content and logs a spurious warning on every
eval run, even though the judge sends no tools and never needs AFC.
Explicitly disabling it removes the warning without changing the
request sent to the model (the field is client-side only).
@roanny

roanny commented Sep 17, 2026

Copy link
Copy Markdown

Verified offline against google-adk 2.9.1 and google-genai 2.23.0, through RubricBasedFinalResponseQualityV1Evaluator.evaluate_invocations with AsyncModels._generate_content stubbed (no network):

  • Criterion without judge_model_config (ADK's default GenerateContentConfig()): the judge call takes the AFC branch and google_genai.models logs the AFC warning.
  • Criterion with judge_model_options.judge_model_config = {"automatic_function_calling": {"disable": true}}, parsed from the eval config JSON: the config reaches the transport with automatic_function_calling.disable=True, the call takes the direct path, and no warning is logged.

#7147 makes that exact config the default fallback, so this confirms it removes the warning for criteria that set no judge_model_config. Thanks for picking it up.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

LLM-as-judge default config triggers google-genai's AFC warning in every eval run

3 participants