Skip to content

feat(gooddata-eval): capture reasoning_steps through the agentic-CLI path - #1712

Merged
Tomkess merged 1 commit into
feat/chat-client-reasoning-stepsfrom
fix/agentic-reasoning-steps
Aug 5, 2026
Merged

feat(gooddata-eval): capture reasoning_steps through the agentic-CLI path#1712
Tomkess merged 1 commit into
feat/chat-client-reasoning-stepsfrom
fix/agentic-reasoning-steps

Conversation

@Tomkess

@Tomkess Tomkess commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Summary

Stacked on #1708. That PR wires ChatResult.reasoning_steps through runner.py's generic single-turn path (ChatBackend.ask()), but never through the agentic-CLI path (cli/agentic_runner.pyevaluate_agentic_*). As a result, agentic_alert_skill, agentic_metric_skill, and agentic_conversation items could never carry a reasoning trace — run_agentic_items builds its own bare ItemReport and reasoning_steps was simply never assigned, regardless of what the platform emitted.

  • alert_skill.py / metric_skill.py / conversation.py: accumulate chat_result.reasoning_steps across every send_message call in each evaluator's run loop; attach to the run/turn result.
  • evaluate_agentic_alert_skill / evaluate_agentic_metric_skill / evaluate_agentic_conversation: return the reasoning steps on pass; attach them to the raised exception on fail (mirrors the existing conversation_id-on-exception idiom in ChatClient.ask()).
  • cli/agentic_runner.py: _dispatch_agentic now returns whatever evaluate_agentic_* returns; run_agentic_items writes it onto ItemReport.reasoning_steps from either the return value or the caught exception.

general_question/guardrail/search_tool/visualization evaluators are untouched — still return None, no behavior change.

Confirmed live (not just unit tests): ran a real agentic_alert_skill question end-to-end against a live workspace and got a populated reasoning trace where before there was none.

Test plan

  • packages/gooddata-eval test suite: 268/268 pass (13 new tests added)
  • ruff check clean
  • Live end-to-end run against a real workspace produced a non-empty reasoning trace for agentic_alert_skill

…path

6001d2f wired ChatResult.reasoning_steps through runner.py's generic
single-turn path only. The agentic-CLI path (cli/agentic_runner.py ->
evaluate_agentic_*) builds its own ItemReport and never touched it, so
agentic_alert_skill/agentic_metric_skill/agentic_conversation items could
never produce a reasoning trace, no matter what the platform emitted.

Accumulates reasoning_steps across every send_message call in each of the
three evaluators' run loops, attaches it to the run/turn result, and
surfaces it from evaluate_agentic_* either as the return value (pass) or
as an attribute on the raised exception (fail) -- mirroring the existing
conversation_id-on-exception idiom in ChatClient.ask(). run_agentic_items
picks it up from either path onto ItemReport.reasoning_steps, which
json_report.py already serializes unconditionally.

general_question/guardrail/search_tool/visualization are left untouched --
their evaluate_agentic_* functions still return None, unchanged.
@Tomkess
Tomkess requested review from hkad98, lupko and pcerny as code owners August 5, 2026 07:34
@coderabbitai

coderabbitai Bot commented Aug 5, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 5e4ac16a-8bc2-4a64-8e11-bfc5ce795f3c

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@Tomkess
Tomkess merged commit f3486db into feat/chat-client-reasoning-steps Aug 5, 2026
1 check passed
@Tomkess
Tomkess deleted the fix/agentic-reasoning-steps branch August 5, 2026 07:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant