feat(gooddata-eval): capture reasoning_steps through the agentic-CLI path - #1712
Merged
Tomkess merged 1 commit intoAug 5, 2026
Merged
Conversation
…path 6001d2f wired ChatResult.reasoning_steps through runner.py's generic single-turn path only. The agentic-CLI path (cli/agentic_runner.py -> evaluate_agentic_*) builds its own ItemReport and never touched it, so agentic_alert_skill/agentic_metric_skill/agentic_conversation items could never produce a reasoning trace, no matter what the platform emitted. Accumulates reasoning_steps across every send_message call in each of the three evaluators' run loops, attaches it to the run/turn result, and surfaces it from evaluate_agentic_* either as the return value (pass) or as an attribute on the raised exception (fail) -- mirroring the existing conversation_id-on-exception idiom in ChatClient.ask(). run_agentic_items picks it up from either path onto ItemReport.reasoning_steps, which json_report.py already serializes unconditionally. general_question/guardrail/search_tool/visualization are left untouched -- their evaluate_agentic_* functions still return None, unchanged.
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stacked on #1708. That PR wires
ChatResult.reasoning_stepsthroughrunner.py's generic single-turn path (ChatBackend.ask()), but never through the agentic-CLI path (cli/agentic_runner.py→evaluate_agentic_*). As a result,agentic_alert_skill,agentic_metric_skill, andagentic_conversationitems could never carry a reasoning trace —run_agentic_itemsbuilds its own bareItemReportandreasoning_stepswas simply never assigned, regardless of what the platform emitted.alert_skill.py/metric_skill.py/conversation.py: accumulatechat_result.reasoning_stepsacross everysend_messagecall in each evaluator's run loop; attach to the run/turn result.evaluate_agentic_alert_skill/evaluate_agentic_metric_skill/evaluate_agentic_conversation: return the reasoning steps on pass; attach them to the raised exception on fail (mirrors the existingconversation_id-on-exception idiom inChatClient.ask()).cli/agentic_runner.py:_dispatch_agenticnow returns whateverevaluate_agentic_*returns;run_agentic_itemswrites it ontoItemReport.reasoning_stepsfrom either the return value or the caught exception.general_question/guardrail/search_tool/visualizationevaluators are untouched — still returnNone, no behavior change.Confirmed live (not just unit tests): ran a real
agentic_alert_skillquestion end-to-end against a live workspace and got a populated reasoning trace where before there was none.Test plan
packages/gooddata-evaltest suite: 268/268 pass (13 new tests added)ruff checkcleanagentic_alert_skill