Repository navigation
Make Smart a native analyst over its specialists' data - #181
Merged
Merged
Conversation
Smart answered "what data do you have?" with a list of specialists, sometimes routed a request for values to the catalog and asked the user to confirm delegation, lost the subject of a follow-up, and ended a two-specialist answer before writing it. - describe_agent returns one combined Semantic View catalog for Smart, read per specialist under the caller's access and the specialist's release pin. - The planner routes a Smart request for values to discovery, the root keeps a bounded history of its thread, and its token limit covers several specialists. - An uncovered request ends as a stated gap with no number; a scope statement or clarification completes a turn. - A plain value whose claim fails is rechecked without the claim, a spoken date is a date, and a computed average may be stated rounded. - Collaboration targets resolve by an unambiguous task or agent name. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A correct answer could still show "[unverified number]": a value repeated in a table and in one unbound sentence was removed everywhere, and a ratio across two specialists had no verified source. - Each number is judged at its position. A draft with an unsupported number gets one rewrite when it still fits the context budget; a number that remains takes its clause with it and leaves no marker. A wrong table cell or headline rebuilds the answer from cells, and a wrong comparison is replaced in place. - The Smart root holds compute_metrics and data_to_chart over the results its specialists return, listed to the model with the evidence id each has in the turn. compute_metrics gains combine (two results on their shared label) and ratio (one column per unit of another). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Smart root could combine results and chart them, but its final message was text only, so the combined table and the chart never reached the user. - The root's tables and charts are saved as steps of its final message and sent in its completion event; the stream replays them as table and chart frames. - A comparison is judged against every result it could mean, so "the fewest" over headcount is no longer corrected from the expense table. - The root passes each specialist only the period that applies to its measure. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Smart answered with identifiers, backticks, "Semantic View" and agent names, which mean nothing to someone who knows the data but not the product. - The catalog carries a plain label for every View, metric and dimension. - The root prompt asks for business names and sources by subject; the Semantic View and serving agent are named only when the user asks for the source. - The root's final text writes identifiers as words outside fenced code. - An uncovered request is declined plainly, without owners or catalogs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A forecast asked of Smart never returned: the specialist ran non-iteratively and could not go from its query to ML, its result table was not evidence, and its turn sometimes ended as an unverified clarification or as tool-call markup. - Discovery lists each owner's enabled abilities; the root delegates forecasts, anomaly checks and change diagnosis to the owning specialist, and says plainly when no owner has the ability. - A Smart specialist runs iteratively with a 40,000-token limit, like a direct turn, and keeps the read-only analysis tools its owner enabled. - An ML result table is evidence. A turn holding results is verified even after a later clarification, and a tool call written as text is never an answer. - A forecast request plans as machine learning even when it asks for history. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- A Smart answer shows its last table and its last chart, headed in plain words, instead of every intermediate result with raw column names. - A follow-up sent the moment an answer lands waits briefly for the previous run to settle instead of being refused as a concurrent run. - Studio follows the reply after a message is sent, wherever the reader had scrolled, and asks for a role when none is active instead of reporting that conversations and specialists failed to load. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… data - A well-formed call to an offered tool that the provider wrote as text markup now runs as that call, through the same gates, outside final composition. Before, the turn spent its one repair and ended without an answer, which left Smart without a specialist's result. - Smart's final text writes "Semantic View" as "data". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… Smart An iterative turn only learned a result's row count, so it wrote blind and its answer was rebuilt from cells or, under Smart, the specialist reported that the value was missing. Smart also spent a model call each on discovery and waiting. - A tool result with a table carries a bounded preview of its rows when it fits the context budget. - Discovery runs before the Smart root's first model call, and the root waits on its own once every owning specialist has started. - ml_execute runs on the compiled query of a semantic_query result named by evidence_id; a query the engine rejects is repairable. - The root and its specialists keep to what was asked: no unrequested totals, ranks or charts, and no repeated highest/lowest line. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A layperson asks Smart for "this every Monday" or "tell me when it passes X". Smart could not do either: a schedule needs consent, a Smart turn is unattended, and the Smart worker only ran under a login session. - propose_automation validates a schedule and writes nothing. The draft travels with the answer; Studio shows a card whose Schedule button creates the automation through the existing endpoint, which now accepts Smart for the session's own user. Capabilities lists Smart's scheduled reports. - A fire queues a Smart root marked `automation:<id>` and waits for it. The Smart worker resolves the marker to the automation at every checkpoint and runs each participant as the owner through the worker execution account, under the automation's role. A changed or disabled automation, a missing execution account, or a refused impersonation fails the run closed. Security review requested: this adds an unattended identity path to the Smart worker. It reuses the scheduled-agent contract and stores no credential. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A heartbeat could undo the completion of its own run: the engine rewrites the whole row an UPDATE read, so the turn went back to running, was recovered and answered twice, and the second completion failed the root. A process now writes a run row in turn, and the completion a parent already received stands. A Smart participant near its token limit answers from the evidence it holds. diagnose_change takes the result's own period and measure columns and can be repaired; the semantic planner groups two named periods instead of comparing their whole range with the one before. A change a result lists as a cell is accepted, a follow-up may repeat the year the user named earlier, and a Smart answer keeps its newest result so the next turn can chart it. Studio names the owner's agents it leaves out until access is verified. The live acceptance lab for Smart moves into tests/benchmark/smart_native. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The verifier replaced correct answers with a stiff rendering when the model noted a figure with its currency or sign, noted a listed contribution as a change, or stated a computed percentage rounded. Those are now matched to the result cells that hold them; a percentage no result holds is still refused. A two-period breakdown is diagnosed without a model call, a schedule proposal stays available to Smart whatever the planner chose, and Studio keeps reading the run tree briefly after the root settles so a status card does not stay on "Queued". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A monthly series stamped on the first of the month and forecast with frequency "M" labelled October's value 30 September, one step early. A requested frequency is now anchored where the series itself is. Smart no longer says it will request an analysis from someone, and keeps a caveat to one sentence. The lab's forecast case names the months it requires and gives the judge the monthly history. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A specialist read the user's whole question as its task and asked about the part another specialist owned; the root then followed up with one of them and answered from that one's results alone. The wider question now reaches a specialist as background, and before the answer is verified the results of every finished participant are taken as evidence. The lab's judge reads each answer three times and the middle score counts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
"May against April" was sometimes planned as one range of both months compared with the 61 days before it, so the answer explained a change that did not happen. Such a plan is now read as the two months. A Smart chart of a label and two measures draws both when the designed spec drew only one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Smart answers like one analyst that owns the data its specialists serve. A person who knows the business, and nothing about Nova, can ask for figures, comparisons across domains, charts, derived numbers, a forecast, the reason a measure moved, or a recurring report, and gets one answer in business language.
diagnose_changetakes the result's own period and measure columns and runs automatically on a two-period breakdown. Two named months are planned as two groups.UPDATEs on one row lose a status change on this engine (measured 3 of 40).Review notes
harness_worker._identity,_automation_user) reuseDelegateExecutor.owner_connectionand fail closed when the automation is disabled, its role changed, or impersonation is not configured. Please review this as a security change._update_runserializes statements within one process only. A cancellation from the API process still relies on the status and generation checks.compute_metrics,data_to_chart,diagnose_change,ml_executeanddescribe_agentchanged, so agents pinned to a release that uses them will report drift until republished.SMART_MAX_TOTAL_TOKENS).Checks
Local, on the last commit: backend unit suite with coverage (7,255 passed),
tests/eval(326 passed), eval scorecard 48/48, blocking Ruff on changed files, answer-contract corpus 65/65 with no false accept; frontendpnpm lint,pnpm build,pnpm test:coverage(1,197 passed).Not run locally: the seeded real-engine integration suite, whose ports are in use by a running dev stack. It is expected to run here in CI.
Live acceptance
backend/tests/benchmark/smart_native/holds the lab: a Finance and an HR agent, gold values from SQL, 23 cases, and a pass rule of gold figures, business wording, required artifacts, no stream error, and an LLM judge at 4 of 5 or better on four dimensions.The target is three consecutive clean passes on the same code. It has not been reached. Full passes on successive commits scored 18, 20, 23, 20, 21, 21, 23, 23 and 21 of 23; each failure was traced to a cause and fixed with a test. The last code change (two named months, two-measure chart) answers the two failures of the last valid pass, and its confirming passes could not run: the model provider returned HTTP 402.
Median time per answer in the last clean passes: about 50 seconds; slowest about 90.
🤖 Generated with Claude Code