cu analyze: default analyzer by file type, markdown anchors, --format both, cu resolve, result deletion

[Arlind Nocaj (ArlindNocaj)]

Makes cu-cli usable by a coding agent with nothing but CU_ENDPOINT (+ az login). Built fresh on the latest main; additive — no existing flag, command or doc text removed. One commit, 16 files (+3 re-recorded cassettes), no new dependencies.

Key improvements

• Usable without CU expertise. cu analyze doc.pdf just works: the analyzer is chosen by file type (prebuilt-documentSearch for documents), no profile setup, no model choice.
• Anchors in the markdown. Every section, table, figure (and with --level paragraph every paragraph) carries <!--s3-->-style ids. Without them an agent has no way to point back to where in the document a statement came from.
• Offline cu resolve. Turns an anchor into page, bounding box, text and surrounding blocks from the saved JSON — no service call. An agent reads the markdown, cites ids, and resolves them on demand only when it needs to look at a figure, a table region or the page itself.
• Two files from one call. --format both --output-file doc writes doc.md and doc.json and lists the paths on stdout. The agent's context is not polluted with the whole document: it can grep/head the markdown for the parts it needs and keep the JSON on disk for cu resolve.

export CU_ENDPOINT=https://<resource>.services.ai.azure.com/
cu analyze report.pdf                                                  # works: markdown with anchors → stdout
cu analyze report.pdf --level paragraph --format both --output-file report   # report.md + report.json
cu resolve report.json p30 --around 1                                  # page, bbox, text, neighbours — offline

Before	After
cu analyze doc.pdf errors: no default_analyzer configured	uses CU_DEFAULT_ANALYZER › profile › prebuilt-documentSearch (documents) / *Search by file type
--llm-input xor --json — two billed calls for two files	--format md|json|both (--json = shorthand, --llm-input hidden alias); --format both --output-file NAME → NAME.md + NAME.json, one call; batch gets both sidecars. Written paths go to stdout, one per line (gog-style: data on stdout, diagnostics on stderr)
Markdown has no way back to the page	<!--s3--> / <!--t0--> / <!--f0--> anchors (+ <!--p30--> with --level paragraph) = service-result array indices; stripping them yields the previous markdown; +0.2 % tokens coarse, +6 % paragraph
—	new cu resolve RESULT.json ID… [--around N] — local, no endpoint
result stays on the service, operation id echoed in JSON	deleted after retrieval (--keep-result opts out); id omitted

Docs: Quickstart section in the root README ("which command when"), two short additive paragraphs in cu-cli/README.md, CHANGELOG Unreleased.

Deliberately left out (follow-ups): separate --md-rich/--md/--map output flags, cu resolve --page/--images, grounded field front matter, --schema ephemeral extraction, agent skill.

Verification: cu-cli/scripts/ci.sh green (headers, ruff, mypy, unit, integration playback) on Python 3.12. test_validate_release.py::test_cli_requires_stable_core_dependency fails on main already (not in ci.sh), unchanged here.

[Arlind Nocaj (ArlindNocaj)]

@microsoft-github-policy-service agree company="Microsoft"

[Yung-Shin Lin (yungshinlintw)]

Given that the result will still deleted automatically by CU service after 24 hours, --keep-result can be misleading. Another possible name is --delay-delete

[Yung-Shin Lin (yungshinlintw)]

We should move the content for quickstart into cu-cli/README.md directly as pypi will get the content from cu-cli/README.md

[Yung-Shin Lin (yungshinlintw)]

Given that we want this to be a quickstart, we can add the following step to allow user to optionally use a multi-page pdf that they can try right away and show the benefit prebuilt-documentSearch

# Uncomment the following to use a sample file
# curl -L https://raw.githubusercontent.com/Azure-Samples/azure-ai-content-understanding-assets/main/document/financial_table_and_chart.pdf -o doc.pdf

cu analyze doc.pdf

[Yung-Shin Lin (yungshinlintw)]

We should not quote the 6.5% token increase as it's not a universal ratio for all files

[Yung-Shin Lin (yungshinlintw)]

We should explain what 'p3' means as it's not clear from the context for readers

[Yung-Shin Lin (yungshinlintw)]

In the "You get" column, please add the following link for users to check out supported prebuilt analyzers:

https://learn.microsoft.com/azure/ai-services/content-understanding/concepts/prebuilt-analyzers

[Yung-Shin Lin (yungshinlintw)]

Please direct user to this section in README for more details in creating custom analyzer:
https://github.com/Azure/content-understanding-toolkit/blob/main/cu-cli/README.md#create-a-custom-analyzer

[Yung-Shin Lin (yungshinlintw)]

We should not mention prebuilt-documentFieldSchema here as it's specifically used as an optional helper when creating custom analyzer, and it's advanced usage, so we do not need to mention it in a quickstart

[Yung-Shin Lin (yungshinlintw)]

For the description for per file, paths on stdout, it is a little too short to be clear for what they mean.

[Copilot]

🟡 Changes recommended

Unresolved critical and moderate findings affect Markdown anchors, format validation, output collisions, modality selection, and result cleanup.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

This PR makes cu-cli more agent-friendly with automatic analyzer selection, anchored Markdown, offline resolution, dual-format output, and result cleanup.

Changes:

• Adds modality-based defaults and CU_DEFAULT_ANALYZER.
• Adds rich Markdown anchors and cu resolve.
• Adds --format both, documentation, tests, recordings, and cleanup behavior.

File summaries

File	Review summary
README.md	Adds Quickstart guidance.
cu-cli/README.md	Documents the new CLI workflow and options.
cu-cli/packages/standalone/tests/unit/test_cli.py	Tests CLI defaults and output behavior.
cu-cli/packages/standalone/tests/unit/commands/test_analyze_contract.py	Updates analyze contract tests.
cu-cli/packages/standalone/tests/integration/recordings/analyze_single.yaml	Refreshes single-file playback.
cu-cli/packages/standalone/tests/integration/recordings/analyze_batch.yaml	Refreshes batch playback.
cu-cli/packages/standalone/src/cu_cli/output.py	Critical (3 votes): SDK result models are not converted before Markdown rendering, so normal Markdown output lacks anchors.
cu-cli/packages/standalone/src/cu_cli/modality.py	Moderate (2 votes): .webm is classified as audio despite documented video precedence.
cu-cli/packages/standalone/src/cu_cli/commands/resolve.py	Adds offline anchor resolution.
cu-cli/packages/standalone/src/cu_cli/commands/analyze.py	Moderate findings: conflicting --llm-input and format flags are not rejected (1 vote); dual-output collision checks cover only JSON (3 votes); conflicting --json and --format md silently selects Markdown (2 votes).
cu-cli/packages/standalone/src/cu_cli/cli.py	Registers resolve and updates help.
cu-cli/packages/core/tests/test_rich_markdown.py	Tests anchor mapping and resolution.
cu-cli/packages/core/tests/test_environment.py	Tests environment registration.
cu-cli/packages/core/tests/test_analysis.py	Tests result deletion.
cu-cli/packages/core/src/cu_cli_core/rich_markdown.py	Moderate findings (1 vote each): paragraph anchors are omitted inside tables or figures; coarse sections may lack page mappings; malformed coordinates can raise ValueError.
cu-cli/packages/core/src/cu_cli_core/environment.py	Registers analyzer environment configuration.
cu-cli/packages/core/src/cu_cli_core/command_spec.py	Defines the new command options.
cu-cli/packages/core/src/cu_cli_core/analysis.py	Moderate (3 votes): cleanup failures are ignored, so retained service results can be reported as successful deletion.
cu-cli/CHANGELOG.md	Records the unreleased changes.

Review details

Suppressed comments (4)

cu-cli/packages/core/src/cu_cli_core/rich_markdown.py:167

• --level paragraph promises an anchor for every paragraph, but this condition drops every paragraph whose span starts inside a table or figure. Those paragraphs remain in the service paragraphs array and are not resolvable by their p<i> ids, so the advertised paragraph-level citation coverage is incomplete. Either emit paragraph anchors for these entries or change the public contract and examples to document the exception.

        for index, paragraph in enumerate(content.get("paragraphs") or []):
            if "span" not in paragraph:
                continue
            start = int(paragraph["span"]["offset"])
            if any(lo <= start < hi for lo, hi in containers):

cu-cli/packages/core/src/cu_cli_core/rich_markdown.py:192

• In coarse mode, paragraph ids are intentionally absent, so a section containing only paragraphs has no child with a bbox. The page calculation therefore leaves pages unset for that section even though the section is anchored and cu resolve is supposed to map anchors back to document location. Derive section pages from pages[].spans (or retain enough paragraph/page metadata) instead of relying only on ids present in the coarse map.

    for entry in ids.values():
        if entry["kind"] == "section":
            pages = sorted({int(ids[c]["bbox"]["page"]) for c in _child_ids(entry, ids) if ids[c].get("bbox")})
            if pages:
                entry["pages"] = pages

cu-cli/packages/core/src/cu_cli_core/rich_markdown.py:57

• A source that matches _SOURCE_RE but contains non-numeric coordinates, such as D(1,a,b,c,d), raises ValueError from this conversion instead of returning None. That aborts rich-markdown rendering for a malformed optional source; catch ValueError so malformed sources are handled like the missing/malformed cases this parser is intended to tolerate.

    values = [float(v) for v in match.group(2).split(",") if v.strip()]

cu-cli/packages/standalone/src/cu_cli/commands/analyze.py:337

• --llm-input is documented as an alias for --format md, but this validation does not reject --llm-input --format both. That combination reaches the both branch, writes JSON and Markdown, and silently ignores the alias instead of enforcing the selected view. Include "both" in the conflict check (and use a format-neutral error message).

    if llm_input and (json_output or output_format == "json"):
        raise CuCliError("--llm-input and --json cannot be combined.", exit_code=VALIDATION_FAILURE)

• Files reviewed: 19/20 changed files
• Comments generated: 5
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Yung-Shin Lin (yungshinlintw)]

We cannot assume that all users want to save the JSON, and we have many customers mainly use markdown. Currently, we don't have direct customers asking anchor from markdown back to JSON for grounding purpose as far as I know (also checked with PMs).

However, the anchors in the md seems to be a useful addition.

While I like that anchor format, I think:

• It should not be default
• It should be merged as an option into CU SDK to_llm_input (Chien Yuan Chang (@chienyuanchang) can help with this for all SDK languages, starting with Python first)
• We should control the generation of anchor with --anchor-level none (default) | coarse | paragraph