Repository navigation
fix(webfetch): enforce native retrieval and shared scoring contracts - #42
lanyue-llk wants to merge 151 commits into
Conversation
lanyue-llk
changed the title
lanyue-llk
changed the title
1a7f2e3 to
da3f01b
Compare
lanyue-llk
changed the title
|
Claude encountered an error after 3s —— View job I'll analyze this and get back to you. |
|
Claude encountered an error after 3s —— View job I'll analyze this and get back to you. |
|
Claude encountered an error after 3s —— View job I'll analyze this and get back to you. |
|
Claude encountered an error after 3s —— View job I'll analyze this and get back to you. |
|
Claude encountered an error after 3s —— View job I'll analyze this and get back to you. |
|
Claude encountered an error after 3s —— View job I'll analyze this and get back to you. |
|
Claude encountered an error after 3s —— View job I'll analyze this and get back to you. |
|
Claude encountered an error after 3s —— View job I'll analyze this and get back to you. |
lanyue-llk
changed the title
|
Claude encountered an error after 3s —— View job I'll analyze this and get back to you. |
|
Claude encountered an error after 3s —— View job I'll analyze this and get back to you. |
|
Claude encountered an error after 3s —— View job I'll analyze this and get back to you. |
|
Claude encountered an error after 3s —— View job I'll analyze this and get back to you. |
lanyue-llk
left a comment
lanyue-llk
left a comment
There was a problem hiding this comment.
审查结论：建议修正下列问题后再合入。
本 PR 将 Web Fetch 扩展为五浏览器、七类任务、每浏览器 430 题，并统一原生接口、正文评分和发布校验。共享任务投影与统一路由选择器是合适的抽象；保留动态任务分类也合理，但调用校验没有同步迁移。资源投影还把单个 URL 当成 srcset 列表解析，导致合法资源地址损坏。
确认两处功能回退，均做了基线对照：目标分支通过，PR 失败；在隔离副本中验证最小修复后，相关测试 63 项通过。现有相关测试分两组共 478 项通过、13 项跳过；启用四种真实本地浏览器后，资源、取消和文件测试 24 项通过、1 项失败，失败原因见第三条。进程清理失败后的隔离行为符合现有安全合同，不列为缺陷。
这次审查覆盖代码、调用合同及相关测试，不改变报告中独立逐结果复核 0/2,150 的范围。
| raise ValueError('Native fetch cannot discard declared retrieval hints: ' + case['id']) | ||
| if case.get('track') == 'dynamic': | ||
| case['track'] = 'web' | ||
| case['retrieval_hints'] = {'version': 3, 'mode': 'native'} |
There was a problem hiding this comment.
[P1] 同步固定原生调用校验，避免动态任务整批无法执行
native_fetch_dataset() 现在给动态任务写入非空的 retrieval_hints={"version":3,"mode":"native"}。但 Provider.check_native_call() 对 uniform_native_parameters=True 的方法仍拒绝所有非空 hints，run_evaluation() 又在任何调用前执行该校验。因此认证入口的 --retrieval-mode native 遇到动态任务时，会直接报 fixed native calls prohibit per-case retrieval hints，Moli、Firecrawl、Tavily 等固定原生方法都受影响。相同场景在 main 可以通过。
最小修复是在共享校验中通过现有解析器识别并允许严格的 v3 native 标记，继续拒绝选择器、延迟和其他逐题辅助参数。补一个从原生数据集转换一直跑到 runner 校验的动态任务测试，避免只测 plan_for_case() 而漏掉调用边界。
| const urls = new Set(); | ||
| const add = (value, base = location.href) => { | ||
| if (!value) return; | ||
| for (const part of String(value).split(',')) { |
There was a problem hiding this comment.
[P2] 只对 srcset 解析候选列表，保留单个资源 URL 中的逗号
add() 对 src、href、poster 等单个 URL 也执行 split(',')。真实 Chromium 页面中，<img src="/image/a,b.png"> 被返回成 /image/a 和 /b.png；/image/thumb.png?crop=10,20 被返回成 ?crop=10 和 /20。这些都是合法 URL，当前投影丢掉真实资源并制造不存在的地址。main 的 Chromium 路径能保留两条完整地址，PR 的共享投影失败；现在其他使用该投影的浏览器也受影响。
把单 URL 属性和 srcset 候选解析分开：前者按完整属性值解析，后者按候选与描述符解析。补路径含逗号、查询含逗号和多候选 srcset 的功能测试。
| self._run_shared(ChromiumFileDeliveryTests().test_inline_redirect_attachment_and_http_error_use_native_navigation) | ||
|
|
||
| def test_pdf_and_inline_attachment_mislabeled_docx_use_shared_extraction(self): | ||
| self._run_shared(ChromiumDocumentDeliveryTests().test_native_document_bytes_use_the_shared_text_extractor) |
There was a problem hiding this comment.
[P2] 将 Moli 文档实测迁移到当前正文合同
这项新增测试复用 output_mode='document' 的下载字节加外部解析测试，但 _run_shared() 实际通过 MoliWebFetchProvider.fetch() 调用当前路由。该路由对直接 PDF 选择原生 Markdown，已不再走共享文档解析器。使用本次评测的真实 Moli 1.1.15 运行，测试在 /fixture.pdf 失败，返回 tool_error：raw download output only supports automatic output or --dump json。未设置 MOLI_BINARY 时此测试被跳过，所以普通 CI 的通过不能证明文档实测通过。
应按当前交付合同拆开原始文件和文档正文的场景：原文件继续验证字节，正文验证原生文本及浏览器查看器文本；原生接口不提供正文的直接文档应保留失败，不恢复外部解析来满足旧断言。删除已退役解析路径专用的断言，并实际启用二进制运行这些测试。
|
Claude encountered an error after 3s —— View job I'll analyze this and get back to you. |
7607b18 to
2166c94
Compare
lanyue-llk
changed the title
cd0b669 to
0598c6a
Compare
lanyue-llk
changed the title
Rebase the session changes onto main 03c9a5e and extend its shared certification, execution, evidence, assembly and reporting owners. Keep command entry points thin, package the participant registry, remove retired runners and historical qualification recovery, and enforce task-selected native outputs without external answer repair. Preserve measured evidence and require complete declared correction scopes. Add functional controls for native documents, media URL parsing, interrupted evidence recovery, fixed versus explicitly live references, scoring regressions, report bindings and release presentation.
Separate resource admission from execution of hash-bound fixed benchmark inputs. Keep source qualification mandatory for new or changed resource tasks, explicit live references and invalid sources. Exercise the public certified entry point with frozen body and resource answers whose historical raw files are absent, preserve the original task bytes, and retain strict admission controls.
lanyue-llk
changed the title
lanyue-llk
changed the title
lanyue-llk
changed the title
1 participant
Native Markdown was rejected when records shared lines, identifiers included annotations, linked cells omitted pipe delimiters, or an escaped exclamation mark preceded a link. The shared parser now preserves visible content and checks fields within their own record and section. Paired acceptance controls cover complete outputs, missing values, and values borrowed from neighboring records.
Retrieval follows one declared policy per task family. The comparison contains Moli 1.1.15, the aggregate fix in lexmount/moli#1501, Lightpanda, Obscura, Chromium, and Kitesurf. Chromium uses the pinned HTML-to-Markdown conversion; the other methods retain native Markdown. Direct media tasks retain original file bytes. Final URLs and language remain diagnostic. A declared transient retry selects its final physical response and preserves every call.
Reference corrections use independent source evidence and apply to every method. Microsoft 365 has one fixed body reference and rejects the retired introduction. Alibaba Cloud uses a fixed international URL and requires replacement calls. Unobserved locale variants, interaction-only controls, a random server tagline, and a phrase formed by joining unrelated navigation and body text are removed from acceptance requirements. The population stays at 430 tasks.
The profile requires review of every successful and failed result. Assembly preserves that scope, and the review boundary rejects failure-only coverage, missing attempts, and unresolved judgments. Existing content reviews can be reused when the raw answer and acceptance contract are unchanged; a new call still needs its identity checked.
The interpretation page's method-by-family interface matrix reads actual retrieval, extraction, and waiting metadata. Network disclosures count recorded DNS modes and distinguish the remote Kitesurf service. Release checks reject altered interface descriptions, network counts, or missing disclosures. Public DoH is not presented as a guarantee of identical destination IPs.
The fixed Moli profile binds commit a737f2a and its immutable binary across all 430 calls, with no inherited results from an earlier build. Calls and outputs stay outside Git. This PR does not publish new report totals.
Validation
git diff --checkpassed.Moli's adapter also preserves HTTP status from an unambiguous same-call native document event. An empty answer accompanied by an explicit native readiness timeout is classified as a timeout. This restores the existing transient retry policy without changing Markdown, retrying content failures, or attributing a subresource error to the document. Adapter scenarios: 79 passed; 59 subtests passed, with failing controls recorded before the fix.