fix(qa): 修复 Markdown 标题式问答前缀解析缺陷 - #976
Merged
Merged
Conversation
QA 分块解析器原本不识别 # Q: / ## 问题: 这类带前缀标题,导致此类文档无法被正确切分问答对;同时缺乏超长 chunk 切分,单条问答可能超过 embedding 上下文上限。
Owner
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 378792dcb2
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Contributor
Author
|
怎么 Re-run failed jobs |
Refactor encode and aencode methods to handle requests with retries and error logging.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
QA 分块解析器原本不识别 # Q: / ## 问题: 这类带前缀标题,导致此类文档无法被正确切分问答对;同时缺乏超长 chunk 切分,单条问答可能超过 embedding 上下文上限。
变更说明
补全 QA 分块解析器的方法说明并登记 v0.7.2.beta1 的两条 changelog,使 docstring 描述与代码行为一致、让工作区已有的 QA 重写与 embed.py 调试日志改动进入版本材料。
processchunk_markdowndocstring 与实际分支行为一致(修订"随时叠加"、"txt 兜底"等不准描述,明确# Q:/# 问题:标题作问题、纯# 标题作分节符);把工作区已有的 QA 重写 + 超长 chunk 切分 + embed.py 超长输入调试日志补录到 changelog v0.7.2.beta1。工程主张与 Owner
验证情况
docstring 描述与 chunk_markdown 分支行为一致
.txt分支不调表格提取矛盾;原第 4 点"txt 中...无命中时兜底"与.txt分支把分隔符作为最高优先级矛盾。修订后两处均与代码一致。changelog 两条登记与工作区 diff 一致
git diff HEAD -- backend/package/yuxi/knowledge/chunking/ragflow_like/parsers/qa.py backend/package/yuxi/models/embed.py。qa.py 含 heading_re 剥#、纯#标题作分节符、.md/.markdown/.mdx/.docx合并分支、.txt改分隔符优先 + 前缀兜底、新增_split_long_qa_chunks;embed.py 新增_log_long_inputs在encode/aencode前以logger.warning打印超 4000 字符输入。QA 解析器代码逻辑零改动
git diff HEAD -- backend/package/yuxi/knowledge/chunking/ragflow_like/parsers/qa.py仅 4 增 6 减,全部分布在chunk_markdowndocstring 与.md系列、else两处分支注释行;_md_question_level、_extract_pairs_from_markdown_headings、_extract_pairs_by_prefix、_split_long_qa_chunks函数体未动。文档构建与工程契约不破坏
verify_engineering_contracts.py。cd docs && pnpm run build(build complete in 13.31s,无 dead link 报错);python3.12 scripts/verify_engineering_contracts.py(通过:46 decisions / 87 docs / 253 web sources);git diff --check(无输出)。