Skip to content

fix(validate): read the when cue of each language in description-no-when - #454

Open
napetrov wants to merge 8 commits into
intel:mainfrom
napetrov:fix/description-when-multilingual
Open

napetrov wants to merge 8 commits into
intel:mainfrom
napetrov:fix/description-when-multilingual

Conversation

@napetrov

@napetrov napetrov commented Nov 7, 2026

Copy link
Copy Markdown
Contributor

Part of #413 (A10).

Summary

_flag_description
  letters outside covered scripts > covered letters -> None (not checked)
  WHEN_RE per language: en, es, fr, de, pt, ru, zh, ja, ko
    Latin cues: word boundaries
    CJK cues: substrings, no word boundaries
  no cue -> description-no-when
  • Bare 使用 and 用于 are not cues (they mean "use").
  • Negative tests cover the common false passes: 当天, 相当, 同时, 实际, 国際, 交際, 一時に, 平時は, 用户说明.

Evidence

  • Before: corpus eval (description-no-when flag rate) CJK 86.6%, Latin 40.0%, total 718,264 flagged.
    After: CJK 48.9%, Latin 37.5%, total 629,620 flagged.
  • Precision sample of rows the check now clears: 39/40 CJK and 30/30 Latin are genuine when-less descriptions.
  • tests/test_description_when_cues.py: 95 passed; ruff, mypy, and make ci-local static and typecheck pass.

Merge Danger

Door: two-way (the check is a warning; reverting restores the English-only check)

Blast Radius: catalog validation warnings only; no change to routing or grading. Some CJK skills that used to warn now pass, and some Latin-script skills with no when cue still warn.

Declined, recorded in the PR: Polish, Dutch, Turkish and Vietnamese cue lists; descriptions in uncovered scripts stay unchecked (documented in docs/external-content.md); zh, ko, es, pt and ru recall gaps; 時には and 平時に treated as scope rather than a trigger.

🤖 Generated with Claude Code

The check only knew English cues, so a skill whose description had a native
when clause in Chinese, Japanese, Korean or another covered language was
flagged as having none (94.7% of CJK skills with a native cue). Add cue lists
for en, es, fr, de, pt, ru, zh, ja and ko; CJK cues match as substrings.
A description whose letters are mostly in a script no list covers is not
checked. Bare 使用 (use) and 用于 (for) are not cues.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
@napetrov
napetrov requested a review from a team as a code owner November 7, 2026 07:46
Copilot AI balanced review requested due to automatic review settings November 7, 2026 07:46

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Multilingual when-clause detection incorrectly accepts occasional-action phrases, capability descriptions, and compound words, suppressing valid description-no-when warnings.

4 open issues
What changed in this PR

4 new findings.

This PR broadens catalog validation’s description-no-when check beyond English, reducing warnings for descriptions with native-language invocation cues.

Changes:

  • Adds multilingual cue patterns and skips descriptions predominantly written in uncovered scripts.
  • Adds positive, negative, and end-to-end regression tests.
  • Documents language coverage, limitations, and changed warning behavior.
File Description
dokimi/​tests/​test_description_when_cues.py Tests native cues, misleading expressions, script exclusions, and catalog warnings.
dokimi/​dokimi/​external/​validate.py Adds multilingual cue detection and script-coverage handling.
dokimi/​docs/​external-content.md Explains supported languages and uncovered-script behavior.
dokimi/​changelog.d/​20261106_120000_napetrov_description-when-multilingual.md Records the validation change and recommends rerunning catalog checks.

Review effort: Balanced

Comment thread dokimi/dokimi/external/validate.py Resolved
Comment thread dokimi/dokimi/external/validate.py Resolved
Comment thread dokimi/dokimi/external/validate.py Resolved
Comment thread dokimi/dokimi/external/validate.py Resolved
@codecov-commenter

codecov-commenter commented Nov 7, 2026 •

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.14%. Comparing base (15d840d) to head (60d7c76).
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #454   +/-   ##
=======================================
  Coverage   95.14%   95.14%           
=======================================
  Files         373      373           
  Lines       49271    49281   +10     
  Branches     8467     8468    +1     
=======================================
+ Hits        46879    46889   +10     
  Misses       1515     1515           
  Partials      877      877           
Files with missing lines Coverage Δ
dokimi/dokimi/external/validate.py 96.33% <100.00%> (+0.13%) ⬆️
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Thread: PRRT_kwDOVBg7g860zpIj
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Thread: PRRT_kwDOVBg7g860zpIx
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Thread: PRRT_kwDOVBg7g860zpI3
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
Thread: PRRT_kwDOVBg7g860zpI_
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
The colon example ended in 생성될 때, which the older 때 cue already
matched, so the test passed with the new cue removed. Use inputs the old
cues do not match; removing the new cue now fails both.

Thread: PRRT_kwDOVBg7g860zpI_
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
A label with one of these words between 트리거 and the colon was a cue
before the bare-noun fix narrowed it; it now is again. 트리거 조건: 새 태그
is a when clause, and the noun alone is still not.

Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
@napetrov

napetrov commented Nov 7, 2026

Copy link
Copy Markdown
Contributor Author

Its four open issues are the threads above, each fixed in the commit named there.

This is the Codecov report. It makes no claim, so nothing changes.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants