[patch] Rank standardized property names by fuzzy similarity - #120
Merged
Merged
Conversation
NameStandardizer.FindStandardPropertyMatch admitted a standard property by substring containment and returned the first one that qualified, so the winner was decided by the order StandardOrder.PropertyNames happens to list them in rather than by how well it matched. "subtitles_track" standardized to "title" instead of "subtitle", and "review_status_flag" to "status" instead of "review_status" -- in both cases the shorter, less specific name won only because it is listed earlier. Candidates are now ranked by ktsu.FuzzySearch, which was already a PackageReference of this project but had no call site anywhere in the repo. Its score rewards consecutive characters, separator boundaries and camelCase boundaries, and penalizes characters the pattern never matched -- the ranking this matcher was approximating by hand. The containment gate is deliberately unchanged. Fuzzy.Contains matches a subsequence rather than a substring, so using it to admit candidates as well as rank them would map keys that are preserved today; since ranking already discards those weaker candidates, widening the gate buys nothing and risks renaming frontmatter a caller meant to keep. Ties still fall back to the standard order. PropertyMerger's word-overlap scorer is the other call site named in the issue and is left alone for now: its candidates only have to share a word, so pairs like "review_notes"/"notes_reviewed" are related without either being a subsequence of the other, and a whole-name score cannot rank them at all. Refs #113 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S9FaM3MJE7n987yxvtxUZD
The Standardize helper asserted the extracted frontmatter was not null and then returned it, which leaves the return flowing a nullable value as far as static analysis is concerned. Throwing AssertFailedException on the null branch makes the contract explicit and carries a message naming the key that produced no frontmatter. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01S9FaM3MJE7n987yxvtxUZD
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



Refs #113.
The defect
NameStandardizer.FindStandardPropertyMatchadmits a standard property by substring containment and returns the first one that qualifies. Containment says a candidate is plausible; it does not say how good it is. So the winner is decided by the orderStandardOrder.PropertyNameshappens to list them in:subtitles_tracktitlesubtitlereview_status_flagstatusreview_statusIn both cases the shorter, less specific name wins only because it is listed earlier (
titlebeforesubtitle,statusbeforereview_status).The change
Candidates are ranked by
ktsu.FuzzySearch, which was already aPackageReferenceofFrontmatter.csprojbut had no call site anywhere in the repo — the repo's ownCLAUDE.mdflags it as referenced-but-unused. Its score rewards consecutive characters, separator boundaries and camelCase boundaries, and penalizes characters the pattern never matched: the ranking this matcher was approximating by hand.FuzzyRanking.Scorescores both directions and keeps the better one, because the caller's gate is itself bidirectional — a candidate qualifies whether it contains the key or is contained by it, and a one-directional score would leave half of the admitted candidates unranked.What is deliberately not changed
The containment gate. The issue's sketch replaces it with
Fuzzy.Contains, but that matches a subsequence, not a substring, so it admits strictly more candidates. Measuring it against the realStandardOrder.PropertyNameslist, the extra admissions are all low- or negative-scoring (video_subtitlesalso matchesvideo/doi;author_editorial_notesalso matchesaudio/area) and every one of them loses on ranking anyway. Since ranking already discards them, widening the gate buys nothing and risks renaming frontmatter a caller meant to preserve — the failure mode here is silent data loss, so the conservative gate is the right default. Ties still fall back to the standard order, so the existing ordering still decides when the scores cannot.PropertyMerger's word-overlap scorer, the other call site named in the issue. Its candidates only have to share a word, and related pairs likereview_notes/notes_reviewedare neither a substring nor a subsequence of one another — so a whole-name fuzzy score returns nothing for them and hands the decision straight back to iteration order, which is worse than what is there today. Scoring per word pair instead does work, but I could not construct a case that pins the resulting ranking change through the public API, and an unpinned behaviour change on a merge path is not worth landing unattended. Detail left on #113.Testing
FuzzyRanking.Score's contract: symmetry, relative ranking, and that it reports no score when neither name is a subsequence of the other (the property that decides thePropertyMergerquestion above).🤖 Generated with Claude Code
https://claude.ai/code/session_01S9FaM3MJE7n987yxvtxUZD
Generated by Claude Code