Repository navigation
Accept ASCII in -Validate wherever BOM-less UTF-8 is allowed - #136
Merged
Merged
Conversation
Conversion to UTF-8 without a BOM leaves ASCII files unchanged, but -Validate compared labels, so the same file failed -Validate "utf-8" as CharsetNotAllowed (exit 2 under -FailOnChanges). A pipeline that converted with -Target utf-8 and then validated "utf-8,utf-8-bom" failed every file whose text was plain English. -Validate now uses the conversion rule from ConversionPolicy and explains the pass with "ASCII is already valid UTF-8 without a BOM." Only the BOM-less utf-8 label accepts ASCII; utf-8-bom alone still rejects it. The whole file is still validated as ASCII, so a non-ASCII byte past the detection sample is reported invalid, as conversion reports it. Tests cover each allowed list, the explanation and the late byte; three mutations (rule disabled, validation skipped, utf-8-bom accepted) fail them. Backlog: BL-38. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
After #132, conversion to UTF-8 without a BOM leaves ASCII files unchanged ("ASCII is already valid UTF-8 without a BOM").
-Validatestill compared labels, so the same pure-ASCII file failed. Reproduced with themasterbuild:-Validatelistutf-8Invalid/CharsetNotAllowed, exit 2 under-FailOnChangesutf-8,utf-8-bomutf-8-bomA pipeline that converted with
-Target utf-8and then validated with-Validate "utf-8,utf-8-bom"(the example inCLI.md) failed every file whose text was plain English. It also contradicted the plan guarantee text "ASCII already counts as BOM-less UTF-8".Change
ConversionPolicyexposes the rule it already applies (IsAsciiAlreadyUtf8) and its explanation (AsciiAlreadyUtf8Reason). Conversion and-Validatenow both use it, so they accept the same files.-Validateaccepts a detected ASCII file when the list allows BOM-lessutf-8, and explains the pass with "ASCII is already valid UTF-8 without a BOM." A list allowing onlyutf-8-bomstill rejects it, because an ASCII file has no BOM.Invalid/StrictValidationFailed, as conversion reports it.CLI.mdnotes the rule in the-Validaterow; backlog entry BL-38.No conversion decision changes, so the semantics version stays at 8.
Tests
In
AsciiAlreadyUtf8Tests:utf-8,utf-8,utf-8-bom,us-asciiandutf-8-bom, asserting result, reason code, explanation and the-FailOnChangesexit code.éafter it:Invalid/StrictValidationFailed, exit 2.Mutation checks (each restored byte-for-byte afterwards)
-ValidateConvertAndValidateNowAgreeAboutTheSameFile)utf-8-bomalso accepts ASCIIVerification
masterafter Refuse a contradicting source choice even when it is the target #135; the backlog conflict was resolved by keeping both entries (BL-37, BL-38).docs/Test-DefectBacklog.ps1passes.gui-smoke. The full smoke run and the four-corpus audit will be on the final committed release candidate.🤖 Generated with Claude Code