Skip to content

Thread the negotiated position encoding into TextDocument (prerequisite for non-UTF-16 encodings) #371

Description

@Firehed

Summary

Position handling is implemented for UTF-16 (the LSP mandatory default and the only
encoding the server currently supports), but the negotiated encoding is not
threaded into TextDocument — documents are always constructed with the default.
This is correct-by-accident today because UTF-16 is the sole option; it becomes a
real defect the moment a second PositionEncoding case (e.g. utf-8 or utf-32) is
added. This issue tracks the wiring and coverage needed before that day.

Context: the UTF-16 interior conversion landed in #366 (boundary) and #370 (interior
reads/writes + round-trip corpus), per RFC 1 0001-foundational-architecture.md §4.9.
Related: #192.

The gaps

  1. Negotiated encoding is not wired into documents. DocumentManager::open()
    constructs TextDocument with the constructor default (PositionEncoding::Utf16),
    not the value resolved on the session (SessionCapabilities::$positionEncoding).
    So even if negotiation selected a non-default encoding, the interior would still
    use UTF-16.

  2. Only one encoding exists. CapabilityNegotiator::SUPPORTED_ENCODINGS holds a
    single entry and PositionEncoding has a single case, so:

    • There is no utf-8 (character = byte offset, identity) or utf-32
      (character = codepoint) conversion arm.
    • The negotiator's "pick the server's most-preferred among those the client
      offered" loop only ever sees one candidate, so multi-entry selection is
      unexercised (PHPStan forces a new match arm when a case is added, but does not
      assert selection behaviour).

Acceptance criteria

  • The encoding resolved during initialize (SessionCapabilities::$positionEncoding)
    is threaded to wherever TextDocument is constructed, so a document carries the
    negotiated encoding rather than the hardcoded default.
  • A test proves a non-default negotiated encoding reaches the document
    boundary (i.e. offsetAt/positionAt use the negotiated encoding, not an
    assumed UTF-16).
  • Adding a second PositionEncoding case is covered by a negotiator test showing
    the server selects its most-preferred encoding among several the client offers
    (multi-entry SUPPORTED_ENCODINGS).
  • If utf-8 and/or utf-32 support is the driver, each new encoding has
    characterToByteOffset / byteToCharacterOffset arms with the same
    surrogate/width coverage PositionEncodingTest already applies to UTF-16.

Notes

  • This is deferred, not a current bug: UTF-16 is the mandatory LSP default and the
    only encoding the server advertises, so the hardcoded default is presently correct.
  • Do not reach for the native mb_ functions when adding encodings: an LSP
    character is a code-unit count, not a codepoint count, and mb_strlen/mb_substr
    count codepoints (an astral codepoint is two UTF-16 units). See the rationale in
    PositionEncoding.

Issue body generated by AI.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions