diff --git a/AGENTS.md b/AGENTS.md index 8d33ada..2ed21da 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -18,7 +18,7 @@ lsp-cli is a high-level commandline tool that makes it possible to query LSP ser * When done, inform the user about difficulties you've met during the work. If you had no difficulties, omit the report. * If you meet any difficulties with LSP protocol or LSP server implementation - (e.g. bugs or non-standard API), write it down into `GOTCHAS.md` + (e.g. bugs or non-standard API), write it down into `docs/GOTCHAS.md` to the relevant section. * Note that you're a consultant, not a product owner. Only the user may make important architectural desicions. @@ -79,9 +79,9 @@ write a comment why you have to do it. Prefer using methods instead of free functions for "do smth with an object" actions. Use OOP where appropriate. -See @SMELLS.md for known code smells in Rust. +See @docs/SMELLS.md for known code smells in Rust. -See @GOTCHAS.md for LSP protocol/servers tricky/buggy details. +See @docs/GOTCHAS.md for LSP protocol/servers tricky/buggy details. # Commands diff --git a/E2E_BUGS.md b/E2E_BUGS.md deleted file mode 100644 index c78958d..0000000 --- a/E2E_BUGS.md +++ /dev/null @@ -1,127 +0,0 @@ -# E2E: why real-server smoke cases have `exceptions` - -## Mechanism - -In `tests/e2e/manifest/query_case.rs`, a `smoke` pair can be `status: queries`, -and each query case carries an optional `exceptions` list. Each entry names a -`command` (one of the real-server query kinds — `grep`, `references`, -`callers`, `callees`, `build-index`, `format`, etc.), an `outcome` (`failure` -or `empty-matches`), an optional expected stderr `message`, and a mandatory -`reason`. - -At runtime (`tests/e2e/real_servers.rs:112-130,303-338`), if a query has a -matching exception, the harness skips the normal "must succeed with real -matches" assertion and instead asserts the *documented* deviant behavior: - -- `failure`: the command must exit non-zero and stderr must contain `message`. -- `empty-matches`: the command must succeed but return an empty `matches` - array. - -So `exceptions` is not error-tolerance or flakiness suppression — it's a -positive assertion of each server's known, reproducible protocol quirk, with -the `reason` pinned in the yaml so the deviation is self-documenting and any -regression still fails loudly. - -## Root causes, grouped - -1. **No background-indexing-completion signal (`build-index` → `failure`, - message "background-work progress").** Several servers advertise - `$/progress`/work-done tokens but never send a terminal "index build - finished" notification the CLI can wait on: clangd (c, cpp, cuda, objc, - objcpp), OmniSharp (cs), vtsls (js/ts), pyright (python), Zuban (python), - typescript-language-server (js/ts). This is the single most common - exception across the suite. - -2. ~~**Workspace-symbol search (`grep`) returns nothing before indexing - finishes.**~~ **Fixed for clangd, vtsls, and pyright.** clangd, vtsls, - pylyzer, pyright, and EmmyLua used to report empty `matches` for - `workspace/symbol` queries issued immediately after startup, since none - exposes a synchronous "ready" signal. This turned out to be the same - class of bug as item 3 below rather than a pure server quirk: - `run_workspace_symbol_query` (`src/commands/symbol_query.rs`) already had - a prime-and-retry path (`prime_workspace_document`, opening a workspace - file to give the server something to index) but it only triggered when - the first `workspace/symbol` call *errored* — an empty-but-successful - `[]` response was accepted as final. Changed the condition to also prime - and retry on an empty result, with a short poll (a few hundred-ms-spaced - attempts) after priming since indexing an opened document is still - asynchronous. Verified against live clangd (`E2E_CASES="cpp/clangd,c/clangd"`, - also spot-checked `objc`/`objcpp`/`cuda`, which share clangd's exact code - path): `grep Order` now reliably returns real matches instead of `[]`, - even from a cold index, across repeated runs. Removed the `grep` - empty-matches exceptions for clangd (cpp/c/objc/objcpp/cuda), vtsls - (typescript/javascript), and pyright/pylyzer (python) — the same fix - applies architecturally to all of them since they're all document-driven - analyzers with no true "ready" signal. vtsls, pyright, and pylyzer - weren't available to verify locally in this sandbox, so those removals - are provisional pending CI; if any disagrees, revert just that server's - exception (same discipline as the Rust `callees` revert below). EmmyLua's - `grep` exception was left in place — untouched pending separate - investigation. - -3. ~~**ts_ls (typescript-language-server) "No Project" errors.**~~ **Fixed.** - For both TS and JS, ts_ls used to throw `No Project` on `grep`, - `references`, `callers`, `callees`, `definition`, `declaration` because - these all called `workspace/symbol` before any document had been opened - via `textDocument/didOpen`, and ts_ls only attaches a TS project on the - first `didOpen`. This was a genuine CLI ordering bug, not just a server - quirk to document around: `references`/`callers`/`callees`/`definition`/ - `declaration` already had a document-symbol-scan fallback - (`exact_named_document_anchors` in `src/commands/symbol_query.rs`) that - opens documents first, but the code called `workspace/symbol` - unconditionally *before* trying that fallback. Reordering - `select_named_anchors` to try the document-scan path first (falling back - to `workspace/symbol` only when it finds nothing) fixed all five. `grep` - has no document-scan alternative, so it now retries once — opening one - workspace file to prime a project — if the first `workspace/symbol` call - fails. Verified against live ts_ls (`E2E_CASES="typescript/ts_ls,javascript/ts_ls"`) - and re-checked for regressions against rust_analyzer/gopls/pyright, which - share the same code path. - -4. **Call-hierarchy has no edges for the fixture.** Go (`gopls`) and Rust - (`rust_analyzer`) report empty `callees` for `SampleOrder`/`sample_order` - because that fixture constructs data directly rather than calling other - functions. **Fixed for Go** by extracting a `newItem` helper so the - fixture has a real outgoing call; verified against live gopls across - multiple CI runs. **Attempted for Rust** the same way (`new_item` - helper) and it passed repeatedly in local testing, but failed - consistently in CI (`Callees returned no semantic matches`, twice in a - row, even with a retry-on-empty added to the test harness for exactly - this class of indexing race). Since local runs and CI disagree - reproducibly rather than intermittently, this looks like a CI-sandbox - limitation of rust-analyzer's project-model/semantic analysis (not a - timing race the harness can retry past), so the Rust fixture and - exception were reverted to the original state — CI is the trustworthy - signal here, not local runs. EmmyLua (lua) still has no outgoing edges - despite a real same-file call existing in source (a genuine EmmyLua - limitation, not fixable via fixture changes), and pylyzer (python) still - has no incoming (`callers`) edges despite a real cross-file caller - existing. - -5. **Server-specific formatting/output bugs.** EmmyLua's `format` returns an - edit whose range falls outside the requested file — a genuine bug in the - server, tolerated as a `failure` exception with matched message - ("returned a line outside"). - -## Related: servers excluded entirely (`status: excluded`) - -These aren't `exceptions` entries but explain further gaps in coverage: - -- `denols` (Deno LSP) rejects the standard shutdown request because it - requires non-null parameters. -- `roslyn_ls` (cs) and `lua_ls` (lua) have lifecycle-level incompatibilities: - no smoke queries at all, or no clean exit after direct shutdown. -- Several Python servers fail to launch/initialize correctly in the isolated - harness: `pylsp` (Mason launcher can't import the module), `pyre` (same), - `pyrefly` (initializes but returns no workspace/document symbols). - -## Bottom line - -The exceptions exist because real LSP servers deviate from the LSP spec's -strict guarantees in ways that are reproducible but server-specific — mainly -(a) no standard signal for "background indexing/build is done," (b) symbol -search racing ahead of indexing, and (c) a couple of servers requiring a -document to be opened before workspace-wide queries work. Rather than -weakening assertions globally, the suite encodes each deviation explicitly -per server/command so real regressions still fail, while known quirks are -pinned and self-documented via `reason`. diff --git a/E2E_TESTS.md b/E2E_TESTS.md deleted file mode 100644 index 21e650e..0000000 --- a/E2E_TESTS.md +++ /dev/null @@ -1,607 +0,0 @@ -# End-to-end test plan - -## Goal - -Exercise the released `lsp-cli` binary against every supported language, every compatible -supported LSP server, and every top-level subcommand. Keep the suite useful both as a fast pull -request check and as an exhaustive compatibility check. - -The tests must validate user-visible behavior: exit status, stdout, stderr, filesystem effects, -server lifecycle, and semantically relevant LSP results. They must not depend on private Rust APIs. - -## Local dev environment - -The real-server E2E targets (`test-real-server-e2e`, `test-real-server-smoke-e2e`, -`test-server-provisioning-e2e`) need `go`, `java`, `node`/`npm`, and `dotnet` on `PATH`, matching -what CI installs in `.github/workflows/ci.yml` and `e2e.yml`. Run `make download-dev-env` to fetch -project-local copies into `.env/` (not a system-wide install), then `source activate.sh` from the -repo root to put them on `PATH` for the current shell. - -## Working definition of supported - -At pinned `lsp-cli-data` revision `a71b45d8f0402c9aea220922d713eeead5880b72`, the data tree has: - -- 362 filetype configurations; -- 362 LSP configurations; -- 336 detectable filetype IDs (a non-empty `extensions` or `patterns` list); -- 359 distinct LSP configurations associated with those detectable filetypes; -- 850 compatible detectable-filetype/LSP pairs. - -The working E2E scope is now the 336 detectable IDs (see `tests/e2e/cases/*.yaml`, one file per -language ID, for the full list). The original ten source-project languages -(`c cpp cs cuda go java javascript kotlin lua objc objcpp python rust typescript`, plus the -`gomod`/`gowork` metadata projects) keep real symbol-query coverage; the remaining 320 IDs were -added as `kind: metadata` cases (detection plus, where a compatible server is downloadable, a -capabilities-only smoke check), matching the `gomod`/`gowork` pattern rather than building a real -per-language source fixture and query profile for each one. - -The remaining 26 filetype configurations have no detection rules. They are configuration catalog -entries, but cannot currently drive a project-based E2E test. Separately test that the whole data -tree parses and that catalog commands describe it consistently. - -This is a product-policy boundary rather than an implementation fact. Before declaring the suite -complete, the product owner must confirm one of these definitions: - -1. **Detectable support (recommended):** exhaustive real-server tests cover the 336 detectable IDs, - 359 relevant servers, and 850 compatible pairs. -2. **Configured support:** all 362 filetype and server configs are considered supported. This first - requires adding detection rules, test projects, and provisioning for the presently inactive - catalog entries. - -Pros of detectable support: executable now, objective, and automatically derived from the shipped -data. Cons: `lsp-cli languages` currently appears to expose a wider catalog than this definition. - -Pros of configured support: the word “supported” matches every shipped YAML entry. Cons: most of -the required projects and provisioning do not exist, and millions of incompatible Cartesian cases -would still need to be excluded. - -## Non-goals - -- Do not run a Cartesian product of every language, server, command, and option. Only configured - language/server relationships are meaningful. -- Do not require a server to implement an optional LSP capability. -- Do not put language-specific parsing or source-code knowledge into production `lsp-cli` code. -- Do not make tracked playground files writable test state. -- Do not add a Rust dependency without explicit permission. The existing `tempfile`, `serde`, - `serde_json`, and `serde_yaml` dependencies are sufficient for the planned harness. - -## Test projects - -### Existing projects - -Reuse the projects under `playground/` for: - -| Filetype ID | Directory | -|---|---| -| `c` | `playground/c` | -| `cpp` | `playground/cpp` | -| `cs` | `playground/csharp` | -| `go` | `playground/go` | -| `java` | `playground/java` | -| `javascript` | `playground/js` | -| `lua` | `playground/lua` | -| `python` | `playground/python` | -| `rust` | `playground/rust` | -| `typescript` | `playground/typescript` | - -### Additional projects - -Source projects are committed for: - -- `playground/cuda` -- `playground/kotlin` -- `playground/objc` -- `playground/objcpp` - -Minimal detection fixtures are committed under `playground/gomod` and `playground/gowork`. These -IDs describe Go workspace metadata, not source languages, so they can cover detection, file -listing, server selection, initialization, and lifecycle, but cannot independently provide -meaningful symbol or call-hierarchy assertions. - -Every source-language project should be small, valid, and multi-file. Where the language permits, -it should contain: - -- one stable workspace symbol; -- functions and methods; -- a declaration separated from its definition; -- references from more than one file; -- a caller and callee chain; -- types and fields; -- a file whose formatting can be made deterministically incorrect; -- a deterministic source mutation that produces one diagnostic. - -Prefer equivalent domain concepts and symbol names across projects when natural. Do not force a -language into constructs it does not support merely to make fixtures textually identical. - -### Existing playground audit - -The ten existing playgrounds were audited against these requirements on 2026-09-05. `Present` -means the committed source provides the semantic shape; `missing` identifies follow-up work; -`unverified` means the required compiler or runtime was not installed for this audit. Under the -strict declaration rule, an interface, trait, protocol, or declaration file is required when the -language can express one without relying on comments or third-party syntax. - -| Project | Valid, small, multi-file | Stable workspace symbol | Functions and methods | Separate declaration | Cross-file references | Caller/callee chain | Types and fields | Formatting mutation | Diagnostic mutation | -|---|---|---|---|---|---|---|---|---|---| -| C | Present; portable `compile_flags.txt` configures clangd | `Order` | Functions present; methods not applicable | Present in `order.h` | Present | Present | Present | Missing recipe; baseline is not formatter-clean | Missing recipe | -| C++ | **Invalid:** undefined `f()` and `g()` prevent linking; portable `compile_flags.txt` configures clangd | `playground::Order` | Present | Present in `order.hpp` | Present | Present | Present | Missing recipe; baseline is not formatter-clean | Missing recipe | -| C# | Unverified; `dotnet` unavailable | `Order` | Present | **Missing:** an interface can provide it | Present | Present | Present | Missing recipe | Missing recipe | -| Go | Unverified; `go` unavailable | `Order` | Present | **Missing:** an interface can provide it | Present | Present | Present | Missing recipe; baseline is visibly not `gofmt`-clean | Missing recipe | -| Java | Unverified; JDK and Maven unavailable | `Order` | Present | **Missing:** an interface can provide it | Present | Present | Present | Missing recipe | Missing recipe | -| JavaScript | Present; exercised with Node.js | `Order` | Present | **Missing:** a declaration file can provide it | Present | Present | Present | Missing recipe | Missing recipe | -| Lua | Unverified; Lua unavailable | **Missing:** only local functions are declared | Functions present; methods missing | Not applicable: Lua has no native declaration construct | Present for `format_timestamp` | Present | Partial: a module table field exists, but no structured domain type | Missing recipe | Missing recipe | -| Python | Present; exercised with Python | `Order` | Present | **Missing:** a protocol or abstract base can provide it | Present | Present | Present | Missing recipe; baseline is visibly not formatter-clean | Missing recipe | -| Rust | **Invalid as a standalone project:** Cargo treats it as an undeclared root-workspace member | `Order` | Present | **Missing:** a trait can provide it | Present | Present | Present | Missing recipe | Missing recipe | -| TypeScript | Unverified; local TypeScript compiler unavailable | `Order` | Present | **Missing:** an interface can provide it | Present | Present | Present | Missing recipe | Missing recipe | - -The C sources compile and link, while the C++ sources compile but fail at link time because the -calls added in `main.cpp` have no definitions. JavaScript and Python execute successfully. The Rust -check fails before compilation because the nested package is neither a root-workspace member nor -excluded from that workspace. C and C++ now use portable `compile_flags.txt` files because a -tracked compilation database cannot keep its required absolute directory valid after the fixture -is copied into an isolated E2E sandbox. - -No playground currently defines the exact source edit and expected diagnostic needed for a stable -mutation test. Those recipes should live in manifest data rather than language-specific Rust test -code. The next project-phase item will repair and normalize the fixtures; this audit intentionally -does not mix those changes with the inventory. - -Tests securely create randomized sandboxes with the `tempfile` crate under the user's -`XDG_RUNTIME_DIR`, `XDG_CACHE_HOME`, or `$HOME/.cache`, in that order, then copy a project there -before formatting it or introducing diagnostics. Real test state must not use the ambient system -`/tmp`. This keeps the repository clean, avoids uncontrolled parent root markers, keeps Unix socket -paths short, and allows safe parallel execution. - -Pros of committed playgrounds: humans can reproduce failures with the same projects. Cons: each -language fixture must evolve with its toolchain and server ecosystem. - -An alternative is to generate every project during test setup. That reduces committed files, but -makes failures harder to inspect and manual reproduction less convenient; do not use it for the -baseline projects. - -## Coverage model - -`lsp-cli` currently has 24 canonical top-level subcommands. Factor them by responsibility instead -of multiplying all commands by all language/server pairs. - -| Scope | Subcommands | Required coverage | -|---|---|---| -| Global CLI | `commands`, `languages`, `servers`, `completion`, `agent-skill`, `update` | Focused binary-level cases, independent of real servers | -| Detection and filesystem | `detect`, `list-files` | Every detectable filetype ID | -| LSP requests | `server-capabilities`, `diagnostics`, `format`, `grep`, `list-symbols`, `list-functions`, `references`, `callers`, `callees`, `definition`, `declaration`, `build-index` | Every compatible language/server pair, capability-aware | -| Process lifecycle | `run`, `daemon`, `stop`, `stop-all` | Every distinct relevant server where applicable, with grouped lifecycle scenarios | - -### Capability-aware expectations - -For each compatible pair, first record or inspect the server's initialized capabilities. A command -passes if it either: - -- succeeds and returns the expected semantic result; or -- returns the documented, user-facing unsupported-capability error when the server does not - advertise the required capability. - -Formatting, declarations, diagnostics, workspace symbols, and call hierarchy are optional or vary -substantially between servers. Treating every unsupported operation as a suite failure would test -an assumption the LSP specification does not make. - -Capability advertisement is not enough by itself: when a server advertises a capability, exercise -the corresponding command and assert its behavior. - -### Option coverage - -Distribute option variants across the matrix using explicit cases; do not create another full -cross-product. Cover at least: - -- automatic selection, `--lang`, and `--lsp`; -- text and `--json` output; -- direct execution, `--detach`, and `--no-detach`; -- `--limit`, `--files-with-matches`, and `--full`; -- `--wait-for-index`; -- `format`, `format --check`, and `format --stdout`; -- successful operations, unsupported capabilities, missing executables, server crashes, malformed - replies, and timeouts; -- `--download` once per supported installation mechanism, rather than redundantly for every query. - -JSON assertions should deserialize and compare stable semantic fields. Text assertions should -avoid full snapshots when server versions can legitimately change ordering, signatures, or detail. - -## Harness design - -Use a normal Cargo integration-test crate that invokes the built binary through -`CARGO_BIN_EXE_lsp-cli`. - -Proposed layout: - -```text -tests/ - e2e.rs - e2e/ - harness.rs - manifest.rs - catalog.rs - detection.rs - queries.rs - lifecycle.rs - update.rs - cases/ - suite.yaml - .yaml -``` - -Keep every Rust file under 600 lines. Move repeated process setup and assertions into helpers as -soon as a second test needs them. - -`harness.rs` should provide methods for actions on an E2E context, for example: - -- create an isolated home, configuration root, runtime root, and workspace copy; -- construct an `lsp-cli` process with deterministic environment variables; -- run a command with a deadline and capture stdout/stderr/status; -- parse JSON output; -- introduce a formatting or diagnostic mutation; -- find and terminate remaining child processes; -- stop daemons and report their runtime state after failure. - -Each test process should set at least: - -- `HOME` to an isolated temporary home; -- `XDG_CONFIG_HOME` to an isolated configuration directory; -- `XDG_RUNTIME_DIR` to an isolated daemon directory; -- `LSP_DATA` to the pinned repository submodule; -- `PATH` to the explicitly provisioned toolchain/server environment. - -Do not rely on a developer's user configuration, downloaded server cache, daemon sockets, current -shell, or ambient server versions. - -The manifest directory should include stable case data, provisioning metadata, expected -capabilities, and documented exclusions. `tests/e2e/cases/suite.yaml` owns global command coverage, -while each `tests/e2e/cases/.yaml` owns one project and its configured server behavior. -A validation test should fail when: - -- a detectable filetype lacks a project; -- a configured E2E case names an incompatible or missing data config; -- an exclusion lacks a reason; -- two cases select the same user-visible server ambiguously; -- a new top-level subcommand has no assigned coverage class. - -The version 8 manifest assigns every canonical command to a coverage strategy, derives one -preferred smoke-matrix server for every source-language project from `data/lsp-cli.yaml`, and uses -`coverage: complete`. The compatible inventory—16 detectable languages, 57 relevant servers, and -141 language/server pairs—is resolved directly from the pinned data. Case YAML contains only -E2E-specific behavior, avoiding a second copy of each server's `filetypes` list. - -The suite manifest also assigns each of the 57 relevant server configs exactly one provisioning -disposition. Twenty-two general-purpose servers are downloaded through the production Mason path; -the other 35 have explicit policy or technical exclusions. Shared server setup owns installer and -runtime prerequisites so language/server behavior overlays do not duplicate them. - -### Extending the manifest - -To cover an existing detectable filetype, add its small project under `playground/` and one case -file named after the filetype ID. Compatible servers are discovered from `data/lsp/*.yaml`; add a -`pairs` entry only when the E2E suite has executable behavior or a reviewed exclusion for that -pair. To introduce a genuinely new filetype or server, first add its YAML config and commit it in -the `data` submodule, then update the submodule revision. Add a project/case file here for a new -detectable filetype, and add pair-specific E2E behavior when it is ready. - -Pair entries are sparse E2E behavior overlays and use the LSP YAML filename stem as their stable -config ID. Bare compatibility entries are rejected because compatibility belongs to `data`. The -test runner loads the configured user-visible server name for `--lsp`; do not duplicate it in the -manifest. The first server in each source language's production preference list is also its -merge-gate smoke server. -Manifest validation resolves that user-visible name to one compatible LSP config and requires the -corresponding pair to exist. Each preferred pair has a tagged `smoke` disposition: either a generic -query suite or an exclusion with a mandatory reviewed reason. Executable pairs keep provisioning -and runtime host programs in the shared `setup` block. Source-language query profiles declare -shared semantic terms, expected symbols, and format paths; pair entries keep only deadlines and -narrowly scoped known-result exceptions. Language-specific -prerequisites and expectations belong in YAML, not in the Rust runner. The first provisioning -method is `download`; add other mechanisms as typed methods when needed instead of branching on -server names. - -Every distinct preferred server has exactly one explicit lifecycle-owner pair. Its tagged -`lifecycle` disposition either runs grouped daemon scenarios or records a reviewed exclusion; -direct `run` may be excluded independently when only detached operation is reliable. This keeps -the chosen project stable when a shared server gains another filetype without repeating process -tests for every compatible pair. - -The query runner obtains raw initialized capabilities through `server-capabilities --json`, then -executes every LSP query command. Advertised capabilities require a successful semantic response; -missing capabilities require the command's user-facing unsupported error. Known deviations must -name the command, expected outcome, reason, and a stable error fragment for expected failures. -`E2E_CASE=/` selects one configured executable or explicitly excluded case for -manual diagnosis without changing the all-cases CI default. Selecting a compatible pair without -E2E behavior fails with a clear error instead of silently running no tests. - -### Preferred server matrix - -| Languages | LSP config ID | -| --- | --- | -| C, C++, CUDA, Objective-C, Objective-C++ | `clangd` | -| C# | `roslyn_ls` | -| Go | `gopls` | -| Java | `jdtls` | -| JavaScript, TypeScript | `ts_ls` | -| Kotlin | `kotlin_lsp` | -| Lua | `lua_ls` | -| Python | `pyright` | -| Rust | `rust_analyzer` | - -These selections come from the shipped production preferences, not duplicated manifest flags or -language-specific runner branches. Real-server tests use `--download` with a clean isolated home, -so Mason's current registry release selects and installs the server version on every run. Failure -diagnostics must retain the resolved package source ID so an upstream version change can be -identified after the fact. - -In `coverage: complete` mode, manifest validation makes a new detectable filetype fail until its -project is declared. New compatible relationships are automatically part of the resolved inventory; -the later exhaustive-matrix checks track whether each has executable behavior or an exclusion. -Partial mode remains available for isolated manifest fixtures and staged downstream suites. - -## Special command strategies - -### `run` - -`run` replaces the current process with the language server on Unix. The foundation smoke uses a -deterministic server marker to prove replacement. Real-server coverage should additionally use -piped stdio for a minimal `initialize` / `initialized` / `shutdown` / `exit` exchange. Test -selection and exec errors separately. - -### `daemon`, `stop`, and `stop-all` - -For every applicable server, run a grouped lifecycle scenario: - -1. start a daemon in an isolated runtime directory; -2. issue at least two queries with `--detach` and verify reuse; -3. stop the exact daemon; -4. verify that a later query starts or connects according to documented behavior; -5. start multiple isolated daemons and verify `stop-all` removes all of them. - -On failure, print socket paths, process state, selected command line, workspace root, and bounded -server stderr. Cleanup must run even after an assertion failure. - -### `update` - -The production default uses the lsp-cli-data GitHub release endpoint. The narrowly scoped -`LSP_CLI_DATA_RELEASE_API_URL` override redirects only release metadata lookup, allowing the binary -E2E test to serve metadata and a valid archive locally. This keeps the success path deterministic -without changing normal update behavior. - -### Diagnostics and formatting - -Start from a valid temporary workspace. Apply one language-specific mutation recorded in the -manifest, run the command, assert the expected file/range/message class, and discard the temporary -copy. Do not commit permanently broken source files that could interfere with unrelated queries. - -### Indexing - -`build-index` should be tested against every server that has a usable background-work completion -signal. For other servers, assert the intended bounded timeout or no-op policy. Do not infer -completion from a fixed sleep. - -## Real-server provisioning - -Do not install LSP servers separately. Every real-server case must pass `--download`, allowing the -production Mason integration to select the current registry package, install it inside the case's -isolated home, and return the resolved executable. This applies uniformly to direct archives and -npm, PyPI, Cargo, Go, NuGet, GitHub, or generic package sources supported by the downloader. - -Language SDKs and package-manager runtimes remain explicit host prerequisites. Keep their resolver -commands in the manifest so a missing prerequisite produces a case-specific error rather than a -silent skip. - -Do not silently skip a required pair because its executable is absent. A CI lane either provisions -the server or reports the pair as an explicit, reviewed exclusion. - -Downloading these external test tools requires product-owner approval under the repository's -dependency policy. Latest-Mason server downloads and official free SDK/runtime provisioning are -approved. They do not become Rust package dependencies, but they remain operational dependencies -with maintenance, security, licensing, storage, and network consequences. Proprietary and -unsupported-platform prerequisites are excluded. - -### Provisioning inventory - -Run the complete downloadable-server inventory manually with: - -```sh -make test-server-provisioning-e2e -``` - -Set `E2E_SERVER=` to diagnose one downloadable server. The test copies the server's -owner project into an isolated context, stages its declared host programs, invokes `detect` with -`--download`, and verifies that exactly one selected command resolves inside the isolated home. -It does not initialize the server or substitute for the later language/server behavior matrix. - -The downloadable inventory is: - -```text -basedpyright clangd denols emmylua_ls gopls jdtls jedi_language_server -kotlin_language_server kotlin_lsp lua_ls omnisharp pylsp pylyzer pyre pyrefly pyright -roslyn_ls rust_analyzer ts_ls ty vtsls zuban -``` - -Exclusions cover specialized framework, lint, formatting, spelling, security, AI, and adapter -servers; deprecated servers; `sourcekit` on the Linux lane; configs without current Mason packages; -and `java_language_server`, whose Mason source-build recipe is not supported by lsp-cli. These are -provisioning decisions only. On the approved Linux x86_64 lane, all 31 compatible pairs backed by -downloadable servers have an explicit query, capabilities-only, or reviewed-exclusion disposition. -The other 110 pairs inherit their server's reviewed provisioning exclusion, avoiding duplicate -policy text while keeping all 141 compatible pairs classified. - -Always using Mason latest detects upstream compatibility changes immediately and avoids maintaining -a second installation path. The tradeoff is a nondeterministic merge gate: a registry or server -release can break an unchanged commit, and reproducing the failure depends on the recorded source -ID remaining available upstream. - -## CI plan - -### Pull requests - -Run: - -- all existing unit tests and checks through `make test`; -- global and detection E2E tests; -- one preferred current-Mason server per source language; -- every relevant subcommand across that smoke matrix; -- manifest/data consistency checks. - -### Nightly exhaustive matrix - -Run all 141 compatible pairs, sharded by language and server installation family. Use fail-fast -disabled so one broken server does not hide the rest of the compatibility report. - -The planner resolves installation families from the current Mason registry rather than copying -that registry metadata into `cases/`. Each shard receives a comma-separated `E2E_CASES` selection. -Suite-level smoke, lifecycle, and provisioning deadlines provide the common defaults; cases only -declare intentional overrides. - -Do not share homes, daemon runtime directories, or mutable workspaces between parallel jobs. CI may -cache immutable download transport data, but each case must retain isolated runtime state and must -not substitute a separately installed server for `--download`. - -Every real-server case explicitly tears down its isolated home and temporary roots before the next -case starts. This includes Mason packages, Go module/build caches, and other server download state; -only immutable Rust build artifacts are shared by CI. - -### Manual workflow - -Allow selection of one language, one server, or one installation family. This is needed to debug a -nightly failure without rerunning the complete matrix. - -Pros of split CI: fast merge feedback plus exhaustive coverage. Cons: a regression affecting a -non-preferred server may be found the following night rather than on the originating pull request. - -Running all 141 pairs on every pull request gives earlier detection, but has much higher latency, -cost, rate-limit exposure, and upstream-flake risk. It is not the recommended default. - -## Failure policy - -Classify failures as: - -1. `lsp-cli` regression; -2. playground/manifest drift; -3. provisioning or network failure; -4. upstream server behavior change; -5. known server limitation; -6. unsupported LSP capability with the expected user-facing response. - -Only category 6 is an immediate passing outcome. Known limitations must be explicit manifest -entries and, when they concern protocol or server behavior, documented in `GOTCHAS.md`. Do not add -unbounded retries. A retry may cover an identified transient installation/network step, but must -not conceal query or protocol failures. - -If a hard-to-debug defect is fixed, add a focused regression test in addition to the broad matrix. -Also consider whether a type invariant, runtime check, clearer trace, or state-dump helper can make -that class of defect easier to diagnose. - -## Execution phases - -### Phase 0: approve boundaries - -- [x] Confirm detectable support versus configured support: use the 16 detectable IDs. -- [x] Confirm that unsupported optional capabilities count as a passing, asserted outcome. -- [x] Approve latest-Mason external server provisioning through `--download`. -- [x] Approve a narrow HTTP endpoint seam for deterministic `update` E2E coverage. -- [x] Confirm PR smoke plus nightly exhaustive CI cadence. - -### Phase 1: foundation - -- [x] Add `tests/e2e.rs` and compact harness modules. -- [x] Add the initial manifest schema and validation. -- [x] Isolate all environment and runtime state. -- [x] Implement deadlines, cleanup guards, JSON helpers, and useful failure diagnostics. -- [x] Prove the harness with Rust/rust-analyzer. -- [x] Cover all 24 subcommand paths with either a real server or a deterministic local fixture. - -### Phase 2: projects - -- [x] Audit the ten existing playgrounds against the common semantic requirements. -- [x] Remove duplicated setup patterns within each class of fixture. -- [x] Add CUDA, Kotlin, Objective-C, and Objective-C++ projects. -- [x] Add `gomod` and `gowork` detection fixtures. -- [x] Update `playground/README.md` with manual reproduction commands. - -### Phase 3: preferred-server smoke matrix - -Manual LSP verification follows server selection and downloader support so it runs against servers -resolved by the same current Mason registry used in CI rather than ambient installations. - -The 2026-09-05 and 2026-09-06 manual surveys used isolated `tempfile` sandboxes and current Mason -packages. A checksum-pinned Go 1.27.1 SDK was provisioned only inside each Go survey sandbox. No -survey state used the ambient system `/tmp` or modified a tracked playground. The harness sets both -`TMPDIR` and the JVM's `java.io.tmpdir`, because the latter does not inherit the former. - -| Project | Server source | Verified behavior | Remaining blocker or limitation | -|---|---|---|---| -| CUDA | `pkg:github/clangd/clangd@22.1.6` | Detection, files, capabilities, diagnostics, document symbols/functions, definition/declaration, references, callers/callees, `format --stdout`, direct execution, daemon reuse, and stop | Immediate `grep Order` returned no workspace symbols; `build-index` reported no background-work progress | -| Objective-C | `pkg:github/clangd/clangd@22.1.6` | Same applicable paths as CUDA, with clean diagnostics and semantic results | Immediate workspace-symbol grep was empty; `build-index` exposed no progress | -| Objective-C++ | `pkg:github/clangd/clangd@22.1.6` | Same applicable paths as Objective-C, with clean diagnostics and semantic results | Immediate workspace-symbol grep was empty; `build-index` exposed no progress | -| Kotlin | Mason generic `kotlin-lsp` package, version `kotlin-lsp/v262.9593.0` | Template rendering, download, detection, and file listing passed; foreground and detached initialization were attempted; daemon creation and stop passed | The current packaged `intellij-server` reports that its build has expired and exits before LSP initialization | -| Go module metadata | `pkg:golang/golang.org/x/tools/gopls@v0.23.0` with Go 1.27.1 | Detection, file listing, foreground and detached capabilities, daemon creation/reuse, and stop passed | Metadata-only fixture intentionally has no source-level semantic assertions | -| Go workspace metadata | `pkg:golang/golang.org/x/tools/gopls@v0.23.0` with Go 1.27.1 | Detection, file listing, foreground and detached capabilities, daemon creation/reuse, and stop passed | Metadata-only fixture intentionally has no source-level semantic assertions | - -The clangd projects use portable `compile_flags.txt` files. CUDA is parsed as C++ with its CUDA -qualifiers defined as empty macros, keeping semantic queries deterministic without requiring a -CUDA SDK; this fixture validates lsp-cli/LSP behavior, not CUDA compilation. Committed playgrounds -must not contain `compile_commands.json`, whose required absolute working directories become stale -when the harness copies a project. - -- [x] Configure one production preference per source language and derive the smoke matrix from it. -- [x] Provision servers through `--download`; add no separate installers or Rust dependencies. -- [x] Run each relevant command manually against every new project and record each success or - classified upstream limitation. -- [x] Implement capability-aware query assertions. -- [x] Implement direct/detached lifecycle scenarios. -- [x] Add the pull-request E2E job. - -### Phase 4: exhaustive compatibility - -- [x] Resolve all 141 compatible pairs from pinned data without duplicating `filetypes` in cases. -- [x] Provision every non-excluded server and required SDK. -- [x] Record reviewed exceptions and platform constraints. -- [x] Add sharded nightly and manual workflows. -- [x] Verify failures retain server version, command line, capabilities, stderr summary, and cleanup - state. - -### Phase 5: hardening - -- [x] Run `make test`. -- [x] Run the full latest-Mason E2E matrix from a clean environment. -- [x] Check every new or edited test file for boilerplate and duplication. -- [x] Check every source file remains below 600 lines. -- [x] Add regression tests for every bug uncovered during rollout. -- [x] Add LSP/server-specific discoveries to `GOTCHAS.md`. -- [x] Document upstream server version identification and nightly triage in [E2E_TRIAGE.md](E2E_TRIAGE.md). - -## Definition of done - -The work is complete when: - -- every accepted supported language has a committed project or justified metadata-only fixture; -- every compatible supported language/server pair has an executable manifest entry; -- every top-level subcommand has binary-level E2E coverage in its appropriate scope; -- advertised capabilities are exercised and unsupported capabilities have asserted user-facing - behavior; -- direct, detached, stop, and stop-all lifecycle paths are covered; -- formatting and diagnostics cannot dirty tracked files; -- required servers are provisioned through `--download`, and resolved source IDs are retained; -- PR smoke, nightly exhaustive, and manual targeted workflows are documented and passing; -- `make test` passes; -- known protocol/server deviations are recorded in `GOTCHAS.md`; -- no required case is silently skipped. - -## Architectural consequences and limitations - -- The data catalog becomes an enforceable compatibility contract: adding a detectable filetype or - compatible LSP config requires E2E ownership. -- Real-server E2E tests are inherently slower and less hermetic than protocol tests. Unit tests and - fake-server integration tests remain necessary for precise edge cases. -- Capability-aware results mean “all commands tested” does not mean “all commands succeed on every - server.” It means every applicable success path and every inapplicable user-facing response is - verified. -- Latest-server testing creates recurring upstream-flake, security-review, and reproducibility - risks even when this repository is unchanged. -- Some servers require proprietary, platform-specific, or unusually heavy SDKs. Their treatment - must be an explicit product decision rather than an automatic skip. -- The current difference between the 362 configured filetypes and 16 detectable filetypes may need - a future terminology or behavior change in `languages`; this plan exposes but does not decide - that product question. diff --git a/E2E_TRIAGE.md b/E2E_TRIAGE.md deleted file mode 100644 index 16209ef..0000000 --- a/E2E_TRIAGE.md +++ /dev/null @@ -1,120 +0,0 @@ -# E2E compatibility triage - -This guide is for maintainers investigating the scheduled **End-to-end compatibility** workflow -or the real-server pull-request job. These tests install the current package selected by the Mason -registry, so an unchanged lsp-cli commit can fail after an upstream release. - -## Find the failing pair - -Pairs use `/`, for example `python/pyright`. The server component is the -filename stem under `data/lsp/`, not necessarily the executable or display name. - -1. Open the workflow summary and find the pair's executable or excluded classification. -2. Open the failed `/` matrix job. -3. Read the `E2E failed case IDs` footer. It lists the specific failed pairs collected from the - longer diagnostics above it. - -If the footer says the IDs are unavailable, failure occurred before a case emitted its labelled -diagnostic. Start with the planner, build, or test-runner error immediately above the footer. - -## Identify the upstream version - -A failed case retains this block before deleting its isolated home: - -```text -server package source IDs: -pkg:npm/pyright@1.1.409 -server command line: -... -server capabilities: -... -server stderr summary: -... -cleanup state: -... -``` - -Treat each complete `pkg:/@` source ID as the authoritative -package identity. Preserve the whole value: versions and package names can contain prefixes, -scopes, or backend-specific suffixes. The executable name and `initialize` response version may -describe a product differently and are supporting evidence, not replacements for the source ID. - -`` normally means provisioning failed before a -receipt was written. A missing or malformed receipt is reported separately. Inspect the preceding -download/install error and do not infer a version from an older run. - -Outside the isolated suite, successful downloads store JSON receipts under -`~/.local/share/lsp-cli/receipts/`; the `source_id` field has the same meaning. E2E case homes and -their receipts are intentionally removed after every case, including failures, so use the retained -diagnostic rather than a path printed earlier in the log. - -Successful E2E cases do not print retained failure context. The suite follows Mason latest, so a -later rerun may resolve a different source ID. Compare IDs from available failing runs and record -the ID in an issue when exact upstream identity matters; the suite does not promise that the -registry will retain an older version for reproduction. - -## Reproduce a narrow case - -Use a config ID from `tests/e2e/cases/`, and run from the repository root: - -```sh -E2E_CASE=python/pyright make test-real-server-smoke-e2e -``` - -If the failure involves direct or detached process behavior, run the combined query and lifecycle -target with the same selector: - -```sh -E2E_CASE=java/jdtls make test-real-server-e2e -``` - -For installation failures, isolate provisioning before starting the server: - -```sh -E2E_SERVER=pyright make test-server-provisioning-e2e -``` - -These commands download external tools and require the host programs declared by the manifest. -They also use the current Mason registry; compare the reproduced source ID with the original before -concluding that behavior changed locally. - -The manual **End-to-end compatibility** workflow can select `language`, `server`, or -`installation-family`. The value is respectively a case language ID, an LSP config ID, or one of: - -```text -cargo generic github golang npm nuget pypi -``` - -For example, with the GitHub CLI: - -```sh -gh workflow run e2e.yml -f selector=server -f value=pyright -``` - -Use `all` with an empty value for the complete matrix. The generated workflow summary shows all -selected pairs, including reviewed exclusions, before runnable pairs are sharded. - -## Classify the failure - -Check evidence in this order: - -1. **Planner or manifest:** an unknown selector, missing registry package, or validation failure - happened before a server case ran. -2. **Provisioning or network:** no completed receipt, package-manager output, HTTP failure, or an - absent host program points to installation rather than LSP behavior. Use the provisioning-only - command above. -3. **Startup or shutdown:** use the retained command line and server stderr. SDK incompatibility, - launcher failure, crash, and failure to exit are distinct from query-result drift. -4. **Protocol or capability:** compare the retained capabilities with the command exercised by the - case. An unadvertised optional capability passes only when lsp-cli returns its expected - user-facing unsupported error. -5. **Semantic result:** compare stable names and the pair's reviewed exceptions. Do not weaken an - expectation until the same source ID reproduces the behavior or an upstream change is confirmed. -6. **Cleanup:** inspect sandbox and runtime roots independently of the primary failure. A successful - query with a retained root is still a cleanup regression. - -Classify the result as an lsp-cli regression, fixture/manifest drift, provisioning or network -failure, upstream behavior change, known server limitation, or expected unsupported capability. -Do not add unbounded retries or turn crashes and provisioning failures into accepted query -exceptions. A confirmed stable server limitation belongs in both its manifest disposition and -`GOTCHAS.md`; an lsp-cli defect needs a focused regression test. diff --git a/DOWNLOAD.md b/docs/DOWNLOAD.md similarity index 100% rename from DOWNLOAD.md rename to docs/DOWNLOAD.md diff --git a/GOTCHAS.md b/docs/GOTCHAS.md similarity index 100% rename from GOTCHAS.md rename to docs/GOTCHAS.md diff --git a/SMELLS.md b/docs/SMELLS.md similarity index 100% rename from SMELLS.md rename to docs/SMELLS.md diff --git a/SUBCOMMAND_RESEARCH.md b/docs/SUBCOMMAND_RESEARCH.md similarity index 100% rename from SUBCOMMAND_RESEARCH.md rename to docs/SUBCOMMAND_RESEARCH.md diff --git a/TODO.md b/docs/TODO.md similarity index 100% rename from TODO.md rename to docs/TODO.md diff --git a/tests/e2e/Readme.md b/tests/e2e/Readme.md new file mode 100644 index 0000000..09e9a13 --- /dev/null +++ b/tests/e2e/Readme.md @@ -0,0 +1,443 @@ +# End-to-end tests + +## Goal + +Exercise the released `lsp-cli` binary against every supported language, every compatible +supported LSP server, and every top-level subcommand. Keep the suite useful both as a fast pull +request check and as an exhaustive compatibility check. + +The tests validate user-visible behavior: exit status, stdout, stderr, filesystem effects, server +lifecycle, and semantically relevant LSP results. They do not depend on private Rust APIs. + +## Scope + +"Supported" means **detectable support**: real-server tests cover every filetype ID that has a +detection rule (a non-empty `extensions` or `patterns` list), every LSP config compatible with +those filetypes, and every resulting compatible language/server pair. A filetype configuration +with no detection rule is a configuration catalog entry only — it cannot drive a project-based E2E +test, so those entries are covered separately by parsing/catalog consistency checks, not real +queries. + +The current inventory (detectable filetypes, relevant servers, compatible pairs) is derived +directly from the pinned `lsp-cli-data` submodule revision and drifts as that data changes; treat +`tests/e2e/cases/` (one YAML file per language ID) and `tests/e2e/cases/suite.yaml` +(`schema-version`, `coverage: complete`) as the source of truth for current counts rather than any +number written here. + +Open product question (not yet decided): the full data catalog has more configured filetypes than +detectable ones. Whether "supported" should eventually mean every configured filetype (which would +require adding detection rules, test projects, and provisioning for the currently inactive catalog +entries) is a product-policy call, not an implementation fact. + +## Non-goals + +- Do not run a Cartesian product of every language, server, command, and option. Only configured + language/server relationships are meaningful. +- Do not require a server to implement an optional LSP capability. +- Do not put language-specific parsing or source-code knowledge into production `lsp-cli` code. +- Do not make tracked playground files writable test state. +- Do not add a Rust dependency without explicit permission. The existing `tempfile`, `serde`, + `serde_json`, and `serde_yaml` dependencies are sufficient for the harness. + +## Local dev environment + +The real-server E2E targets (`test-real-server-e2e`, `test-real-server-smoke-e2e`, +`test-server-provisioning-e2e`) need `go`, `java`, `node`/`npm`, and `dotnet` on `PATH`, matching +what CI installs in `.github/workflows/ci.yml` and `e2e.yml`. Run `make download-dev-env` to fetch +project-local copies into `.env/` (not a system-wide install), then `source activate.sh` from the +repo root to put them on `PATH` for the current shell. + +## Test projects (playgrounds) + +Real-server cases use small, committed multi-file projects under `playground/`; see +`playground/README.md` for the per-project layout and manual reproduction commands. Where the +language permits, every source-language project should contain: + +- one stable workspace symbol; +- functions and methods; +- a declaration separated from its definition; +- references from more than one file; +- a caller and callee chain; +- types and fields; +- a file whose formatting can be made deterministically incorrect; +- a deterministic source mutation that produces one diagnostic. + +Prefer equivalent domain concepts and symbol names across projects when natural. Do not force a +language into constructs it does not support merely to make fixtures textually identical. + +Filetype IDs that describe workspace/module metadata rather than a source language (e.g. +`gomod`/`gowork`) use minimal detection-only fixtures: they cover detection, file listing, server +selection, initialization, and lifecycle, but cannot independently provide meaningful symbol or +call-hierarchy assertions. + +Tests securely create randomized sandboxes with the `tempfile` crate under the user's +`XDG_RUNTIME_DIR`, `XDG_CACHE_HOME`, or `$HOME/.cache`, in that order, then copy a project there +before formatting it or introducing diagnostics. Real test state must not use the ambient system +`/tmp`. This keeps the repository clean, avoids uncontrolled parent root markers, keeps Unix socket +paths short, and allows safe parallel execution. + +Committed playgrounds (vs. generating every project during test setup) trade "each language +fixture must evolve with its toolchain and server ecosystem" for "humans can reproduce failures +with the same projects" — keep using committed projects for the baseline fixtures. + +## Coverage model + +`lsp-cli`'s canonical top-level subcommands are factored by responsibility instead of multiplying +all commands by all language/server pairs: + +| Scope | Subcommands | Required coverage | +|---|---|---| +| Global CLI | `commands`, `languages`, `servers`, `completion`, `agent-skill`, `update` | Focused binary-level cases, independent of real servers | +| Detection and filesystem | `detect`, `list-files` | Every detectable filetype ID | +| LSP requests | `server-capabilities`, `diagnostics`, `format`, `grep`, `list-symbols`, `list-functions`, `references`, `callers`, `callees`, `definition`, `declaration`, `build-index` | Every compatible language/server pair, capability-aware | +| Process lifecycle | `run`, `daemon`, `stop`, `stop-all` | Every distinct relevant server where applicable, with grouped lifecycle scenarios | + +### Capability-aware expectations + +For each compatible pair, first record or inspect the server's initialized capabilities. A command +passes if it either: + +- succeeds and returns the expected semantic result; or +- returns the documented, user-facing unsupported-capability error when the server does not + advertise the required capability. + +Capability advertisement is not enough by itself: when a server advertises a capability, exercise +the corresponding command and assert its behavior. + +### Option coverage + +Distribute option variants across the matrix using explicit cases; do not create another full +cross-product. Cover at least: + +- automatic selection, `--lang`, and `--lsp`; +- text and `--json` output; +- direct execution, `--detach`, and `--no-detach`; +- `--limit`, `--files-with-matches`, and `--full`; +- `--wait-for-index`; +- `format`, `format --check`, and `format --stdout`; +- successful operations, unsupported capabilities, missing executables, server crashes, malformed + replies, and timeouts; +- `--download` once per supported installation mechanism, rather than redundantly for every query. + +JSON assertions deserialize and compare stable semantic fields. Text assertions avoid full +snapshots when server versions can legitimately change ordering, signatures, or detail. + +## Harness & manifest layout + +A normal Cargo integration-test crate invokes the built binary through `CARGO_BIN_EXE_lsp-cli`: + +```text +tests/ + e2e.rs + e2e/ + harness.rs + manifest.rs / manifest/ + catalog.rs + queries.rs + lifecycle.rs + update.rs + cases/ + suite.yaml + .yaml +``` + +Keep every Rust file under 600 lines; move repeated process setup and assertions into helpers as +soon as a second test needs them. + +`harness.rs` provides methods for actions on an E2E context: creating an isolated home, +configuration root, runtime root, and workspace copy; constructing an `lsp-cli` process with +deterministic environment variables; running a command with a deadline and capturing +stdout/stderr/status; parsing JSON output; introducing a formatting or diagnostic mutation; finding +and terminating remaining child processes; stopping daemons and reporting their runtime state after +failure. + +Each test process sets at least: + +- `HOME` to an isolated temporary home; +- `XDG_CONFIG_HOME` to an isolated configuration directory; +- `XDG_RUNTIME_DIR` to an isolated daemon directory; +- `LSP_DATA` to the pinned repository submodule; +- `PATH` to the explicitly provisioned toolchain/server environment. + +Do not rely on a developer's user configuration, downloaded server cache, daemon sockets, current +shell, or ambient server versions. + +`tests/e2e/cases/suite.yaml` owns global command coverage and assigns every canonical command a +coverage strategy; each `tests/e2e/cases/.yaml` owns one project and its configured +server behavior. Case YAML contains only E2E-specific behavior — it does not duplicate each +server's `filetypes` list, which is derived from `data/lsp-cli.yaml`. Pair entries are sparse E2E +behavior overlays keyed by the LSP YAML filename stem as their stable config ID; bare compatibility +entries are rejected because compatibility belongs to `data`. + +A validation test fails when: + +- a detectable filetype lacks a project; +- a configured E2E case names an incompatible or missing data config; +- an exclusion lacks a reason; +- two cases select the same user-visible server ambiguously; +- a new top-level subcommand has no assigned coverage class. + +Every relevant server has exactly one preferred pair per source language, tagged with a `smoke` +disposition (a generic query suite, or an exclusion with a mandatory reviewed reason) and exactly +one lifecycle-owner pair, tagged with a `lifecycle` disposition (grouped daemon scenarios, or a +reviewed exclusion). Direct `run` may be excluded independently when only detached operation is +reliable. This keeps the chosen project stable when a shared server gains another filetype, without +repeating process tests for every compatible pair. Source-language query profiles declare shared +semantic terms, expected symbols, and format paths; pair entries keep only deadlines and narrowly +scoped known-result exceptions (see "Real-server exceptions" below). Language-specific +prerequisites and expectations belong in YAML, not in the Rust runner. + +### Extending the manifest + +To cover an existing detectable filetype, add its small project under `playground/` and one case +file named after the filetype ID. Compatible servers are discovered from `data/lsp/*.yaml`; add a +`pairs` entry only when the E2E suite has executable behavior or a reviewed exclusion for that +pair. To introduce a genuinely new filetype or server, first add its YAML config and commit it in +the `data` submodule, then update the submodule revision, then add a project/case file here and any +pair-specific E2E behavior. + +The query runner obtains raw initialized capabilities through `server-capabilities --json`, then +executes every LSP query command. Advertised capabilities require a successful semantic response; +missing capabilities require the command's user-facing unsupported error. + +## Real-server exceptions + +In `tests/e2e/manifest/query_case.rs`, a `smoke` pair can be `status: queries`, and each query case +carries an optional `exceptions` list. Each entry names a `command` (one of the real-server query +kinds — `grep`, `references`, `callers`, `callees`, `build-index`, `format`, etc.), an `outcome` +(`failure` or `empty-matches`), an optional expected stderr `message`, and a mandatory `reason`. + +At runtime (`tests/e2e/real_servers.rs`), if a query has a matching exception, the harness skips +the normal "must succeed with real matches" assertion and instead asserts the *documented* deviant +behavior: + +- `failure`: the command must exit non-zero and stderr must contain `message`. +- `empty-matches`: the command must succeed but return an empty `matches` array. + +`exceptions` is not error-tolerance or flakiness suppression — it's a positive assertion of each +server's known, reproducible protocol quirk, with the `reason` pinned in the YAML so the deviation +is self-documenting and any regression still fails loudly. + +### Known root causes + +- **No background-indexing-completion signal** (`build-index` → `failure`, message + "background-work progress"). Several servers advertise `$/progress`/work-done tokens but never + send a terminal "index build finished" notification the CLI can wait on. This is the single most + common exception across the suite. +- **Workspace-symbol search (`grep`) racing indexing.** A server with no synchronous "ready" signal + can return empty `matches` for `workspace/symbol` issued immediately after startup. + `run_workspace_symbol_query` (`src/commands/symbol_query.rs`) primes the server by opening a + workspace document and retries with a short poll when the first `workspace/symbol` call is empty + or errors; servers that still race past that retry keep an `empty-matches` exception. +- **Call-hierarchy has no edges for the fixture.** A server reports empty `callees`/`callers` when + the fixture doesn't happen to exercise a real call edge for the queried symbol, or (for a couple + of servers) as a genuine analysis limitation independent of the fixture. +- **Server-specific formatting/output bugs**, e.g. a `format` edit whose range falls outside the + requested file — tolerated as a `failure` exception with a matched message. + +### Servers excluded entirely (`status: excluded`) + +- `denols` (Deno LSP) rejects the standard shutdown request because it requires non-null params. +- `roslyn_ls` (cs) and `lua_ls` (lua) have lifecycle-level incompatibilities: no smoke queries at + all, or no clean exit after direct shutdown. +- Several Python servers fail to launch/initialize correctly in the isolated harness: `pylsp` + (Mason launcher can't import the module), `pyre` (same), `pyrefly` (initializes but returns no + workspace/document symbols). + +Real LSP servers deviate from the LSP spec's strict guarantees in ways that are reproducible but +server-specific. Rather than weakening assertions globally, the suite encodes each deviation +explicitly per server/command so real regressions still fail loudly, while known quirks are pinned +and self-documented via `reason`. + +## Provisioning + +Do not install LSP servers separately. Every real-server case passes `--download`, letting the +production Mason integration select the current registry package, install it inside the case's +isolated home, and return the resolved executable. This applies uniformly to direct archives and +npm, PyPI, Cargo, Go, NuGet, GitHub, or generic package sources supported by the downloader. + +Language SDKs and package-manager runtimes remain explicit host prerequisites, with their resolver +commands kept in the manifest so a missing prerequisite produces a case-specific error rather than +a silent skip. A CI lane either provisions the server or reports the pair as an explicit, reviewed +exclusion — a required pair is never silently skipped because its executable is absent. + +Downloading these external test tools requires product-owner approval under the repository's +dependency policy. Latest-Mason server downloads and official free SDK/runtime provisioning are +approved; they do not become Rust package dependencies, but remain operational dependencies with +maintenance, security, licensing, storage, and network consequences. Proprietary and +unsupported-platform prerequisites are excluded. + +Always using Mason latest detects upstream compatibility changes immediately and avoids maintaining +a second installation path. The tradeoff is a nondeterministic merge gate: a registry or server +release can break an unchanged commit, and reproducing the failure depends on the recorded source +ID remaining available upstream. + +Run the complete downloadable-server inventory manually with: + +```sh +make test-server-provisioning-e2e +``` + +Set `E2E_SERVER=` to diagnose one downloadable server. The test copies the server's +owner project into an isolated context, stages its declared host programs, invokes `detect` with +`--download`, and verifies that exactly one selected command resolves inside the isolated home. It +does not initialize the server or substitute for the later language/server behavior matrix. + +## Running and reproducing tests + +From the repository root: + +```sh +# smoke: one preferred server per source language, all relevant subcommands +E2E_CASE=python/pyright make test-real-server-smoke-e2e + +# combined query + lifecycle target for one pair +E2E_CASE=java/jdtls make test-real-server-e2e + +# provisioning only, for one downloadable server +E2E_SERVER=pyright make test-server-provisioning-e2e +``` + +`E2E_CASE=/` (or `E2E_CASES=` with a comma-separated list, used by sharded CI) +selects one configured executable or explicitly excluded case without changing the all-cases +default. Selecting a compatible pair with no E2E behavior fails with a clear error instead of +silently running no tests. + +These commands download external tools and require the host programs declared by the manifest, and +they use the current Mason registry — compare the resulting source ID with an earlier run before +concluding that local behavior has changed. + +The manual **End-to-end compatibility** GitHub Actions workflow can select `language`, `server`, or +`installation-family` (the value is respectively a case language ID, an LSP config ID, or one of +`cargo generic github golang npm nuget pypi`): + +```sh +gh workflow run e2e.yml -f selector=server -f value=pyright +``` + +Use `selector=all` with an empty value for the complete matrix. The generated workflow summary +shows all selected pairs, including reviewed exclusions, before runnable pairs are sharded. + +## CI plan + +**Pull requests** run: all existing unit tests and checks via `make test`; global and detection E2E +tests; one preferred current-Mason server per source language; every relevant subcommand across +that smoke matrix; manifest/data consistency checks. + +**Nightly** runs all compatible pairs, sharded by language and server installation family, with +fail-fast disabled so one broken server doesn't hide the rest of the compatibility report. The +planner resolves installation families from the current Mason registry rather than copying that +registry metadata into `cases/`. Suite-level smoke, lifecycle, and provisioning deadlines provide +common defaults; cases only declare intentional overrides. + +Jobs never share homes, daemon runtime directories, or mutable workspaces. CI may cache immutable +download transport data, but each case retains isolated runtime state and never substitutes a +separately installed server for `--download`. Every real-server case tears down its isolated home +and temporary roots (Mason packages, Go module/build caches, other server download state) before +the next case starts; only immutable Rust build artifacts are shared by CI. + +Split CI (fast PR smoke + exhaustive nightly) trades "a regression affecting a non-preferred server +may surface the following night rather than on the originating PR" for much lower latency, cost, +rate-limit exposure, and upstream-flake risk on every PR — this is the deliberate default over +running everything on every PR. + +## Triage a CI failure + +Pairs use `/`, e.g. `python/pyright`. The server component is the +filename stem under `data/lsp/`, not necessarily the executable or display name. + +1. Open the workflow summary and find the pair's executable or excluded classification. +2. Open the failed `/` matrix job. +3. Read the `E2E failed case IDs` footer, which lists the specific failed pairs collected from the + diagnostics above it. If the footer says the IDs are unavailable, failure occurred before a case + emitted its labelled diagnostic — start with the planner, build, or test-runner error + immediately above the footer. + +### Identify the upstream version + +A failed case retains this block before deleting its isolated home: + +```text +server package source IDs: +pkg:npm/pyright@1.1.409 +server command line: +... +server capabilities: +... +server stderr summary: +... +cleanup state: +... +``` + +Treat each complete `pkg:/@` source ID as the authoritative +package identity — preserve the whole value, since versions and package names can contain +prefixes, scopes, or backend-specific suffixes. The executable name and `initialize` response +version may describe a product differently and are supporting evidence, not replacements for the +source ID. + +`` normally means provisioning failed before a +receipt was written; inspect the preceding download/install error rather than inferring a version +from an older run. Outside the isolated suite, successful downloads store JSON receipts under +`~/.local/share/lsp-cli/receipts/` (same `source_id` field meaning). E2E case homes and their +receipts are intentionally removed after every case, including failures, so use the retained +diagnostic rather than a path printed earlier in the log. + +Successful E2E cases print no retained failure context. The suite follows Mason latest, so a later +rerun may resolve a different source ID — compare IDs from available failing runs and record the ID +in an issue when exact upstream identity matters; the suite does not promise the registry will +retain an older version for reproduction. + +### Classify the failure + +Check evidence in this order: + +1. **Planner or manifest:** an unknown selector, missing registry package, or validation failure + happened before a server case ran. +2. **Provisioning or network:** no completed receipt, package-manager output, HTTP failure, or an + absent host program points to installation rather than LSP behavior — use + `make test-server-provisioning-e2e`. +3. **Startup or shutdown:** use the retained command line and server stderr. SDK incompatibility, + launcher failure, crash, and failure to exit are distinct from query-result drift. +4. **Protocol or capability:** compare the retained capabilities with the command exercised by the + case. An unadvertised optional capability passes only when lsp-cli returns its expected + user-facing unsupported error. +5. **Semantic result:** compare stable names and the pair's reviewed exceptions (see "Real-server + exceptions" above). Do not weaken an expectation until the same source ID reproduces the + behavior or an upstream change is confirmed. +6. **Cleanup:** inspect sandbox and runtime roots independently of the primary failure — a + successful query with a retained root is still a cleanup regression. + +## Failure policy + +Classify failures as: + +1. `lsp-cli` regression; +2. playground/manifest drift; +3. provisioning or network failure; +4. upstream server behavior change; +5. known server limitation; +6. unsupported LSP capability with the expected user-facing response. + +Only category 6 is an immediate passing outcome. Known limitations must be explicit manifest +entries and, when they concern protocol or server behavior, documented in `docs/GOTCHAS.md`. Do not add +unbounded retries — a retry may cover an identified transient installation/network step, but must +not conceal query or protocol failures. + +If a hard-to-debug defect is fixed, add a focused regression test in addition to the broad matrix, +and consider whether a type invariant, runtime check, clearer trace, or state-dump helper can make +that class of defect easier to diagnose next time. + +## Definition of done (for a new language/server) + +- The language has a committed project or a justified metadata-only fixture. +- Every compatible supported language/server pair has an executable manifest entry. +- Every top-level subcommand exercised has binary-level E2E coverage in its appropriate scope. +- Advertised capabilities are exercised; unsupported capabilities have asserted user-facing + behavior. +- Direct, detached, stop, and stop-all lifecycle paths are covered where applicable. +- Formatting and diagnostics tests cannot dirty tracked files. +- Required servers are provisioned through `--download`, and resolved source IDs are retained on + failure. +- Known protocol/server deviations are recorded in `docs/GOTCHAS.md`. +- No required case is silently skipped.