Skip to content

test run reports passing results against stale assemblies when the project has compilation errors #51

Description

@Niaobu

Summary

unityctl test run executes and reports results even when the project currently has unresolved C# compilation errors. In that state Unity runs the test suite against the last successfully compiled assemblies (stale DLLs in Library/ScriptAssemblies), so the command can return a green/passing result — or a result with a different test count — that does not reflect the code on disk. There is no warning that the code under test never compiled.

This is a correctness/safety problem: a passing test run after an edit that doesn't compile is silently meaningless, and an agent or human reading the result will believe their change is verified when it was never built.

Expected behavior

test run should refuse to run (or at minimum loudly warn) when compilation has failed, consistent with how play enter and record start already behave:

Cannot enter play mode - compilation errors exist. Fix the errors and try again.

Proposed: return a COMPILATION_ERROR result (non-zero CLI exit) such as
Cannot run tests - compilation errors exist. Fix the errors and try again.

Actual behavior

Tests run against stale assemblies and the command reports success (or a stale test list), with no indication that the current sources failed to compile.

Repro

  1. Have a project with at least one edit-mode test that passes.
  2. unityctl asset refresh → confirm a clean compile.
  3. Introduce a compile error in any C# script in an assembly that the tests depend on (e.g. add this wont compile; to a runtime script).
  4. unityctl asset refresh → reports the compilation error (correct).
  5. unityctl test runobserve it still reports a passing/green result (running against the previously compiled assemblies) instead of failing on the pre-existing compile error.

Root cause

HandleTestRun (UnityCtl.UnityPackage/Editor/UnityCtlClient.cs, ~L1054) builds a TestRunnerApi filter and calls Execute(...) immediately. It never checks EditorUtility.scriptCompilationFailed (or EditorApplication.isCompiling) before running.

The Bridge has no precondition gate for test.run either — its CommandConfig (UnityCtl.Bridge/BridgeEndpoints.cs, ~L66) only sets a timeout and CompletionEvent = TestFinished. By contrast, the play-enter path checks compilation state and short-circuits with COMPILATION_ERROR (BridgeEndpoints.cs ~L523-534), and record start does the same (~L756). EditorUtility.scriptCompilationFailed is already used elsewhere in the plugin (UnityCtlClient.cs ~L1503), so the signal is readily available.

Suggested fix

Guard test run on EditorUtility.scriptCompilationFailed before executing — either in HandleTestRun (return an error result) or as a Bridge precondition mirroring the play-enter/record-start gate — so a stale assembly can never produce a misleading green result. Reuse the existing COMPILATION_ERROR shape and message style for consistency.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions