Add a JSON-RPC server mode to F# Interactive

FSharp.slnx

@@ -27,6 +27,9 @@
     <Project Path="tests/FSharp.Compiler.ComponentTests/FSharp.Compiler.ComponentTests.fsproj">
       <BuildType Solution="Proto|*" Project="Debug" />
     </Project>
+    <Project Path="tests/FSharp.Compiler.Interactive.Server.Tests/FSharp.Compiler.Interactive.Server.Tests.fsproj">
+      <BuildType Solution="Proto|*" Project="Debug" />
+    </Project>
     <Project Path="tests/FSharp.Compiler.Private.Scripting.UnitTests/FSharp.Compiler.Private.Scripting.UnitTests.fsproj">
       <BuildType Solution="Proto|*" Project="Debug" />
     </Project>

VisualFSharp.slnx

@@ -40,6 +40,9 @@
     <Project Path="tests/FSharp.Compiler.ComponentTests/FSharp.Compiler.ComponentTests.fsproj">
       <BuildType Solution="Proto|*" Project="Debug" />
     </Project>
+    <Project Path="tests/FSharp.Compiler.Interactive.Server.Tests/FSharp.Compiler.Interactive.Server.Tests.fsproj">
+      <BuildType Solution="Proto|*" Project="Debug" />
+    </Project>
     <Project Path="tests/FSharp.Compiler.LanguageServer.Tests/FSharp.Compiler.LanguageServer.Tests.fsproj">
       <BuildType Solution="Proto|*" Project="Debug" />
     </Project>

buildtools/AssemblyCheck/SkipVerifyEmbeddedPdb.txt

@@ -1,6 +1,7 @@
 FSharp.Build.UnitTests.dll
 FSharp.Benchmarks.Common.dll
 FSharp.Compiler.Benchmarks.dll
+FSharp.Compiler.Interactive.Server.Tests.dll
 FSharp.Compiler.ComponentTests.dll
 FSharp.Test.Utilities.dll
 FSharp.Compiler.LanguageServer.Tests.dll

docs/release-notes/.FSharp.Compiler.Service/11.0.100.md

@@ -1,8 +1,13 @@
+### Added
+
+* F# Interactive gains a JSON-RPC server mode, `--fsi-server-jsonrpc:<pipe name>`, in which a host submits interactions over a named pipe and receives structured results — diagnostics with positions, escaping exceptions, the values each interaction bound, and the session's own process id — instead of recovering them by looking for a `SERVER-PROMPT>` marker in the output text. Program output continues to flow through the redirected console streams. The pipe admits only the user running the session; `--fsi-server-client-pid:<pid>` names the host process whose exit ends the session. `FsiEvaluationSession` exposes both options as `JsonRpcServerPipeName` and `JsonRpcClientProcessId`. The mode is part of the .NET fsi only. ([PR #20396](https://github.com/dotnet/fsharp/pull/20396))
+
 ### Fixed
 
 * Fix internal error "no 'value__' field found for enumeration type" when an `enum` constraint is checked against an enum declared in the same recursive group (`module rec` or an `and` group). The constraint is now solved once the representations of the group are established. ([Issue #14580](https://github.com/dotnet/fsharp/issues/14580), [PR #20454](https://github.com/dotnet/fsharp/pull/20454))
 * Better diagnostic for a bare `enum` constraint: `'T : enum` now reports FS0699 "An 'enum' constraint must be of the form 'enum<type>'" instead of "Unexpected identifier: 'enum (4)'", and parser messages for invalid constraints no longer leak internal ` (2)`/` (3)`/` (4)` markers. ([Issue #14580](https://github.com/dotnet/fsharp/issues/14580), [PR #20454](https://github.com/dotnet/fsharp/pull/20454))
 * Fix FSI `--multiemit+ --debug-` unnecessarily generating and loading portable PDB data for every submission. ([Issue #17306](https://github.com/dotnet/fsharp/issues/17306), [PR #20394](https://github.com/dotnet/fsharp/pull/20394), [downstream report](https://github.com/bryanedds/Nu/issues/1090))
+* `FsiEvaluationSession.EvalInteraction` and `EvalInteractionNonThrowing` evaluate every `;;`-separated interaction in the text they are given, as their documentation has always said, instead of silently discarding everything after the first. ([PR #20396](https://github.com/dotnet/fsharp/pull/20396))
 * Fix reference-assembly MVID collisions when a public member is renamed in an early file of a large (>~32 file) project under `--optimize-`. The per-file signature hashes were folded with a left-shift-by-one combiner over a 32-bit `Hash`, which truncated the contribution of any file more than ~32 positions from the end of the compile order, so MSBuild's `CopyRefAssembly` saw an unchanged MVID and kept a stale reference assembly (surfacing as `FS0039` downstream). The combiner is now an FNV-1a multiply mix over a 64-bit `Hash`. ([Issue #20389](https://github.com/dotnet/fsharp/issues/20389))
 * Fix internal error FS0192 "Iterate2D" when a `[<ReflectedDefinition(true)>]` parameter auto-quotes an argument that captures a not-yet-generalized use of an inferred generically-recursive function. The auto-quoted (`Expr.WithValue`) copy now keeps a fresh link to the recursive-value use so it receives the same inferred type arguments as the executable expression at the letrec point. ([Issue #20379](https://github.com/dotnet/fsharp/issues/20379))
 * Fix `StackOverflowException` when checking a long `seq { ... }` body. ([PR #20480](https://github.com/dotnet/fsharp/pull/20480))

eng/Packages.props

@@ -116,6 +116,10 @@
     <PackageVersion Include="System.Reflection.Metadata" Version="$(SystemReflectionMetadataVersion)" />
     <PackageVersion Include="System.Runtime.CompilerServices.Unsafe" Version="$(SystemRuntimeCompilerServicesUnsafeVersion)" />
     <PackageVersion Include="System.Security.Cryptography.Xml" Version="$(SystemSecurityCryptographyXmlVersion)" />
+    <!-- The JSON-RPC transport of the F# Interactive server mode and of the interactive window.
+         Pinned to the version Roslyn's packages already force into vsintegration, so that the two
+         agree and a Roslyn bump does not turn into a downgrade conflict. -->
+    <PackageVersion Include="StreamJsonRpc" Version="2.26.10" />
     <PackageVersion Include="xunit.v3.assert" Version="$(XunitVersion)" />
     <PackageVersion Include="xunit.v3.common" Version="$(XunitVersion)" />
     <PackageVersion Include="xunit.v3.extensibility.core" Version="$(XunitVersion)" />

eng/Signing.props

@@ -3,6 +3,11 @@
   <ItemGroup>
     <FileSignInfo Include="Nerdbank.Streams.dll" CertificateName="None" />
     <FileSignInfo Include="Newtonsoft.Json.dll" CertificateName="None" />
+    <!-- StreamJsonRpc's own serialization stack, shipped for real by fsi's JSON-RPC server mode. -->
+    <FileSignInfo Include="MessagePack.dll" CertificateName="None" />
+    <FileSignInfo Include="MessagePack.Annotations.dll" CertificateName="None" />
+    <FileSignInfo Include="Nerdbank.MessagePack.dll" CertificateName="None" />
+    <FileSignInfo Include="PolyType.dll" CertificateName="None" />
   </ItemGroup>
 
   <PropertyGroup>

eng/tests/TestSplit.fsx

@@ -48,6 +48,7 @@ let otherProjects =
         "tests/FSharp.Core.UnitTests/FSharp.Core.UnitTests.fsproj",                          2,     "all"
         "tests/FSharp.Compiler.Service.Tests/FSharp.Compiler.Service.Tests.fsproj",          2,     "all"
         "tests/FSharp.Compiler.Private.Scripting.UnitTests/FSharp.Compiler.Private.Scripting.UnitTests.fsproj", 2, "all"
+        "tests/FSharp.Compiler.Interactive.Server.Tests/FSharp.Compiler.Interactive.Server.Tests.fsproj",   2,     "coreclr"
         "tests/fsharp/FSharpSuite.Tests.fsproj",                                             3,     "desktop"
     ]
 

src/Compiler/Interactive/fsi.fs

@@ -988,11 +988,19 @@ type internal FsiCommandLineOptions(fsi: FsiEvaluationSessionHostConfig, argv: s
     let mutable fsiServerOutputCodePage = None
     let mutable fsiLCID = None
 
+    let mutable fsiServerJsonRpcPipe = ""
+    let mutable fsiServerClientProcessId = None
+
     // internal options
     let mutable probeToSeeIfConsoleWorks = true
     let mutable peekAheadOnConsoleToPermitTyping = true
 
-    let isInteractiveServer () = fsiServerName <> ""
+    let isJsonRpcServer () = fsiServerJsonRpcPipe <> ""
+
+    // Neither server mode has a user at a console, so neither uses the console reader.
+    let isInteractiveServer () =
+        fsiServerName <> "" || isJsonRpcServer ()
+
     let recordExplicitArg arg = explicitArgs <- explicitArgs @ [ arg ]
 
     let executableFileNameWithoutExtension =
@@ -1074,6 +1082,8 @@ type internal FsiCommandLineOptions(fsi: FsiEvaluationSessionHostConfig, argv: s
                 [ // Make internal fsi-server* options. Do not print in the help. They are used by VFSI.
                     CompilerOption("fsi-server-report-references", "", OptionString(fun s -> writeReferencesAndExit <- Some s), None, None)
                     CompilerOption("fsi-server", "", OptionString(fun s -> fsiServerName <- s), None, None) // "FSI server mode on given named channel");
+                    CompilerOption("fsi-server-jsonrpc", "", OptionString(fun s -> fsiServerJsonRpcPipe <- s), None, None) // "FSI server mode speaking JSON-RPC over the given named pipe"
+                    CompilerOption("fsi-server-client-pid", "", OptionInt(fun n -> fsiServerClientProcessId <- Some n), None, None) // "Process id of the host; the JSON-RPC server exits when it does"
                     CompilerOption("fsi-server-input-codepage", "", OptionInt(fun n -> fsiServerInputCodePage <- Some(n)), None, None) // " Set the input codepage for the console");
                     CompilerOption("fsi-server-output-codepage", "", OptionInt(fun n -> fsiServerOutputCodePage <- Some(n)), None, None) // " Set the output codepage for the console");
                     CompilerOption(
@@ -1385,6 +1395,16 @@ type internal FsiCommandLineOptions(fsi: FsiEvaluationSessionHostConfig, argv: s
 
     member _.IsInteractiveServer = isInteractiveServer ()
 
+    member _.IsJsonRpcServer = isJsonRpcServer ()
+
+    member _.JsonRpcServerPipeName =
+        if isJsonRpcServer () then
+            Some fsiServerJsonRpcPipe
+        else
+            None
+
+    member _.JsonRpcClientProcessId = fsiServerClientProcessId
+
     member _.ProbeToSeeIfConsoleWorks = probeToSeeIfConsoleWorks
 
     member _.EnableConsoleKeyProcessing = enableConsoleKeyProcessing
@@ -1477,7 +1497,9 @@ type internal FsiConsolePrompt(fsiOptions: FsiCommandLineOptions, fsiConsoleOutp
     // A prompt gets "printed ahead" at start up. Tells users to start type while initialisation completes.
     // A prompt can be skipped by "silent directives", e.g. ones sent to FSI by VS.
     let mutable dropPrompt = 0
-    let mutable showPrompt = true
+
+    // A JSON-RPC host learns an interaction finished from the response to its request.
+    let mutable showPrompt = not fsiOptions.IsJsonRpcServer
 
     // NOTE: SERVER-PROMPT is not user displayed, rather it's a prefix that code elsewhere
     // uses to identify the prompt, see service\FsPkgs\FSharp.VS.FSI\fsiSessionToolWindow.fs
@@ -1520,17 +1542,13 @@ type internal FsiConsoleInput
 
     let consoleOpt =
         // The "console.fs" code does a limited form of "TAB-completion".
-        // Currently, it turns on if it looks like we have a console.
-        if fsiOptions.EnableConsoleKeyProcessing then
+        // Currently, it turns on if it looks like we have a console. A session driven by a host has
+        // no user at a console, whatever the probe would say.
+        if fsiOptions.EnableConsoleKeyProcessing && not fsiOptions.IsInteractiveServer then
             fsi.GetOptionalConsoleReadLine(fsiOptions.ProbeToSeeIfConsoleWorks)
         else
             None
 
-    // When VFSI is running, there should be no "console", and in particular the console.fs readline code should not to run.
-    do
-        if fsiOptions.IsInteractiveServer then
-            assert consoleOpt.IsNone
-
     /// This threading event gets set after the first-line-reader has finished its work
     let consoleReaderStartupDone = new ManualResetEvent(false)
 
@@ -4413,11 +4431,37 @@ type FsiInteractionProcessor
         let tokenizer =
             fsiStdinLexerProvider.CreateBufferLexer(scriptFileName, lexbuf, diagnosticsLogger)
 
-        currState
-        |> InteractiveCatch diagnosticsLogger (fun istate ->
-            let expr = ParseInteraction tcConfigB.diagnosticsOptions tokenizer
-            ExecuteParsedInteractionOnMainThread(ctok, diagnosticsLogger, expr, istate, cancellationToken))
-        |> commitResult
+        // The text may hold several interactions, as standard input would. Each one that completes
+        // is committed before the next is parsed, so that a failure later in the text keeps what ran
+        // before it; the value reported is that of the last interaction that produced one.
+        let rec run istate lastValue =
+            let errorsBefore = diagnosticsLogger.ErrorCount
+
+            let istate, status =
+                istate
+                |> InteractiveCatch diagnosticsLogger (fun istate ->
+                    match ParseInteraction tcConfigB.diagnosticsOptions tokenizer with
+                    | Some(ParsedScriptInteraction.Definitions([], _)) -> istate, Completed lastValue
+                    | expr -> ExecuteParsedInteractionOnMainThread(ctok, diagnosticsLogger, expr, istate, cancellationToken))
+
+            let status =
+                match status with
+                | Completed value -> Completed(Option.orElse lastValue value)
+                | status -> status
+
+            match status with
+            | Completed value when
+                diagnosticsLogger.ErrorCount = errorsBefore
+                && not tokenizer.LexBuffer.IsPastEndOfStream
+                ->
+                if cancellationToken.IsCancellationRequested then
+                    istate, CtrlC
+                else
+                    setCurrState istate
+                    run istate value
+            | _ -> istate, status
+
+        run currState None |> commitResult
 
     member this.EvalScript(ctok, scriptPath, diagnosticsLogger) =
         // Todo: this runs the script as expected but errors are displayed one line to far in debugger
@@ -4959,6 +5003,10 @@ type FsiEvaluationSession
     /// A host calls this to get the active language ID if provided by fsi-server-lcid
     member _.LCID = fsiOptions.FsiLCID
 
+    member _.JsonRpcServerPipeName = fsiOptions.JsonRpcServerPipeName
+
+    member _.JsonRpcClientProcessId = fsiOptions.JsonRpcClientProcessId
+
     /// A host calls this to report an unhandled exception in a standard way, e.g. an exception on the GUI thread gets printed to stderr
     member x.ReportUnhandledException exn = x.ReportUnhandledExceptionSafe true exn
 
@@ -5108,7 +5156,9 @@ type FsiEvaluationSession
         // We later switch to doing interaction-by-interaction processing on the "event loop" thread
         let ctokRun = AssumeCompilationThreadWithoutEvidence()
 
-        if fsiOptions.IsInteractiveServer then
+        // The JSON-RPC server carries interrupts on its own connection and is started by the
+        // process entry point.
+        if fsiOptions.IsInteractiveServer && not fsiOptions.IsJsonRpcServer then
             SpawnInteractiveServer(fsi, fsiOptions, fsiConsoleOutput)
 
         use _ = UseBuildPhase BuildPhase.Interactive
@@ -5127,7 +5177,10 @@ type FsiEvaluationSession
                 | _ -> ())
 
             fsiInteractionProcessor.LoadInitialFiles(ctokRun, diagnosticsLogger)
-            fsiInteractionProcessor.StartStdinReadAndProcessThread(tcConfigB.diagnosticsOptions, diagnosticsLogger)
+
+            // Interactions arrive on the control channel, leaving stdin to the script.
+            if not fsiOptions.IsJsonRpcServer then
+                fsiInteractionProcessor.StartStdinReadAndProcessThread(tcConfigB.diagnosticsOptions, diagnosticsLogger)
 
             DriveFsiEventLoop(fsi, fsiInterruptController, fsiConsoleOutput)
 

src/Compiler/Interactive/fsi.fsi

@@ -310,6 +310,13 @@ type FsiEvaluationSession =
     /// A host calls this to get the active language ID if provided by fsi-server-lcid
     member LCID: int option
 
+    /// The named pipe requested with `--fsi-server-jsonrpc`, when a host is to drive the session over
+    /// JSON-RPC instead of standard input.
+    member JsonRpcServerPipeName: string option
+
+    /// The host process named with `--fsi-server-client-pid`, whose exit ends the session.
+    member JsonRpcClientProcessId: int option
+
     /// A host calls this to report an unhandled exception in a standard way, e.g. an exception on the GUI thread gets printed to stderr
     member ReportUnhandledException: exn: exn -> unit
 

src/Microsoft.FSharp.Compiler/Microsoft.FSharp.Compiler.fsproj

@@ -26,6 +26,18 @@
     <NuspecProperty Include="configuration=$(Configuration)" />
   </ItemGroup>
 
+  <!-- fsi's JSON-RPC server ships StreamJsonRpc and its dependency closure next to fsi.dll. A source-only
+       build has no server mode (see src/fsi/fsi.targets), so it ships none of these. The nuspec takes
+       them as one token: an item's Include cannot combine `name=` with an item list, so the list is
+       kept in a property and expanded into <file> entries here. -->
+  <PropertyGroup Condition="'$(DotNetBuildSourceOnly)' != 'true'">
+    <FsiJsonRpcServerAssemblies>MessagePack.dll;MessagePack.Annotations.dll;Microsoft.VisualStudio.Threading.dll;Microsoft.VisualStudio.Validation.dll;Nerdbank.MessagePack.dll;Nerdbank.Streams.dll;Newtonsoft.Json.dll;PolyType.dll;StreamJsonRpc.dll</FsiJsonRpcServerAssemblies>
+    <FsiJsonRpcServerFiles>$([System.Text.RegularExpressions.Regex]::Replace('$(FsiJsonRpcServerAssemblies)', '([^;]+);?', '&lt;file src=&quot;fsi\$(Configuration)\$(FSharpNetCoreProductTargetFramework)\$1&quot; target=&quot;lib\$(FSharpNetCoreProductTargetFramework)&quot; /&gt; '))</FsiJsonRpcServerFiles>
+  </PropertyGroup>
+
+  <ItemGroup>
+    <NuspecProperty Include="fsiJsonRpcServerFiles=$(FsiJsonRpcServerFiles)" />
+  </ItemGroup>
   <ItemGroup Condition="'$(IgnoreMibc)' != 'true'">
     <NuspecProperty Include="mibcRoot=$(ArtifactsDir)\mibc" />
   </ItemGroup>

src/Microsoft.FSharp.Compiler/Microsoft.FSharp.Compiler.nuspec

@@ -28,6 +28,10 @@
         <!-- assemblies -->
         <file src="fsc\$configuration$\$fSharpNetCoreProductTargetFramework$\fsc.dll"                      target="lib\$fSharpNetCoreProductTargetFramework$" />
         <file src="fsi\$configuration$\$fSharpNetCoreProductTargetFramework$\fsi.dll"                      target="lib\$fSharpNetCoreProductTargetFramework$" />
+        <!-- StreamJsonRpc and its dependency closure, which fsi's JSON-RPC server mode loads at run time.
+             The SDK lays out exactly what this lib folder holds, so a missing file here is a missing
+             file in every installed SDK. Filled in by the project file; empty in a source-only build. -->
+        $fsiJsonRpcServerFiles$
         <!-- FSI needs the .NET-only FSharp.Core APIs for IAsyncDisposable support (dotnet/fsharp#14454). -->
         <file src="FSharp.Core\$configuration$\$fSharpCoreShippedNetTargetFramework$\FSharp.Core.dll"      target="lib\$fSharpNetCoreProductTargetFramework$" />
         <file src="FSharp.Core\$configuration$\$fSharpCoreShippedNetTargetFramework$\FSharp.Core.xml"      target="lib\$fSharpNetCoreProductTargetFramework$" />

src/fsi/fsi.targets

@@ -22,13 +22,26 @@
     <DefineConstants>$(DefineConstants);FSI_SHADOW_COPY_REFERENCES;FSI_SERVER</DefineConstants>
   </PropertyGroup>
 
+  <!-- The JSON-RPC server needs StreamJsonRpc, which the .NET Framework fsi does not carry and a
+       source-only build cannot provide, so the mode belongs to the .NET build alone. -->
+  <PropertyGroup Condition="'$(TargetFramework)' != 'net472' and '$(DotNetBuildSourceOnly)' != 'true'">
+    <FsiJsonRpcServer>true</FsiJsonRpcServer>
+    <DefineConstants>$(DefineConstants);FSI_JSONRPC_SERVER</DefineConstants>
+  </PropertyGroup>
+
   <!--  The fsi application does not provide a referencable public interface  -->
   <PropertyGroup>
     <NoOptimizationData>true</NoOptimizationData>
     <NoInterfaceData>true</NoInterfaceData>
     <CompressMetadata>true</CompressMetadata>
   </PropertyGroup>
 
+  <ItemGroup Condition="'$(FsiJsonRpcServer)' == 'true'">
+    <Compile Include="$(MSBuildThisFileDirectory)interactiveProtocol.fs" />
+    <Compile Include="$(MSBuildThisFileDirectory)fsiserver.fs" />
+    <PackageReference Include="StreamJsonRpc" />
+  </ItemGroup>
+
   <ItemGroup>
     <EmbeddedText Include="$(MSBuildThisFileDirectory)..\LegacyMSBuildResolver\LegacyResolver.txt">
       <Link>LegacyResolver.txt</Link>