Fix empty metadata tool results and UTF-8 read boundaries

Author: KSemenenko

CHANGELOG.md

@@ -2,6 +2,13 @@
 
 All notable changes to ManagedCode.FileContext are documented here.
 
+## Unreleased
+
+- Preserve UTF-8 byte accounting when a surrogate pair crosses a full-read buffer boundary.
+- Return a missing-file failure from the metadata tool so session restoration cannot turn a null result into empty content; keep the nullable direct API unchanged.
+- Verify tool-result ordering, empty/error outputs, restored-session follow-ups, and concurrent multi-file operations through real filesystem and LlmTck tests.
+- Update Meziantou.Analyzer to 3.0.203. Retain OpenAI 2.12.0 because Microsoft.Extensions.AI.OpenAI 10.9.0 requires OpenAI below 2.13.0.
+
 ## 0.0.2 - 2026-09-03
 
 - Enforce a 95% product line-coverage gate in local and CI coverage runs.

Directory.Packages.props

@@ -4,7 +4,7 @@
     <CentralPackageTransitivePinningEnabled>false</CentralPackageTransitivePinningEnabled>
   </PropertyGroup>
   <ItemGroup>
-    <GlobalPackageReference Include="Meziantou.Analyzer" Version="3.0.200" PrivateAssets="all" />
+    <GlobalPackageReference Include="Meziantou.Analyzer" Version="3.0.203" PrivateAssets="all" />
     <GlobalPackageReference Include="Roslynator.Analyzers" Version="5.0.0" PrivateAssets="all" />
     <GlobalPackageReference Include="SonarAnalyzer.CSharp" Version="10.33.0.1635" PrivateAssets="all" />
   </ItemGroup>

docs/Features/file-context.md

@@ -13,7 +13,7 @@ In scope: standard file access, bounded line navigation, metadata, Markdown grap
 3. The adapter implements the complete Agent Framework `AgentFileStore` contract using only `IStorage`.
 4. Standard tools come from Agent Framework `FileAccessProvider`: read, list, grep, write, delete, replace, and replace-lines.
 5. Write tools are disabled by default. When enabled, Agent Framework approval remains required unless the host explicitly disables it.
-6. Full reads fail before allocation when metadata exceeds `MaximumFullReadBytes`.
+6. Full reads fail before allocation when metadata exceeds `MaximumFullReadBytes`. Decoded UTF-8 limits retain encoder state across buffer boundaries, including split surrogate pairs.
 7. Range reads are 1-based by line, stream sequentially, return continuation metadata, and stop at configured line/byte limits.
 8. Content search is case-insensitive regular expression search with a timeout, file/match/result limits, and optional standard glob filtering.
 9. Directory listing returns direct child directories before direct child files and does not leak the configured storage prefix.
@@ -48,12 +48,30 @@ flowchart TD
 ## Failure flows
 
 - Unsafe paths fail with `ArgumentException`; the storage provider is not called.
-- Missing files return `null` through the `AgentFileStore` and metadata contracts; a range read throws `FileNotFoundException`.
+- Missing files return `null` through the direct `AgentFileStore` and `IFileContext.GetInfoAsync` contracts; range reads and the `file_context_info` tool throw `FileNotFoundException`.
 - Storage failures become `IOException` values with the operation and safe logical path, preserving the provider's safe problem detail.
 - Invalid or catastrophic regex patterns fail deterministically; a regex timeout does not hang the agent invocation.
 - Oversized files, result sets, or graph exports stop at configured boundaries and report truncation or a clear limit failure.
 - Graph operations with no matching Markdown input fail clearly instead of inventing an empty knowledge base.
 
+## Empty results and conversation history
+
+An empty file or no search matches is a valid tool outcome. With the tested Agent Framework function-invocation and OpenAI chat pipeline, these become a `role: tool` message with the matching `tool_call_id`: the content contains serialized `""` or `[]`, respectively. The metadata tool reports a missing file as a failure instead of returning `null`, because the tested Agent Framework session roundtrip converts a null function result into empty wire content. Range reads retain their structured window metadata even when their content is empty. Tool exceptions also produce a matching error result during the normal function-invocation loop.
+
+FileContext does not persist agent sessions, synthesize fallback assistant responses, or repair interrupted model/tool turns. The host owns those concerns: it must preserve call/result pairs when saving or replaying history and handle cancellation, approval pauses, and provider failures before reusing an incomplete turn. A final assistant message does not replace a tool result. Completed tool turns are also tested through Agent Framework session serialization/restoration and a subsequent user request.
+
+## Multiple files and concurrency
+
+A model response can request several file tools, each with its own path and call ID. The integration tests verify that all results, including a failed sibling call, reach the next model request. Single-file read/write/range tools do not accept a batch of paths; list, grep, and Markdown graph operations already work across a scoped collection.
+
+`FunctionInvokingChatClient` processes calls sequentially by default. A host using `UseFunctionInvocation` can enable concurrent execution:
+
+```csharp
+.UseFunctionInvocation(configure: client => client.AllowConcurrentInvocation = true)
+```
+
+Independent writes and range reads on eight different files are tested concurrently against the real filesystem provider. Other storage providers must support the host's chosen concurrency. FileContext adds no multi-file transaction or same-file write locking; serialize dependent operations and writes to the same path. Write-tool enablement and approval requirements still apply.
+
 ## Verification scenarios
 
 1. Write, read, exists, list, grep, and delete through `ManagedCodeStorageFileStore` against real filesystem storage.

docs/Testing/index.md

@@ -6,6 +6,8 @@ The suite is integration-first:
 - graph tests run the real `ManagedCode.MarkdownLd.Kb` parser, graph builder, search, and serializers;
 - dependency-injection tests resolve the actual default and keyed Agent Framework contracts;
 - end-to-end tests run Agent Framework function invocation against a real LlmTck HTTP replay service and verify every advertised read, list, grep, write, delete, replace, range, metadata, graph-search, and graph-export tool;
+- protocol tests inspect outgoing HTTP messages for matching call/result IDs after empty results, missing files, tool failures, mutations, and multiple calls with sequential or concurrent invocation enabled, including a subsequent request after session serialization/restoration;
+- concurrent storage tests write and range-read eight independent files through one shared adapter/service;
 - a sparse 1 GiB filesystem test reads bounded line windows repeatedly, rejects full-file loading, caps allocations, and proves that an oversized line fails before it can be buffered in memory.
 
 Every filesystem test owns a unique temporary root and removes it on disposal. Test execution is serialized so process-wide allocation assertions cannot be distorted by another test. No `IStorage`, Agent Framework, Markdown-LD, or LlmTck mocks are used.

src/ManagedCode.FileContext/FileContextToolDescriptions.cs

@@ -9,7 +9,7 @@ internal static class FileContextToolDescriptions
         "Read a bounded, one-based line range from a text file. Use this instead of a full read for large files.";
     public const string StartLine = "One-based first line to read.";
     public const string LineCount = "Number of lines to return; omitted uses the configured default.";
-    public const string GetInfo = "Return file size, media type, and last-modified time without reading its content.";
+    public const string GetInfo = "Return file size, media type, and last-modified time without reading its content. Fails if the file does not exist.";
     public const string SearchMarkdownGraph =
         "Build a linked-data knowledge graph from scoped Markdown files and search its concepts and relationships.";
     public const string GraphQuery = "Concept or relationship query.";

src/ManagedCode.FileContext/FileContextTools.cs

@@ -15,11 +15,12 @@ public Task<FileContextRange> ReadRangeAsync(
     }
 
     [Description(FileContextToolDescriptions.GetInfo)]
-    public Task<FileContextInfo?> GetInfoAsync(
+    public async Task<FileContextInfo> GetInfoAsync(
         [Description(FileContextToolDescriptions.RelativeFilePath)] string path,
         CancellationToken cancellationToken = default)
     {
-        return fileContext.GetInfoAsync(path, cancellationToken);
+        return await fileContext.GetInfoAsync(path, cancellationToken).ConfigureAwait(false)
+            ?? throw new FileNotFoundException($"File '{path}' was not found.", path);
     }
 
     [Description(FileContextToolDescriptions.SearchMarkdownGraph)]

src/ManagedCode.FileContext/StorageTextReader.cs

@@ -14,20 +14,29 @@ public static async Task<string> ReadAsync(
     {
         using var reader = new StreamReader(stream, Encoding.UTF8, true, ReaderBufferSize, leaveOpen: true);
         var buffer = new char[CharacterBufferLength];
+        var encoder = Encoding.UTF8.GetEncoder();
+        var encodedBuffer = new byte[Encoding.UTF8.GetMaxByteCount(CharacterBufferLength)];
         var builder = new StringBuilder();
         var bytesRead = 0L;
 
         while (await reader.ReadAsync(buffer.AsMemory(), cancellationToken).ConfigureAwait(false) is var read && read > 0)
         {
-            bytesRead += Encoding.UTF8.GetByteCount(buffer.AsSpan(0, read));
-            if (bytesRead > limit)
-            {
-                throw new IOException($"Decoded file content exceeds the configured {limit}-byte limit.");
-            }
+            bytesRead += encoder.GetBytes(buffer.AsSpan(0, read), encodedBuffer, flush: false);
+            EnsureWithinLimit(bytesRead, limit);
 
             builder.Append(buffer, 0, read);
         }
 
+        bytesRead += encoder.GetBytes(ReadOnlySpan<char>.Empty, encodedBuffer, flush: true);
+        EnsureWithinLimit(bytesRead, limit);
         return builder.ToString();
     }
+
+    private static void EnsureWithinLimit(long bytesRead, long limit)
+    {
+        if (bytesRead > limit)
+        {
+            throw new IOException($"Decoded file content exceeds the configured {limit}-byte limit.");
+        }
+    }
 }

tests/ManagedCode.FileContext.Tests/FileContextBoundaryTests.cs

@@ -12,6 +12,21 @@ public async Task FileQueries_WhenFileIsMissing_ReturnAbsenceOrClearFailure()
         await Should.ThrowAsync<FileNotFoundException>(() => service.ReadRangeAsync("missing.txt"));
     }
 
+    [Fact]
+    public async Task MarkdownGraph_WhenSiblingDirectorySharesPrefix_ExcludesSiblingDocuments()
+    {
+        await using var scope = await TestStorageScope.CreateAsync();
+        var store = new ManagedCodeStorageFileStore(scope.Storage);
+        var service = new FileContextService(store);
+        await store.WriteAsync("docs/included.md", "# Included document");
+        await store.WriteAsync("docs-other/excluded.md", "# Excluded sibling");
+
+        var result = await service.ExportMarkdownGraphAsync(MarkdownGraphFormat.Mermaid, "docs");
+
+        result.DocumentCount.ShouldBe(1);
+        result.Content.ShouldNotContain("Excluded sibling");
+    }
+
     [Fact]
     public async Task ReadRange_WhenArgumentsAreOutsideConfiguredBounds_RejectsRequest()
     {

tests/ManagedCode.FileContext.Tests/LlmTck/FileAccessMutationLlmTckTests.cs

@@ -60,6 +60,9 @@ public async Task Agent_WhenLlmRequestsMutationTool_ChangesRealFileSystemContent
         (await store.ReadAsync(FileName)).ShouldBe(expectedContent);
         LlmTckToolReplay.RecordedRequests.Count.ShouldBe(2);
         LlmTckToolReplay.RecordedRequests[0].ShouldContain(toolName);
+        var results = LlmTckToolAssertions.AssertClosedCalls(
+            LlmTckToolReplay.RecordedRequests[1], $"call-{toolName}");
+        results[0].GetProperty("content").GetString().ShouldNotBeNullOrWhiteSpace();
         var assertions = await host.GetAssertionsAsync();
         assertions.Matched.ShouldBe(2);
         assertions.Unmatched.ShouldBe(0);

tests/ManagedCode.FileContext.Tests/LlmTck/FileToolResultProtocolTests.cs

@@ -0,0 +1,112 @@
+using System.ClientModel;
+using System.Text.Json;
+using ManagedCode.LlmTck.Configuration;
+using ManagedCode.LlmTck.Models;
+using Microsoft.Agents.AI;
+using Microsoft.Extensions.AI;
+using OpenAI;
+
+namespace ManagedCode.FileContext.Tests.LlmTck;
+
+public sealed class FileToolResultProtocolTests(Xunit.Abstractions.ITestOutputHelper output)
+{
+    [Theory]
+    [InlineData(FileAccessProvider.ReadFileToolName, "{\"fileName\":\"empty.txt\"}", "\"\"")]
+    [InlineData(FileAccessProvider.ReadFileToolName, "{\"fileName\":\"missing.txt\"}", "not found")]
+    [InlineData(FileAccessProvider.LsToolName, "{\"directory\":\"empty\"}", "[]")]
+    [InlineData(FileAccessProvider.GrepToolName, "{\"regexPattern\":\"absent\",\"directory\":\"\"}", "[]")]
+    [InlineData(FileContextToolNames.GetInfo, "{\"path\":\"missing.txt\"}", "Error: Function failed.")]
+    [InlineData(FileContextToolNames.ReadRange, "{\"path\":\"empty.txt\"}", "\"content\": \"\"")]
+    [InlineData(FileContextToolNames.ReadRange, "{\"path\":\"first.txt\",\"startLine\":100}", "\"content\": \"\"")]
+    [InlineData(FileContextToolNames.ReadRange, "{\"path\":\"missing.txt\"}", "Error: Function failed.")]
+    [InlineData(FileContextToolNames.ReadRange, "{\"path\":\"../escape.txt\"}", "Error: Function failed.")]
+    [InlineData(FileContextToolNames.SearchMarkdownGraph, "{\"query\":\"absent\"}", "Error: Function failed.")]
+    public async Task EmptyOrFailedTool_StillSendsMatchingResult(string toolName, string arguments, string expectedContent)
+    {
+        var requests = await RunToolLoopAsync(LlmTckToolReplay.CreateResponse("call-empty", toolName, arguments));
+
+        foreach (var request in requests)
+        {
+            var results = LlmTckToolAssertions.AssertClosedCalls(request, "call-empty");
+            output.WriteLine(results[0].GetRawText());
+            results[0].GetProperty("content").ValueKind.ShouldBe(JsonValueKind.String);
+            results[0].GetProperty("content").GetString()!.ShouldContain(expectedContent);
+        }
+    }
+
+    [Theory]
+    [InlineData(false)]
+    [InlineData(true)]
+    public async Task MultipleFileCalls_InOneTurn_ReturnEveryResult(bool concurrent)
+    {
+        var first = LlmTckToolReplay.CreateResponse("call-first", FileAccessProvider.ReadFileToolName, "{\"fileName\":\"first.txt\"}");
+        var second = LlmTckToolReplay.CreateResponse("call-second", FileContextToolNames.ReadRange, "{\"path\":\"second.txt\"}");
+        var missing = LlmTckToolReplay.CreateResponse("call-missing", FileContextToolNames.ReadRange, "{\"path\":\"missing.txt\"}");
+
+        var requests = await RunToolLoopAsync($"[{first},{second},{missing}]", concurrent);
+
+        foreach (var request in requests)
+        {
+            var results = LlmTckToolAssertions.AssertClosedCalls(request, "call-first", "call-second", "call-missing");
+            results.Single(result => string.Equals(result.GetProperty("tool_call_id").GetString(), "call-first", StringComparison.Ordinal))
+                .GetProperty("content").GetString()!.ShouldContain("first-content");
+            results.Single(result => string.Equals(result.GetProperty("tool_call_id").GetString(), "call-second", StringComparison.Ordinal))
+                .GetProperty("content").GetString()!.ShouldContain("second-content");
+            results.Single(result => string.Equals(result.GetProperty("tool_call_id").GetString(), "call-missing", StringComparison.Ordinal))
+                .GetProperty("content").GetString().ShouldNotBeNullOrWhiteSpace();
+        }
+    }
+
+    [Fact]
+    public async Task LiteralNullAssistantText_IsNotInterpretedAsToolReplay()
+    {
+        var requests = await RunToolLoopAsync(
+            LlmTckToolReplay.CreateResponse("read", FileAccessProvider.ReadFileToolName, "{\"fileName\":\"first.txt\"}"),
+            finalResponse: "null");
+
+        foreach (var request in requests)
+        {
+            LlmTckToolAssertions.AssertClosedCalls(request, "read");
+        }
+    }
+
+    private static async Task<string[]> RunToolLoopAsync(string replay, bool concurrent = false, string finalResponse = "Inspection completed.")
+    {
+        const string prompt = "Inspect the files.";
+        LlmTckToolReplay.Reset();
+        var host = new LlmTckTestHost(() => new LlmTckConfigurationBuilder()
+            .AddModel(FileContextAgentLlmTckTests.Model, LlmTckModelKind.Chat)
+            .AddChatScenario("protocol", scenario => scenario.ForModel(FileContextAgentLlmTckTests.Model)
+                .WhenUserContains(prompt).Responds(replay).Responds(finalResponse).Responds("Follow-up completed."))
+            .Build());
+        await using var hostLifetime = host.ConfigureAwait(false);
+        await host.StartAsync().ConfigureAwait(false);
+        var storage = await TestStorageScope.CreateAsync().ConfigureAwait(false);
+        await using var storageLifetime = storage.ConfigureAwait(false);
+        var store = new ManagedCodeStorageFileStore(storage.Storage);
+        await store.WriteAsync("empty.txt", string.Empty).ConfigureAwait(false);
+        await store.WriteAsync("first.txt", "first-content").ConfigureAwait(false);
+        await store.WriteAsync("second.txt", "second-content").ConfigureAwait(false);
+        using var provider = new FileContextProvider(store, new FileContextService(store),
+            new FileContextOptions { RequireReadToolApproval = false });
+        var sdk = new OpenAIClient(new ApiKeyCredential("not-required-by-llm-tck"),
+            new OpenAIClientOptions { Endpoint = LlmTckToolReplay.CreateRouteUri(host.Endpoint) });
+        using var client = sdk.GetChatClient(FileContextAgentLlmTckTests.Model).AsIChatClient().AsBuilder()
+            .UseAIContextProviders(provider)
+            .UseFunctionInvocation(configure: options => options.AllowConcurrentInvocation = concurrent).Build();
+        var agent = new ChatClientAgent(client, new ChatClientAgentOptions { UseProvidedChatClientAsIs = true });
+
+        var session = await agent.CreateSessionAsync().ConfigureAwait(false);
+        var response = await agent.RunAsync(prompt, session).ConfigureAwait(false);
+        response.Text.ShouldBe(finalResponse);
+        var saved = await agent.SerializeSessionAsync(session).ConfigureAwait(false);
+        var restored = await agent.DeserializeSessionAsync(saved).ConfigureAwait(false);
+        var followUp = await agent.RunAsync("Continue the inspection.", restored).ConfigureAwait(false);
+        followUp.Text.ShouldBe("Follow-up completed.");
+        LlmTckToolReplay.RecordedRequests.Count.ShouldBe(3);
+        var assertions = await host.GetAssertionsAsync().ConfigureAwait(false);
+        assertions.Matched.ShouldBe(3);
+        assertions.Unmatched.ShouldBe(0);
+        return LlmTckToolReplay.RecordedRequests.Skip(1).ToArray();
+    }
+}