Outcome receipts for Semantic Kernel plugins that call external web tools

[auxiliar-ag]

Semantic Kernel already draws one useful boundary for you: a [KernelFunction] parameter typed as Kernel, KernelArguments, ILoggerFactory, CultureInfo, or CancellationToken is automatically excluded from the LLM-facing function schema — the model only sees the parameters it needs to fill in. Plumbing stays hidden; the model-facing surface stays small. I've been applying that same split one layer down, inside plugin functions that call an external web service (search, extraction, page retrieval): keep the transport details out of what the model reads, but hand back a small, structured verdict instead of the raw payload.

Here's the gap that motivates it. An HTTP status code only tells you the wire transfer worked; it says nothing about whether the search came back empty, the cache handed you something stale, or the body fails a schema check downstream. Hand that response to the agent loop unfiltered and "no exception was raised" quietly becomes "the task is done" — which stays invisible until a silently-swapped fallback provider starts returning the same empty result, with nothing in the trace to flag it.

Two booleans, not one status flag

{
  "tool_name": "web_search",
  "outcome": "invalid_result",
  "transport_success": true,
  "task_success": false,
  "reason": "empty_results",
  "latency_ms": 11.4,
  "attempts": 1,
  "fallback_used": false,
  "fallback_reason": null
}

I split the verdict into transport_success and task_success rather than folding both into a single status string, because they fail independently and want different responses. A thrown exception from the transport (timeout, non-2xx, connection refused) justifies an immediate retry or a fallback attempt. A 200 that fails validation is a different situation — retrying the same provider with the same query is unlikely to help, so the more useful move is trying a different provider or giving up cleanly with a reason attached.

Wiring it into a KernelFunction

The function itself stays thin: [KernelFunction("web_search")] takes query and a CancellationToken (the latter invisible to the model, per the opening point), hands off to a plain C# service that owns the retry/fallback/redaction logic, and serializes whatever comes back. Registration and invocation use the ordinary plugin surface — builder.Plugins.AddFromObject(new WebSearchPlugin(service), "WebTools") to add it, kernel.InvokeAsync("WebTools", "web_search", new KernelArguments { ["query"] = ... }) to call it. No custom Kernel subclass, no non-standard invocation path — the receipt logic lives inside an ordinary plugin method.

One easy-to-miss point: a failed transport call's exception message might contain a query-string API key or a bearer token in an authorization header, which can end up sitting in reason unless something strips it first. I run known secret patterns through a redaction step before they reach the receipt — cheap to add, awkward to retrofit after a key first shows up in a shared trace.

To be clear about scope: this is a pattern you'd adapt into your own plugin, not something you install. There's no package beyond Microsoft.SemanticKernel itself — just a receipt record, a small service class, and one KernelFunction. Everything above was built and exercised against Microsoft.SemanticKernel 1.79.0 with deterministic fixture transports; no live provider was called while testing it.

Zooming out one level: once a plugin is reporting task_success per call, the next question tends to be which provider earns that flag more often across a whole corpus of calls, not just this one. That's a different, longer-running measurement — closer to a provider comparison benchmark than anything a single receipt is meant to answer.

Question for other Semantic Kernel users: where would you rather own this check — inline inside the KernelFunction, as shown here, or centralized in a registered IFunctionInvocationFilter so every function gets it without each author wiring it by hand? The filter route also reaches calls made outside the automatic tool-calling loop, which scoping the check to IAutoFunctionInvocationFilter alone wouldn't cover.

Disclosure: I work with NativePort, and used AI assistance to draft this post; both the code and the write-up were checked against microsoft/semantic-kernel's current C# source and docs before publication.