Evaluate replacing per-plugin Python shims with a Rust-native managed-plugin hook boundary

[lucarlig]

Summary

Evaluate whether the managed Rust-backed plugins in cpex-plugins should keep per-plugin Python gateway shims, or whether the hook boundary should move lower so the gateway can execute them through a Rust-native contract.

The current model works, but it keeps a Python package and Python import surface on every managed Rust plugin even when the actual plugin behavior is already implemented in Rust.

This issue is to make that tradeoff explicit and decide whether we should:

1. keep the current Python package ABI and just reduce boilerplate,
2. introduce a shared Rust-native hook bridge while still publishing Python packages, or
3. move the hook/runtime boundary directly into IBM/mcp-context-forge and potentially relocate some of this responsibility there.

Current state

Today this repo is explicitly organized around Rust plugins published as Python packages:

• README.md and DEVELOPING.md define the managed layout as plugins/rust/python-package/<slug>/
• each plugin publishes a Python entry point under [project.entry-points."cpex.plugins"]
• each plugin manifest declares kind in Python module.object form
• tools/plugin_catalog.py validates that Python package/module/object contract
• ADR-048 in IBM/mcp-context-forge explicitly chose PyO3-backed packages because the gateway still imports plugins as normal Python modules

That means the Python layer is not just incidental packaging today; it is part of the current plugin ABI.

At the same time, the amount of Python-specific logic varies a lot across plugins:

Mostly thin gateway shims today

• pii_filter
• secrets_detection
• rate_limiter
• url_reputation

These already look like compatibility wrappers around Rust-owned behavior.

Still keep meaningful Python-side behavior

• retry_with_backoff
• encoded_exfil_detection

These still retain non-trivial Python config, fallback, or hook logic, so removing Python there is not just a packaging cleanup.

Why reconsider this

Even when the Python layer is thin, it still has cost:

• every plugin needs a Python module/object entry point and manifest wiring
• every plugin has Python packaging and import-surface conventions to maintain
• test coverage has to keep validating the Python package contract, not just the Rust behavior
• shared hook/result translation logic is duplicated or reimplemented per plugin
• the runtime story remains “Rust behind Python” rather than “native Rust plugin execution”
• architectural ownership is split: plugin behavior lives here, but the actual plugin ABI is still largely defined by the framework repo

This is not automatically wrong. The question is whether that tradeoff is still the right one now that the managed set has been extracted.

Options

Option A: Keep the current Python package ABI, but standardize the bridge harder

What this means:

• keep publishing cpex-* Python packages
• keep Python module:object entry points and manifest kind
• move more hook/result construction into shared Rust helpers or code generation
• reduce each plugin shim to the smallest practical compatibility surface

Pros:

• lowest-risk path
• compatible with ADR-048 and the current gateway loading model
• no framework ABI break in mcp-context-forge
• still lets us delete a lot of repetitive per-plugin Python glue
• can be done incrementally plugin by plugin

Cons:

• Python remains part of the runtime ABI and packaging model
• we still carry Python package validation, entry-point wiring, and shim maintenance forever
• does not fully answer whether Rust-native execution should be the long-term model

Option B: Define a Rust-native plugin hook ABI, but keep this repo as the plugin home

What this means:

• the gateway/framework learns how to load and invoke managed Rust plugins directly through a stable Rust-side contract
• cpex-plugins becomes the home of those Rust-native plugins and shared bridge crates
• Python packages become optional compatibility artifacts or disappear entirely for plugins that do not need Python-side behavior

Pros:

• removes per-plugin Python shims where they are only compatibility glue
• makes hook/result translation a shared native layer instead of repeated package boilerplate
• gives a cleaner architectural story for “Rust-backed managed plugins”
• still preserves the repo split that ADR-048 wanted for independent plugin ownership and release cadence

Cons:

• this is a cross-repo ABI change, not a local repo cleanup
• mcp-context-forge would need new loading/runtime machinery
• release, compatibility, and version skew management may get harder before they get easier
• some plugins still have real Python logic today, so migration would not be uniform
• operators may lose the simplicity of “install normal Python packages and import plugins normally” unless we preserve that experience another way

Option C: Move the native hook/runtime boundary directly into IBM/mcp-context-forge

What this means:

• accept that the true ABI owner is the framework repo
• implement Rust-native hook loading/execution there first
• then decide whether cpex-plugins should only hold reusable engines, or whether some managed plugin code should move back into the framework repo

Pros:

• puts ABI ownership where the runtime decisions actually live
• avoids pretending this can be solved entirely inside cpex-plugins
• may reduce cross-repo churn if native execution is tightly coupled to gateway internals
• cleaner if plugin “loading” and hook lifecycle are framework concerns first and packaging concerns second

Cons:

• reintroduces some of the coupling ADR-048 was trying to remove
• may pull plugin release and gateway release concerns back together
• risks rebuilding the same monorepo pressure that extraction was intended to reduce
• could make independently versioned managed plugins less real in practice

Option D: Keep the current architecture and only remove redundant Python fallbacks

What this means:

• treat ADR-048 as the long-term architecture
• keep Python package entry points as the official plugin contract
• continue deleting Python fallback/parity logic where it no longer adds value
• do not pursue a Rust-native loading model

Pros:

• simplest operating model
• aligned with the current repo validator, release workflow, and ADR
• avoids a large cross-repo redesign

Cons:

• leaves the current Python shim model in place indefinitely
• keeps the architectural boundary slightly awkward for plugins that are functionally Rust-native already
• may leave us with recurring requests to “just remove the shim” without a principled answer

Recommendation

Do this in two stages instead of picking a repo move blindly.

Stage 1

In cpex-plugins, reduce the accidental Python surface as far as possible without changing the framework ABI:

• identify the exact hook/result/violation/config patterns repeated across plugins
• push more of that into shared Rust helpers or generation
• classify which plugins are truly shim-only vs which still need meaningful Python behavior

Stage 2

Open a companion architecture decision in IBM/mcp-context-forge to decide whether a direct Rust-native managed-plugin ABI should exist at all.

That decision should come before we decide to move code into the framework repo or redesign this repo around native loading.

If the framework decides the long-term ABI remains Python package imports, then the answer here is Option A/D.
If the framework decides Rust-native plugin loading is strategic, then we can choose between Option B and Option C with a real ABI owner and migration plan.

Acceptance criteria

• Document the exact reasons the current Python shim exists for each managed plugin.
• Separate "Python needed for framework ABI" from "Python needed for plugin-specific behavior".
• Decide whether the long-term managed-plugin ABI is still Python module:object, or whether a Rust-native ABI should be introduced.
• If the answer is "stay Python ABI", define the shared bridge/codegen work that minimizes per-plugin shim code.
• If the answer is "introduce Rust-native ABI", open the required companion issue(s) in IBM/mcp-context-forge and define the migration boundary.
• Make an explicit call on whether any code should move into IBM/mcp-context-forge as part of that decision, instead of letting the boundary drift implicitly.

Related

• Move plugin-owned testing into cpex-plugins to support removing redundant Python fallbacks #20
• Add to_rust_native_policy() method to RetryWithBackoffPlugin #21
• Track OpenTelemetry support at the Rust plugin boundary #27
• IBM/mcp-context-forge ADR-048: Extract Rust-Backed Plugins First and Preserve Python Examples Separately

[lucarlig]

contextforge-org/contextforge-plugins-framework#12 will fix close this