diff --git a/AGENTS.md b/AGENTS.md index 6ce9add..51c391e 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,4 +1,4 @@ -# CLAUDE.md +# AGENTS.md ## Git @@ -8,11 +8,171 @@ This is a monorepo of standalone plugin packages for the ContextForge Plugin Extensibility (CPEX) Framework. Each plugin lives in its own top-level directory with independent build configuration. -- Plugins are Rust+Python (PyO3/maturin) or pure Python. +- Plugins are implemented as **pure Python** or **pure Rust**. Each plugin uses one language for its core logic — there is no dual-path where a plugin ships both Rust and Python implementations with a Rust fallback. For Rust plugins, Python entry points (PyO3/maturin) are a packaging and distribution layer only, not a parallel implementation. - Each plugin has its own `pyproject.toml`, `Cargo.toml`, `Makefile`, and `tests/`. - Package names follow the pattern `cpex-` (e.g., `cpex-rate-limiter`). - `mcpgateway` is a runtime dependency provided by the host gateway — never declare it in `pyproject.toml`. +## Testing Strategy + +### Test Location by Type + +- **Unit tests**: Located within each plugin's own directory + - Python: `plugins/rust/python-package//tests/` (current hybrid) or `plugins/python//tests/` (pure Python) + - Rust: inline `mod tests` within source files (e.g., `src/lib.rs`) + - Test individual plugin functionality in isolation + - Fast, deterministic tests + - Run during plugin development and CI + - Scope: Plugin logic, Rust functions, Python bindings + +- **Plugin-framework integration tests**: Located in `plugins/rust/python-package//tests/` + - Test plugin integration with the local plugin framework (PyO3 bindings, Python ↔ Rust interface) + - Run via `make test-integration` within the plugin directory + - Scope: PyO3 entry points, plugin loading by the Python framework, hook dispatch + +- **Gateway integration tests**: Located in `mcp-context-forge/tests/integration/` + - Test plugin integration with the full gateway + - Test cross-plugin interactions + - Test plugin lifecycle management + - Scope: Plugin loading in gateway context, hook execution, framework interaction + +- **E2E tests**: Located in `mcp-context-forge/tests/e2e/` + - Test complete workflows with plugins enabled + - Test plugin behavior in realistic scenarios + - Test multi-gateway plugin coordination + - Scope: Full request/response cycles, real-world usage patterns + +### Cross-Repository Testing Coordination + +When developing a plugin: + +1. Write unit tests in the plugin's own directory (Rust: inline `mod tests`; Python: `plugins/rust/python-package//tests/`) and plugin-framework integration tests in `plugins/rust/python-package//tests/` +2. Run local tests: `make test-all` and `make test-integration` from plugin directory +3. After plugin PR is merged, coordinate with `mcp-context-forge` team +4. Write gateway integration/E2E tests in `mcp-context-forge/tests/` +5. Ensure both repositories' CI passes before release + +See `mcp-context-forge/tests/AGENTS.md` for integration/E2E test conventions. + +## Plugin Development Workflows + +### Current Workflow: Rust + Python Hybrid + +**Architecture:** +- Plugin logic implemented entirely in Rust — no Python fallback implementation +- Python entry points (PyO3/maturin) are a packaging and distribution layer only +- Published as Python packages to PyPI +- Loaded by Python-based plugin framework in gateway + +**Why Python Entry Points?** +The plugin framework is currently implemented in Python (`mcpgateway/plugins/framework/`). Python entry points allow the framework to discover and load plugins dynamically. This is a transitional packaging layer — all plugin logic remains in Rust. This is not a dual-path architecture. + +**Development Steps:** + +1. **Create Plugin** (in `cpex-plugins`): + ```bash + cd cpex-plugins + make plugin-scaffold # Interactive plugin generator + ``` + +2. **Implement Plugin** (in `cpex-plugins/plugins/rust/python-package//`): + - Write Rust core logic in `src/` + - Implement Python bindings in `cpex_/plugin.py` + - Update `plugin-manifest.yaml` + +3. **Write Tests**: + ```bash + cd plugins/rust/python-package/ + # Add Rust unit tests inline in src/ using mod tests + # Add Python unit tests in tests/ + # Add plugin-framework integration tests in tests/ (run via make test-integration) + make test-all # Run Rust + Python unit tests + make test-integration # Run plugin-framework integration tests + ``` + +4. **Build and Install**: + ```bash + uv sync --dev + make install # Build Rust extension and install + ``` + +5. **Create PR in cpex-plugins**: + - Include unit tests and plugin-framework integration tests + - Ensure `make ci` passes + - Tag release: `-v` + +6. **Gateway Integration Testing** (in `mcp-context-forge`): + - Install plugin: `pip install cpex-` + - Configure in `plugins/config.yaml` + - Write integration tests in `tests/integration/` + - Write E2E tests in `tests/e2e/` + +7. **Release**: + - Tag in cpex-plugins triggers PyPI publish + - Update mcp-context-forge dependencies + - Deploy with new plugin version + +### Future Workflow: Pure Rust + +**Architecture (Post-Framework Migration):** +- Plugins implemented in pure Rust +- Plugin framework migrated to Rust +- No Python entry points needed +- Direct Rust-to-Rust plugin loading +- Published to Cargo registry + +**What Changes:** +- Remove `pyproject.toml` and maturin configuration +- Remove Python entry points (`cpex_/plugin.py`) +- Remove PyO3 bindings +- Pure Rust crate structure: `plugins/rust//` +- Cargo-based dependency management + +**Development Steps (Future):** + +1. **Create Plugin** (in `cpex-plugins`): + ```bash + cd cpex-plugins + cargo new --lib plugins/rust/ + ``` + +2. **Implement Plugin** (in `cpex-plugins/plugins/rust//`): + - Write Rust plugin in `src/lib.rs` + - Implement plugin traits from Rust framework + - Update `Cargo.toml` + +3. **Write Unit Tests** (inline `mod tests` in source files): + ```bash + cd plugins/rust/ + cargo test # Run Rust tests + ``` + +4. **Build**: + ```bash + cargo build --release + ``` + +5. **Create PR in cpex-plugins**: + - Include unit tests + - Ensure `cargo test` passes + - Version in `Cargo.toml` + +6. **Integration Testing** (in `mcp-context-forge`): + - Add plugin as Cargo dependency + - Configure in Rust plugin framework + - Write integration tests in `tests/integration/` + - Write E2E tests in `tests/e2e/` + +7. **Release**: + - Publish to Cargo registry + - Update mcp-context-forge `Cargo.toml` + - Deploy with new plugin version + +**Migration Timeline:** +- Current: Hybrid Rust + Python (transitional) +- Future: Pure Rust (after framework migration) +- Python components will be removed in future releases + ## Build & Test From within a plugin directory (e.g., `rate_limiter/`): @@ -41,4 +201,4 @@ When bumping a plugin version, update all of these: 2. `cpex_/plugin-manifest.yaml` — the `version` field. 3. `Cargo.lock` — updates automatically on the next build. -Tag releases as `-v` (e.g., `rate-limiter-v0.0.2`) on `main` to trigger the PyPI publish workflow. +Tag releases as `-v` (e.g., `rate-limiter-v0.0.2`) on `main` to trigger the PyPI publish workflow. \ No newline at end of file diff --git a/README.md b/README.md index d90aceb..52ea022 100644 --- a/README.md +++ b/README.md @@ -24,6 +24,42 @@ Python integration tests live under `plugins/tests//`; Rust unit tests liv Rust crates are owned by the top-level workspace in `Cargo.toml`. Python package names follow `cpex-`, Python modules follow `cpex_`, plugin manifests must declare a top-level `kind` in `module.object` form, and `pyproject.toml` must publish the matching `module:object` reference under `[project.entry-points."cpex.plugins"]`. Release tags use the hyphenated slug form `-v`, for example `rate-limiter-v0.0.2`. +## Testing Strategy + +Testing spans two repositories: + +- **Unit tests**: within each plugin's own directory — Python in `plugins/rust/python-package//tests/`, Rust inline via `mod tests` in source files +- **Plugin-framework integration tests**: `plugins/rust/python-package//tests/` — test PyO3 bindings and plugin loading by the Python framework (`make test-integration`) +- **Gateway integration tests**: `mcp-context-forge/tests/integration/` — test plugin integration with the full gateway +- **E2E tests**: `mcp-context-forge/tests/e2e/` — test complete workflows with plugins + +Unit tests and plugin-framework integration tests live in the plugin's own directory. Gateway integration and E2E tests live in `mcp-context-forge`. + +See [TESTING.md](TESTING.md) for detailed testing guidelines and cross-repository coordination. + +## Plugin Development + +### Current Architecture (Transitional) + +Plugins are implemented as **pure Python** or **pure Rust** — each plugin uses one language for its logic. There is no dual-path where a plugin ships both Rust and Python implementations with a Rust fallback. + +For Rust plugins, the current approach wraps the Rust implementation with PyO3/maturin bindings as a packaging layer: +- Plugin logic implemented entirely in Rust +- Python entry points (PyO3/maturin) are a packaging and distribution layer only, not a parallel implementation +- Published as Python packages to PyPI +- Loaded by Python-based plugin framework in `mcp-context-forge` + +### Future Architecture + +After the plugin framework is migrated to Rust: +- Plugins will be **pure Rust** implementations +- No Python entry points needed +- Direct Rust-to-Rust plugin loading +- Published to Cargo registry + +See [DEVELOPING.md](DEVELOPING.md) for detailed workflows for both current and future development. + + ## Creating a New Plugin Use the plugin scaffold generator to create a new plugin with all required files and structure: @@ -68,3 +104,45 @@ make plugin-scaffold # Create new plugin (interactive) ``` The catalog and validator used by CI live in `tools/plugin_catalog.py`. + +## Quick Start + +### Develop a Plugin + +```bash +cd plugins/rust/python-package/ +uv sync --dev # Install dependencies +make install # Build Rust extension +make test-all # Run unit tests +``` + +### Plugin-Framework Integration Testing + +After unit tests pass, run plugin-framework integration tests within `cpex-plugins`: + +```bash +cd plugins/rust/python-package/ +make test-integration # Test PyO3 bindings and framework loading +``` + +### Gateway Integration Testing + +After the plugin PR is merged, coordinate with `mcp-context-forge`: + +```bash +cd mcp-context-forge +pip install /path/to/cpex-plugins/plugins/rust/python-package/ +# Configure plugin in plugins/config.yaml +pytest tests/integration/ # Run gateway integration tests +pytest tests/e2e/ # Run E2E tests +``` + +See [TESTING.md](TESTING.md) for cross-repository testing workflow. + +## Documentation + +- [AGENTS.md](AGENTS.md) - AI coding assistant guidelines +- [DEVELOPING.md](DEVELOPING.md) - Plugin development workflows +- [TESTING.md](TESTING.md) - Testing strategy and guidelines +- [CONTRIBUTING.md](CONTRIBUTING.md) - Contribution guidelines +- [SECURITY.md](SECURITY.md) - Security policy \ No newline at end of file diff --git a/TESTING.md b/TESTING.md index 6d99f98..12c490a 100644 --- a/TESTING.md +++ b/TESTING.md @@ -1,8 +1,105 @@ # Testing cpex-plugins -Testing is split into two layers: +## Testing Architecture -## 1. Repo Contract Tests +Testing spans two repositories. `cpex-plugins` owns unit tests and plugin-framework integration tests; `mcp-context-forge` owns gateway integration and E2E tests. + +### Unit Tests (cpex-plugins) + +**Location**: Within each plugin's own directory +- Python: `plugins/rust/python-package//tests/` (current hybrid) or `plugins/python//tests/` (pure Python) +- Rust: inline `mod tests` within source files (e.g., `src/lib.rs`) + +**Scope**: +- Individual plugin functionality in isolation +- Rust core logic and functions +- Python bindings and entry points +- Plugin configuration validation +- Fast, deterministic tests + +**Purpose**: +- Provide fast feedback during plugin development +- Validate plugin logic independently of the gateway +- Ensure plugin contracts are met +- Test edge cases and error handling + +**Run Locally**: +```bash +cd plugins/rust/python-package/ +make test-all # Runs both Rust and Python unit tests +``` + +### Plugin-Framework Integration Tests (cpex-plugins) + +**Location**: `cpex-plugins/tests/` (plugin-specific `tests/` directories) + +**Scope**: +- PyO3 entry points and Python ↔ Rust interface +- Plugin loading by the Python plugin framework +- Hook dispatch through the framework layer +- Coverage of PyO3 paths (run as part of Rust coverage) + +**Purpose**: +- Validate that the Rust implementation is correctly exposed through PyO3 bindings +- Ensure the plugin framework can discover, load, and invoke the plugin +- Keep PyO3 code paths covered without requiring a full gateway + +**Run Locally**: +```bash +cd plugins/rust/python-package/ +make test-integration # Runs plugin-framework integration tests +``` + +### Gateway Integration Tests (mcp-context-forge) + +**Location**: `mcp-context-forge/tests/integration/` + +**Scope**: +- Plugin integration with the full gateway +- Plugin loading and initialization in gateway context +- Hook execution within the gateway framework +- Cross-plugin interactions +- Plugin lifecycle management + +**Purpose**: +- Validate plugin behavior within the gateway +- Test framework-plugin contracts at the gateway level +- Ensure plugins work together correctly +- Test plugin configuration and registration + +**Run Locally**: +```bash +cd mcp-context-forge +pytest tests/integration/ +``` + +### E2E Tests (mcp-context-forge) + +**Location**: `mcp-context-forge/tests/e2e/` + +**Scope**: +- Complete request/response workflows +- Realistic usage scenarios +- Multi-gateway plugin coordination +- Performance and load testing with plugins + +**Purpose**: +- Validate end-to-end functionality +- Test real-world usage patterns +- Ensure system-level correctness +- Catch integration issues + +**Run Locally**: +```bash +cd mcp-context-forge +pytest tests/e2e/ +``` + +## Testing Layers + +`cpex-plugins` has three local testing layers. Gateway integration and E2E tests live in `mcp-context-forge`. + +### 1. Repo Contract Tests These validate monorepo conventions and are enforced in CI before plugin builds run. @@ -23,9 +120,9 @@ They verify: - changed-plugin detection for CI - canonical release tag resolution -## 2. Plugin Tests +### 2. Plugin Unit Tests -Each plugin has its own Rust and Python test suite. +Each plugin has its own Rust and Python unit test suite. ```bash cd plugins/rust/python-package/rate_limiter @@ -45,7 +142,18 @@ make plugin-test PLUGIN=rate_limiter `make plugin-test` runs the selected plugin's `make ci` target, including stub verification, build, bench compilation where configured, install, and Python tests. -## 3. Rust Coverage +### 3. Plugin-Framework Integration Tests + +Each plugin also has integration tests between the plugin and the Python plugin framework. These live in the plugin's `tests/` directory alongside unit tests and test the PyO3 interface — ensuring the Rust implementation is correctly exposed through Python bindings and that the framework can discover, load, and invoke the plugin. + +```bash +cd plugins/rust/python-package/rate_limiter +make test-integration +``` + +These tests are distinct from gateway integration tests in `mcp-context-forge`: they exercise the plugin ↔ framework boundary within this repository, without requiring a running gateway. + +## 4. Rust Coverage CI enforces at least 90% line coverage for each Rust plugin selected by the plugin catalog. The coverage job instruments Rust, runs Rust unit tests, then runs each plugin's repo-level Python integration tests so PyO3 paths are counted. @@ -85,7 +193,7 @@ Rust unit tests use `cargo nextest run`. Coverage uses `cargo llvm-cov nextest - Criterion benchmarks are verified in CI with `cargo nextest run --benches -E 'kind(bench)' --no-run`, which compiles benchmark test targets without rerunning normal unit tests or collecting noisy performance measurements on shared CI runners. -## 4. Mutation Testing +## 5. Mutation Testing Mutation testing runs in PR CI on Ubuntu for Rust code touched by the pull request diff. It is also available locally through cargo-mutants and runs Rust tests with nextest. @@ -98,6 +206,120 @@ make plugin-mutants PLUGIN=retry_with_backoff `.cargo/mutants.toml` sets `test_tool = "nextest"`, selects the `mutants` nextest profile, and keeps `cap_lints = false` so Rust warnings are not downgraded during mutant builds. The `mutants` profile keeps fail-fast enabled because cargo-mutants only needs one failing test to mark a mutant as caught. CI installs `cargo-mutants` with `cargo install cargo-mutants --version 27.0.0 --locked` and runs `cargo mutants "${cargo_args[@]}"`, using `--in-diff cargo-mutants.diff` for Rust source changes and full-package mutation for mutation-tooling config changes. +## Cross-Repository Testing Workflow + +### Development Workflow + +1. **Develop Plugin in cpex-plugins**: + ```bash + cd cpex-plugins/plugins/rust/python-package/ + # Implement plugin logic + # Write unit tests in tests/ + # Write plugin-framework integration tests in tests/ + make test-all # Run unit tests + make test-integration # Run plugin-framework integration tests + ``` + +2. **Create PR in cpex-plugins**: + - Include unit tests and plugin-framework integration tests + - Ensure `make ci` passes + - Get PR reviewed and merged + +3. **Coordinate with mcp-context-forge**: + - Notify mcp-context-forge team of new plugin + - Discuss integration test requirements + - Plan E2E test scenarios + +4. **Write Integration Tests in mcp-context-forge**: + ```bash + cd mcp-context-forge + # Install plugin: pip install cpex- + # Configure in plugins/config.yaml + # Write tests in tests/integration/ + pytest tests/integration/ + ``` + +5. **Write E2E Tests in mcp-context-forge**: + ```bash + cd mcp-context-forge + # Write tests in tests/e2e/ + pytest tests/e2e/ + ``` + +6. **Create PR in mcp-context-forge**: + - Include integration and E2E tests + - Ensure all tests pass + - Get PR reviewed and merged + +7. **Release**: + - Tag plugin in cpex-plugins: `-v` + - Update mcp-context-forge dependencies + - Deploy with new plugin version + +### Testing Coordination Guidelines + +**When to Write Unit Tests (cpex-plugins)**: +- Testing plugin logic in isolation +- Testing Rust functions and algorithms +- Testing Python bindings +- Testing configuration validation +- Testing error handling and edge cases + +**When to Write Plugin-Framework Integration Tests (cpex-plugins)**: +- Testing PyO3 entry points end-to-end +- Testing plugin loading by the Python framework +- Testing hook dispatch through the framework layer +- Ensuring PyO3 paths are covered in Rust coverage + +**When to Write Gateway Integration Tests (mcp-context-forge)**: +- Testing plugin loading and initialization in the gateway +- Testing hook execution in the full gateway framework +- Testing plugin interactions with gateway services +- Testing cross-plugin behavior +- Testing plugin lifecycle (enable/disable/reload) + +**When to Write E2E Tests (mcp-context-forge)**: +- Testing complete request/response flows +- Testing realistic usage scenarios +- Testing performance with plugins enabled +- Testing multi-gateway coordination +- Testing production-like configurations + +### CI Coordination + +**cpex-plugins CI**: +- Runs repo contract tests +- Runs plugin unit tests +- Runs plugin-framework integration tests (`make test-integration`) +- Builds and packages plugins +- Publishes to PyPI on release tags + +**mcp-context-forge CI**: +- Runs integration tests with latest plugin versions +- Runs E2E tests with plugins enabled +- Validates plugin compatibility +- Tests plugin upgrades + +### Test Coverage Expectations + +**Unit Tests (cpex-plugins)**: +- Aim for >90% code coverage of plugin logic +- Cover all public APIs and entry points +- Test error paths and edge cases +- Fast execution (<1 second per test) + +**Integration Tests (mcp-context-forge)**: +- Cover all plugin hooks +- Test plugin configuration variations +- Test plugin interactions +- Moderate execution time (<5 seconds per test) + +**E2E Tests (mcp-context-forge)**: +- Cover critical user workflows +- Test realistic scenarios +- Test performance characteristics +- Slower execution acceptable (seconds to minutes) + ## CI Behavior Repo contract tests run in their own CI workflow. The Rust plugin CI workflow uses the same plugin catalog to select affected plugin build, integration, and coverage jobs. @@ -108,3 +330,121 @@ Per-plugin build/test jobs are then scoped by the plugin catalog: - shared workflow, workspace, root orchestration, docs, test, and tool changes run all managed plugin jobs Release CI validates the tag and plugin metadata before any artifact is published. + +## Testing Best Practices + +### Unit Tests + +- **Fast**: Each test should complete in milliseconds +- **Isolated**: No external dependencies (network, filesystem, database) +- **Deterministic**: Same input always produces same output +- **Focused**: Test one thing per test +- **Clear**: Test names describe what is being tested + +### Integration Tests + +- **Realistic**: Use actual gateway framework components +- **Scoped**: Test specific integration points +- **Stable**: Use test fixtures and mocks for external services +- **Documented**: Explain what integration is being tested + +### E2E Tests + +- **Complete**: Test full workflows from start to finish +- **Representative**: Use realistic data and scenarios +- **Robust**: Handle timing and async operations correctly +- **Maintainable**: Use page objects and test helpers + +## Running Tests + +### Local Development + +```bash +# In cpex-plugins +cd plugins/rust/python-package/ +make test-all # Run unit tests (Rust + Python) +make test-integration # Run plugin-framework integration tests + +# In mcp-context-forge +cd mcp-context-forge +pytest tests/integration/ # Run gateway integration tests +pytest tests/e2e/ # Run E2E tests +``` + +### CI Pipeline + +```bash +# cpex-plugins CI +make plugins-validate # Validate repo structure +make plugin-test PLUGIN= # Run unit tests for specific plugin +# make test-integration is run as part of the coverage job + +# mcp-context-forge CI +make test # Run unit tests +pytest tests/integration/ # Run gateway integration tests +pytest tests/e2e/ # Run E2E tests +``` + +## Debugging Test Failures + +### Unit Test Failures (cpex-plugins) + +1. Run tests locally: `make test-all` +2. Check Rust test output: `cargo test -- --nocapture` +3. Check Python test output: `pytest -v` +4. Use debugger: `rust-gdb` or `pdb` + +### Plugin-Framework Integration Test Failures (cpex-plugins) + +1. Run tests locally: `make test-integration` +2. Check PyO3 binding output: `pytest -v tests/` +3. Verify Rust extension is built: `make install` +4. Check framework loading: `pytest -vv tests/` + +### Gateway Integration Test Failures (mcp-context-forge) + +1. Check plugin installation: `pip list | grep cpex` +2. Verify plugin configuration: `cat plugins/config.yaml` +3. Check gateway logs: `tail -f logs/gateway.log` +4. Run with verbose output: `pytest -vv tests/integration/` + +### E2E Test Failures (mcp-context-forge) + +1. Check full system logs +2. Verify all services are running +3. Check network connectivity +4. Run with debug logging: `LOG_LEVEL=DEBUG pytest tests/e2e/` + +## Test Documentation + +For detailed testing conventions in mcp-context-forge, see: +- `mcp-context-forge/tests/AGENTS.md` - Testing conventions and workflows +- `mcp-context-forge/plugins/AGENTS.md` - Plugin framework testing + +## Future: Pure Rust Testing + +After the plugin framework is migrated to Rust: + +### Unit Tests (cpex-plugins) + +```bash +cd plugins/rust/ +cargo test # Run Rust tests +cargo test -- --nocapture # With output +``` + +### Integration Tests (mcp-context-forge) + +```bash +cd mcp-context-forge +cargo test --test integration # Run integration tests +``` + +### E2E Tests (mcp-context-forge) + +```bash +cd mcp-context-forge +cargo test --test e2e # Run E2E tests +``` + +Python test infrastructure will be removed after framework migration. \ No newline at end of file