diff --git a/reflexio/server/README.md b/reflexio/server/README.md index 53e7d567..bbb5e59f 100644 --- a/reflexio/server/README.md +++ b/reflexio/server/README.md @@ -72,7 +72,7 @@ Description: FastAPI backend server that processes user interactions to generate - **Identity/config**: `GET /api/whoami`, `GET /api/my_config`, `GET /api/get_config`, `POST /api/set_config`, `POST /api/update_config` - **Publish/direct writes**: `POST /api/publish_interaction`, `POST /api/add_user_profile`, `POST /api/add_user_playbook`, `POST /api/add_agent_playbook` - **Retrieval**: `POST /api/get_requests`, `POST /api/get_interactions`, `GET /api/get_all_interactions`, `POST /api/get_profiles`, `GET /api/get_all_profiles`, `POST /api/get_user_playbooks`, `POST /api/get_agent_playbooks`, `POST /api/get_agent_success_evaluation_results` -- **Search/stats**: `POST /api/search`, `POST /api/search_profiles`, `POST /api/rerank_user_profiles`, `POST /api/search_interactions`, `POST /api/search_user_playbooks`, `POST /api/search_agent_playbooks`, `GET /api/storage_stats`, `GET /api/get_profile_statistics`, `POST /api/get_dashboard_stats`, `POST /api/get_playbook_application_stats` +- **Search/stats/provenance**: `POST /api/search`, `POST /api/search_profiles`, `POST /api/rerank_user_profiles`, `POST /api/search_interactions`, `POST /api/search_user_playbooks`, `POST /api/search_agent_playbooks`, `POST /api/get_learning_provenance`, `GET /api/storage_stats`, `GET /api/get_profile_statistics`, `POST /api/get_dashboard_stats`, `POST /api/get_playbook_application_stats` - **Profile lifecycle**: `POST /api/rerun_profile_generation`, `POST /api/manual_profile_generation`, `POST /api/upgrade_all_profiles`, `POST /api/downgrade_all_profiles`, `GET /api/profile_change_log`, `PUT /api/update_user_profile`, `DELETE /api/delete_profile`, `DELETE /api/delete_profiles_by_ids`, `DELETE /api/delete_all_profiles` - **Playbook lifecycle**: `POST /api/rerun_playbook_generation`, `POST /api/manual_playbook_generation`, `POST /api/run_playbook_aggregation`, `GET /api/playbook_aggregation_change_logs`, `POST /api/upgrade_all_user_playbooks`, `POST /api/downgrade_all_user_playbooks`, `PUT /api/update_agent_playbook_status`, `PUT /api/update_agent_playbook`, `PUT /api/update_user_playbook`, `DELETE /api/delete_agent_playbook`, `DELETE /api/delete_user_playbook`, `DELETE /api/delete_agent_playbooks_by_ids`, `DELETE /api/delete_user_playbooks_by_ids`, `DELETE /api/delete_all_playbooks`, `DELETE /api/delete_all_user_playbooks`, `DELETE /api/delete_all_agent_playbooks` - **Evaluation**: `POST /api/get_evaluation_overview`, `POST /api/evaluations/regenerate`, `GET /api/evaluations/regenerate/{job_id}`, `DELETE /api/evaluations/regenerate/{job_id}`, `POST /api/evaluations/grade_on_demand`, `GET /api/evaluations/shadow_comparisons/recent` @@ -185,7 +185,8 @@ python -m reflexio.server.scripts.manage_invitation_codes list --show-used - **Playbook memory**: `playbook/` extracts user playbooks, consolidates them against existing rows, aggregates them into agent playbooks, and tracks aggregation change logs. - **Evaluation**: `agent_success_evaluation/service.py`, `agent_success_evaluation/runner.py`, `agent_success_evaluation/scheduler.py`, `agent_success_evaluation/components/evaluator.py`, `shadow_comparison/`, and `evaluation_overview/` handle session grading, per-turn shadow verdicts, regeneration jobs, and dashboard-facing rollups. - **Async clarification**: `extraction/` and `reflection/` manage resumable agent runs, pending tool calls, prior-answer search, and long-horizon reflection updates. -- **Search preparation**: `pre_retrieval/` and `unified_search_service.py` handle query reformulation, document expansion, embeddings, and cross-entity search orchestration. +- **Search preparation**: `pre_retrieval/`, `retrieval/`, and `unified_search_service.py` handle query reformulation, document expansion, relevance/recency ranking, embeddings, and cross-entity search orchestration. +- **Governance**: `governance/` coordinates user export/erasure, audit events, privacy-safe subject references, and idempotent purge/rebuild workflows over storage governance primitives. - **Optimization/integrations**: `playbook_optimizer/` and `braintrust/` run candidate playbook optimization, rollout support, and Braintrust export/sync. - **Lineage**: `lineage/` resolves active records across superseded chains and schedules tombstone garbage collection for profile/playbook storage. - **Persistence/config**: `storage/`, `configurator/`, and `operation_state_utils.py` provide storage abstractions, config loading, locks, bookmarks, progress, and cancellation. @@ -477,8 +478,8 @@ Pre-computed embeddings passed to storage methods via `query_embedding` paramete | File | Purpose | |------|---------| -| `storage_base/` | BaseStorage interface split by domain (`_profiles.py`, `_playbook.py`, `_requests.py`, `_operations.py`, `_agent_run.py`, `_lineage.py`, `_shadow_verdicts.py`, `_stall_state.py`, `_share_links.py`) | -| `sqlite_storage/` | SQLite-backed implementation split across the same domains, including lineage/tombstone support in `_lineage.py` | +| `storage_base/` | BaseStorage interface split by domain (`_profiles.py`, `_playbook.py`, `_requests.py`, `_operations.py`, `_agent_run.py`, `_governance.py`, `_lineage.py`, `_shadow_verdicts.py`, `_stall_state.py`, `_share_links.py`) | +| `sqlite_storage/` | SQLite-backed implementation split across the same domains, including governance/audit tables in `_governance.py` and lineage/tombstone support in `_lineage.py` | | `retention.py`, `retention_mixin.py` | Data retention and cleanup helpers | | `constants.py`, `error.py` | Storage constants and shared errors | @@ -630,6 +631,7 @@ All services follow BaseGenerationService: - [Code Map (root README)](../README.md) -- high-level overview of all Reflexio components - [API Endpoints README](api_endpoints/README.md) -- RequestContext contract and handler/helper map - [Services README](services/README.md) -- per-directory index of the business-logic layer +- [Governance Service README](services/governance/README.md) -- user export/erasure, audit, and purge workflow map - [Prompt Bank README](prompt/prompt_bank/README.md) -- versioned prompt template system - [Playbook Service README](services/playbook/README.md) -- playbook extraction, aggregation, and deduplication pipeline - [Site Variables README](site_var/README.md) -- global configuration and feature flags diff --git a/reflexio/server/api_endpoints/README.md b/reflexio/server/api_endpoints/README.md index b191ae9e..4df89da6 100644 --- a/reflexio/server/api_endpoints/README.md +++ b/reflexio/server/api_endpoints/README.md @@ -25,7 +25,8 @@ api.py (core_router + sub-routers) - **`RequestContext` is the context-passing contract** — handlers receive it via `Depends` and never reach for storage/config/prompts globally. - **Route handlers call `Reflexio` through `get_reflexio(org_id)`** — these helper files do **not** instantiate `Reflexio()` directly. -- **Auth is injected, not implemented here** — the OS app uses `default_get_org_id` / `DEFAULT_ORG_ID`; the enterprise extension swaps in authenticated org resolution (see `reflexio_ext/server/api_endpoints/`). +- **Learning provenance currently lives in `../api.py`** — `POST /api/get_learning_provenance` reconstructs provenance for profiles/user playbooks/agent playbooks from storage and generation metadata. +- **Auth is injected, not implemented here** — the OSS app uses `default_get_org_id` / `DEFAULT_ORG_ID`; the enterprise extension swaps in authenticated org resolution (see `reflexio_ext/server/api_endpoints/`). ## Requirements / Problems to Avoid diff --git a/reflexio/server/services/README.md b/reflexio/server/services/README.md index f90be9e4..cee1711b 100644 --- a/reflexio/server/services/README.md +++ b/reflexio/server/services/README.md @@ -51,17 +51,18 @@ strings before deleting old import paths in the same PR. | `evaluation_overview/` | Dashboard/read-side rollups: `service.py` entry point, `components/` aggregation helpers, and root `eval_sampler.py` shared with regenerate jobs. See [README](evaluation_overview/README.md). | | `playbook_optimizer/` | Scenario-based playbook optimization: mature flat package with `optimizer.py`, `scheduler.py`, `rollout.py`, `judge.py`, `models.py`, `scenario_resolver.py`, `gepa_adapter.py`, and `assistant_webhook.py`. See [README](playbook_optimizer/README.md). | | `braintrust/` | Braintrust export/sync: `service.py`, `client.py`, `_cron.py`, `_encryption.py`. | +| `governance/` | `GovernanceService` for user export/erasure, audit events, privacy-safe subject references, and idempotent purge workflows. See [README](governance/README.md). | | `lineage/` | Current-record resolution and tombstone GC: `resolve.py`, `gc_scheduler.py`. | | `pre_retrieval/` | `QueryReformulator` (`_query_reformulator.py`) + `DocumentExpander` (`_document_expander.py`) - query rewrite and doc expansion for recall. Compact by design; see [README](pre_retrieval/README.md). | | `tagging/` | `TaggingService` (`service.py`) + deferred `tagging_scheduler.py` - post-generation profile/playbook tagging. Compact by design; see [README](tagging/README.md). | | `unified_search_service.py` | `run_unified_search()` — two-phase parallel search across profiles / agent playbooks / user playbooks. | -| `retrieval/` | `relevance_floor.py` — result relevance thresholding. | +| `retrieval/` | `relevance_floor.py` and `recency.py` — result relevance thresholding plus recency-aware ranking helpers. | ## Persistence & Config | Path | Purpose | |------|---------| -| `storage/` | `storage_base/` (`BaseStorage` split by domain, including `_lineage.py`) + `sqlite_storage/` (including lineage/tombstones) + `retention*.py`. Access via `request_context.storage` only. | +| `storage/` | `storage_base/` (`BaseStorage` split by domain, including `_governance.py` and `_lineage.py`) + `sqlite_storage/` (including governance/audit tables and lineage/tombstones) + `retention*.py`. Access via `request_context.storage` only. | | `configurator/` | `DefaultConfigurator` — loads YAML config and creates the storage backend. | ## Key Rules diff --git a/reflexio/server/services/governance/README.md b/reflexio/server/services/governance/README.md new file mode 100644 index 00000000..98ab89aa --- /dev/null +++ b/reflexio/server/services/governance/README.md @@ -0,0 +1,40 @@ +# server/services/governance +Description: Governance service layer for user data export, erasure, audit events, and idempotent purge operation tracking. + +## Main Entry Points + +| File | Purpose | +|------|---------| +| `service.py` | `GovernanceService` orchestrates user export and user erasure workflows against `request_context.storage`. | +| `subject_refs.py` | Stable privacy-safe reference helpers (`subject_ref`, `request_ref`, `stable_id`) used in audit and purge records. | + +## Purpose + +1. **User export** - Build a bundle of a user's profiles, interactions, requests, sessions, and user playbooks, then append an audit event. +2. **User erasure** - Prepare idempotent purge targets, delete user-owned rows, hide/rebuild impacted agent playbooks, and complete or fail the purge with audit state. +3. **Privacy-safe bookkeeping** - Store hashed subject/request references instead of raw user or request identifiers in governance audit/purge rows. + +## Architecture Pattern + +`GovernanceService` is a focused orchestration layer over storage primitives. It does not own persistence directly; backend-neutral contracts live in `storage/storage_base/_governance.py` and the SQLite implementation lives in `storage/sqlite_storage/_governance.py`. + +``` +GovernanceService + -> storage.begin_purge_operation / prepare_governance_erase_targets + -> storage.apply_governance_user_data_delete + -> storage.apply_governance_agent_playbook_rebuild + -> storage.complete_purge_operation_with_audit +``` + +## Key Contracts + +- `export_user(user_id, request_id) -> UserExportResult` +- `erase_user(user_id, request_id) -> UserEraseResult` +- Domain schemas: `models/api_schema/domain/governance.py` +- Retention config: `Config.governance_retention` / `GovernanceRetentionConfig` + +## Requirements / Problems to Avoid + +- **Do not store raw identifiers in audit/purge references** — use `subject_refs.py` helpers. +- **Do not bypass storage governance methods** — purge target preparation, deletion, rebuild, and audit completion are idempotency-sensitive. +- **Keep governance storage backend-neutral** — add abstract methods to `storage_base/_governance.py` before implementation-specific code.