diff --git a/docs/testing/e2e_testing.md b/docs/testing/e2e_testing.md index 50d442bec..0305b8dd4 100644 --- a/docs/testing/e2e_testing.md +++ b/docs/testing/e2e_testing.md @@ -15,8 +15,9 @@ This guide describes how to run, extend, and understand the Lightspeed Core Stac 7. [Configuration Files](#configuration-files) 8. [Feature Files and Steps](#feature-files-and-steps) 9. [Gherkin Keywords in Feature Files](#gherkin-keywords-in-feature-files) -10. [Writing New Scenarios](#writing-new-scenarios) -11. [Troubleshooting](#troubleshooting) +10. [Choosing the Test Layer: E2E or Integration?](#choosing-the-test-layer-e2e-or-integration) +11. [Writing New Scenarios](#writing-new-scenarios) +12. [Troubleshooting](#troubleshooting) --- @@ -336,9 +337,50 @@ Here, **Given** sets state, **When** performs the HTTP call, **Then** and **And* --- +## Choosing the Test Layer: E2E or Integration? + +Before writing a scenario, decide whether it belongs here at all. The suite has +three layers, and the boundary between the top two is strict: + +| Layer | Location | Talks to | May touch `src/`? | +|---|---|---|---| +| Unit | `tests/unit/` | one function or class, everything else mocked | yes | +| Integration | `tests/integration/` (pytest) | real configuration loading, real database, real pipelines in-process; external services (OGX, LLM providers) mocked; repo CLIs as subprocesses | yes | +| E2E | `tests/e2e/` (behave) | a deployed stack, through its public surfaces only: the HTTP API, container lifecycle and logs, configuration files the harness applies | **never** | + +The rule for e2e: **a step definition must not import from, invoke, or shell out +to anything under `src/`.** The moment it does, the scenario stops proving what a +deployed stack does and starts proving what a checked-out source tree does — that +is an integration test, and it belongs in `tests/integration/` as pytest, where it +runs in seconds without Docker. + +A quick test: *could this scenario run unchanged against a container image, with +no source checkout on the machine?* If yes, it is e2e. If it needs +`src/lightspeed_stack.py`, `src/ogx_configuration.py`, a Python import from the +service, or a subprocess of a repo entrypoint, it is integration. + +Typical consequences: + +- Configuration **validation**, **migration** (`--migrate-config`) and run.yaml + **synthesis** are integration concerns: they exercise CLIs and the config + pipeline, not a running service. See `tests/integration/test_unified_synthesis.py` + and `tests/integration/test_unified_mode_cli.py`. +- **Boot** scenarios (apply a config, restart, hit `readiness` and `query`) and + **log-evidence** scenarios (`docker logs `) are e2e: they observe the + deployed stack from outside. +- If a scenario needs a generated artifact as its starting point (for example a + migrated configuration), commit the artifact as a fixture and add an + integration test that guards it against drift, rather than generating it inside + the e2e step. + +The integration side of this boundary is described in +[tests/integration/README.md](../../tests/integration/README.md#what-to-test). + +--- + ## Writing New Scenarios -1. **Choose or add a feature file** under `tests/e2e/features/` and use existing steps where possible. If you add a new file, **add it to `tests/e2e/test_list.txt`** so the suite runs it. +1. **Confirm the scenario is e2e at all** — see [Choosing the Test Layer](#choosing-the-test-layer-e2e-or-integration). Then **choose or add a feature file** under `tests/e2e/features/` and use existing steps where possible. If you add a new file, **add it to `tests/e2e/test_list.txt`** so the suite runs it. 2. **Use tags** for mode-dependent or config-dependent behavior (`@skip-in-library-mode`, `@Authorized`, etc.). **Adding a tag that switches configuration** (e.g. a new feature-level or scenario-level config) usually means you must also add or change a **Lightspeed Stack config** file under `configuration/server-mode/` or `library-mode/` and wire the tag in `environment.py` (e.g. in `before_feature` / `after_feature` or `before_scenario` / `after_scenario`) so the config is applied and the container restarted when the tag is active. 3. **Use placeholders** `{MODEL}` and `{PROVIDER}` in request bodies so the same scenario works with different backends. 4. **Add step definitions** in the appropriate `features/steps/*.py` if you need new steps; reuse `context` for host, port, auth, and responses. diff --git a/docs/testing/testing.md b/docs/testing/testing.md index 4331f1dd8..cee759c36 100644 --- a/docs/testing/testing.md +++ b/docs/testing/testing.md @@ -132,6 +132,7 @@ As specified in Definition of Done, new changes need to be covered by tests. Integration tests are based on the [Pytest framework](https://docs.pytest.org/en/) and code coverage is measured by the plugin [pytest-cov](https://github.com/pytest-dev/pytest-cov). For mocking and patching, the [unittest framework](https://docs.python.org/3/library/unittest.html) is used. * Defined in [tests/integration](https://github.com/lightspeed-core/lightspeed-stack/tree/main/tests/integration) +* **Integration or e2e?** Integration tests may touch `src/` (in-process pipelines, repo CLIs as subprocesses); e2e tests never do. See [Choosing the Test Layer](e2e_testing.md#choosing-the-test-layer-e2e-or-integration). diff --git a/src/ogx_configuration.py b/src/ogx_configuration.py index d5092bc41..bf6b06457 100644 --- a/src/ogx_configuration.py +++ b/src/ogx_configuration.py @@ -30,7 +30,7 @@ from pydantic import SecretStr import constants -from log import get_logger +from log import get_logger, setup_logging logger = get_logger(__name__) @@ -1550,6 +1550,10 @@ def main() -> None: run.yaml needs to exist; otherwise the legacy path enriches the ``--input`` run.yaml in place. Server-mode container entrypoints rely on this dispatch to serve both modes with a single invocation. + + Configures logging first so the INFO lines this module emits reach the + container log: run as a bare script there is no handler on the root + logger, and ``logging.lastResort`` would drop everything below WARNING. """ parser = ArgumentParser( description="Generate the OGX run configuration from a " @@ -1577,6 +1581,15 @@ def main() -> None: ) args = parser.parse_args() + # Configure logging before doing any work. This module runs as a bare + # script from the container entrypoint (scripts/ogx-entrypoint.sh), so + # nothing has installed a handler on the root logger; Python's lastResort + # then emits WARNING and above only, and every INFO line this module + # writes -- including which config shape was detected and where the + # synthesized run.yaml was written -- is silently dropped. AsyncOgxClient + # already does this for the in-process path, for the same reason. + setup_logging() + with open(args.config, encoding="utf-8") as f: config = yaml.safe_load(f) diff --git a/tests/configuration/unified-mode/lightspeed-stack-invalid-config-and-legacy.yaml b/tests/configuration/unified-mode/lightspeed-stack-invalid-config-and-legacy.yaml new file mode 100644 index 000000000..a33693698 --- /dev/null +++ b/tests/configuration/unified-mode/lightspeed-stack-invalid-config-and-legacy.yaml @@ -0,0 +1,25 @@ +name: Lightspeed Core Service (LCS) +service: + host: 0.0.0.0 + port: 8080 + auth_enabled: false + workers: 1 + color_log: true + access_log: true +llama_stack: + # Library mode - embeds the stack in-process + use_as_library_client: true + config: + profile: tests/configuration/run.yaml + # INVALID: config block plus the legacy path (mutual exclusion, R3) + library_client_config_path: tests/configuration/run.yaml +user_data_collection: + feedback_enabled: true + feedback_storage: "/tmp/data/feedback" + transcripts_enabled: true + transcripts_storage: "/tmp/data/transcripts" +authentication: + module: "noop" +inference: + default_provider: openai + default_model: gpt-4o-mini diff --git a/tests/configuration/unified-mode/lightspeed-stack-invalid-providers-and-legacy.yaml b/tests/configuration/unified-mode/lightspeed-stack-invalid-providers-and-legacy.yaml new file mode 100644 index 000000000..19c299ab9 --- /dev/null +++ b/tests/configuration/unified-mode/lightspeed-stack-invalid-providers-and-legacy.yaml @@ -0,0 +1,31 @@ +name: Lightspeed Core Service (LCS) +service: + host: 0.0.0.0 + port: 8080 + auth_enabled: false + workers: 1 + color_log: true + access_log: true +llama_stack: + # Library mode - embeds the stack in-process + use_as_library_client: true + # INVALID: synthesis input plus the legacy path (mutual exclusion, R3) + library_client_config_path: tests/configuration/run.yaml +user_data_collection: + feedback_enabled: true + feedback_storage: "/tmp/data/feedback" + transcripts_enabled: true + transcripts_storage: "/tmp/data/transcripts" +authentication: + module: "noop" +inference: + default_provider: openai + default_model: gpt-4o-mini + # Unified synthesis input (Decision S5): the high-level provider entry + # replaces the default baseline's openai provider by id at synthesis time. + providers: + - type: openai + id: openai + api_key_env: OPENAI_API_KEY + allowed_models: + - ${env.E2E_OPENAI_MODEL:=gpt-4o-mini} diff --git a/tests/configuration/unified-mode/lightspeed-stack-invalid-version-legacy-unified-body.yaml b/tests/configuration/unified-mode/lightspeed-stack-invalid-version-legacy-unified-body.yaml new file mode 100644 index 000000000..9ae7b389d --- /dev/null +++ b/tests/configuration/unified-mode/lightspeed-stack-invalid-version-legacy-unified-body.yaml @@ -0,0 +1,31 @@ +name: Lightspeed Core Service (LCS) +service: + host: 0.0.0.0 + port: 8080 + auth_enabled: false + workers: 1 + color_log: true + access_log: true +llama_stack: + # Library mode - embeds the stack in-process + use_as_library_client: true +user_data_collection: + feedback_enabled: true + feedback_storage: "/tmp/data/feedback" + transcripts_enabled: true + transcripts_storage: "/tmp/data/transcripts" +authentication: + module: "noop" +inference: + default_provider: openai + default_model: gpt-4o-mini + # Unified synthesis input (Decision S5): the high-level provider entry + # replaces the default baseline's openai provider by id at synthesis time. + providers: + - type: openai + id: openai + api_key_env: OPENAI_API_KEY + allowed_models: + - ${env.E2E_OPENAI_MODEL:=gpt-4o-mini} +# INVALID: explicit legacy marker on a unified-shaped body (R11, LCORE-2872) +config_format_version: legacy diff --git a/tests/configuration/unified-mode/lightspeed-stack-legacy-for-migration.yaml b/tests/configuration/unified-mode/lightspeed-stack-legacy-for-migration.yaml new file mode 100644 index 000000000..396b24fcb --- /dev/null +++ b/tests/configuration/unified-mode/lightspeed-stack-legacy-for-migration.yaml @@ -0,0 +1,23 @@ +name: Lightspeed Core Service (LCS) +service: + host: 0.0.0.0 + port: 8080 + auth_enabled: false + workers: 1 + color_log: true + access_log: true +llama_stack: + # Library mode - embeds the stack in-process + use_as_library_client: true + # Legacy two-file shape: external run.yaml, no synthesis input + library_client_config_path: tests/e2e/configs/run-ci.yaml +user_data_collection: + feedback_enabled: true + feedback_storage: "/tmp/data/feedback" + transcripts_enabled: true + transcripts_storage: "/tmp/data/transcripts" +authentication: + module: "noop" +inference: + default_provider: openai + default_model: gpt-4o-mini diff --git a/tests/e2e/README.md b/tests/e2e/README.md index cdecbfd1f..e7f88ea80 100644 --- a/tests/e2e/README.md +++ b/tests/e2e/README.md @@ -2,8 +2,10 @@ End-to-end tests for the Lightspeed Core Stack REST API (Behave, Gherkin). -**Full guide:** [docs/e2e_testing.md](../../docs/e2e_testing.md) — how to run, environment variables, deployment modes, tags and hooks, Gherkin keywords, configuration, and troubleshooting. +**Full guide:** [docs/testing/e2e_testing.md](../../docs/testing/e2e_testing.md) — how to run, environment variables, deployment modes, tags and hooks, Gherkin keywords, configuration, and troubleshooting. * Tests: `tests/e2e/features/*.feature` * Step definitions: `tests/e2e/features/steps/` * Feature list (run order): `test_list.txt` + +**Not sure a scenario is e2e?** Steps must never touch `src/`; validation, migration and synthesis live in `tests/integration/`. See [Choosing the Test Layer](../../docs/testing/e2e_testing.md#choosing-the-test-layer-e2e-or-integration). diff --git a/tests/e2e/configuration/library-mode/lightspeed-stack-legacy.yaml b/tests/e2e/configuration/library-mode/lightspeed-stack-legacy.yaml new file mode 100644 index 000000000..6b4ea3c09 --- /dev/null +++ b/tests/e2e/configuration/library-mode/lightspeed-stack-legacy.yaml @@ -0,0 +1,47 @@ +name: Lightspeed Core Service (LCS) +service: + host: 0.0.0.0 + port: 8080 + auth_enabled: false + workers: 1 + color_log: true + access_log: true +llama_stack: + # Legacy two-file shape (R2 deprecation window): external run.yaml consumed + # via library_client_config_path; no unified synthesis input. Kept as a + # dedicated fixture because the standard library-mode baseline migrated to + # unified mode in LCORE-2342, which silently removed legacy boot coverage. + use_as_library_client: true + library_client_config_path: run.yaml +user_data_collection: + feedback_enabled: true + feedback_storage: "/tmp/data/feedback" + transcripts_enabled: true + transcripts_storage: "/tmp/data/transcripts" +authentication: + module: "noop" +inference: + default_provider: openai + default_model: gpt-4o-mini +rag: + byok: + stores: + - rag_id: e2e-test-docs + backend: faiss + embedding_model: sentence-transformers/all-mpnet-base-v2 + embedding_dimension: 768 + vector_db_id: ${env.FAISS_VECTOR_STORE_ID} + db_path: ${env.KV_RAG_PATH:=~/.llama/storage/rag/kv_store.db} + score_multiplier: 1.0 + retrieval: + tool: + sources: + - e2e-test-docs + +shields: + - name: pii-redaction + provider_id: redaction + config: + rules: + - pattern: '\d+' + replacement: '[NUM]' diff --git a/tests/e2e/configuration/unified-mode/README.md b/tests/e2e/configuration/unified-mode/README.md new file mode 100644 index 000000000..98131ba4f --- /dev/null +++ b/tests/e2e/configuration/unified-mode/README.md @@ -0,0 +1,25 @@ +# Unified-mode e2e configuration fixtures + +Fixtures for the `unified-mode-*.feature` files (LCORE-2341/LCORE-2343). +Same layout as the parent directory: `library-mode/` and `server-mode/` +variants differing only in the `llama_stack` block; the harness resolves +`//` via the standard `configure_service` logic. + +All profile-based fixtures reference `run.yaml` — the repo-root copy the CI +harness materializes from `tests/e2e/configs/run-.yaml` — so they stay +provider-agnostic across the providers matrix. + +Only bootable fixtures live here. The validation-only and synthesis-only +inputs (invalid configs, `native_override` shapes) belong to the integration +layer — `tests/configuration/unified-mode/` and +`tests/integration/test_unified_synthesis.py` — because e2e steps never run +`src/` CLIs (see `docs/testing/e2e_testing.md`, "Choosing the Test Layer"). + +| Fixture | Purpose | +|---|---| +| `lightspeed-stack-unified-providers.yaml` | Minimal unified config driven only by top-level `inference.providers` (default baseline, R1/S5). openai-specific — used by `@openai-only` scenarios. | +| `lightspeed-stack-unified-config-only.yaml` | Unified config driven only by `llama_stack.config` (`profile: run.yaml`, R1). | +| `lightspeed-stack-unified-relative-profile.yaml` | Same shape as config-only; exists to pin R8 (relative `profile:` resolves against the config file's directory) as a distinct intent. | +| `lightspeed-stack-unified-absolute-profile.yaml` | `profile:` as a container-absolute path (differs per mode subdir). | +| `lightspeed-stack-legacy-for-migration.yaml` | Legacy half of the migration fixture pair; paired with `tests/e2e/configs/run-ci.yaml`. Deliberately free of enrichment sections so migrate→synthesize round-trips losslessly (see LCORE-3370). Input to the drift guard below; never booted. | +| `lightspeed-stack-unified-migrated.yaml` | **Committed** output of `--migrate-config` for the pair above. Booted by `unified-mode-migration.feature` (`@openai-only`: it inlines the openai run-ci.yaml). `tests/integration/test_unified_mode_cli.py::test_committed_migrated_fixture_matches_cli_output` fails when the CLI output drifts; its docstring has the regeneration command. | diff --git a/tests/e2e/configuration/unified-mode/library-mode/lightspeed-stack-legacy-for-migration.yaml b/tests/e2e/configuration/unified-mode/library-mode/lightspeed-stack-legacy-for-migration.yaml new file mode 100644 index 000000000..6393142b5 --- /dev/null +++ b/tests/e2e/configuration/unified-mode/library-mode/lightspeed-stack-legacy-for-migration.yaml @@ -0,0 +1,23 @@ +name: Lightspeed Core Service (LCS) +service: + host: 0.0.0.0 + port: 8080 + auth_enabled: false + workers: 1 + color_log: true + access_log: true +llama_stack: + # Library mode - embeds the stack in-process + use_as_library_client: true + # Legacy two-file shape: external run.yaml, no synthesis input + library_client_config_path: run.yaml +user_data_collection: + feedback_enabled: true + feedback_storage: "/tmp/data/feedback" + transcripts_enabled: true + transcripts_storage: "/tmp/data/transcripts" +authentication: + module: "noop" +inference: + default_provider: openai + default_model: gpt-4o-mini diff --git a/tests/e2e/configuration/unified-mode/library-mode/lightspeed-stack-unified-absolute-profile.yaml b/tests/e2e/configuration/unified-mode/library-mode/lightspeed-stack-unified-absolute-profile.yaml new file mode 100644 index 000000000..089fb0afb --- /dev/null +++ b/tests/e2e/configuration/unified-mode/library-mode/lightspeed-stack-unified-absolute-profile.yaml @@ -0,0 +1,24 @@ +name: Lightspeed Core Service (LCS) +service: + host: 0.0.0.0 + port: 8080 + auth_enabled: false + workers: 1 + color_log: true + access_log: true +llama_stack: + # Library mode - embeds the stack in-process + use_as_library_client: true + config: + # Absolute path as mounted in the library-mode container + profile: /app-root/run.yaml +user_data_collection: + feedback_enabled: true + feedback_storage: "/tmp/data/feedback" + transcripts_enabled: true + transcripts_storage: "/tmp/data/transcripts" +authentication: + module: "noop" +inference: + default_provider: openai + default_model: gpt-4o-mini diff --git a/tests/e2e/configuration/unified-mode/library-mode/lightspeed-stack-unified-config-only.yaml b/tests/e2e/configuration/unified-mode/library-mode/lightspeed-stack-unified-config-only.yaml new file mode 100644 index 000000000..b3df26828 --- /dev/null +++ b/tests/e2e/configuration/unified-mode/library-mode/lightspeed-stack-unified-config-only.yaml @@ -0,0 +1,24 @@ +name: Lightspeed Core Service (LCS) +service: + host: 0.0.0.0 + port: 8080 + auth_enabled: false + workers: 1 + color_log: true + access_log: true +llama_stack: + # Library mode - embeds the stack in-process + use_as_library_client: true + config: + # Synthesis baseline: the CI-materialized run.yaml (provider-agnostic) + profile: run.yaml +user_data_collection: + feedback_enabled: true + feedback_storage: "/tmp/data/feedback" + transcripts_enabled: true + transcripts_storage: "/tmp/data/transcripts" +authentication: + module: "noop" +inference: + default_provider: openai + default_model: gpt-4o-mini diff --git a/tests/e2e/configuration/unified-mode/library-mode/lightspeed-stack-unified-migrated.yaml b/tests/e2e/configuration/unified-mode/library-mode/lightspeed-stack-unified-migrated.yaml new file mode 100644 index 000000000..576b1950a --- /dev/null +++ b/tests/e2e/configuration/unified-mode/library-mode/lightspeed-stack-unified-migrated.yaml @@ -0,0 +1,126 @@ +authentication: + module: noop +inference: + default_model: gpt-4o-mini + default_provider: openai +name: Lightspeed Core Service (LCS) +ogx: + config: + baseline: empty + native_override: + apis: + - responses + - batches + - files + - inference + - tool_runtime + - conversations + - vector_io + distro_name: starter + providers: + batches: + - config: + sqlstore: + backend: sql_default + table_name: batches + provider_id: reference + provider_type: inline::reference + files: + - config: + metadata_store: + backend: sql_default + table_name: files_metadata + storage_dir: ~/.llama/storage/files + provider_id: meta-reference-files + provider_type: inline::localfs + inference: + - config: + allowed_models: + - ${env.E2E_OPENAI_MODEL:=gpt-4o-mini} + api_key: ${env.OPENAI_API_KEY} + provider_id: openai + provider_type: remote::openai + - config: {} + provider_id: sentence-transformers + provider_type: inline::sentence-transformers + responses: + - config: + persistence: + responses: + backend: sql_default + table_name: agents_responses + provider_id: builtin + provider_type: inline::builtin + tool_runtime: + - config: {} + provider_id: file-search + provider_type: inline::file-search + - config: {} + provider_id: model-context-protocol + provider_type: remote::model-context-protocol + vector_io: + - config: + persistence: + backend: kv_rag + namespace: vector_io::faiss + provider_id: faiss + provider_type: inline::faiss + registered_resources: + models: + - metadata: + embedding_dimension: 768 + model_id: all-mpnet-base-v2 + model_type: embedding + provider_id: sentence-transformers + provider_model_id: all-mpnet-base-v2 + vector_stores: [] + server: + port: 8321 + storage: + backends: + kv_default: + db_path: ${env.KV_STORE_PATH:=~/.llama/storage/kv_store.db} + type: kv_sqlite + kv_rag: + db_path: ${env.KV_RAG_PATH:=~/.llama/storage/rag/kv_store.db} + type: kv_sqlite + sql_default: + db_path: ${env.SQL_STORE_PATH:=~/.llama/storage/sql_store.db} + type: sql_sqlite + stores: + connectors: + backend: sql_default + table_name: connectors + conversations: + backend: sql_default + table_name: openai_conversations + inference: + backend: sql_default + max_write_queue_size: 10000 + num_writers: 4 + table_name: inference_store + metadata: + backend: kv_default + namespace: registry + prompts: + backend: sql_default + table_name: prompts + vector_stores: + default_embedding_model: + model_id: all-mpnet-base-v2 + provider_id: sentence-transformers + default_provider_id: faiss + version: 2 + use_as_library_client: true +service: + access_log: true + auth_enabled: false + color_log: true + host: 0.0.0.0 + port: 8080 + workers: 1 +user_data_collection: + feedback_enabled: true + feedback_storage: /tmp/data/feedback + transcripts_enabled: true + transcripts_storage: /tmp/data/transcripts diff --git a/tests/e2e/configuration/unified-mode/library-mode/lightspeed-stack-unified-providers.yaml b/tests/e2e/configuration/unified-mode/library-mode/lightspeed-stack-unified-providers.yaml new file mode 100644 index 000000000..731c39b5d --- /dev/null +++ b/tests/e2e/configuration/unified-mode/library-mode/lightspeed-stack-unified-providers.yaml @@ -0,0 +1,29 @@ +name: Lightspeed Core Service (LCS) +service: + host: 0.0.0.0 + port: 8080 + auth_enabled: false + workers: 1 + color_log: true + access_log: true +llama_stack: + # Library mode - embeds the stack in-process + use_as_library_client: true +user_data_collection: + feedback_enabled: true + feedback_storage: "/tmp/data/feedback" + transcripts_enabled: true + transcripts_storage: "/tmp/data/transcripts" +authentication: + module: "noop" +inference: + default_provider: openai + default_model: gpt-4o-mini + # Unified synthesis input (Decision S5): the high-level provider entry + # replaces the default baseline's openai provider by id at synthesis time. + providers: + - type: openai + id: openai + api_key_env: OPENAI_API_KEY + allowed_models: + - ${env.E2E_OPENAI_MODEL:=gpt-4o-mini} diff --git a/tests/e2e/configuration/unified-mode/library-mode/lightspeed-stack-unified-relative-profile.yaml b/tests/e2e/configuration/unified-mode/library-mode/lightspeed-stack-unified-relative-profile.yaml new file mode 100644 index 000000000..228b2d40b --- /dev/null +++ b/tests/e2e/configuration/unified-mode/library-mode/lightspeed-stack-unified-relative-profile.yaml @@ -0,0 +1,24 @@ +name: Lightspeed Core Service (LCS) +service: + host: 0.0.0.0 + port: 8080 + auth_enabled: false + workers: 1 + color_log: true + access_log: true +llama_stack: + # Library mode - embeds the stack in-process + use_as_library_client: true + config: + # R8: relative profile resolves against this file's loaded location + profile: run.yaml +user_data_collection: + feedback_enabled: true + feedback_storage: "/tmp/data/feedback" + transcripts_enabled: true + transcripts_storage: "/tmp/data/transcripts" +authentication: + module: "noop" +inference: + default_provider: openai + default_model: gpt-4o-mini diff --git a/tests/e2e/configuration/unified-mode/server-mode/lightspeed-stack-legacy-for-migration.yaml b/tests/e2e/configuration/unified-mode/server-mode/lightspeed-stack-legacy-for-migration.yaml new file mode 100644 index 000000000..18890c9fc --- /dev/null +++ b/tests/e2e/configuration/unified-mode/server-mode/lightspeed-stack-legacy-for-migration.yaml @@ -0,0 +1,23 @@ +name: Lightspeed Core Service (LCS) +service: + host: 0.0.0.0 + port: 8080 + auth_enabled: false + workers: 1 + color_log: true + access_log: true +llama_stack: + # Server mode - connects to the separate ogx service + use_as_library_client: false + url: http://${env.E2E_OGX_HOSTNAME}:8321 + api_key: xyzzy +user_data_collection: + feedback_enabled: true + feedback_storage: "/tmp/data/feedback" + transcripts_enabled: true + transcripts_storage: "/tmp/data/transcripts" +authentication: + module: "noop" +inference: + default_provider: openai + default_model: gpt-4o-mini diff --git a/tests/e2e/configuration/unified-mode/server-mode/lightspeed-stack-unified-absolute-profile.yaml b/tests/e2e/configuration/unified-mode/server-mode/lightspeed-stack-unified-absolute-profile.yaml new file mode 100644 index 000000000..7f9b31dd5 --- /dev/null +++ b/tests/e2e/configuration/unified-mode/server-mode/lightspeed-stack-unified-absolute-profile.yaml @@ -0,0 +1,26 @@ +name: Lightspeed Core Service (LCS) +service: + host: 0.0.0.0 + port: 8080 + auth_enabled: false + workers: 1 + color_log: true + access_log: true +llama_stack: + # Server mode - connects to the separate ogx service + use_as_library_client: false + url: http://${env.E2E_OGX_HOSTNAME}:8321 + api_key: xyzzy + config: + # Absolute path as mounted in the ogx container + profile: /opt/app-root/run.yaml +user_data_collection: + feedback_enabled: true + feedback_storage: "/tmp/data/feedback" + transcripts_enabled: true + transcripts_storage: "/tmp/data/transcripts" +authentication: + module: "noop" +inference: + default_provider: openai + default_model: gpt-4o-mini diff --git a/tests/e2e/configuration/unified-mode/server-mode/lightspeed-stack-unified-config-only.yaml b/tests/e2e/configuration/unified-mode/server-mode/lightspeed-stack-unified-config-only.yaml new file mode 100644 index 000000000..34ace93fa --- /dev/null +++ b/tests/e2e/configuration/unified-mode/server-mode/lightspeed-stack-unified-config-only.yaml @@ -0,0 +1,26 @@ +name: Lightspeed Core Service (LCS) +service: + host: 0.0.0.0 + port: 8080 + auth_enabled: false + workers: 1 + color_log: true + access_log: true +llama_stack: + # Server mode - connects to the separate ogx service + use_as_library_client: false + url: http://${env.E2E_OGX_HOSTNAME}:8321 + api_key: xyzzy + config: + # Synthesis baseline: the CI-materialized run.yaml (provider-agnostic) + profile: run.yaml +user_data_collection: + feedback_enabled: true + feedback_storage: "/tmp/data/feedback" + transcripts_enabled: true + transcripts_storage: "/tmp/data/transcripts" +authentication: + module: "noop" +inference: + default_provider: openai + default_model: gpt-4o-mini diff --git a/tests/e2e/configuration/unified-mode/server-mode/lightspeed-stack-unified-migrated.yaml b/tests/e2e/configuration/unified-mode/server-mode/lightspeed-stack-unified-migrated.yaml new file mode 100644 index 000000000..7a31fc7af --- /dev/null +++ b/tests/e2e/configuration/unified-mode/server-mode/lightspeed-stack-unified-migrated.yaml @@ -0,0 +1,128 @@ +authentication: + module: noop +inference: + default_model: gpt-4o-mini + default_provider: openai +name: Lightspeed Core Service (LCS) +ogx: + api_key: xyzzy + config: + baseline: empty + native_override: + apis: + - responses + - batches + - files + - inference + - tool_runtime + - conversations + - vector_io + distro_name: starter + providers: + batches: + - config: + sqlstore: + backend: sql_default + table_name: batches + provider_id: reference + provider_type: inline::reference + files: + - config: + metadata_store: + backend: sql_default + table_name: files_metadata + storage_dir: ~/.llama/storage/files + provider_id: meta-reference-files + provider_type: inline::localfs + inference: + - config: + allowed_models: + - ${env.E2E_OPENAI_MODEL:=gpt-4o-mini} + api_key: ${env.OPENAI_API_KEY} + provider_id: openai + provider_type: remote::openai + - config: {} + provider_id: sentence-transformers + provider_type: inline::sentence-transformers + responses: + - config: + persistence: + responses: + backend: sql_default + table_name: agents_responses + provider_id: builtin + provider_type: inline::builtin + tool_runtime: + - config: {} + provider_id: file-search + provider_type: inline::file-search + - config: {} + provider_id: model-context-protocol + provider_type: remote::model-context-protocol + vector_io: + - config: + persistence: + backend: kv_rag + namespace: vector_io::faiss + provider_id: faiss + provider_type: inline::faiss + registered_resources: + models: + - metadata: + embedding_dimension: 768 + model_id: all-mpnet-base-v2 + model_type: embedding + provider_id: sentence-transformers + provider_model_id: all-mpnet-base-v2 + vector_stores: [] + server: + port: 8321 + storage: + backends: + kv_default: + db_path: ${env.KV_STORE_PATH:=~/.llama/storage/kv_store.db} + type: kv_sqlite + kv_rag: + db_path: ${env.KV_RAG_PATH:=~/.llama/storage/rag/kv_store.db} + type: kv_sqlite + sql_default: + db_path: ${env.SQL_STORE_PATH:=~/.llama/storage/sql_store.db} + type: sql_sqlite + stores: + connectors: + backend: sql_default + table_name: connectors + conversations: + backend: sql_default + table_name: openai_conversations + inference: + backend: sql_default + max_write_queue_size: 10000 + num_writers: 4 + table_name: inference_store + metadata: + backend: kv_default + namespace: registry + prompts: + backend: sql_default + table_name: prompts + vector_stores: + default_embedding_model: + model_id: all-mpnet-base-v2 + provider_id: sentence-transformers + default_provider_id: faiss + version: 2 + url: http://${env.E2E_OGX_HOSTNAME}:8321 + use_as_library_client: false +service: + access_log: true + auth_enabled: false + color_log: true + host: 0.0.0.0 + port: 8080 + workers: 1 +user_data_collection: + feedback_enabled: true + feedback_storage: /tmp/data/feedback + transcripts_enabled: true + transcripts_storage: /tmp/data/transcripts diff --git a/tests/e2e/configuration/unified-mode/server-mode/lightspeed-stack-unified-providers.yaml b/tests/e2e/configuration/unified-mode/server-mode/lightspeed-stack-unified-providers.yaml new file mode 100644 index 000000000..a40465cca --- /dev/null +++ b/tests/e2e/configuration/unified-mode/server-mode/lightspeed-stack-unified-providers.yaml @@ -0,0 +1,31 @@ +name: Lightspeed Core Service (LCS) +service: + host: 0.0.0.0 + port: 8080 + auth_enabled: false + workers: 1 + color_log: true + access_log: true +llama_stack: + # Server mode - connects to the separate ogx service + use_as_library_client: false + url: http://${env.E2E_OGX_HOSTNAME}:8321 + api_key: xyzzy +user_data_collection: + feedback_enabled: true + feedback_storage: "/tmp/data/feedback" + transcripts_enabled: true + transcripts_storage: "/tmp/data/transcripts" +authentication: + module: "noop" +inference: + default_provider: openai + default_model: gpt-4o-mini + # Unified synthesis input (Decision S5): the high-level provider entry + # replaces the default baseline's openai provider by id at synthesis time. + providers: + - type: openai + id: openai + api_key_env: OPENAI_API_KEY + allowed_models: + - ${env.E2E_OPENAI_MODEL:=gpt-4o-mini} diff --git a/tests/e2e/configuration/unified-mode/server-mode/lightspeed-stack-unified-relative-profile.yaml b/tests/e2e/configuration/unified-mode/server-mode/lightspeed-stack-unified-relative-profile.yaml new file mode 100644 index 000000000..0647f4c87 --- /dev/null +++ b/tests/e2e/configuration/unified-mode/server-mode/lightspeed-stack-unified-relative-profile.yaml @@ -0,0 +1,26 @@ +name: Lightspeed Core Service (LCS) +service: + host: 0.0.0.0 + port: 8080 + auth_enabled: false + workers: 1 + color_log: true + access_log: true +llama_stack: + # Server mode - connects to the separate ogx service + use_as_library_client: false + url: http://${env.E2E_OGX_HOSTNAME}:8321 + api_key: xyzzy + config: + # R8: relative profile resolves against this file's loaded location + profile: run.yaml +user_data_collection: + feedback_enabled: true + feedback_storage: "/tmp/data/feedback" + transcripts_enabled: true + transcripts_storage: "/tmp/data/transcripts" +authentication: + module: "noop" +inference: + default_provider: openai + default_model: gpt-4o-mini diff --git a/tests/e2e/features/environment.py b/tests/e2e/features/environment.py index 4fc2346fc..95d349ffe 100644 --- a/tests/e2e/features/environment.py +++ b/tests/e2e/features/environment.py @@ -242,6 +242,21 @@ def before_scenario(context: Context, scenario: Scenario) -> None: scenario.skip("Skipped in Prow (requires Docker Compose services)") return + # Skip openai-specific scenarios on non-openai provider matrices: the + # providers workflow runs the full test list with E2E_DEFAULT_PROVIDER_OVERRIDE + # set (azure/watsonx/...), and fixtures that hardcode an openai provider + # (e.g. the unified-mode inference.providers fixture) cannot serve queries + # for those models. + provider_override = os.getenv("E2E_DEFAULT_PROVIDER_OVERRIDE", "") + if "openai-only" in scenario.effective_tags and provider_override not in ( + "", + "openai", + ): + scenario.skip( + f"Skipped on provider matrix '{provider_override}' (openai-only fixture)" + ) + return + # In Prow, verify the lightspeed port-forward is alive before each scenario. # Port-forwards can silently die between scenarios (e.g. pod restart, TCP reset). if is_prow_environment(): @@ -530,7 +545,15 @@ def after_feature(context: Context, feature: Feature) -> None: remove_config_backup(backup_path) if not context.is_library_mode: restart_container("ogx") - restart_container("lightspeed-stack") + # restart_container hard-fails for lightspeed-stack when the + # service does not accept HTTP in time. That is right inside a + # scenario, but this runs in after_feature: an exception here is a + # hook error that takes down the whole run rather than failing one + # scenario. Warn and let the next feature's own restart surface it. + try: + restart_container("lightspeed-stack") + except AssertionError as exc: + print(f"⚠ after_feature restore: lightspeed-stack not ready ({exc})") reset_active_lightspeed_stack_config_basename() else: remove_config_backup(backup_path) diff --git a/tests/e2e/features/steps/common.py b/tests/e2e/features/steps/common.py index ba49db388..5ce825486 100644 --- a/tests/e2e/features/steps/common.py +++ b/tests/e2e/features/steps/common.py @@ -213,7 +213,7 @@ def restart_service_without_restoring_ogx(context: Context) -> None: if getattr(context, "lightspeed_stack_skip_restart", False): context.lightspeed_stack_skip_restart = False return - restart_lightspeed_stack_service(skip_ogx_restore=True, wait_http=False) + restart_lightspeed_stack_service(skip_ogx_restore=True) @given("The system is in default state") diff --git a/tests/e2e/features/steps/proxy.py b/tests/e2e/features/steps/proxy.py index ddb8d4c50..f5651b6cb 100644 --- a/tests/e2e/features/steps/proxy.py +++ b/tests/e2e/features/steps/proxy.py @@ -42,7 +42,6 @@ from tests.e2e.utils.utils import ( is_prow_environment, restart_container, - wait_for_lightspeed_stack_http_ready, ) _CLUSTER_INTERCEPTION_PROXY_PORTS = frozenset( @@ -339,7 +338,6 @@ def restart_ogx(context: Context) -> None: def restart_lightspeed_stack(context: Context) -> None: """Restart the Lightspeed Stack container.""" restart_container("lightspeed-stack") - wait_for_lightspeed_stack_http_ready() # --- Tunnel Proxy Steps --- diff --git a/tests/e2e/features/steps/tls.py b/tests/e2e/features/steps/tls.py index a87715d2d..443704184 100644 --- a/tests/e2e/features/steps/tls.py +++ b/tests/e2e/features/steps/tls.py @@ -98,7 +98,6 @@ def _restart_lightspeed_after_ogx_tls(context: Context) -> None: """ from tests.e2e.utils.utils import ( restart_container, - wait_for_lightspeed_stack_http_ready, ) scenario = getattr(getattr(context, "scenario", None), "name", "") or "?" @@ -111,7 +110,6 @@ def _restart_lightspeed_after_ogx_tls(context: Context) -> None: flush=True, ) restart_container("lightspeed-stack") - wait_for_lightspeed_stack_http_ready() def restart_ogx_for_tls_feature(context: Context) -> None: diff --git a/tests/e2e/features/steps/unified_mode.py b/tests/e2e/features/steps/unified_mode.py new file mode 100644 index 000000000..3e110ed2c --- /dev/null +++ b/tests/e2e/features/steps/unified_mode.py @@ -0,0 +1,107 @@ +"""Step definitions for the unified-mode e2e features (LCORE-2343). + +Only the startup-log evidence step lives here. Everything else the five +``unified-mode-*.feature`` files need — applying a configuration, restarting +containers, hitting ``readiness`` and ``query`` — resolves through the generic +steps, and the configuration validation, migration and synthesis assertions +moved to ``tests/integration/`` (``test_unified_mode_cli.py``, +``test_unified_synthesis.py``): e2e steps observe the deployed stack from +outside and never import from or execute anything under ``src/``. See +``docs/testing/e2e_testing.md``, "Choosing the Test Layer". + +The log step is mode-aware: in server mode the synthesis evidence is emitted by +the ogx container (entrypoint + config CLI), not the lightspeed-stack +container the Gherkin names; the scenario's intent (R10: the synthesized path +is logged at startup) is asserted against the container that actually +synthesizes. +""" + +import re +import subprocess + +from behave import then # pyright: ignore +from behave.runner import Context + + +def _container_started_at(container: str) -> str: + """Return the container's last start timestamp (RFC 3339) from ``docker inspect``.""" + result = subprocess.run( + ["docker", "inspect", "-f", "{{.State.StartedAt}}", container], + capture_output=True, + text=True, + timeout=60, + check=False, + ) + assert ( + result.returncode == 0 + ), f"docker inspect {container} failed: {result.stderr[-500:]}" + started_at = result.stdout.strip() + assert started_at, f"docker inspect {container} returned no StartedAt" + return started_at + + +@then("the lightspeed-stack container logs contain synthesized run.yaml") +def container_logs_show_synthesis(context: Context) -> None: + """Assert the container that synthesizes logged the synthesized-config path. + + Library mode: the lightspeed-stack container itself synthesizes in-process + and logs "Using synthesized OGX config at ". Server mode: synthesis + happens in the ogx container (entrypoint + config CLI), which + echoes the generated-config path — the Gherkin names lightspeed-stack, but + the scenario's intent (R10: the path is logged at startup) can only be + observed on the synthesizing container. Deviation agreed in planning (Q2). + + ``docker logs`` accumulates across ``docker restart``, and the CI baseline + configurations already synthesize (library) or generate (server) on the + very first compose boot — so an unscoped read would pass on evidence from + an earlier boot. The read is therefore limited to lines since the + container's current ``StartedAt``, i.e. the restart the scenario just + performed under the unified fixture. + + The pattern must carry a path and must be unique to synthesis. The + entrypoint's own lines cannot provide that: it echoes "(mode + auto-detected)" before generation runs and unconditionally, and it echoes + "Using generated config: " identically for the synthesis and the + legacy-enrichment branch (scripts/ogx-entrypoint.sh). Matching either + would let the scenario pass when the unified fixture never reached the + container and the entrypoint enriched a run.yaml instead — which is the + one thing this scenario exists to rule out. The fallback line "Using + original config:" does not discriminate either; it is printed only when + generation *failed*, so it is absent from a successful enrichment too. + + So both modes match a line that only the synthesis path writes: + src/client/ogx.py in library mode, src/ogx_configuration.py in server + mode. The latter reaches the container log only because main() configures + logging — as a bare script nothing installs a root handler and + logging.lastResort drops everything below WARNING. + """ + if context.is_library_mode: + container = "lightspeed-stack" + pattern = r"Using synthesized OGX config at \S+" + else: + container = "ogx" + pattern = r"Wrote synthesized OGX configuration to \S+" + + started_at = _container_started_at(container) + result = subprocess.run( + ["docker", "logs", "--since", started_at, container], + capture_output=True, + text=True, + timeout=60, + check=False, + ) + assert ( + result.returncode == 0 + ), f"docker logs {container} failed: {result.stderr[-500:]}" + logs = result.stdout + result.stderr + # Checked before the pattern assert: when the entrypoint genuinely fell + # back, the pattern is absent too, and this is the message that says why. + if not context.is_library_mode: + assert "Using original config:" not in logs, ( + f"{container} fell back to the original run.yaml on this boot; " + "the unified fixture was not synthesized" + ) + assert re.search(pattern, logs), ( + f"{container} logs since {started_at} carry no synthesis-path evidence " + f"(pattern {pattern!r} not found)" + ) diff --git a/tests/e2e/features/unified-mode-boot.feature b/tests/e2e/features/unified-mode-boot.feature index 1724dd622..feab84a8d 100644 --- a/tests/e2e/features/unified-mode-boot.feature +++ b/tests/e2e/features/unified-mode-boot.feature @@ -1,4 +1,4 @@ -@cfg_unified @skip +@cfg_unified @skip-in-prow Feature: Unified mode configuration boot Background: @@ -10,7 +10,7 @@ Feature: Unified mode configuration boot # --- library mode (@skip-in-server-mode) --- - @skip-in-server-mode + @skip-in-server-mode @openai-only Scenario: Unified config with inference.providers boots and serves requests in library mode Given The service uses the lightspeed-stack-unified-providers.yaml configuration And The service is restarted @@ -62,7 +62,7 @@ Feature: Unified mode configuration boot # --- server mode (@skip-in-library-mode) --- - @skip-in-library-mode + @skip-in-library-mode @openai-only Scenario: Unified config with inference.providers boots and serves requests in server mode Given The service uses the lightspeed-stack-unified-providers.yaml configuration And OGX is restarted diff --git a/tests/e2e/features/unified-mode-legacy.feature b/tests/e2e/features/unified-mode-legacy.feature index dea933804..c27fe523e 100644 --- a/tests/e2e/features/unified-mode-legacy.feature +++ b/tests/e2e/features/unified-mode-legacy.feature @@ -1,4 +1,4 @@ -@cfg_unified @skip +@cfg_unified @skip-in-prow Feature: Legacy two-file configuration during deprecation window Background: @@ -12,7 +12,10 @@ Feature: Legacy two-file configuration during deprecation window @skip-in-server-mode Scenario: Legacy two-file configuration still boots and serves requests in library mode - Given The service uses the lightspeed-stack.yaml configuration + # lightspeed-stack-legacy.yaml (not the standard baseline): LCORE-2342 + # migrated the library-mode baseline to unified mode, so only a dedicated + # legacy-shaped fixture still exercises the deprecated two-file path (R2). + Given The service uses the lightspeed-stack-legacy.yaml configuration And The service is restarted When I access endpoint "readiness" using HTTP GET method Then The status code of the response is 200 diff --git a/tests/e2e/features/unified-mode-migration.feature b/tests/e2e/features/unified-mode-migration.feature index 7dbca73b1..8462ccc2c 100644 --- a/tests/e2e/features/unified-mode-migration.feature +++ b/tests/e2e/features/unified-mode-migration.feature @@ -1,4 +1,4 @@ -@cfg_unified @skip +@cfg_unified @skip-in-prow Feature: Legacy to unified configuration migration Background: @@ -8,24 +8,20 @@ Feature: Legacy to unified configuration migration And the Lightspeed stack configuration directory is "tests/e2e/configuration/unified-mode" - Scenario: migrate-config produces a unified configuration from a legacy pair - When lightspeed-stack --migrate-config is run for the legacy migration fixture pair - Then the file lightspeed-stack-unified-migrated.yaml contains native_override - And the file lightspeed-stack-unified-migrated.yaml does not contain library_client_config_path - - - Scenario: migrate then synthesize round-trips to the original run.yaml - When lightspeed-stack --migrate-config is run for the legacy migration fixture pair - And the active unified configuration is synthesized to run.yaml - Then the synthesized run.yaml parses to the same data as the legacy migration fixture run.yaml - + # The --migrate-config CLI itself (output shape, owner-only mode, migrate-then- + # synthesize round trip) is covered by tests/integration/test_unified_mode_cli.py: + # e2e steps never run src/ CLIs (docs/testing/e2e_testing.md, "Choosing the + # Test Layer"). The scenarios below boot the committed + # lightspeed-stack-unified-migrated.yaml fixture — generated once from the + # legacy migration fixture pair (lightspeed-stack-legacy-for-migration.yaml + + # tests/e2e/configs/run-ci.yaml) and guarded against CLI drift by that same + # integration module. It inlines the openai run-ci.yaml, hence @openai-only. # --- library mode (@skip-in-server-mode) --- - @skip-in-server-mode - Scenario: Migrated unified configuration drives byte-identical OGX behavior in library mode - Given lightspeed-stack --migrate-config is run for the legacy migration fixture pair - And The service uses the lightspeed-stack-unified-migrated.yaml configuration + @skip-in-server-mode @openai-only + Scenario: Migrated unified configuration boots and serves queries in library mode + Given The service uses the lightspeed-stack-unified-migrated.yaml configuration And The service is restarted When I access endpoint "readiness" using HTTP GET method Then The status code of the response is 200 @@ -38,10 +34,9 @@ Feature: Legacy to unified configuration migration # --- server mode (@skip-in-library-mode) --- - @skip-in-library-mode - Scenario: Migrated unified configuration drives byte-identical OGX behavior in server mode - Given lightspeed-stack --migrate-config is run for the legacy migration fixture pair - And The service uses the lightspeed-stack-unified-migrated.yaml configuration + @skip-in-library-mode @openai-only + Scenario: Migrated unified configuration boots and serves queries in server mode + Given The service uses the lightspeed-stack-unified-migrated.yaml configuration And OGX is restarted And Lightspeed Stack is restarted When I access endpoint "readiness" using HTTP GET method diff --git a/tests/e2e/features/unified-mode-synthesis.feature b/tests/e2e/features/unified-mode-synthesis.feature index 69f67c44d..528a4683a 100644 --- a/tests/e2e/features/unified-mode-synthesis.feature +++ b/tests/e2e/features/unified-mode-synthesis.feature @@ -1,4 +1,4 @@ -@cfg_unified @skip +@cfg_unified @skip-in-prow Feature: Unified mode configuration synthesis Background: @@ -7,42 +7,17 @@ Feature: Unified mode configuration synthesis And the Lightspeed stack configuration directory is "tests/e2e/configuration/unified-mode" - Scenario: native_override replaces an overlapping scalar key - Given The service uses the lightspeed-stack-unified-native-override-scalar.yaml configuration - When the active unified configuration is synthesized to run.yaml - Then the synthesized run.yaml contains the native_override scalar value for safety.excluded_categories - - - Scenario: native_override replaces an overlapping list key wholesale - Given The service uses the lightspeed-stack-unified-native-override-list.yaml configuration - When the active unified configuration is synthesized to run.yaml - Then the synthesized run.yaml contains exactly the native_override list for apis - - - Scenario: LCORE-emitted secrets remain as environment references on disk - Given The service uses the lightspeed-stack-unified-providers.yaml configuration - When the active unified configuration is synthesized to run.yaml - Then the synthesized run.yaml contains ${env.OPENAI_API_KEY} - And the synthesized run.yaml does not contain the resolved OPENAI_API_KEY value - - - Scenario: Synthesized run.yaml is written with owner-only permissions - Given The service uses the lightspeed-stack-unified-providers.yaml configuration - When the active unified configuration is synthesized to run.yaml - Then the synthesized run.yaml file permissions are 0600 - - - Scenario: synthesized-config-output overrides the default synthesis location - Given The service uses the lightspeed-stack-unified-providers.yaml configuration - And lightspeed-stack is started with --synthesized-config-output set to a custom path - When the active unified configuration is synthesized to run.yaml - Then the synthesized run.yaml is written to the custom output path - And the default synthesized run.yaml path does not exist - + # Synthesis semantics — native_override replacement (R5), secrets kept as + # environment references (R6), owner-only output mode (R10) and the + # --synthesized-config-output override — are covered in-process by + # tests/integration/test_unified_synthesis.py: e2e steps never run src/ CLIs + # (docs/testing/e2e_testing.md, "Choosing the Test Layer"). What remains here + # is the one thing only a deployed stack can show: the synthesized path is + # logged at startup (R10). # --- library mode (@skip-in-server-mode) --- - @skip-in-server-mode + @skip-in-server-mode @openai-only Scenario: Synthesized run.yaml path is logged at startup in library mode Given The service uses the lightspeed-stack-unified-providers.yaml configuration And The service is restarted @@ -51,7 +26,7 @@ Feature: Unified mode configuration synthesis # --- server mode (@skip-in-library-mode) --- - @skip-in-library-mode + @skip-in-library-mode @openai-only Scenario: Synthesized run.yaml path is logged at startup in server mode Given The service uses the lightspeed-stack-unified-providers.yaml configuration And OGX is restarted diff --git a/tests/e2e/features/unified-mode-validation.feature b/tests/e2e/features/unified-mode-validation.feature deleted file mode 100644 index ab7a09038..000000000 --- a/tests/e2e/features/unified-mode-validation.feature +++ /dev/null @@ -1,26 +0,0 @@ -@cfg_unified @skip -Feature: Unified mode configuration validation - - Background: - Given The service is started locally - And The system is in default state - And the Lightspeed stack configuration directory is "tests/e2e/configuration/unified-mode" - - - Scenario: inference.providers together with library_client_config_path fails at load - Given The service uses the lightspeed-stack-invalid-providers-and-legacy.yaml configuration - When configuration validation is attempted for the active configuration - Then the validation error contains --migrate-config - - - Scenario: ogx.config together with library_client_config_path fails at load - Given The service uses the lightspeed-stack-invalid-config-and-legacy.yaml configuration - When configuration validation is attempted for the active configuration - Then the validation error contains --migrate-config - - - - Scenario: config_format_version legacy with unified-shaped body fails at load - Given The service uses the lightspeed-stack-invalid-version-legacy-unified-body.yaml configuration - When configuration validation is attempted for the active configuration - Then the validation error contains config_format_version diff --git a/tests/e2e/test_list.txt b/tests/e2e/test_list.txt index 79f34fcb5..e80a173cd 100644 --- a/tests/e2e/test_list.txt +++ b/tests/e2e/test_list.txt @@ -40,7 +40,6 @@ features/tls-tlsv13.feature features/degraded_mode_startup.feature features/unified-mode-boot.feature features/unified-mode-legacy.feature -features/unified-mode-validation.feature features/unified-mode-migration.feature features/unified-mode-synthesis.feature features/okp_rag.feature diff --git a/tests/e2e/utils/utils.py b/tests/e2e/utils/utils.py index fb35203a3..c10e87d49 100644 --- a/tests/e2e/utils/utils.py +++ b/tests/e2e/utils/utils.py @@ -519,6 +519,11 @@ def restart_container(container_name: str) -> None: Raises: subprocess.CalledProcessError: if the `docker restart` command fails. subprocess.TimeoutExpired: if the `docker restart` command times out. + AssertionError: for ``lightspeed-stack``, if the service does not + accept HTTP within ``wait_for_lightspeed_stack_http_ready``'s + budget. Docker health itself stays a soft failure; the HTTP wait + does not, so callers that must not fail (teardown hooks) have to + guard the call. """ if is_prow_environment(): restart_pod(container_name) @@ -548,6 +553,14 @@ def restart_container(container_name: str) -> None: # that restart the container don't time out. wait_for_container_health(container_name) + # Docker health can report healthy before uvicorn binds the published + # port (the documented race wait_for_lightspeed_stack_http_ready exists + # for). Unified-mode first boots are the slowest restarts in the suite + # and hit that window reliably, so close it here for every restart + # rather than only in the proxy steps. + if container_name == "lightspeed-stack": + wait_for_lightspeed_stack_http_ready() + if container_name == "ogx": from tests.e2e.features.steps.health import ( reset_ogx_disrupt_once_tracking, @@ -556,19 +569,15 @@ def restart_container(container_name: str) -> None: reset_ogx_disrupt_once_tracking() -def restart_lightspeed_stack_service( - *, wait_http: bool = False, skip_ogx_restore: bool = False -) -> None: +def restart_lightspeed_stack_service(*, skip_ogx_restore: bool = False) -> None: """Restart the lightspeed-stack container used by Behave steps. - Wraps ``restart_container("lightspeed-stack")`` and optionally polls the - host-mapped port so step modules share one LCS restart path. + Wraps ``restart_container("lightspeed-stack")`` so step modules share one + LCS restart path. That path already waits for Docker health and then for + HTTP on the host-mapped port, so callers need no wait of their own. Parameters: ---------- - wait_http: When True, also call ``wait_for_lightspeed_stack_http_ready`` - after Docker health. Default False — generic ``The service is - restarted`` relies on Docker health only; proxy/tls steps opt in. skip_ogx_restore: When True on Prow/Konflux, tell e2e-ops not to bring llama back before recreating LCS (degraded-mode startup). """ @@ -577,8 +586,6 @@ def restart_lightspeed_stack_service( os.environ["E2E_SKIP_OGX_RESTORE_ON_LCS_RESTART"] = "1" try: restart_container("lightspeed-stack") - if wait_http: - wait_for_lightspeed_stack_http_ready() finally: if skip_ogx_restore: if previous is None: @@ -588,8 +595,9 @@ def restart_lightspeed_stack_service( def wait_for_lightspeed_stack_http_ready( - max_attempts: int = 80, + timeout_s: float = 120.0, delay_s: float = 1.5, + request_timeout_s: float = 5.0, ) -> None: """Block until Lightspeed Stack accepts HTTP on the host-mapped port. @@ -600,10 +608,21 @@ def wait_for_lightspeed_stack_http_ready( Treats HTTP 200 and 401 as success: the process is listening. Auth-enabled configs (e.g. RBAC jwk-token) return 401 on probes without a Bearer token. + Bounded by a single monotonic deadline covering both the requests and the + sleeps, and each request is additionally capped at the time remaining, so + the wait stays within ``timeout_s`` plus at most one request timeout — + ``requests`` applies its scalar ``timeout`` to the connect and the read + phase separately, so an attempt started just under the deadline can + overrun by that much. An attempt-counted loop cannot give even that + guarantee: with a per-request timeout the worst case is + ``attempts * request_timeout + (attempts - 1) * delay``, which for the + previous defaults was 518.5s while the failure message reported 120s. + Parameters: ---------- - max_attempts: Maximum GET attempts. + timeout_s: Total wall-clock budget for becoming reachable. delay_s: Sleep between attempts. + request_timeout_s: Per-request timeout, clamped to the time remaining. Raises: ------ AssertionError: If ``/liveness`` does not return an accepted status in time. @@ -613,26 +632,35 @@ def wait_for_lightspeed_stack_http_ready( host = os.getenv("E2E_LSC_HOSTNAME", "localhost") port = os.getenv("E2E_LSC_PORT", "8080") url = f"http://{host}:{port}/liveness" - for attempt in range(max_attempts): + started = time.monotonic() + deadline = started + timeout_s + attempt = 0 + while True: + remaining = deadline - time.monotonic() + if remaining <= 0: + break + attempt += 1 try: - response = requests.get(url, timeout=5) + response = requests.get(url, timeout=min(request_timeout_s, remaining)) if response.status_code in (200, 401): return detail = response.text[:200].replace("\n", " ") print( - f"⏱ HTTP wait LSC {attempt + 1}/{max_attempts} " + f"⏱ HTTP wait LSC attempt {attempt} " f"({url} -> {response.status_code}: {detail})..." ) except requests.RequestException as exc: print( - f"⏱ HTTP wait LSC {attempt + 1}/{max_attempts} " + f"⏱ HTTP wait LSC attempt {attempt} " f"({url} -> {exc.__class__.__name__}: {exc})..." ) - if attempt < max_attempts - 1: - time.sleep(delay_s) + if time.monotonic() + delay_s >= deadline: + break + time.sleep(delay_s) + elapsed = time.monotonic() - started raise AssertionError( f"Lightspeed Stack did not become reachable at {url!r} " - f"after {max_attempts} attempts (~{max_attempts * delay_s:.0f}s)" + f"after {attempt} attempts / {elapsed:.0f}s (budget {timeout_s:.0f}s)" ) diff --git a/tests/integration/README.md b/tests/integration/README.md index 6863e4869..e0949b787 100644 --- a/tests/integration/README.md +++ b/tests/integration/README.md @@ -219,6 +219,9 @@ Integration tests should verify: 3. **External mocks only** - Mock only external services (OGX, external APIs) 4. **Error handling** - HTTP status codes, error messages 5. **Data flow** - Database persistence, cache updates, etc. +6. **CLI contracts** - Repo entrypoints (`src/lightspeed_stack.py --migrate-config`, `--dump-configuration`, `src/ogx_configuration.py`) run as subprocesses: exit codes, messages, written files and their modes + +Anything that needs a *deployed* stack — HTTP against a running service, container restarts, container logs — is an e2e concern instead. Conversely, e2e steps must never touch `src/`; see [Choosing the Test Layer](../../docs/testing/e2e_testing.md#choosing-the-test-layer-e2e-or-integration). ### What NOT to Test diff --git a/tests/integration/test_unified_mode_cli.py b/tests/integration/test_unified_mode_cli.py new file mode 100644 index 000000000..3987af8dc --- /dev/null +++ b/tests/integration/test_unified_mode_cli.py @@ -0,0 +1,177 @@ +"""Integration tests for the unified-mode CLI contracts (LCORE-2343). + +These cover the surfaces the unified-mode e2e features used to exercise by +shelling out to ``src/`` — which e2e steps must not do (see +``docs/testing/e2e_testing.md``, "Choosing the Test Layer"): configuration +validation through ``lightspeed_stack.py --dump-configuration``, +legacy-to-unified migration through ``--migrate-config``, and the committed +migrated e2e fixture that replaced the migration step in +``unified-mode-migration.feature``. + +Everything here runs the real entrypoint as a subprocess from the repository +root, the way operators and the container entrypoint invoke it; the in-process +half of the same pipeline lives in ``test_unified_synthesis.py``. +""" + +import os +import stat +import subprocess +import sys +from pathlib import Path +from typing import Any + +import pytest +import yaml + +from ogx_configuration import synthesize_configuration + +_REPO_ROOT = Path(__file__).resolve().parents[2] +_ENTRYPOINT = _REPO_ROOT / "src" / "lightspeed_stack.py" +_FIXTURES = _REPO_ROOT / "tests" / "configuration" / "unified-mode" +_E2E_FIXTURES = _REPO_ROOT / "tests" / "e2e" / "configuration" / "unified-mode" +# The run.yaml the e2e harness materializes at the repo root in CI; the +# committed migrated e2e fixtures were generated against it. +_E2E_RUN_YAML = _REPO_ROOT / "tests" / "e2e" / "configs" / "run-ci.yaml" +_MIGRATED_FIXTURE = "lightspeed-stack-unified-migrated.yaml" +_LEGACY_PAIR_FIXTURE = "lightspeed-stack-legacy-for-migration.yaml" +_CLI_TIMEOUT_SECONDS = 120 + + +def _run_cli(*args: str) -> subprocess.CompletedProcess[str]: + """Run the service entrypoint as a subprocess from the repo root, never raising.""" + return subprocess.run( + [sys.executable, str(_ENTRYPOINT), *args], + cwd=_REPO_ROOT, + capture_output=True, + text=True, + timeout=_CLI_TIMEOUT_SECONDS, + check=False, + ) + + +def _load_yaml(path: Path) -> Any: + """Parse a YAML file.""" + return yaml.safe_load(path.read_text(encoding="utf-8")) + + +def _migrate(lcs_config: Path, run_yaml: Path, output: Path) -> None: + """Run ``--migrate-config`` for a legacy pair and assert it succeeded.""" + result = _run_cli( + "--migrate-config", + "--run-yaml", + str(run_yaml), + "-c", + str(lcs_config), + "--migrate-output", + str(output), + ) + assert result.returncode == 0 and output.is_file(), ( + f"--migrate-config failed (rc={result.returncode}).\n" + f"stdout:\n{result.stdout}\nstderr:\n{result.stderr}" + ) + + +# --------------------------------------------------------------------------- +# Validation through the CLI (R3 mutual exclusion, R11 format-version marker) +# --------------------------------------------------------------------------- + + +@pytest.mark.parametrize( + ("fixture", "expected_message"), + [ + pytest.param( + "lightspeed-stack-invalid-providers-and-legacy.yaml", + "--migrate-config", + id="inference-providers-plus-legacy-path", + ), + pytest.param( + "lightspeed-stack-invalid-config-and-legacy.yaml", + "--migrate-config", + id="config-block-plus-legacy-path", + ), + pytest.param( + "lightspeed-stack-invalid-version-legacy-unified-body.yaml", + "config_format_version", + id="legacy-version-marker-on-unified-body", + ), + ], +) +def test_cli_rejects_invalid_configuration(fixture: str, expected_message: str) -> None: + """``--dump-configuration`` exits non-zero and names the problem. + + ``main()`` loads (and thereby validates) the configuration before any dump + handling, so a failed cross-field validation surfaces as a non-zero exit + with the Pydantic message on stderr. The fixtures point their legacy path + at an existing run.yaml so the captured failure is the intended + cross-field error, not a file-not-found. + """ + result = _run_cli("--dump-configuration", "-c", str(_FIXTURES / fixture)) + output = result.stdout + result.stderr + + assert result.returncode != 0, ( + f"expected {fixture} to fail validation, but the load succeeded. " + f"Output:\n{output}" + ) + assert expected_message in output, ( + f"validation failure for {fixture} does not mention " + f"{expected_message!r}. Full output:\n{output}" + ) + + +# --------------------------------------------------------------------------- +# --migrate-config contract +# --------------------------------------------------------------------------- + + +def test_cli_migrate_config_writes_unified_owner_only(tmp_path: Path) -> None: + """``--migrate-config`` emits a unified file, owner-only, that round-trips. + + The output carries the run.yaml as ``native_override`` and drops the legacy + ``library_client_config_path``; it is written 0600 because migrated files + may carry lifted secrets (R10); and synthesizing it reproduces the pair's + run.yaml data (migrate-then-synthesize round trip). + """ + output = tmp_path / "unified.yaml" + _migrate(_FIXTURES / _LEGACY_PAIR_FIXTURE, _E2E_RUN_YAML, output) + + mode = stat.S_IMODE(os.stat(output).st_mode) + assert mode == 0o600, f"migrated file mode is {oct(mode)}, expected 0o600" + + text = output.read_text(encoding="utf-8") + migrated = yaml.safe_load(text) + assert migrated["ogx"]["config"][ + "native_override" + ], "migrated config carries no native_override" + assert "library_client_config_path" not in text + + synthesized = synthesize_configuration(migrated, config_file_dir=str(tmp_path)) + assert synthesized == _load_yaml(_E2E_RUN_YAML) + + +@pytest.mark.parametrize("mode", ["library-mode", "server-mode"]) +def test_committed_migrated_fixture_matches_cli_output( + tmp_path: Path, mode: str +) -> None: + """The committed migrated e2e fixture is exactly what the CLI produces today. + + ``unified-mode-migration.feature`` boots + ``tests/e2e/configuration/unified-mode//lightspeed-stack-unified-migrated.yaml`` + instead of generating it in a step (e2e steps never run ``src/`` CLIs). + This guard fails the moment ``--migrate-config`` output drifts from the + committed file. To refresh the fixture, run from the repo root with + ``DIR=tests/e2e/configuration/unified-mode/``:: + + uv run python src/lightspeed_stack.py --migrate-config \\ + --run-yaml tests/e2e/configs/run-ci.yaml \\ + -c $DIR/lightspeed-stack-legacy-for-migration.yaml \\ + --migrate-output $DIR/lightspeed-stack-unified-migrated.yaml + chmod 644 $DIR/lightspeed-stack-unified-migrated.yaml + """ + output = tmp_path / "migrated.yaml" + _migrate(_E2E_FIXTURES / mode / _LEGACY_PAIR_FIXTURE, _E2E_RUN_YAML, output) + + committed = _E2E_FIXTURES / mode / _MIGRATED_FIXTURE + assert _load_yaml(output) == _load_yaml(committed), ( + f"{committed} no longer matches --migrate-config output; regenerate it " + "(see this test's docstring)" + ) diff --git a/tests/integration/test_unified_synthesis.py b/tests/integration/test_unified_synthesis.py index fa7c8d1e6..b5a7f6105 100644 --- a/tests/integration/test_unified_synthesis.py +++ b/tests/integration/test_unified_synthesis.py @@ -23,6 +23,8 @@ import yaml from pydantic import ValidationError +import constants +from client.ogx import AsyncOgxClientHolder from configuration import configuration from ogx_configuration import ( CONDITIONAL_OPENAI_PROVIDER_ID, @@ -522,3 +524,81 @@ def test_load_accepts_minimal_unified_config(tmp_path: Path) -> None: loaded = configuration.configuration assert loaded.ogx.config is None assert loaded.inference.providers[0].type == "openai" + + +# --------------------------------------------------------------------------- +# R6: emitted secrets stay environment references on disk +# --------------------------------------------------------------------------- + + +def test_synthesized_secrets_stay_env_references( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + """A provider's ``api_key_env`` lands on disk as ``${env.NAME}``, never resolved. + + The synthesizer must emit the environment reference for OGX to resolve at + its own startup; the literal secret value must never be written (R6). + """ + secret = "sk-resolved-secret-that-must-not-land-on-disk" + monkeypatch.setenv("OPENAI_API_KEY", secret) + lcs_dict = _base_config_dict() + lcs_dict["ogx"] = {"use_as_library_client": True} + lcs_dict["inference"] = { + "providers": [{"type": "openai", "api_key_env": "OPENAI_API_KEY"}] + } + _, out_path = _load_and_synthesize(tmp_path, lcs_dict) + + text = out_path.read_text(encoding="utf-8") + assert "${env.OPENAI_API_KEY}" in text + assert secret not in text + + +# --------------------------------------------------------------------------- +# R11: explicit config_format_version must agree with the detected shape +# --------------------------------------------------------------------------- + + +def test_load_rejects_legacy_version_marker_on_unified_body(tmp_path: Path) -> None: + """``config_format_version: legacy`` on a unified-shaped body fails the real load.""" + lcs_dict = _base_config_dict() + lcs_dict["ogx"] = {"use_as_library_client": True} + lcs_dict["inference"] = { + "providers": [{"type": "openai", "api_key_env": "OPENAI_API_KEY"}] + } + lcs_dict["config_format_version"] = "legacy" + cfg_path = _write_yaml(tmp_path / "lightspeed-stack.yaml", lcs_dict) + with pytest.raises(ValidationError, match="config_format_version"): + configuration.load_configuration(str(cfg_path)) + + +# --------------------------------------------------------------------------- +# --synthesized-config-output: the override the workers honour +# --------------------------------------------------------------------------- + + +def test_synthesized_config_output_override( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + """The synthesized-config-output override redirects library-mode synthesis. + + ``--synthesized-config-output`` reaches the uvicorn workers as + ``LIGHTSPEED_STACK_SYNTHESIZED_CONFIG_PATH``; the client holder must write + the synthesized run.yaml there and leave the default location untouched. + """ + lcs_dict = _base_config_dict() + lcs_dict["ogx"] = { + "use_as_library_client": True, + "config": {"baseline": "empty", "native_override": {"version": 2}}, + } + cfg_path = _write_yaml(tmp_path / "lightspeed-stack.yaml", lcs_dict) + custom_output = tmp_path / "custom-run.yaml" + monkeypatch.chdir(tmp_path) + monkeypatch.setenv(constants.CONFIG_PATH_ENV_VAR, str(cfg_path)) + monkeypatch.setenv(constants.SYNTHESIZED_CONFIG_PATH_ENV_VAR, str(custom_output)) + + # pylint: disable-next=protected-access + written = AsyncOgxClientHolder()._synthesize_library_config() + + assert Path(written) == custom_output + assert isinstance(yaml.safe_load(custom_output.read_text(encoding="utf-8")), dict) + assert not (tmp_path / constants.DEFAULT_SYNTHESIZED_CONFIG_PATH).exists()