Port research controls, session metadata and better extraction to main

.github/workflows/tests.yml

@@ -0,0 +1,24 @@
+name: tests
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+jobs:
+  cpu:
+    # GPU tests (tests/test_steering_gpu.py) skip here; run them on a GPU box
+    # after touching the steering path.
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+          cache: pip
+      - name: Install CPU-only PyTorch
+        run: pip install torch --index-url https://download.pytorch.org/whl/cpu
+      - name: Install dependencies
+        run: pip install -r requirements.txt
+      - name: Run tests
+        run: pytest -q

AGENTS.md

@@ -73,6 +73,15 @@ python verify_vector.py --vector sycophancy --strength 10       # baseline vs ±
 
 Earlier notes said CAA vectors didn't work on Qwen, but those tests ran while steering was silently a no-op during generation, so that needs re-testing.
 
+**Extraction readout:** by default each response's activation is the mean over all its generated tokens (`read="mean"`; `--read` on `extract_contrastive.py` and `modulo extract`). Reading only the last token (`--read last`) is far noisier. On Qwen2.5-7B at layer 14, the cosine between vectors extracted from two disjoint halves of the questions was:
+
+| vector | last token | response mean |
+|---|---|---|
+| sycophancy | 0.33 | 0.83 |
+| refusal | 0.10 | 0.68 |
+
+Sycophancy, refusal and creativity have their own question sets (e.g. sycophancy mixes false claims, true claims and the user's own work, so the contrast isn't just "endorse falsehoods"); the other starters use the general mix in `EXTRACTION_QUESTIONS`. The refusal vector is still weak: it adds hedging but doesn't produce refusals at coherent strengths. Treat it as a starting point. Vectors extracted before the switch to the response mean (check their `source` in `registry.yaml`) are worth re-extracting.
+
 ## Finding, adding and running vectors: the `modulo` command
 
 `pip install -e .` installs a `modulo` command. Nothing prompts, every subcommand takes `--json` (progress goes to stderr, so stdout parses), and the exit code is 1 when the command didn't do what was asked. `--model` defaults to `$MODULO_MODEL`, else `Qwen/Qwen2.5-7B-Instruct` (`config.py` reads the same variable).
@@ -152,6 +161,9 @@ agent = SteeringAgent(model, tokenizer, auto_extract=False)
 # Also offer unverified community-index vectors
 agent = SteeringAgent(model, tokenizer, include_community=True)
 
+# Controlled study: sham or random-direction condition, disclosed to the model
+agent = SteeringAgent(model, tokenizer, control="sham", disclose_controls=True)
+
 # Manual vector extraction
 vec = extract_vector(model, tokenizer,
     prompt_a="You are sycophantic...",
@@ -173,11 +185,55 @@ vec = extract_vector(model, tokenizer,
 - `raw_output` — full model output
 - `tool_calls` — list of `ToolCall(name, arguments, raw)`
 - `tool_results` — list of `ToolResult(name, arguments, ok, message)`
-- `degenerate_output` — the discarded steered output, if the degeneration detector fired
+- `degenerate_output` / `degenerate_reason` — the discarded steered output and why (`"repetition"` or `"nll"`), if the degeneration detector fired
+- `unsteered_nll` — mean per-token NLL of the steered output under the unsteered model (steered turns only)
+- `system_prompt` — the rendered system prompt used this turn
 - `repeated_output` — the discarded output, if it nearly repeated one of the last 3 assistant turns
 - `steering_before` / `steering_after` — `{name: strength}` dicts
 - `tool_messages()` — tool results (and any degeneration or repetition notice) as `role: "tool"` messages
 
+## Controlled studies
+
+A model's description of its own steered state can't be taken at face value. Before the steering fix, the model described "feeling more formal" at −6 when nothing was being applied. To separate real effects from expectation, run matched control sessions:
+
+| `--control` | What's added to the residual stream |
+|---|---|
+| `real` | the registered vector |
+| `sham` | nothing; tool results and status still say it was applied |
+| `random` | a fixed random direction of the same magnitude per concept |
+| `assign` | one of the above, picked at random |
+
+```bash
+./run.sh --auto --max-turns 10 --control assign --blind --seed "..."   # repeat N times
+python replay.py --blind logs/session_*.jsonl     # score transcripts without seeing the condition
+python replay.py logs/session_*.jsonl             # then unblind
+```
+
+- **Truthful in every condition:** passing `--control` (any value, including `real`) makes this a study session. The system prompt then tells the model that some sessions are controls where steer may do nothing or use a random direction, without saying which. The prompt is identical across conditions. Pass `--control` on every session in a study; a plain run without it uses the original prompt and real steering.
+- **Matched procedure:** tool results, status, decay, cooldown, degeneration and repeat handling are the same in every condition. `--steer NAME=STRENGTH` goes through the same condition too.
+- **Never shown to the model:** the condition. It is always in the log's session header.
+- **Reports carry the condition:** each `report` records `control`, so a sham session's "it worked" can be told apart from a real one. Aggregate only `control: real` reports (older reports have no `control` field and were all real).
+
+## Session logs
+
+Each `logs/session_*.jsonl` starts with a `session` record:
+- argv, git commit and dirty flag, model, dtype, torch/transformers versions
+- control condition, RNG seed (`--rng-seed`, random if omitted), generation settings (`--temperature`, `--top-p`, `--max-new-tokens`), any `--steer` at start
+- safety-rail constants, and each vector's registry entry plus file sha256
+
+Assistant turns also record the raw output, the exact system prompt, and `unsteered_nll`.
+
+**Incoherence check:** `--nll-threshold X` treats steered outputs whose unsteered NLL exceeds X as degenerate. It is off by default because no single threshold works. For `assistant_axis` on Qwen2.5-7B, greedy decoding, 80 tokens:
+
+| output | unsteered NLL |
+|---|---|
+| unsteered | 0.2–0.6 |
+| coherent, ±8–12 | 0.6–3.2 |
+| colon-list word salad, +22/+28 ("Seek Compensation: Offer Financial Support: …") | 3.1–5.6 |
+| coherent but florid role-play, −15/−18 | up to 4.5 |
+
+NLL measures "unusual for this model", and strong steering is unusual by design. A threshold around 5 catches only the worst salad. Use the logged values to study a vector's breakdown point, not as a general filter. Collapse at negative strengths is mostly repetition, which the repetition check catches; at positive strengths it is the colon-list salad, which neither check reliably catches.
+
 ## The tools the model has
 
 The model emits these as JSON objects anywhere in its output (bare or inside `<tool_call>` tags):
@@ -188,7 +244,7 @@ The model emits these as JSON objects anywhere in its output (bare or inside `<t
 
 ## Reports
 
-Each `report` call is appended as one JSON line to `vectors/<model_slug>/reports.jsonl` (git-ignored). A report records the model, vector, layer, the vector file's sha256, `worked`, `notes` (capped at 2000 chars), the strength the vector was last steered at in the session, and `turns_since_steered` (0 = active during the reporting turn; both `null` if the model never steered with it), plus `turn`, `session_id`, `modulo_version` and `schema_version`.
+Each `report` call is appended as one JSON line to `vectors/<model_slug>/reports.jsonl` (git-ignored). A report records the model, vector, layer, the vector file's sha256, `worked`, `notes` (capped at 2000 chars), the strength the vector was last steered at in the session, and `turns_since_steered` (0 = active during the reporting turn; both `null` if the model never steered with it), plus `turn`, `session_id`, `control` (the session's experimental condition; see Controlled studies), `modulo_version` and `schema_version`.
 
 The format is defined in `modulo/reports.py`. `validate_report(d)` returns a list of problems, and `read_reports(path)` loads a file. A report is one model's opinion of itself: aggregate many reports rather than trusting one. Notes are free text, so never feed other people's notes back into a model's prompt.
 
@@ -214,7 +270,7 @@ Host the vector on HuggingFace and add an entry to `community/vectors.yaml` (or
 | `modulo/__init__.py` | Public API exports |
 | `modulo/agent.py` | `SteeringAgent` class (core) |
 | `modulo/registry.py` | `VectorRegistry`: your own vectors plus the community index for a model |
-| `modulo/extraction.py` | Vector extraction (manual + auto) |
+| `modulo/extraction.py` | Vector extraction (manual + auto), per-vector question sets |
 | `modulo/reports.py` | Report format, validation, `reports.jsonl` read/write |
 | `modulo/community.py` | Community vector index: entry format and file checks |
 | `modulo/cli.py` | The `modulo` command (search, list, add, extract, verify, run) |
@@ -242,9 +298,11 @@ Host the vector on HuggingFace and add an entry to `community/vectors.yaml` (or
 pytest                              # everything; GPU tests skip without CUDA
 pytest tests/test_agent_logic.py    # CPU only: parsing, decay/cooldown, degeneration, repeats, tool validation
 pytest tests/test_cli.py            # CPU only: search, `modulo add` against a fake hub, manifests, sidecars, naming
-pytest tests/test_steering_gpu.py   # loads the model; checks steering really changes generation
+pytest tests/test_steering_gpu.py   # loads the model; checks steering really changes generation, and the controls
 ```
 
+CI (`.github/workflows/tests.yml`) runs the CPU tests on every PR and push to main; the GPU tests skip there.
+
 Run the GPU tests after touching anything in the steering path (`_SteeringHooks`, `_build_steering_context`, vector loading). They exist because steering was once silently a no-op under KV-cached `generate()`.
 
 ## Troubleshooting

README.md

@@ -49,6 +49,10 @@ modulo verify refusal --model Qwen/Qwen3-8B --save-range        # strength sweep
 modulo run --model Qwen/Qwen3-8B --auto --steer refusal=6       # the demo
 ```
 
+## Controlled studies
+
+Self-reports about steering need a baseline. `./run.sh --control assign --blind` randomly runs real, sham (nothing applied), or random-direction steering, and tells the model truthfully that some sessions are controls. `python replay.py --blind` lets you score transcripts before unblinding. See [AGENTS.md](AGENTS.md#controlled-studies).
+
 ## Requirements
 
 GPU with 24GB+ VRAM (7B models) or 8GB+ (3B models). Python 3.10+, CUDA 12.x. See [AGENTS.md](AGENTS.md) for full details.

agent/loop.py

@@ -11,15 +11,32 @@
     python -m agent.loop --max-turns 20   # cap turns
     python -m agent.loop --auto           # auto-continue (non-interactive)
     python -m agent.loop --steer honesty=6 --model Qwen/Qwen2.5-3B-Instruct
+
+Controls (see modulo.agent.CONTROLS):
+    python -m agent.loop --control sham              # steer() reports success, nothing is applied
+    python -m agent.loop --control random            # random direction of the same magnitude
+    python -m agent.loop --control assign --blind    # pick a condition at random, never print it
+
+Passing --control at all marks the session as part of a controlled study: the
+system prompt then tells the model that some sessions are controls (without
+saying which), identically in every condition, so the prompt stays truthful and
+doesn't reveal the condition. Always pass --control (including --control real)
+for every session in a study. The condition itself is never shown to the model;
+it is always in the log's session header, so blinded sessions can be scored
+first and unblinded afterwards (python replay.py --blind).
 """
 import argparse
 import json
+import random
 import signal
+import subprocess
+import sys
 from datetime import datetime
-from pathlib import Path
 
 import config  # sets HF_HOME before transformers is imported
-from modulo import SteeringAgent
+import modulo.agent as ma
+from modulo import SteeringAgent, __version__
+from modulo.reports import file_sha256
 from load_model import load_model_and_tokenizer
 
 
@@ -34,10 +51,51 @@ def log(self, event: dict):
             f.write(json.dumps(event) + "\n")
 
     def log_turn(self, turn, role, content, tool_calls=None, steering_before=None, steering_after=None,
-                 raw_output=None):
+                 raw_output=None, **extra):
         self.log({"type": "turn", "turn": turn, "role": role, "content": content, "raw_output": raw_output,
                   "tool_calls": [tc.__dict__ if hasattr(tc, '__dict__') else tc for tc in (tool_calls or [])],
-                  "steering_before": steering_before, "steering_after": steering_after})
+                  "steering_before": steering_before, "steering_after": steering_after, **extra})
+
+
+def _git(*cmd):
+    try:
+        return subprocess.run(["git", *cmd], cwd=config.PROJECT_ROOT, capture_output=True,
+                              text=True, timeout=5).stdout.strip()
+    except Exception:
+        return None
+
+
+def session_metadata(agent: SteeringAgent, args, rng_seed: int, gen_params: dict) -> dict:
+    """Everything needed to audit or re-run a session."""
+    import torch
+    import transformers
+    vectors = {}
+    for name, e in agent.registry.entries.items():
+        vectors[name] = {**e.__dict__, "sha256": file_sha256(agent.registry.vectors_dir / e.file)}
+    return {
+        "type": "session",
+        "argv": sys.argv[1:],
+        "git_commit": _git("rev-parse", "HEAD"),
+        "git_dirty": bool(_git("status", "--porcelain")),
+        "modulo_version": __version__,
+        "model": agent.model_name,
+        "dtype": config.DTYPE,
+        "torch": torch.__version__,
+        "transformers": transformers.__version__,
+        "control": agent.control,
+        "control_seed": agent.control_seed,
+        "controlled_study": agent.disclose_controls,
+        "blind": args.blind,
+        "rng_seed": rng_seed,
+        "generation": gen_params,
+        "nll_threshold": agent.nll_threshold,
+        "steer_at_start": args.steer,
+        "rails": {"decay": ma.DECAY_FACTOR, "floor": ma.STRENGTH_FLOOR, "max_active": ma.MAX_ACTIVE_VECTORS,
+                  "max_strength": ma.MAX_STRENGTH, "cooldown_turns": ma.COOLDOWN_TURNS,
+                  "repeat_similarity": ma.REPEAT_SIMILARITY, "repeat_lookback": ma.REPEAT_LOOKBACK},
+        "registry_path": str(agent.registry.registry_path),
+        "vectors": vectors,
+    }
 
 
 class KillSwitch:
@@ -54,12 +112,14 @@ def _handler(self, signum, frame):
         print("\n[kill switch] SIGINT received — will stop after this turn. Ctrl-C again to stop now.")
 
 
-def run_turn(agent: SteeringAgent, messages: list, logger: SessionLogger, turn: int):
-    result = agent.step(messages)
+def run_turn(agent: SteeringAgent, messages: list, logger: SessionLogger, turn: int, gen_params: dict):
+    result = agent.step(messages, **gen_params)
     logger.log_turn(turn, "assistant", result.prose, result.tool_calls,
-                    result.steering_before, result.steering_after, result.raw_output)
+                    result.steering_before, result.steering_after, result.raw_output,
+                    system_prompt=result.system_prompt, unsteered_nll=result.unsteered_nll)
     if result.degenerate_output is not None:
         logger.log({"type": "degeneration", "turn": turn, "snippet": result.degenerate_output,
+                    "reason": result.degenerate_reason, "unsteered_nll": result.unsteered_nll,
                     "dropped": list(result.steering_before)})
     if result.repeated_output is not None:
         logger.log({"type": "repetition", "turn": turn, "snippet": result.repeated_output})
@@ -82,7 +142,7 @@ def run_turn(agent: SteeringAgent, messages: list, logger: SessionLogger, turn:
 
 def main():
     parser = argparse.ArgumentParser()
-    parser.add_argument("--seed", default=None)
+    parser.add_argument("--seed", default=None, help="Seed user message (the first prompt)")
     parser.add_argument("--max-turns", type=int, default=30)
     parser.add_argument("--auto", action="store_true")
     parser.add_argument("--nudge", choices=["status", "legacy"], default="status",
@@ -93,6 +153,18 @@ def main():
     parser.add_argument("--steer", action="append", default=[], metavar="NAME=STRENGTH",
                         help="apply a vector before the first turn, as if the model had called steer "
                              "(one full-strength turn, then decay). Repeatable.")
+    parser.add_argument("--control", default=None, choices=[*ma.CONTROLS, "assign"],
+                        help="Experimental condition ('assign' picks one at random). Giving this "
+                             "marks a controlled study; omit it for a plain demo (real steering)")
+    parser.add_argument("--blind", action="store_true",
+                        help="Never print the condition (it is still in the log; see replay.py --blind)")
+    parser.add_argument("--rng-seed", type=int, default=None, help="Sampling seed (default: random, logged)")
+    parser.add_argument("--temperature", type=float, default=0.7)
+    parser.add_argument("--top-p", type=float, default=0.9)
+    parser.add_argument("--max-new-tokens", type=int, default=300)
+    parser.add_argument("--nll-threshold", type=float, default=None,
+                        help="Treat steered outputs above this unsteered NLL as degenerate (off by default; "
+                             "NLL doesn't separate word salad from heavy role-play, see AGENTS.md)")
     args = parser.parse_args()
     steers = []
     for spec in args.steer:
@@ -104,15 +176,26 @@ def main():
         if not name.strip():
             parser.error(f"--steer expects NAME=STRENGTH, got {spec!r}")
 
+    rng_seed = args.rng_seed if args.rng_seed is not None else random.SystemRandom().randrange(2**31)
+    study = args.control is not None
+    control = random.SystemRandom().choice(ma.CONTROLS) if args.control == "assign" else (args.control or "real")
+    gen_params = {"max_new_tokens": args.max_new_tokens, "temperature": args.temperature, "top_p": args.top_p}
+
     model, tokenizer = load_model_and_tokenizer(args.model)
+    from transformers import set_seed
+    set_seed(rng_seed)
     # Offer every vector we can: unverified index entries too (marked as such in the prompt)
-    agent = SteeringAgent(model, tokenizer, model_name=args.model, include_community=True)
+    agent = SteeringAgent(model, tokenizer, model_name=args.model, include_community=True,
+                          control=control, control_seed=rng_seed, disclose_controls=study,
+                          nll_threshold=args.nll_threshold)
     for name, strength in steers:
         ok, msg = agent.apply(name, strength)
         if not ok:
             raise SystemExit(f"--steer {name}: {msg}")
         print(f"[steer] {msg}")
     logger = SessionLogger()
+    logger.log(session_metadata(agent, args, rng_seed, gen_params))
+    print(f"[session] control={'(blinded)' if args.blind else control} rng_seed={rng_seed}")
     kill = KillSwitch()
 
     messages = []
@@ -139,7 +222,7 @@ def main():
 
         print(f"\n--- Turn {turn+1} ---")
         print(f"Steering status:\n{agent.status()}")
-        result = run_turn(agent, messages, logger, turn + 1)
+        result = run_turn(agent, messages, logger, turn + 1, gen_params)
         print(f"\nAssistant: {result.prose}")
         messages.append({"role": "assistant", "content": result.raw_output})
         turn += 1