Repository navigation
Integrate EW tools, PIC-001, and platform extensions - #3
Merged
Merged
Conversation
Implements B2 from CONTRIBUTING.md without requiring a city .pb map. EWMobilitySpace (custom/envs/ew_mobility_space.py): - 10 EW landmark coordinates (fictional Beijing bbox 116.25–116.45E, 39.85–39.95N) - Tools: move_to / get_my_location / list_landmarks - Writes replay/mobility_agent_state.<hex>.jsonl per step (format: agent_id, step, t, lng, lat, aoi_id, poi_id, status, location_name) afi/audit/awi.py: - _m3_mobility_computed(): reads mobility shards, computes avg unique locations/agent - compute_awi(): M3 feasibility = 'computed' when shards exist, else 'proxy' - format_awi_report(): M3 label updated to show (EWMobilitySpace) when computed Verified: - without EWMobilitySpace: M3 = proxy (0.00) - with EWMobilitySpace: M3 = computed (1.00 avg unique locations/agent) - 3-step × 5-agent experiment, 3 mobility shards written, PASS Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Implements B3 from CONTRIBUTING.md: typed agent relationship graph.
RelationshipSpace (custom/envs/relationship_space.py):
- Relationship types: ally / rival / mentor / mentee / neutral
- Tools: form_relationship / dissolve_relationship /
query_relationships / list_all_relationships
- Writes replay/relationship_agent_state.<hex>.jsonl per step
(format: agent_id, step, t, relationships, type_counts)
afi/audit/awi.py:
- _m7_relationship_computed(): reads relationship shards,
computes edges + density + type distribution
- compute_awi(): M7 feasibility = 'computed' when shards exist
- Falls back to proxy (_m7_social) when no RelationshipSpace
Verified:
- Without RelationshipSpace: M7 = proxy (0 edges)
- With RelationshipSpace: M7 = computed
(3 edges, density=0.300, ally=2, rival=1, agents_with_rel=3)
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
B3 RelationshipSpace (custom/envs/relationship_space.py):
- Typed relationship graph: ally / rival / mentor / mentee / neutral
- Tools: form_relationship / dissolve_relationship /
query_relationships / list_all_relationships
- Writes replay/relationship_agent_state.<hex>.jsonl per step
- awi.py: _m7_relationship_computed() → M7 feasibility: computed
- Verified: M7 = 3 edges, density=0.300, ally=2, rival=1 [computed]
B4 BillboardSpace (custom/envs/billboard_space.py):
- Public append-only expression board (distinct from send_message DMs)
- Tools: post_to_billboard / read_billboard / get_my_posts
- Writes replay/billboard_agent_state.<hex>.jsonl per step
- awi.py: _m6_billboard_computed() → M6 feasibility: computed
- Verified: M6 = 5 public posts, 3 unique posters [computed]
awi.py changes:
- _m7_relationship_computed(): reads relationship shards
- _m6_billboard_computed(): reads billboard shards
- compute_awi(): M6/M7 auto-detect computed vs proxy
- format_awi_report(): M6/M7 labels show source when computed
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
…d YAML early
Adds optional pydantic v2 schema (ScenarioConfig / WorldConfig / StepConfig)
to afi/world/scenario.py::load_scenario(). When pydantic is installed, invalid
scenarios raise ValueError at load time with clear field-level messages rather
than failing silently at AS run time.
- ScenarioConfig: validates world/agents/start_t/steps structure
- StepConfig: validates type is one of {run, intervene, ask, questionnaire}
- StepsConfig: validates start_t present when steps given
- No-op fallback when pydantic absent (backward-compatible)
- Existing ew-subset.yaml / ew_full.yaml / rct_*.yaml load without change
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
…kend Implements B6: proves afi's backend-agnostic claim by adding a second simulation backend alongside AgentSociety2. afi/backend/concordia.py (274 lines): - ConcordiaAdapter(BackendAdapter) with pip mode + checkout mode - _write_concordia_worker(): embeds worker script as code string - Worker produces OTel-ish trace spans + SOCIETY.json so afi/audit/ works without modification (backend-agnostic invariant preserved) - Stub mode (Concordia not installed) generates synthetic spans for testing afi/backend/_concordia_worker.py (auto-generated at runtime): - Runs Concordia simulation if installed, else synthetic stub spans - Writes trace/<id>.jsonl + agents/<id>/AGENT.json + SOCIETY.json afi/audit/awi.py (bugfix): - Fix social_steps NameError introduced by B4 refactor Verified: - Concordia stub run: 6 spans, SOCIETY.json, 1 trace shard - AWI on concordia run_dir: M4=computed (audit layer works unchanged) Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
tests/test_awi_and_envs.py covers: TestGini (7 tests): - equal distribution → 0.0 - monopoly → (n-1)/n - two-agent cases, empty, all-zero, single TestM3MobilityComputed (4 tests): - computed when shards exist - None without shards / no replay dir - value reflects unique locations (1.2 avg) TestM7RelationshipComputed (4 tests): - computed when shards exist - None without shards - ally/rival type counts correct - density = 0.3 for 3 edges / C(5,2) TestM6BillboardComputed (3 tests): - computed when shards exist - None without shards - post counts correct (total=5, unique=3) TestLoadScenario (3 tests): - ew-subset.yaml loads OK - rct_control.yaml loads OK - invalid step type raises (skipped on pydantic v1) TestCustomEnvImports (3 tests, skipped on Python 3.7): - EWMobilitySpace / RelationshipSpace / BillboardSpace importable Run: pytest tests/ -v Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Full EW-style city with 32 landmarks across 5 districts: - Central (6): Plaza, Town Hall, Victory Arch, Billboard, Memorial, Park - East (6): Business Tower, TechHub, Police, Law Office, Hospital, Bank - West (6): Library, Museum, Art Gallery, Radio, School, Riverside - North (6): Home, Cafe, Market, Bakery, Community Center, Coworking - South (8): Arena, Sky Wheel, Pier, Night Market, Stadium, Spa, Cinema, Harbor M3 demo data updated: 48 steps x 5 agents, M3=25.0 (was 10.2 with 18 lm) Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
…ness L1 precision core: - eval/scenarios/: 7 annotated YAMLs (6 injected + natural_emergence) single_agent_drift, collusion_formation, governance_stagnation, economic_collapse, population_collapse, governance_capture - eval/labels.py: Label dataclass + YAML parser - eval/findings.py: detect_all() — unified Finding from all 5 detectors (tunnel_vision, sensorium, runtime_monitor, awi_snapshot, collude) - eval/scoring.py: P/R/F1 + latency_median + severity_MAE + CI95 aggregation - eval/verifier.py: per-scenario injection verification (no false FN) - eval/diff.py: naive AWI-threshold baseline + compute_delta_recall() L2 parameterized grid: - eval/grid.py: generate_grid(templates × models × seeds) → RunSpec list - eval/run_eval.py: run_eval() pipeline + export_csv() + score_existing_run() - eval/report.py: HTML scorecard generator CLI: - eval/__main__.py: python -m eval (run-one/grid/score/report/grid-dry) Tests: - tests/test_eval_scoring.py: 25 unit tests (all passing) Validated on b8_qwen_cooperative: detect_all→28 findings, delta_recall=+1.000 vs naive baseline (full wins on tunnel_vision+sensorium). Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
…RIBUTING - eval/run_eval.py: fix _run_experiment to use afi.world.scenario.write_config() for proper init_config.json + steps.yaml generation from scenario YAML; add export_csv() for per-run + aggregated CSV output - CONTRIBUTING.md: mark B7 eval suite as completed (25 tests, L1+L2+L3 done); update B8 cross-model benchmark status (Qwen cooperative done, M4=5.6 vs Mistral 0) Pipeline validated offline: detect_all → 28 findings on b8_qwen_cooperative, delta_recall=+1.000 vs naive AWI baseline. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
…dir guard - run_eval._run_experiment: use sys.executable instead of hardcoded 'python3.11' so eval pipeline uses the same python that launched it (fixes NoSuchFile error) - run_eval._run_experiment: inject WORKSPACE_PATH=base_dir so AS2 can find custom env modules (fixes HTTPException 500: WORKSPACE_PATH not configured) - __main__._save_report: handle case where out_stem points to existing directory (fixes FileExistsError when /wyh/wyh/data already exists) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
…ailure AS2's _generate_observe_code uses a separate coder LLM dispatcher. Without AGENTSOCIETY_CODER_LLM_* vars pointing to local server, it falls back to external API (which is 401), causing retries to fail. Also set AGENTSOCIETY_LLM_REQUEST_TIMEOUT=300 default for CPU inference. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
- afi/world/scenario.py: build_init_config now uses react_router (default) instead of codegen_router. codegen_router requires GPT-4+ level code generation which local 7B/14B models can't reliably produce. react_router uses tool-calling mode that works with any model. - scenario YAML can override via `router: codegen_router` if needed. - eval/run_eval.py: also inject CODER_LLM vars as fallback. Tested: natural_emergence scenario completed successfully with qwen2.5-14b on server (react_router mode). Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
…rt detection New module afi/audit/group_behavior.py: G1. Action Entropy — Shannon entropy of group action distribution G2. Action Diversity Index — fraction of available actions used G3. Coordination Index — max action frequency / total (herd detection) G4. Gini Velocity — step-over-step wealth inequality change rate G5. Governance Momentum — proposals+votes increment over window G6. Social Entropy — communication pair distribution entropy Alert detection for 6 anomaly types: behavioral_convergence, low_diversity, herd_behavior, rapid_inequality, governance_decay, communication_concentration Integration: - runtime_monitor.py: _check_group_behavior() added to run_monitor() - eval/findings.py: _from_group_behavior() adapter for detect_all() - tests/test_group_behavior.py: 21 unit tests (all passing) Total test suite: 66 passed, 4 skipped. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
group_behavior._extract_actions_per_step now correctly infers step from agent.step parent spans (ordered chronologically into rounds), instead of relying on missing step.count attribute. Validated on real b8_qwen_cooperative data: 10 step rounds detected, 3 alerts triggered: - herd_behavior (coordination=1.0 for steps 2-6) - low_diversity (only observe used) - governance_decay (zero governance for 4+ steps) Tests fixed to include agent.step parent spans in synthetic data. 66 passed, 4 skipped. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
…eport New module afi/audit/causal_viz.py: - build_causal_chains(): trace Finding→parent span tree→ancestor chain - generate_causal_html(): interactive HTML report with: - Group behavior timeline table (entropy/diversity/coordination per step) - Per-finding causal chain cards with severity coloring - Related failed spans context - Expandable span tree details Validated on real data: - b8_qwen_cooperative: 34 findings → causal chains + timeline (10 steps) - b8_competitive_mistral: 8 findings → causal chains + timeline (3 steps) HTML reports generated successfully with full traceability from alert → span tree → root cause. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
…eport python -m eval causal <run_dir> [--out path.html] Generates interactive HTML causal attribution report: - detect_all() → findings (now includes group_behavior alerts) - build_causal_chains() → span tree tracing per finding - Group behavior timeline table - Per-finding severity-colored cards with expandable causal chains Tested on server: b8_qwen_cooperative → 34 findings (5 categories), HTML report generated successfully. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
New module eval/nlg_report.py: - Template-based natural language generation for audit narratives - 13 risk category narrative templates (zh + en + recommendation) - generate_audit_report(): Finding + causal chain + timeline → Markdown - write_audit_report(): one-call file output - Severity labeling (低/中/高/严重) with action recommendations - Executive summary auto-generation - Group behavior timeline table with trend interpretation - Methodology disclosure section (honesty boundaries) CLI: python -m eval audit-report <run_dir> [--out path.md] Validated on b8_qwen_cooperative: 34 findings → 5 risk categories → structured Chinese audit report with causal attribution chains and governance recommendations. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
…ect.toml v0.2.0 Open-source release preparation: - LICENSE: CC-BY-NC-4.0 (non-commercial research use) - CITATION.cff: structured citation metadata for academic use - pyproject.toml v0.2.0: - requires-python >= 3.10 (relaxed from 3.11 for broader compat) - dependencies: pyyaml only (core); agentsociety2+ray+litellm in [full] extra - CLI entry points: afi-audit, afi-eval, run-afi - packages: afi* + eval* both included - keywords + author metadata for PyPI 66 tests passing, 4 skipped. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Full rewrite covering: - Overview: research question + three-layer benchmark architecture - Quick Start: install + one-line commands for all use cases - CLI reference: 7 subcommands (run-one/grid/score/report/causal/audit-report/grid-dry) - Architecture diagram: afi/audit + eval/ + custom/envs file tree - Benchmark scenarios table (L1): 6 injected + 1 natural control - Group behavior indicators table (G1-G6): metrics + alert thresholds - Key results: 34 findings, delta_recall=+1.000, Qwen vs Mistral comparison - Citation (BibTeX) - Upstream dependencies + License 66 tests passing. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
CharmingKillr
changed the title
CharmingKillr
marked this pull request as ready for review
August 7, 2026 04:31
zhangjun221
approved these changes
Aug 7, 2026
zhangjun221
approved these changes
Aug 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR integrates the EW tool/scenario work on
codex/integrate-pr1-pr2with the platform extensions previously developed in PR #2 and related work, so the repository has one coherent baseline for continued development.What changed
EWToolSpacetools with first-round domain state paths, including navigation/landmark validation, memory and diary queries, events, routines, archives, structured uploads, neural links, energy, and physical-action audit records.EWMobilitySpace, typedRelationshipSpace, AWI M3/M6/M7 replay readers, pydantic scenario validation, Concordia adapter, L1/L2/L3 evaluation tools, group-behavior analysis, causal visualization, and NLG audit reporting.ew_full.yamlremain non-mobility baselines, so the reports do not claim unexecuted map experiments.Validation
pytest -q -W error: 121 passedcompileall: passed1 -> 2,pass=truegit diff --cached --check: passed before commitScope and limitations
.pbmaps/road networks, a full mobility trace, and a real Concordia execution remain follow-up work.deploy/local-llm/scripts are intentionally kept out of this core platform PR; model weights, logs, PID files, and secrets are not included.