BROKEN Feat/issue 591 scale validation

[bedaHovorka]

Scale validation for Goal 1 — 5-train correctness & 20-train performance (#591)

Summary

Goal 1 SP7 (#591): validate that the multi-train simulation meets the scale and
performance bar — a 5-train scenario where all trains complete their routes on a
shared topology, and a 20-train stress scenario with a real-time ratio ≥ 1× and
no deadlocks/livelocks across 10 consecutive runs.

Adds MultiTrainScaleValidationTest (5-train completeness + 20-train ×10 stress)
on the praha-hlavni-nadrazi.xml fixture, exposed to commonTest via
NetworkResources.

These tests double as the deferred, final verification of Goal 1's own
success criteria ("5+ trains running simultaneously", "performance remains
acceptable up to 20 concurrent trains") — LONG_TERM_GOALS.md marks Goal 1
✅ COMPLETE, but that scale claim was never actually exercised until this PR.
Treat this test suite as the acceptance gate for that outstanding "Goal 1B"
slice. Per docs/GOAL_10_EXPECTATIONS_CRITIQUE.md §4/§5, the 5-train Praha
scenario is also recommended as Goal 10 (#532)'s Stage-A paramount example
going forward, so this fixture is expected to stay load-bearing beyond #591.

⚠️ Status: BROKEN — root cause identified, CI currently blocked one layer earlier

Headline finding: the hang is a CPU livelock, not a wait-for deadlock.
Full analysis in docs/ISSUE_591_SCALE_LIVELOCK_ANALYSIS.md (traffic-simulation-expert,
TEAM.md): MultiTrainLoop.reserveEntryPath enumerates every topological
entry→exit path (760 for N-Bypass → S-Vrs-3 on Praha) and calls
reservePath(trainName, inIo, outIo) once per candidate — but that call
ignores which candidate was passed, so after the first AllPathsBlocked the
remaining ~759 calls are guaranteed-identical ~2.7s failures. One dispatcher
iteration costs ~34 minutes inside a single kDisco event, freezing sim time at
t≈21.5 with trains 1–4 parked mid-journey. 5-train passes because its routes
are block-disjoint, so the hot loop is never entered.

Current CI blocker (secondary, plumbing): the latest run
(28812932322,
2026-07-06) (or later) no longer times out at 10 minutes — it now fails fast in 2m14s.
Both fiveTrainCompleteness and twentyTrainStress throw
org.koin.core.error.NoBeanDefFoundException for CollisionDetectionService
before the simulation runs at all. MultiTrainScaleValidationTest uses
CommonKoinTestBase's default commonCoreTestModule (core-test/src/commonMain),
which never received the CollisionDetectionService binding that the JVM-only
CoreTestModule.kt has — a gap from Goal 3's collision-detection merge landing
on develop/goal-10 after this branch's commonTest fixture was written. This
needs a one-line Koin binding fix before the livelock above can even be
re-observed in CI.

Scope — resolved, now #591-only

The base was retargeted from develop to goal-10 (review comment,
2026-06-30). The branch is now 7 commits ahead of goal-10, all
test(#591)/docs(#591):

• test(#591) expose praha-hlavni-nadrazi.xml fixture to commonTest
• test(#591) add 5-train Praha correctness validation
• test(#591) add 20-train Praha stress scenario (draft)
• docs(#591) scale-validation design spec and plan
• docs(#591) livelock root-cause analysis
• docs(#591) Goal 9 resolution options research
• docs(goal-10) Goal 10 (Goal 10: AI Dispatcher Routing #532) expectations critique

The previously-flagged production-code scope creep (#569, #582, #584, #587,
#589, #618) is no longer part of this PR's diff — those commits are now
goal-10 base history, not PR-owned changes. Review concern #2 ("scope far
wider than #591") is resolved.

Research & design docs added this cycle

• docs/ISSUE_591_SCALE_LIVELOCK_ANALYSIS.md — livelock root cause (see above)
• docs/ISSUE_591_GOAL9_RESOLUTION_OPTIONS.md — 8 options (A–H) mapping
  shipped Goal 9 features (event-driven retry, reroute, temporal admission
  control, contention telemetry, etc.) onto the scale problem, with an explicit
  note that Goal 9 alone cannot fix the P0 hot-loop defect, route-search cost,
  or the full-path capacity ceiling
• docs/GOAL_10_EXPECTATIONS_CRITIQUE.md — critiques Goal 10: AI Dispatcher Routing #532's description
  (no declared canon, stale dependency claims, falsified 500ms decision
  criterion, non-determinism vs. acceptance) and proposes staged A/B/C success
  criteria with this PR's Praha 5-train scenario as the Stage-A example
• docs/superpowers/specs/2026-06-30-591-scale-validation-design.md +
  docs/superpowers/plans/2026-06-30-591-scale-validation.md — design spec and
  implementation plan for the scale-validation work itself

Related Issues

• Closes Ex Goal 1 SP7: Scale validation for 5-train correctness and 20-train performance #591 (pending — acceptance criteria not yet met)
• Feeds Goal 10 (Goal 10: AI Dispatcher Routing #532) as the proposed Stage-A paramount example
• Builds on Goal 1 → Goal 10 prereq: expose block-release/occupancy events to external (non-train) subscribers #569, Goal 1 SP3: Add block release events and wait-and-enter behaviour #582, Goal 1 SP4: Create ThreeTrainLoop prototype scenario #584, Goal 1 SP5: Validate two-train concurrency without deadlock or ordering errors #587, Goal 1 SP6: Run 1000-iteration deterministic race test for three-train scenario #589, Fix train-completion cleanup: unlock switches in unregister (I1) + emit legacy BlockOccupancyEvent on release paths (I2) #618 (now base history via goal-10)

Acceptance criteria (#591)

• 5-train scenario completes all routes, no exceptions/occupancy errors —
  unverified (CI blocked by Koin binding gap, see Status)
• 20-train scenario real-time ratio ≥ 1× — unverified (same blocker)
• No deadlocks/livelocks in 10 consecutive 20-train runs — root-caused as
  a livelock (see Status); fix not yet implemented
• Performance bottleneck summary documented if ratio falls short — n/a
  until green

Recommended next steps (sequencing)

1. Fix the commonCoreTestModule Koin gap — add the CollisionDetectionService
   binding so the tests can run far enough to re-exercise the livelock in CI.
2. P0 (livelock analysis §7): break reserveEntryPath's hot loop — return on
   AllPathsBlocked instead of retrying 759 guaranteed-identical candidates.
3. Address still-open review items: pin kdisco off 0.6.0-SNAPSHOT, add a
   per-scenario timeout to twentyTrainStress (assertTimeoutPreemptively or
   a kDisco step ceiling) so a future hang fails fast instead of hitting the
   CI wall, consider tagging the 20-train stress as integration-test.
4. Land the P0 fix, get both tests green, close out Ex Goal 1 SP7: Scale validation for 5-train correctness and 20-train performance #591's acceptance criteria.
5. Longer-term: Goal 9 options doc's P1/P2 (event-driven retry, route caching,
   incremental reservation, watchdog) and the Goal 10 (Goal 10: AI Dispatcher Routing #532) re-description
   proposed in the critique doc.

Test Coverage

New tests (core/src/commonTest/.../sim/MultiTrainScaleValidationTest.kt):

• 5-train completeness (fiveTrainCompleteness)
• 20-train stress, 10 runs (twentyTrainStress)

Not run to green on this branch — currently blocked by the Koin binding
gap (step 1 above), and once that's fixed, by the livelock itself (step 2).

CI/CD Status

• ❌ build — fails in 2m14s: both MultiTrainScaleValidationTest methods
  throw NoBeanDefFoundException for CollisionDetectionService (Koin
  plumbing gap, not the livelock — see Status)
• ⏸ fast-sim, JVM/native smoke comparison — skipped (dependent on build)
• ⏸ SonarQube — pending

[bedaHovorka]

Review — request changes

Short review per request (no implementation, no local build). Findings ranked
by severity.

🔴 1. CI is red on timeout, not on an assertion — suite hangs

Both build runs fail because Run unit tests times out after 10 min. This
is a deadlock/livelock in the multi-train paths — the exact failure mode #591's
acceptance criteria forbid ("No deadlocks or livelocks in 10 consecutive 20-train
runs"). Until the hang is found, all four #591 acceptance checkboxes are
unverifiable. The PR title "BROKEN" reflects this; please root-cause before
re-running.

🔴 2. Scope is far wider than #591

14 commits ahead of develop, spanning #569, #582, #584, #587, #589, #618 plus
#591 — including production changes (BlockEvent, onBlockEvent/onSimulationEvent,
configureIntermediateSemaphores, switch-unlock in unregister, kDisco
Resource gating). #591 is test-only. Recommend splitting the production work
into its own PRs and landing #591's tests on top once green; this also makes
bisecting the hang tractable.

🟠 3. kDisco bumped to 0.6.0-SNAPSHOT

gradle.properties: kdiscoVersion 0.5.0 → 0.6.0-SNAPSHOT. CI checkKdisco
warns "kDisco 0.6.0-SNAPSHOT not found in mavenLocal". A snapshot is not
reproducible and breaks downstream :fast-sim/native redistribution. Pin to a
released kDisco version before merge.

🟠 4. twentyTrainStress has no per-run timeout

MultiTrainScaleValidationTest runs 10 sequential scenarios inside one @Test
with no isolation/timeout, so a hang in run 1 blocks the whole suite until the
10-min CI wall. Same risk from the 1000-iteration ThreeTrainLoopRaceTest
(#589). Add a per-scenario cap (JUnit assertTimeoutPreemptively, or a kDisco
step/time ceiling) so a hang fails fast with a named scenario instead of
hitting the global CI timeout.

🟡 5. Full-path reservation may not satisfy the scale bar

MultiTrainLoop reserves the full entry-to-exit path and "serializes trains
that share any block" (its own KDoc). With 20 trains over 5 north-entries / 6
south-exits on Praha and MAX_CONCURRENT_TRAINS=20, contention is high; if no
fully-free path is found before endTime=1200, trainsExited == 20 can't hold.
Either confirm Praha admits ~20 near-disjoint paths, or the scale bar needs the
incremental-reservation slice first (the KDoc already calls this out as a future
slice).

🟡 6. Stress test lives in commonTest but measures wall-clock

twentyTrainStress uses currentTimeMillisKMP() for the real-time ratio and
runs 10× — on native (:fast-sim) that ratio is meaningless and the budget is
large. Consider tagging the 20-train stress as integration-test (matches the
pattern in cdf7ecb, which tagged PathReservation tests as integration) and
keeping only the 5-train correctness check in commonTest.

---

No code changes made, no local build run (per request).

[bedaHovorka]

🔴 2. Scope is far wider than #591

14 commits ahead of develop, spanning #569, #582, #584, #587, #589, #618 plus #591 — including production changes (BlockEvent, onBlockEvent/onSimulationEvent, configureIntermediateSemaphores, switch-unlock in unregister, kDisco Resource gating). #591 is test-only. Recommend splitting the production work into its own PRs and landing #591's tests on top once green; this also makes bisecting the hang tractable.

changed base

[bedaHovorka]

@copilot : This needs very complex analyse and deep research, what logic/behaviour is missing in simulation model -> please enagne traffic simulation expert from @TEAM.md and create .md with analyse/research output

[Copilot]

@copilot : This needs very complex analyse and deep research, what logic/behaviour is missing in simulation model -> please enagne traffic simulation expert from @TEAM.md and create .md...

Deep analysis done as traffic-simulation-expert (TEAM.md) — published in docs/ISSUE_591_SCALE_LIVELOCK_ANALYSIS.md (commit fb89fc7).

Key findings (reproduced locally with jstack + debug logs):

• The hang is a CPU livelock, not a wait-for deadlock. MultiTrainLoop.reserveEntryPath loops over all topological entry→exit paths (measured 760 for N-Bypass → S-Vrs-3 on Praha) and calls reservePath(trainName, inIo, outIo) once per candidate — but that call ignores the candidate path, so after the first AllPathsBlocked all remaining 759 calls are guaranteed identical failures. Each call re-runs a full exhaustive route enumeration (~2.7 s), so one dispatcher iteration costs ~34 min inside a single kDisco event, freezing sim time at t≈21.5 — trains 1–4 park mid-journey and never release blocks.
• Why 5-train passes: its routes are block-disjoint, so the first candidate always succeeds and the hot loop is never entered.
• Missing simulation-model behaviour (§5 of the doc): incremental signal-to-signal (Fahrstraße) route reservation with release-behind-train, deadlock-avoidance for opposing routes, event-driven dispatcher retry on block-release events (Goal 1 → Goal 10 prereq: expose block-release/occupancy events to external (non-train) subscribers #569 API already exists), cached/k-shortest route search (current findAllPaths enumerates everything before applying maxPaths), and a livelock/deadlock watchdog. The kDisco Resource gate is currently inert (acquired/released within one non-suspending section).
• Recommended sequencing (§7): P0 break the hot loop (return on AllPathsBlocked), P1 event-driven retry + route caching, P2 incremental reservation + watchdog. Until P0–P1 land, Ex Goal 1 SP7: Scale validation for 5-train correctness and 20-train performance #591's acceptance criteria are unreachable on this branch.

Per request, analysis only — no production code changed.

[bedaHovorka]

needs at least goal 3, 9. Undeterministic reasoning in goal 10

[bedaHovorka]

@copilot Please elaborate in new .md file, how can features from Goal 9 helps to solve this problem. Please do research, not implement. I expected comprehensive document with described many options to solve it.

[Copilot]

Research done as traffic-simulation-expert — published in docs/ISSUE_591_GOAL9_RESOLUTION_OPTIONS.md (commit 8a29b43), cross-linked from §5.3 of the parent analysis. Research only, no implementation.

The document catalogues 8 options (A–H) mapping shipped Goal 9 features onto the scale problem, each with mechanism, trade-offs, gaps, and cost/benefit:

• A — event-driven dispatcher retry on ConflictDetectedEvent (SP1) + BlockEvent.BlockReleased (Goal 1 → Goal 10 prereq: expose block-release/occupancy events to external (non-train) subscribers #569), replacing per-sim-second polling
• B — HoldTrain resolutions as a principled backoff policy (cheapest wire-in)
• C — Reroute resolutions to spread trains across the Praha ladder (the only option raising effective concurrency toward 20 trains under full-path reservation) — gated on route caching: generateRerouteCandidates scans all ~110 InOut pairs at ~2.7 s/exhaustive search, so calling it from the dispatcher today would worsen the livelock
• D — SpeedAdjust for temporal de-confliction (fewer hard stops ⇒ shorter block holds)
• E — wiring the inert TemporalConflictDetector (SP2) as predictive admission control, the only mechanism that reduces how often AllPathsBlocked occurs at all
• F — AutoConflictResolutionService (SP5) as the single decision seam — exactly the Goal 10 wiring point its KDoc reserves
• G — preference learning (SC4) with an explicit determinism analysis for the "10 consecutive runs" criterion (addresses the non-deterministic-reasoning concern: acceptance runs use the base deterministic ranker; learned/AI policies stay behind the SP5 seam)
• H — Goal 9 contention telemetry (blockedSince, conflict-event wait-for edges) as the watchdog substrate that makes "no deadlock/livelock" directly assertable instead of a CI-timeout inference

§5 states what Goal 9 cannot solve (P0 hot-loop defect, route-search cost, full-path capacity ceiling), §6 gives a comparative matrix + wave-based sequencing, and §7 covers the Goal 3/Goal 10 cross-dependencies from this thread.

[bedaHovorka]

§5 states what Goal 9 cannot solve (P0 hot-loop defect, route-search cost, full-path capacity ceiling), §6 gives a comparative matrix + wave-based sequencing, and §7 covers the Goal 3/Goal 10 cross-dependencies from this thread.

@copilot what we expect from Goal 10 #532? Please elaborate in next new .md file. I expected comprehensive document with described critics and improvement, how to change Goal 10 description. Consider that we have only limited sources and do only small examples now. This example will be paramount of our effort in future.

[Copilot]

Research done as traffic-simulation-expert — published in docs/GOAL_10_EXPECTATIONS_CRITIQUE.md (commit e0bec9a), cross-linked from §7 of the Goal 9 options doc. Research only, no implementation.

The document answers "what do we expect from Goal 10 (#532)" and how to change its description:

• §2 — the three current sources compared: LONG_TERM_GOALS.md, the Goal 10: AI Dispatcher Routing #532 issue body (rule-based dispatcher), and the agent-based companion plan (Koog/Ollama/DSL, ~40 sub-issues)
• §3 Critique (C1–C7) — no declared canon between the sources; stale dependency claims (Goal 9 and Goal 3 are now ✅ COMPLETE, while the real blockers — the Ex Goal 1 SP7: Scale validation for 5-train correctness and 20-train performance #591 dispatcher defects and Goal 6 metrics — are uncaptured); the 500 ms decision criterion is falsified by measurement (~2.7 s/search on Praha) and mis-typed for a discrete-event engine; non-determinism vs. acceptance (the "undeterministic reasoning" concern from this thread); scope vs. the limited-resources constraint; the paramount example never named; duplication risk with shipped Goal 9 machinery
• §4 — positive definition: four expectations (autonomy, authority handover, explainability, substitutability), each mapped to an existing seam
• §5 — paramount example recommendation: Praha 5-train with dispatcher-chosen routes as the Stage-A acceptance scenario (upgrading Ex Goal 1 SP7: Scale validation for 5-train correctness and 20-train performance #591's green test from scripted to dispatcher-selected routes); vyhybna stays the SP4 smoke slice; 20-train demoted to measured-not-gated until Ex Goal 1 SP7: Scale validation for 5-train correctness and 20-train performance #591 P1/P2 land
• §6 — drop-in replacement text for the Goal 10 description: staged A/B/C criteria (deterministic MVP → modes+explainability → pluggable planners/LLM as stretch), determinism clause for acceptance runs, real-time-ratio budget replacing 500 ms, planner-vs-planner comparison replacing the unfalsifiable "human dispatcher parity", corrected dependencies, agent-submodule placement — with a per-edit rationale table
• §7 — concrete edit checklist for issue Goal 10: AI Dispatcher Routing #532 (incl. re-ordering the SP1/SP3 LLM programme behind Stage A under the limited-resources constraint)
• §8 — self-critique of the proposal's own risks

[bedaHovorka]

@copilot

• paramount example for Goal-10 finaly remains shunting loop only, paramount example for 1B will be praha hlavni nadrazi
• Goal-9 is not realistic and in future will be prepared more realistic update Goal-9B, after Goal-1B

[Copilot]

Recorded these decisions in the research docs (e1d6aab):

• docs/GOAL_10_EXPECTATIONS_CRITIQUE.md — superseding-decision note at the top plus inline notes at R3 and §5: Goal 10's paramount example remains the vyhybna.xml shunting loop only; the Praha hlavní nádraží 5-train scenario moves to Goal 1B as its paramount example. The original §5/§6 recommendation is kept as historical analysis.
• docs/ISSUE_591_GOAL9_RESOLUTION_OPTIONS.md — note that Goal 9 as shipped is judged not realistic and a more realistic Goal 9B update will follow after Goal 1B; the options catalogue now reads as inventory for Goal 9B, and the cross-reference to the critique doc reflects the new paramount-example split.

[bedaHovorka]

@copilot rebased to goal-10, check if is in doc is not anything totally outdated

[Copilot]

Re-checked all three research docs against the rebased goal-10 base and marked what was overtaken by events (a0f0c6a):

Still valid (verified in code): the livelock root causes — MultiTrainLoop.reserveEntryPath's path-independent reservePath hot loop (now lines 289–352, unchanged) and the uncached exhaustive route search — plus the §7 P0–P2 sequencing.

Marked as superseded/updated:

• ISSUE_591_SCALE_LIVELOCK_ANALYSIS.md — CI header is historical (failure mode is now the NoBeanDefFoundException Koin gap, not the 10-min timeout); noted the SP0 dispatcher extraction (SP0.1: Extract RuleBasedDispatcher from ShuntingLoop; scaffold :dispatcher-agent module #720–SP0.10: AgentLoopDriver — paced sense-decide-act loop via SimulationController #739: declarative decide() seam, AgentLoopDriver in :dispatcher-agent) as the wiring point for §5.3/§7 fixes — currently ShuntingLoop-only, needs revision for station-scale; noted Train fixes Fix visual backward jump of train animation at segment boundaries #633/Fix visual backward jump of train animation at segment boundaries #718 now in base; kDisco pinned to released 0.6.0.
• GOAL_10_EXPECTATIONS_CRITIQUE.md — C1 (no canon) resolved by the 2026-07-06 Goal 10: AI Dispatcher Routing #532 rewrite + superseding-plan comment; agent module exists as :dispatcher-agent; the "Critical Files in core/.../context/" concern is addressed (seam lives in core/sim/); determinism (R5) adopted as Stage A3; Goal 6 SP1 (Goal 6 SP1: metrics domain model + collection API #717) landed.
• ISSUE_591_GOAL9_RESOLUTION_OPTIONS.md — §3 inventory re-verified present in core/sim/conflict/; options' "dispatcher wiring point" redirected to the SP0 seam instead of MultiTrainLoop's inline loop.

[bedaHovorka]

FYI : GLM Model in #633 also found this race condition in Front/Tail and retrival of approvedTrains in MultiTrainLoop

49a7588

[bedaHovorka]

FYI : second fix in model - train navigation calling:

in #718
c6d27c7

[bedaHovorka]

both Train model fixies merged to develop, and since is this PR rebased on develop, is also here

[bedaHovorka]

routing issue #742 fixed.

[bedaHovorka]

See also resolved issues in kdisco

[bedaHovorka]

in goal 10 was extracted dispatcher - for more complex station must be revided : ee9260d

[bedaHovorka]

please check current version, which was merged from #720

[bedaHovorka]

and rework from imperative to declarative forms of station control by Dispatcher

#735

[bedaHovorka]

dispatcher-agent/src/main/kotlin/cz/vutbr/fit/interlockSim/dispatcher/AgentLoopDriver.kt #739 #740

[bedaHovorka]

Goal 10 preparation (#535) is about "goodbye" hardoded shunting loop and use general code to use any xml topology

From other site, xml must have unique ids of its elements tu identify objects by dispatcher.

[bedaHovorka]

also in another bugfix add names of elements to log.

here is before that fix
… t=21.3 BLOCK_TRANSITION: Train 2 leaving block

you don't know from it which block is it

[bedaHovorka]

compass_artifact_wf-0594016f-0705-55c3-9303-3d3e50c6242a_text_markdown.md

about agent memory