CPU Multithreading - Single/Multithread support in Options

docs/multicore.md

@@ -0,0 +1,63 @@
+# Multi-core calculation support — dev handoff notes
+
+## Changelog (PR summary)
+
+**Adds multi-core support for the four "iterate candidates, rank outcomes" features, with a user toggle (default: multi-core) and automatic single-core fallback.**
+
+Parallelized features:
+- Passive tree node power / heat map / "Show Power Report" (`CalcsTab:PowerBuilder`)
+- Compare tab power report (`CompareTab`)
+- Anoint notable stat-sorting (`NotableDBControl`)
+- Item DB stat-sorting (`ItemDBControl`)
+
+### Architecture
+
+Workers are one-shot isolated Lua VMs started via the engine's existing `LaunchSubScript` primitive (no engine changes). Each worker bootstraps a headless PoB (HeadlessWrapper pattern), reconstructs the build from an in-memory `SaveDB` XML snapshot, computes its batch using **the same per-candidate functions the single-core path runs**, and returns one JSON payload (dkjson). The main thread partitions candidates, aggregates progress via subList calls, validates results, and merges by stable IDs.
+
+### New files
+- `src/Modules/ParallelRunner.lua` — worker pool / job manager: launch, cancel, progress aggregation, baseline validation, per-session sticky fallback, dev-mode job log (`src/parallel_log.txt`)
+- `src/CalcWorker.lua` — worker script (read via `io.open`, passed as scriptText like `UpdateCheck.lua`): hardened headless stubs, script-path-relative `io.open` resolution (worker threads don't inherit a guaranteed cwd), env probe mode
+- `src/Modules/PowerCalcTasks.lua` — shared per-candidate compute logic + JSON number sentinels + baseline extraction, loaded by both main thread and workers
+- `spec/System/TestParallelTasks_spec.lua` — equivalence tests: single-core vs worker entry points in-process, through a JSON round trip, for all four features; merge staleness tests
+
+### Modified files
+- `src/Launch.lua` — `OnSubError` now notifies CUSTOM subscript callbacks (`(nil, errMsg)` — audited compatible with all existing users); nil-guards in `OnSubError`/`OnSubFinished` for aborted subscripts; dev-mode F7 = worker environment probe (report written to `src/probe_report.txt`)
+- `src/Classes/CalcsTab.lua` — `PowerBuilder(filter)` optional candidate filter (nil = byte-identical legacy behavior); `EnumeratePowerCandidates`/`BuildPowerBatches` (modKey-grouped, weight-balanced batches carrying authoritative `node.path`/`node.depends` id lists); dual-path `BuildPower` with duplicate-launch guard; `LaunchParallelPowerBuild`/`MergeParallelPower` (staleness via `outputRevision`)
+- `src/Classes/CompareTab.lua` — `ComparePowerBuilder` split into enumerate / `ComputeComparePowerCandidate` / format-row so both paths share candidate computation; dual-path report runner
+- `src/Classes/NotableDBControl.lua`, `src/Classes/ItemDBControl.lua` — `ListBuilder` split into filter / measure / finish; dual-path `Draw`; candidates keyed by node id / item name
+- `src/Modules/Main.lua` — `computationMode` ("MULTI"/"SINGLE", default MULTI) and `workerCount` (0 = auto) settings, persisted in `Settings.xml`; Options popup UI; loads `ParallelRunner`/`PowerCalcTasks` as globals
+- `src/Modules/Build.lua` — cancels outstanding worker jobs on shutdown
+- `src/Modules/DataLegionLookUpTableHelper.lua` — **standalone bugfix**: `loadJewelFile` could cache empty jewel data to `<jewel>.bin` when `Inflate` is stubbed (headless/busted), silently corrupting timeless jewel stats for the full app afterwards. Now rejects empty data, reads an existing `.bin` directly when `NewFileSearch` is unavailable, and never writes the shared cache from headless contexts
+
+### Correctness guarantees
+- Workers and single-core run identical code; results cross the thread boundary as JSON with `inf`/NaN sentinels
+- Every worker returns a baseline stat vector recomputed from its reconstructed build; any relative deviation > 1e-3 from the main thread's baseline fails the whole job → automatic single-core fallback (sticky for the session, cleared by toggling the option or a successful F7 probe)
+- Main thread sends each candidate's live `path`/`depends` node ids: a spec rebuilt from XML resolves equal-length shortest-path ties in different `pairs()` order, which would otherwise change `pathPower` vs what the UI displays
+- Results merge only if `build.outputRevision` is unchanged since the snapshot; any build change mid-job cancels/discards and re-runs
+- Worker errors surface once via `OnSubError` → fallback; workers never write shared caches
+- Headless/CI: `HeadlessWrapper.lua` stubs `LaunchSubScript` → everything transparently uses the single-core path; no platform checks anywhere (pure feature detection)
+
+### Verified
+- `spec/System/TestParallelTasks_spec.lua`: 6/6 (all four features numerically identical through the worker entry points + JSON round trip)
+- Full existing spec suite: zero regressions vs pristine master baseline (same harness, failure sets identical)
+- Cross-process fresh-VM E2E: 781/781 nodes bit-identical on a 3.13 test build; pass on a live 3.28 build with a Brutal Restraint timeless jewel
+- Real engine: worker baseline probe exact match on all stats; heat map on a level-95 build computes 4,348 candidates in ~15 s (8 workers) vs ~48 s single-core
+- Engine facts established via probe: subscript VMs receive funcList injections (incl. `GetUserPath`) but not `NewFileSearch`/`Inflate`/`Deflate`; worker-thread cwd is launch-dependent (both handled in `CalcWorker.lua`)
+
+### Performance (measured, 5950X SMT-off, level-95 build, full-depth heat map)
+- Single-core: ~48 s (one core busy)
+- Multi-core, 8 workers (auto default): ~15–27 s depending on system load
+- Fixed overhead per job: ~5–7 s worker bootstrap (one-shot VMs must load all game data + build) + ~1–2 s snapshot/merge — this is the Amdahl floor; raise `worker threads` to physical cores − 1 to approach it
+
+---
+
+## Future speed projects (ordered by expected impact)
+
+1. **Persistent warm workers (engine change — removes the ~7 s floor).** `LaunchSubScript` VMs are one-shot; every job pays a full bootstrap. A persistent worker/channel primitive in SimpleGraphic (send job → receive result → VM stays alive) would drop heat-map latency to pure compute (~3–6 s at 15 workers). Largest win, requires C++ runtime work upstream.
+2. **Result caching across tree toggles.** Allocating then un-allocating a node currently recomputes everything twice. Caching the last few power-result sets keyed by a spec/state hash makes "undo a change" instant. Cheap, and directly targets the most common interaction.
+3. **Trim worker bootstrap.** Profile `timing.initMs` vs `bootstrapMs` (already reported by workers): skip unique/rare item DB loading outside itemdb mode, skip tree versions the build doesn't use, lazy-load legion/tattoo data unless present, and cache Data modules as precompiled bytecode (`string.dump`). Realistic target: 7 s → 3–4 s.
+4. **Wave scheduling for stragglers.** The job ends when the slowest worker finishes; per-candidate cost varies (masteries, long paths). Splitting into ~2× more batches than workers and launching a second wave as results return evens out the tail without engine changes.
+5. **Stat-scoped calculation (benefits single-core too).** Each candidate runs a full offence+defence pipeline even when the heat map only needs one stat (e.g. Life). A calculation mode that skips unneeded sections could cut per-candidate cost several-fold — invasive, touches `calcs.perform`.
+6. **Parallelize the Timeless Jewel search.** "Find Timeless Jewel" brute-forces thousands of seeds through the same kind of loop and is a notorious multi-minute wait — it fits the existing ParallelRunner/batch pattern well and may be the single most user-visible follow-up.
+7. **Progressive merge.** Color the tree per-worker as payloads arrive instead of waiting for the last one — no wall-time change, large perceived-latency change.
+8. **Auto worker count tuning.** Auto currently clamps at 8 for memory safety; scale the cap with detected physical cores and available RAM instead (e.g. `min(cores − 1, RAM_GB / 1.5, 16)`).

docs/rundown.md

@@ -135,6 +135,8 @@
     Patch notes written in plain text.
 * **CONTRIBUTING.md**
     Contribution guides.
+* **CalcWorker.lua**
+    Background calculation worker script, executed in isolated Lua VMs on worker threads via `LaunchSubScript` (see `Modules/ParallelRunner.lua`). Bootstraps a headless copy of the program (like `HeadlessWrapper.lua`), reconstructs the build from an XML snapshot, computes a batch of candidates via `Modules/PowerCalcTasks.lua`, and returns the results as JSON.
 * **GameVersions.lua**
     Contains global variables to identify and convert outdated builds. Also contains global table of passive skill tree versions used to upgrade to newer skill tree versions.
 * **HeadlessWrapper.lua**

spec/System/TestParallelTasks_spec.lua

@@ -0,0 +1,228 @@
+-- Tests for the multi-core calculation support (Modules/ParallelRunner,
+-- Modules/PowerCalcTasks, CalcWorker.lua).
+--
+-- Real worker threads can't run headless (LaunchSubScript is stubbed), so
+-- these tests run the worker entry points (computeBatch) in-process on the
+-- same build, round-trip the payloads through JSON exactly as the thread
+-- boundary would, merge them, and require results identical to the
+-- single-core code path.
+local dkjson = require "dkjson"
+
+local function jsonRoundTrip(payload)
+	return dkjson.decode(dkjson.encode(payload))
+end
+
+local function numEq(a, b)
+	if a == nil and b == nil then
+		return true
+	end
+	if type(a) ~= "number" or type(b) ~= "number" then
+		return a == b
+	end
+	if a ~= a and b ~= b then
+		return true -- both NaN
+	end
+	return math.abs(a - b) <= math.max(1e-9, 1e-9 * math.max(math.abs(a), math.abs(b)))
+end
+
+describe("TestParallelTasks", function()
+	before_each(function()
+		newBuild()
+	end)
+
+	it("number sentinels survive the JSON boundary", function()
+		local enc = PowerCalcTasks.encodeNumber
+		local dec = PowerCalcTasks.decodeNumber
+		assert.are.equals(5.25, dec(jsonRoundTrip({ v = enc(5.25) }).v))
+		assert.are.equals(math.huge, dec(jsonRoundTrip({ v = enc(math.huge) }).v))
+		assert.are.equals(-math.huge, dec(jsonRoundTrip({ v = enc(-math.huge) }).v))
+		assert.are.equals(0, dec(jsonRoundTrip({ v = enc(0/0) }).v)) -- NaN guarded to 0
+		assert.are.equals(nil, dec(nil))
+	end)
+
+	it("nodePower worker batches merge to the same result as single-core PowerBuilder", function()
+		local calcsTab = build.calcsTab
+		calcsTab.powerStat = data.powerStatList[4] -- Combined DPS
+		calcsTab.nodePowerMaxDepth = 4
+
+		-- Single-core reference (direct call; outside a coroutine it never yields)
+		calcsTab:PowerBuilder()
+		local refPower = { }
+		for nodeId, node in pairs(build.spec.nodes) do
+			refPower[nodeId] = {
+				singleStat = node.power and node.power.singleStat,
+				pathPower = node.power and node.power.pathPower,
+			}
+		end
+		local refPowerMax = copyTable(calcsTab.powerMax)
+
+		-- Parallel simulation: enumerate, batch exactly as production does
+		-- (including authoritative path data), compute, JSON round trip, merge
+		local cand = calcsTab:EnumeratePowerCandidates()
+		assert.True(cand.total > 0)
+		local batches = calcsTab:BuildPowerBatches(cand, 3)
+		local common = { powerStatIndex = 4, nodePowerMaxDepth = 4 }
+		local payloads = { }
+		for i, batch in ipairs(batches) do
+			payloads[i] = jsonRoundTrip(PowerCalcTasks.nodePower.computeBatch(build, common, batch, "", nil))
+			-- Worker payloads carry a baseline for divergence detection
+			assert.is_table(payloads[i].baseline)
+		end
+		calcsTab:MergeParallelPower(payloads, build.outputRevision)
+
+		for nodeId, ref in pairs(refPower) do
+			local node = build.spec.nodes[nodeId]
+			assert.True(numEq(ref.singleStat, node.power and node.power.singleStat),
+				"singleStat mismatch on node "..tostring(nodeId))
+			assert.True(numEq(ref.pathPower, node.power and node.power.pathPower),
+				"pathPower mismatch on node "..tostring(nodeId))
+		end
+		for key, value in pairs(refPowerMax) do
+			assert.True(numEq(value, calcsTab.powerMax[key]), "powerMax mismatch on "..key)
+		end
+	end)
+
+	it("nodePower merge discards results from a stale build revision", function()
+		local calcsTab = build.calcsTab
+		calcsTab.powerBuildFlag = false
+		calcsTab:MergeParallelPower({ }, build.outputRevision - 1)
+		assert.True(calcsTab.powerBuildFlag)
+	end)
+
+	it("notable measureOne matches the worker batch path", function()
+		local itemsTab = build.itemsTab
+		itemsTab.displayItem = new("Item", "Rarity: RARE\nTest Amulet\nJade Amulet\n+30 to Dexterity")
+		itemsTab.anointEnchantSlot = 1
+
+		-- Collect a handful of anointable notables
+		local nodeIds = { }
+		for nodeId, node in pairs(build.spec.tree.nodes) do
+			if node.recipe and #node.recipe >= 1 and node.modKey ~= "" then
+				table.insert(nodeIds, nodeId)
+				if #nodeIds >= 15 then
+					break
+				end
+			end
+		end
+		assert.True(#nodeIds > 0)
+
+		-- Single-core reference via the shared measureOne
+		local sortDetail = PowerCalcTasks.findSortDetail("CombinedDPS")
+		local calcFunc = build.calcsTab:GetMiscCalculator()
+		local itemType = itemsTab.displayItem.base.type
+		local calcBase = calcFunc({ repSlotName = itemType, repItem = itemsTab:anointItem(nil) })
+		local ref = { }
+		for _, nodeId in ipairs(nodeIds) do
+			ref[nodeId] = PowerCalcTasks.notable.measureOne(itemsTab, calcFunc, calcBase, sortDetail, itemType, build.spec.tree.nodes[nodeId])
+		end
+
+		-- Worker path
+		local payload = jsonRoundTrip(PowerCalcTasks.notable.computeBatch(build,
+			{ sortMode = "CombinedDPS", anointEnchantSlot = 1 },
+			{ nodeIds = nodeIds },
+			itemsTab.displayItem:BuildRaw(), nil))
+		for _, nodeId in ipairs(nodeIds) do
+			local workerPower = PowerCalcTasks.decodeNumber(payload.results[tostring(nodeId)])
+			assert.True(numEq(ref[nodeId], workerPower), "measuredPower mismatch on node "..tostring(nodeId))
+		end
+	end)
+
+	it("itemdb measureOne matches the worker batch path", function()
+		-- Item DBs load over frames in the background
+		local guard = 0
+		while main.onFrameFuncs["LoadItems"] and guard < 1000 do
+			runCallback("OnFrame")
+			guard = guard + 1
+		end
+		local names = { }
+		for name in pairs(main.uniqueDB.list) do
+			table.insert(names, name)
+			if #names >= 10 then
+				break
+			end
+		end
+		assert.True(#names > 0)
+
+		local itemsTab = build.itemsTab
+		local sortDetail = PowerCalcTasks.findSortDetail("Life")
+		local calcFunc = build.calcsTab:GetMiscCalculator()
+		local ref = { }
+		for _, name in ipairs(names) do
+			ref[name] = PowerCalcTasks.itemdb.measureOne(itemsTab, calcFunc, sortDetail, sortDetail.sortMode, false, main.uniqueDB.list[name])
+		end
+
+		local payload = jsonRoundTrip(PowerCalcTasks.itemdb.computeBatch(build,
+			{ sortMode = "Life", dbType = "UNIQUE" },
+			{ names = names }, "", nil))
+		for _, name in ipairs(names) do
+			local workerPower = PowerCalcTasks.decodeNumber(payload.results[name])
+			assert.True(numEq(ref[name], workerPower), "measuredPower mismatch on item "..name)
+		end
+	end)
+
+	it("compare descriptors compute identically through the worker entry point", function()
+		local primaryFile = io.open("../spec/TestBuilds/3.13/OccVortex.xml", "r")
+		local compareFile = io.open("../spec/TestBuilds/3.13/Mirage Archer Toxic Rain.xml", "r")
+		assert.is_not_nil(primaryFile)
+		assert.is_not_nil(compareFile)
+		local primaryXml = primaryFile:read("*a")
+		primaryFile:close()
+		local compareXml = compareFile:read("*a")
+		compareFile:close()
+
+		loadBuildFromXML(primaryXml, "Primary")
+		for i = 1, 5 do
+			runCallback("OnFrame")
+		end
+		local compareTab = build.compareTab
+		local compareEntry = new("CompareEntry", compareXml, "Comparison")
+		compareEntry.xmlText = compareXml
+		local powerStat = data.powerStatList[4] -- Combined DPS
+		local categories = { treeNodes = true, items = true, skillGems = true, supportGems = true, config = true }
+
+		-- Single-core reference (run the real coroutine to completion)
+		local co = coroutine.create(function()
+			compareTab:ComparePowerBuilder(compareEntry, powerStat, categories)
+		end)
+		repeat
+			local res, errMsg = coroutine.resume(co)
+			assert.True(res, errMsg)
+		until coroutine.status(co) == "dead"
+		local refResults = compareTab.comparePowerResults
+		assert.True(#refResults > 0)
+
+		-- Parallel simulation
+		local descriptors = compareTab:EnumerateComparePowerCandidates(compareEntry, categories)
+		for i, desc in ipairs(descriptors) do
+			desc.i = i
+		end
+		local calcFunc, calcBase = compareTab.calcs.getMiscCalculator(build)
+		local fmtCtx = compareTab:MakeComparePowerFormatContext(compareEntry, powerStat, calcBase)
+		local resultByIndex = { }
+		for _, slice in ipairs(ParallelRunner:PartitionList(descriptors, 3)) do
+			local payload = jsonRoundTrip(PowerCalcTasks.compare.computeBatch(build,
+				{ powerStatIndex = 4 }, { descriptors = slice }, compareXml, nil))
+			for _, res in ipairs(payload.results) do
+				resultByIndex[res.i] = res
+			end
+		end
+		local parResults = { }
+		for i, desc in ipairs(descriptors) do
+			local res = resultByIndex[i]
+			if res and res.ok and res.impact then
+				local row = compareTab:BuildComparePowerRow(desc, PowerCalcTasks.decodeNumber(res.impact), res.extra or { }, fmtCtx)
+				if row then
+					table.insert(parResults, row)
+				end
+			end
+		end
+
+		assert.are.equals(#refResults, #parResults)
+		for i = 1, #refResults do
+			assert.are.equals(refResults[i].category, parResults[i].category)
+			assert.are.equals(refResults[i].name, parResults[i].name)
+			assert.True(numEq(refResults[i].impact, parResults[i].impact), "impact mismatch on row "..i)
+			assert.are.equals(refResults[i].combinedImpactStr, parResults[i].combinedImpactStr)
+		end
+	end)
+end)