diff --git a/docs/multicore.md b/docs/multicore.md new file mode 100644 index 00000000000..eb76ff3ecd3 --- /dev/null +++ b/docs/multicore.md @@ -0,0 +1,63 @@ +# Multi-core calculation support — dev handoff notes + +## Changelog (PR summary) + +**Adds multi-core support for the four "iterate candidates, rank outcomes" features, with a user toggle (default: multi-core) and automatic single-core fallback.** + +Parallelized features: +- Passive tree node power / heat map / "Show Power Report" (`CalcsTab:PowerBuilder`) +- Compare tab power report (`CompareTab`) +- Anoint notable stat-sorting (`NotableDBControl`) +- Item DB stat-sorting (`ItemDBControl`) + +### Architecture + +Workers are one-shot isolated Lua VMs started via the engine's existing `LaunchSubScript` primitive (no engine changes). Each worker bootstraps a headless PoB (HeadlessWrapper pattern), reconstructs the build from an in-memory `SaveDB` XML snapshot, computes its batch using **the same per-candidate functions the single-core path runs**, and returns one JSON payload (dkjson). The main thread partitions candidates, aggregates progress via subList calls, validates results, and merges by stable IDs. + +### New files +- `src/Modules/ParallelRunner.lua` — worker pool / job manager: launch, cancel, progress aggregation, baseline validation, per-session sticky fallback, dev-mode job log (`src/parallel_log.txt`) +- `src/CalcWorker.lua` — worker script (read via `io.open`, passed as scriptText like `UpdateCheck.lua`): hardened headless stubs, script-path-relative `io.open` resolution (worker threads don't inherit a guaranteed cwd), env probe mode +- `src/Modules/PowerCalcTasks.lua` — shared per-candidate compute logic + JSON number sentinels + baseline extraction, loaded by both main thread and workers +- `spec/System/TestParallelTasks_spec.lua` — equivalence tests: single-core vs worker entry points in-process, through a JSON round trip, for all four features; merge staleness tests + +### Modified files +- `src/Launch.lua` — `OnSubError` now notifies CUSTOM subscript callbacks (`(nil, errMsg)` — audited compatible with all existing users); nil-guards in `OnSubError`/`OnSubFinished` for aborted subscripts; dev-mode F7 = worker environment probe (report written to `src/probe_report.txt`) +- `src/Classes/CalcsTab.lua` — `PowerBuilder(filter)` optional candidate filter (nil = byte-identical legacy behavior); `EnumeratePowerCandidates`/`BuildPowerBatches` (modKey-grouped, weight-balanced batches carrying authoritative `node.path`/`node.depends` id lists); dual-path `BuildPower` with duplicate-launch guard; `LaunchParallelPowerBuild`/`MergeParallelPower` (staleness via `outputRevision`) +- `src/Classes/CompareTab.lua` — `ComparePowerBuilder` split into enumerate / `ComputeComparePowerCandidate` / format-row so both paths share candidate computation; dual-path report runner +- `src/Classes/NotableDBControl.lua`, `src/Classes/ItemDBControl.lua` — `ListBuilder` split into filter / measure / finish; dual-path `Draw`; candidates keyed by node id / item name +- `src/Modules/Main.lua` — `computationMode` ("MULTI"/"SINGLE", default MULTI) and `workerCount` (0 = auto) settings, persisted in `Settings.xml`; Options popup UI; loads `ParallelRunner`/`PowerCalcTasks` as globals +- `src/Modules/Build.lua` — cancels outstanding worker jobs on shutdown +- `src/Modules/DataLegionLookUpTableHelper.lua` — **standalone bugfix**: `loadJewelFile` could cache empty jewel data to `.bin` when `Inflate` is stubbed (headless/busted), silently corrupting timeless jewel stats for the full app afterwards. Now rejects empty data, reads an existing `.bin` directly when `NewFileSearch` is unavailable, and never writes the shared cache from headless contexts + +### Correctness guarantees +- Workers and single-core run identical code; results cross the thread boundary as JSON with `inf`/NaN sentinels +- Every worker returns a baseline stat vector recomputed from its reconstructed build; any relative deviation > 1e-3 from the main thread's baseline fails the whole job → automatic single-core fallback (sticky for the session, cleared by toggling the option or a successful F7 probe) +- Main thread sends each candidate's live `path`/`depends` node ids: a spec rebuilt from XML resolves equal-length shortest-path ties in different `pairs()` order, which would otherwise change `pathPower` vs what the UI displays +- Results merge only if `build.outputRevision` is unchanged since the snapshot; any build change mid-job cancels/discards and re-runs +- Worker errors surface once via `OnSubError` → fallback; workers never write shared caches +- Headless/CI: `HeadlessWrapper.lua` stubs `LaunchSubScript` → everything transparently uses the single-core path; no platform checks anywhere (pure feature detection) + +### Verified +- `spec/System/TestParallelTasks_spec.lua`: 6/6 (all four features numerically identical through the worker entry points + JSON round trip) +- Full existing spec suite: zero regressions vs pristine master baseline (same harness, failure sets identical) +- Cross-process fresh-VM E2E: 781/781 nodes bit-identical on a 3.13 test build; pass on a live 3.28 build with a Brutal Restraint timeless jewel +- Real engine: worker baseline probe exact match on all stats; heat map on a level-95 build computes 4,348 candidates in ~15 s (8 workers) vs ~48 s single-core +- Engine facts established via probe: subscript VMs receive funcList injections (incl. `GetUserPath`) but not `NewFileSearch`/`Inflate`/`Deflate`; worker-thread cwd is launch-dependent (both handled in `CalcWorker.lua`) + +### Performance (measured, 5950X SMT-off, level-95 build, full-depth heat map) +- Single-core: ~48 s (one core busy) +- Multi-core, 8 workers (auto default): ~15–27 s depending on system load +- Fixed overhead per job: ~5–7 s worker bootstrap (one-shot VMs must load all game data + build) + ~1–2 s snapshot/merge — this is the Amdahl floor; raise `worker threads` to physical cores − 1 to approach it + +--- + +## Future speed projects (ordered by expected impact) + +1. **Persistent warm workers (engine change — removes the ~7 s floor).** `LaunchSubScript` VMs are one-shot; every job pays a full bootstrap. A persistent worker/channel primitive in SimpleGraphic (send job → receive result → VM stays alive) would drop heat-map latency to pure compute (~3–6 s at 15 workers). Largest win, requires C++ runtime work upstream. +2. **Result caching across tree toggles.** Allocating then un-allocating a node currently recomputes everything twice. Caching the last few power-result sets keyed by a spec/state hash makes "undo a change" instant. Cheap, and directly targets the most common interaction. +3. **Trim worker bootstrap.** Profile `timing.initMs` vs `bootstrapMs` (already reported by workers): skip unique/rare item DB loading outside itemdb mode, skip tree versions the build doesn't use, lazy-load legion/tattoo data unless present, and cache Data modules as precompiled bytecode (`string.dump`). Realistic target: 7 s → 3–4 s. +4. **Wave scheduling for stragglers.** The job ends when the slowest worker finishes; per-candidate cost varies (masteries, long paths). Splitting into ~2× more batches than workers and launching a second wave as results return evens out the tail without engine changes. +5. **Stat-scoped calculation (benefits single-core too).** Each candidate runs a full offence+defence pipeline even when the heat map only needs one stat (e.g. Life). A calculation mode that skips unneeded sections could cut per-candidate cost several-fold — invasive, touches `calcs.perform`. +6. **Parallelize the Timeless Jewel search.** "Find Timeless Jewel" brute-forces thousands of seeds through the same kind of loop and is a notorious multi-minute wait — it fits the existing ParallelRunner/batch pattern well and may be the single most user-visible follow-up. +7. **Progressive merge.** Color the tree per-worker as payloads arrive instead of waiting for the last one — no wall-time change, large perceived-latency change. +8. **Auto worker count tuning.** Auto currently clamps at 8 for memory safety; scale the cap with detected physical cores and available RAM instead (e.g. `min(cores − 1, RAM_GB / 1.5, 16)`). diff --git a/docs/rundown.md b/docs/rundown.md index 03fde7149dd..5cb93603bad 100644 --- a/docs/rundown.md +++ b/docs/rundown.md @@ -135,6 +135,8 @@ Patch notes written in plain text. * **CONTRIBUTING.md** Contribution guides. +* **CalcWorker.lua** + Background calculation worker script, executed in isolated Lua VMs on worker threads via `LaunchSubScript` (see `Modules/ParallelRunner.lua`). Bootstraps a headless copy of the program (like `HeadlessWrapper.lua`), reconstructs the build from an XML snapshot, computes a batch of candidates via `Modules/PowerCalcTasks.lua`, and returns the results as JSON. * **GameVersions.lua** Contains global variables to identify and convert outdated builds. Also contains global table of passive skill tree versions used to upgrade to newer skill tree versions. * **HeadlessWrapper.lua** diff --git a/runtime/Path of Building.exe b/runtime/Path of Building.exe new file mode 100644 index 00000000000..5e7ecf7b732 Binary files /dev/null and b/runtime/Path of Building.exe differ diff --git a/spec/System/TestParallelTasks_spec.lua b/spec/System/TestParallelTasks_spec.lua new file mode 100644 index 00000000000..a3378198973 --- /dev/null +++ b/spec/System/TestParallelTasks_spec.lua @@ -0,0 +1,228 @@ +-- Tests for the multi-core calculation support (Modules/ParallelRunner, +-- Modules/PowerCalcTasks, CalcWorker.lua). +-- +-- Real worker threads can't run headless (LaunchSubScript is stubbed), so +-- these tests run the worker entry points (computeBatch) in-process on the +-- same build, round-trip the payloads through JSON exactly as the thread +-- boundary would, merge them, and require results identical to the +-- single-core code path. +local dkjson = require "dkjson" + +local function jsonRoundTrip(payload) + return dkjson.decode(dkjson.encode(payload)) +end + +local function numEq(a, b) + if a == nil and b == nil then + return true + end + if type(a) ~= "number" or type(b) ~= "number" then + return a == b + end + if a ~= a and b ~= b then + return true -- both NaN + end + return math.abs(a - b) <= math.max(1e-9, 1e-9 * math.max(math.abs(a), math.abs(b))) +end + +describe("TestParallelTasks", function() + before_each(function() + newBuild() + end) + + it("number sentinels survive the JSON boundary", function() + local enc = PowerCalcTasks.encodeNumber + local dec = PowerCalcTasks.decodeNumber + assert.are.equals(5.25, dec(jsonRoundTrip({ v = enc(5.25) }).v)) + assert.are.equals(math.huge, dec(jsonRoundTrip({ v = enc(math.huge) }).v)) + assert.are.equals(-math.huge, dec(jsonRoundTrip({ v = enc(-math.huge) }).v)) + assert.are.equals(0, dec(jsonRoundTrip({ v = enc(0/0) }).v)) -- NaN guarded to 0 + assert.are.equals(nil, dec(nil)) + end) + + it("nodePower worker batches merge to the same result as single-core PowerBuilder", function() + local calcsTab = build.calcsTab + calcsTab.powerStat = data.powerStatList[4] -- Combined DPS + calcsTab.nodePowerMaxDepth = 4 + + -- Single-core reference (direct call; outside a coroutine it never yields) + calcsTab:PowerBuilder() + local refPower = { } + for nodeId, node in pairs(build.spec.nodes) do + refPower[nodeId] = { + singleStat = node.power and node.power.singleStat, + pathPower = node.power and node.power.pathPower, + } + end + local refPowerMax = copyTable(calcsTab.powerMax) + + -- Parallel simulation: enumerate, batch exactly as production does + -- (including authoritative path data), compute, JSON round trip, merge + local cand = calcsTab:EnumeratePowerCandidates() + assert.True(cand.total > 0) + local batches = calcsTab:BuildPowerBatches(cand, 3) + local common = { powerStatIndex = 4, nodePowerMaxDepth = 4 } + local payloads = { } + for i, batch in ipairs(batches) do + payloads[i] = jsonRoundTrip(PowerCalcTasks.nodePower.computeBatch(build, common, batch, "", nil)) + -- Worker payloads carry a baseline for divergence detection + assert.is_table(payloads[i].baseline) + end + calcsTab:MergeParallelPower(payloads, build.outputRevision) + + for nodeId, ref in pairs(refPower) do + local node = build.spec.nodes[nodeId] + assert.True(numEq(ref.singleStat, node.power and node.power.singleStat), + "singleStat mismatch on node "..tostring(nodeId)) + assert.True(numEq(ref.pathPower, node.power and node.power.pathPower), + "pathPower mismatch on node "..tostring(nodeId)) + end + for key, value in pairs(refPowerMax) do + assert.True(numEq(value, calcsTab.powerMax[key]), "powerMax mismatch on "..key) + end + end) + + it("nodePower merge discards results from a stale build revision", function() + local calcsTab = build.calcsTab + calcsTab.powerBuildFlag = false + calcsTab:MergeParallelPower({ }, build.outputRevision - 1) + assert.True(calcsTab.powerBuildFlag) + end) + + it("notable measureOne matches the worker batch path", function() + local itemsTab = build.itemsTab + itemsTab.displayItem = new("Item", "Rarity: RARE\nTest Amulet\nJade Amulet\n+30 to Dexterity") + itemsTab.anointEnchantSlot = 1 + + -- Collect a handful of anointable notables + local nodeIds = { } + for nodeId, node in pairs(build.spec.tree.nodes) do + if node.recipe and #node.recipe >= 1 and node.modKey ~= "" then + table.insert(nodeIds, nodeId) + if #nodeIds >= 15 then + break + end + end + end + assert.True(#nodeIds > 0) + + -- Single-core reference via the shared measureOne + local sortDetail = PowerCalcTasks.findSortDetail("CombinedDPS") + local calcFunc = build.calcsTab:GetMiscCalculator() + local itemType = itemsTab.displayItem.base.type + local calcBase = calcFunc({ repSlotName = itemType, repItem = itemsTab:anointItem(nil) }) + local ref = { } + for _, nodeId in ipairs(nodeIds) do + ref[nodeId] = PowerCalcTasks.notable.measureOne(itemsTab, calcFunc, calcBase, sortDetail, itemType, build.spec.tree.nodes[nodeId]) + end + + -- Worker path + local payload = jsonRoundTrip(PowerCalcTasks.notable.computeBatch(build, + { sortMode = "CombinedDPS", anointEnchantSlot = 1 }, + { nodeIds = nodeIds }, + itemsTab.displayItem:BuildRaw(), nil)) + for _, nodeId in ipairs(nodeIds) do + local workerPower = PowerCalcTasks.decodeNumber(payload.results[tostring(nodeId)]) + assert.True(numEq(ref[nodeId], workerPower), "measuredPower mismatch on node "..tostring(nodeId)) + end + end) + + it("itemdb measureOne matches the worker batch path", function() + -- Item DBs load over frames in the background + local guard = 0 + while main.onFrameFuncs["LoadItems"] and guard < 1000 do + runCallback("OnFrame") + guard = guard + 1 + end + local names = { } + for name in pairs(main.uniqueDB.list) do + table.insert(names, name) + if #names >= 10 then + break + end + end + assert.True(#names > 0) + + local itemsTab = build.itemsTab + local sortDetail = PowerCalcTasks.findSortDetail("Life") + local calcFunc = build.calcsTab:GetMiscCalculator() + local ref = { } + for _, name in ipairs(names) do + ref[name] = PowerCalcTasks.itemdb.measureOne(itemsTab, calcFunc, sortDetail, sortDetail.sortMode, false, main.uniqueDB.list[name]) + end + + local payload = jsonRoundTrip(PowerCalcTasks.itemdb.computeBatch(build, + { sortMode = "Life", dbType = "UNIQUE" }, + { names = names }, "", nil)) + for _, name in ipairs(names) do + local workerPower = PowerCalcTasks.decodeNumber(payload.results[name]) + assert.True(numEq(ref[name], workerPower), "measuredPower mismatch on item "..name) + end + end) + + it("compare descriptors compute identically through the worker entry point", function() + local primaryFile = io.open("../spec/TestBuilds/3.13/OccVortex.xml", "r") + local compareFile = io.open("../spec/TestBuilds/3.13/Mirage Archer Toxic Rain.xml", "r") + assert.is_not_nil(primaryFile) + assert.is_not_nil(compareFile) + local primaryXml = primaryFile:read("*a") + primaryFile:close() + local compareXml = compareFile:read("*a") + compareFile:close() + + loadBuildFromXML(primaryXml, "Primary") + for i = 1, 5 do + runCallback("OnFrame") + end + local compareTab = build.compareTab + local compareEntry = new("CompareEntry", compareXml, "Comparison") + compareEntry.xmlText = compareXml + local powerStat = data.powerStatList[4] -- Combined DPS + local categories = { treeNodes = true, items = true, skillGems = true, supportGems = true, config = true } + + -- Single-core reference (run the real coroutine to completion) + local co = coroutine.create(function() + compareTab:ComparePowerBuilder(compareEntry, powerStat, categories) + end) + repeat + local res, errMsg = coroutine.resume(co) + assert.True(res, errMsg) + until coroutine.status(co) == "dead" + local refResults = compareTab.comparePowerResults + assert.True(#refResults > 0) + + -- Parallel simulation + local descriptors = compareTab:EnumerateComparePowerCandidates(compareEntry, categories) + for i, desc in ipairs(descriptors) do + desc.i = i + end + local calcFunc, calcBase = compareTab.calcs.getMiscCalculator(build) + local fmtCtx = compareTab:MakeComparePowerFormatContext(compareEntry, powerStat, calcBase) + local resultByIndex = { } + for _, slice in ipairs(ParallelRunner:PartitionList(descriptors, 3)) do + local payload = jsonRoundTrip(PowerCalcTasks.compare.computeBatch(build, + { powerStatIndex = 4 }, { descriptors = slice }, compareXml, nil)) + for _, res in ipairs(payload.results) do + resultByIndex[res.i] = res + end + end + local parResults = { } + for i, desc in ipairs(descriptors) do + local res = resultByIndex[i] + if res and res.ok and res.impact then + local row = compareTab:BuildComparePowerRow(desc, PowerCalcTasks.decodeNumber(res.impact), res.extra or { }, fmtCtx) + if row then + table.insert(parResults, row) + end + end + end + + assert.are.equals(#refResults, #parResults) + for i = 1, #refResults do + assert.are.equals(refResults[i].category, parResults[i].category) + assert.are.equals(refResults[i].name, parResults[i].name) + assert.True(numEq(refResults[i].impact, parResults[i].impact), "impact mismatch on row "..i) + assert.are.equals(refResults[i].combinedImpactStr, parResults[i].combinedImpactStr) + end + end) +end) diff --git a/src/CalcWorker.lua b/src/CalcWorker.lua new file mode 100644 index 00000000000..a11772e19b3 --- /dev/null +++ b/src/CalcWorker.lua @@ -0,0 +1,429 @@ +#@ +-- Path of Building +-- +-- Module: Calc Worker +-- Background calculation worker, launched via LaunchSubScript by Modules/ParallelRunner. +-- Runs in an isolated Lua VM on its own thread: bootstraps a headless copy of the +-- program (modelled on HeadlessWrapper.lua), reconstructs the build from an XML +-- snapshot, computes the requested batch of candidates, and returns a JSON string. +-- +-- Available in this VM: the Lua standard library, require() for bundled binary/lua +-- modules, plus engine functions injected via funcList (GetScriptPath etc.) and +-- main-thread callbacks via subList (ConPrintf, ParallelWorkerProgress). + +local jobId, workerIdx, scriptPath, workerMode, buildXML, auxText, specJSON = ... + +-- Keep any engine-injected functions before defining stubs +local engineScriptPath = GetScriptPath +local engineRuntimePath = GetRuntimePath +local engineUserPath = GetUserPath +local engineWorkDir = GetWorkDir +local engineConPrintf = ConPrintf +local engineNewFileSearch = NewFileSearch +local engineInflate = Inflate +local engineDeflate = Deflate + +-- Millisecond timer; on Windows os.clock() is wall time, which is all we need +-- for timing telemetry and the calc engine's internal pacing checks +local osClock = os.clock +local function nowMs() + return osClock() * 1000 +end + +-- JSON codec, needed both for the work spec and to report errors; loaded before +-- anything else so failures later on can still be serialized +local dkjson +do + local ok, mod = pcall(require, "dkjson") + if ok and mod then + dkjson = mod + else + local candidates = { } + if engineRuntimePath then + candidates[#candidates + 1] = engineRuntimePath().."/lua/dkjson.lua" + end + candidates[#candidates + 1] = scriptPath.."/../runtime/lua/dkjson.lua" + for _, path in ipairs(candidates) do + local func = loadfile(path) + if func then + dkjson = func() + break + end + end + end + if not dkjson then + error("CalcWorker: unable to load dkjson") + end +end + +------------------------------------------------------------------------------ +-- Headless environment stubs (hardened copy of HeadlessWrapper.lua) +------------------------------------------------------------------------------ + +-- Callbacks +local callbackTable = { } +local mainObject +function runCallback(name, ...) + if callbackTable[name] then + return callbackTable[name](...) + elseif mainObject and mainObject[name] then + return mainObject[name](mainObject, ...) + end +end +function SetCallback(name, func) + callbackTable[name] = func +end +function GetCallback(name) + return callbackTable[name] +end +function SetMainObject(obj) + mainObject = obj +end + +-- Image Handles +local imageHandleClass = { } +imageHandleClass.__index = imageHandleClass +function NewImageHandle() + return setmetatable({ }, imageHandleClass) +end +function imageHandleClass:Load(fileName, ...) + self.valid = true +end +function imageHandleClass:Unload() + self.valid = false +end +function imageHandleClass:IsValid() + return self.valid +end +function imageHandleClass:SetLoadingPriority(pri) end +function imageHandleClass:ImageSize() + return 1, 1 +end + +-- Rendering +function RenderInit(flag, ...) end +function GetScreenSize() + return 1920, 1080 +end +function GetScreenScale() + return 1 +end +function GetVirtualScreenSize() + return GetScreenSize() +end +function GetDPIScaleOverridePercent() + return 1 +end +function SetDPIScaleOverridePercent(scale) end +function SetClearColor(r, g, b, a) end +function SetDrawLayer(layer, subLayer) end +function SetViewport(x, y, width, height) end +function SetDrawColor(r, g, b, a) end +function DrawImage(imgHandle, left, top, width, height, tcLeft, tcTop, tcRight, tcBottom) end +function DrawImageQuad(imageHandle, x1, y1, x2, y2, x3, y3, x4, y4, s1, t1, s2, t2, s3, t3, s4, t4) end +function DrawString(left, top, align, height, font, text) end +function DrawStringWidth(height, font, text) + return 1 +end +function DrawStringCursorIndex(height, font, text, cursorX, cursorY) + return 0 +end +function StripEscapes(text) + return text:gsub("%^%d",""):gsub("%^x%x%x%x%x%x%x","") +end +function GetAsyncCount() + return 0 +end + +-- Search Handles; prefer the engine implementation if this VM has one. +-- The stub returning nil makes data loaders (e.g. timeless jewel LUTs) fall +-- back to reading .bin caches directly — they must NOT get fake handles +NewFileSearch = engineNewFileSearch or function() end + +-- General Functions +function SetWindowTitle(title) end +function GetCursorPos() + return 0, 0 +end +function SetCursorPos(x, y) end +function ShowCursor(doShow) end +function IsKeyDown(keyName) end +function Copy(text) end +function Paste() end +-- Prefer the engine implementations if this VM has them. The stubs return nil +-- (NOT "") so callers treat compression as unavailable instead of caching +-- empty data — an empty string here once poisoned the shared timeless jewel +-- .bin cache on disk +Deflate = engineDeflate or function(data) + return nil +end +Inflate = engineInflate or function(data) + return nil +end +function GetTime() + return nowMs() +end +-- Prefer real engine paths where they were injected; fall back to the path the +-- main thread passed as an argument +GetScriptPath = engineScriptPath or function() + return scriptPath +end +GetRuntimePath = engineRuntimePath or function() + return scriptPath.."/../runtime" +end +GetUserPath = engineUserPath or function() + return "" +end +GetWorkDir = engineWorkDir or function() + return "" +end +function GetCloudProvider(fullPath) + return nil, nil, nil +end +function MakeDir(path) end +function RemoveDir(path) end +function SetWorkDir(path) end +-- Workers may not spawn further workers +function LaunchSubScript(scriptText, funcList, subList, ...) end +function AbortSubScript(ssID) end +function IsSubScriptRunning(ssID) end +function LoadModule(fileName, ...) + if not fileName:match("%.lua") then + fileName = fileName .. ".lua" + end + local func, err = loadfile(fileName) + if not func then + -- The worker thread's working directory may not be the script path; + -- retry with an absolute path + func, err = loadfile(GetScriptPath().."/"..fileName) + end + if func then + return func(...) + else + error("LoadModule() error loading '"..fileName.."': "..err) + end +end +function PCall(func, ...) + local ret = { pcall(func, ...) } + if ret[1] then + table.remove(ret, 1) + return nil, unpack(ret) + else + return ret[2] + end +end +function PLoadModule(fileName, ...) + if not fileName:match("%.lua") then + fileName = fileName .. ".lua" + end + local func, err = loadfile(fileName) + if not func then + func, err = loadfile(GetScriptPath().."/"..fileName) + end + if func then + return PCall(func, ...) + else + error("PLoadModule() error loading '"..fileName.."': "..err) + end +end +-- ConPrintf is normally marshalled to the main thread via subList; keep a +-- local fallback in case it wasn't injected +ConPrintf = ConPrintf or function(fmt, ...) + print(string.format(fmt, ...)) +end +function ConPrintTable(tbl, noRecurse) end +function ConExecute(cmd) end +function ConClear() end +function SpawnProcess(cmdName, args) end +function OpenURL(url) end +function SetProfiling(isEnabled) end +function Restart() end +function Exit() end +function TakeScreenshot() end + +-- Launch.lua indexes these at load time +arg = arg or { } +jit = jit or { opt = { start = function() end } } + +local l_require = require +function require(name) + -- Don't hard-fail if lcurl isn't loadable in this VM; nothing that runs in + -- a worker actually performs HTTP requests + if name == "lcurl.safe" then + local ok, mod = pcall(l_require, name) + return ok and mod or nil + end + return l_require(name) +end + +-- Relative paths resolve against the process working directory, which is only +-- guaranteed to be the script path for the main thread. Resolve relative opens +-- against the script path first so file access behaves like the main thread +-- (e.g. "TreeData//tree.lua" in PassiveTree). +local l_ioOpen = io.open +local function isRelativePath(fileName) + return type(fileName) == "string" and not fileName:match("^%a:[/\\]") and not fileName:match("^[/\\]") +end +function io.open(fileName, ...) + if isRelativePath(fileName) then + local file = l_ioOpen(scriptPath.."/"..fileName, ...) + if file then + return file + end + end + return l_ioOpen(fileName, ...) +end + +------------------------------------------------------------------------------ +-- Worker body +------------------------------------------------------------------------------ + +local function canOpen(path) + local file = l_ioOpen(path, "rb") + if file then + file:close() + return true + end + return false +end + +-- Filled in as early as possible so error payloads can carry environment +-- diagnostics even when bootstrap fails +local envProbe + +local function checkForStartupError() + if mainObject and mainObject.promptMsg then + error("worker startup failed: "..tostring(mainObject.promptMsg)) + end +end + +local function runWorker() + local timing = { } + local startTime = nowMs() + envProbe = { + scriptPathArg = scriptPath, + workDir = engineWorkDir and engineWorkDir() or "?", + treeOpenRelative = canOpen("TreeData/3_28/tree.lua"), + treeOpenAbsolute = canOpen(scriptPath.."/TreeData/3_28/tree.lua"), + } + + -- Bootstrap the program + local launchDoFile, dofileErr = loadfile(scriptPath.."/Launch.lua") + if not launchDoFile then + error("unable to load Launch.lua: "..tostring(dofileErr)) + end + launchDoFile() + -- The worker must never kick off an update check + mainObject.CheckForUpdate = function() end + runCallback("OnInit") + checkForStartupError() + timing.initMs = nowMs() - startTime + + local spec = dkjson.decode(specJSON or "") or { } + local common = spec.common or { } + local batch = spec.batch or { } + + -- Reconstruct the build from the XML snapshot. Init has queued a mode change + -- to an empty build; replacing it before the first frame means the worker + -- never wastes time building that placeholder (or loading its tree version) + local build + if buildXML and #buildXML > 0 then + mainObject.main:SetMode("BUILD", false, "CalcWorker", buildXML) + runCallback("OnFrame") + checkForStartupError() + build = mainObject.main.modes["BUILD"] + -- Let deferred frame work settle (initial calc pass, item DB loading for + -- modes that need it); each frame is cheap once the flags have cleared + local framesLeft = 1000 + while framesLeft > 0 do + local needMoreFrames = build.buildFlag or build.modFlag + if workerMode == "itemdb" and mainObject.main.onFrameFuncs["LoadItems"] then + needMoreFrames = true + end + if not needMoreFrames then + break + end + runCallback("OnFrame") + checkForStartupError() + framesLeft = framesLeft - 1 + end + else + runCallback("OnFrame") -- Need at least one frame for everything to initialise + checkForStartupError() + end + timing.bootstrapMs = nowMs() - startTime + + local result + if workerMode == "probe" then + -- Environment probe: report facts about this VM so assumptions about the + -- worker environment can be verified at runtime instead of guessed at + local probe = { + luaVersion = _VERSION, + jitVersion = (rawget(_G, "jit") and jit.version) or "none", + hasLoadfile = type(loadfile) == "function", + hasIo = type(io) == "table", + packagePath = package and package.path or "?", + injectedGetScriptPath = engineScriptPath ~= nil, + injectedGetRuntimePath = engineRuntimePath ~= nil, + injectedGetUserPath = engineUserPath ~= nil, + injectedGetWorkDir = engineWorkDir ~= nil, + injectedConPrintf = engineConPrintf ~= nil, + injectedNewFileSearch = engineNewFileSearch ~= nil, + injectedInflate = engineInflate ~= nil, + buildXMLBytes = buildXML and #buildXML or 0, + specJSONBytes = specJSON and #specJSON or 0, + buildLoaded = build ~= nil, + buildName = build and build.buildName or "none", + } + for key, value in pairs(envProbe or { }) do + probe[key] = value + end + result = { probe = probe } + if build then + local calcStart = nowMs() + local tasks = LoadModule("Modules/PowerCalcTasks") + local calcFunc, calcBase = build.calcsTab:GetMiscCalculator() + result.baseline = tasks.extractBaseline(calcBase, nil, false) + -- Time a couple of representative calcFunc calls + calcFunc({ }, false) + probe.singleCalcMs = nowMs() - calcStart + timing.calcMs = nowMs() - calcStart + end + if ParallelWorkerProgress then + ParallelWorkerProgress(jobId, workerIdx, 1, 1) + end + else + if not build then + error("no build XML provided for mode "..tostring(workerMode)) + end + local tasks = LoadModule("Modules/PowerCalcTasks") + local task = tasks[workerMode] + if not task then + error("unknown worker mode: "..tostring(workerMode)) + end + local calcStart = nowMs() + local progressFunc + if ParallelWorkerProgress then + progressFunc = function(done, total) + ParallelWorkerProgress(jobId, workerIdx, done, total) + end + end + result = task.computeBatch(build, common, batch, auxText, progressFunc) + timing.calcMs = nowMs() - calcStart + end + + timing.totalMs = nowMs() - startTime + result.timing = timing + return dkjson.encode(result) +end + +local ok, resultOrErr = xpcall(runWorker, function(err) + return tostring(err).."\n"..debug.traceback() +end) +if ok then + return resultOrErr +end +-- Return the failure as a structured payload so the main thread gets a clean +-- error message (an uncaught error would also work via OnSubError, but with +-- less context); include the environment diagnostics for probe reports +return dkjson.encode({ error = resultOrErr, probe = envProbe }) diff --git a/src/Classes/CalcsTab.lua b/src/Classes/CalcsTab.lua index 690103ba096..3088f119564 100644 --- a/src/Classes/CalcsTab.lua +++ b/src/Classes/CalcsTab.lua @@ -6,6 +6,7 @@ local pairs = pairs local ipairs = ipairs local t_insert = table.insert +local t_remove = table.remove local m_max = math.max local m_floor = math.floor @@ -450,12 +451,27 @@ function CalcsTabClass:BuildOutput() self.miscCalculator = { self.calcs.getMiscCalculator(self.build) } end --- Controls the coroutine that calculates node power +-- Controls the coroutine (or worker job) that calculates node power function CalcsTabClass:BuildPower() + if self.powerBuildFlag and self.build.buildFlag then + -- A main output rebuild is pending and will set powerBuildFlag again + -- when it completes; starting now would immediately be cancelled and + -- restarted (which is expensive with background workers) + return + end if self.powerBuildFlag then self.powerBuildFlag = false self.powerMax = nil - self.powerBuilder = coroutine.create(self.PowerBuilder) + self.powerBuilder = nil + if self.powerJob then + -- A rebuild was requested while workers were still running; their + -- results are stale, so cancel and start over + self.powerJob:Cancel() + self.powerJob = nil + end + if not (ParallelRunner:IsAvailable() and self:LaunchParallelPowerBuild()) then + self.powerBuilder = coroutine.create(self.PowerBuilder) + end end if self.powerBuilder then local res, errMsg = coroutine.resume(self.powerBuilder, self) @@ -472,7 +488,10 @@ function CalcsTabClass:BuildPower() end -- Estimate the offensive and defensive power of all unallocated nodes -function CalcsTabClass:PowerBuilder() +-- filter is optional (used by parallel workers): { nodes = , +-- cluster = } restricts which candidates are +-- calculated; behaviour with no filter is unchanged +function CalcsTabClass:PowerBuilder(filter) -- local timer_start = GetTime() local useFullDPS = self.powerStat and self.powerStat.stat == "FullDPS" local calcFunc, calcBase = self:GetMiscCalculator() @@ -544,7 +563,7 @@ function CalcsTabClass:PowerBuilder() if node.type == "Mastery" then node.power.masteryEffects = { } end - if node.modKey ~= "" and not self.mainEnv.grantedPassives[nodeId] then + if node.modKey ~= "" and not self.mainEnv.grantedPassives[nodeId] and (not filter or filter.nodes[nodeId]) then if node.type == "Mastery" and node.allMasteryOptions then if not (self.nodePowerMaxDepth and self.nodePowerMaxDepth < node.pathDist) then t_insert(masteryNodeList, node) @@ -569,8 +588,8 @@ function CalcsTabClass:PowerBuilder() distanceMap = nil table.sort(distanceList, function(a, b) return a[1] < b[1] end) -- Count eligible cluster nodes - for _, node in pairs(self.build.spec.tree.clusterNodeMap) do - if not node.alloc and node.modKey ~= "" and not self.mainEnv.grantedPassives[node.id] then + for nodeName, node in pairs(self.build.spec.tree.clusterNodeMap) do + if not node.alloc and node.modKey ~= "" and not self.mainEnv.grantedPassives[node.id] and (not filter or filter.cluster[nodeName]) then total = total + 1 end end @@ -624,6 +643,9 @@ function CalcsTabClass:PowerBuilder() end end nodeIndex = nodeIndex + 1 + if self.workerProgressFunc and nodeIndex % 10 == 0 then + self.workerProgressFunc(nodeIndex, total) + end if coroutine.running() and GetTime() - start > 100 then if self.build.powerBuilderProgressCallback then self.build.powerBuilderProgressCallback(m_floor(nodeIndex/total*100)) @@ -669,6 +691,9 @@ function CalcsTabClass:PowerBuilder() end end nodeIndex = nodeIndex + 1 + if self.workerProgressFunc and nodeIndex % 10 == 0 then + self.workerProgressFunc(nodeIndex, total) + end if coroutine.running() and GetTime() - start > 100 then if self.build.powerBuilderProgressCallback then self.build.powerBuilderProgressCallback(m_floor(nodeIndex/total*100)) @@ -697,6 +722,9 @@ function CalcsTabClass:PowerBuilder() node.power.singleStat = self:CalculatePowerStat(self.powerStat, output, calcBase) end nodeIndex = nodeIndex + 1 + if self.workerProgressFunc and nodeIndex % 10 == 0 then + self.workerProgressFunc(nodeIndex, total) + end if coroutine.running() and GetTime() - start > 100 then if self.build.powerBuilderProgressCallback then self.build.powerBuilderProgressCallback(m_floor(nodeIndex/total*100)) @@ -709,6 +737,269 @@ function CalcsTabClass:PowerBuilder() self.powerMax = newPowerMax self.powerBuilderInitialized = true -- ConPrintf("Power Build time: %d ms", GetTime() - timer_start) + return newPowerMax +end + +-- Enumerate the candidates PowerBuilder would calculate, grouped so that nodes +-- sharing a modKey stay together (they share one cache entry in PowerBuilder). +-- Returns { groups = { { weight, nodeIds } ... }, clusterNames = { ... }, total } +function CalcsTabClass:EnumeratePowerCandidates() + local groupByKey = { } + local groups = { } + local clusterNames = { } + local total = 0 + local function nodeGroup(key) + if not groupByKey[key] then + groupByKey[key] = { weight = 0, nodeIds = { } } + t_insert(groups, groupByKey[key]) + end + return groupByKey[key] + end + for nodeId, node in pairs(self.build.spec.nodes) do + if node.modKey ~= "" and not self.mainEnv.grantedPassives[nodeId] + and not (self.nodePowerMaxDepth and self.nodePowerMaxDepth < (node.pathDist or 1000)) then + if node.type == "Mastery" and node.allMasteryOptions then + -- Mastery effect pseudo-nodes each cost one calculation + local count = 0 + for _, masteryEffect in ipairs(node.masteryEffects or { }) do + local assignedNodeId = isValueInTable(self.build.spec.masterySelections, masteryEffect.effect) + if not assignedNodeId or assignedNodeId == node.id then + count = count + 1 + end + end + if count > 0 then + local group = nodeGroup("mastery:"..nodeId) + group.weight = group.weight + count + t_insert(group.nodeIds, nodeId) + total = total + count + end + else + local group = nodeGroup(node.alloc and (node.modKey.."_remove") or node.modKey) + group.weight = group.weight + 1 + t_insert(group.nodeIds, nodeId) + total = total + 1 + end + end + end + for nodeName, node in pairs(self.build.spec.tree.clusterNodeMap) do + if not node.alloc and node.modKey ~= "" and not self.mainEnv.grantedPassives[node.id] then + t_insert(clusterNames, nodeName) + total = total + 1 + end + end + return { groups = groups, clusterNames = clusterNames, total = total } +end + +-- Distribute enumerated candidates across up to workerCount batches (heaviest +-- modKey groups first, so batches stay balanced), attaching each candidate's +-- live path data +function CalcsTabClass:BuildPowerBatches(cand, workerCount) + local batches = { } + for i = 1, workerCount do + batches[i] = { weight = 0, nodeIds = { }, clusterNames = { } } + end + table.sort(cand.groups, function(a, b) return a.weight > b.weight end) + for _, group in ipairs(cand.groups) do + local lightest = batches[1] + for i = 2, workerCount do + if batches[i].weight < lightest.weight then + lightest = batches[i] + end + end + lightest.weight = lightest.weight + group.weight + for _, nodeId in ipairs(group.nodeIds) do + t_insert(lightest.nodeIds, nodeId) + end + end + for i, name in ipairs(cand.clusterNames) do + t_insert(batches[((i - 1) % workerCount) + 1].clusterNames, name) + end + -- Drop empty batches, strip local weights + for i = #batches, 1, -1 do + if #batches[i].nodeIds == 0 and #batches[i].clusterNames == 0 then + t_remove(batches, i) + else + batches[i].weight = nil + end + end + -- Send each candidate's live path/depends node ids along with the batch. + -- A spec rebuilt from XML resolves equal-length shortest-path ties in a + -- different pairs() order, so without this workers could compute pathPower + -- along a different (equally short) path than the one the UI displays. + for _, batch in ipairs(batches) do + batch.paths = { } + batch.depends = { } + for _, nodeId in ipairs(batch.nodeIds) do + local node = self.build.spec.nodes[nodeId] + if node then + if node.alloc and node.depends then + local ids = { } + for _, depNode in ipairs(node.depends) do + t_insert(ids, depNode.id) + end + batch.depends[tostring(nodeId)] = ids + elseif node.path then + local ids = { } + for _, pathNode in ipairs(node.path) do + t_insert(ids, pathNode.id) + end + batch.paths[tostring(nodeId)] = ids + end + end + end + end + return batches +end + +-- Try to run the power build on background workers; returns true if a job was +-- launched, false if the caller should use the single-core coroutine instead +function CalcsTabClass:LaunchParallelPowerBuild() + local cand = self:EnumeratePowerCandidates() + if not ParallelRunner:ShouldParallelize(cand.total) then + return false + end + local buildXML = self.build:SaveDB("parallelPower") + if not buildXML then + return false + end + local revision = self.build.outputRevision + local calcFunc, calcBase = self:GetMiscCalculator() + local powerStat = self.powerStat + local useFullDPS = powerStat and powerStat.stat == "FullDPS" + local powerStatIndex + if powerStat then + for i, stat in ipairs(data.powerStatList) do + if stat == powerStat then + powerStatIndex = i + break + end + end + end + local batches = self:BuildPowerBatches(cand, ParallelRunner:GetWorkerCount()) + -- The tree view reads powerMax while the heat map is enabled; the coroutine + -- path initializes it on its first resume, so the worker path must provide + -- the same neutral values until the merge overwrites them + local neutralPowerMax = { + singleStat = 0, + offence = 0, + offencePerPoint = 0, + defence = 0, + defencePerPoint = 0 + } + local job = ParallelRunner:LaunchJob({ + mode = "nodePower", + buildXML = buildXML, + common = { + powerStatIndex = powerStatIndex, + nodePowerMaxDepth = self.nodePowerMaxDepth, + }, + batches = batches, + total = cand.total, + expectedBaseline = PowerCalcTasks.extractBaseline(calcBase, powerStat, useFullDPS), + onProgress = function(percent) + if self.build.powerBuilderProgressCallback then + self.build.powerBuilderProgressCallback(percent) + end + end, + onComplete = function(payloads) + self.powerJob = nil + self:MergeParallelPower(payloads, revision) + end, + onError = function(errMsg) + -- Fall back to the single-core coroutine path + self.powerJob = nil + self.powerBuilder = coroutine.create(self.PowerBuilder) + end, + }) + if not job then + return false + end + self.powerMax = neutralPowerMax + self.powerJob = job + return true +end + +-- Apply worker results onto the live build; mirrors the state PowerBuilder +-- leaves behind (node.power tables, powerMax, completion callback) +function CalcsTabClass:MergeParallelPower(payloads, revision) + if self.build.outputRevision ~= revision then + -- The build changed while workers were running; discard and rebuild + ParallelRunner:Log("merge discarded: revision %s vs %s", tostring(revision), tostring(self.build.outputRevision)) + self.powerBuildFlag = true + return + end + local spec = self.build.spec + local decodeNumber = PowerCalcTasks.decodeNumber + for nodeId, node in pairs(spec.nodes) do + wipeTable(node.power) + if node.type == "Mastery" then + node.power.masteryEffects = { } + end + end + for _, node in pairs(spec.tree.clusterNodeMap) do + if node.power then + wipeTable(node.power) + else + node.power = { } + end + end + local newPowerMax = { + singleStat = 0, + offence = 0, + offencePerPoint = 0, + defence = 0, + defencePerPoint = 0 + } + for _, payload in pairs(payloads) do + local results = payload.results or { } + for idStr, power in pairs(results.nodes or { }) do + local node = spec.nodes[tonumber(idStr)] + if node then + node.power.singleStat = decodeNumber(power.singleStat) + node.power.pathPower = decodeNumber(power.pathPower) + node.power.offence = decodeNumber(power.offence) + node.power.defence = decodeNumber(power.defence) + if power.masteryEffects then + node.power.masteryEffects = node.power.masteryEffects or { } + for effIdStr, effect in pairs(power.masteryEffects) do + node.power.masteryEffects[tonumber(effIdStr)] = { + singleStat = decodeNumber(effect.singleStat), + pathPower = decodeNumber(effect.pathPower), + offence = decodeNumber(effect.offence), + defence = decodeNumber(effect.defence), + } + end + end + end + end + for name, power in pairs(results.cluster or { }) do + local node = spec.tree.clusterNodeMap[name] + if node then + node.power = node.power or { } + node.power.singleStat = decodeNumber(power.singleStat) + end + end + if results.powerMax then + for key in pairs(newPowerMax) do + newPowerMax[key] = m_max(newPowerMax[key], decodeNumber(results.powerMax[key]) or 0) + end + end + end + local mergedNodes, poweredNodes = 0, 0 + for _, payload in pairs(payloads) do + for _, power in pairs((payload.results or { }).nodes or { }) do + mergedNodes = mergedNodes + 1 + if power.singleStat then + poweredNodes = poweredNodes + 1 + end + end + end + ParallelRunner:Log("merge applied: %d node results (%d with singleStat), powerMax.singleStat=%s", mergedNodes, poweredNodes, tostring(newPowerMax.singleStat)) + self.powerMax = newPowerMax + self.powerBuilderInitialized = true + if self.build.powerBuilderCallback then + self.build.powerBuilderCallback() + end end function CalcsTabClass:CalculatePowerStat(selection, original, modified) diff --git a/src/Classes/CompareTab.lua b/src/Classes/CompareTab.lua index 24506313c6c..12466f08434 100644 --- a/src/Classes/CompareTab.lua +++ b/src/Classes/CompareTab.lua @@ -2435,39 +2435,21 @@ function CompareTabClass:GetSocketGroupLabel(group) return table.concat(names, " + ") end --- Coroutine: calculate power of compared build elements against primary build -function CompareTabClass:ComparePowerBuilder(compareEntry, powerStat, categories) - local results = {} - local useFullDPS = powerStat.stat == "FullDPS" - - -- Get calculator for primary build - local calcFunc, calcBase = self.calcs.getMiscCalculator(self.primaryBuild) - - -- Find display stat for formatting - local displayStat = nil - for _, ds in ipairs(self.primaryBuild.displayStats) do - if ds.stat == powerStat.stat then - displayStat = ds - break - end - end - if not displayStat then - displayStat = { fmt = ".1f" } - end - - local total = 0 - local processed = 0 - local start = GetTime() - - -- Count total work items for progress +-- Enumerate the candidates the compare power report would calculate, as an +-- ordered list of descriptors. Descriptors carry only stable identifiers +-- (node ids, slot names, list indices, config var names) so they can be sent +-- to worker VMs which resolve them against their own reconstructed builds. +function CompareTabClass:EnumerateComparePowerCandidates(compareEntry, categories) + local descriptors = {} if categories.treeNodes then local compareNodes = compareEntry.spec and compareEntry.spec.allocNodes or {} local primaryNodes = self.primaryBuild.spec and self.primaryBuild.spec.allocNodes or {} - for nodeId, node in pairs(compareNodes) do + for nodeId, _ in pairs(compareNodes) do if type(nodeId) == "number" and nodeId < CLUSTER_NODE_OFFSET and not primaryNodes[nodeId] then local pNode = self.primaryBuild.spec.nodes[nodeId] - if pNode and (pNode.type == "Normal" or pNode.type == "Notable" or pNode.type == "Keystone") and not pNode.ascendancyName then - total = total + 1 + if pNode and (pNode.type == "Normal" or pNode.type == "Notable" or pNode.type == "Keystone") + and not pNode.ascendancyName and pNode.modKey ~= "" then + t_insert(descriptors, { kind = "treeNode", nodeId = nodeId }) end end end @@ -2477,34 +2459,43 @@ function CompareTabClass:ComparePowerBuilder(compareEntry, powerStat, categories if self:ShouldShowRing3(compareEntry) then t_insert(baseSlots, 10, "Ring 3") end - -- we only use jewel slots if both sides have them available self:AddAbyssSockets(compareEntry, baseSlots, true) - for _, slotName in ipairs(baseSlots) do local cSlot = compareEntry.itemsTab and compareEntry.itemsTab.slots[slotName] local cItem = cSlot and compareEntry.itemsTab.items[cSlot.selItemId] - if cItem then - total = total + 1 + local pSlot = self.primaryBuild.itemsTab and self.primaryBuild.itemsTab.slots[slotName] + local pItem = pSlot and self.primaryBuild.itemsTab.items[pSlot.selItemId] + if cItem and cItem.raw and not (pItem and pItem.name == cItem.name) then + t_insert(descriptors, { kind = "item", slotName = slotName }) end end - -- Count jewels for progress tracking - local jewelSlots = self:GetJewelComparisonSlots(compareEntry) - for _, jEntry in ipairs(jewelSlots) do - if jEntry.cItem then - total = total + 1 + -- Jewels; identified by their socket node id (GetJewelComparisonSlots + -- returns a nodeId-sorted list in any VM) + for _, jEntry in ipairs(self:GetJewelComparisonSlots(compareEntry)) do + if jEntry.cItem and jEntry.cItem.raw and not (jEntry.pItem and jEntry.pItem.name == jEntry.cItem.name) then + t_insert(descriptors, { kind = "jewel", nodeId = jEntry.nodeId }) end end end if categories.skillGems then local cGroups = compareEntry.skillsTab and compareEntry.skillsTab.socketGroupList or {} - total = total + #cGroups + local pGroups = self.primaryBuild.skillsTab and self.primaryBuild.skillsTab.socketGroupList or {} + local pSignatures = {} + for _, group in ipairs(pGroups) do + pSignatures[self:GetSocketGroupSignature(group)] = true + end + for groupIndex, cGroup in ipairs(cGroups) do + local sig = self:GetSocketGroupSignature(cGroup) + if sig ~= "" and not pSignatures[sig] then + t_insert(descriptors, { kind = "skillGem", groupIndex = groupIndex }) + end + end end if categories.supportGems then local cMainGroup = compareEntry.skillsTab and compareEntry.skillsTab.socketGroupList[compareEntry.mainSocketGroup] local pMainGroup = self.primaryBuild.skillsTab and self.primaryBuild.skillsTab.socketGroupList[self.primaryBuild.mainSocketGroup] if cMainGroup and pMainGroup then - -- Count support gems in compared build's main group not in primary's main group local pSupportNames = {} for _, gem in ipairs(pMainGroup.gemList or {}) do local ge = self:GetGemGrantedEffect(gem) @@ -2513,12 +2504,12 @@ function CompareTabClass:ComparePowerBuilder(compareEntry, powerStat, categories if name then pSupportNames[name] = true end end end - for _, gem in ipairs(cMainGroup.gemList or {}) do - local ge = self:GetGemGrantedEffect(gem) + for gemIndex, cGem in ipairs(cMainGroup.gemList or {}) do + local ge = self:GetGemGrantedEffect(cGem) if ge and ge.support then - local name = ge.name or gem.nameSpec + local name = ge.name or cGem.nameSpec if name and not pSupportNames[name] then - total = total + 1 + t_insert(descriptors, { kind = "supportGem", gemIndex = gemIndex }) end end end @@ -2531,524 +2522,609 @@ function CompareTabClass:ComparePowerBuilder(compareEntry, powerStat, categories if varData.var and varData.apply and varData.type ~= "text" then local pVal, cVal = self:NormalizeConfigVals(varData, pInput[varData.var], cInput[varData.var]) if pVal ~= cVal then - total = total + 1 + t_insert(descriptors, { kind = "config", var = varData.var }) end end end end + return descriptors +end - if total == 0 then - self.comparePowerResults = results - self.comparePowerProgress = 100 - return - end +-- Create the shared calculation context for computing compare power candidates +function CompareTabClass:MakeComparePowerContext(compareEntry, powerStat) + local ctx = { + compareEntry = compareEntry, + powerStat = powerStat, + useFullDPS = powerStat.stat == "FullDPS", + cache = {}, + } + ctx.calcFunc, ctx.calcBase = self.calcs.getMiscCalculator(self.primaryBuild) + return ctx +end - -- Get baseline stat value for percentage calculation +-- Create the formatting context used to turn raw impact numbers into result +-- rows; main thread only (formatting depends on UI settings) +function CompareTabClass:MakeComparePowerFormatContext(compareEntry, powerStat, calcBase) + local displayStat = nil + for _, ds in ipairs(self.primaryBuild.displayStats) do + if ds.stat == powerStat.stat then + displayStat = ds + break + end + end + if not displayStat then + displayStat = { fmt = ".1f" } + end local baseStatValue = calcBase[powerStat.stat] or 0 if powerStat.transform then baseStatValue = powerStat.transform(baseStatValue) end + return { + compareEntry = compareEntry, + powerStat = powerStat, + displayStat = displayStat, + baseStatValue = baseStatValue, + } +end - -- Helper to format an impact value and compute percentage - local function formatImpact(impact) - local displayVal = impact * ((displayStat.pc or displayStat.mod) and 100 or 1) - local rawNumStr = s_format("%" .. displayStat.fmt, displayVal) - local isZero = (tonumber(rawNumStr) == 0) - local numStr = formatNumSep(rawNumStr) - - -- Determine color - local isPositive = (displayVal > 0 and not displayStat.lowerIsBetter) or (displayVal < 0 and displayStat.lowerIsBetter) - local isNegative = (displayVal < 0 and not displayStat.lowerIsBetter) or (displayVal > 0 and displayStat.lowerIsBetter) - local color = isPositive and colorCodes.POSITIVE or isNegative and colorCodes.NEGATIVE or "^7" - local sign = displayVal > 0 and "+" or "" - local str = color .. sign .. numStr - - -- Compute percentage change - local percent = 0 - if baseStatValue ~= 0 then - percent = (impact / math.abs(baseStatValue)) * 100 - end +-- Format an impact value and compute percentage +function CompareTabClass:FormatComparePowerImpact(fmtCtx, impact) + local displayStat = fmtCtx.displayStat + local displayVal = impact * ((displayStat.pc or displayStat.mod) and 100 or 1) + local rawNumStr = s_format("%" .. displayStat.fmt, displayVal) + local isZero = (tonumber(rawNumStr) == 0) + local numStr = formatNumSep(rawNumStr) - -- Build combined string: "+1,234.5 (+4.3%)" - local combinedStr = str - if percent ~= 0 then - local pctStr = s_format("%+.1f%%", percent) - combinedStr = str .. " ^7(" .. color .. pctStr .. "^7)" - end + -- Determine color + local isPositive = (displayVal > 0 and not displayStat.lowerIsBetter) or (displayVal < 0 and displayStat.lowerIsBetter) + local isNegative = (displayVal < 0 and not displayStat.lowerIsBetter) or (displayVal > 0 and displayStat.lowerIsBetter) + local color = isPositive and colorCodes.POSITIVE or isNegative and colorCodes.NEGATIVE or "^7" + local sign = displayVal > 0 and "+" or "" + local str = color .. sign .. numStr - return str, displayVal, combinedStr, percent, isZero + -- Compute percentage change + local percent = 0 + if fmtCtx.baseStatValue ~= 0 then + percent = (impact / math.abs(fmtCtx.baseStatValue)) * 100 end - -- ========================================== - -- Tree Nodes - -- ========================================== - if categories.treeNodes then - local compareNodes = compareEntry.spec and compareEntry.spec.allocNodes or {} - local primaryNodes = self.primaryBuild.spec and self.primaryBuild.spec.allocNodes or {} - local cache = {} + -- Build combined string: "+1,234.5 (+4.3%)" + local combinedStr = str + if percent ~= 0 then + local pctStr = s_format("%+.1f%%", percent) + combinedStr = str .. " ^7(" .. color .. pctStr .. "^7)" + end - for nodeId, _ in pairs(compareNodes) do - if type(nodeId) == "number" and nodeId < CLUSTER_NODE_OFFSET and not primaryNodes[nodeId] then - local pNode = self.primaryBuild.spec.nodes[nodeId] - if pNode and (pNode.type == "Normal" or pNode.type == "Notable" or pNode.type == "Keystone") - and not pNode.ascendancyName and pNode.modKey ~= "" then - local output - if cache[pNode.modKey] then - output = cache[pNode.modKey] - else - output = calcFunc({ addNodes = { [pNode] = true } }, useFullDPS) - cache[pNode.modKey] = output - end - local impact = self.primaryBuild.calcsTab:CalculatePowerStat(powerStat, output, calcBase) - local pathDist = pNode.pathDist or 0 - if pathDist == 0 then - pathDist = #(pNode.path or {}) - if pathDist == 0 then pathDist = 1 end - end - local perPoint = impact / pathDist - local impactStr, impactVal, combinedImpactStr, impactPercent, impactIsZero = formatImpact(impact) - local perPointStr = formatImpact(perPoint) - - if not impactIsZero then - t_insert(results, { - category = "Tree", - categoryColor = "^7", - nameColor = "^7", - name = pNode.dn, - nodeId = nodeId, - impact = impactVal, - impactStr = impactStr, - impactPercent = impactPercent, - combinedImpactStr = combinedImpactStr, - pathDist = pathDist, - perPoint = perPoint * ((displayStat.pc or displayStat.mod) and 100 or 1), - perPointStr = perPointStr, - }) - end + return str, displayVal, combinedStr, percent, isZero +end - processed = processed + 1 - if coroutine.running() and GetTime() - start > 100 then - self.comparePowerProgress = m_floor(processed / total * 100) - coroutine.yield() - start = GetTime() +-- Compute the raw impact of one candidate descriptor against the primary +-- build. Runs identically on the main thread and inside worker VMs. +-- Returns ok, impact, extra: ok=false means the calculation errored (impact +-- holds the message); impact=nil means the candidate resolved to nothing. +function CompareTabClass:ComputeComparePowerCandidate(ctx, desc) + local compareEntry = ctx.compareEntry + if desc.kind == "treeNode" then + local pNode = self.primaryBuild.spec.nodes[desc.nodeId] + if not (pNode and pNode.modKey ~= "") then + return true, nil + end + local output = ctx.cache[pNode.modKey] + if not output then + output = ctx.calcFunc({ addNodes = { [pNode] = true } }, ctx.useFullDPS) + ctx.cache[pNode.modKey] = output + end + local impact = self.primaryBuild.calcsTab:CalculatePowerStat(ctx.powerStat, output, ctx.calcBase) + local pathDist = pNode.pathDist or 0 + if pathDist == 0 then + pathDist = #(pNode.path or {}) + if pathDist == 0 then pathDist = 1 end + end + return true, impact, { pathDist = pathDist } + elseif desc.kind == "item" then + local slotName = desc.slotName + local cSlot = compareEntry.itemsTab and compareEntry.itemsTab.slots[slotName] + local cItem = cSlot and compareEntry.itemsTab.items[cSlot.selItemId] + if not (cItem and cItem.raw) then + return true, nil + end + local newItem = new("Item", cItem.raw) + newItem:NormaliseQuality() + + -- if our comparison has abyssal jewels, but the primary build + -- doesn't, add those temporarily to the build to work around + -- calcfunc not being able to take in multiple items + local cmpJewels = {} + local oldEquipped = {} + if newItem.abyssalSocketCount > 0 then + for idx = 1, newItem.abyssalSocketCount do + local abyssSlotName = string.format("%s Abyssal Socket %d", slotName, idx) + local cmpJewelSlot = compareEntry.itemsTab.slots[abyssSlotName] + -- save old id and unequip existing + local primaryJewelSlot = self.primaryBuild.itemsTab.slots[abyssSlotName] + oldEquipped[abyssSlotName] = primaryJewelSlot.selItemId + primaryJewelSlot:SetSelItemId(0) + if cmpJewelSlot.selItemId > 0 then + local cmpJewel = compareEntry.itemsTab.items[cmpJewelSlot.selItemId] + -- due to a previous bug where jewel slots didn't + -- get cleared when becoming inactive, so the item + -- might not exist + if cmpJewel then + -- copy item + local itemCopy = new("Item", cmpJewel:BuildRaw()) + table.insert(cmpJewels, itemCopy) + self.primaryBuild.itemsTab:AddItem(itemCopy, false) + + -- equip copied + self.primaryBuild.itemsTab.slots[abyssSlotName]:SetSelItemId(itemCopy.id) end end end end - end - -- ========================================== - -- Items - -- ========================================== - if categories.items then - local baseSlots = { "Weapon 1", "Weapon 2", "Helmet", "Body Armour", "Gloves", "Boots", "Amulet", "Ring 1", "Ring 2", "Belt", "Flask 1", "Flask 2", "Flask 3", "Flask 4", "Flask 5" } - if self:ShouldShowRing3(compareEntry) then - t_insert(baseSlots, 10, "Ring 3") - end + local output = ctx.calcFunc({ repSlotName = slotName, repItem = newItem }, ctx.useFullDPS) + local impact = self.primaryBuild.calcsTab:CalculatePowerStat(ctx.powerStat, output, ctx.calcBase) - -- we only use jewel slots if both sides have them available - self:AddAbyssSockets(compareEntry, baseSlots, true) - - for _, slotName in ipairs(baseSlots) do - local cSlot = compareEntry.itemsTab and compareEntry.itemsTab.slots[slotName] - local cItem = cSlot and compareEntry.itemsTab.items[cSlot.selItemId] - local pSlot = self.primaryBuild.itemsTab and self.primaryBuild.itemsTab.slots[slotName] - local pItem = pSlot and self.primaryBuild.itemsTab.items[pSlot.selItemId] - if cItem and cItem.raw and not (pItem and pItem.name == cItem.name) then - local newItem = new("Item", cItem.raw) - newItem:NormaliseQuality() - - -- if our comparison has abyssal jewels, but the primary build - -- doesn't, add those temporarily to the build to work around - -- calcfunc not being able to take in multiple items - local cmpJewels = {} - local oldEquipped = {} - if newItem.abyssalSocketCount > 0 then - for idx = 1, newItem.abyssalSocketCount do - local abyssSlotName = string.format("%s Abyssal Socket %d", slotName, idx) - local cmpJewelSlot = compareEntry.itemsTab.slots[abyssSlotName] - -- save old id and unequip existing - local primaryJewelSlot = self.primaryBuild.itemsTab.slots[abyssSlotName] - oldEquipped[abyssSlotName] = primaryJewelSlot.selItemId - primaryJewelSlot:SetSelItemId(0) - if cmpJewelSlot.selItemId > 0 then - local cmpJewel = compareEntry.itemsTab.items[cmpJewelSlot.selItemId] - -- due to a previous bug where jewel slots didn't - -- get cleared when becoming inactive, so the item - -- might not exist - if cmpJewel then - -- copy item - local itemCopy = new("Item", cmpJewel:BuildRaw()) - table.insert(cmpJewels, itemCopy) - self.primaryBuild.itemsTab:AddItem(itemCopy, false) - - -- equip copied - self.primaryBuild.itemsTab.slots[abyssSlotName]:SetSelItemId(itemCopy.id) - end + -- restore abyss jewel state + if newItem.abyssalSocketCount > 0 then + for k, v in pairs(oldEquipped) do + self.primaryBuild.itemsTab.slots[k]:SetSelItemId(v) + end + for _, item in ipairs(cmpJewels) do + self.primaryBuild.itemsTab:DeleteItem(item) + end + end + return true, impact, { cmpJewelCount = #cmpJewels } + elseif desc.kind == "jewel" then + local nodeId = desc.nodeId + local cSpec = compareEntry.spec + local cItemId = cSpec and cSpec.jewels and cSpec.jewels[nodeId] + local cItem = cItemId and compareEntry.itemsTab.items[cItemId] + if not (cItem and cItem.raw) then + return true, nil + end + local newItem = new("Item", cItem.raw) + newItem:NormaliseQuality() + local pSpec = self.primaryBuild.spec + local pNodeAllocated = pSpec.allocNodes and pSpec.allocNodes[nodeId] and true or false + local bestImpactVal = nil + if pNodeAllocated then + -- Socket is allocated in primary build, test directly in that socket + local output = ctx.calcFunc({ repSlotName = "Jewel "..nodeId, repItem = newItem }, ctx.useFullDPS) + bestImpactVal = self.primaryBuild.calcsTab:CalculatePowerStat(ctx.powerStat, output, ctx.calcBase) + else + -- Socket is NOT allocated in primary build; try the jewel in every + -- jewel socket on the primary build's tree, temporarily allocating + -- unallocated sockets via addNodes so CalcSetup doesn't skip them + if not ctx.primaryJewelSockets then + ctx.primaryJewelSockets = {} + for _, slot in ipairs(self.primaryBuild.itemsTab.orderedSlots) do + if slot.nodeId then + local node = pSpec.nodes[slot.nodeId] + if node then + t_insert(ctx.primaryJewelSockets, { + slotName = slot.slotName, + nodeId = slot.nodeId, + node = node, + allocated = pSpec.allocNodes and pSpec.allocNodes[slot.nodeId] and true or false, + }) end end end - - - local output = calcFunc({ repSlotName = slotName, repItem = newItem }, useFullDPS) - local impact = self.primaryBuild.calcsTab:CalculatePowerStat(powerStat, output, calcBase) - local impactStr, impactVal, combinedImpactStr, impactPercent, impactIsZero = formatImpact(impact) - - -- restore abyss jewel state - if newItem.abyssalSocketCount > 0 then - for k, v in pairs(oldEquipped) do - self.primaryBuild.itemsTab.slots[k]:SetSelItemId(v) - end - for _, item in ipairs(cmpJewels) do - self.primaryBuild.itemsTab:DeleteItem(item) - end + end + for _, socketInfo in ipairs(ctx.primaryJewelSockets) do + local override = { repSlotName = socketInfo.slotName, repItem = newItem } + if not socketInfo.allocated then + override.addNodes = { [socketInfo.node] = true } end - - if not impactIsZero then - -- Get rarity color for item name - local rarityColor = colorCodes[cItem.rarity] or colorCodes.NORMAL - - - local abyssJewelText = #cmpJewels > 0 and - s_format(", %d Jewel%s", #cmpJewels, #cmpJewels > 1 and "s" or "") or "" - t_insert(results, { - category = "Item", - categoryColor = rarityColor, - nameColor = rarityColor, - name = (cItem.name or "Unknown") .. ", " .. slotName .. abyssJewelText, - itemObj = newItem, - slotName = slotName, - impact = impactVal, - impactStr = impactStr, - impactPercent = impactPercent, - combinedImpactStr = combinedImpactStr, - pathDist = nil, - perPoint = nil, - perPointStr = nil, - }) + local output = ctx.calcFunc(override, ctx.useFullDPS) + local impact = self.primaryBuild.calcsTab:CalculatePowerStat(ctx.powerStat, output, ctx.calcBase) + if bestImpactVal == nil or impact > bestImpactVal then + bestImpactVal = impact end end - processed = processed + 1 - if coroutine.running() and GetTime() - start > 100 then - self.comparePowerProgress = m_floor(processed / total * 100) - coroutine.yield() - start = GetTime() - end end - end + return true, bestImpactVal, { bestSocket = not pNodeAllocated } + elseif desc.kind == "skillGem" then + local cGroup = compareEntry.skillsTab and compareEntry.skillsTab.socketGroupList[desc.groupIndex] + if not cGroup then + return true, nil + end + local pGroups = self.primaryBuild.skillsTab and self.primaryBuild.skillsTab.socketGroupList or {} + -- Temporarily add this socket group to primary build and recalculate + t_insert(pGroups, cGroup) + self.primaryBuild.buildFlag = true - -- ========================================== - -- Jewels (included as items) - -- ========================================== - if categories.items then - -- Build list of jewel socket info in the primary build for fallback testing - -- Each entry has { slotName, nodeId, node, allocated } - local pSpec = self.primaryBuild.spec - local primaryJewelSockets = {} - for _, slot in ipairs(self.primaryBuild.itemsTab.orderedSlots) do - if slot.nodeId then - local node = pSpec.nodes[slot.nodeId] - local allocated = pSpec.allocNodes and pSpec.allocNodes[slot.nodeId] and true or false - if node then - t_insert(primaryJewelSockets, { - slotName = slot.slotName, - nodeId = slot.nodeId, - node = node, - allocated = allocated, - }) - end - end + -- Get a fresh calculator with the added group (pcall to guarantee cleanup) + local ok, gemCalcFunc, gemCalcBase = pcall(function() + return self.calcs.getMiscCalculator(self.primaryBuild) + end) + + -- Always remove the temporarily added group + t_remove(pGroups) + self.primaryBuild.buildFlag = true + + if not ok then + -- gemCalcFunc contains the error message on failure + return false, tostring(gemCalcFunc) end + return true, self.primaryBuild.calcsTab:CalculatePowerStat(ctx.powerStat, gemCalcBase, ctx.calcBase) + elseif desc.kind == "supportGem" then + local cMainGroup = compareEntry.skillsTab and compareEntry.skillsTab.socketGroupList[compareEntry.mainSocketGroup] + local pMainGroup = self.primaryBuild.skillsTab and self.primaryBuild.skillsTab.socketGroupList[self.primaryBuild.mainSocketGroup] + local cGem = cMainGroup and cMainGroup.gemList and cMainGroup.gemList[desc.gemIndex] + if not (cGem and pMainGroup) then + return true, nil + end + -- Create a temporary copy of this support gem + local tempGem = { + nameSpec = cGem.nameSpec, + level = cGem.level, + quality = cGem.quality, + qualityId = cGem.qualityId, + enabled = cGem.enabled, + grantedEffect = cGem.grantedEffect, + gemData = cGem.gemData, + count = cGem.count, + enableGlobal1 = cGem.enableGlobal1, + enableGlobal2 = cGem.enableGlobal2, + } - local jewelSlots = self:GetJewelComparisonSlots(compareEntry) - for _, jEntry in ipairs(jewelSlots) do - if jEntry.cItem and jEntry.cItem.raw and not (jEntry.pItem and jEntry.pItem.name == jEntry.cItem.name) then - local newItem = new("Item", jEntry.cItem.raw) - newItem:NormaliseQuality() + -- Temporarily add to primary build's main socket group + t_insert(pMainGroup.gemList, tempGem) + self.primaryBuild.buildFlag = true - local bestImpactVal = nil - local bestSlotLabel = jEntry.label + local ok, sgCalcFunc, sgCalcBase = pcall(function() + return self.calcs.getMiscCalculator(self.primaryBuild) + end) - if jEntry.pNodeAllocated then - -- Socket is allocated in primary build, test directly in that socket - local output = calcFunc({ repSlotName = jEntry.cSlotName, repItem = newItem }, useFullDPS) - bestImpactVal = self.primaryBuild.calcsTab:CalculatePowerStat(powerStat, output, calcBase) - else - -- Socket is NOT allocated in primary build; try the jewel in every - -- jewel socket on the primary build's tree, temporarily allocating - -- unallocated sockets via addNodes so CalcSetup doesn't skip them - for _, socketInfo in ipairs(primaryJewelSockets) do - local override = { repSlotName = socketInfo.slotName, repItem = newItem } - if not socketInfo.allocated then - override.addNodes = { [socketInfo.node] = true } - end - local output = calcFunc(override, useFullDPS) - local impact = self.primaryBuild.calcsTab:CalculatePowerStat(powerStat, output, calcBase) - if bestImpactVal == nil or impact > bestImpactVal then - bestImpactVal = impact - bestSlotLabel = jEntry.label .. " (best socket)" - end - end - end + -- Always remove the temporarily added gem + t_remove(pMainGroup.gemList) + self.primaryBuild.buildFlag = true - if bestImpactVal ~= nil then - local impactStr, impactVal, combinedImpactStr, impactPercent, impactIsZero = formatImpact(bestImpactVal) - if not impactIsZero then - local rarityColor = colorCodes[jEntry.cItem.rarity] or colorCodes.NORMAL - - t_insert(results, { - category = "Item", - categoryColor = rarityColor, - nameColor = rarityColor, - name = (jEntry.cItem.name or "Unknown") .. ", " .. bestSlotLabel, - itemObj = newItem, - impact = impactVal, - impactStr = impactStr, - impactPercent = impactPercent, - combinedImpactStr = combinedImpactStr, - pathDist = nil, - perPoint = nil, - perPointStr = nil, - }) - end - end - end - processed = processed + 1 - if coroutine.running() and GetTime() - start > 100 then - self.comparePowerProgress = m_floor(processed / total * 100) - coroutine.yield() - start = GetTime() + if not ok then + return false, tostring(sgCalcFunc) + end + return true, self.primaryBuild.calcsTab:CalculatePowerStat(ctx.powerStat, sgCalcBase, ctx.calcBase) + elseif desc.kind == "config" then + local varData + for _, vd in ipairs(self.configOptions) do + if vd.var == desc.var then + varData = vd + break end end - end - - -- ========================================== - -- Skill Gems (socket groups) - -- ========================================== - if categories.skillGems then - local cGroups = compareEntry.skillsTab and compareEntry.skillsTab.socketGroupList or {} - local pGroups = self.primaryBuild.skillsTab and self.primaryBuild.skillsTab.socketGroupList or {} - - -- Build signature set for primary groups - local pSignatures = {} - for _, group in ipairs(pGroups) do - pSignatures[self:GetSocketGroupSignature(group)] = true + if not varData then + return true, nil end + local pInput = self.primaryBuild.configTab.input + local cInput = compareEntry.configTab.input or {} + local cVal = cInput[varData.var] - for _, cGroup in ipairs(cGroups) do - local sig = self:GetSocketGroupSignature(cGroup) - if sig ~= "" and not pSignatures[sig] then - -- Temporarily add this socket group to primary build and recalculate - t_insert(pGroups, cGroup) - self.primaryBuild.buildFlag = true + -- Save original value + local savedVal = pInput[varData.var] - -- Get a fresh calculator with the added group (pcall to guarantee cleanup) - local ok, gemCalcFunc, gemCalcBase = pcall(function() - return self.calcs.getMiscCalculator(self.primaryBuild) - end) + -- Apply compare build's config value + pInput[varData.var] = cVal - -- Always remove the temporarily added group - t_remove(pGroups) - self.primaryBuild.buildFlag = true + -- Rebuild and calculate (pcall to guarantee restore on error) + local ok, cfgCalcFunc, cfgCalcBase = pcall(function() + self.primaryBuild.configTab:BuildModList() + self.primaryBuild.buildFlag = true + return self.calcs.getMiscCalculator(self.primaryBuild) + end) - if not ok then - -- gemCalcFunc contains the error message on failure; skip this group - ConPrintf("Compare power (gem): %s", tostring(gemCalcFunc)) - else - local impact = self.primaryBuild.calcsTab:CalculatePowerStat(powerStat, gemCalcBase, calcBase) - local impactStr, impactVal, combinedImpactStr, impactPercent, impactIsZero = formatImpact(impact) - if not impactIsZero then - local label = self:GetSocketGroupLabel(cGroup) - - t_insert(results, { - category = "Skill gem", - categoryColor = colorCodes.GEM, - nameColor = colorCodes.GEM, - name = label, - impact = impactVal, - impactStr = impactStr, - impactPercent = impactPercent, - combinedImpactStr = combinedImpactStr, - pathDist = nil, - perPoint = nil, - perPointStr = nil, - }) - end - end - end - processed = processed + 1 - if coroutine.running() and GetTime() - start > 100 then - self.comparePowerProgress = m_floor(processed / total * 100) - coroutine.yield() - start = GetTime() - end + -- Always restore original value + pInput[varData.var] = savedVal + self.primaryBuild.configTab:BuildModList() + self.primaryBuild.buildFlag = true + + if not ok then + return false, tostring(cfgCalcFunc) end + return true, self.primaryBuild.calcsTab:CalculatePowerStat(ctx.powerStat, cfgCalcBase, ctx.calcBase) end + return true, nil +end - -- ========================================== - -- Support Gems (from compared build's active skill) - -- ========================================== - if categories.supportGems then +-- Turn a computed impact into a result row for the report list; returns nil +-- when the (formatted) impact is zero. Main thread only. +function CompareTabClass:BuildComparePowerRow(desc, impact, extra, fmtCtx) + local compareEntry = fmtCtx.compareEntry + local impactStr, impactVal, combinedImpactStr, impactPercent, impactIsZero = self:FormatComparePowerImpact(fmtCtx, impact) + if impactIsZero then + return nil + end + if desc.kind == "treeNode" then + local pNode = self.primaryBuild.spec.nodes[desc.nodeId] + if not pNode then + return nil + end + local pathDist = extra.pathDist or 1 + local perPoint = impact / pathDist + local perPointStr = self:FormatComparePowerImpact(fmtCtx, perPoint) + return { + category = "Tree", + categoryColor = "^7", + nameColor = "^7", + name = pNode.dn, + nodeId = desc.nodeId, + impact = impactVal, + impactStr = impactStr, + impactPercent = impactPercent, + combinedImpactStr = combinedImpactStr, + pathDist = pathDist, + perPoint = perPoint * ((fmtCtx.displayStat.pc or fmtCtx.displayStat.mod) and 100 or 1), + perPointStr = perPointStr, + } + elseif desc.kind == "item" then + local cSlot = compareEntry.itemsTab and compareEntry.itemsTab.slots[desc.slotName] + local cItem = cSlot and compareEntry.itemsTab.items[cSlot.selItemId] + if not (cItem and cItem.raw) then + return nil + end + local newItem = new("Item", cItem.raw) + newItem:NormaliseQuality() + local rarityColor = colorCodes[cItem.rarity] or colorCodes.NORMAL + local cmpJewelCount = extra.cmpJewelCount or 0 + local abyssJewelText = cmpJewelCount > 0 and + s_format(", %d Jewel%s", cmpJewelCount, cmpJewelCount > 1 and "s" or "") or "" + return { + category = "Item", + categoryColor = rarityColor, + nameColor = rarityColor, + name = (cItem.name or "Unknown") .. ", " .. desc.slotName .. abyssJewelText, + itemObj = newItem, + slotName = desc.slotName, + impact = impactVal, + impactStr = impactStr, + impactPercent = impactPercent, + combinedImpactStr = combinedImpactStr, + pathDist = nil, + perPoint = nil, + perPointStr = nil, + } + elseif desc.kind == "jewel" then + if not fmtCtx.jewelEntryByNode then + fmtCtx.jewelEntryByNode = {} + for _, jEntry in ipairs(self:GetJewelComparisonSlots(compareEntry)) do + fmtCtx.jewelEntryByNode[jEntry.nodeId] = jEntry + end + end + local jEntry = fmtCtx.jewelEntryByNode[desc.nodeId] + if not (jEntry and jEntry.cItem and jEntry.cItem.raw) then + return nil + end + local newItem = new("Item", jEntry.cItem.raw) + newItem:NormaliseQuality() + local rarityColor = colorCodes[jEntry.cItem.rarity] or colorCodes.NORMAL + local bestSlotLabel = jEntry.label .. (extra.bestSocket and " (best socket)" or "") + return { + category = "Item", + categoryColor = rarityColor, + nameColor = rarityColor, + name = (jEntry.cItem.name or "Unknown") .. ", " .. bestSlotLabel, + itemObj = newItem, + impact = impactVal, + impactStr = impactStr, + impactPercent = impactPercent, + combinedImpactStr = combinedImpactStr, + pathDist = nil, + perPoint = nil, + perPointStr = nil, + } + elseif desc.kind == "skillGem" then + local cGroup = compareEntry.skillsTab and compareEntry.skillsTab.socketGroupList[desc.groupIndex] + if not cGroup then + return nil + end + return { + category = "Skill gem", + categoryColor = colorCodes.GEM, + nameColor = colorCodes.GEM, + name = self:GetSocketGroupLabel(cGroup), + impact = impactVal, + impactStr = impactStr, + impactPercent = impactPercent, + combinedImpactStr = combinedImpactStr, + pathDist = nil, + perPoint = nil, + perPointStr = nil, + } + elseif desc.kind == "supportGem" then local cMainGroup = compareEntry.skillsTab and compareEntry.skillsTab.socketGroupList[compareEntry.mainSocketGroup] - local pMainGroup = self.primaryBuild.skillsTab and self.primaryBuild.skillsTab.socketGroupList[self.primaryBuild.mainSocketGroup] - - if cMainGroup and pMainGroup then - -- Collect support gem names already in primary build's main group - local pSupportNames = {} - for _, gem in ipairs(pMainGroup.gemList or {}) do - local ge = self:GetGemGrantedEffect(gem) - if ge and ge.support then - local name = ge.name or gem.nameSpec - if name then pSupportNames[name] = true end - end + local cGem = cMainGroup and cMainGroup.gemList and cMainGroup.gemList[desc.gemIndex] + if not cGem then + return nil + end + local cGrantedEffect = self:GetGemGrantedEffect(cGem) + local name = (cGrantedEffect and cGrantedEffect.name) or cGem.nameSpec + return { + category = "Support gem", + categoryColor = colorCodes.GEM, + nameColor = colorCodes.GEM, + name = name, + impact = impactVal, + impactStr = impactStr, + impactPercent = impactPercent, + combinedImpactStr = combinedImpactStr, + pathDist = nil, + perPoint = nil, + perPointStr = nil, + } + elseif desc.kind == "config" then + local varData + for _, vd in ipairs(self.configOptions) do + if vd.var == desc.var then + varData = vd + break end + end + if not varData then + return nil + end + local function stripColors(s) + return s:gsub("%^%x", ""):gsub("%^x%x%x%x%x%x%x", "") + end + local pVal = (self.primaryBuild.configTab.input or {})[varData.var] + local cVal = (compareEntry.configTab.input or {})[varData.var] + local displayName = varData.label or varData.var + displayName = displayName:gsub(":$", "") + local pDisplay = stripColors(self:FormatConfigValue(varData, pVal)) + local cDisplay = stripColors(self:FormatConfigValue(varData, cVal)) + return { + category = "Config", + categoryColor = colorCodes.FRACTURED, + nameColor = "^7", + name = displayName .. " (" .. pDisplay .. " -> " .. cDisplay .. ")", + impact = impactVal, + impactStr = impactStr, + impactPercent = impactPercent, + combinedImpactStr = combinedImpactStr, + pathDist = nil, + perPoint = nil, + perPointStr = nil, + } + end + return nil +end - for _, cGem in ipairs(cMainGroup.gemList or {}) do - local cGrantedEffect = self:GetGemGrantedEffect(cGem) - if cGrantedEffect and cGrantedEffect.support then - local name = cGrantedEffect.name or cGem.nameSpec - if name and not pSupportNames[name] then - -- Create a temporary copy of this support gem - local tempGem = { - nameSpec = cGem.nameSpec, - level = cGem.level, - quality = cGem.quality, - qualityId = cGem.qualityId, - enabled = cGem.enabled, - grantedEffect = cGem.grantedEffect, - gemData = cGem.gemData, - count = cGem.count, - enableGlobal1 = cGem.enableGlobal1, - enableGlobal2 = cGem.enableGlobal2, - } - - -- Temporarily add to primary build's main socket group - t_insert(pMainGroup.gemList, tempGem) - self.primaryBuild.buildFlag = true +-- Coroutine: calculate power of compared build elements against primary build +function CompareTabClass:ComparePowerBuilder(compareEntry, powerStat, categories) + local results = {} + local descriptors = self:EnumerateComparePowerCandidates(compareEntry, categories) + local total = #descriptors - local ok, sgCalcFunc, sgCalcBase = pcall(function() - return self.calcs.getMiscCalculator(self.primaryBuild) - end) + if total == 0 then + self.comparePowerResults = results + self.comparePowerProgress = 100 + return + end - -- Always remove the temporarily added gem - t_remove(pMainGroup.gemList) - self.primaryBuild.buildFlag = true + local ctx = self:MakeComparePowerContext(compareEntry, powerStat) + local fmtCtx = self:MakeComparePowerFormatContext(compareEntry, powerStat, ctx.calcBase) + local start = GetTime() - if not ok then - ConPrintf("Compare power (support gem): %s", tostring(sgCalcFunc)) - else - local impact = self.primaryBuild.calcsTab:CalculatePowerStat(powerStat, sgCalcBase, calcBase) - local impactStr, impactVal, combinedImpactStr, impactPercent, impactIsZero = formatImpact(impact) - - if not impactIsZero then - t_insert(results, { - category = "Support gem", - categoryColor = colorCodes.GEM, - nameColor = colorCodes.GEM, - name = name, - impact = impactVal, - impactStr = impactStr, - impactPercent = impactPercent, - combinedImpactStr = combinedImpactStr, - pathDist = nil, - perPoint = nil, - perPointStr = nil, - }) - end - end - processed = processed + 1 - if coroutine.running() and GetTime() - start > 100 then - self.comparePowerProgress = m_floor(processed / total * 100) - coroutine.yield() - start = GetTime() - end - end - end + for processed, desc in ipairs(descriptors) do + local ok, impact, extra = self:ComputeComparePowerCandidate(ctx, desc) + if not ok then + ConPrintf("Compare power (%s): %s", desc.kind, tostring(impact)) + elseif impact then + local row = self:BuildComparePowerRow(desc, impact, extra or {}, fmtCtx) + if row then + t_insert(results, row) end end + if coroutine.running() and GetTime() - start > 100 then + self.comparePowerProgress = m_floor(processed / total * 100) + coroutine.yield() + start = GetTime() + end end - -- ========================================== - -- Config Options - -- ========================================== - if categories.config then - local pInput = self.primaryBuild.configTab.input - local cInput = compareEntry.configTab.input or {} + self.comparePowerResults = results + self.comparePowerProgress = 100 +end - local function stripColors(s) - return s:gsub("%^%x", ""):gsub("%^x%x%x%x%x%x%x", "") +-- Try to run the compare power report on background workers; returns true if a +-- job was launched, false if the caller should use the single-core coroutine +function CompareTabClass:LaunchParallelComparePower(compareEntry) + if not ParallelRunner:IsAvailable() or not compareEntry.xmlText then + return false + end + local powerStat = self.comparePowerStat + local powerStatIndex + for i, stat in ipairs(data.powerStatList) do + if stat == powerStat then + powerStatIndex = i + break end - - for _, varData in ipairs(self.configOptions) do - if varData.var and varData.apply and varData.type ~= "text" then - local pVal = pInput[varData.var] - local cVal = cInput[varData.var] - local pNorm, cNorm = self:NormalizeConfigVals(varData, pVal, cVal) - - if pNorm ~= cNorm then - -- Save original value - local savedVal = pInput[varData.var] - - -- Apply compare build's config value - pInput[varData.var] = cVal - - -- Rebuild and calculate (pcall to guarantee restore on error) - local ok, cfgCalcFunc, cfgCalcBase = pcall(function() - self.primaryBuild.configTab:BuildModList() - self.primaryBuild.buildFlag = true - return self.calcs.getMiscCalculator(self.primaryBuild) - end) - - -- Always restore original value - pInput[varData.var] = savedVal - self.primaryBuild.configTab:BuildModList() - self.primaryBuild.buildFlag = true - - if not ok then - -- cfgCalcFunc contains the error message on failure; skip this config - ConPrintf("Compare power (config): %s", tostring(cfgCalcFunc)) - else - local impact = self.primaryBuild.calcsTab:CalculatePowerStat(powerStat, cfgCalcBase, calcBase) - local impactStr, impactVal, combinedImpactStr, impactPercent, impactIsZero = formatImpact(impact) - - -- Only include configs with non-zero impact - if not impactIsZero then - -- Build display name with value change description - local displayName = varData.label or varData.var - displayName = displayName:gsub(":$", "") - - local pDisplay = stripColors(self:FormatConfigValue(varData, pVal)) - local cDisplay = stripColors(self:FormatConfigValue(varData, cVal)) - - t_insert(results, { - category = "Config", - categoryColor = colorCodes.FRACTURED, - nameColor = "^7", - name = displayName .. " (" .. pDisplay .. " -> " .. cDisplay .. ")", - impact = impactVal, - impactStr = impactStr, - impactPercent = impactPercent, - combinedImpactStr = combinedImpactStr, - pathDist = nil, - perPoint = nil, - perPointStr = nil, - }) + end + if not powerStatIndex then + return false + end + local descriptors = self:EnumerateComparePowerCandidates(compareEntry, self.comparePowerCategories) + -- Skill gem and config candidates each cost a full recalculation, so weight + -- compare candidates higher than plain calcFunc calls + if not ParallelRunner:ShouldParallelize(#descriptors, 3) then + return false + end + local buildXML = self.primaryBuild:SaveDB("parallelCompare") + if not buildXML then + return false + end + local revision = self.primaryBuild.outputRevision + local calcFunc, calcBase = self.calcs.getMiscCalculator(self.primaryBuild) + local fmtCtx = self:MakeComparePowerFormatContext(compareEntry, powerStat, calcBase) + for i, desc in ipairs(descriptors) do + desc.i = i + end + local batches = { } + for i, slice in ipairs(ParallelRunner:PartitionList(descriptors, ParallelRunner:GetWorkerCount())) do + batches[i] = { descriptors = slice } + end + local job + job = ParallelRunner:LaunchJob({ + mode = "compare", + buildXML = buildXML, + auxText = compareEntry.xmlText, + common = { powerStatIndex = powerStatIndex }, + batches = batches, + total = #descriptors, + expectedBaseline = PowerCalcTasks.extractBaseline(calcBase, powerStat, powerStat.stat == "FullDPS"), + onProgress = function(percent) + self.comparePowerProgress = percent + end, + onComplete = function(payloads) + if self.comparePowerJob ~= job then + return + end + self.comparePowerJob = nil + if self.primaryBuild.outputRevision ~= revision or self.comparePowerCompareId ~= compareEntry then + -- The build or comparison changed while workers were running + self.comparePowerDirty = true + return + end + local resultByIndex = { } + for _, payload in pairs(payloads) do + for _, res in ipairs(payload.results or { }) do + resultByIndex[res.i] = res + end + end + local results = { } + for i, desc in ipairs(descriptors) do + local res = resultByIndex[i] + if res then + if not res.ok then + ConPrintf("Compare power (%s): %s", desc.kind, tostring(res.err)) + elseif res.impact then + local row = self:BuildComparePowerRow(desc, PowerCalcTasks.decodeNumber(res.impact), res.extra or { }, fmtCtx) + if row then + t_insert(results, row) end end - - processed = processed + 1 - if coroutine.running() and GetTime() - start > 100 then - self.comparePowerProgress = m_floor(processed / total * 100) - coroutine.yield() - start = GetTime() - end end end - end + self.comparePowerResults = results + self.comparePowerProgress = 100 + self.comparePowerListSynced = false + end, + onError = function() + if self.comparePowerJob ~= job then + return + end + self.comparePowerJob = nil + -- Fall back to the single-core coroutine path + self.comparePowerCoroutine = coroutine.create(function() + self:ComparePowerBuilder(compareEntry, self.comparePowerStat, self.comparePowerCategories) + end) + end, + }) + if not job then + return false end - - self.comparePowerResults = results - self.comparePowerProgress = 100 + self.comparePowerJob = job + return true end -- Drive the compare power report coroutine @@ -3065,9 +3141,18 @@ function CompareTabClass:RunComparePowerReport(compareEntry) self.comparePowerResults = nil self.comparePowerProgress = 0 self.comparePowerListSynced = false - self.comparePowerCoroutine = coroutine.create(function() - self:ComparePowerBuilder(compareEntry, self.comparePowerStat, self.comparePowerCategories) - end) + self.comparePowerCoroutine = nil + if self.comparePowerJob then + -- A rebuild was requested while workers were still running; their + -- results are stale, so cancel and start over + self.comparePowerJob:Cancel() + self.comparePowerJob = nil + end + if not self:LaunchParallelComparePower(compareEntry) then + self.comparePowerCoroutine = coroutine.create(function() + self:ComparePowerBuilder(compareEntry, self.comparePowerStat, self.comparePowerCategories) + end) + end end -- Resume coroutine diff --git a/src/Classes/ItemDBControl.lua b/src/Classes/ItemDBControl.lua index 3136d1a8476..7bc9efc7c5f 100644 --- a/src/Classes/ItemDBControl.lua +++ b/src/Classes/ItemDBControl.lua @@ -214,39 +214,18 @@ function ItemDBClass:BuildSortOrder() t_insert(self.sortOrder, self.sortControl.NAME) end -function ItemDBClass:ListBuilder() +function ItemDBClass:BuildFilteredList() local list = { } for id, item in pairs(self.db.list) do if self:DoesItemMatchFilters(item) then t_insert(list, item) end end + return list +end - if self.sortDetail and self.sortDetail.stat then -- stat-based - local useFullDPS = self.sortDetail.stat == "FullDPS" - local start = GetTime() - local calcFunc, calcBase = self.itemsTab.build.calcsTab:GetMiscCalculator(self.build) - for itemIndex, item in ipairs(list) do - item.measuredPower = 0 - for slotName, slot in pairs(self.itemsTab.slots) do - if self.itemsTab:IsItemValidForSlot(item, slotName) and not slot.inactive and (not slot.weaponSet or slot.weaponSet == (self.itemsTab.activeItemSet.useSecondWeaponSet and 2 or 1)) then - local output = calcFunc(item.base.flask and { toggleFlask = item } or item.base.tincture and { toggleTincture = item } or { repSlotName = slotName, repItem = item }, useFullDPS) - local measuredPower = output.Minion and output.Minion[self.sortMode] or output[self.sortMode] or 0 - if self.sortDetail.transform then - measuredPower = self.sortDetail.transform(measuredPower) - end - item.measuredPower = m_max(item.measuredPower, measuredPower) - end - end - local now = GetTime() - if now - start > 50 then - self.defaultText = "^7Sorting... ("..m_floor(itemIndex/#list*100).."%)" - coroutine.yield() - start = now - end - end - end - +-- Sort the list and publish it +function ItemDBClass:FinishList(list) table.sort(list, function(a, b) for _, data in ipairs(self.sortOrder) do local aVal = a[data.key] @@ -273,6 +252,100 @@ function ItemDBClass:ListBuilder() self.defaultText = "^7No items found that match those filters." end +function ItemDBClass:ListBuilder() + local list = self:BuildFilteredList() + + if self.sortDetail and self.sortDetail.stat then -- stat-based + local useFullDPS = self.sortDetail.stat == "FullDPS" + local start = GetTime() + local calcFunc, calcBase = self.itemsTab.build.calcsTab:GetMiscCalculator(self.build) + for itemIndex, item in ipairs(list) do + item.measuredPower = PowerCalcTasks.itemdb.measureOne(self.itemsTab, calcFunc, self.sortDetail, self.sortMode, useFullDPS, item) + local now = GetTime() + if now - start > 50 then + self.defaultText = "^7Sorting... ("..m_floor(itemIndex/#list*100).."%)" + coroutine.yield() + start = now + end + end + end + + self:FinishList(list) +end + +-- Try to run the stat sort on background workers; returns true if a job was +-- launched, false if the caller should use the single-core coroutine instead +function ItemDBClass:LaunchParallelListBuild() + if not (self.sortDetail and self.sortDetail.stat) or self.db.loading or not ParallelRunner:IsAvailable() then + return false + end + local list = self:BuildFilteredList() + -- Items are checked against every slot they fit, so weight them higher + -- than single-calculation candidates + if not ParallelRunner:ShouldParallelize(#list, 2) then + return false + end + local build = self.itemsTab.build + local buildXML = build:SaveDB("parallelSort") + if not buildXML then + return false + end + local useFullDPS = self.sortDetail.stat == "FullDPS" + local calcFunc, calcBase = build.calcsTab:GetMiscCalculator() + local names = { } + for _, item in ipairs(list) do + t_insert(names, item.name) + end + local batches = { } + for i, nameSlice in ipairs(ParallelRunner:PartitionList(names, ParallelRunner:GetWorkerCount())) do + batches[i] = { names = nameSlice } + end + local job + job = ParallelRunner:LaunchJob({ + mode = "itemdb", + buildXML = buildXML, + common = { + sortMode = self.sortDetail.sortMode, + dbType = self.dbType, + }, + batches = batches, + total = #list, + expectedBaseline = PowerCalcTasks.extractBaseline(calcBase, self.sortDetail, useFullDPS), + onProgress = function(percent) + self.defaultText = "^7Sorting... ("..percent.."%)" + end, + onComplete = function(payloads) + if self.parallelJob ~= job then + return + end + self.parallelJob = nil + local powerByName = { } + for _, payload in pairs(payloads) do + for name, power in pairs(payload.results or { }) do + powerByName[name] = PowerCalcTasks.decodeNumber(power) + end + end + for _, item in ipairs(list) do + item.measuredPower = powerByName[item.name] or 0 + end + self:FinishList(list) + end, + onError = function() + if self.parallelJob ~= job then + return + end + self.parallelJob = nil + self.listBuilder = coroutine.create(self.ListBuilder) + end, + }) + if not job then + return false + end + self.parallelJob = job + self.defaultText = "^7Sorting..." + return true +end + function ItemDBClass:Draw(viewPort) if self.itemsTab.build.outputRevision ~= self.listOutputRevision then self.listBuildFlag = true @@ -280,8 +353,17 @@ function ItemDBClass:Draw(viewPort) if self.listBuildFlag then self.listBuildFlag = false wipeTable(self.list) - self.listBuilder = coroutine.create(self.ListBuilder) + self.listBuilder = nil + if self.parallelJob then + -- A rebuild was requested while workers were still running; their + -- results are stale, so cancel and start over + self.parallelJob:Cancel() + self.parallelJob = nil + end self.listOutputRevision = self.itemsTab.build.outputRevision + if not self:LaunchParallelListBuild() then + self.listBuilder = coroutine.create(self.ListBuilder) + end end if self.listBuilder and not self.db.loading then local res, errMsg = coroutine.resume(self.listBuilder, self) diff --git a/src/Classes/NotableDBControl.lua b/src/Classes/NotableDBControl.lua index d08ec4080c9..8ae01b41611 100644 --- a/src/Classes/NotableDBControl.lua +++ b/src/Classes/NotableDBControl.lua @@ -124,41 +124,28 @@ function NotableDBClass:CalculatePowerStat(selection, original, modified) return originalValue - modifiedValue end -function NotableDBClass:ListBuilder() +function NotableDBClass:BuildFilteredList() local list = { } for id, node in pairs(self.db) do if self:DoesNotableMatchFilters(node) then t_insert(list, node) end end + return list +end - if self.sortDetail and self.sortDetail.stat then -- stat-based - local cache = { } +-- Rescale infinite results, sort the list and publish it +function NotableDBClass:FinishList(list) + if self.sortDetail and self.sortDetail.stat then local infinites = { } - local start = GetTime() - local calcFunc = self.itemsTab.build.calcsTab:GetMiscCalculator() - local itemType = self.itemsTab.displayItem.base.type - local calcBase = calcFunc({ repSlotName = itemType, repItem = self.itemsTab:anointItem(nil) }) self.sortMaxPower = 0 - for nodeIndex, node in ipairs(list) do - node.measuredPower = 0 - if node.modKey ~= "" then - local output = calcFunc({ repSlotName = itemType, repItem = self.itemsTab:anointItem(node) }) - node.measuredPower = self:CalculatePowerStat(self.sortDetail, output, calcBase) - if node.measuredPower == m_huge then - t_insert(infinites, node) - else - self.sortMaxPower = m_max(self.sortMaxPower, node.measuredPower) - end - end - local now = GetTime() - if now - start > 50 then - self.defaultText = "^7Sorting... ("..m_floor(nodeIndex/#list*100).."%)" - coroutine.yield() - start = now + for _, node in ipairs(list) do + if node.measuredPower == m_huge then + t_insert(infinites, node) + else + self.sortMaxPower = m_max(self.sortMaxPower, node.measuredPower or 0) end end - if #infinites > 0 then self.sortMaxPower = self.sortMaxPower * 2 for _, node in ipairs(infinites) do @@ -193,6 +180,103 @@ function NotableDBClass:ListBuilder() self.defaultText = "^7No notables found that match those filters." end +function NotableDBClass:ListBuilder() + local list = self:BuildFilteredList() + + if self.sortDetail and self.sortDetail.stat then -- stat-based + local start = GetTime() + local calcFunc = self.itemsTab.build.calcsTab:GetMiscCalculator() + local itemType = self.itemsTab.displayItem.base.type + local calcBase = calcFunc({ repSlotName = itemType, repItem = self.itemsTab:anointItem(nil) }) + self.sortMaxPower = 0 + for nodeIndex, node in ipairs(list) do + node.measuredPower = PowerCalcTasks.notable.measureOne(self.itemsTab, calcFunc, calcBase, self.sortDetail, itemType, node) + if node.measuredPower ~= m_huge then + self.sortMaxPower = m_max(self.sortMaxPower, node.measuredPower) + end + local now = GetTime() + if now - start > 50 then + self.defaultText = "^7Sorting... ("..m_floor(nodeIndex/#list*100).."%)" + coroutine.yield() + start = now + end + end + end + + self:FinishList(list) +end + +-- Try to run the stat sort on background workers; returns true if a job was +-- launched, false if the caller should use the single-core coroutine instead +function NotableDBClass:LaunchParallelListBuild() + if not (self.sortDetail and self.sortDetail.stat) or not ParallelRunner:IsAvailable() then + return false + end + local list = self:BuildFilteredList() + if not ParallelRunner:ShouldParallelize(#list) then + return false + end + local build = self.itemsTab.build + local buildXML = build:SaveDB("parallelSort") + if not buildXML then + return false + end + local calcFunc, calcBase = build.calcsTab:GetMiscCalculator() + local nodeIds = { } + for _, node in ipairs(list) do + t_insert(nodeIds, node.id) + end + local batches = { } + for i, ids in ipairs(ParallelRunner:PartitionList(nodeIds, ParallelRunner:GetWorkerCount())) do + batches[i] = { nodeIds = ids } + end + local job + job = ParallelRunner:LaunchJob({ + mode = "notable", + buildXML = buildXML, + auxText = self.itemsTab.displayItem:BuildRaw(), + common = { + sortMode = self.sortDetail.sortMode, + anointEnchantSlot = self.itemsTab.anointEnchantSlot or 1, + }, + batches = batches, + total = #list, + expectedBaseline = PowerCalcTasks.extractBaseline(calcBase, self.sortDetail, false), + onProgress = function(percent) + self.defaultText = "^7Sorting... ("..percent.."%)" + end, + onComplete = function(payloads) + if self.parallelJob ~= job then + return + end + self.parallelJob = nil + local powerById = { } + for _, payload in pairs(payloads) do + for idStr, power in pairs(payload.results or { }) do + powerById[tonumber(idStr)] = PowerCalcTasks.decodeNumber(power) + end + end + for _, node in ipairs(list) do + node.measuredPower = powerById[node.id] or 0 + end + self:FinishList(list) + end, + onError = function() + if self.parallelJob ~= job then + return + end + self.parallelJob = nil + self.listBuilder = coroutine.create(self.ListBuilder) + end, + }) + if not job then + return false + end + self.parallelJob = job + self.defaultText = "^7Sorting..." + return true +end + ---@param viewPort table function NotableDBClass:Draw(viewPort) if self.itemsTab.build.outputRevision ~= self.listOutputRevision then @@ -201,8 +285,17 @@ function NotableDBClass:Draw(viewPort) if self.listBuildFlag then self.listBuildFlag = false wipeTable(self.list) - self.listBuilder = coroutine.create(self.ListBuilder) + self.listBuilder = nil + if self.parallelJob then + -- A rebuild was requested while workers were still running; their + -- results are stale, so cancel and start over + self.parallelJob:Cancel() + self.parallelJob = nil + end self.listOutputRevision = self.itemsTab.build.outputRevision + if not self:LaunchParallelListBuild() then + self.listBuilder = coroutine.create(self.ListBuilder) + end end if self.listBuilder then local res, errMsg = coroutine.resume(self.listBuilder, self) diff --git a/src/Launch.lua b/src/Launch.lua index e018879cadc..c6c44c7cc9b 100644 --- a/src/Launch.lua +++ b/src/Launch.lua @@ -146,6 +146,14 @@ function launch:OnKeyDown(key, doubleClick) local before = collectgarbage("count") collectgarbage("collect") ConPrintf("%dkB => %dkB", before, collectgarbage("count")) + elseif key == "F7" and self.devMode then + -- Probe the parallel worker environment using the currently loaded build + local buildMode = self.main and self.main.modes and self.main.modes["BUILD"] + if ParallelRunner and buildMode and buildMode.spec then + ParallelRunner:RunProbe(buildMode) + else + ConPrintf("Probe: no build loaded") + end elseif key == "PAUSE" and self.devMode and profiler then if profiling then profiler.stop() @@ -207,6 +215,10 @@ function launch:OnSubCall(func, ...) end function launch:OnSubError(id, errMsg) + if not self.subScripts[id] then + -- Subscript was aborted (e.g. cancelled worker job); ignore the late event + return + end if self.subScripts[id].type == "UPDATE" then self:ShowErrMsg("In update thread: %s", errMsg) self.updateCheckRunning = false @@ -215,11 +227,22 @@ function launch:OnSubError(id, errMsg) if errMsg then self:ShowErrMsg("In download callback: %s", errMsg) end + elseif self.subScripts[id].type == "CUSTOM" then + if self.subScripts[id].callback then + local cbErrMsg = PCall(self.subScripts[id].callback, nil, errMsg) + if cbErrMsg then + self:ShowErrMsg("In subscript callback: %s", cbErrMsg) + end + end end self.subScripts[id] = nil end function launch:OnSubFinished(id, ...) + if not self.subScripts[id] then + -- Subscript was aborted (e.g. cancelled worker job); ignore the late event + return + end if self.subScripts[id].type == "UPDATE" then self.updateAvailable, self.updateErrMsg = ... self.updateCheckRunning = false diff --git a/src/Modules/Build.lua b/src/Modules/Build.lua index b7376f5d6bb..ac1ef19baed 100644 --- a/src/Modules/Build.lua +++ b/src/Modules/Build.lua @@ -915,6 +915,14 @@ function buildMode:CanExit(mode) end function buildMode:Shutdown() + -- All parallel jobs compute against this build; their results are useless now + ParallelRunner:CancelAll() + if self.calcsTab then + self.calcsTab.powerJob = nil + end + if self.compareTab then + self.compareTab.comparePowerJob = nil + end if launch.devMode and (not main.disableDevAutoSave) and self.targetVersion and not self.abortSave then if self.dbFileName then self:SaveDBFile() diff --git a/src/Modules/DataLegionLookUpTableHelper.lua b/src/Modules/DataLegionLookUpTableHelper.lua index 05b6982ff9e..4a7bbccc05d 100644 --- a/src/Modules/DataLegionLookUpTableHelper.lua +++ b/src/Modules/DataLegionLookUpTableHelper.lua @@ -50,9 +50,25 @@ local function loadJewelFile(jewelTypeName) jewelData = uncompressedFile:read("*a") uncompressedFile:close() end - if jewelData then + if jewelData and #jewelData > 0 then return jewelData end + jewelData = nil + end + + -- Headless contexts (busted tests, calculation workers) have no working + -- NewFileSearch, so file freshness can't be compared above; trust an + -- existing non-empty .bin cache, which the full application keeps fresh + if not uncompressedFileAttr.fileName and not compressedFileAttr.fileName then + local uncompressedFile = io.open(scriptPath .. jewelTypeName .. ".bin", "rb") + if uncompressedFile then + jewelData = uncompressedFile:read("*a") + uncompressedFile:close() + if jewelData and #jewelData > 0 then + return jewelData + end + jewelData = nil + end end ConPrintf("Failed to load " .. scriptPath .. jewelTypeName .. ".bin, or data is out of date, falling back to compressed file") @@ -63,10 +79,18 @@ local function loadJewelFile(jewelTypeName) elseif splitFile ~= "" then jewelData = Inflate(splitFile) end + if jewelData == "" then + -- Inflate is stubbed out in headless environments; treat as missing + -- rather than caching empty data + jewelData = nil + end if jewelData == nil then ConPrintf("Failed to load either file: " .. jewelTypeName .. ".zip, " .. jewelTypeName .. ".bin") - else + elseif compressedFileAttr.fileName or splitFile ~= "" then + -- Cache the uncompressed data for future loads. Only done when a real + -- NewFileSearch located the compressed file: headless workers and tests + -- must never write the cache shared with the full application local uncompressedFile = io.open(scriptPath .. jewelTypeName .. ".bin", "wb+") if uncompressedFile then uncompressedFile:write(jewelData) diff --git a/src/Modules/Main.lua b/src/Modules/Main.lua index beae31b7beb..9929eb565ac 100644 --- a/src/Modules/Main.lua +++ b/src/Modules/Main.lua @@ -26,6 +26,8 @@ LoadModule("Modules/BuildSiteTools") -- Load as global so other modules can access the same instance ToastNotification = LoadModule("Modules/ToastNotification") +ParallelRunner = LoadModule("Modules/ParallelRunner") +PowerCalcTasks = LoadModule("Modules/PowerCalcTasks") --[[if launch.devMode then for skillName, skill in pairs(data.enchantments.Helmet) do @@ -119,6 +121,8 @@ function main:Init() self.showFlavourText = true self.showAnimations = true self.showAllItemAffixes = true + self.computationMode = "MULTI" + self.workerCount = 0 self.errorReadingSettings = false if not SetDPIScaleOverridePercent then SetDPIScaleOverridePercent = function(scale) end end @@ -630,6 +634,12 @@ function main:LoadSettings(ignoreBuild) self.dpiScaleOverridePercent = tonumber(node.attrib.dpiScaleOverridePercent) or 0 SetDPIScaleOverridePercent(self.dpiScaleOverridePercent) end + if node.attrib.computationMode == "MULTI" or node.attrib.computationMode == "SINGLE" then + self.computationMode = node.attrib.computationMode + end + if node.attrib.workerCount then + self.workerCount = m_min(m_max(tonumber(node.attrib.workerCount) or 0, 0), 16) + end end end end @@ -762,6 +772,8 @@ function main:SaveSettings() showAnimations = tostring(self.showAnimations), showAllItemAffixes = tostring(self.showAllItemAffixes), dpiScaleOverridePercent = tostring(self.dpiScaleOverridePercent), + computationMode = self.computationMode, + workerCount = tostring(self.workerCount or 0), } }) local res, errMsg = common.xml.SaveXMLFile(setXML, self.userPath.."Settings.xml") if not res then @@ -845,11 +857,13 @@ function main:OpenOptionsPopup(savedState) showFlavourText = self.showFlavourText, showAnimations = self.showAnimations, showAllItemAffixes = self.showAllItemAffixes, - dpiScaleOverridePercent = self.dpiScaleOverridePercent + dpiScaleOverridePercent = self.dpiScaleOverridePercent, + computationMode = self.computationMode, + workerCount = self.workerCount } -- NOTE: Height needs to be adjusted if more menu options are added - local oneColumnHeightReq = 850 -- Min height required to not split menu into two columns + local oneColumnHeightReq = 880 -- Min height required to not split menu into two columns local columnWidth = 600 local startingY = 20 @@ -954,6 +968,29 @@ function main:OpenOptionsPopup(savedState) controls.nodePowerTheme.tooltipText = "Changes the colour scheme used for the node power display on the passive tree." controls.nodePowerTheme:SelByValue(self.nodePowerTheme, "theme") + nextRow() + controls.computationMode = new("DropDownControl", { "TOPLEFT", controls.sectionAnchor, "TOPLEFT" }, { currentX + defaultLabelPlacementX, currentY, 170, 18 }, { + { label = "Multi-core (recommended)", mode = "MULTI" }, + { label = "Single-core", mode = "SINGLE" }, + }, function(index, value) + self.computationMode = value.mode + if ParallelRunner then + -- Give parallel mode another chance if a previous failure disabled it + ParallelRunner.disabledThisSession = false + end + end) + controls.computationModeLabel = new("LabelControl", { "RIGHT", controls.computationMode, "LEFT" }, { defaultLabelSpacingPx, 0, 0, 16 }, "^7Computation mode:") + controls.computationMode.tooltipText = "Multi-core runs heavy calculations (tree heat map / power report, compare tab power report,\nitem and anoint list sorting) on background worker threads, using multiple CPU cores.\nEach worker uses extra memory while it runs. If results ever look wrong, switch to single-core.\nSingle-core uses the original in-process calculation." + controls.computationMode:SelByValue(self.computationMode, "mode") + controls.workerCount = new("EditControl", { "LEFT", controls.computationMode, "RIGHT" }, { 4, 0, 60, 18 }, tostring(self.workerCount or 0), nil, "%D", 2, function(buf) + self.workerCount = m_min(m_max(tonumber(buf) or 0, 0), 16) + end) + controls.workerCountLabel = new("LabelControl", { "LEFT", controls.workerCount, "RIGHT" }, { 4, 0, 0, 16 }, "^7worker threads (0 = auto)") + controls.workerCount.tooltipText = function() + local detected = tonumber(os.getenv("NUMBER_OF_PROCESSORS") or "") or 2 + return "Number of background worker threads used in multi-core mode.\n0 = automatic: one less than the number of CPU cores, up to 8.\nOn this system, automatic uses "..m_max(1, m_min(detected - 1, 8)).." workers." + end + nextRow() controls.colorPositive = new("EditControl", { "TOPLEFT", controls.sectionAnchor, "TOPLEFT" }, { currentX + defaultLabelPlacementX, currentY, 100, 18 }, tostring(self.colorPositive:gsub('^(^)', '0')), nil, nil, 8, function(buf) local match = string.match(buf, "0x%x+") @@ -1193,6 +1230,8 @@ function main:OpenOptionsPopup(savedState) self.showAllItemAffixes = savedState.showAllItemAffixes self.dpiScaleOverridePercent = savedState.dpiScaleOverridePercent SetDPIScaleOverridePercent(self.dpiScaleOverridePercent) + self.computationMode = savedState.computationMode + self.workerCount = savedState.workerCount main:ClosePopup() end) diff --git a/src/Modules/ParallelRunner.lua b/src/Modules/ParallelRunner.lua new file mode 100644 index 00000000000..bc78ae4da32 --- /dev/null +++ b/src/Modules/ParallelRunner.lua @@ -0,0 +1,361 @@ +-- Path of Building +-- +-- Module: Parallel Runner +-- Worker pool for running batched calculations on background threads. +-- Each worker is an isolated Lua VM started via the engine's LaunchSubScript; +-- work is described by a mode string plus JSON payloads, and results come back +-- as a single JSON string per worker. If a worker fails, or its results diverge +-- from the main thread's baseline, the whole job fails and the caller falls +-- back to the single-core code path. +-- +local dkjson = require "dkjson" + +local ipairs = ipairs +local pairs = pairs +local t_insert = table.insert +local m_abs = math.abs +local m_max = math.max +local m_min = math.min +local m_floor = math.floor +local s_format = string.format + +local ParallelRunner = { + jobCount = 0, + activeJobs = { }, + -- Set after a worker error or baseline mismatch so we don't repeatedly pay + -- worker bootstrap costs for a broken configuration. Cleared on restart or + -- when the user changes the computation mode in Options. + disabledThisSession = false, +} + +-- In dev mode, mirror job lifecycle events to a log file so worker problems +-- can be diagnosed without console access +local function logEvent(fmt, ...) + if not (launch and launch.devMode) then + return + end + local file = io.open(GetScriptPath().."/parallel_log.txt", "a") + if file then + file:write(os.date("%H:%M:%S"), " ", s_format(fmt, ...), "\n") + file:close() + end +end + +local jobClass = { } +jobClass.__index = jobClass + +-- Cancel all outstanding workers of this job; late finish/error events for +-- aborted subscripts are ignored (nil-guarded in launch:OnSubFinished/OnSubError) +function jobClass:Cancel() + if self.cancelled then + return + end + self.cancelled = true + for i, id in pairs(self.workers) do + AbortSubScript(id) + launch.subScripts[id] = nil + self.workers[i] = nil + end + ParallelRunner.activeJobs[self.id] = nil +end + +function ParallelRunner:IsAvailable() + -- LaunchSubScript is engine-provided; headless mode stubs it to return nil, + -- and forks without it should transparently stay on the single-core path + if type(LaunchSubScript) ~= "function" then + return false + end + if self.disabledThisSession then + return false + end + return not main or main.computationMode ~= "SINGLE" +end + +function ParallelRunner:GetWorkerCount() + local count = main and main.workerCount or 0 + if count == 0 then + -- Auto: leave one core for the UI thread + local detected = tonumber(os.getenv("NUMBER_OF_PROCESSORS") or "") or 2 + return m_max(1, m_min(detected - 1, 8)) + end + return m_max(1, m_min(count, 16)) +end + +-- Decide whether a job is big enough to be worth the per-worker bootstrap cost. +-- weight is a relative estimate of per-candidate cost (1 = one cached calcFunc call) +function ParallelRunner:ShouldParallelize(candidateCount, weight) + if not self:IsAvailable() then + return false + end + return candidateCount * (weight or 1) >= 300 +end + +-- Split a flat list into up to numBatches round-robin slices +function ParallelRunner:PartitionList(list, numBatches) + local batches = { } + local n = m_min(numBatches, #list) + for i = 1, n do + batches[i] = { } + end + for i, item in ipairs(list) do + t_insert(batches[((i - 1) % n) + 1], item) + end + return batches +end + +-- Compare the baseline stats computed by a worker against the main thread's. +-- Any disagreement means the build didn't survive the XML round trip intact, +-- so per-candidate results can't be trusted. Returns a description of the +-- first mismatch, or nil if everything agrees. +local function validateBaseline(got, expected) + if type(got) ~= "table" then + return "no baseline returned" + end + for stat, value in pairs(expected) do + local workerValue = got[stat] + if type(workerValue) ~= "number" then + return s_format("stat %s missing", stat) + end + if m_abs(workerValue - value) > m_max(1e-6, 1e-3 * m_max(m_abs(workerValue), m_abs(value))) then + return s_format("stat %s: worker %g vs main %g", stat, workerValue, value) + end + end +end + +-- Launch a job: +-- desc = { +-- mode = "probe"/"nodePower"/"compare"/"notable"/"itemdb", +-- buildXML = , +-- auxText = , +-- common = , +-- batches = , +-- total = , +-- expectedBaseline = number, validated against each worker>, +-- onProgress = function(percent), +-- onComplete = function(payloads), -- array of decoded worker payloads +-- onError = function(errMsg), -- called at most once; caller should fall back +-- } +-- Returns the job (with job:Cancel()), or nil + error message if workers +-- could not be launched at all (caller should fall back synchronously). +function ParallelRunner:LaunchJob(desc) + local scriptText, errMsg = self:GetWorkerScript() + if not scriptText then + return nil, errMsg + end + self.jobCount = self.jobCount + 1 + local job = setmetatable({ + id = self.jobCount, + desc = desc, + workers = { }, + payloads = { }, + progress = { }, + pending = 0, + lastPercent = -1, + }, jobClass) + for i, batch in ipairs(desc.batches) do + local specJSON = dkjson.encode({ common = desc.common, batch = batch }) + local id = LaunchSubScript(scriptText, + "GetScriptPath,GetRuntimePath,GetWorkDir,MakeDir,GetUserPath", + "ConPrintf,ParallelWorkerProgress", + job.id, i, GetScriptPath(), desc.mode, desc.buildXML or "", desc.auxText or "", specJSON) + if not id then + job:Cancel() + return nil, "Unable to launch background worker" + end + job.workers[i] = id + job.pending = job.pending + 1 + launch:RegisterSubScript(id, function(resultJSON, workerErrMsg) + self:OnWorkerDone(job, i, resultJSON, workerErrMsg) + end) + end + if job.pending == 0 then + return nil, "No work to distribute" + end + self.activeJobs[job.id] = job + logEvent("job %d launched: mode=%s workers=%d total=%s buildXML=%dB", job.id, desc.mode, job.pending, tostring(desc.total), #(desc.buildXML or "")) + return job +end + +function ParallelRunner:OnWorkerDone(job, workerIdx, resultJSON, errMsg) + if job.cancelled or job.failed then + return + end + job.workers[workerIdx] = nil + if errMsg or not resultJSON then + self:FailJob(job, errMsg or "worker returned no result") + return + end + local payload, _, decodeErr = dkjson.decode(resultJSON) + if type(payload) ~= "table" then + self:FailJob(job, "malformed worker result: "..tostring(decodeErr)) + return + end + if payload.error then + self:FailJob(job, payload.error, payload) + return + end + if job.desc.expectedBaseline then + local mismatch = validateBaseline(payload.baseline, job.desc.expectedBaseline) + if mismatch then + self:FailJob(job, "worker results diverged from main thread ("..mismatch..")") + return + end + end + job.payloads[workerIdx] = payload + job.pending = job.pending - 1 + logEvent("job %d worker %d done: %dB result, %d pending", job.id, workerIdx, #resultJSON, job.pending) + if job.pending == 0 then + self.activeJobs[job.id] = nil + if job.desc.onComplete then + logEvent("job %d complete, merging", job.id) + job.desc.onComplete(job.payloads) + end + end +end + +function ParallelRunner:FailJob(job, errMsg, payload) + job.failed = true + job:Cancel() + self.disabledThisSession = true + ConPrintf("Parallel computation failed, falling back to single-core: %s", tostring(errMsg)) + logEvent("job %d FAILED: %s", job.id, tostring(errMsg)) + if job.desc.onError then + job.desc.onError(errMsg, payload) + end +end + +function ParallelRunner:CancelAll() + for _, job in pairs(self.activeJobs) do + job:Cancel() + end +end + +-- Public wrapper so call sites can add their own entries to the dev log +function ParallelRunner:Log(fmt, ...) + logEvent(fmt, ...) +end + +function ParallelRunner:GetWorkerScript() + if not self.workerScript then + local file = io.open("CalcWorker.lua", "r") + if not file then + file = io.open(GetScriptPath().."/CalcWorker.lua", "r") + end + if not file then + return nil, "CalcWorker.lua not found" + end + self.workerScript = file:read("*a") + file:close() + end + return self.workerScript +end + +-- Dev-mode probe: launches a single worker that bootstraps a headless program, +-- reloads the given build from XML, and reports facts about the worker VM +-- (injected functions, timings, baseline agreement). Results go to the console +-- and to probe_report.txt next to the scripts. +function ParallelRunner:RunProbe(build) + local reportLines = { } + local function report(fmt, ...) + local line = s_format(fmt, ...) + ConPrintf("%s", line) + t_insert(reportLines, line) + end + local function writeReport() + local file = io.open(GetScriptPath().."/probe_report.txt", "w") + if file then + file:write(table.concat(reportLines, "\n"), "\n") + file:close() + end + end + local buildXML = build:SaveDB("probe") + if not buildXML then + report("Probe: unable to serialize build") + writeReport() + return + end + local expectedBaseline + if build.calcsTab and build.calcsTab.miscCalculator then + local calcFunc, calcBase = build.calcsTab:GetMiscCalculator() + expectedBaseline = PowerCalcTasks.extractBaseline(calcBase, nil, false) + end + local startTime = GetTime() + report("Probe: launching worker (build XML: %d bytes)...", #buildXML) + local job, errMsg = self:LaunchJob({ + mode = "probe", + buildXML = buildXML, + common = { }, + batches = { { } }, + total = 1, + -- Baselines are compared manually below so the probe can report every + -- stat instead of failing on the first mismatch + onComplete = function(payloads) + local payload = payloads[1] + report("Probe: completed in %d ms (wall)", GetTime() - startTime) + for key, value in pairs(payload.probe or { }) do + report("Probe: %s = %s", key, tostring(value)) + end + for key, value in pairs(payload.timing or { }) do + report("Probe: timing.%s = %s", key, tostring(value)) + end + local mismatches = 0 + for stat, value in pairs(expectedBaseline or { }) do + local workerValue = payload.baseline and payload.baseline[stat] + if type(workerValue) ~= "number" then + report("Probe: BASELINE %s: main %.6g, worker MISSING", stat, value) + mismatches = mismatches + 1 + elseif m_abs(workerValue - value) > m_max(1e-6, 1e-3 * m_max(m_abs(workerValue), m_abs(value))) then + report("Probe: BASELINE %s: main %.6g, worker %.6g (%+.3f%%)", stat, value, workerValue, (workerValue - value) / value * 100) + mismatches = mismatches + 1 + else + report("Probe: baseline %s: main %.6g, worker %.6g OK", stat, value, workerValue) + end + end + if mismatches == 0 then + report("Probe: baseline validated OK against main thread") + -- A successful probe proves the environment works; re-enable if a + -- previous failure had disabled parallel mode + self.disabledThisSession = false + else + report("Probe: %d baseline stats DIVERGED", mismatches) + end + writeReport() + end, + onError = function(probeErrMsg, payload) + report("Probe: FAILED: %s", tostring(probeErrMsg)) + if payload and payload.probe then + for key, value in pairs(payload.probe) do + report("Probe: env.%s = %s", key, tostring(value)) + end + end + writeReport() + end, + }) + if not job then + report("Probe: could not launch worker: %s", tostring(errMsg)) + writeReport() + end +end + +-- Called from worker threads via launch:OnSubCall; aggregates per-worker +-- progress counters into one percentage for the job's progress callback +function ParallelWorkerProgress(jobId, workerIdx, done, total) + local job = ParallelRunner.activeJobs[jobId] + if not job or job.cancelled or job.failed then + return + end + job.progress[workerIdx] = done + if job.desc.onProgress and job.desc.total and job.desc.total > 0 then + local doneSum = 0 + for _, workerDone in pairs(job.progress) do + doneSum = doneSum + workerDone + end + local percent = m_min(m_floor(doneSum / job.desc.total * 100), 100) + if percent ~= job.lastPercent then + job.lastPercent = percent + job.desc.onProgress(percent) + end + end +end + +return ParallelRunner diff --git a/src/Modules/PowerCalcTasks.lua b/src/Modules/PowerCalcTasks.lua new file mode 100644 index 00000000000..3175332a050 --- /dev/null +++ b/src/Modules/PowerCalcTasks.lua @@ -0,0 +1,341 @@ +-- Path of Building +-- +-- Module: Power Calc Tasks +-- Shared per-candidate calculation logic for the parallelizable "iterate over +-- many candidates and rank them" features (node power / compare / anoint +-- notables / item DB sorting). Loaded both by the main thread (single-core code +-- paths and result merging) and by CalcWorker.lua inside worker VMs, so the two +-- paths always run identical calculation code. +-- +local ipairs = ipairs +local pairs = pairs +local type = type +local t_insert = table.insert +local m_huge = math.huge +local m_max = math.max + +local tasks = { } + +local function isFiniteNumber(value) + -- NaN compares unequal to itself; math.huge can't survive a JSON round trip + return type(value) == "number" and value == value and value ~= m_huge and value ~= -m_huge +end + +-- Baseline stats used to validate that a worker's reconstructed build produces +-- the same numbers as the main thread's live build. Mirrors the stats consumed +-- by CalcsTab:CalculateCombinedOffDefStat plus the active power stat. +local baselineStatList = { + "LifeUnreserved", "Life", "Armour", "EnergyShield", "EnergyShieldRecoveryCap", + "Evasion", "LifeRegenRecovery", "EnergyShieldRegenRecovery", "CombinedDPS", +} + +function tasks.extractBaseline(output, powerStat, useFullDPS) + local baseline = { } + if type(output) ~= "table" then + return baseline + end + for _, stat in ipairs(baselineStatList) do + if isFiniteNumber(output[stat]) then + baseline[stat] = output[stat] + end + end + if type(output.Minion) == "table" and isFiniteNumber(output.Minion.CombinedDPS) then + baseline["Minion.CombinedDPS"] = output.Minion.CombinedDPS + end + if useFullDPS and isFiniteNumber(output.FullDPS) then + baseline["FullDPS"] = output.FullDPS + end + if powerStat and powerStat.stat and isFiniteNumber(output[powerStat.stat]) then + baseline["powerStat."..powerStat.stat] = output[powerStat.stat] + end + return baseline +end + +-- JSON-safe number encoding: math.huge/-math.huge/NaN can't cross the JSON +-- boundary, so they're carried as sentinel strings +function tasks.encodeNumber(value) + if type(value) ~= "number" then + return nil + end + if value ~= value then + return 0 + elseif value == m_huge then + return "inf" + elseif value == -m_huge then + return "-inf" + end + return value +end + +function tasks.decodeNumber(value) + if value == "inf" then + return m_huge + elseif value == "-inf" then + return -m_huge + end + return tonumber(value) +end + +-- Resolve a DB sort drop-list selection (built from data.powerStatList by +-- NotableDBControl/ItemDBControl:BuildSortOrder) from its sortMode key; used to +-- reconstruct the same selection inside a worker VM +function tasks.findSortDetail(sortMode) + for _, stat in ipairs(data.powerStatList) do + if (stat.itemField or stat.stat) == sortMode then + return { + sortMode = sortMode, + itemField = stat.itemField, + stat = stat.stat, + transform = stat.transform, + } + end + end +end + +------------------------------------------------------------------------------ +-- Node power (heat map / power report) +------------------------------------------------------------------------------ + +tasks.nodePower = { } + +-- Worker side: run PowerBuilder for the given subset of nodes/cluster notables +-- and serialize the node.power results +function tasks.nodePower.computeBatch(build, common, batch, auxText, progressFunc) + local calcsTab = build.calcsTab + calcsTab.powerStat = common.powerStatIndex and data.powerStatList[common.powerStatIndex] or nil + calcsTab.nodePowerMaxDepth = common.nodePowerMaxDepth + local filter = { nodes = { }, cluster = { } } + for _, nodeId in ipairs(batch.nodeIds or { }) do + filter.nodes[nodeId] = true + end + for _, name in ipairs(batch.clusterNames or { }) do + filter.cluster[name] = true + end + -- Overwrite this spec's path/depends data with the main thread's; a spec + -- rebuilt from XML resolves equal-length shortest-path ties differently, + -- which would change pathPower for nodes with multiple viable paths + for idStr, ids in pairs(batch.paths or { }) do + local node = build.spec.nodes[tonumber(idStr)] + if node then + local path = { } + for _, pathId in ipairs(ids) do + local pathNode = build.spec.nodes[pathId] + if pathNode then + t_insert(path, pathNode) + end + end + if #path > 0 then + node.path = path + node.pathDist = #path + end + end + end + for idStr, ids in pairs(batch.depends or { }) do + local node = build.spec.nodes[tonumber(idStr)] + if node then + local depends = { } + for _, depId in ipairs(ids) do + local depNode = build.spec.nodes[depId] + if depNode then + t_insert(depends, depNode) + end + end + node.depends = depends + end + end + calcsTab.workerProgressFunc = progressFunc + local newPowerMax = calcsTab:PowerBuilder(filter) + calcsTab.workerProgressFunc = nil + + local encodeNumber = tasks.encodeNumber + local results = { nodes = { }, cluster = { }, powerMax = { } } + for key, value in pairs(newPowerMax or { }) do + results.powerMax[key] = encodeNumber(value) + end + for nodeId in pairs(filter.nodes) do + local node = build.spec.nodes[nodeId] + if node and node.power then + local out = { + singleStat = encodeNumber(node.power.singleStat), + pathPower = encodeNumber(node.power.pathPower), + offence = encodeNumber(node.power.offence), + defence = encodeNumber(node.power.defence), + } + if node.power.masteryEffects then + out.masteryEffects = { } + for effectId, effect in pairs(node.power.masteryEffects) do + out.masteryEffects[tostring(effectId)] = { + singleStat = encodeNumber(effect.singleStat), + pathPower = encodeNumber(effect.pathPower), + offence = encodeNumber(effect.offence), + defence = encodeNumber(effect.defence), + } + end + end + results.nodes[tostring(nodeId)] = out + end + end + for name in pairs(filter.cluster) do + local node = build.spec.tree.clusterNodeMap[name] + if node and node.power then + results.cluster[name] = { singleStat = encodeNumber(node.power.singleStat) } + end + end + + local powerStat = calcsTab.powerStat + local useFullDPS = powerStat and powerStat.stat == "FullDPS" + local calcFunc, calcBase = calcsTab:GetMiscCalculator() + return { + results = results, + baseline = tasks.extractBaseline(calcBase, powerStat, useFullDPS), + } +end + +------------------------------------------------------------------------------ +-- Anoint notable sorting (NotableDBControl) +------------------------------------------------------------------------------ + +tasks.notable = { } + +-- Measure the impact of anointing one notable onto the display item. +-- Identical to the original NotableDBControl:ListBuilder loop body. +function tasks.notable.measureOne(itemsTab, calcFunc, calcBase, sortDetail, itemType, node) + if node.modKey == "" then + return 0 + end + local output = calcFunc({ repSlotName = itemType, repItem = itemsTab:anointItem(node) }) + local original, modified = output, calcBase + if modified.Minion then + original = original.Minion + modified = modified.Minion + end + local originalValue = original[sortDetail.stat] or 0 + local modifiedValue = modified[sortDetail.stat] or 0 + if sortDetail.transform then + originalValue = sortDetail.transform(originalValue) + modifiedValue = sortDetail.transform(modifiedValue) + end + return originalValue - modifiedValue +end + +-- Worker side: auxText is the display item's raw text +function tasks.notable.computeBatch(build, common, batch, auxText, progressFunc) + local itemsTab = build.itemsTab + itemsTab.displayItem = new("Item", auxText) + itemsTab.anointEnchantSlot = common.anointEnchantSlot or 1 + local sortDetail = tasks.findSortDetail(common.sortMode) + if not sortDetail or not sortDetail.stat then + error("notable: unknown sort mode "..tostring(common.sortMode)) + end + local calcFunc, miscCalcBase = build.calcsTab:GetMiscCalculator() + local itemType = itemsTab.displayItem.base.type + local calcBase = calcFunc({ repSlotName = itemType, repItem = itemsTab:anointItem(nil) }) + local results = { } + local nodeIds = batch.nodeIds or { } + for index, nodeId in ipairs(nodeIds) do + local node = build.spec.tree.nodes[nodeId] + if node then + results[tostring(nodeId)] = tasks.encodeNumber(tasks.notable.measureOne(itemsTab, calcFunc, calcBase, sortDetail, itemType, node)) + end + if progressFunc and index % 10 == 0 then + progressFunc(index, #nodeIds) + end + end + return { + results = results, + baseline = tasks.extractBaseline(miscCalcBase, sortDetail, false), + } +end + +------------------------------------------------------------------------------ +-- Compare tab power report (CompareTab) +------------------------------------------------------------------------------ + +tasks.compare = { } + +-- Worker side: auxText is the comparison build's XML; descriptors identify +-- candidates by stable ids and are computed via the same CompareTab methods +-- the single-core path uses +function tasks.compare.computeBatch(build, common, batch, auxText, progressFunc) + local compareTab = build.compareTab + if not compareTab then + error("compare: build has no compare tab") + end + local powerStat = data.powerStatList[common.powerStatIndex] + if not powerStat then + error("compare: unknown power stat index "..tostring(common.powerStatIndex)) + end + local compareEntry = new("CompareEntry", auxText, "CalcWorker comparison") + local ctx = compareTab:MakeComparePowerContext(compareEntry, powerStat) + local results = { } + local descriptors = batch.descriptors or { } + for index, desc in ipairs(descriptors) do + local ok, impact, extra = compareTab:ComputeComparePowerCandidate(ctx, desc) + t_insert(results, { + i = desc.i, + ok = ok and true or false, + impact = ok and impact and tasks.encodeNumber(impact) or nil, + err = not ok and tostring(impact) or nil, + extra = extra, + }) + if progressFunc and index % 5 == 0 then + progressFunc(index, #descriptors) + end + end + return { + results = results, + baseline = tasks.extractBaseline(ctx.calcBase, powerStat, ctx.useFullDPS), + } +end + +------------------------------------------------------------------------------ +-- Item DB sorting (ItemDBControl) +------------------------------------------------------------------------------ + +tasks.itemdb = { } + +-- Measure an item's best impact across all slots it fits. +-- Identical to the original ItemDBControl:ListBuilder loop body. +function tasks.itemdb.measureOne(itemsTab, calcFunc, sortDetail, sortMode, useFullDPS, item) + local measuredPower = 0 + for slotName, slot in pairs(itemsTab.slots) do + if itemsTab:IsItemValidForSlot(item, slotName) and not slot.inactive and (not slot.weaponSet or slot.weaponSet == (itemsTab.activeItemSet.useSecondWeaponSet and 2 or 1)) then + local output = calcFunc(item.base.flask and { toggleFlask = item } or item.base.tincture and { toggleTincture = item } or { repSlotName = slotName, repItem = item }, useFullDPS) + local power = output.Minion and output.Minion[sortMode] or output[sortMode] or 0 + if sortDetail.transform then + power = sortDetail.transform(power) + end + measuredPower = m_max(measuredPower, power) + end + end + return measuredPower +end + +-- Worker side: candidates are item names in the shared unique/rare DB +function tasks.itemdb.computeBatch(build, common, batch, auxText, progressFunc) + local itemsTab = build.itemsTab + local sortDetail = tasks.findSortDetail(common.sortMode) + if not sortDetail or not sortDetail.stat then + error("itemdb: unknown sort mode "..tostring(common.sortMode)) + end + local useFullDPS = sortDetail.stat == "FullDPS" + local db = common.dbType == "RARE" and main.rareDB or main.uniqueDB + local calcFunc, calcBase = build.calcsTab:GetMiscCalculator() + local results = { } + local names = batch.names or { } + for index, name in ipairs(names) do + local item = db.list[name] + if item then + results[name] = tasks.encodeNumber(tasks.itemdb.measureOne(itemsTab, calcFunc, sortDetail, sortDetail.sortMode, useFullDPS, item)) + end + if progressFunc and index % 10 == 0 then + progressFunc(index, #names) + end + end + return { + results = results, + baseline = tasks.extractBaseline(calcBase, sortDetail, useFullDPS), + } +end + +return tasks diff --git a/src/parallel_log.txt b/src/parallel_log.txt new file mode 100644 index 00000000000..54ad0c28cce --- /dev/null +++ b/src/parallel_log.txt @@ -0,0 +1,103 @@ +20:26:05 job 1 launched: mode=nodePower workers=8 total=4348 buildXML=56056B +20:26:17 job 1 worker 8 done: 29479B result, 7 pending +20:26:17 job 1 worker 2 done: 30721B result, 6 pending +20:26:18 job 1 worker 4 done: 28443B result, 5 pending +20:26:18 job 1 worker 5 done: 29249B result, 4 pending +20:26:19 job 1 worker 6 done: 29317B result, 3 pending +20:26:19 job 1 worker 7 done: 31172B result, 2 pending +20:26:20 job 1 worker 1 done: 29046B result, 1 pending +20:26:21 job 1 worker 3 done: 31851B result, 0 pending +20:26:21 job 1 complete, merging +20:26:21 merge applied: 2671 node results (2552 with singleStat), powerMax.singleStat=0 +20:28:31 job 2 launched: mode=nodePower workers=8 total=4348 buildXML=56056B +20:28:45 job 2 worker 2 done: 21231B result, 7 pending +20:28:46 job 2 worker 7 done: 20830B result, 6 pending +20:28:47 job 2 worker 8 done: 20891B result, 5 pending +20:28:47 job 2 worker 1 done: 20824B result, 4 pending +20:28:48 job 2 worker 3 done: 21252B result, 3 pending +20:28:48 job 2 worker 4 done: 21363B result, 2 pending +20:28:48 job 2 worker 5 done: 20719B result, 1 pending +20:28:49 job 2 worker 6 done: 21110B result, 0 pending +20:28:49 job 2 complete, merging +20:28:49 merge applied: 2671 node results (2671 with singleStat), powerMax.singleStat=0 +20:28:51 job 3 launched: mode=nodePower workers=8 total=4348 buildXML=56462B +20:29:11 job 3 worker 5 done: 26456B result, 7 pending +20:29:12 job 3 worker 7 done: 26833B result, 6 pending +20:29:13 job 3 worker 8 done: 26669B result, 5 pending +20:29:14 job 3 worker 1 done: 26444B result, 4 pending +20:29:14 job 3 worker 2 done: 28394B result, 3 pending +20:29:15 job 3 worker 3 done: 28048B result, 2 pending +20:29:16 job 3 worker 4 done: 27750B result, 1 pending +20:29:16 job 3 worker 6 done: 26920B result, 0 pending +20:29:16 job 3 complete, merging +20:29:16 merge applied: 2671 node results (2671 with singleStat), powerMax.singleStat=1811177.1023715 +20:30:35 job 4 launched: mode=nodePower workers=8 total=4348 buildXML=56473B +20:30:36 job 5 launched: mode=nodePower workers=8 total=4348 buildXML=56462B +20:30:55 job 5 worker 3 done: 27915B result, 7 pending +20:30:56 job 5 worker 2 done: 28277B result, 6 pending +20:30:57 job 5 worker 4 done: 27617B result, 5 pending +20:30:57 job 5 worker 5 done: 26405B result, 4 pending +20:30:58 job 5 worker 6 done: 26920B result, 3 pending +20:30:59 job 5 worker 7 done: 26719B result, 2 pending +20:30:59 job 5 worker 8 done: 26641B result, 1 pending +20:31:00 job 5 worker 1 done: 26335B result, 0 pending +20:31:00 job 5 complete, merging +20:31:00 merge applied: 2671 node results (2671 with singleStat), powerMax.singleStat=1811177.1023715 +20:33:05 job 6 launched: mode=nodePower workers=8 total=4348 buildXML=56473B +20:33:12 job 7 launched: mode=nodePower workers=8 total=4348 buildXML=56462B +20:33:15 job 8 launched: mode=nodePower workers=8 total=4348 buildXML=56473B +20:33:18 job 9 launched: mode=nodePower workers=8 total=4348 buildXML=56462B +20:33:37 job 9 worker 3 done: 27926B result, 7 pending +20:33:38 job 9 worker 8 done: 26664B result, 6 pending +20:33:39 job 9 worker 1 done: 26335B result, 5 pending +20:33:40 job 9 worker 2 done: 28266B result, 4 pending +20:33:41 job 9 worker 4 done: 27617B result, 3 pending +20:33:41 job 9 worker 5 done: 26405B result, 2 pending +20:33:42 job 9 worker 6 done: 26920B result, 1 pending +20:33:43 job 9 worker 7 done: 26719B result, 0 pending +20:33:43 job 9 complete, merging +20:33:43 merge applied: 2671 node results (2671 with singleStat), powerMax.singleStat=1811177.1023715 +23:41:17 job 1 launched: mode=nodePower workers=12 total=4348 buildXML=56462B +23:41:20 job 2 launched: mode=nodePower workers=12 total=4348 buildXML=56462B +23:41:40 job 2 worker 10 done: 18596B result, 11 pending +23:41:41 job 2 worker 7 done: 17987B result, 10 pending +23:41:41 job 2 worker 8 done: 17968B result, 9 pending +23:41:42 job 2 worker 11 done: 16783B result, 8 pending +23:41:42 job 2 worker 12 done: 18085B result, 7 pending +23:41:43 job 2 worker 1 done: 18078B result, 6 pending +23:41:43 job 2 worker 2 done: 19757B result, 5 pending +23:41:44 job 2 worker 3 done: 18742B result, 4 pending +23:41:44 job 2 worker 4 done: 18747B result, 3 pending +23:41:44 job 2 worker 5 done: 17663B result, 2 pending +23:41:45 job 2 worker 6 done: 18979B result, 1 pending +23:41:45 job 2 worker 9 done: 17406B result, 0 pending +23:41:45 job 2 complete, merging +23:41:45 merge applied: 2671 node results (2671 with singleStat), powerMax.singleStat=1811177.1023715 +23:41:46 job 3 launched: mode=nodePower workers=12 total=4348 buildXML=56473B +23:41:53 job 4 launched: mode=nodePower workers=12 total=4348 buildXML=56462B +23:42:17 job 5 launched: mode=nodePower workers=12 total=4348 buildXML=56473B +23:42:36 job 5 worker 6 done: 18147B result, 11 pending +23:42:36 job 5 worker 8 done: 18167B result, 10 pending +23:42:37 job 5 worker 9 done: 17909B result, 9 pending +23:42:37 job 5 worker 10 done: 18073B result, 8 pending +23:42:38 job 5 worker 11 done: 18015B result, 7 pending +23:42:38 job 5 worker 12 done: 18556B result, 6 pending +23:42:39 job 5 worker 1 done: 18184B result, 5 pending +23:42:39 job 5 worker 2 done: 19436B result, 4 pending +23:42:40 job 5 worker 3 done: 18359B result, 3 pending +23:42:41 job 5 worker 4 done: 17919B result, 2 pending +23:42:41 job 5 worker 5 done: 17225B result, 1 pending +23:42:41 job 5 worker 7 done: 18901B result, 0 pending +23:42:41 job 5 complete, merging +23:42:41 merge applied: 2671 node results (2671 with singleStat), powerMax.singleStat=1825249.3469207 +23:42:52 job 6 launched: mode=nodePower workers=8 total=4348 buildXML=56461B +23:43:09 job 6 worker 5 done: 26643B result, 7 pending +23:43:11 job 6 worker 6 done: 26614B result, 6 pending +23:43:12 job 6 worker 7 done: 25371B result, 5 pending +23:43:13 job 6 worker 1 done: 26736B result, 4 pending +23:43:13 job 6 worker 2 done: 28473B result, 3 pending +23:43:14 job 6 worker 3 done: 28836B result, 2 pending +23:43:14 job 6 worker 4 done: 26997B result, 1 pending +23:43:15 job 6 worker 8 done: 27151B result, 0 pending +23:43:15 job 6 complete, merging +23:43:15 merge applied: 2671 node results (2671 with singleStat), powerMax.singleStat=1811177.1023715 diff --git a/src/probe_report.txt b/src/probe_report.txt new file mode 100644 index 00000000000..b223315d273 --- /dev/null +++ b/src/probe_report.txt @@ -0,0 +1,37 @@ +Probe: launching worker (build XML: 56056 bytes)... +Probe: completed in 7611 ms (wall) +Probe: buildLoaded = true +Probe: scriptPathArg = C:/Users/Albi/Desktop/PoBMC/src +Probe: singleCalcMs = 13 +Probe: specJSONBytes = 24 +Probe: buildXMLBytes = 56056 +Probe: treeOpenAbsolute = true +Probe: treeOpenRelative = true +Probe: hasIo = true +Probe: hasLoadfile = true +Probe: injectedConPrintf = true +Probe: injectedGetRuntimePath = true +Probe: injectedGetScriptPath = true +Probe: injectedGetUserPath = true +Probe: injectedGetWorkDir = true +Probe: buildName = CalcWorker +Probe: injectedNewFileSearch = false +Probe: jitVersion = LuaJIT 2.1.1753364724 +Probe: luaVersion = Lua 5.1 +Probe: packagePath = .\?.lua;C:\Users\Albi\Desktop\PoBMC\runtime\lua\?.lua;C:\Users\Albi\Desktop\PoBMC\runtime\lua\?\init.lua; +Probe: workDir = C:/Users/Albi/Desktop/PoBMC/src +Probe: injectedInflate = false +Probe: timing.calcMs = 13 +Probe: timing.bootstrapMs = 7592 +Probe: timing.totalMs = 7605 +Probe: timing.initMs = 660.00000000006 +Probe: baseline EnergyShieldRecoveryCap: main 0, worker 0 OK +Probe: baseline LifeUnreserved: main 4590, worker 4590 OK +Probe: baseline Life: main 4634, worker 4634 OK +Probe: baseline CombinedDPS: main 1.78378e+07, worker 1.78378e+07 OK +Probe: baseline Armour: main 31853, worker 31853 OK +Probe: baseline LifeRegenRecovery: main 352.2, worker 352.2 OK +Probe: baseline EnergyShieldRegenRecovery: main 10, worker 10 OK +Probe: baseline Evasion: main 16811, worker 16811 OK +Probe: baseline EnergyShield: main 0, worker 0 OK +Probe: baseline validated OK against main thread