Conversation
HeatmapSubsystem: refactored the AddHeatmapActor to handle null and return early dropping the need to do if/else to only if null. logic remains relatively the same. Also checked a couple of old todo's to see if they are relevant. As they no longer are, the comments have been updated to reflect it correctly
Add video walkthrough to BUILD-AND-RUN docs
Replace embedded YouTube iframe by thumbnail link as GitHub Markdown strips <iframe> tags for security
Add robust macOS screenshot flow and packaging guidance. Implement macOS-specific preflight write probe, FScreenshotRequest delegate handler, async PNG compression/save, timeout handling, and state reset; add BeginDestroy override to unbind the screenshot delegate. Introduce platform-specific Engine.ini files (Mac/Windows) and move renderer MegaLights setting into per-platform configs; add local mac signing setting to DefaultEngine.ini. Update .gitignore to exclude hdf5 share and MobiusCaptures test data. Expand docs/BUILD-AND-RUN.md with macOS packaging entitlements and runtime Files & Folders permission guidance. Will perform a clean clone and test following documentation steps to ensure the process is correct and no steps are missed. Once confirmed will create tutorial video to aid mac users
Update docs/BUILD-AND-RUN.md to clarify Unreal Engine build workflow: recommend opening the .uproject and allowing Unreal to rebuild missing modules automatically, and only generate IDE/Visual Studio project files and manually build the editor if the automatic rebuild fails or IDE integration is required. Replace "Xcode Command Line Tools" with requiring full Xcode selected via xcode-select and add Unreal 5.5 Xcode compatibility details (min 15.2.0, max 16.9.0) plus note that local setup used Xcode 16.4.
Clarify macOS build and packaging guidance in BUILD-AND-RUN.md. Reword Xcode compatibility to state UE5.5 supports Xcode 15.2.0 through 16.9.0 and note the docs were last tested with Xcode 16.4. Add a known macOS limitation: heatmap Gaussian blur depends on Epic's OpenCV plugin and Mac OpenCV libs are not shipped, requiring a custom plugin or code replacement. Improve Unreal build instructions to prefer letting the editor rebuild missing modules and explain IDE integration options (Visual Studio/Xcode) only when needed. Expand macOS packaging notes to describe local entitlement templates, warn that they may not be present in a fresh checkout, instruct developers to run a local packaging pass to generate and review them, and remind contributors not to commit local packaging resources or generated StagedBuilds. Also adjust wording to make clear disabling the App Sandbox does not bypass macOS privacy prompts.
Dev mac update documentation Can confirm this hasn't broken windows packaging operations and earlier tests of Mac appear correct
Move HDF5 entity metadata from the data runnable into AgentDataSubsystem (CachedHdf5Entities) so entity info remains available after runnable teardown. Update SetEntityInfoByIndex and SetEntityRenderingByIndex to read from the cached subsystem array. Release the raw HDF5 sample buffer earlier in Run() to reduce peak memory usage. In MassEntitySpawnSubsystem: - add RegisteredPedestrianTemplateID - explicitly DestroyTemplate on file switch to prevent stale FSimulationFragment references from being retained in the TemplateRegistry - clear SharedSimulationFragment before recreating it - use MoveTemp when creating shared fragments Also cache HDF5 entities during BuildPedestrianMovementFragmentData, reset JSONObject where appropriate, and apply minor cleanup/comment improvements. Note: This change mitigates the current suspected lifetime/memory issue, but it does not complete the larger loading/parsing overhaul. A follow-up refactor is planned once the current issue is resolved to separate format-specific loading from the shared data parsing path. That will allow JSON, HDF5, and future formats to produce a consistent intermediate representation, reducing duplicated logic, limiting format-specific assumptions, and making eventual streaming support significantly easier to implement. AI transparency: Claude assisted with investigation of the suspected issue and potential fixes. ChatGPT was used to polish the wording of this commit message. All code changes and final review decisions remain mine.
Introduce a lightweight memory-tracing helper (MemoryTraceHelper.h) and instrument key places (AgentDataSubsystem, MassEntitySpawnSubsystem, runnable lifecycle) with FMobiusMemSnapshot to diagnose allocation/regression issues (compiled out in Shipping). Unify simulation-entity paths by extracting JSON entity rows into Hdf5Data.Entities and renaming CachedHdf5Entities to CachedEntityData; remove legacy JSON-only fallback code and simplify entity parsing paths to avoid dangling JSONObject lifetimes. Add a ProjectMobiusTests module with automated memory tests (simulation fragment lifecycle, reload cycles, and entity cache lifecycle) and register the tests; update .uproject to include the new tests module and add the module Build.cs. Minor logging and cleanup improvements around runnable/switch/gc points to help track memory during loads and teardown. Memory management is better but still need to address it further. Used Claude to generate text memory logger.
Wraps up the FSimulationFragment heap indirection started in the previous commit. SimulationData is now a TSharedPtr<TMap> so the ~4 GB block can be freed on file switch without waiting for the Mass archetype that permanently holds the fragment struct. Updated every call site that poked the member directly - PedestrianInitializeMOP, PedestrianMovementProcessor, AgentDataSubsystem (JSON + HDF5 paths), MassEntitySpawnSubsystem, FloorStatsWidget. MassEntitySpawnSubsystem now runs a second GC pass after clearing the template, and the processing runnable ends with FMemory::Trim() so pages actually return to the OS. Added a few FMobiusMemSnapshot probes around the Juelich convert and flow-counter reset for delta tracking. Used Claude (AI) to re-audit FlowCounter.cpp / StatisticSubsystem.cpp after the earlier HasAgentBeenCountedInFlowCounter nullptr fix to identify any other potential issues of the same shape. It turned up several - IsAgentLocationInAFlowCounterBand, SendDataToFlowCounter and SendArrayDataToFlowCounter were all missing the IsActorBeingDestroyed guard; the FlashBarrierColor timer captured raw this and was never cleared in EndPlay (now WeakObjectPtr + EndPlay cleanup); and GetHoveredAgentInfoMeshData was quietly MoveTemp-ing its backing member, so a second read per update returned default-constructed data (now a plain copy, matching the Selected equivalent). Added null guards on the pillar meshes and trigger box, and tagged the two GT-only reader paths with comments so a future non-GT caller will trip a visible reminder instead of a race. Deleted dead code the audit surfaced: unused StatisticSubsystem TObjectPtr on AFlowCounter, ThreadSafeNewAgentDataQueue + its FBuckectTempData struct, a stale BucketData copy inside a hot loop, and a block of commented-out AsyncTask scaffolding in SendArrayDataToFlowCounter. Project config: ProjectMobiusTests moved to Editor type with AutomationController added so automation macros load in editor builds. DefaultGame.ini flipped CrashReporter on and ForDistribution off for dev iteration. RightBracket added as an extra console key.
Instrument memory tracing and improve async/thread safety across multiple systems. Key changes: - Add and include Util/MemoryTraceHelper and define LogMobiusMemory for MobiusCore; add FMobiusMemSnapshot probes in FlowCounter, StatisticSubsystem, StatisticActorManagementSubsystem, MassEntitySpawnSubsystem and AgentDataSubsystem to help diagnose memory retention during file switches and lifecycle operations. - HeatmapSubsystem: snapshot Heatmaps into TWeakObjectPtr arrays for safe iteration on worker threads and re-validate actors before use. - RuntimeMeshBuilder: reuse RuntimeDatasmithAnchor (call Reset), guard async work with TWeakObjectPtr, and capture ExistingRunnable by value when dispatching deletion to the game thread. - MobiusIpcClient: marshal OnMessage delegate execution to the game thread using AsyncTask and a weak/shared pointer check to avoid touching UObject state from background thread. - AgentDataSubsystem: add ClearPerFileState to drop per-file caches and drain queues between file switches. - Add MobiusCore dependency to ProjectMobiusTests.Build.cs so tests can access the new utilities. Overall the patch is focused on reducing transient UObject retention, eliminating race conditions in background tasks, and providing runtime memory diagnostics to trace leaks and allocator behavior.
Fixed several significant memory leaks occurring when switching between .udatasmith, .ifc, and .fbx files. These fixes reduce the process-memory delta per switch from ~749 MB to ~303 MB. Core Datasmith & Runtime Fixes Plugged the "Deferred-Purge" Race: Instead of destroying and re-spawning ADatasmithRuntimeActor (which deferred EndPlay and kept ~840 UObjects rooted), we now reuse the actor across loads. Implemented Two-Tick Deferral: Added a two-tick delay after calling Reset() to ensure FSceneImporter::Tick completes its cleanup before the next import starts. A single tick was found to be insufficient due to FTimerManager execution order. Proactive Garbage Collection: Forced a CollectGarbage sweep at the start of the load continuation to prevent the memory footprint from doubling before the engine's natural GC cycle. Actor & Component Cleanup Material & Timer Leaks: Fixed leaks in AFlowCounter (MIDs) and UScreenFacingWorldWidgetComp (Active timers) by explicitly clearing them in EndPlay. Visualizer Optimization: Fixed a monotonic growth bug in HeatmapPixelTextureVisualizer where the index buffer (MeshTriangles) was never emptied. Thread Safety: Switched to TWeakObjectPtr for async mesh builds to prevent crashes or leaks if an actor is destroyed mid-build. Logic & Subsystem Fixes Buffer Drains: Fixed a bug where MeshLoaderRunnable was nulled without being deleted, causing massive leaks in ProceduralMeshComponent buffers. Subsystem Resets: Added ResetForFileSwitch hooks to the Statistics and Widget subsystems to clear agent data and unbind delegates, preventing dangling references to destroyed widgets. Verification Results: Live Refs: Confirmed via FReferenceChainSearch (dropped from 838 to 1). Stability: Confirmed flat object counts for BoxComponent and DeformableQuadComponent across multiple switches. Memory: Residual growth in obj list is now confirmed as PendingKill assets awaiting engine GC, not active leaks. Will assess current blueprint code for any missed object/delegate cleanups AI Usage: Claude was used to identify potential leak sights, reading debug profiling logs and assist with patches a notable patch was moving from an object to a TWeakObjectPtr in HeatmapPixelTextureVisualizer.cpp
Add ReleaseDatasmithSceneResources() and call it from EndPlay to flush rendering commands, detach static mesh components, clear physics meshes, release render resources and wait on fences so heavy GPU/CPU buffers owned by DatasmithRuntime are freed. Move MasterTypeCache to a class-level static (and clear it on EndPlay) so UMaterial* entries don't become stale across PIE sessions; clear DatasmithMaterialsMap and destroy the RuntimeDatasmithAnchor. Also include minor header/formatting cleanups. Additionally add/import updated splash/icon assets and PNG variants, and set ProjectDisplayedTitle="Mobius Viewer" in DefaultGame.ini. UAsset LFS blobs updated accordingly.
Remove dynamic binding to GameInstance (OnPedestrianVectorFileUpdated) in UMassEntitySpawnSubsystem::Deinitialize by checking GetMobiusGameInstance(GetWorld()) and calling RemoveDynamic before Super::Deinitialize to avoid dangling callbacks. Also update Git LFS OIDs/sizes for BP_MobiusPawn.uasset and BP_ProjectMobiusController.uasset (asset blob changes).
Add a TWeakObjectPtr<UProjectMobiusGameInstance> CachedGameInstance to store the game instance when binding OnPedestrianVectorFileChanged. Set the cache in GetScreenshotRequiredSubsystemsAndData when casting succeeds, and use it in EndPlay to RemoveDynamic and clear the cached pointer. This removes the GetWorld()-based lookup and ensures safe unbinding/cleanup of the delegate.
the toggle text would change from on and off, the feature made sense to me at the time but ultimately is confusing and lead to users turning on and off debug features repeatedly as they were unsure what state the toggle meant when on or off
Replace hard thread kills and manual Exit() calls with proper join/wait semantics to avoid Assimp/HDF5 leaks and race conditions (WaitForCompletion in destructors, remove manual Exit()). Drop stale delegate bindings before stopping runnables and bind completion delegates before starting new runnables to avoid late broadcasts reaching freed/stale state. Reset runnable pointers and data-load flags (bIsDataLoaded) when switching files to prevent using previous-run data. Make async/game-thread interactions safer: use TWeakObjectPtr in ParallelFor->AsyncTask closures for HeatmapSubsystem spawns, and guard CreateHeatmap calls behind weak-pointer validity checks. Fix widget lifecycle for AgentInfoDisplay: move slate resource release/cleanup to ReleaseSlateResources/RebuildWidget, rebind delegates in RebuildWidget, and remove BeginDestroy override. Add null/IsValid checks and early returns in various MassAI processors (heatmap/time-dilation/niagara) to prevent crashes when subsystems or actors are missing. Also minor whitespace/license formatting cleanup. Overall these changes reduce crashes, memory leaks, and race conditions when stopping/starting background runnables and interacting with game-thread subsystems.
Discovered the edited resolutions sizes did not transfer over correctly, have since corrected this and sharpened the image with the assist of ChatGPT to sharpen the image
Refactor heatmap and runtime mesh building to emit geometry in multiple sections and pump those sections across frames to avoid large game-thread hitches. Adds tile-based mesh construction, per-tile/per-section material assignment, and a ticked EmitNextTileSection/EmitNextChunkSection flow with FinalizeTileEmit/FinalizeMeshEmit to stagger CreateMeshSection calls. Introduces SplitSubmeshByTriCap and submesh chunking to cap tris per section, enables async cooking for procedural meshes, defers Datasmith heatmap broadcast until components are ready, and adds detailed timing/logging via UMobiusCustomLoggerSubsystem. Also hardens checks for multi-section meshes, cleans up ticker handles in EndPlay, and preserves legacy single-section fallback when batching is disabled.
Add adaptive door spawn strategy and safer async/memory handling across the project. Key changes: - RuntimeMeshBuilder: cancel in-flight FBX tickers and async runnables during mesh resets, guard late async callbacks, abort stale spawner entries, and call DecideAndExecuteSpawnStrategy after Datasmith load. - New DecideAndExecuteSpawnStrategy: consults PerformanceUtilSubsystem (instant + smoothed FPS) and physical memory to choose between immediate flush or batched spawning; reports a user warning and logs when batched mode is used. Added SpawnFPSThreshold and MemFreeThreshold tunables. - FlowCounterSpawnerComponent: add FlushRemainingSpawns() to bypass frame throttles and AbortSpawning() to discard queued doors; wire AbortSpawning() call from mesh reset. - PerformanceUtilSubsystem: add SmoothedFPS and accessors, compute smoothed FPS in CalculateCurrentFPS, and expose getters for use by spawn logic. - MobiusCustomLoggerSubsystem: adjust FlushToDisk ordering and emit a warning log when logging is disabled; ensure flush delegate broadcasting remains. - FlowCounterWidget: make geometry handling robust (use MyGeometry from NativeTick), move style update into an internal function, improve sizing/measurement and early-exit checks. - Mass AI / Agent data: add timing logs around runnable creation and deserialization, remove unnecessary sleeps, add optional file-switch diagnostics, improve GC/trim sequencing, ensure SimulationData backing map is emptied before resetting shared pointer, and stabilize spawned handle cleanup. - MassEntitySpawnSubsystem: opt-in diagnostics and safer teardown flow during file switches; minor logging and housekeeping improvements. - Agent data runnable: small logging/timing instrumentation for performance debugging. - UI/Error window: increase default size. - Tests: add FSimulationFragmentSharedBackingClearTest to verify clearing the shared backing map behavior. These changes aim to prevent stale async work from re-adding old geometry, reduce memory retention during data/file switches, and make door-flow spawning adaptive to runtime performance and memory conditions while providing diagnostic hooks for investigation.
removed trim, from the thread data importer, calling trim on a non game thread is not allowed and will trigger assertion fails
Promote EditorScriptingUtilities to the public deps of MobiusEditor and remove the MobiusWidgets module from ProjectMobius. Remove direct includes for HeatmapVisualization and TimeDilation from AgentDataSubsystem, and remove MobiusWidgetSubsystem usage and reset in MassEntitySpawnSubsystem (still reset StatisticSubsystem). Also apply minor formatting/whitespace cleanups in build files. These changes reduce coupling to widget/heatmap subsystems and adjust dependency visibility.
Remove the EditorScriptingUtilities dependency and replace usages of UEditorAssetLibrary with asset-registry / ObjectTools / AssetTools helpers. MobiusEditor.Build.cs: drop "EditorScriptingUtilities" from public deps. GenerateDatasmithMaterialsCommandlet.cpp: add includes (AssetToolsModule, IAssetTools, ObjectTools), implement AssetExists, DeleteAssetAtPath, DuplicateAssetToPath helpers, and replace all UEditorAssetLibrary checks/Deletes/Duplicate calls with the new helpers. MobiusEditorModule.cpp: replace UEditorAssetLibrary existence checks with AssetRegistry::GetAssetsByPackageName checks. This keeps asset operations behavior while removing the EditorScriptingUtilities plugin dependency.
Hdf5 memory cleanup All validation tests have been completed successfully. Additional testing on legacy hardware is still recommended to ensure acceptable performance on older systems, particularly devices running Windows 10 and older macOS versions. Due to limited access to such hardware, broader coverage will rely on community testing and feedback. The primary goals of this PR have been achieved, with several additional issues resolved along the way, resulting in improved stability and performance. Based on current results, the codebase is in a suitable state for merging. With this work complete, development can now proceed to the next phase: restructuring the data import pipeline and integrating B-Risk. Any further issues identified should be raised as tickets and addressed as part of upcoming feature development.
Prevent the TODO scanner from running on pushes to the repository's default branch. Replaced push.branches with push.branches-ignore: - main and added a job-level guard (if: github.ref_name != github.event.repository.default_branch) so the workflow won't execute for default-branch pushes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The dev branch requires to be updated with the latest changes from main for next development cycle