Benchmark replay by Dreamsorcerer · Pull Request #3347 · dimensionalOS/dimos

Dreamsorcerer · 2026-08-04T12:52:49Z

No description provided.

mintlify · 2026-08-04T12:52:53Z

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated (UTC)
dimensional	🟢 Ready	View Preview	Aug 4, 2026, 12:53 PM

💡 Tip: Enable Workflows to automatically generate PRs for you.

greptile-apps · 2026-08-04T12:56:08Z

Greptile Summary

The PR adds configurable replay speed and windowing and introduces a CodSpeed benchmark for draining the Go2 replay pipeline.

Forwards replay speed, seek, and duration from global configuration into ReplayConnection.
Adds a bounded replay benchmark with stream-delivery validity floors and lifecycle cleanup.
Adds a self-hosted CodSpeed workflow and the corresponding test dependency.
Documents the new replay CLI options.

Confidence Score: 5/5

The PR appears safe to merge based on the follow-up evidence available.

No blocking failure remains within the eligible follow-up review scope.

Important Files Changed

Filename	Overview
.github/workflows/codspeed.yml	Adds the self-hosted, containerized CodSpeed workflow with replay-data preparation and wall-time benchmark execution.
dimos/core/global_config.py	Adds process-wide replay speed, seek, and duration settings exposed through the dynamic CLI configuration.
dimos/robot/unitree/go2/connection.py	Forwards replay timing and window settings into the memory replay implementation.
dimos/robot/unitree/go2/test_replay_benchmark.py	Adds a bounded end-to-end benchmark that validates delivery floors and waits for stream quiescence.
pyproject.toml	Adds the pinned pytest-codspeed benchmark dependency.
uv.lock	Locks pytest-codspeed 5.0.3 and its resolved artifacts.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  CLI["Replay CLI options"] --> Config["GlobalConfig"]
  Config --> Connection["ReplayConnection"]
  Connection --> Store["SQLite replay store"]
  Store --> Streams["Go2 replay streams"]
  Streams --> Coordinator["Unitree Go2 blueprint"]
  Coordinator --> Watchers["Benchmark subscribers"]
  Watchers --> Gate["Count floors + quiescence"]
  Gate --> CodSpeed["Wall-time result"]

_{Reviews (9): Last reviewed commit: "Fix" | Re-trigger Greptile}

codecov · 2026-08-04T13:01:31Z

❌ 1 Tests Failed:

Tests completed	Failed	Passed	Skipped
4210	1	4209	69

View the top 1 failed test(s) by shortest run time

dimos.robot.unitree.go2.test_replay_benchmark::test_go2_replay_drain_walltime

Stack Traces | 83.4s run time

+ Exception Group Traceback (most recent call last):
  |   File ".../dimos/dimos/.venv/lib/python3.12........./site-packages/_pytest/runner.py", line 341, in from_call
  |     result: TResult | None = func()
  |                              ^^^^^^
  |   File ".../dimos/dimos/.venv/lib/python3.12........./site-packages/_pytest/runner.py", line 242, in <lambda>
  |     lambda: runtest_hook(item=item, **kwds), when=when, reraise=reraise
  |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File ".../dimos/dimos/.venv/lib/python3.12....../site-packages/pluggy/_hooks.py", line 512, in __call__
  |     return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File ".../dimos/dimos/.venv/lib/python3.12....../site-packages/pluggy/_manager.py", line 120, in _hookexec
  |     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File ".../dimos/dimos/.venv/lib/python3.12.............................................................../site-packages/pluggy/_callers.py", line 167, in _multicall
  |     raise exception
  |   File ".../dimos/dimos/.venv/lib/python3.12.............................................................../site-packages/pluggy/_callers.py", line 139, in _multicall
  |     teardown.throw(exception)
  |   File ".../dimos/dimos/.venv/lib/python3.12....../site-packages/_pytest/threadexception.py", line 92, in pytest_runtest_call
  |     yield from thread_exception_runtest_hook()
  |   File ".../dimos/dimos/.venv/lib/python3.12....../site-packages/_pytest/threadexception.py", line 68, in thread_exception_runtest_hook
  |     yield
  |   File ".../dimos/dimos/.venv/lib/python3.12.............................................................../site-packages/pluggy/_callers.py", line 139, in _multicall
  |     teardown.throw(exception)
  |   File ".../dimos/dimos/.venv/lib/python3.12....../site-packages/_pytest/unraisableexception.py", line 95, in pytest_runtest_call
  |     yield from unraisable_exception_runtest_hook()
  |   File ".../dimos/dimos/.venv/lib/python3.12....../site-packages/_pytest/unraisableexception.py", line 70, in unraisable_exception_runtest_hook
  |     yield
  |   File ".../dimos/dimos/.venv/lib/python3.12.............................................................../site-packages/pluggy/_callers.py", line 139, in _multicall
  |     teardown.throw(exception)
  |   File ".../dimos/dimos/.venv/lib/python3.12....../site-packages/_pytest/logging.py", line 846, in pytest_runtest_call
  |     yield from self._runtest_for(item, "call")
  |   File ".../dimos/dimos/.venv/lib/python3.12....../site-packages/_pytest/logging.py", line 829, in _runtest_for
  |     yield
  |   File ".../dimos/dimos/.venv/lib/python3.12.............................................................../site-packages/pluggy/_callers.py", line 139, in _multicall
  |     teardown.throw(exception)
  |   File ".../dimos/dimos/.venv/lib/python3.12.............................................................../site-packages/pluggy/_callers.py", line 53, in run_old_style_hookwrapper
  |     return result.get_result()
  |            ^^^^^^^^^^^^^^^^^^^
  |   File ".../dimos/dimos/.venv/lib/python3.12............/site-packages/pluggy/_result.py", line 103, in get_result
  |     raise exc.with_traceback(tb)
  |   File ".../dimos/dimos/.venv/lib/python3.12.............................................................../site-packages/pluggy/_callers.py", line 38, in run_old_style_hookwrapper
  |     res = yield
  |           ^^^^^
  |   File ".../dimos/dimos/.venv/lib/python3.12.............................................................../site-packages/pluggy/_callers.py", line 139, in _multicall
  |     teardown.throw(exception)
  |   File ".../dimos/dimos/.venv/lib/python3.12.../site-packages/_pytest/capture.py", line 898, in pytest_runtest_call
  |     return (yield)
  |             ^^^^^
  |   File ".../dimos/dimos/.venv/lib/python3.12.............................................................../site-packages/pluggy/_callers.py", line 139, in _multicall
  |     teardown.throw(exception)
  |   File ".../dimos/dimos/.venv/lib/python3.12.............................................................../site-packages/pluggy/_callers.py", line 53, in run_old_style_hookwrapper
  |     return result.get_result()
  |            ^^^^^^^^^^^^^^^^^^^
  |   File ".../dimos/dimos/.venv/lib/python3.12............/site-packages/pluggy/_result.py", line 103, in get_result
  |     raise exc.with_traceback(tb)
  |   File ".../dimos/dimos/.venv/lib/python3.12.............................................................../site-packages/pluggy/_callers.py", line 38, in run_old_style_hookwrapper
  |     res = yield
  |           ^^^^^
  |   File ".../dimos/dimos/.venv/lib/python3.12.............................................................../site-packages/pluggy/_callers.py", line 139, in _multicall
  |     teardown.throw(exception)
  |   File ".../dimos/dimos/.venv/lib/python3.12.............................................................../site-packages/pluggy/_callers.py", line 53, in run_old_style_hookwrapper
  |     return result.get_result()
  |            ^^^^^^^^^^^^^^^^^^^
  |   File ".../dimos/dimos/.venv/lib/python3.12............/site-packages/pluggy/_result.py", line 103, in get_result
  |     raise exc.with_traceback(tb)
  |   File ".../dimos/dimos/.venv/lib/python3.12.............................................................../site-packages/pluggy/_callers.py", line 38, in run_old_style_hookwrapper
  |     res = yield
  |           ^^^^^
  |   File ".../dimos/dimos/.venv/lib/python3.12.............................................................../site-packages/pluggy/_callers.py", line 139, in _multicall
  |     teardown.throw(exception)
  |   File ".../dimos/dimos/.venv/lib/python3.12.../site-packages/_pytest/skipping.py", line 257, in pytest_runtest_call
  |     return (yield)
  |             ^^^^^
  |   File ".../dimos/dimos/.venv/lib/python3.12.............................................................../site-packages/pluggy/_callers.py", line 121, in _multicall
  |     res = hook_impl.function(*args)
  |           ^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File ".../dimos/dimos/.venv/lib/python3.12........./site-packages/_pytest/runner.py", line 174, in pytest_runtest_call
  |     item.runtest()
  |   File ".../dimos/dimos/.venv/lib/python3.12....../site-packages/_pytest/python.py", line 1627, in runtest
  |     self.ihook.pytest_pyfunc_call(pyfuncitem=self)
  |   File ".../dimos/dimos/.venv/lib/python3.12....../site-packages/pluggy/_hooks.py", line 512, in __call__
  |     return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File ".../dimos/dimos/.venv/lib/python3.12....../site-packages/pluggy/_manager.py", line 120, in _hookexec
  |     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File ".../dimos/dimos/.venv/lib/python3.12.............................................................../site-packages/pluggy/_callers.py", line 167, in _multicall
  |     raise exception
  |   File ".../dimos/dimos/.venv/lib/python3.12.............................................................../site-packages/pluggy/_callers.py", line 139, in _multicall
  |     teardown.throw(exception)
  |   File ".../dimos/dimos/.venv/lib/python3.12.............................................................../site-packages/pluggy/_callers.py", line 53, in run_old_style_hookwrapper
  |     return result.get_result()
  |            ^^^^^^^^^^^^^^^^^^^
  |   File ".../dimos/dimos/.venv/lib/python3.12............/site-packages/pluggy/_result.py", line 103, in get_result
  |     raise exc.with_traceback(tb)
  |   File ".../dimos/dimos/.venv/lib/python3.12.............................................................../site-packages/pluggy/_callers.py", line 38, in run_old_style_hookwrapper
  |     res = yield
  |           ^^^^^
  |   File ".../dimos/dimos/.venv/lib/python3.12.............................................................../site-packages/pluggy/_callers.py", line 121, in _multicall
  |     res = hook_impl.function(*args)
  |           ^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File ".../dimos/dimos/.venv/lib/python3.12....../site-packages/_pytest/python.py", line 159, in pytest_pyfunc_call
  |     result = testfunction(**testargs)
  |              ^^^^^^^^^^^^^^^^^^^^^^^^
  |   File ".../unitree/go2/test_replay_benchmark.py", line 185, in test_go2_replay_drain_walltime
  |     benchmark.pedantic(build_and_drain, teardown=teardown, rounds=1, warmup_rounds=0)
  |   File ".../dimos/dimos/.venv/lib/python3.12.../site-packages/pytest_codspeed/plugin.py", line 396, in pedantic
  |     result = target(*args, **kwargs)
  |              ^^^^^^^^^^^^^^^^^^^^^^^
  |   File ".../unitree/go2/test_replay_benchmark.py", line 159, in build_and_drain
  |     state["coordinator"] = ModuleCoordinator.build(blueprint)
  |                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File ".../core/coordination/module_coordinator.py", line 368, in build
  |     coordinator.start_all_modules()
  |   File ".../core/coordination/module_coordinator.py", line 257, in start_all_modules
  |     safe_thread_map(modules, lambda m: m.start())
  |   File ".../dimos/utils/safe_thread_map.py", line 84, in safe_thread_map
  |     raise ExceptionGroup("safe_thread_map failed", errors)
  | ExceptionGroup: safe_thread_map failed (1 sub-exception)
  +-+---------------- 1 ----------------
    | dimos.protocol.rpc.rpc_utils.RemoteError: [Remote builtins.RuntimeError] Unable to locate turbojpeg library automatically. You may specify the turbojpeg library path manually.
    | e.g. jpeg = TurboJPEG(lib_path)
    | 
    | Remote traceback:
    | Traceback (most recent call last):
    |   File ".../protocol/rpc/pubsubrpc.py", line 280, in execute_and_respond
    |     response = f(*args[0], **args[1])
    |                ^^^^^^^^^^^^^^^^^^^^^^
    |   File ".../protocol/rpc/spec.py", line 116, in override_f
    |     return getattr(module, fname)(*args, **kwargs)
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File ".../unitree/go2/connection.py", line 348, in start
    |     self.register_disposable(self.connection.video_stream().subscribe(onimage))
    |                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File ".../dimos/dimos/.venv/lib/python3.12.../reactivex/observable/observable.py", line 142, in subscribe
    |     current_thread_scheduler.schedule(set_disposable)
    |   File ".../dimos/dimos/.venv/lib/python3.12.../reactivex/scheduler/trampolinescheduler.py", line 49, in schedule
    |     return self.schedule_absolute(self.now, action, state=state)
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File ".../dimos/dimos/.venv/lib/python3.12.../reactivex/scheduler/trampolinescheduler.py", line 95, in schedule_absolute
    |     self.get_trampoline().run(item)
    |   File ".../dimos/dimos/.venv/lib/python3.12.../reactivex/scheduler/trampoline.py", line 30, in run
    |     self._run()
    |   File ".../dimos/dimos/.venv/lib/python3.12.../reactivex/scheduler/trampoline.py", line 51, in _run
    |     item.invoke()
    |   File ".../dimos/dimos/.venv/lib/python3.12.../reactivex/scheduler/scheduleditem.py", line 25, in invoke
    |     ret = self.scheduler.invoke_action(self.action, state=self.state)
    |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File ".../dimos/dimos/.venv/lib/python3.12.../reactivex/scheduler/scheduler.py", line 105, in invoke_action
    |     ret = action(self, state)
    |           ^^^^^^^^^^^^^^^^^^^
    |   File ".../dimos/dimos/.venv/lib/python3.12.../reactivex/observable/observable.py", line 127, in set_disposable
    |     if not auto_detach_observer.fail(ex):
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File ".../dimos/dimos/.venv/lib/python3.12.../reactivex/observer/autodetachobserver.py", line 64, in fail
    |     self._on_error(exn)
    |   File ".../dimos/dimos/.venv/lib/python3.12.../reactivex/internal/basic.py", line 34, in default_error
    |     raise err
    |   File ".../dimos/dimos/.venv/lib/python3.12.../reactivex/observable/observable.py", line 125, in set_disposable
    |     subscriber = self._subscribe_core(auto_detach_observer, scheduler)
    |                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File ".../dimos/dimos/.venv/lib/python3.12.../reactivex/observable/observable.py", line 49, in _subscribe_core
    |     return self._subscribe(observer, scheduler) if self._subscribe else Disposable()
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File ".../dimos/memory2/replay.py", line 250, in subscribe
    |     first_ts, first_data = next(iterator)
    |                            ^^^^^^^^^^^^^^
    |   File ".../dimos/memory2/replay.py", line 243, in make_iterator
    |     yield (obs.ts, decode(obs))
    |                    ^^^^^^^^^^^
    |   File ".../dimos/memory2/replay.py", line 136, in _decode
    |     data = obs.data
    |            ^^^^^^^^
    |   File ".../memory2/type/observation.py", line 181, in data
    |     loaded = self._loader()
    |              ^^^^^^^^^^^^^^
    |   File ".../dimos/memory2/backend.py", line 91, in loader
    |     return codec.decode(raw)
    |            ^^^^^^^^^^^^^^^^^
    |   File ".../memory2/codecs/jpeg.py", line 39, in decode
    |     return Image.lcm_jpeg_decode(data)
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File ".../msgs/sensor_msgs/Image.py", line 604, in lcm_jpeg_decode
    |     jpeg = TurboJPEG()
    |            ^^^^^^^^^^^
    |   File ".../dimos/dimos/.venv/lib/python3.12/site-packages/turbojpeg.py", line 301, in __init__
    |     self.__find_turbojpeg() if lib_path is None else lib_path)
    |     ^^^^^^^^^^^^^^^^^^^^^^^
    |   File ".../dimos/dimos/.venv/lib/python3.12/site-packages/turbojpeg.py", line 996, in __find_turbojpeg
    |     raise RuntimeError(
    | RuntimeError: Unable to locate turbojpeg library automatically. You may specify the turbojpeg library path manually.
    | e.g. jpeg = TurboJPEG(lib_path)
    | 
    | 
    | The above exception was the direct cause of the following exception:
    | 
    | Traceback (most recent call last):
    |   File ".../dimos/utils/safe_thread_map.py", line 68, in safe_thread_map
    |     outcomes[idx] = fut.result()
    |                     ^^^^^^^^^^^^
    |   File "........./github/home/.local.../uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12....../concurrent/futures/_base.py", line 449, in result
    |     return self.__get_result()
    |            ^^^^^^^^^^^^^^^^^^^
    |   File "........./github/home/.local.../uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12....../concurrent/futures/_base.py", line 401, in __get_result
    |     raise self._exception
    |   File "........./github/home/.local.../uv/python/cpython-3.12.12-linux-x86_64-gnu/lib/python3.12.../concurrent/futures/thread.py", line 59, in run
    |     result = self.fn(*self.args, **self.kwargs)
    |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File ".../core/coordination/module_coordinator.py", line 257, in <lambda>
    |     safe_thread_map(modules, lambda m: m.start())
    |                                        ^^^^^^^^^
    |   File ".../dimos/core/rpc_client.py", line 93, in __call__
    |     result, unsub_fn = self._rpc.call_sync(
    |                        ^^^^^^^^^^^^^^^^^^^^
    |   File ".../protocol/rpc/spec.py", line 88, in call_sync
    |     raise result
    | RuntimeError: Unable to locate turbojpeg library automatically. You may specify the turbojpeg library path manually.
    | e.g. jpeg = TurboJPEG(lib_path)
    +------------------------------------

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}

… into sam/benchmark-replay

codspeed-hq · 2026-08-10T16:24:45Z

Congrats! CodSpeed is installed 🎉

🆕 1 new benchmark was detected.

You will start to see performance impacts in the reports once the benchmarks are run from your default branch.

⚠️

Unknown Walltime execution environment detected

Using the Walltime instrument on standard Hosted Runners will lead to inconsistent data.

For the most accurate results, we recommend using CodSpeed Macro Runners: bare-metal machines fine-tuned for performance measurement consistency.

Detected benchmark

test_go2_replay_drain_walltime: 39.5 s

Benchmark replay

1b8c639

Dreamsorcerer requested review from leshy, mustafab0, paul-nechifor and spomichter as code owners August 4, 2026 12:52

mintlify Bot deployed to staging - docs August 4, 2026 12:53 View deployment

Merge branch 'main' into sam/benchmark-replay

77c8f6b

mintlify Bot deployed to staging - docs August 4, 2026 14:04 View deployment

Dreamsorcerer added 2 commits August 4, 2026 17:06

Fix

badf42c

Merge branch 'sam/benchmark-replay' of github.com:dimensionalOS/dimos…

c0079c7

… into sam/benchmark-replay

mintlify Bot deployed to staging - docs August 4, 2026 16:09 View deployment

Dreamsorcerer added 7 commits August 4, 2026 20:11

Fix

11a09b1

Fix

82a52d1

Fix

92ca3fa

Fix

8c41c87

Fix

8189b7b

Fix

628aeb6

Fix

ed607ed

Dreamsorcerer had a problem deploying to cachix August 6, 2026 17:38 — with GitHub Actions Error

Merge branch 'main' into sam/benchmark-replay

439c2e6

mintlify Bot deployed to staging - docs August 7, 2026 15:14 View deployment

Dreamsorcerer added 4 commits August 7, 2026 16:31

Fix

3ef4edb

Update codspeed.yml

5c03ee6

Fix

d63aafb

Merge branch 'sam/benchmark-replay' of github.com:dimensionalOS/dimos…

c0eeb39

… into sam/benchmark-replay

mintlify Bot deployed to staging - docs August 10, 2026 13:34 View deployment

Fix

227bd81

Dreamsorcerer added 2 commits August 10, 2026 16:18

Fix

fec1a35

Fix

8fcddca

Navigation Menu

Uh oh!

Benchmark replay - #3347

Benchmark replay#3347
Dreamsorcerer wants to merge 19 commits into
mainfrom
sam/benchmark-replay

Dreamsorcerer commented Aug 4, 2026

Uh oh!

mintlify Bot commented Aug 4, 2026 •

edited

Loading

Uh oh!

greptile-apps Bot commented Aug 4, 2026 •

edited

Loading

Uh oh!

codecov Bot commented Aug 4, 2026 •

edited

Loading

Uh oh!

codspeed-hq Bot commented Aug 10, 2026

Detected benchmark

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

Dreamsorcerer commented Aug 4, 2026

Uh oh!

mintlify Bot commented Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

greptile-apps Bot commented Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 5/5

Important Files Changed

Flowchart

Uh oh!

codecov Bot commented Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

❌ 1 Tests Failed:

Uh oh!

codspeed-hq Bot commented Aug 10, 2026

Congrats! CodSpeed is installed 🎉

Detected benchmark

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

mintlify Bot commented Aug 4, 2026 •

edited

Loading

greptile-apps Bot commented Aug 4, 2026 •

edited

Loading

codecov Bot commented Aug 4, 2026 •

edited

Loading