Idle models - #4465
Draft
dtrawins wants to merge 12 commits into
Draft
Conversation
Adds an opt-in idle timeout that unloads an LLM graph's heavy resources (freeing GPU/CPU memory) after a period with no inference, and lazily reloads on the next request, so the GPU can be shared with other workloads. Follows llama.cpp's --sleep-idle-seconds model. - Config: idle_unload_timeout_seconds on the mediapipe config entry (0 = disabled, default). Added to the JSON schema. Phase-1 scope: restricted to LLM continuous-batching graphs (HttpLLMCalculator); rejected for graphs with Python nodes or non-LLM calculators at config validation. - State machine: new UNLOADED state + UnloadEvent. AVAILABLE->UNLOADED on idle; UNLOADED wakes via the existing reload path. isAvailable() is false for UNLOADED, convertToModelStatus() maps it to AVAILABLE so health/ readiness still see the servable (it auto-reloads). - Concurrency: a per-definition recursive lifecycleMtx serializes reload/retire/unload/wakeUp so the watcher and config threads never race on graph state or side packets. unload() is non-blocking, tears down only after confirming the AVAILABLE->UNLOADED transition, and skips while requests or inferences are in flight. The idle timeout is cached in an atomic for lock-free watcher reads. - In-flight guard: an RAII ActiveInferenceGuard (held for the executor's lifetime) keeps a shared_ptr inference counter so a generation that outlives the idle timeout is never unloaded mid-stream; completing an inference refreshes the activity timestamp. lastActivityTimeNs and the counter are shared_ptr so they outlive the definition if an executor is still running during retire. - Wake-up failure is retryable: if the lazy reload fails, the graph reverts to UNLOADED (not a wedged failed state) so the next request re-attempts the wake and self-heals once the underlying issue is resolved; the current request gets a clean error. - Metrics: ovms_graph_loaded gauge (1 loaded / 0 unloaded) per graph. - Composes with --cache_dir so wake-up is a cache import, not a recompile. Tested: state-machine + schema unit tests; in-flight-guard and wake-failure unit tests; functional unload/reload/idle-reset/disabled-default/concurrency with a real model; and end-to-end on an Intel Arc GPU (idle unload frees resources, long generations are not unloaded mid-stream, wake-up reloads and serves, soak shows no leak, failed wake self-heals). No regressions vs main. Implements #4141 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
atobiszei
reviewed
Aug 21, 2026
Comment on lines
+1633
to
+1640
| if (groupManager_ && groupManager_->isEnabled()) { | ||
| auto status = groupManager_->ensureGroupLoaded(modelName, const_cast<ModelManager&>(*this)); | ||
| if (!status.ok()) { | ||
| SPDLOG_ERROR("Failed to load group for model '{}': {}", modelName, status.string()); | ||
| return status; | ||
| } | ||
| groupManager_->recordActivity(); | ||
| } |
Collaborator
There was a problem hiding this comment.
A lot of code in OVMS depend on a assumption that model loading is done in single thread in model manager either during startup or in watcher loop. I think adding model loading in getModelInstance & createPipeline is just unsafe without adding a lot of tests concurrency tests which is unrealistic.
I think what we should do is:
- Make watcher loop handle loading tasks one by one instead of just calling loadConfig().
- Make createPipeline & getModelInstance create and push task for watcher loop as the next task to process.
With that approach we can minimize tests required to cover this implementation.
atobiszei
reviewed
Aug 24, 2026
|
|
||
| // On-demand group loading for idle model management | ||
| if (groupManager_ && groupManager_->isEnabled()) { | ||
| auto status = groupManager_->ensureGroupLoaded(modelName, const_cast<ModelManager&>(*this)); |
Collaborator
There was a problem hiding this comment.
this method is const for a reason (const_cast)
atobiszei
reviewed
Aug 28, 2026
Comment on lines
+167
to
+175
| // In idle management mode, report configured models as ready even if no instance is loaded | ||
| auto* groupMgr = manager.getGroupManager(); | ||
| if (groupMgr && groupMgr->isEnabled()) { | ||
| std::string group = groupMgr->getGroupForServable(name); | ||
| if (!group.empty()) { | ||
| response->set_ready(true); | ||
| return StatusCode::OK; | ||
| } | ||
| } |
Collaborator
There was a problem hiding this comment.
KFS/CAPI should not know about this feature. We shoudl just convert status. So possibly we need some lower level status translation, or additional ModelStatus code that would translate as READY.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🛠 Summary
This is a follow up to the PR #4332
with extension of the feature according to #4456
🧪 Checklist
``