Add idle model unload for mediapipe LLM graphs - #4486
Draft
atobiszei wants to merge 11 commits into
Draft
Conversation
Cherry-picked from idle-models branch (65f2bab). Adds UNLOADED state machine, per-graph idle unload, ActiveInferenceGuard, config/schema/metrics additions, and related tests.
Cherry-picked from idle-models branch (ceb9f4d). Adds ModelGroupManager, group_name config field, CLI idle_unload_timeout_seconds, status codes, and model_group_manager tests.
atobiszei
commented
Aug 28, 2026
| // Urgent reload with existing config (inference-triggered) | ||
| // Urgent reload (inference-triggered wake-up or on-demand load) | ||
| if (!def) | ||
| return StatusCode::MEDIAPIPE_DEFINITION_NOT_LOADED_ANYMORE; |
Collaborator
Author
There was a problem hiding this comment.
requesting load without taskGraphConfig is INTERNAL_ERROR. Check in L150 doesn't make sense anymore as well.
atobiszei
commented
Aug 28, 2026
| /** | ||
| * @brief Model group manager for idle load/unload | ||
| */ | ||
| std::unique_ptr<ModelGroupManager> groupManager_; |
Collaborator
Author
There was a problem hiding this comment.
Rename: groupManager_ -> servableGroupManager
atobiszei
commented
Aug 28, 2026
| return models; | ||
| } | ||
|
|
||
| const std::unordered_map<std::string, ModelConfig>& getServedModelConfigs() const { |
atobiszei
commented
Aug 28, 2026
| return servedModelConfigs; | ||
| } | ||
|
|
||
| ModelGroupManager* getGroupManager() const { |
Collaborator
Author
There was a problem hiding this comment.
We should not expose group manager, and spill such logic to eg kfs_frontend/capi_frontend. We could have additional state that we would treat as READY/LOADED for model readiness, so querying.
What happens when we query model metadata and model is sleeping/unloaded?
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cherry-picked from idle-models branch (65f2bab).