Models hotswap

[staryj]

Hello,

Does LocalAI provide an ability to swap models automatically without manual unload of the running one?
If yes, how to setup it? If no, do you plan to add such feature?

For example, I have Model1 loaded and then I've called Model2. The available resources required by Model2 are not enough, so before running it Model1 will be stopped.
Similar behavior is implemented in https://github.com/mostlygeek/llama-swap/ and it's very useful to change the model on the fly, especially when several users access them (agents and family)

Thanks in advance

[lqp]

AFAIK you can set "Max Active Backends" setting option to 1.

But then you lose the ability to run two smal models (f.e. MoE Gemma and quantized Anima) at the same time.

[staryj]

Thanks! Limitation of active backends works. But yes, in such setup I won't be able to automatically load 2+ small models

[lqp]

Alternatively, you can completely disable autofit ("gpu_layers: -1 ") for all models you use in combination, manually fit them in VRAM. This way hotswap works too, but it much more hassle.