Rpc models inference

[blenderman94]

Hello. i ben trying out rpc and its fun. but i dont seems to be able to control or at least i dont seen in the network that anything about how to control the split ratio. also semingly no doucemntary about . my problem that while it works it least half of my gpu empty.

[LostRuins]

You control the split ratio with --tensor_split, it works exactly the same way as adding a new GPU. Think of it as a virtual GPU device.

[blenderman94]

could be. but it seems to be igored like most of the flags if your client is cpu and your server is a gpu.

[LostRuins]

It should work, you can set tensor split even if your client is CPU mode

[blenderman94]

by suprise it wont work. i see that kccp dials the server and ut says its connected. yet we se even an attempt to load it there. ./kccp --model /mymodel.gguf --tensor_split 0.3 0.7 --rpcmode connect --rpctargets xxx.xxx.xxx.xxx:5551 --contextsize somenumber. could you please provide some hint on how to use it.

here is piece of a log from my terminal with some details sanitized

Use RPC server: xxx.xxx.xxx.xxx:5551
llama_model_loader: loaded meta data with 39 key-value pairs and 866 tensors from /mymodel.gguf (version GGUF V3 (latest))
print_info: file format = GGUF V3 (latest)
print_info: file type = (guessed) unknown, may not work
print_info: file size = 15.65 GiB (4.92 BPW)
llama_prepare_model_devices: using device RPC0 (xxx.xxx.xxx.xxx:5551) (unknown id) - 7592 MiB free
init_tokenizer: initializing tokenizer for type 2
.... many logs
load_tensors: offloading 0 repeating layers to GPU
load_tensors: offloaded 0/66 layers to GPU
load_tensors: CPU model buffer size = 4935.19 MiB
load_tensors: CPU_REPACK model buffer size = 10835.16 MiB
...................................................................................Kilőve

exact scenario.
1 desktop pc with a gtx1080 (8gib vram has also 32gib cpu ram but ddr3 no avx2) as server launched with ./kccp --usecuda --rpmcmode host
2 laptop (16gib cpu ram on ddr4 with avx2) /kccp --model /mymodel.gguf --tensor_split 0.3 0.7 --rpcmode connect --rpctargets xxx.xxx.xxx.xxx:5551 --contextsize somenumber.

if i use th gui and dont touch splitting (then it loads but the proprtions are screwed)

[LostRuins]

btw the GUI can set tensor split too, in the hardware tab.

But to clarify, do you have 1 GPU or multiple GPU? Cause tensor split only works properly when there is more than 1 GPU. if you have a single GPU, what you should do is set --gpulayers to determine how much to offload instead. So, remove --tensor_split and control it with --gpulayers instead

[blenderman94]

that ok but its hardly gives any sarisfactory answer like if you have a couple dozen pc s in rpc ? because half of my devices are android and they have no gui. the peoblemas is soon as you use more then 2 devices handling splitting without kccp and likely llama itself is starting to hard to control. if you swap device gpu pc first then laptop , maybe your vr headset and so on.

[LostRuins]

I added a small update to the rolling build, now I think if you have multiple rpc targets connected, tensor_split should work too.