Running with a Predefined Number of Cores

[michael-shaked]

Hi @kherud ,

First of all, thank you for your great project!

I'm trying to run the library with a specific number of cores for parallel processing, but I'm not sure how to set the number of cores to be used by the library. I'd like to configure the library to use a predefined number of cores rather than relying on the default setting.
How can I do it?

Regards,
Michael

[drycoffee]

You can use setThreads in the ModelParameters:

ModelParameters modelParams = new ModelParameters().setThreads(4);
LlamaModel model = new LlamaModel(modelParams);

[bernardladenthin]

I noticed the repository hasn't seen recent activity, so I've taken the liberty of forking it and continuing development at: https://github.com/bernardladenthin/java-llama.cpp

See also #125

[bernardladenthin]

The API for this already exists and is exercised in the downstream fork at
https://github.com/bernardladenthin/java-llama.cpp:
ModelParameters.setThreads(int) and setThreadsBatch(int) map to the
--threads / --threads-batch CLI flags (ModelParameters.java:51,61).