It would be useful to know the progress of loading an LLM. This would be displayed to a user with a progress bar or similar.
The API would look something like this
public LlamaModel(ModelParameters parameters, Consumer<LLamaLoadProgress>)
Where LLamaLoadProgress is something like
class LLMLoadProgress{
/**
* A value from 0 to 1 representing the overall load progress of the file given by [fileName].
* If you don't want downloads of small files to skew this value, check for [isWeightsProgress], as that file is usually the one we care about for tracking its loading progress, because it is extremely large.
*/
double fraction
/**
* Additional information about the loading progress
*/
String text
/**
* Whether the progress is of downloading the model (true) or loading it from disk (false)
*/
boolean isDownloading
/**
* Whether this represents the progress of downloading the extremely large weights file (true) or some auxiliary file like the tokenizer config.
*/
boolean isWeightsProgress
/**
* The file that its load has progressed by amount indicated by this [LLMLoadProgress]. If the file is the weights, [isWeightsProgress] will be true, and for other files [isWeightsProgress] will be false.
*/
@Nullable String fileName
/**
* The amount of bytes loaded from the file given by [fileName].
*/
long bytesLoaded
/**
* The byte size of the file given by [fileName]. Will be null as long as the file's size is not known.
*/
@Nullable Long fileTotalByteSize
}
It would be useful to know the progress of loading an LLM. This would be displayed to a user with a progress bar or similar.
The API would look something like this
Where LLamaLoadProgress is something like