llama-kt development

[Doomsdayrs]

Greetings!

Currently, I am working on creating a Kotlin/MP wrapper for libllama.
I will publish code on a later point, as soon as I finish cleaning up and publishing what I am working on currently.

I am making this post to figure out two functions.

    // Token logits obtained from the last call to llama_eval()
    // The logits for the last token are stored in the last row
    // Can be mutated in order to change the probabilities of the next token
    // Rows: n_tokens
    // Cols: n_vocab
    LLAMA_API float * llama_get_logits(struct llama_context * ctx);

    // Get the embeddings for the input
    // shape: [n_embd] (1-dimensional)
    LLAMA_API float * llama_get_embeddings(struct llama_context * ctx);

Where does one get n_embd & n_tokens?

[PriNova]

llama_get_logits are used in

llama.cpp/main.cpp

Line 123 in 31572d9

auto logits = llama_get_logits(ctx);

and

llama.cpp/main.cpp

Line 373 in 31572d9

auto logits = llama_get_logits(ctx);

llama_get_embeddings are not directly used at the moment, but called in

llama.cpp/main.cpp

Line 340 in 31572d9

const auto embeddings = llama_get_embeddings(ctx);

without usage of the result.

[Doomsdayrs]

Oh boi, I love code that is not used! 😄

[PriNova]

Keep in mind that GGML and LLAMA is/will be a library. This means that there may be functions that are not all used in an application. However, these functions may be used in Whisper.cpp or other apps. If functions become obsolete in the future, they can be annotated as 'deprecated'.

Depending on which IDE you use, these functions will be displayed as unused. Nevertheless, a port should take into account all functions in the current version, even if they are not currently in use.

I am very curious how the performance of the Kotlin port will be. Especially in using Kotlin/Native and coroutines to parallelize some parts of the code.

Since GGML and LLAMA are available as a library, a cinterop connection would also be a very nice solution to code applications in kotlin.