what does u and s stand for in ubatch and sbatch?

[CharlesJu1]

same as title。

[danbev]

sbatch stands for sequence-aware batch which can be found from this comment. My understanding of what this means is that for recurrent models they can benefit from processing batches with sequences of equal length. The way this works is that if there are multiple sequences in a batch (llama_batch that is passed to llama_decode) they will be split into ubatches of equal sequence length.
The sbatch will manage this (hence the name sequence aware batch) and produce an ubatch which will then be used to build the computation graph, set the inputs, and then compute the graph (the forward pass). So the ubatch is what is actually passed to the model and I think it stands for update batch.
This process will then continue by handling the remaining tokens for the sequences, again making sure that they are of equal size.

There are some more details in the following comment which also contains an example:
#7531 (comment)

[slaren]

ubatch comes from "micro batch". The main reason it exists is to enable pipeline parallelism when processing large batches with multiple GPUs, because each micro batch can be processed in parallel (to some extent).

[danbev]

@slaren Ah, that makes more sense than update. Thanks for that!

[robbiemu]

ubatch comes from "micro batch". The main reason it exists is to enable pipeline parallelism when processing large batches with multiple GPUs, because each micro batch can be processed in parallel (to some extent).

I happen to be working on some code that generates logits and I had a similar doubt. Can you verify if setting n_batch and n_ubatch below n_ctx but giving the model a chunk of tokens with Len = n_ctx will still exercise the full context (as per use with this script https://gist.githubusercontent.com/Ttl/0d51f739dc59254b4b2183e259c97d82/raw/7af6aa7adf3492f6cf0877718708a73c0c1377cf/llama_kl.py or llama-imatrix for that matter). I'm building a tool that is not random (based on discussions here) for analytical imatrix generation and, I'm just wondering if I should advise people not to set the batch values below the context size.

This question stems a bit from this comment: #5263 (comment)

[slaren]

n_batch and n_ubatch don't affect how much of the context you can use, it is just a limit to how many tokens you can put in a single batch. For the most part, using large batches is a tradeoff between performance and memory usage. I am not familiar with the python API, but you would probably want to reset the KV cache after evaluating every chunk. Normally this would be done by calling llama_kv_cache_clear before starting a new chunk.

[robbiemu]

that is interesting. clearing the cache was not done with the earlier kl-divergence script.