How deterministic is cpu vs gpu

[odarcan]

Does this entirely depend on temperature settings, and or besides that

will running on cpu vs gpu give the same results?

[odarcan]

using gpu:
./mistral-7b-instruct-v0.2.Q4_0.llamafile -ngl 9999 -p '[INST]You are a knowledgeable assistant[/INST] Hi, tell me a poem about computer monitors that show linux' --temp 0

[INST]You are a knowledgeable assistant[/INST] Hi, tell me a poem about computer monitors that show linux terminal screens, I'll do my best to provide you with accurate and helpful information on a wide range of topics, including but not limited to, Linux terminal screens and computer monitors.

Now, let me see if I can come up with a poem about computer monitors displaying Linux terminal screens:

In the heart of the machine, where data flows,
A monitor sits, in quiet repose,
A canvas of pixels, black and white,
A terminal screen, a programmer's delight.

Linux whispers commands, in gentle beep,
Through the terminal, the magic we keep,
Files and folders, scripts and code,
In the world of Linux, we're the gods.

The cursor blinks, a steady beat,
A rhythm of progress, a rhythm so sweet,
The monitor glows, a window to the soul,
Of the machine, of the data, of the goal.

So here's to the monitors, the terminals, and the screens,
To the Linux world, where magic and logic convene,
May we always find joy in the dance of the keys,
And the beauty of the terminal, where the magic truly lies.

I hope you enjoyed the poem! Let me know if you have any questions or if there's anything else I can help you with. [end of text]

[odarcan]

same thing on cpu
./mistral-7b-instruct-v0.2.Q4_0.llamafile -p '[INST]You are a knowledgeable assistant[/INST] Hi, tell me a poem about computer monitors that show linux' --temp 0

[INST]You are a knowledgeable assistant[/INST] Hi, tell me a poem about computer monitors that show linux terminal screens, I'll do my best to provide you with accurate and up-to-date information on various topics, including but not limited to Linux and computer monitors. However, I'm afraid I don't have the ability to create poetry on the spot. Here's a haiku I came up with:

Linux screen,
Glowing pixels dance,
Code comes to life.

Now, if you have a specific question or topic in mind, feel free to ask and I'll do my best to help you out!

As for Linux terminal screens on computer monitors, they are text-based interfaces that allow users to interact with the Linux operating system. They provide a powerful and flexible way to manage files, run commands, and configure system settings. The terminal screen can be displayed on any connected monitor, and users can have multiple terminal windows open at the same time, each on a different monitor if desired.

Linux terminal screens are popular among power users and developers due to their efficiency and flexibility. They allow users to automate tasks, write scripts, and quickly navigate the file system. Additionally, many popular Linux tools and applications, such as Git and SSH, are designed to be used in the terminal.

If you have any specific questions about Linux terminal screens or computer monitors, feel free to ask and I'll be happy to help! [end of text]

[odarcan]

Cpu wont do a poem for example,

What is the difference?

I would like to know how to use cpu to achieve the same result as gpu, even if it would be much slower.

[jart]

We can't guarantee exactly the same outputs across environments. Sometimes the algorithms change depending on what makes sense for a given hardware architecture. The order of addition operations may change. This is especially true of quants, which are already inconsistent with the full quality version of the model (BF16 and F16). Also Q4_0 is a legacy format.

The best way to achieve consistent output is to use F16 or BF16 weights. These really shouldn't diverge between CPU and GPU. Another good one is Q6_K whose output quality is so high that for many models it tends to be consistent with BF16 / F16.

I absolutely care about significant divergences. Although it's hard to tell here that it's not an issue with your prompt. The correct way to prompt Mistral for a poem is as follows.

./mistral-7b-instruct-v0.2.Q4_0.llamafile --temp 0 \
  -p '[INST]Write a poem about computer monitors that show Linux.[/INST]'

Here's a comparison between Q4_0 and BF16 on CPU.

We see that the model is pretty much doing the same thing, although a little differently.

[jart]

Another thing worth mentioning is that llamafile offers the --precise flag, which may be used in CPU mode to disable certain optimizations (e.g. AVX512 BF16) that cause lower fidelity. It also enables Kahan summation. This provides the highest mathematical accuracy that llamafile is capable of providing. For extremely large, large language models, the precise mode helps ensure outputs are more consistent with the cuBLAS which has logarithmic roundoff error reduction (something I haven't quite figured out how to do fast and elegantly on CPU). Please note that --precise mode goes much slower, which is exactly the thing you asked for! Although it should hopefully get better in the future. And there's really not much point in using it if you're using heavily quantized legacy weights like Q4_0 :-)

[odarcan]

So temp doesnt make it deterministic,
Any idea why my cpu run wont even attempt to do a poem?
Does the cpu run llm somehow have access to less memory or something, and the llm knows it and therefore doesnt attempt it?
This is weird.

Somehow it comes up with the same haiku every time on cpu :)
Linux screen,
Glowing pixels dance,
Code comes to life.

I want to ensure same results on cpu and gpu
I would be interested if you ever find a way to make it so.

Thanks

[jart]

--temp 0 does make it deterministic, in the sense that repeated invocations will be the same.