Complete revamp of model loading to allow for more discrete control

[manyoso]

Large revamp of model loading.

[cebtenzzre]

I think there should be a comment explaining the significance of negative infinity here.

[manyoso]

This is not negative infinite. It is the smallest possible positive floating point value closest to zero.

[cebtenzzre]

Oops, sometimes I forget that FLT_MIN has nothing to do with INT_MIN. I'm thinking there should be a brief comment like // small nonzero value to make it clear that it is arbitrary.

[manyoso]

Added comments.

[cebtenzzre]

Also, attempting to switch models while a model is loading is allowed - it changes the model name but not the progress value, and it ends up queueing a series of model loads. We should either abort the model load via the progress callback, or prevent the user from doing this.

[cebtenzzre]

Is this dialog still necessary, when selecting a model that previously failed to load? Especially the "Model loading error..." at the top.

[cebtenzzre]

This combo box is too wide now that the Clone/Remove buttons are gone:

Also, half of the generation settings are off the edge of the dialog (not visible).

edit: Apparently also seen in the 2.7.0 release on Windows, see the Discord.

[manyoso]

All of the above should be fixed with this newest version. Thanks for the quality review.

[ThiloteE]

Nitpick: Models with long names bleed into symbols.
Shouldn't prevent a merge, but wanted to mention it.

[ThiloteE]

Also, having multiple conversations with different models and switching between them causes long (few seconds long) "switching context" messages.
.
Is it because the prompt is evaluated on CPU immediatelly?

[manyoso]

Nitpick: Models with long names bleed into symbols. Shouldn't prevent a merge, but wanted to mention it.

Fixed.

[manyoso]

Also, having multiple conversations with different models and switching between them causes long (few seconds long) "switching context" messages.

This has to do with the save/restore of context that can be slow under vulkan. We're looking at ways to speed it up.

[ThiloteE]

Thank you. Can confirm it is much better now. To be honest though, adding slightly to the width wouldn't hurt, because most models on huggingface do have long model names, especially the mergers. Also, if users have multiple model cards for them (e.g. various system prompts in different languages) then those model names might become even longer to avoid confusion.

[cebtenzzre]

A. Things look wrong at the minimum window size - and in general, when the window width is less than 3/4 of my screen.

B. I liked the font size of the regenerate button better before:

Than it is now:

C.

This combo box is too wide now that the Clone/Remove buttons are gone:

This is still the case, the combo box goes off the edge of the dialog.

D. If you:

1. Select a model and chat with it
2. Unload the model
3. Instead of clicking reload, or selecting a different model, select the same model again
4. Your chat session has now been cleared??

E. The way we handle crashes during model loading is still broken - not only is there a dialog without OK/Cancel buttons that appears when the user explicitly tries to load the same model again, but the model switcher actually completely freezes now, and the app has to be restarted in order to continue. And you can never load that model again until you successfully load a different one.

[manyoso]

I don't know what you mean by "This is still the case, the combo box goes off the edge of the dialog." as I don't see that.

The sizing issues with min width and so on are not germane to this PR. I'd like to address them in another PR. Moreover, they are going to go away when the new UI revamp comes so I don't want to spend much time on them.

[manyoso]

I've fixed D. I'm not sure about E and what we want to do. It should probably be addressed in a new PR.

[cebtenzzre]

The sizing issues with min width and so on are not germane to this PR. I'd like to address them in another PR. Moreover, they are going to go away when the new UI revamp comes so I don't want to spend much time on them.

Ok. But the clear chat button didn't overlap before because the model selection box was smaller, so this is a regression, even if you think it's not important.

I don't know what you mean by "This is still the case, the combo box goes off the edge of the dialog." as I don't see that.

On my Mac:

Notice how:

• The "Generation Settings" header that is supposed to be centered is now shifted to the right
• The right column of generation settings is cut off (as in, I cannot see the right half of each inputbox)
• The expanded combobox to select a model stretches outside of the settings dialog and onto the dimmed background behind it (to the right)

[manyoso]

Ok. But the clear chat button didn't overlap before because the model selection box was smaller, so this is a regression, even if you think it's not important.

Got it, thanks.

On my Mac:

Yeah, I don't see anything like this. I'll check on my mac.

[barnett-yuxiang]

LGTM

[manyoso]

I cannot reproduce what you see on the mac with the model settings. Further, this change does not seem to have any impact on the model settings or that combo box. I'm not sure why you see what you see @cebtenzzre

[manyoso]

The model loading error you get when app crashes and we display on reload is now gone.

[ThiloteE]

fixed #1660