Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

server: comment --threads option behavior #6230

Closed
Tracked by #6233
phymbert opened this issue Mar 22, 2024 · 4 comments · Fixed by #6254
Closed
Tracked by #6233

server: comment --threads option behavior #6230

phymbert opened this issue Mar 22, 2024 · 4 comments · Fixed by #6254
Assignees
@phymbert
Labels
documentation Improvements or additions to documentation enhancement New feature or request server/webui

Comments

@phymbert
Copy link
Collaborator

phymbert commented Mar 22, 2024 •
edited
Loading

As we are using batching, I am wondering what is the purpose of --threads N parameter in the server.

Should we remove it ?
@phymbert phymbert added the enhancement New feature or request label Mar 22, 2024
@ggerganov
Copy link
Member

It's used for bs = 1 decoding:

https://github.com/ggerganov/llama.cpp/blob/b2075fd6a578f5685060df4baa90ae9e48e98c70/llama.cpp#L9082

@phymbert phymbert self-assigned this Mar 22, 2024
@phymbert
Copy link
Collaborator Author

OK, later I will add a this precision in the readme, also specifying this is not used at all when using --ngl.

@phymbert phymbert reopened this Mar 22, 2024
@phymbert phymbert added the documentation Improvements or additions to documentation label Mar 22, 2024
@phymbert phymbert changed the title server: remove --threads option server: comment --threads option behavior Mar 22, 2024
@phymbert phymbert mentioned this issue Mar 22, 2024
Closed
16 tasks
@Jeximo
Copy link
Contributor

Jeximo commented Mar 22, 2024

As we are using batching

What about those who don't use batching?

@phymbert
Copy link
Collaborator Author

As we are using batching

What about those who don't use batching?

The server is using batching by design

@phymbert phymbert mentioned this issue Mar 23, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@phymbert phymbert

Labels
documentation Improvements or additions to documentation enhancement New feature or request server/webui
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

server: docs: --threads and --threads, --ubatch-size, --log-disable ggml-org/llama.cpp
3 participants
@ggerganov @phymbert @Jeximo