Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multi-threaded quantization #1075

Merged
merged 6 commits into from
Apr 20, 2023
Merged

Multi-threaded quantization #1075

merged 6 commits into from
Apr 20, 2023

Commits on Apr 19, 2023

  1. Multi-threading quantization.

    Not much gain for simple quantizations, bit it will be important
    for quantizations that require more CPU cycles.
    Kawrakow committed Apr 19, 2023
    Configuration menu
    Copy the full SHA
    d2f9266 View commit details
    Browse the repository at this point in the history

Commits on Apr 20, 2023

  1. Multi-threading for quantize-stats

    It now does the job in ~14 seconds on my Mac for
    Q4_0, Q4_1 and Q4_2. Single-threaded it was taking
    more than 2 minutes after adding the more elaborate
    version of Q4_2.
    Kawrakow committed Apr 20, 2023
    Configuration menu
    Copy the full SHA
    ce05fc0 View commit details
    Browse the repository at this point in the history
  2. Reviewer comments

    Kawrakow committed Apr 20, 2023
    Configuration menu
    Copy the full SHA
    b65e559 View commit details
    Browse the repository at this point in the history
  3. Avoiding compiler confusion

    After changing chunk_size to const int as suggested by
    @ggerganov, clang and GCC starting to warn me that I don't
    need to capture it in the lambda. So, I removed it from the
    capture list. But that makes the MSVC build fail. So,
    making it a constexpr to make every compiler happy.
    Kawrakow committed Apr 20, 2023
    Configuration menu
    Copy the full SHA
    7fae1c4 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    0ae02eb View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    b3545d9 View commit details
    Browse the repository at this point in the history