Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: use scalar type to dispatch to different gptq_marlin kernels #689

Merged
merged 1 commit into from
Sep 11, 2024

Conversation

AlpinDale
Copy link
Member

Use ScalarType instead of num_bits (in combination with has_zp) to preform the dispatching for gptq_marlin this sets the stage for folding fp8_marlin.cu into gptq_marlin.cu since now we can move dequant_8bit in fp8_marlin.cu as a dequant<T, vllm::kFE4M3fn.id()>(int q) specialization. I did not fold fp8_marlin.cu into gptq_marlin.cu in this PR to avoid excessive compile times for gptq_marlin.cu for now.

In-order to support passing scalar type as a template parameter in C++17, it has to be serialized to something that can be passed as a template parameter. This per introduces the concept of serializing the type into a 64 bit int id (that can be passed as a parameter) alongside a deserialization routine (from_id if the template needs to access the traits of the type). If/when we move to make C++20 the lowest standard support this serialization/deserialization can be removed as C++20 introduces passing literal class types as template parameters (see: https://en.cppreference.com/w/cpp/language/template_parameters)

@AlpinDale AlpinDale merged commit 6144150 into main Sep 11, 2024
5 checks passed
@AlpinDale AlpinDale deleted the gptq-scalar-type branch September 11, 2024 08:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant