Support 1 * 128 and 128 * 128 block-wise quant?

[zfan2356]

In the CUDA 12.9 cuBLASLt documentation, I noticed support for 1×128 and 128×128 block-wise quantization methods. However, I found that nvmath-python currently lacks bindings for this type of quantize approach. I wonder do we have any plan for support this approach?

https://docs.nvidia.com/cuda/cublas/index.html#cublasltmatmulmatrixscale-t

[szkarpinski]

Hi @zfan2356 , thank you for raising this issue. The lower-level nvmath.bindings for those scaling types should be included in the next release of nvmath-python. I'll also start working on adding support to our higher-level host APIs in nvmath.linalg.advanced.matmul.

[zfan2356]

@szkarpinski Thanks for your reply! I also recommend supporting the pointer array mode for Group GEMM. Both of these features are commonly used in modern LLM pretrain. nvmath-python is a user-friendly Python library, and I hope it can support these two features as soon as possible. Thank you very much!