How to "fit" Mistral Small 24b on a RTX 5090?

[celsowm]

Hi !
I am trying to use Mistral Small 24b on our 5090 using sglang but no success at al :(

I tried both the safetensor version as well the fp8 using this official docker blackwell: link

Both everytime I got OOM

Any hints?

[b8zhong]

Try

Reducing --mem-fraction-static
Using a NVFP4 checkpoint (https://huggingface.co/llmat/Mistral-Small-Instruct-2409-NVFP4). The memory footprint is lower