Flashinfer adoption by vLLM

[prasad-nair-amd]

The flashinfer github page says , flashinfer is adopted by vLLM. Does this mean if we are doing inference using vLLM flashinfer is used ??

[ghost]

You would still have to install FlashInfer ("pip install flashinfer-python"). Otherwise, you'll see this warning pop up during inference:

(EngineCore_0 pid=20942) WARNING 08-11 15:32:01 [topk_topp_sampler.py:61] FlashInfer is not available. Falling back to the PyTorch-native implementation of top-p & top-k sampling. For the best performance, please install FlashInfer.