Hardware-Agnostic Models in vLLM
- PyTorch: 1337 events in the last 90 days
- Previous: earlier the same day · How Shopify built a continual learning loop with PyTorch and vLLM
What happened
TL;DR To achieve state-of-the-art performance at the frontier, vLLM is changing its internal implementation in ways that make it incompatible with fullgraph torch.compile. This may have consequences for users who care about out-of-tree accelerators, older GPUs, or more exotic models. To address this, we are introducing a new set of “HW agnostic” layers in vLLM. These layers will ensure vLLM can continue to move at t…
Summary assembled by rule from the sources below
Why it's spreading
Sources
Official
- Hardware-Agnostic Models in vLLMPyTorch Blog