As a tech enthusiast who specializes in hardware for local large language model (LLM) inference, I’m always on the lookout for the best performance-per-dollar.

This often leads me to explore older datacenter hardware. Recently, I got my hands on an Nvidia Tesla V100 with 32GB of VRAM. This card was a powerhouse in its day, but in August 2025, is it still a smart choice for a home lab focused on running quantized LLMs? I put it through its paces to find out.

The main appeal of the V100 is its generous 32GB of HBM2 VRAM and high memory bandwidth, two of the most critical factors for running large language models.

For those of us running quantized models, VRAM is often the biggest bottleneck, and 32GB opens the door to running larger, more capable models or using much longer context windows with smaller models.

My Benchmark Setup

To get a clear picture of its performance, I installed the V100 in my test bench running Ubuntu 24.04 LTS with Nvidia drivers version 575.57.08.

For the inference engine, I used the llama.cpp server. I interacted with the models through the Open WebUI front end. All models were tested on text summarization, using Unsloth dynamic quants, specifically the Q4_K_XL quantization.

I focused my testing purely on inference, as this is the primary use case for most home lab builders

Inference Performance Benchmarks

I tested the V100 with a few different models to see how it handled various sizes and architectures. Here’s how it performed.

Qwen3 30B A3B (Mixture-of-Experts)

I started with a Mixture-of-Experts (MoE) model, which can be demanding. The V100 handled it well, and I was particularly impressed by the context size it could manage.

With the Qwen3 30B model, I was able to load a 70,000-token context, which is quite substantial. At this maximum context, the generation speed was still a usable 12.32 tokens per second. This demonstrates the primary strength of this card: its ability to handle very large contexts thanks to its ample VRAM.

Qwen3 32B (Dense Model)

Next, I tested a standard dense model of a similar size to see how the performance compared.

For the dense 32B model, the VRAM allowed for a maximum context of 28,000 tokens. At that point, the inference speed was just under 11 tokens per second. While the context size is smaller than with the MoE model, 28K is still very respectable and more than what many consumer cards can handle with a model of this size.

Looking at the competition for the Qwen3 32B model, an RTX 3090 with 10,000 tokens context generates tokens at 24.52 t/s. A dual RTX 5060 Ti 16GB with the same 10K context, generate at rate 13.15 t/s. The newer and much more expensive RTX 5090 handles a 28k context with an inference speed of 30.71 t/s.

The V100 is slower in raw token generation than the RTX 3090. However, it was able to handle a much larger context with the Qwen3 32B model (28K vs 10K) and an even larger one with the Qwen3 30B MoE model (70K). This highlights the core trade-off: raw speed versus VRAM capacity and context length.

Nvidia Llama 3.3 Nemotron Super 49B

To push the limits, I loaded the largest model I thought was feasible, the Nemotron Super 49B.

I was successfully able to run this 49B parameter model, which is a significant achievement for a single GPU setup. With a 7,000-token context, the card produced tokens at a rate of over 17 t/s.

This is a practical speed for interactive use, showing that the V100 can indeed handle larger models that are out of reach for cards with less VRAM.

Competitive Landscape and Pricing

Performance numbers are only part of the story; value is what truly matters. As of August 2025, here is how the Tesla V100 stacks up against some of its main competitors in the new and used markets.

Challenges and Considerations

The Tesla V100 is not a simple plug-and-play solution. As a server card, it uses a blower-style cooler designed for high-airflow server chassis. To use it in a desktop, you will need to ensure adequate case airflow or fashion a custom cooling solution to prevent it from overheating.

A significant software consideration is that Nvidia is expected to drop support for the Volta architecture, which the V100 is based on, in the next major CUDA Toolkit release (version 13). This could create compatibility issues with future software, although the LLM community is known for finding workarounds.

Furthermore, the V100 lacks hardware support for newer data formats like FP8 and BF16, and it doesn’t have features like Flash Attention. However, for our primary use case of running GGUF-quantized models, this is less of a concern. Formats like Q4_K_M are implemented at the software level, so the underlying hardware format support is not a direct limitation.

Conclusion: Is It a Good Buy?

So, should you buy a Tesla V100 32GB in 2025 for your local LLM setup? The answer is a conditional yes.

The V100 is a viable and compelling option if your priority is maximizing VRAM to run larger models or use extremely long context windows. Its ability to handle a 49B model or a 70K context with a 30B model is something that cheaper cards like the RTX 3090 simply cannot do. It occupies a unique space where it offers more VRAM than the consumer “value king” RTX 3090 and significantly more memory bandwidth than a dual-GPU setup like two RTX 5060 Tis.

However, it is an older card with impending end-of-life for CUDA support, and its raw inference speed is lower than that of the RTX 3090. You must also be prepared to handle its specific cooling needs.

It makes sense to buy the V100 if you find it at a compelling price (in the 1100 range or lower) and your primary goal is inference on models that require more than 24GB of VRAM. If you prioritize raw generation speed for smaller models and shorter contexts, a used RTX 3090 likely offers better value. The V100 is a specialized tool for the enthusiast who understands its strengths and weaknesses and values VRAM capacity above all else.