As a tech enthusiast who specializes in hardware for local large language model (LLM) inference, I’m always on the lookout for the best performance-per-dollar.

This often leads me to explore older datacenter hardware. Recently, I got my hands on an Nvidia Tesla V100 with 32GB of VRAM. This card was a powerhouse in its day, but in August 2026, is it still a smart choice for a home lab focused on running quantized LLMs? I put it through its paces to find out.

The main appeal of the V100 is its generous 32GB of HBM2 VRAM and high memory bandwidth, two of the most critical factors for running large language models.

For those of us running quantized models, VRAM is often the biggest bottleneck, and 32GB opens the door to running larger, more capable models or using much longer context windows with smaller models.

My Benchmark Setup

To get a clear picture of its performance, I installed the V100 in my test bench running Ubuntu 24.04 LTS with Nvidia drivers version 580.142 and CUDA 12.8.

For the inference engine, I used the llama.cpp server. I interacted with the models through the Open WebUI front end. All models were tested on text summarization, using Unsloth dynamic quants, specifically the Q4_K_XL quantization.

I focused my testing purely on inference, as this is the primary use case for most home lab builders

Inference Performance Benchmarks

We repeated our tests using newer model versions, the latest NVIDIA drivers, and the latest version of llama.cpp (build: 10404) to provide a more up-to-date picture of the V100’s performance. Here’s how it performed.

Qwen3.6 35B A3B (Mixture-of-Experts)

I started with a Mixture-of-Experts (MoE) model, which can be demanding. The V100 handled it well, and I was particularly impressed by the context size it could manage.

With the Qwen3 30B model, I was able to load a 70,000-token context, which is quite substantial. At this maximum context, the generation speed was still a usable 12.32 tokens per second. This demonstrates the primary strength of this card: its ability to handle very large contexts thanks to its ample VRAM.

Qwen3.6 27B (Dense Model)

Next, I tested a standard dense model of a similar size to see how the performance compared.

For the dense 27B model, the VRAM allowed for a maximum context of 128,000 tokens. At that point, the inference speed was just under 11 tokens per second. While the context size is smaller than with the MoE model, 28K is still very respectable and more than what many consumer cards can handle with a model of this size.

Muse Glimmer 30B (Dense Model)

Nvidia Llama 3.3 Nemotron Super 49B

To push the limits, I loaded the largest model I thought was feasible, the Nemotron Super 49B.

I was successfully able to run this 49B parameter model, which is a significant achievement for a single GPU setup. With a 7,000-token context, the card produced tokens at a rate of over 17 t/s.

This is a practical speed for interactive use, showing that the V100 can indeed handle larger models that are out of reach for cards with less VRAM.

Competitive Landscape and Pricing

Performance numbers are only part of the story; value is what truly matters. As of August 2026, here is how the Tesla V100 stacks up against some of its main competitors in the new and used markets.

V100 vs RTX 3090 vs RTX 5090

The raw performance gap is much clearer when we look at both prompt processing and token generation. The RTX 5090 is in a different performance class, while the V100 remains competitive with the RTX 3090 in some smaller-context workloads.

Qwen3.6 27B

On the 27B model, the RTX 3090 is slightly faster than the V100 for generation at larger contexts, while the RTX 5090 is more than twice as fast. The V100’s advantage is that its 32GB of VRAM allows it to reach 128K context, where the 24GB RTX 3090 cannot.

Qwen3.6 35B.A3B MoE

The MoE benchmark shows a much larger advantage for the newer cards. The RTX 3090 is around 37% faster than the V100 at 4K generation, while the RTX 5090 is roughly 2.5x faster, but the V100 can still run very large 256K contexts.

Overall, the RTX 5090 is the clear performance winner, and the RTX 3090 is the better choice if generation speed is the priority. The V100’s reason to exist in a home lab is different: 32GB of VRAM at a low used price, making it useful for models and context sizes that a 24GB RTX 3090 cannot accommodate.

Conclusion: Is It a Good Buy?

So, is the Tesla V100 32GB still a good buy for a local LLM setup in 2026? The answer is a qualified yes, but only if its strengths match your specific workload.

The V100’s biggest advantage remains its 32GB of HBM2 VRAM. In testing, that extra capacity allowed it to run models such as the 49B Nemotron Super and handle context sizes reaching 256K with the Qwen3.6 35B.A3B. For enthusiasts who want to experiment with larger quantized models or very long contexts without spending several thousand dollars on a modern 32GB GPU, that capability is difficult to ignore.

At the same time, the V100 is not a performance champion. The RTX 3090 is generally faster for token generation, while the RTX 5090 is in an entirely different class. The V100 also comes from an older generation, so its long-term software and CUDA support, power consumption, and datacenter-oriented cooling requirements are important considerations for a home lab.

Ultimately, the V100 makes the most sense when VRAM capacity is the priority and you can find a well-priced used card. If you mainly run smaller models and care more about generation speed, a used RTX 3090 is the better all-around choice. But if your goal is to squeeze the largest models and longest contexts possible out of a relatively inexpensive single GPU, the 32GB V100 remains a surprisingly capable and interesting option in 2026.