Computer Hardware Required to Run LLaMA AI Model Locally (GPU, CPU, RAM, SSD)

Large language models (LLMs) are powerful tools capable of generating natural language texts for various tasks such as programming, text summarization, role-playing, or serving as general AI assistants. One of the most advanced LLMs is LLaMA (Large Language Model Meta AI), a 70-billion-parameter model developed by Meta AI, a research division of Facebook.

To run the LLaMA model at home, you will need a computer equipped with a powerful GPU, capable of handling the substantial data and computational demands required for inferencing. In this article, we will discuss some of the hardware requirements necessary to run LLaMA and Llama-2 locally.

There are different methods for running LLaMA models on consumer hardware. The most common approach involves using a single NVIDIA GeForce RTX 3090 GPU. This GPU, with its 24 GB of memory, suffices for running a Llama model. For instance, one can use an RTX 3090, an ExLlamaV2 model loader, and a 4-bit quantized LLaMA or Llama-2 30B model, achieving approximately 30 to 40 tokens per second, which is huge.

However, to run the larger 65B model, a dual GPU setup is necessary. This configuration allows the model weights to fit within the VRAM. Popular combinations include 2x RTX 3090s or an RTX 3090 paired with an RTX 4090.

Dual GPU PC build to run Llama large language model in 2024:

Running the LLaMA model on a CPU is also an option. This approach requires the GGML/GGUF version (LLaMA, Vicuna, Alpaca) of the model and a software named llama.cpp. Suitable CPUs for running LLaMA include the Core i7 12900K and Ryzen 9 5900X. For more information on this topic, refer to the CPU section.

Before we continue I want to mention that this guide is tailored toward PC users. We also have a guide about best Mac for large language models. You can check it out if your are a Mac user.

Let’s examine some of the PC hardware requirements necessary to operate the LLaMA model on a desktop PC:

GPU for running LLaMA and Llama-2

The GPU is the most crucial component of computer hardware for running LLaMA on a consumer-grade machine, as it handles the majority of the processing needed to operate the model. The performance of the GPU directly influences the inference speed.

While different variations and implementations of the model might demand less powerful hardware, the GPU remains the most vital part of the system.

GPU Requirements for 4-Bit Quantized LLaMA Models:

Example of inference speed using ExLlama,  RTX 4090, and Intel i9-12900K  CPU

LLaMA-7B

To run LLaMA-7B effectively, it is recommended to have a GPU with a minimum of 6GB VRAM. A suitable GPU example for this model is the RTX 3060, which offers a 8GB VRAM version. Other GPUs such as the GTX 1660, 2060, AMD 5700 XT, or RTX 3050, which also have 6GB VRAM, can serve as good options to support LLaMA-7B.

LLaMA-13B

For optimal performance with LLaMA-13B, a GPU with at least 10GB VRAM is suggested. Examples of GPUs that meet this requirement include the AMD 6900 XT, RTX 2060 12GB, 3060 12GB, 3080, or A2000. These GPUs provide the necessary VRAM capacity to handle the demands of a 13B model.

LLaMA-30B

To ensure smooth operation of LLaMA-30B, it is advisable to use a GPU with a minimum of 20GB VRAM. The RTX 3080 20GB, A4500, A5000, 3090, 4090, 6000, or Tesla V100 are examples of GPUs that offer the required VRAM capacity. These GPUs enable efficient processing and memory management for LLaMA-30B.

LLaMA-65B and 70B

LLaMA-65B and 70B performs optimally when paired with a GPU that has a minimum of 40GB VRAM. Suitable examples of GPUs for this model include the A100 40GB, 2x3090, 2x4090, A40, RTX A6000, or 8000. These GPUs provide the VRAM capacity to handle LLaMA-65B and Llama-2 70B weights.

CPU for LLaMA

In addition to the GPU, you will also need a CPU that can support the GPU and handle other tasks, such as data loading and preprocessing. If you are running a Llama model entirely on the GPU (GPTQ model) CPU is not so crucial. What the CPU does, is to helps load your prompt faster, where the LLM inference is entirely done on the GPU.

Good CPUs for LLaMA include the Intel Core i9-10900K, i7-12700K, Core i7 13700K or Ryzen 9 5900X and Ryzen 9 7900X, 7950X.

It is important to note that this article focuses on a build that is using the GPU for inference. However, there are LLaMa models specifically optimized for CPU use. For example, GGML/GGUF provides a solution to overcome the limitations posed by GPU memory when dealing with large models. Therefore, if you are constrained by budget for a GPU, or if you already own one but wish to test models that exceed your VRAM capacity, it is advisable to run a software called llama.cpp with GGUF model files. Yet, it’s crucial to consider CPU inference as a combo between the processor and RAM, since CPU inference is bottlenecked by RAM and benefits from a CPU that supports higher memory bandwidth and more channels. Hence, opting for a CPU with these specifications is the preferred choice.

Models for Llama CPU based inference:

• Core i9 13900K (2 channels,  works with DDR5-6000 @ 96 GB/s)
• Ryzen 9 7950x (2 channels,  works with DDR5-6000 @ 96 GB/s)

This is an example of running llama.cpp with a Ryzen 7 3700X and 128GB RAM @ 3600 MHz.

Memory (RAM) for LLaMA computer

Besides the GPU and CPU, you will also need sufficient RAM (Random Access Memory) and storage space to store the model parameters and data. The RAM requirement for the 4-bit LLaMA-30B is 32 GB, which allows the entire model to be held in memory without swapping to disk. However, if you don’t have the required amount, the model will still load, but not as quickly. For larger models or longer texts, you may want to consider using more RAM, such as 64 GB.

In a situations where you use CPU for inference, the bandwidth between the CPU and the memory is a critical factor. I’d like to emphasize its importance. When generating a single token, the entire model needs to be read from memory once. Suppose your have Core i9-10900X (4 channel support) and DDR4-3600 memory, this means a throughput of 115 GB/s and your model size is 13 GB. In that case the inference speed will be around 8 tokens per second, regardless of how fast your CPU is or how many parallel cores it has.

The amount of RAM depends on the type of GGML /GGUF quantization and the model (LLaMA, Alpaca, Wizard, Vicuna etc.) you are using.

These are the the memory (RAM) requirements for LLaMA model used on the CPU:

Inference speed for 13B model with 4-bit quantization, based on memory (RAM) speed when running on CPU:

Storage

The minimum storage requirement for LLaMA is 1 TB NVMe SSD, which can store the model files and data files with fast read and write speeds. However, for more data or backup purposes, you may want to use more storage space, such as 2 TB or 4 TB of SSD.

Choose high-speed storage. Opt for a PCIe 4.0 NVMe SSD with excellent sequential speeds to facilitate fast data transfer between storage and system RAM.

How does model quantization effect the choice of GPU?

Quantized LLMs (Language Models) use fewer bits to store and process the model’s weights and activations. This makes them faster and more efficient for GPU deployment.

4-bit quantized LLMs use only 4 bits per weight. This means they take up much less memory and computation time than full-precision models. They can run smoothly on GPUs with low VRAM capacities.

8-bit quantized LLMs use 8 bits per weight. This still reduces memory and computation costs compared to full-precision models, but not as much as 4-bit quantization. They need more GPU memory and computational power to run well. They are more suitable for GPUs with high VRAM capacities and computational capabilities.

To sum up, 4-bit quantized LLMs are more efficient and can run on GPUs with low VRAM capacities. 8-bit quantized LLMs are slightly less efficient and need GPUs with high VRAM capacities and computational capabilities.

As you can see, the precision of an LLaMAhas a direct impact on its GPU memory requirements and computational demands. Native (32-bit) LLMs require the most GPU memory and computational power, while 4-bit and lower quantized LLMs require the least.

The suitable GPU for an LLaMA will depend on its precision and the specific tasks that you want to use it for. If you need to run a large LLaMA (30B+), then you will need a GPU with a large VRAM capacity. If you only need to run a small LLaMA (7B), then you can get away with using a GPU with a smaller VRAM capacity.

It is important to note that the accuracy of the model will also decrease as the quantization level decreases. This is because the reduced precision can lead to errors in the model’s predictions.

The best quantization level for you will depend on your specific needs and requirements. If you need a model that is small and efficient, then you may want to consider using a 2-bit or 3-bit quantized model. However, if you need a model that is highly accurate, then you may want to use a 5 or more bit model.

Does a dual GPU setup deliver better performance than a single one when used with LLaMA?

Adding a second GPU may not speed up text generation as much as expected. Some tests have shown surprising results where lower-end GPUs were faster than higher-end GPUs in generating tokens/second.

Dual GPU setups have more VRAM in total, but each GPU still has its own VRAM limit. The 30B LLaMA needs about 20GB VRAM, so two RTX 3090 GPUs (each with 24GB VRAM) still have only 24GB VRAM available, so splitting the model between the two will not increase the speed. The model should fit in the VRAM of one GPU to run to gain a max speed performance. .

However, if the model is large (65B+) and cannot fit within the VRAM of a single GPU, using multiple GPUs can indeed pseed up the inference compared to running the same model in split mode between the GPU and RAM. In these instances, each GPU can manage a segment of the model, with the model weights being distributed among them.

Therefore, multiple GPUs are commonly used when dealing with large models.

What is faster for inference speed for 65B Llama model – Dual RTX 3090/4090 or Mac M2 Pro/Max/Ultra?

Using Apple M1 or M2 M3 Pro/Max is the recommended option for running large language models on a laptop. These processors utilize Apple’s unified memory architecture (UMA), offering fast and low-latency memory access. This results in improved speed of output. The M1/M2 Pro supports up to 200 GB/s unified memory bandwidth, while the M1/M2 Max supports up to 400 GB/s and M1/M2/M3 Ultra 800 GB/s. For example MacBook Pro M2 Max using Llama.cpp can run 7B model with 65 t/s, 13B model with 30  t/s, and 65B model with 5 t/s.

However in terms of inference speed dual setup of RTX 3090/4090 GPUs is faster compared to the Mac M2 Pro/Max/Ultra. Two RTX 4090s can run 65b models at a speed of 20 tokens per second, while two affordable secondhand RTX 3090s achieve 15 tokens per second with Exllama. Additionally, the Mac evaluates prompts slower, making the dual GPU setup more appealing.

Hints and Tips when choosing PC hardware for LLaMA

Build around the GPU

Create a platform that includes the motherboard, CPU, and RAM. The GPU handles training and inference, while the CPU, RAM, and storage manage data loading. Select a motherboard with PCIe 4.0 (or 5.0) support, multiple NVMe drive slots, x16 GPU slots, and four memory slots. CPUs with high single-threaded speed, like Ryzen 5000/7000 or Intel’s 12th/13th gen, are recommended.

LLM and VRAM

For optimal performance in terms of response quality, it is recommended to run the 8-bit 13B LLM or the 4-bit 30B model on a GPU with at least 20GB VRAM. Both models provide similar quality responses, and VRAM availability should be the deciding factor. Consider options like the RTX 30 series or RTX 40 series, such as the RTX 3090 24GB,  RTX 4090 24GB for the best local performance.

Speed Comparison

The 13B model generally runs faster than the 30B model in terms of tokens generated per second. While the exact speed difference may vary, the 13B model tends to offer a noticeable improvement in generation speed compared to the 30B model.

RAM requirements

Aim for higher speed DDR5 memory and at least 1.5 times the VRAM capacity or double the VRAM for optimal performance.

PCIe 4.0 NVMe SSD

The importance of a high sequential speed PCIe 4.0 NVMe SSD is mainly for the initial model load into VRAM. Once the model is loaded, the SSD’s impact on generation speed (tokens/second) is minimal.

Sufficient Regular RAM

Having enough regular RAM, preferably double the VRAM capacity, is essential for the initial model load. Once the model is loaded, its impact on the actual generation speed is limited.

CPU Single-Threaded Speed

The CPU’s single-threaded speed is important primarily for the initial model load rather than running the model during generation. The CPU’s role is more prominent in tasks such as prompt preprocessing, model loading, and other non-GPU-dependent operations.

Power supply and case

Invest in a high-quality power supply with sufficient capacity to power all components. Choose a spacious case with good airflow for optimal thermals.

DDR5 and future platforms

While DDR5 and future platforms like Zen 4 or AM5 offer advantages, stability and compatibility can vary. Consider investing in a high-end motherboard with good PCIe slot layout and memory support for future upgradeability.

Remember, while these hints and tips provide insights based on experience, individual system configurations and performance may vary. It’s always advisable to experiment and benchmark different setups to find the most suitable solution for your specific needs.