Ollama server requirements: how much GPU and RAM do you actually need?

There is no single set of Ollama server requirements that works for every model. The model itself matters, but so do quantisation, context length, concurrent users, the runtime and whether some of the work can be handled by system RAM and CPU.

That means choosing an AI server by GPU name alone is usually the wrong place to start. At Whiterock Platforms, we prefer to start with the workload and work backwards towards the lowest-cost hardware that should run it sensibly.

If you already know you want help putting a model online, see our AI Hosting service. If you are comparing standard website or application hosting instead, see Web Hosting.

A quick way to estimate model weight memory

A useful first approximation for quantised model weights is: parameters × bits per parameter ÷ 8.

Using that deliberately simplified calculation, an 8-billion-parameter model at 4-bit quantisation is roughly 4 GB of model weights. A 14B model is roughly 7 GB, a 32B model roughly 16 GB and a 70B model roughly 35 GB.

Those are not GPU requirements. They are only theoretical starting points for the model weights. Real usage needs additional memory for the runtime, model metadata, context and other working data.

Why VRAM requirements are higher than the model file alone

Loading the model is only part of the job. The server also needs working memory while generating responses. Context length can materially change that requirement because the runtime has to retain information about the tokens being processed.

A model that fits comfortably with a short prompt may have much less headroom when asked to work with a long document, a large conversation history or several users at the same time.

For that reason, we would not size a production Ollama server by asking only “does the model fit in VRAM?” The more useful question is “does it fit with enough headroom for the way people will actually use it?”

Practical starting points for GPU memory

These are starting points for investigation rather than guaranteed minimum specifications:

Small 7B–8B quantised models: an 8 GB-class GPU can be worth testing if the model, context and expected concurrency are modest.

13B–14B quantised models: 12–16 GB of VRAM gives considerably more room and is a sensible range to assess.

30B–32B quantised models: 24 GB-class hardware becomes much more useful, although the exact requirement still depends heavily on quantisation and context.

Very large models: once model weights and runtime requirements move beyond a single GPU's available VRAM, the choices include higher-memory hardware, multiple GPUs or some degree of CPU and RAM offload.

System RAM still matters

GPU memory gets most of the attention, but system RAM matters too. The operating system, Ollama and supporting services all need memory, and workloads that cannot be held fully in VRAM may use system RAM as part of the execution path.

A machine with a suitable GPU but barely enough system RAM can therefore still be a poor AI server. We normally want meaningful headroom rather than designing a machine that operates permanently at its memory limit.

Context length can change the answer

Context is one of the easiest things to overlook when comparing AI servers. A short interactive assistant and a system that routinely processes long documents may use the same base model but have very different memory requirements.

If you intend to use large context windows, include that requirement when sizing the server. Buying a GPU based only on model parameter count can leave a machine with little useful headroom once real prompts arrive.

One user and ten users are different workloads

Concurrency changes both memory pressure and the amount of compute required. A local assistant used by one person can tolerate a configuration that would feel painfully slow if several customers were waiting for responses at the same time.

This is why an AI hosting quote should ideally include an estimate of simultaneous users, not just monthly traffic.

Can CPU offload reduce the GPU requirement?

Often, yes. Efficient runtimes can place only part of a model on the GPU and use system RAM and CPU for the remainder. That can make a model runnable on hardware with less VRAM.

The trade-off is usually speed. A configuration that technically runs is not automatically a good production configuration, so latency and expected usage still need testing.

Does an older GPU still make sense for Ollama?

Sometimes. For inference, usable VRAM can be more valuable than simply having the newest GPU generation. An older card with more memory may suit some workloads better than a newer card with less memory.

Compatibility has to be checked first. Driver support, CUDA or other runtime requirements, model software and the specific GPU generation all matter. We would not recommend old hardware simply because it is cheap.

The five things we ask before sizing an Ollama server

Before recommending hardware, we want to know:

1. The exact model. A model name or Ollama tag is far more useful than simply saying “an LLM”.

2. Quantisation. Q4, Q5, Q8 and full-precision variants can have very different memory requirements.

3. Context requirement. A chatbot with short prompts is different from a system reading long documents.

4. Concurrent users. One developer and a customer-facing application are not the same workload.

5. Performance expectation. “It runs” and “it responds quickly enough for paying users” are different standards.

When dedicated GPU hosting may be unnecessary

Dedicated GPU hosting is not automatically the cheapest answer. If a model is used only occasionally, an API or usage-based service can cost less than keeping a dedicated GPU online all month.

Dedicated infrastructure starts to become more attractive when you need predictable availability, fixed monthly cost, control over the model or environment, private deployment, or enough regular usage that metered services become expensive.

How Whiterock approaches AI server sizing

Our approach is deliberately simple: start with the model and workload, calculate a realistic memory requirement, check software compatibility and then choose the least expensive configuration that still has useful headroom.

We are building our own compatibility and benchmark data around this process. As we test more GPUs and models, this guide will be updated with measured performance rather than pretending that theoretical specifications tell the whole story.

If you already have an Ollama model running locally, send us the model, your current hardware and the number of people you expect to use it. Our managed AI hosting service includes a fit check to help work out the lowest-cost sensible server for the workload.

Ollama server requirements FAQ

Can Ollama run without a GPU?
Yes. Ollama workloads can run using CPU resources, although generation is normally much slower than a suitable GPU-backed setup. CPU execution can still be useful for testing, low-volume workloads or partial offload.

Is 8 GB of VRAM enough for Ollama?
It can be enough for some smaller quantised models, but model size alone does not decide this. Quantisation, context length and expected usage need to be considered as well.

Does Ollama need lots of system RAM?
The amount depends on the model and how much work is kept on the GPU. More system RAM is particularly useful when part of the model is being offloaded from VRAM.

Can an older NVIDIA GPU run Ollama?
Some older GPUs can still be useful for supported inference workloads. Compatibility with the required driver and runtime should be checked before choosing hardware.