Running AI locally is easy now. Choosing what to run is not.
We are at an inflection point in AI. Small, open models can now perform as well as cloud models like GPT or Claude on many tasks. This was not the case a year ago.
Why local
Privacy, cost and ownership
Running AI locally means your data can stay with you. It gives you more control over the models and infrastructure you depend on, and it can replace a recurring API bill with hardware you own.
The promise is straightforward. Working out what to run is not.
The difficulty with local AI
There is no single model to choose
If you want to use a cloud model, the decision is relatively simple: what is the best model I can get at an acceptable price? There is already a lot of good information to help answer that question. Most developers end up using GPT or Claude; teams using open models often start with whatever their API provider recommends.
Local AI is different. A model is only one part of the system. To find the best setup for your workload, you have to make several choices together.
- 01Which model suits the workload?
- 02Which hardware should run it?
- 03Which agent harness should you use?
- 04Which inference engine should serve it?
- 05Which model variation, quantization or pruning method should you use?
- 06Which runtime configuration should you choose, such as speculative decoding?
Getting the best combination should not require becoming an expert in every one of these layers.
How people benchmark local AI today
The available information is fragmented
The information around local AI is still sparse, isolated and often unreliable. That is the other side of having so many choices: there are more ways to optimize a system, but also more ways to compare the wrong things.
A lot of local-model discussion starts and ends with tokens per second, FLOPs or memory bandwidth. These numbers matter, but choosing an entire setup from them alone is naive.
Similar throughput. Very different task time.
Decode throughput at 32K context
tokens per second · higher is better
Laguna S 2.1
Q4_K_M
Nemotron 3 Super
UD-Q3_K_M
Median GDPVal task time
minutes per task · lower is better
Laguna S 2.1
Q4_K_M
Nemotron 3 Super
UD-Q3_K_M
Better specifications do not guarantee a faster task.
FP32 compute
TFLOPS · higher is better
M4 Max
128GB · 40-core GPU
DGX Spark
GB10 · 128GB
Memory bandwidth
GB/s · higher is better
M4 Max
128GB · 40-core GPU
DGX Spark
GB10 · 128GB
Median GDPVal task time
prefill + decode · lower is better
M4 Max
128GB · 40-core GPU
DGX Spark
GB10 · 128GB
A model name alone is not a benchmark result. The exact variation has to be tested.
A heavy quant of a better, larger model is not automatically better.
GDPVal score
higher is better
Ornith 397B
IQ2_XXS
Ornith 35B
Q4_K_M
Total parameters
billions of parameters
Ornith 397B
IQ2_XXS
Ornith 35B
Q4_K_M
GGUF disk size
GiB on disk
Ornith 397B
IQ2_XXS
Ornith 35B
Q4_K_M
How local.ai solves this
We compare the complete setup
Our aim is to take that burden off you and identify the best solution for your constraints: your workload, budget and time.
That means comparing the whole combination, not one part in isolation.
If you are buying hardware, local.ai helps show what gives you the best results across the current model landscape. If you already have hardware, it helps you choose the model and setup that give you the right trade-off between intelligence and time.
The measurements
Intelligence, speed and cost
Every combination we test has three outcomes that matter: how well it completes the work, how long the work takes, and what the setup costs.
Cost is driven mainly by the hardware you need, with estimated electricity cost included where the measurement supports it. Speed is not just prefill time or decode throughput; it is the end-to-end time a real task takes to complete. Intelligence is measured with evaluations that mimic the kinds of workloads people actually use AI for.
Intelligence is the local.ai Intelligence Index, a fixed weighted combination of three benchmark scores; the exact formula is below. Speed applies the same weights to each evaluation's median end-to-end task time, which gives the typical task behind Tasks per Hour and, together with hardware cost, the Value Index.
We run the workloads, measure what happens and keep the dimensions connected. That is what turns a pile of benchmark numbers into a useful answer.
Intelligence Index
Intelligence Index = 2% × τ²-bench + 36% × GAIA + 62% × GDPval
Each benchmark score is a percentage from 0 to 100, so the index is also a value from 0 to 100.
- τ²-bench2%
- Agentic tool use: multi-turn tasks where the model must call tools and follow policies to resolve a request.
- GAIA36%
- General assistant tasks that need reasoning, browsing and tool use to reach a verifiable answer.
- GDPval62%
- Economically valuable work: realistic professional deliverables graded against expert output.
- A benchmark score is the best completed measured run for that exact model variation, meaning the same quantization and serving configuration, shown as whole percents.
- A model only gets an index once all three benchmarks have a completed score. A missing benchmark is never counted as zero and the weights are never redistributed, so models with incomplete evaluations appear as in progress without an index.
- The same weights apply to each evaluation's median end-to-end task time. That weighted time is the typical task behind Tasks per Hour.
- The weights above are the defaults everyone sees. You can change them in any chart's settings; custom weights are normalised to add up to 100% and every chart and table re-ranks live.
Worked example. A model scoring 80% on τ²-bench, 70% on GAIA and 85% on GDPval has an Intelligence Index of 2% × 80 + 36% × 70 + 62% × 85 = 79.5.
Where this goes next
A broader map of local AI
The landscape changes quickly, so this is not a finished map. We will keep adding representative workloads, models, hardware, inference engines and configurations as we can measure them properly.
The principle will stay the same: compare complete setups on real work, keep missing measurements missing, and make the result useful to someone deciding what to run.
Independence
An independent reference
local.ai is presented as an independent reference. The underlying speed and evaluation data are produced by Exo Labs. We have no commercial relationship with the hardware vendors or model providers referenced on this site.