What Is AI Inference? A Plain-English Guide

Every time you ask an AI a question, a model somewhere does a piece of work to answer you. It's the part of AI you actually use, and the part businesses pay for again and again. This guide explains what it is, how it works and why it matters, from scratch.

What is AI inference?

AI inference is the work a trained model does each time you give it a prompt and it gives something back: an answer, a piece of code, an image. Training is how a model learns. Inference is how it gets used. Think of a chef: going to culinary school is training, and cooking tonight's order is inference. School happens once. Orders never stop.

Figure 1 · Two jobs, very different rhythms

1. TrainingThe model reads huge amounts of text and adjusts billions of numbers until it gets good at predicting language. Takes weeks or months on thousands of chips. Happens before the model is released.

↓
produces a file of those numbers, called the model’s weights

2. InferencePeople send questions (prompts). The model uses what it learned to write answers. Happens every time anyone uses it, all day, from millions of people and software programs.

What happens when you hit enter

A model doesn't write your answer in one go. It writes it one small piece at a time. Those pieces are called tokens. A token is a chunk of text, often a bit shorter than a word.

Figure 2 · One question, step by step

Step 1 · You ask“Why is the sky blue?”

↓

Step 2 · Split into tokensWhy is the sky blue?

↓

Step 3 · Read (“prefill”)The model reads all of your tokens at once and builds up an understanding of the question.

↓

Step 4 · Write (“decode”), one token at a timeBecause sunlight scatters …
Each new token is chosen based on everything before it, then added to the pile. Repeat until the answer is done.

That’s why answers appear word by word. The model really is producing them in sequence.

Reading is fast because the model can look at all your tokens together. Writing is slower per token because each new token has to wait for the one before it. Longer answers mean more tokens, more waiting and more cost.

Where this actually runs

Models run on GPUs, chips that are very good at doing lots of math at the same time. The model's weights have to sit in the GPU's memory, and big models need several GPUs working together. Those GPUs live in data centers, and somebody has to rent, run and power them.

Figure 3 · The inference stack

Apps and agentsA chat app, a coding tool, an AI agent doing a task.

What you see

↓

Model providersThe companies that offer a model through an API, including labs and inference companies.

What you pay for

↓

Serving softwareTakes incoming requests, groups them, keeps track of memory, and decides what runs where.

Where speed and cost are won

↓

GPUsThe chips that do the math and hold the model.

Rented by the hour

↓

Data centersBuildings with power, cooling and networking. Companies that rent out GPUs, often called neoclouds, operate here.

Built with debt

Why it costs money, and what drives the cost

Most AI providers charge per token, usually quoted per million. They typically charge more for the tokens a model writes than for the ones it reads. Behind that price is a simple fact: someone is paying for GPUs by the hour whether or not they're busy.

That makes utilization the number that matters most. A GPU that's always working spreads its cost over many answers. A GPU that sits idle costs the same and earns nothing. Serving software tries to keep chips busy by handling many people's requests together, which is called batching.

Figure 4 · Same chip, same rent (illustrative)

GPU kept busycost per answer: low

GPU mostly idlecost per answer: high

The percentages are illustrative, not measured data. The point is the shape: the bill is fixed, so the more useful work you fit in, the cheaper each answer gets.

Another cost is memory. As a conversation grows, the model keeps notes on it so it doesn't have to re-read everything for each new token. Those notes, called the KV cache, take up space on the GPU. That's one reason very long conversations and long documents cost more to serve.

The three-way tradeoff

Anyone running a model is balancing three things, and improving one usually costs you something on the others.

Figure 5 · What providers trade off

Quality: bigger models tend to give better answers. Speed: how fast the reply starts and finishes. Cost: what each answer costs to produce. Smaller or specialized models, better chips and smarter serving software all try to bend this triangle.

Why this matters for AI as a whole

Training makes headlines, but inference is where AI turns into a business. Every chat, every coding session and every step an AI agent takes is inference. When a company says it’s selling AI, mostly it’s selling inference.

That's why the numbers we track at State of Inference keep coming back to it:

Companies are committing hundreds of billions of dollars to compute they expect inference demand to fill. See what Anthropic's S-1 showed about revenue per dollar of compute.
Much of the GPU capacity is financed with debt, and that financing is under watch. See Nvidia's reported neocloud exposure and the neocloud debt test.
Prices per token have been falling while usage grows, so what businesses actually spend is the thing to watch. See our AI spending tracker.
Who sells inference, and how they're funded, is on our providers list.

Plain-English glossary

Prompt: What you send to the model: your question, plus any documents or instructions.
Token: A small chunk of text. Models read and write in tokens, and providers usually price by them.
Model weights: The billions of numbers a model learned during training. The model is, in practice, this file.
GPU: A chip built to do many calculations in parallel. The workhorse for running AI models.
Latency: How long you wait. Usually measured to the first token, and then to the whole answer.
Throughput: How much work a system gets done overall, for example tokens per second across all users.
Batching: Handling many people’s requests together on the same GPU so it stays busy.
Utilization: How much of the time a GPU is doing useful work instead of sitting idle.
KV cache: The model’s running notes on the conversation so far. It saves re-reading, and takes up GPU memory.
Open-weight model: A model whose weights are published, so anyone can download and run it on their own hardware or a provider’s.
Neocloud: A company that rents out GPU capacity, often built specifically for AI workloads.

This guide is a simplified explanation for newcomers. Real systems have more moving parts, and details vary by model and provider. Spot something wrong or unclear? Email info@stateofinference.com.