Understanding Token Economics for Large Language Models
Large language model (LLM) developers and operators often face a common challenge: determining how much a single token actually costs. While API providers publish pricing tiers, the true expense depends on a combination of hardware, workload characteristics, and runtime optimisations. This article synthesises practical measurement techniques and recent market data to give a clear picture of token costs for both self‑hosted and API‑based deployments.
Calculating Cost on Self‑Hosted GPUs
When running LLM inference on dedicated GPUs, the cost per million tokens can be derived from a simple formula:
$ per million tokens = ($/hr ÷ 3600) ÷ tokens per second × 1,000,000
The difficulty lies in accurately measuring tokens per second. This metric fluctuates with batch size, quantisation, engine version, GPU model, and concurrent system activity.
- Batch size – Larger batches improve throughput but may increase GPU utilisation.
- Quantisation – Reducing precision can boost speed at a potential quality cost.
- Engine flags – Settings such as
max_num_seqsin vLLM directly impact throughput. - Caching – Prefix caches can make repeated prompts appear cheaper; each measurement should use a cold start unless cache performance is being evaluated.
To account for variability, measurements should be repeated several times. A single run can be misleading because the same hardware can yield different token‑per‑second rates due to background processes or temperature throttling. For example, an Ollama server produced $8.77, $8.86, $11.26, and $10.02 per million tokens across four runs.
Automated Measurement with Throttle‑Pro
A tool named Throttle‑Pro automates the process of measuring token cost, applying controlled changes, and determining statistically significant improvements. The typical workflow is:
pipx install throttle-pro throttle check --url http://localhost:8000 --model <your-model> --gpu-hourly-rate <your $/hr>
Running the command three to four times establishes a reliable baseline before any configuration is altered. The tool then reports whether a change is CHEAPER, MORE EXPENSIVE, or a NO WINNER if the improvement falls within the noise floor.
Market‑Wide Token Prices (October 2026)
API pricing continues to evolve rapidly. Recent data show that efficient models such as DeepSeek, Gemini Flash, and GPT mini classes charge under $1 per million input tokens, whereas frontier models (e.g., GPT‑5.6 Sol, Claude Opus 5) range from $5 to $10 per million input tokens. Output tokens are consistently more expensive, often three to ten times the input rate.
Key highlights include:
- GPT‑5.6 Luna: $0.20 / million input, $1.20 / million output (August 2026).
- Gemini 3.8 Flash: $0.75 / million input, $3.75 / million output (September 2026).
- Frontier models: $5–$10 / million input, $25–$50 / million output.
These figures underscore the importance of selecting an appropriate model tier and optimizing token usage. For developers seeking real‑time comparisons, tools like Price Per Token and LLM Prices provide up‑to‑date rate tables across providers.
Practical Takeaways
Accurate cost estimation requires repeated, controlled measurements and awareness of hidden factors such as caching and system noise. Self‑hosted environments benefit most from tools that automate benchmarking and statistical analysis.
When evaluating LLM deployments:
- Run multiple benchmark iterations to establish a confidence interval.
- Measure token throughput under realistic batch sizes and workload mixes.
- Consider the trade‑off between model quality and token cost; lower‑cost models may suffice for many use cases.
- Regularly compare on‑premise costs against API pricing, especially as market rates shift.
By integrating systematic measurement with current pricing data, operators can make informed decisions that balance performance, quality, and cost.
Leave a Reply