Self-host versus API

Why it matters

Every team that ships a model feature is asked, usually by someone holding a cloud invoice or a compliance questionnaire, whether it should run the model itself. The question sounds like cost and is really five questions: where the data may go, what the bill looks like as volume grows, how much control the product needs, whether the model you need is even available to run, and who carries the pager. Interviewers ask it because it is the build-versus-buy decision of this decade, and because a candidate who answers "self-host, it is cheaper" without doing the accelerator arithmetic has told them everything.

The idea

Data and compliance

Some data cannot leave a jurisdiction, a network or a named processor. Regulated health, financial and government workloads, and contracts that forbid third-party processing, decide the question before cost enters it. The nuance is that "API" no longer means one thing: providers offer regional endpoints, zero-retention terms and deployments inside your own cloud account, and a managed open-weight endpoint in-region satisfies most residency rules without a single accelerator of your own. Self-hosting is the answer only when no processor is acceptable, or when the audit needs to see the weights.

Cost at scale: the crossover

An API bills per token, so its cost is a line through the origin. Self-hosting has a fixed cost that exists at zero traffic and rises slowly after it: accelerators by the hour whether busy or idle, and engineers to run them. The two lines cross at a volume, and everything about the decision is where that crossing sits.

api_cost   = tokens_a_day x price_per_token x 30
self_host  = accelerators x 720 accelerator-hours x hourly_rate / utilisation
           + engineers x loaded_monthly_cost
crossover  = the volume at which api_cost equals self_host

Two terms move the crossover far more than the hourly rate. Utilisation: a fleet sized for the daily peak idles most of the night, and a model that serves 20 percent of its possible tokens costs five times its nominal rate per token. Engineering time: the serving stack, the upgrades and the on-call are salaries that rarely appear in the comparison. A single accelerator running a small quantised model with continuous batching can produce on the order of a few thousand tokens a second, roughly 200 million tokens a day at full utilisation, and that number is what the fleet is sized from.

The arithmetic at the two ends is stark. Fifty million tokens a day of classification on a small API tier at $0.10 per million is $5 a day, $150 a month; no accelerator running 720 hours plus a fraction of an engineer gets near it. Two billion tokens a day on a frontier tier at $8 per million is $16,000 a day, about $480,000 a month, and an open-weight model that passes the same eval on a well-utilised fleet can undercut that comfortably. Most products live between those ends, which is why the answer is usually a split rather than a side.

Latency and control

Self-hosting gives you the knobs the previous lessons described: the batch limits, a quantised checkpoint, speculative decoding tuned to your traffic, a custom fine-tune served at full weight, and a model version pinned until you choose to move. It also gives you the latency floor, because the model sits next to your service with no shared queue in front of it. An API gives you none of those directly; it gives you tiers, effort, caching and a deprecation calendar you do not control. For a product whose behaviour is tuned against one exact model, a forced upgrade is a regression you did not schedule.

Capability

The frontier models are API-only. Open-weight families, Llama, DeepSeek, Qwen, GLM and Mistral among them, close the gap on most tasks and run anywhere, but the hardest reasoning, the longest contexts and the newest capabilities arrive at the API first. The honest test is the eval set: if an open-weight model passes it at the quality bar, capability is not the deciding factor; if it does not, no amount of cost arithmetic matters.

Operations

Running a model is running a service with unusual hardware. Someone owns the serving stack and its upgrades, accelerator scheduling and the fight for capacity, model version rollouts with evals, the KV-cache and batching tuning from earlier in this module, capacity for the peak, and the pager when a replica wedges at three in the morning. A team of three cannot absorb it; a platform team of ten with other accelerator workloads often can.

The hybrid, and the middle path

Most mature deployments split. The API serves frontier tasks: hard reasoning, agent planning, anything where the eval says only the top tier passes. Self-hosted small models serve the high-volume, well-defined work: classification, extraction, embedding, routing, guardrails. Between the two sits the managed open-weight provider: open models served by someone else, per token, often in-region, with pinned versions and fine-tune hosting. It captures most of the control and compliance of self-hosting with none of the operations, and for many teams it is the right answer to a question they thought had two options.

A decision list

• Can the data leave your boundary at all? If not, self-host or a deployment inside your account.
• Does an open-weight model pass your eval at the quality bar? If not, API, and the rest is moot.
• What is the crossover at realistic utilisation with engineers counted? Below it, API or managed open-weight.
• Do you need a pinned version, a custom fine-tune at full weight, or a latency floor? Managed open-weight first, self-host if it cannot deliver.
• Who carries the pager? If nobody can, the answer is not self-host.