Self-hosted LLM or API: when running a model in your own cloud pays off

Three options, not two

The question “own model or API” is framed too narrowly. In practice there are three options, and they differ mainly in who sees the data and who carries the operations.

Moreover, the decision is not made once for the whole company, but separately for each workload. An internal chatbot over documentation, data extraction from contracts and a coding assistant have different volumes, different data sensitivity and different demands on model quality.

What open-weight models can do today

In October 2026, the range of models with open weights is broad and licensed for commercial use too. A few examples verified in primary sources:

• gpt-oss from OpenAI – released on 5 August 2025 under the Apache 2.0 license; according to OpenAI, the larger gpt-oss-120b variant runs on a single GPU with 80 GB of memory, while the smaller gpt-oss-20b gets by with 16 GB.
• DeepSeek V4-Flash – according to the model card on Hugging Face, MIT license, 284 billion parameters (13 billion active) and a one-million-token context.
• Mistral – the model overview lists Mistral Small 4 and Mistral Large 3 among its open-weight models under Apache 2.0. According to a report from 6 October 2026, Mistral Large 4 is in public preview via the API, and the vendor has promised open weights by the end of October 2026.

Two things need to be said openly. First, model size determines hardware: a model that fits on a single 48 GB or 80 GB card is in a different cost league from a model with hundreds of billions of parameters that needs a whole eight-GPU node. Second, public leaderboards do not tell you how a model will perform on your tasks and in Czech. Before you start counting money, build your own test set from real inputs and use it to compare the candidate with the model you use via an API.

Break-even: a method instead of a magic number

An API is paid for by consumption, your own GPUs by time. The break-even point therefore depends on one thing: how many tokens per month you actually push through hardware you pay for around the clock.

monthly GPU cost      = hourly node price × 720 h
blended API price     = (input share × input price) + (output share × output price)   [USD / 1M tokens]
break-even volume     = monthly GPU cost / blended API price                           [M tokens / month]
required throughput   = break-even volume / (720 × 3600 s × utilization)              [tokens / s]

A worked example with public price lists

The inputs come from public AWS price lists as of 6 October 2026; everything else is an assumption and is labeled as such.

• GPU: a p5.48xlarge instance (8× H100) under EC2 Capacity Blocks in Stockholm cost USD 37.76 per hour (USD 4.72 per accelerator) until 6 October 2026. From 7 October 2026, AWS announces on the same page a uniform rate of USD 5.97 per accelerator in all regions for the P5 family, i.e. 8 × 5.97 = USD 47.76 per hour; the calculation below uses this new rate. This is the price of reserved time blocks, which changes regularly; the on-demand price differs.
• API: the open-weight model DeepSeek v3.2 costs USD 0.74 per million input tokens and USD 2.22 per million output tokens on Amazon Bedrock in the same region.
• Assumption: a 3 : 1 ratio of input to output tokens (typical for RAG and agents with long context; yours may differ).

The calculation deliberately contains no “typical” GPU performance figure. Throughput depends on the model, quantization, context length and degree of concurrency, and the only trustworthy value is the one you measure with your own load test. The question then becomes simple: can your model sustain the required throughput on this node over the long term, and do you even have that much traffic?

The middle path: models in hyperscalers' EU regions

For most companies, the most sensible starting point is models run by the cloud provider they already have a contract, identities and networking with. But the deployment details matter.

Azure AI Foundry

The deployment type determines where data is processed. According to Microsoft's documentation, with the Global type data may be processed in any Azure region, with the Data Zone type only within the given zone (US, EU or APAC), and with the regional type in the geography chosen by the customer. The EU Data Zone follows the EU Data Boundary, which may also include EFTA countries such as Norway and Switzerland. Data at rest stays in the chosen geography.

On retention: according to the data privacy page, prompts and responses are not available to OpenAI or other model providers and are not used to train foundation models. However, the service includes abuse monitoring, during which prompts and responses may be stored for human review. This step is dropped only for customers approved for “modified abuse monitoring”, which has to be applied for.

Amazon Bedrock

With Bedrock, watch the inference profile type. According to the cross-region inference documentation, a geographic profile (for example EU) processes the request in one of the regions of that geography, whereas a global profile may pick any supported commercial region worldwide. For accounts with sensitive data, it is therefore advisable to block global profiles by policy.

Bedrock controls retention through the data_retention_mode parameter. According to the documentation, the value none means that AWS does not write requests or responses to persistent storage and does not share them with the model provider.

When self-hosting makes sense even without a financial break-even

Price per token is only one criterion. Self-hosting tends to be justified in these situations:

• Data must not leave your environment. Contractual obligations to customers, classified or especially sensitive information, or an internal policy that does not allow even temporary storage at a third party. Your own model in your VPC or VNet is the only option where you control retention entirely yourself.
• You need a stable model version. With a vendor API you have no control over the version, and older models get retired. Weights you have stored will not change until you change them yourself – which matters for validated processes and audit trails.
• It is a narrow, high-volume task. Classification, extraction, summarization or embeddings can often be handled by a small model on a single card, and the break-even calculation comes out completely differently than for a large model.
• You need fine-tuning or your own adapters and full control over what happens to them.
• You need control over latency. Dedicated capacity close to the application removes shared limits and response-time fluctuations. That holds, however, only if your capacity is sized for peak load.

When not to do it

• Traffic is bursty or small – the GPU would sit idle most of the time.
• The task requires a frontier model (complex agentic workflows, demanding reasoning over code) and the open-weight candidate falls short on your test set.
• The team has no experience with GPUs on Kubernetes and no capacity to staff on-call for yet another critical service.
• The real problem is not the model but disorganized infrastructure and data – more on that in our article on infrastructure chaos before deploying AI.

What to serve the model with

Choosing an inference server has become simpler over the past year.

What matters in practice is that both vLLM and KServe offer an OpenAI-compatible API. If your application calls the model through an internal gateway with this interface, you can switch between an API and self-hosting workload by workload, and the decision is not irreversible.

# Minimal vLLM start with an OpenAI-compatible API (model only as an example)
vllm serve openai/gpt-oss-20b \
  --host 0.0.0.0 --port 8000 \
  --max-model-len 32768 \
  --api-key "$VLLM_API_KEY"

# The application only changes the base URL:
# OPENAI_BASE_URL=http://llm-gateway.internal:8000/v1

Operating costs people forget

The hourly GPU price is the visible item. The less visible ones decide whether someone cancels the project six months in.

• Capacity availability. GPU instances are not always available in your chosen region. Sort out quotas, reservations and a fallback region before you sign an internal SLA, not after the first outage.
• Cold start. Large model weights run to tens or hundreds of gigabytes. Downloading them and loading them into memory determines how fast you can scale and how long recovery takes after a node failure. For large models, scale-to-zero is more theory than practice.
• Upgrades. GPU drivers, the device plugin, the inference server and new model versions form another dependency chain alongside regular cluster upgrades. Rolling out a new model version deserves the same discipline as zero-downtime deployment for applications.
• Evaluation. Every change of model, quantization or prompt needs a quality regression test. Without one, you will not notice that outputs have gotten worse.
• Security and audit. Your own model means your own prompt logs. You have to decide what gets logged, who has access to it and how long it is kept – exactly what you held against the vendor with the API. This ties in with security monitoring and logging.
• Weight provenance. Download weights and container images from verified sources, keep them in your own registry, and restrict egress from inference pods with a network policy.
• People. The combination of Kubernetes, GPU and LLM know-how is rare. Allow time for onboarding, and keep in mind that the service needs backup coverage.

What to do on Monday

Key takeaways

• Break-even is a question of utilization, not GPU price. An 8× H100 node at USD 47.76/h only pays for itself against an API price of USD 1.11 per million tokens at around 31 billion tokens per month – and that is without redundancy or staff time.
• Measure performance, do not borrow it. Throughput depends on the model, quantization and concurrency; only your own measured value belongs in the calculation.
• The middle path requires the right configuration. EU Data Zone or a regional deployment in Azure, a geographic inference profile in Bedrock; global variants may process data outside the EU.
• Verify retention for every model. The newest models are starting to come with mandatory retention of up to 30 days for abuse-monitoring purposes.
• Self-hosting is justified mainly by data and version stability. The financial argument works for small models with large, steady volume.
• Do not choose TGI for new projects. The standard for production serving is vLLM, on Kubernetes possibly with KServe.
• An OpenAI-compatible gateway keeps the door open. It lets you move workloads between APIs and self-hosting as prices and terms change.