What it does
Provides managed, enterprise-grade access to OpenAI's foundation models hosted entirely within Azure infrastructure:
Chat and completion models — GPT-4o, GPT-4, GPT-4 Turbo, o1, o3, and older GPT-3.5 variants for text generation, summarisation, Q&A, and reasoning tasks.
Embedding models — text-embedding-ada-002, text-embedding-3-small/large for converting text into vector representations used in semantic search and RAG pipelines.
Image generation — DALL-E 3 for generating images from text prompts.
Audio — Whisper for speech-to-text transcription; TTS models for text-to-speech synthesis.
Fine-tuning — ability to fine-tune selected base models (GPT-4o mini, GPT-3.5) on custom training data to adapt model behaviour to specific domains or styles.
Data stays within the customer's Azure tenant — no traffic is routed to OpenAI's consumer endpoints.
When you'd use it
You need GPT-class language model capabilities inside an enterprise Azure environment with data residency, private networking, and compliance controls.
You are building chat interfaces, summarisation pipelines, code assistants, or document Q&A features in your own application.
You need embeddings to power a RAG (Retrieval-Augmented Generation) pipeline over private data.
You want to fine-tune a model on domain-specific data to improve accuracy or reduce prompt length.
Sample use case
A large insurance company builds an internal claims assistant. Adjusters describe a claim in natural language; the assistant uses GPT-4o (via Azure OpenAI) to extract key facts, cross-reference policy terms retrieved from Azure AI Search, and draft a recommended settlement decision. The entire interaction is logged for audit, and no customer data ever leaves the company's Azure tenant.
Example reference architecture
.png)
Pricing & How Costs Work
Azure OpenAI Service has two distinct billing models: Pay-as-you-go (PAYG) and Provisioned Throughput Units (PTUs). The right choice depends on traffic volume, latency requirements, and predictability of spend.
Pay-as-you-go — per token billing
PAYG charges per token consumed, split into input (prompt) tokens and output (completion) tokens. Output tokens are more expensive than input tokens for most models.
Model family | Billing unit | Notes |
|---|---|---|
GPT-4o | Per 1,000 input tokens + per 1,000 output tokens | Most capable, highest rate |
GPT-4o mini | Per 1,000 input tokens + per 1,000 output tokens | Cheaper, lower capability |
o1 / o3 (reasoning) | Per 1,000 input tokens + per 1,000 output tokens | Also bills for internal reasoning tokens which are not returned in the response but are consumed |
text-embedding-3-large/small | Per 1,000 input tokens | Output tokens do not apply to embeddings |
DALL-E 3 | Per image generated | Varies by resolution (1024×1024, 1792×1024, etc.) |
Whisper | Per audio minute | Transcription priced per minute of audio |
TTS | Per 1 million characters | Text-to-speech synthesis |
Fine-tuned model inference | Per 1,000 tokens + hosting fee | Additional per-hour charge for keeping a fine-tuned deployment active |
Key mechanics:
A token is approximately 4 characters of English text, or ¾ of a word. A 1,000-word document is roughly 1,300–1,400 tokens.
The system prompt, conversation history, and any retrieved context (RAG chunks) all count as input tokens on every call. Long contexts and multi-turn conversations accumulate cost rapidly.
Reasoning tokens (o1/o3 models): the model thinks internally before responding. These internal reasoning steps are billed as input tokens but are invisible in the API response — a single o1 call can consume 5–20× the visible token count.
Context window size determines the maximum tokens per call but does not itself affect cost — only tokens actually submitted and generated are billed.
Cached prompt tokens: Azure OpenAI supports prompt caching on some models. Repeated identical prompt prefixes are charged at a reduced rate (typically 50% discount on the cached portion).
Provisioned Throughput Units (PTUs) — reserved capacity
PTUs reserve a fixed amount of model throughput (tokens per minute) for your exclusive use. This is a capacity commitment, not a consumption model.
Aspect | PTU model |
|---|---|
Billing | Per PTU per hour (committed, regardless of actual usage) |
Commitment terms | Hourly (on-demand PTU — flexible), 1-month, or 1-year reserved (lower hourly rate) |
Benefit | Predictable latency; no throttling from shared capacity; consistent throughput |
Drawback | You pay for capacity even when idle; requires forecasting peak token throughput |
Minimum purchase | Varies by model — typically 25–50 PTUs minimum |
PTUs are suited to production workloads with consistent, high-volume traffic. PAYG is suited to variable or development workloads.
What Drives Costs
Cost driver | Why it matters |
|---|---|
Input token volume | Every token in the prompt — system message, history, RAG context, user input — is billed on every call. Long system prompts repeated across thousands of calls are a major hidden cost. |
Output token volume | The model's generated response. Verbose completions cost more; instruct the model to be concise when appropriate. |
Model selection | GPT-4o costs significantly more per token than GPT-4o mini. Using a more powerful model than the task requires is a common source of unnecessary spend. |
Reasoning tokens (o1/o3) | Internal thinking tokens are billed but invisible — o1/o3 calls can cost 10–20× an equivalent GPT-4o call for simple tasks. |
Conversation history length | Multi-turn chat applications re-send the full conversation history on every turn. A 20-turn conversation means turn 20 sends ~19 turns of history as input tokens. |
RAG chunk size and count | Retrieval-augmented calls inject document chunks into the prompt. Large chunks or a high top-k value inflate input token counts per call. |
Fine-tuned deployment hosting | Active fine-tuned model deployments accrue an hourly charge even with zero traffic. |
PTU idle capacity | Provisioned capacity is billed whether used or not — under-utilised PTUs waste committed spend. |
Image resolution (DALL-E) | Larger image dimensions cost more per generation. |
Cost optimisation levers:
Right-size the model: use GPT-4o mini or GPT-3.5 for classification, extraction, or simple Q&A tasks; reserve GPT-4o for reasoning-heavy steps.
Trim conversation history: implement a sliding window or summarisation strategy to cap the number of turns re-sent as context.
Optimise RAG chunk strategy: reduce chunk size or top-k count; use reranking to pass only the most relevant chunks to the LLM.
Use prompt caching: structure prompts so the static system prompt prefix is identical across calls to benefit from cached token discounts.
Delete idle fine-tuned deployments: hosting charges stop when the deployment is removed; redeploy on demand.
Use PTUs at high utilisation only: PTUs break even versus PAYG at roughly 60–70% utilisation; below that, PAYG is cheaper.
Avoid o1/o3 for tasks that don't require deep reasoning: the reasoning token overhead makes these models extremely expensive for simple tasks.
Azure Cost Data — Meters & Meter Subcategories
Azure OpenAI Service appears under the Cognitive Services service family in Cost Management. Resource type: Microsoft.CognitiveServices/accounts with kind OpenAI.
Meter name | Meter subcategory | What it counts |
|---|---|---|
GPT-4o Input Tokens | Standard | Input (prompt) tokens consumed by GPT-4o calls |
GPT-4o Output Tokens | Standard | Output (completion) tokens generated by GPT-4o calls |
GPT-4o mini Input Tokens | Standard | Input tokens for GPT-4o mini calls |
GPT-4o mini Output Tokens | Standard | Output tokens for GPT-4o mini calls |
o1 Input Tokens | Standard | Input + reasoning tokens for o1 model calls |
o1 Output Tokens | Standard | Output tokens for o1 model calls |
text-embedding-3-large Tokens | Standard | Tokens submitted to the embedding model |
text-embedding-ada-002 Tokens | Standard | Tokens submitted to the legacy embedding model |
DALL-E 3 Images | Standard | Images generated, grouped by resolution |
Whisper Audio Minutes | Standard | Minutes of audio transcribed |
Fine-tuned Model Hosting | Standard | Hourly charge per active fine-tuned deployment |
Fine-tuned Model Training | Standard | Token count in training dataset during a fine-tuning job |
Provisioned Throughput Units | PTU (Hourly / Monthly / Annual) | PTUs reserved; meter subcategory reflects commitment term |
Meter subcategory notes:
Each model has separate input and output token meters — this is important for cost analysis because input:output ratios vary significantly by use case (summarisation generates fewer output tokens; long-form generation generates many).
Reasoning tokens on o1/o3 are billed through the input token meter for those models. There is no separate "reasoning token" meter — the quantity will appear higher than the visible prompt length.
PTU meters use a subcategory reflecting the commitment type (
Provisioned Throughput Unit - Hourly,- Monthly,- Yearly). PTU cost appears as a flat quantity even in periods of zero inference activity.The
MeterNameincludes the model name and token direction (input/output), making it possible to build a cost-per-model breakdown directly from billing export data without needing Azure Monitor metrics.Multiple model deployments within the same Azure OpenAI resource generate separate meter rows in billing exports, distinguishable by
MeterNameandAdditionalInfofields which include the deployment name.
Common Developer Mistakes That Drive Up Costs
Using GPT-4o for every task regardless of complexity. Teams often default to the most capable model. Using GPT-4o for classification, routing, or simple extraction tasks that GPT-4o mini or even a Cognitive Services API would handle equally well is the single most common source of unnecessary spend.
Sending full conversation history on every turn. Multi-turn chat implementations often re-send the entire conversation transcript as input tokens on every request. A 20-turn conversation means turn 20 submits ~19 turns as context — the input token cost grows quadratically as the conversation lengthens.
Verbose system prompts repeated at scale. A 1,000-token system prompt sent on every request in a high-volume pipeline costs as much as 1,000 tokens of user input per call. Teams rarely account for this when estimating costs.
Not accounting for reasoning tokens in o1/o3. Developers see a short prompt and expect a predictable cost, not realising that o1/o3 models consume internal reasoning tokens (invisible in the response) that can be 5–20× the visible prompt length.
Leaving fine-tuned deployments active when idle. Fine-tuned model deployments accrue an hourly hosting charge whether or not they receive traffic. Deployments created for a proof-of-concept and then forgotten are a common source of persistent background cost.
Oversized RAG chunks passed as context. Retrieving large document chunks and passing all of them to the model inflates input tokens per call. Teams often set top-k and chunk size conservatively high without profiling the actual retrieval quality trade-off.
Not enabling prompt caching. For workloads with a static system prompt prefix, prompt caching offers ~50% discount on the cached portion. Failing to structure prompts to take advantage of this is a straightforward missed saving.
What Could Make Your Bill Go Up or Down Next Month
Could go UP:
A new feature launch or marketing campaign drives a sudden increase in user traffic and therefore LLM call volume.
A developer increases the
max_tokenslimit in completions "just in case", causing the model to generate longer responses across all calls.A new o1/o3 model is adopted for a use case that previously used GPT-4o — reasoning token costs are significantly higher.
Conversation history pruning logic has a bug and starts passing unbounded history, silently inflating input tokens per call.
A fine-tuned deployment is created for testing and not cleaned up at sprint end.
PTU utilisation drops below the break-even threshold (typically ~65%), making PTU more expensive than equivalent PAYG consumption would have been.
Could go DOWN:
Prompt caching is enabled and the system prompt accounts for a large share of input tokens per call.
Model is downgraded from GPT-4o to GPT-4o mini for a subset of lower-complexity tasks.
Conversation history is capped with a sliding window or summarisation strategy.
RAG chunk size or top-k count is reduced after retrieval quality testing shows smaller context is sufficient.
A PTU reservation is purchased, replacing PAYG for a consistently high-volume workload.
A seasonal traffic drop (e.g. post-holiday) reduces call volume on a PAYG deployment.
Most Common Optimisation Techniques
Technique | Mechanism | Typical saving |
|---|---|---|
Right-size model selection | Use GPT-4o mini or AI Language for simpler tasks; reserve GPT-4o/o1 for reasoning-heavy steps | 50–90% reduction on affected calls |
Prompt caching | Structure prompts so the system prefix is identical across calls to benefit from cached input token discounts | ~50% on input tokens for the cached portion |
Conversation history pruning | Implement a sliding window or summarise older turns; cap history at N tokens | Linear reduction in input tokens per turn as conversation length grows |
RAG optimisation | Reduce chunk size; lower top-k; use a reranker to filter before LLM call | 20–50% reduction in input tokens per RAG call |
PTU for high-volume predictable workloads | Replace PAYG with PTUs when sustained utilisation exceeds ~65% | 30–50% reduction vs PAYG at equivalent throughput |
Semantic caching via APIM | Cache LLM responses at the gateway layer for semantically similar queries | Highly variable — 10–40% in FAQ or support scenarios |
Delete idle fine-tuned deployments | Remove deployments not receiving traffic; redeploy on demand | Eliminates 100% of hosting cost for idle deployments |
Avoid o1/o3 where reasoning depth is not needed | Use GPT-4o for tasks that don't require deep chain-of-thought reasoning | Reduces cost by 5–20× on affected calls |