--- title: "Azure Open AI" slug: "azure-open-ai" updated: 2026-07-13T15:58:27Z published: 2026-07-13T15:58:27Z canonical: "azure-cost-management-playbook.turbo360.com/azure-open-ai" --- > ## Documentation Index > Fetch the complete documentation index at: https://azure-cost-management-playbook.turbo360.com/llms.txt > Use this file to discover all available pages before exploring further. # Azure Open AI ## What it does Provides managed, enterprise-grade access to OpenAI's foundation models hosted entirely within Azure infrastructure: - **Chat and completion models** — GPT-4o, GPT-4, GPT-4 Turbo, o1, o3, and older GPT-3.5 variants for text generation, summarisation, Q&A, and reasoning tasks. - **Embedding models** — text-embedding-ada-002, text-embedding-3-small/large for converting text into vector representations used in semantic search and RAG pipelines. - **Image generation** — DALL-E 3 for generating images from text prompts. - **Audio** — Whisper for speech-to-text transcription; TTS models for text-to-speech synthesis. - **Fine-tuning** — ability to fine-tune selected base models (GPT-4o mini, GPT-3.5) on custom training data to adapt model behaviour to specific domains or styles. Data stays within the customer's Azure tenant — no traffic is routed to OpenAI's consumer endpoints. ## When you'd use it - You need GPT-class language model capabilities inside an enterprise Azure environment with data residency, private networking, and compliance controls. - You are building chat interfaces, summarisation pipelines, code assistants, or document Q&A features in your own application. - You need embeddings to power a RAG (Retrieval-Augmented Generation) pipeline over private data. - You want to fine-tune a model on domain-specific data to improve accuracy or reduce prompt length. ## Sample use case A large insurance company builds an internal claims assistant. Adjusters describe a claim in natural language; the assistant uses GPT-4o (via Azure OpenAI) to extract key facts, cross-reference policy terms retrieved from Azure AI Search, and draft a recommended settlement decision. The entire interaction is logged for audit, and no customer data ever leaves the company's Azure tenant. ## Example reference architecture ![](https://cdn.document360.io/4b481a5c-c963-4b45-80c8-bc19500830f4/Images/Documentation/image(77).png) ## Pricing & How Costs Work Azure OpenAI Service has two distinct billing models: **Pay-as-you-go (PAYG)** and **Provisioned Throughput Units (PTUs)**. The right choice depends on traffic volume, latency requirements, and predictability of spend. ### Pay-as-you-go — per token billing PAYG charges per **token** consumed, split into input (prompt) tokens and output (completion) tokens. Output tokens are more expensive than input tokens for most models. | **Model family** | **Billing unit** | **Notes** | | --- | --- | --- | | GPT-4o | Per 1,000 input tokens + per 1,000 output tokens | Most capable, highest rate | | GPT-4o mini | Per 1,000 input tokens + per 1,000 output tokens | Cheaper, lower capability | | o1 / o3 (reasoning) | Per 1,000 input tokens + per 1,000 output tokens | Also bills for internal **reasoning tokens** which are not returned in the response but are consumed | | text-embedding-3-large/small | Per 1,000 input tokens | Output tokens do not apply to embeddings | | DALL-E 3 | Per image generated | Varies by resolution (1024×1024, 1792×1024, etc.) | | Whisper | Per audio minute | Transcription priced per minute of audio | | TTS | Per 1 million characters | Text-to-speech synthesis | | Fine-tuned model inference | Per 1,000 tokens + hosting fee | Additional per-hour charge for keeping a fine-tuned deployment active | **Key mechanics:** - A **token** is approximately 4 characters of English text, or ¾ of a word. A 1,000-word document is roughly 1,300–1,400 tokens. - The **system prompt**, **conversation history**, and any **retrieved context** (RAG chunks) all count as input tokens on every call. Long contexts and multi-turn conversations accumulate cost rapidly. - **Reasoning tokens** (o1/o3 models): the model thinks internally before responding. These internal reasoning steps are billed as input tokens but are invisible in the API response — a single o1 call can consume 5–20× the visible token count. - **Context window size** determines the maximum tokens per call but does not itself affect cost — only tokens actually submitted and generated are billed. - **Cached prompt tokens:** Azure OpenAI supports prompt caching on some models. Repeated identical prompt prefixes are charged at a reduced rate (typically 50% discount on the cached portion). ### Provisioned Throughput Units (PTUs) — reserved capacity PTUs reserve a fixed amount of model throughput (tokens per minute) for your exclusive use. This is a capacity commitment, not a consumption model. | **Aspect** | **PTU model** | | --- | --- | | Billing | Per PTU per hour (committed, regardless of actual usage) | | Commitment terms | Hourly (on-demand PTU — flexible), 1-month, or 1-year reserved (lower hourly rate) | | Benefit | Predictable latency; no throttling from shared capacity; consistent throughput | | Drawback | You pay for capacity even when idle; requires forecasting peak token throughput | | Minimum purchase | Varies by model — typically 25–50 PTUs minimum | PTUs are suited to production workloads with consistent, high-volume traffic. PAYG is suited to variable or development workloads. ## What Drives Costs | **Cost driver** | **Why it matters** | | --- | --- | | **Input token volume** | Every token in the prompt — system message, history, RAG context, user input — is billed on every call. Long system prompts repeated across thousands of calls are a major hidden cost. | | **Output token volume** | The model's generated response. Verbose completions cost more; instruct the model to be concise when appropriate. | | **Model selection** | GPT-4o costs significantly more per token than GPT-4o mini. Using a more powerful model than the task requires is a common source of unnecessary spend. | | **Reasoning tokens (o1/o3)** | Internal thinking tokens are billed but invisible — o1/o3 calls can cost 10–20× an equivalent GPT-4o call for simple tasks. | | **Conversation history length** | Multi-turn chat applications re-send the full conversation history on every turn. A 20-turn conversation means turn 20 sends ~19 turns of history as input tokens. | | **RAG chunk size and count** | Retrieval-augmented calls inject document chunks into the prompt. Large chunks or a high top-k value inflate input token counts per call. | | **Fine-tuned deployment hosting** | Active fine-tuned model deployments accrue an hourly charge even with zero traffic. | | **PTU idle capacity** | Provisioned capacity is billed whether used or not — under-utilised PTUs waste committed spend. | | **Image resolution (DALL-E)** | Larger image dimensions cost more per generation. | **Cost optimisation levers:** - **Right-size the model:** use GPT-4o mini or GPT-3.5 for classification, extraction, or simple Q&A tasks; reserve GPT-4o for reasoning-heavy steps. - **Trim conversation history:** implement a sliding window or summarisation strategy to cap the number of turns re-sent as context. - **Optimise RAG chunk strategy:** reduce chunk size or top-k count; use reranking to pass only the most relevant chunks to the LLM. - **Use prompt caching:** structure prompts so the static system prompt prefix is identical across calls to benefit from cached token discounts. - **Delete idle fine-tuned deployments:** hosting charges stop when the deployment is removed; redeploy on demand. - **Use PTUs at high utilisation only:** PTUs break even versus PAYG at roughly 60–70% utilisation; below that, PAYG is cheaper. - **Avoid o1/o3 for tasks that don't require deep reasoning:** the reasoning token overhead makes these models extremely expensive for simple tasks. ## Azure Cost Data — Meters & Meter Subcategories Azure OpenAI Service appears under the **Cognitive Services** service family in Cost Management. Resource type: `Microsoft.CognitiveServices/accounts` with kind `OpenAI`. | **Meter name** | **Meter subcategory** | **What it counts** | | --- | --- | --- | | GPT-4o Input Tokens | Standard | Input (prompt) tokens consumed by GPT-4o calls | | GPT-4o Output Tokens | Standard | Output (completion) tokens generated by GPT-4o calls | | GPT-4o mini Input Tokens | Standard | Input tokens for GPT-4o mini calls | | GPT-4o mini Output Tokens | Standard | Output tokens for GPT-4o mini calls | | o1 Input Tokens | Standard | Input + reasoning tokens for o1 model calls | | o1 Output Tokens | Standard | Output tokens for o1 model calls | | text-embedding-3-large Tokens | Standard | Tokens submitted to the embedding model | | text-embedding-ada-002 Tokens | Standard | Tokens submitted to the legacy embedding model | | DALL-E 3 Images | Standard | Images generated, grouped by resolution | | Whisper Audio Minutes | Standard | Minutes of audio transcribed | | Fine-tuned Model Hosting | Standard | Hourly charge per active fine-tuned deployment | | Fine-tuned Model Training | Standard | Token count in training dataset during a fine-tuning job | | Provisioned Throughput Units | PTU (Hourly / Monthly / Annual) | PTUs reserved; meter subcategory reflects commitment term | **Meter subcategory notes:** - Each model has **separate input and output token meters** — this is important for cost analysis because input:output ratios vary significantly by use case (summarisation generates fewer output tokens; long-form generation generates many). - Reasoning tokens on o1/o3 are billed through the **input token meter** for those models. There is no separate "reasoning token" meter — the quantity will appear higher than the visible prompt length. - PTU meters use a subcategory reflecting the commitment type (`Provisioned Throughput Unit - Hourly`, `- Monthly`, `- Yearly`). PTU cost appears as a flat quantity even in periods of zero inference activity. - The `MeterName` includes the model name and token direction (input/output), making it possible to build a cost-per-model breakdown directly from billing export data without needing Azure Monitor metrics. - Multiple model deployments within the same Azure OpenAI resource generate separate meter rows in billing exports, distinguishable by `MeterName` and `AdditionalInfo` fields which include the deployment name. --- ## Common Developer Mistakes That Drive Up Costs - **Using GPT-4o for every task regardless of complexity.** Teams often default to the most capable model. Using GPT-4o for classification, routing, or simple extraction tasks that GPT-4o mini or even a Cognitive Services API would handle equally well is the single most common source of unnecessary spend. - **Sending full conversation history on every turn.** Multi-turn chat implementations often re-send the entire conversation transcript as input tokens on every request. A 20-turn conversation means turn 20 submits ~19 turns as context — the input token cost grows quadratically as the conversation lengthens. - **Verbose system prompts repeated at scale.** A 1,000-token system prompt sent on every request in a high-volume pipeline costs as much as 1,000 tokens of user input per call. Teams rarely account for this when estimating costs. - **Not accounting for reasoning tokens in o1/o3.** Developers see a short prompt and expect a predictable cost, not realising that o1/o3 models consume internal reasoning tokens (invisible in the response) that can be 5–20× the visible prompt length. - **Leaving fine-tuned deployments active when idle.** Fine-tuned model deployments accrue an hourly hosting charge whether or not they receive traffic. Deployments created for a proof-of-concept and then forgotten are a common source of persistent background cost. - **Oversized RAG chunks passed as context.** Retrieving large document chunks and passing all of them to the model inflates input tokens per call. Teams often set top-k and chunk size conservatively high without profiling the actual retrieval quality trade-off. - **Not enabling prompt caching.** For workloads with a static system prompt prefix, prompt caching offers ~50% discount on the cached portion. Failing to structure prompts to take advantage of this is a straightforward missed saving. --- ## What Could Make Your Bill Go Up or Down Next Month **Could go UP:** - A new feature launch or marketing campaign drives a sudden increase in user traffic and therefore LLM call volume. - A developer increases the `max_tokens` limit in completions "just in case", causing the model to generate longer responses across all calls. - A new o1/o3 model is adopted for a use case that previously used GPT-4o — reasoning token costs are significantly higher. - Conversation history pruning logic has a bug and starts passing unbounded history, silently inflating input tokens per call. - A fine-tuned deployment is created for testing and not cleaned up at sprint end. - PTU utilisation drops below the break-even threshold (typically ~65%), making PTU more expensive than equivalent PAYG consumption would have been. **Could go DOWN:** - Prompt caching is enabled and the system prompt accounts for a large share of input tokens per call. - Model is downgraded from GPT-4o to GPT-4o mini for a subset of lower-complexity tasks. - Conversation history is capped with a sliding window or summarisation strategy. - RAG chunk size or top-k count is reduced after retrieval quality testing shows smaller context is sufficient. - A PTU reservation is purchased, replacing PAYG for a consistently high-volume workload. - A seasonal traffic drop (e.g. post-holiday) reduces call volume on a PAYG deployment. --- ## Most Common Optimisation Techniques | **Technique** | **Mechanism** | **Typical saving** | | --- | --- | --- | | **Right-size model selection** | Use GPT-4o mini or AI Language for simpler tasks; reserve GPT-4o/o1 for reasoning-heavy steps | 50–90% reduction on affected calls | | **Prompt caching** | Structure prompts so the system prefix is identical across calls to benefit from cached input token discounts | ~50% on input tokens for the cached portion | | **Conversation history pruning** | Implement a sliding window or summarise older turns; cap history at N tokens | Linear reduction in input tokens per turn as conversation length grows | | **RAG optimisation** | Reduce chunk size; lower top-k; use a reranker to filter before LLM call | 20–50% reduction in input tokens per RAG call | | **PTU for high-volume predictable workloads** | Replace PAYG with PTUs when sustained utilisation exceeds ~65% | 30–50% reduction vs PAYG at equivalent throughput | | **Semantic caching via APIM** | Cache LLM responses at the gateway layer for semantically similar queries | Highly variable — 10–40% in FAQ or support scenarios | | **Delete idle fine-tuned deployments** | Remove deployments not receiving traffic; redeploy on demand | Eliminates 100% of hosting cost for idle deployments | | **Avoid o1/o3 where reasoning depth is not needed** | Use GPT-4o for tasks that don't require deep chain-of-thought reasoning | Reduces cost by 5–20× on affected calls |