Azure Machine Learning

What it does

A full MLOps platform for the end-to-end lifecycle of custom machine learning models:

• Training and fine-tuning — run experiment jobs on managed GPU/CPU clusters, serverless compute, or attached Kubernetes clusters. Supports PyTorch, TensorFlow, scikit-learn, Hugging Face, and more.

• Experiment tracking — logs metrics, parameters, and artefacts for every run; compare runs in the studio UI.

• Model registry — version and govern trained models with lineage back to the training data and code that produced them.

• Managed online endpoints — deploy models as scalable, low-latency REST endpoints with autoscaling and blue/green deployment support.

• Batch endpoints — trigger high-throughput asynchronous scoring jobs over large datasets.

• Pipelines — author and schedule multi-step ML workflows (data prep → training → evaluation → registration → deployment).

• Feature store — centralise feature engineering logic for reuse across models and teams.

• Responsible AI dashboard — model explainability, fairness, error analysis, and causal analysis tools built into the platform.

When you'd use it

• You need to train or fine-tune a model on your own data rather than consuming a pre-built foundation model.

• You are running large-scale batch inference over millions of records.

• You need reproducible, auditable ML pipelines with lineage tracking for regulatory or compliance reasons.

• You want to host open-source models (Hugging Face, LLaMA, Mistral) on infrastructure you control, with autoscaling.

• You are operationalising ML at scale and need a model registry, deployment approvals, and monitoring in one platform.

Sample use case

A retail bank trains a custom credit-risk scoring model on five years of transactional data. Azure ML manages the training pipeline: data is loaded from Azure Data Lake, feature engineering runs as a pipeline step, a gradient-boosted model is trained on a GPU cluster, evaluated against a holdout set, and registered in the model registry with full lineage. The approved model is then deployed to a managed online endpoint called in real time by the loan origination system.

Example reference architecture

Pricing & How Costs Work

Azure ML is primarily a compute-consumption service. The platform management layer (workspace, pipelines, registry) has no direct charge; costs come from the compute and storage resources it uses.

Compute — the dominant cost

VM pricing follows standard Azure VM rates — the instance SKU (e.g. Standard_DS3_v2, Standard_NC24ads_A100_v4) determines the hourly rate. GPU VMs are significantly more expensive than CPU VMs.

Storage

Additional services provisioned by the workspace

Key pricing mechanics:

• Compute clusters scale to zero when idle (if min_instances = 0) — they only bill during active job runs. This is the primary cost-control mechanism for training workloads.

• Compute instances do not scale to zero automatically — a notebook VM left running overnight bills continuously. Auto-shutdown schedules are critical.

• Managed online endpoints bill continuously for every deployed instance. High availability (multiple instances, multiple deployments) multiplies cost directly.

• Spot/low-priority VMs are available for training jobs at up to 80% discount vs standard priority, with the risk of preemption. Not suitable for production endpoints.

• Reserved VM pricing (1-year or 3-year) reduces compute costs for long-running or predictable workloads like always-on online endpoints.

What Drives Costs

Cost optimisation levers:

• Set min_instances = 0 on compute clusters so they scale to zero between jobs.

• Configure auto-shutdown on compute instances — set a 30–60 minute idle timeout for developer VMs.

• Use spot/low-priority VMs for training and evaluation jobs that can tolerate preemption.

• Delete unused online endpoint deployments — each deployment accrues cost even with zero traffic.

• Right-size online endpoint instances: profile actual memory and CPU usage before choosing the VM SKU.

• Use batch endpoints instead of online endpoints for asynchronous, non-latency-sensitive scoring.

• Apply Azure Reserved VM Instances to always-on endpoint infrastructure to reduce hourly rates.

• Implement storage lifecycle policies to archive or delete old experiment outputs and model checkpoints.

• Use hyperparameter sweep budgets (max_total_trials) to cap the number of child runs in sweep jobs.

Azure Cost Data — Meters & Meter Subcategories

Azure ML appears under the Machine Learning service family in Cost Management. Resource type: Microsoft.MachineLearningServices/workspaces.

Meter subcategory notes:

• The meter subcategory identifies the compute type: Compute Cluster, Compute Instance, Managed Online Endpoint, or Batch — this is the primary way to split training costs from inference costs in billing exports.

• VM SKU names appear directly in the MeterName field (e.g. Standard NC6s v3), making it possible to see cost by VM type without additional configuration.

• Compute cluster meters only accrue during job execution (assuming min_instances = 0). In billing exports, this appears as a non-continuous cost pattern — spikes during training periods, zero otherwise. Online endpoint meters are continuous.

• Spot/low-priority compute appears on a separate meter with Low Priority in the subcategory, carrying a lower rate. This is useful for validating that sweep jobs are using spot pricing as intended.

• The Azure ML workspace itself (Microsoft.MachineLearningServices/workspaces) does not generate a billable meter — only the compute and storage resources it uses do.

• Use resource tags on the workspace (e.g. team=data-science, project=credit-risk) and ensure they are inherited by compute clusters and endpoints to enable cost attribution in Cost Management.

Common Developer Mistakes That Drive Up Costs

• Not setting min_instances = 0 on compute clusters. If the minimum node count is left at 1 or more, the cluster runs continuously even between jobs. A single idle Standard_DS3_v2 node running around the clock adds up quickly — and teams often forget to check this on clusters created during initial setup.

• Forgetting to stop compute instances (notebook VMs). Developer VMs do not auto-shut down by default. An active compute instance left running after end of day bills the full VM rate overnight and across weekends. This is the most frequent source of avoidable ML cost in practice.

• Over-provisioning GPU VMs for training. Teams default to A100 or V100 VMs for model training without profiling whether the workload is actually GPU-bound. Many training jobs run on data preprocessing or Python overhead, not GPU operations, making a fraction of the GPU time actually useful.

• Running hyperparameter sweeps without a trial budget. A sweep job with no max_total_trials cap can spin up dozens or hundreds of parallel child runs, each on a full VM. Without a budget, a careless sweep configuration can exhaust a month's compute budget in hours.

• Online endpoints with unnecessary replicas. Teams add replicas for availability without checking actual query-per-second requirements. Each replica is a full VM instance billing 24×7. An endpoint serving 10 requests/minute does not need 3 replicas.

• Using dedicated priority for training jobs that could use spot. Spot (low-priority) VMs are available at up to 80% discount for training workloads. Teams often default to dedicated priority out of habit, paying full price for jobs that could tolerate occasional preemption.

• Accumulating unused model artefacts and training outputs. Each training run writes model checkpoints, outputs, and logs to the default Storage account. Without lifecycle management, storage costs grow linearly with training history and are often overlooked.

What Could Make Your Bill Go Up or Down Next Month

Could go UP:

• A new training campaign (e.g. model retraining on fresh data) adds significant GPU-cluster hours that didn't exist last month.

• A developer left a compute instance (notebook VM) running over a long weekend.

• A hyperparameter sweep was run without a trial cap, spawning many more child runs than expected.

• An online endpoint was scaled from 1 to 3 instances for a load test and not scaled back down.

• A new team member creates their own compute cluster with min_instances > 0 without realising the cost implication.

• A model experiment uses a larger GPU SKU than the previous month's equivalent job.

Could go DOWN:

• Auto-shutdown is configured on all compute instances (e.g. 30-minute idle timeout).

• Training jobs are migrated from dedicated to low-priority (spot) VMs.

• An online endpoint is scaled from 2 instances to 1 after profiling shows peak load is well within a single instance's capacity.

• Old training outputs and model checkpoints are deleted or archived to cold storage via a lifecycle policy.

• A batch endpoint replaces an online endpoint for an asynchronous scoring workload, eliminating 24×7 compute cost.

• Serverless compute is adopted for ad-hoc training jobs, removing the need for persistent cluster provisioning.

Most Common Optimisation Techniques