Complex pipeline scenarios
Develop and implement complex pipeline scenarios
The objective names hybrid pipelines, VM templates, and self-hosted runners or agents. These are the scenarios where the simple hosted-agent story runs out.
Hybrid: cloud control plane, on-premises execution
The usual shape is Azure Pipelines (or GitHub Actions) orchestrating, while the work runs on machines you own inside your network.
The critical property: the agent initiates its control connection outbound. No internet-initiated inbound port is needed for Azure Pipelines to assign work. Traffic from the agent to a database, server, or other deployment target is a separate path, so the agent still needs private line of sight to those targets. A GitHub self-hosted runner follows the same pull model and requires outbound HTTPS over port 443 to GitHub.
Environments with VM resources
Registering VMs as resources in an environment gives you deployment history per
machine and unlocks the rolling strategy — which, as covered elsewhere, is
VM-only. Tags on the VM resources let a deployment job target a subset.
Scaling self-hosted capacity
| Approach | Trade-off |
|---|---|
| Fixed agent pool | Simple; idle cost at trough, queueing at peak |
| VM Scale Set agents | Elastic; agent VMs run in your Azure subscription while Azure Pipelines manages scaling |
| Managed DevOps Pools | Elastic; agents run in Microsoft-managed Azure infrastructure — current recommendation |
VM Scale Set agents use Azure Marketplace or custom images in your subscription. Managed DevOps Pools can use Azure Pipelines, Marketplace, or Azure Compute Gallery images and can expose multiple images in one pool. Choose the image as the durable VM template for the tools and operating system each workload needs.
Container jobs and service containers
Running a job in a container pins its OS, tools, and dependencies to an image, reducing drift from software installed on the host. A reused self-hosted agent can still retain machine-level caches and configuration, and container jobs use the host agent's Docker configuration, so workspace and host hygiene remain separate responsibilities. Service containers provide job-scoped dependencies such as a database or message broker and are managed for that job's lifetime.
Primary sources
- https://learn.microsoft.com/en-us/credentials/certifications/resources/study-guides/az-400
- https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/agents
- https://docs.github.com/en/actions/reference/runners/self-hosted-runners
- https://learn.microsoft.com/en-us/azure/devops/pipelines/process/container-phases
- https://learn.microsoft.com/en-us/azure/devops/pipelines/process/service-containers
- https://learn.microsoft.com/en-us/azure/devops/pipelines/process/environments-virtual-machines
- https://learn.microsoft.com/en-us/azure/devops/pipelines/process/deployment-jobs
- https://learn.microsoft.com/en-us/azure/devops/managed-devops-pools/migrate-from-scale-set-agents