Key concepts for new Azure Pipelines. (Part 1)

A trigger initiates a pipeline execution.

• A pipeline consists of multiple stages and can deploy across several environments.
• Stages organize jobs within a pipeline, with each stage containing one or more jobs.
• Jobs run on individual agents, though they can also be agentless.
• Agents execute jobs comprised of multiple steps.
• Steps, representing the smallest pipeline components, can be either tasks or scripts.
• Tasks are pre-built scripts that perform specific actions like API calls or artifact publishing.
• Artifacts are collections of files or packages produced during a run.

Azure Pipelines Key Terms        

Agent

An agent is a computing resource that runs jobs in a pipeline. Each job runs on a single agent, which could be Microsoft-hosted (e.g., an Ubuntu agent) or self-hosted.

Approvals

Approvals define validations required before a deployment runs. They act as checks, pausing the pipeline until all required approvals are completed.

Artifact

An artifact is a collection of files or packages generated during a pipeline run. These artifacts are used in later stages for deployment or distribution.

Continuous Integration (CI)

CI automates the testing and building of code whenever changes are made, ensuring early bug detection. It produces artifacts used in continuous delivery pipelines.

Continuous Delivery (CD)

CD automates code deployment across multiple stages (test, production, etc.), ensuring quality through continuous testing and monitoring.

Deployment

A deployment executes tasks in a stage, including tests, artifact deployment, and environment configuration. In YAML pipelines, a deployment job defines deployment strategies like rolling, canary, and run-once.

Deployment Group

A set of deployment target machines with installed agents. It enables grouping of agents for targeted deployments.

Environment

A collection of resources (VMs, containers, web apps) where applications are deployed. Pipelines deploy to environments after builds and tests are completed.

Job

A job contains steps that run on an agent. Jobs can run in parallel or sequentially within a stage. Agentless jobs execute tasks directly in Azure DevOps without an agent.

Pipeline

A pipeline defines the CI/CD process for an application. It consists of multiple stages, handling build, test, and deployment workflows.

Release

In classic pipelines, a release is a versioned set of artifacts along with deployment settings. In YAML pipelines, builds and releases are combined into a single multi-stage pipeline.

Run

A single execution of a pipeline, including logs, test results, and step executions. In classic pipelines, a build represents a run.

Script

A step in a pipeline that runs custom code using the command line, PowerShell, or Bash. Unlike tasks, scripts are manually written for pipeline-specific automation.

Stage

A logical boundary in a pipeline representing a phase (e.g., Build, QA, Production). Stages can have dependencies, approvals, and conditions for execution.

Step

The smallest unit of a pipeline is either a script or a task. Multiple steps make up a job.

Task

A pre-packaged script or procedure in a pipeline that automates specific actions, such as running tests or deploying artifacts.

Trigger

Triggers define when a pipeline runs, such as on a code push, scheduled time, or after another build completes.

Library

Centralized storage for secure files and variable groups, allowing values and secrets to be shared across multiple pipelines.

Pipeline runs        

For each pipeline run, Azure Pipelines:

• Processes the pipeline configuration.
• Requests one or more agents to execute jobs.
• Assigns jobs to agents and gathers results.

For each job, an agent:

• Prepares the environment.
• Executes each step of the job.
• Reports the results.

Jobs can succeed, fail, be canceled, or remain incomplete. Understanding these outcomes helps with troubleshooting.

Pipeline processing

Pipeline Processing in Azure Pipelines

To process a pipeline for a run, Azure Pipelines follows these steps:

1. Initial Processing:
2. Stage Execution:
3. Job Execution:

Variables and Processing Order

Understanding the processing order is crucial for variable usage:

• Template Expansion Limitation: Since templates expand before runtime variables exist, you can't use runtime variables in template parameters.
• Authorization Timing: Service connection and environment names can't be resolved using variables because resources must be authorized before the stage starts.
• Variable Scope: Stage and job-level variables aren’t available during authorization. Variable groups are considered resources, so their data is unavailable when checking resource authorization. Pipeline-level variables explicitly defined in the pipeline resource can be used.

Agents in Azure Pipelines

When Azure Pipelines needs to run a job, it requests an agent from the designated pool. The process differs based on whether the agent is Microsoft-hosted or self-hosted.

Parallel Jobs and Agent Assignment

1. Parallel Job Check:
2. Job Queuing:

Microsoft-Hosted Agents

• The Microsoft-hosted agent pool is a globally managed infrastructure split by geography and operating system type.
• Based on the YAML vmImage or Classic Editor pool name, Azure Pipelines selects an agent.
• Every agent is a fresh virtual machine (VM) that has never run pipelines before.
• Once the job is completed, the agent VM is discarded.

Self-Hosted Agents

• Once a parallel slot is free, Azure Pipelines looks for a compatible self-hosted agent.
• Agent Capabilities vs. Pipeline Demands:
• Unlike Microsoft-hosted agents, self-hosted agents are reused across multiple pipeline runs.

Job Preparation on an Agent

Once an agent accepts a job, it performs the following tasks:

1. Downloads and caches all required tasks for the job.
2. Creates a working directory to store source code, artifacts, and output files for the run.

Step Execution in Azure Pipelines

Sequential Execution of Steps

• Steps execute sequentially, meaning each step must be completed or skipped before the next one starts.
• Steps are implemented using tasks, which can run scripts written in Node.js, PowerShell, or other languages.
• The task system handles input and output routing for scripts and provides common services such as: Modifying the system path Creating new pipeline variables

Step Isolation and Environment Variables

• Each step runs in a separate process, ensuring isolation from previous steps.
• Because of this, environment variables do not persist between steps.
• However, scripts and tasks can communicate with the agent using logging commands.
• When a script writes a logging command to standard output, the agent processes it and performs the requested action.

Creating Pipeline Variables via Logging Commands

• Pipeline variables created in one step are automatically converted into environment variables for the next step.
• Bash: echo '##vso[task.setVariable variable=myVar]myValue'
• Powershell: Write-Host "##vso[task.setVariable variable=myVar]myValue"

Result Reporting and Log Collection

• Each step can report warnings, errors, and failures.
• Errors and warnings appear on the pipeline summary page, marking tasks as: Succeeded with issues (warnings) Failed (errors)
• A step fails if: It explicitly reports failure using a ##vso command. The script ends with a nonzero exit code.

Live Logging and Log File Storage

• As steps execute, the agent streams output live to Azure Pipelines.
• At the end of each step, the full log output is uploaded as a file.
• Once the pipeline run is complete, logs are available for download and review.

List pipeline runs

The following command lists the first three pipeline runs that have a status of completed and a result of succeeded and returns the result in table format.

Show pipeline run details

The following command shows details for the pipeline run with the ID 123, returns the results in a table format, and opens your web browser to the Azure Pipelines build results page.

Add a tag to the pipeline run

The following command adds the tag YAML to the pipeline run with the ID 123 and returns the result in JSON format.

az pipelines runs tag add --run-id 123 --tags YAML --output json

[

"YAML"

]

List pipeline run tags

The following command lists the tags for the pipeline run with the ID 123 and returns the result in table format.

az pipelines runs tag list --run-id 123 --output table

Tags

YAML

Delete tag from a pipeline run

The following command deletes the YAML tag from the pipeline run with ID 123.

az pipelines runs tag delete --run-id 123 --tag YAML

YAML vs Classic Pipelines

Azure Pipelines: Automating Development Workflows

Azure Pipelines empowers developers to automate various tasks, from executing simple batch scripts to building a full-fledged CI/CD (Continuous Integration and Continuous Delivery) pipeline for their applications.

Flexible Support for Languages, Platforms, and Tools

Azure Pipelines supports various programming languages, platforms, and tools, making it a versatile choice for development teams. It offers two pipeline options:

1. YAML-based Pipelines – Defined using a configuration file stored in version control.
2. Classic Pipelines – Created using a visual editor.

Defining Pipelines with YAML

• The pipeline configuration is stored in a YAML file named azure-pipelines.yml, which is kept alongside the application code.
• This YAML file follows the same branching structure as the application, allowing each branch to have its customized pipeline configuration.
• Version control integration ensures that any pipeline changes are tracked along with the codebase. Issues or unexpected failures can be easily identified and reverted if necessary.

Creation of your first pipeline: Refer to this document (https://www.linkedin.com/pulse/what-azure-pipelines-ankit-ranjan-devops-engineer--fcj7f/?trackingId=ni%2BEuukwSPqN8jWkeYyhRg%3D%3D)