SlideShare a Scribd company logo
Accordion Pipelines – A Cloud-native declarative Pipelines and
Dynamic workflows
Accordion Ci/cd Cloud-native Workflows
Pavan Kumar Hemadri
Thursday, September 14th, 2023
6 min read
Cisco Tech Blog Home Topics Research Contact Us
Pipelines as code have evolved in the past decade from the early days of Groovy scripts to various
forms of declarative constructs with many open source and commercial offerings. In general,
declarative pipelines use structured syntax with predefined keywords to offer better readability and
abstractions of complexity. Still, they are far from true declarative in spirit by including domain-
specific semantics.
With the increasing adoption of declarative pipelines over conventional CI jobs, standardization, and
pipeline code quality are of utmost importance in large-scale cloud native projects which involve
multiple pipelines from one or more version control repositories. Inheritance of pipelines from one
another helps improve productivity by sharing and reusing pipelines across the projects.
Similarly, the workflows in various OSS or commercial offerings require conditional constructs or
domain-specific language semantics to decide the flow of complex pipelines. This approach
contradicts declarative conventions and is unsuitable for dynamically generating pipelines based on
declarative workflows. An alternative implementation is proposed using reusable and extensible
templates to address these limitations.
In large organizations dealing with multiple projects or in open source development environments,
there is a need to share these templates within the organization or globally across private or public
Git-based version control repositories offered by GitHub, Bit Bucket, Gerrit, etc. This empowers a
true distributed pipelines development ecosystem like go modules in Golang programming language.
This blog discusses how effectively we can automate the workflows with templates on Accordion’s
cloud native CI/CD platform. Accordion pipelines with dynamic workflows are truly declarative, which
can be compared to the likes of Kubernetes YAML, which manifests free from domain-specific
semantics. This brings in powerful dynamic pipelines with a set of manifest files effectively dictated by
the process embedded in the YAML manifest.
Resources
Accordion CI/CD platform
Jenkins Pipelines
Low Code Extensible Pipeline Templates
In the context of CI/CD (Continuous Integration/Continuous Deployment) pipelines, templates refer to
predefined, reusable sets of instructions or configurations that help streamline the process of setting
up and managing pipelines for different projects or components within a software development
environment. Templates are designed to promote consistency, reduce duplication of effort, and
improve the overall efficiency of building, testing, and deploying software.
The Accordion pipeline template’s scope is granular, pervasive, and not specific to a stage or the full
pipeline. In fact, inheritance works at a block level or even for a single step within a stage. One can
create a template for an agent definition or environment or pipeline options, stages, or steps or with
Share
all of them in the most abstract form. Also, these templates can be nested to abstract out the least
relevant data. Here is the sample template:
From the above illustrations, template.yaml is a primary template that contains the template name,
i.e., “master_template,” and different pipeline blocks like environment, options, and stages. Whereas
pipeline.yaml is a purely abstracted pipeline that imports the template from a private GitHub
repository with the URL “https://aci-
github.cisco.com/accordion/pipeline_templates/templates/master_template.yaml” and its git revision
“8f1463c” (this is for pipeline reproducibility). Credentials are optional depending on the repository
type, i.e., public or private. Credentials can be pulled from the CI system’s native credential store or
Kubernetes secrets. The imported template is tagged as “template” in the pipeline – as seen in item
4 in pipeline.yaml. Once a template is imported, one can either inherit the entire template or a
particular block from the template.yaml. In the above example, pipeline.yaml only inherits the block
“stages,” as shown in item 1.
If we were to inherit the entire pipeline from the template, the pipeline.yaml would just be:
The above illustrations demonstrate the behavior of the generation of pipelines dynamically from the
templates. When this template inheritance feature is combined with the feature of policy-based
workflows, this software unveils easy self-serviced pipelines with dynamic workflows. Most of the
work is one-time, i.e., declaration of templates and all workflows, thereby reusing them for various
pipeline requirements.
Share
Dynamic Workflows
Workflows define the flow of a pipeline based on several constructs like parameters, environment, or
even webhook trigger information. Usually, these workflows are developed with the underlying
pipeline’s CI system DSL and custom syntaxes. There is a learning curve involved for pipeline
developers to understand these semantics. The solution shared provides a way of defining semantic-
free workflows with YAML specification.
Accordion’s dynamic workflows have a pure declarative approach in which one can declare the
stages’ (or tasks) names applicable to a workflow and assign a name to the workflow. When the
workflow name is attached to the Gerrit code change or pull request of GitHub / Bit Bucket in the
form of a tag or a comment, the Accordion CICD software applies the workflow name to a given
workflow and template files to render all stages with their definitions and generate the resultant
pipeline dynamically. While the actual pipeline code remains constant as a boilerplate code for any
workflow, just the workflow name gets attached to the code commit or job parameters (external to
the scope of pipeline code) to define the pipeline’s flow.
Dynamic workflows offer:
An abstraction from the workflow policy name to dynamically generate flow with required
stages using templates.
An abstraction defined in the “context” field gives flexibility to support various trigger points.
Here are some examples to illustrate the behavior of policy-based dynamic workflows in Accordion
CICD.
Share
In the above example, pipeline.yaml will remain the same in any workflow. For example, when
reviewing the workflow with the policy name “a7n:lint”
, the policy label “a7n:lint” is attached to a
GitHub or BitBucket PR. The Accordion CI software dynamically generates a pipeline with the blocks
defined in the workflow file and the lint stage by rendering them from templates.
Even if a developer wants to execute a pipeline with a different workflow, only the relevant workflow
name attached to the GitHub pull request changes. For example, upon attaching “a7n:build”
, it
dynamically generates the resultant pipeline for stages build, lint, and test – all this happens with no
changes to the underlying pipeline code. This feature does not need any conditional constructs or
custom DSL syntax in pipelines or workflow files. This also does not require importing multiple
workflow files per workflow for a complex pipeline.
Highlighting important aspects of dynamic workflows:
Ease of use/simplicity: no need to be a pipeline developer to define a workflow. Any user can
define the flow of execution based on the requirement. Integration is straightforward.
Abstraction: confers flexibility and scalability. For example, one can define the number of
workflows with policy labels and stages participating in the pipeline without the knowledge of
underlying pipeline code semantics.
Readability: flows can be easily understood since no conditional and additional logic is used.
Scalability and Flexibility: multiple workflows can be easily imported and referenced in a
pipeline definition.
Real-world example pipeline:
Share
Cisco.com Innovation Labs Research Who We Are Careers Contact 
Help Terms & Conditions Statement Trademarks © 2023 Cisco Systems, Inc.
Conclusion
The Accordion dynamic workflows and templates bring in the power of distributed pipeline
development for large-scale projects, avoiding repeatability of the pipeline code development, low
maintenance, abstraction of secrets, and secure logic. This enables the generation of workflows on
the fly with a selected GitHub label, a comment on a pull request, or any input from any VCS
software. Pure declarative manifests depict the flow of the process from which the pipeline flow is
built dynamically with VCS triggers and selections.
Acknowledgments
We sincerely thank Kalyan Ghosh, Debasish Chakraborty and the Data Center and Cloud Networking
DevOps team for their valuable contributions to this project. Share

More Related Content

PDF
Devops Porto - CI/CD at Gitlab
PDF
Portable CI wGitLab and Github led by Gavin Pickin.pdf
PDF
vodQA Pune (2019) - Jenkins pipeline As code
PPSX
CI-CD Jenkins, GitHub Actions, Tekton
PPTX
Modern CI/CD Pipeline Using Azure DevOps
PDF
Jenkins : Pipeline As Code
PPTX
Introduction to Go
PPTX
GitHub Actions (Nakov at RuseConf, Sept 2022)
Devops Porto - CI/CD at Gitlab
Portable CI wGitLab and Github led by Gavin Pickin.pdf
vodQA Pune (2019) - Jenkins pipeline As code
CI-CD Jenkins, GitHub Actions, Tekton
Modern CI/CD Pipeline Using Azure DevOps
Jenkins : Pipeline As Code
Introduction to Go
GitHub Actions (Nakov at RuseConf, Sept 2022)

Similar to Accordion Pipelines - A Cloud-native declarative Pipelines and Dynamic workflows | Cisco Tech Blog (20)

PDF
How to plan and define your CI-CD pipeline
PDF
CI/CD with Github Actions
PPTX
CICD Pipeline Using Github Actions
PPTX
Pipeline as code - new feature in Jenkins 2
PDF
Top CI/CD Tools Every QA Automation Engineer Should Use
PDF
138-03 Python, Git, GitHub and CI-CD.pdf
PPTX
CI_CD_Pipelines_java_---Presentation.pptx
PDF
Trending pipelines of using CI-CD
PDF
DevOps Patterns to Enable Success in Microservices
PDF
Shift Remote: DevOps: Gitlab ci hands-on experience - Ivan Rimac (Barrage)
PPTX
Devops CI-CD pipeline with Containers
PDF
Getting to Walk with DevOps
PDF
Promise of DevOps
PPTX
Devops
PPTX
Jenkins pipeline as code
PPTX
Ice breaker with dev ops
PDF
CI doesn’t start with Jenkins
PDF
CI CD Pipeline Interview Questions PDF By ScholarHat
PDF
Atlanta Jenkins Area Meetup October 22nd 2015
PPT
icebreakerwithdevops-150218112943-conversion-gate02
How to plan and define your CI-CD pipeline
CI/CD with Github Actions
CICD Pipeline Using Github Actions
Pipeline as code - new feature in Jenkins 2
Top CI/CD Tools Every QA Automation Engineer Should Use
138-03 Python, Git, GitHub and CI-CD.pdf
CI_CD_Pipelines_java_---Presentation.pptx
Trending pipelines of using CI-CD
DevOps Patterns to Enable Success in Microservices
Shift Remote: DevOps: Gitlab ci hands-on experience - Ivan Rimac (Barrage)
Devops CI-CD pipeline with Containers
Getting to Walk with DevOps
Promise of DevOps
Devops
Jenkins pipeline as code
Ice breaker with dev ops
CI doesn’t start with Jenkins
CI CD Pipeline Interview Questions PDF By ScholarHat
Atlanta Jenkins Area Meetup October 22nd 2015
icebreakerwithdevops-150218112943-conversion-gate02
Ad

Recently uploaded (20)

PPTX
VMware vSphere Foundation How to Sell Presentation-Ver1.4-2-14-2024.pptx
PDF
Spectral efficient network and resource selection model in 5G networks
PDF
Reach Out and Touch Someone: Haptics and Empathic Computing
PDF
Approach and Philosophy of On baking technology
PDF
Mobile App Security Testing_ A Comprehensive Guide.pdf
PDF
Advanced methodologies resolving dimensionality complications for autism neur...
PDF
How UI/UX Design Impacts User Retention in Mobile Apps.pdf
PDF
CIFDAQ's Market Insight: SEC Turns Pro Crypto
PDF
Electronic commerce courselecture one. Pdf
PDF
Bridging biosciences and deep learning for revolutionary discoveries: a compr...
PDF
Architecting across the Boundaries of two Complex Domains - Healthcare & Tech...
PDF
7 ChatGPT Prompts to Help You Define Your Ideal Customer Profile.pdf
PDF
Blue Purple Modern Animated Computer Science Presentation.pdf.pdf
PDF
TokAI - TikTok AI Agent : The First AI Application That Analyzes 10,000+ Vira...
PDF
Encapsulation theory and applications.pdf
PDF
Build a system with the filesystem maintained by OSTree @ COSCUP 2025
PPTX
MYSQL Presentation for SQL database connectivity
PPTX
Understanding_Digital_Forensics_Presentation.pptx
PPTX
20250228 LYD VKU AI Blended-Learning.pptx
PPTX
Big Data Technologies - Introduction.pptx
VMware vSphere Foundation How to Sell Presentation-Ver1.4-2-14-2024.pptx
Spectral efficient network and resource selection model in 5G networks
Reach Out and Touch Someone: Haptics and Empathic Computing
Approach and Philosophy of On baking technology
Mobile App Security Testing_ A Comprehensive Guide.pdf
Advanced methodologies resolving dimensionality complications for autism neur...
How UI/UX Design Impacts User Retention in Mobile Apps.pdf
CIFDAQ's Market Insight: SEC Turns Pro Crypto
Electronic commerce courselecture one. Pdf
Bridging biosciences and deep learning for revolutionary discoveries: a compr...
Architecting across the Boundaries of two Complex Domains - Healthcare & Tech...
7 ChatGPT Prompts to Help You Define Your Ideal Customer Profile.pdf
Blue Purple Modern Animated Computer Science Presentation.pdf.pdf
TokAI - TikTok AI Agent : The First AI Application That Analyzes 10,000+ Vira...
Encapsulation theory and applications.pdf
Build a system with the filesystem maintained by OSTree @ COSCUP 2025
MYSQL Presentation for SQL database connectivity
Understanding_Digital_Forensics_Presentation.pptx
20250228 LYD VKU AI Blended-Learning.pptx
Big Data Technologies - Introduction.pptx
Ad

Accordion Pipelines - A Cloud-native declarative Pipelines and Dynamic workflows | Cisco Tech Blog

  • 1. Accordion Pipelines – A Cloud-native declarative Pipelines and Dynamic workflows Accordion Ci/cd Cloud-native Workflows Pavan Kumar Hemadri Thursday, September 14th, 2023 6 min read Cisco Tech Blog Home Topics Research Contact Us Pipelines as code have evolved in the past decade from the early days of Groovy scripts to various forms of declarative constructs with many open source and commercial offerings. In general, declarative pipelines use structured syntax with predefined keywords to offer better readability and abstractions of complexity. Still, they are far from true declarative in spirit by including domain- specific semantics. With the increasing adoption of declarative pipelines over conventional CI jobs, standardization, and pipeline code quality are of utmost importance in large-scale cloud native projects which involve multiple pipelines from one or more version control repositories. Inheritance of pipelines from one another helps improve productivity by sharing and reusing pipelines across the projects. Similarly, the workflows in various OSS or commercial offerings require conditional constructs or domain-specific language semantics to decide the flow of complex pipelines. This approach contradicts declarative conventions and is unsuitable for dynamically generating pipelines based on declarative workflows. An alternative implementation is proposed using reusable and extensible templates to address these limitations. In large organizations dealing with multiple projects or in open source development environments, there is a need to share these templates within the organization or globally across private or public Git-based version control repositories offered by GitHub, Bit Bucket, Gerrit, etc. This empowers a true distributed pipelines development ecosystem like go modules in Golang programming language. This blog discusses how effectively we can automate the workflows with templates on Accordion’s cloud native CI/CD platform. Accordion pipelines with dynamic workflows are truly declarative, which can be compared to the likes of Kubernetes YAML, which manifests free from domain-specific semantics. This brings in powerful dynamic pipelines with a set of manifest files effectively dictated by the process embedded in the YAML manifest. Resources Accordion CI/CD platform Jenkins Pipelines Low Code Extensible Pipeline Templates In the context of CI/CD (Continuous Integration/Continuous Deployment) pipelines, templates refer to predefined, reusable sets of instructions or configurations that help streamline the process of setting up and managing pipelines for different projects or components within a software development environment. Templates are designed to promote consistency, reduce duplication of effort, and improve the overall efficiency of building, testing, and deploying software. The Accordion pipeline template’s scope is granular, pervasive, and not specific to a stage or the full pipeline. In fact, inheritance works at a block level or even for a single step within a stage. One can create a template for an agent definition or environment or pipeline options, stages, or steps or with Share
  • 2. all of them in the most abstract form. Also, these templates can be nested to abstract out the least relevant data. Here is the sample template: From the above illustrations, template.yaml is a primary template that contains the template name, i.e., “master_template,” and different pipeline blocks like environment, options, and stages. Whereas pipeline.yaml is a purely abstracted pipeline that imports the template from a private GitHub repository with the URL “https://aci- github.cisco.com/accordion/pipeline_templates/templates/master_template.yaml” and its git revision “8f1463c” (this is for pipeline reproducibility). Credentials are optional depending on the repository type, i.e., public or private. Credentials can be pulled from the CI system’s native credential store or Kubernetes secrets. The imported template is tagged as “template” in the pipeline – as seen in item 4 in pipeline.yaml. Once a template is imported, one can either inherit the entire template or a particular block from the template.yaml. In the above example, pipeline.yaml only inherits the block “stages,” as shown in item 1. If we were to inherit the entire pipeline from the template, the pipeline.yaml would just be: The above illustrations demonstrate the behavior of the generation of pipelines dynamically from the templates. When this template inheritance feature is combined with the feature of policy-based workflows, this software unveils easy self-serviced pipelines with dynamic workflows. Most of the work is one-time, i.e., declaration of templates and all workflows, thereby reusing them for various pipeline requirements. Share
  • 3. Dynamic Workflows Workflows define the flow of a pipeline based on several constructs like parameters, environment, or even webhook trigger information. Usually, these workflows are developed with the underlying pipeline’s CI system DSL and custom syntaxes. There is a learning curve involved for pipeline developers to understand these semantics. The solution shared provides a way of defining semantic- free workflows with YAML specification. Accordion’s dynamic workflows have a pure declarative approach in which one can declare the stages’ (or tasks) names applicable to a workflow and assign a name to the workflow. When the workflow name is attached to the Gerrit code change or pull request of GitHub / Bit Bucket in the form of a tag or a comment, the Accordion CICD software applies the workflow name to a given workflow and template files to render all stages with their definitions and generate the resultant pipeline dynamically. While the actual pipeline code remains constant as a boilerplate code for any workflow, just the workflow name gets attached to the code commit or job parameters (external to the scope of pipeline code) to define the pipeline’s flow. Dynamic workflows offer: An abstraction from the workflow policy name to dynamically generate flow with required stages using templates. An abstraction defined in the “context” field gives flexibility to support various trigger points. Here are some examples to illustrate the behavior of policy-based dynamic workflows in Accordion CICD. Share
  • 4. In the above example, pipeline.yaml will remain the same in any workflow. For example, when reviewing the workflow with the policy name “a7n:lint” , the policy label “a7n:lint” is attached to a GitHub or BitBucket PR. The Accordion CI software dynamically generates a pipeline with the blocks defined in the workflow file and the lint stage by rendering them from templates. Even if a developer wants to execute a pipeline with a different workflow, only the relevant workflow name attached to the GitHub pull request changes. For example, upon attaching “a7n:build” , it dynamically generates the resultant pipeline for stages build, lint, and test – all this happens with no changes to the underlying pipeline code. This feature does not need any conditional constructs or custom DSL syntax in pipelines or workflow files. This also does not require importing multiple workflow files per workflow for a complex pipeline. Highlighting important aspects of dynamic workflows: Ease of use/simplicity: no need to be a pipeline developer to define a workflow. Any user can define the flow of execution based on the requirement. Integration is straightforward. Abstraction: confers flexibility and scalability. For example, one can define the number of workflows with policy labels and stages participating in the pipeline without the knowledge of underlying pipeline code semantics. Readability: flows can be easily understood since no conditional and additional logic is used. Scalability and Flexibility: multiple workflows can be easily imported and referenced in a pipeline definition. Real-world example pipeline: Share
  • 5. Cisco.com Innovation Labs Research Who We Are Careers Contact  Help Terms & Conditions Statement Trademarks © 2023 Cisco Systems, Inc. Conclusion The Accordion dynamic workflows and templates bring in the power of distributed pipeline development for large-scale projects, avoiding repeatability of the pipeline code development, low maintenance, abstraction of secrets, and secure logic. This enables the generation of workflows on the fly with a selected GitHub label, a comment on a pull request, or any input from any VCS software. Pure declarative manifests depict the flow of the process from which the pipeline flow is built dynamically with VCS triggers and selections. Acknowledgments We sincerely thank Kalyan Ghosh, Debasish Chakraborty and the Data Center and Cloud Networking DevOps team for their valuable contributions to this project. Share