Jenkins Shared Library: pipeline inheritance via Jinja2

Problem

You have a bunch of pipelines in your shared library with a lot of overlapping structure. You would like to have some kind of "base" pipeline which implements all the common parts and extend this pipeline by all your individual pipelines. However, inheritance is not working inside of your shared-library (read this CloudBees post to find out why).

Solution

Use jinja2's template mechanism.

Implementation

TLDR: Find all the code on bitbucket.

We will construct a simple example: we locate our jinja templates under resources within our shared library. Additionally, we write a Python script that traverses all templates and outputs the rendered Groovy pipeline, saving them under vars.

So our final folder structure will look like so:

shared-library
+-- resources
|   +-- scripts
|       +-- generate_pipelines.py
|   +-- template
|       +--base
|          +--pipeline.groovy.j2
|   +-- pipelineTest01.groovy.j2
+-- vars
|   +-- pipelineTest01.groovy

where vars/pipelineTest01.groovy will be the result of executing generate_pipelines.py.

#!/usr/bin/env python3
import jinja2
import pathlib

class Pipeline:
    def __init__(self, tmpl_path):
        self.tmpl_path = tmpl_path
        self.jinja2_env = jinja2.Environment(
            loader=jinja2.FileSystemLoader(self.tmpl_path))

    def generate(self, pipline_tmpl):
        tmpl = self.jinja2_env.get_template(pipline_tmpl)
        return tmpl.render()


def generate_shared_lib():
    root = pathlib.Path(__file__).parent.parent.parent
    tmpl_path = root / 'resources' / 'template'
    pipeline = Pipeline(tmpl_path)

    vars_path = root / 'vars'
    # if vars folder does not exist create it
    vars_path.mkdir(parents=True, exist_ok=True)

    # travers through templates, generate their groovy files
    # and place them under vars
    for path in tmpl_path.glob('*.groovy.j2'):
        path_str = str(path)

        # split off the actual template name which
        # is the last part of the path and generate
        # the pipeline
        pipeline_tmpl = path_str.split('/')[-1]
        pipeline_file = pipeline.generate(pipeline_tmpl)

        # get the groovy file name, i.e. drop the '.j2'
        pipeline_groovy_name = pipeline_tmpl[:-3]

        # place it as a groovy file under vars
        with open(vars_path / pipeline_groovy_name,
                  mode='w') as pipeline_library:
            pipeline_library.write(pipeline_file)


if __name__ == '__main__':
    generate_shared_lib()

Our base pipeline example is the following jinja2 template pipeline.groovy.j2:

def call(Map params = [:]) {
    pipeline {
        agent { label node_label }

        environment {
            SECRET_CREDENTIALS = credentials('super_secret')
            // Custom environment extension
            {% block environment %}
            {% endblock %}
        }

        stages {
            stage('Init') {
                steps {
                    // Do something you do in every pipeline
                    echo 'Init pipeline.'
                    script {
                        // Custom init extension
                        {% block init %}
                        {% endblock %}
                    }
                }
            }

            stage('Checkout') {
                steps{
                    checkout scm
                }
            }

            // Custom stages
            {% block stages %}
            {% endblock %}
        }
    }
}

Not the jinja blocks starting with {%. Those parts will get populated by other pipelines, e.g by pipelineTest01.groovy:

{% extends "base/pipeline.groovy.j2" %}

{% block environment %}
            ANOTHER_SECRET_CREDS = credentials('another_secret')
            TEST = 'Hello world.'
{% endblock %}

{% block stages %}
            stage('Test') {
                steps {
                    echo "Hello ${env.TEST}"
                }
            }
{% endblock %}

Note how this pipeline only populates jinja blocks it needs to customize. Also note that we don't populate every block from the base pipeline, e.g. we leave the Init-block unchanged.

We can now run the Python script and it will generate our pipeline, placing it under /vars/pipelineTest01.groovy:

shared-library apoehlmann$ ./resources/scripts/generate_pipelines.py

This results in the following Groovy file:

def call(Map params = [:]) {
    pipeline {
        agent { label node_label }

        environment {
            SECRET_CREDENTIALS = credentials('super_secret')
            // Custom environment extension
            
            ANOTHER_SECRET_CREDS = credentials('another_secret')
            TEST = 'Hello world.'

        }

        stages {
            stage('Init') {
                steps {
                    // Do something you do in every pipeline
                    echo 'Init pipeline.'
                    script {
                        // Custom init extension
                        
                        
                    }
                }
            }

            stage('Checkout') {
                steps{
                    checkout scm
                }
            }

            // Custom stages
            
            stage('Test') {
                steps {
                    echo "Hello ${env.TEST}"
                }
            }

        }
    }
}

That's basically it.

Conclusion

We have now a way how to template our Jenkins pipelines via Python's Jinja2 and thus implement some kind of "inheritance" in our templates.

Also note that Jinja2 is way more flexible than shown in this post: e.g. it allows to pass template variables to its render() method. Those variables would then get passed to the template which then makes use of them. This allows you to code more complex templates than shown here. It also has so-called include statements which you can use to split up large templates. For a nice intro have a look at this article.

Also as a last note: because we have both the templates and the Python script in our shared library, we could actually write a pipeline that automatically generates the pipelines. But this is left as an exercies for the motivated reader (hint: you will need to come up with a way to execute scripts under resources from within a pipeline step. See e.g. https://stackoverflow.com/a/50827874/4804137 ) :)

Getagged mit:
Jenkins