{"id":10179,"date":"2022-09-06T12:49:36","date_gmt":"2022-09-06T10:49:36","guid":{"rendered":"https:\/\/www.btc-embedded.com\/?p=10179"},"modified":"2024-01-29T12:04:08","modified_gmt":"2024-01-29T10:04:08","slug":"4-steps-to-not-screw-up-jenkins","status":"publish","type":"post","link":"https:\/\/www.btc-embedded.com\/zh-hans\/4-steps-to-not-screw-up-jenkins\/","title":{"rendered":"4 steps to not screw up Jenkins"},"content":{"rendered":"\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

Introduction<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

During the last six years I have worked with many customers who used Jenkins as their central automation server for everything remotely related to software.<\/p>

In my previous role at BTC I spent much time setting up the complex Jenkins setup for BTC internal use. This meant covering multiple programming languages (Java, Matlab, C\/C++, Python) and involving complex test matrices for the supported Matlab and TargetLink versions we support. It\u2019s not the same as developing automotive application software but a tool qualified for standards like ISO 26262 demands a high level of quality.<\/p>

Personally, I\u2019m very fond of Jenkins. But in many of the cases that I encountered, the CI setups have grown into vastly complex systems. Adaptions require experts and developers and testers are wing-clipped by strict permission settings. And instead of providing fast feedback, pipelines are painfully slow, up to the point where even over-night feedback was no longer guaranteed.<\/p>

After all, it’s 2022…<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t
\n\t\t\t

\n\t\t\t\tJenkins is dead!\t\t\t<\/p>\n\t\t\t\t\t\t\t

\n\t\t\t\t\t\t\t\t\t\t\tNietzsche<\/cite>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t\t<\/blockquote>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t
\n\t\t\t
<\/div>\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

Why is Jenkins so slow and complex?<\/h3>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

To be honest, this phenomenon is not exclusive to Jenkins but Jenkins has been around long enough to suffer most from time. Especially scm-centric tools like GitHub Actions, GitLab CI suffer much less, because they don’t aim for a central server but follow a per-repository pattern.<\/p>

Apart from the fact that individual CI pipelines became more complex and involved a bigger variety of tools, the attempt to harmonize processes and bring all teams \/ departments on board led to a major increase of complexity.<\/p>

Different teams have different needs. Applying the same process for everyone often limits the ability to create suitable solutions for individual teams. Furthermore, changes made by one person can easily break the CI setup for everyone involved. IT teams commonly prevented this by thoroughly restricting permissions and access rights. Any changes that a team wished to have, needed to be performed by central IT, creating a bottleneck, frustration and eventually acceptance of a mediocre state.<\/p>

Limited benefits – high maintenance costs. That’s not really what we had hoped to accomplish, right?<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

\n\t\t\t\t
\n\t\t\t\t\t\t\t
\n\t\t\t
<\/div>\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

What can we do about it?<\/h3>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

We can fix this by decentralizing our IT setups, providing a controlled environment, where teams have ownership and by leveraging the benefits of scalable cloud resources.<\/p>

In order to achieve this, we need to:<\/p>

  • Isolate applications and services by moving them into containers<\/li>
  • Move anything that requires manual configuration towards a configuration-as-code approach and apply coding best practices<\/li>
  • Replace manually prepared agent machines with dynamic agents, ideally cloud-based<\/li>
  • Automate pipeline creation to get rid of repetitive manual steps and lower the threshold for developers to onboard new and existing projects<\/li><\/ul>

    \u00a0<\/p>

    This article aims to provide directions to important changes and paradigm shifts. It will not be a full step-by-step guide, but I’ll do my best to link to the plenty available resources that are out there on different aspects that we’ll touch.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

    \n\t\t\t\t
    \n\t\t\t\t\t

    My 4 Steps <\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t

    Step 1 - Why I run Jenkins in Docker<\/h3>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
    \n\t\t\t\t
    \n\t\t\t\t\t\t\t\t\t

    The first step towards a better future for your team\u2019s CI is to say goodbye to installing software on a PC, server or virtual machine. Even if scripts are involved, installation, runtime and maintenance of the software is affected by countless influences from its environment.<\/p>

    So instead of installing Jenkins on Linux, Mac or Windows, we can just run it in a Docker container:<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

    \n\t\t\t\t
    \n\t\t\t\t\t\t\t
    \n\t\t\t
    \n\t\t\t\t\n\t\t\t\t\tdocker run -p 8080:8080 jenkins\/jenkins<\/xmp>\n\t\t\t\t<\/code>\n\t\t\t<\/pre>\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-357536d elementor-widget elementor-widget-text-editor\" data-id=\"357536d\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p>This will spin up a fresh Jenkins instance on http:\/\/localhost:8080 based on the latest version of the jenkins\/jenkins container image from dockerhub. This is nice, but the Jenkins instance is completely vanilla, no users, no config, no plugins. In order to change that, we can create our own Jenkins controller container image.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-0ca9da6 elementor-widget elementor-widget-code-highlight\" data-id=\"0ca9da6\" data-element_type=\"widget\" data-widget_type=\"code-highlight.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<div class=\"prismjs-default copy-to-clipboard \">\n\t\t\t<pre data-line=\"\" class=\"highlight-height language-bash line-numbers\">\n\t\t\t\t<code readonly=\"true\" class=\"language-bash\">\n\t\t\t\t\t<xmp>FROM jenkins\/jenkins:latest\n# disable first-start wizard\nENV JAVA_OPTS -Djenkins.install.runSetupWizard=false\n# copy plugins.txt file\nCOPY plugins.txt .\n# add required plugins based on file\nRUN jenkins-plugin-cli --plugin-file plugins.txt<\/xmp>\n\t\t\t\t<\/code>\n\t\t\t<\/pre>\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-150306c elementor-widget elementor-widget-text-editor\" data-id=\"150306c\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p>The plugins.txt is just a newline separated list of plugin names and version numbers (or \u201clatest\u201d), for example:<\/p><ul><li>blueocean:latest<\/li><li>docker-plugin:latest<\/li><li>git:latest<\/li><li>\u2026<\/li><li>configuration-as-code:latest<\/li><\/ul><p>\u00a0<\/p><p>Like this, you can prepare the the set of plugins according to your needs and build it as an image with a custom tag:<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-0dfd499 elementor-widget elementor-widget-code-highlight\" data-id=\"0dfd499\" data-element_type=\"widget\" data-widget_type=\"code-highlight.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<div class=\"prismjs-default copy-to-clipboard \">\n\t\t\t<pre data-line=\"\" class=\"highlight-height language-bash line-numbers\">\n\t\t\t\t<code readonly=\"true\" class=\"language-bash\">\n\t\t\t\t\t<xmp>docker build . -t jenkins-controller:my-custom-tag<\/xmp>\n\t\t\t\t<\/code>\n\t\t\t<\/pre>\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-f23fa9a elementor-widget elementor-widget-heading\" data-id=\"f23fa9a\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h4 class=\"elementor-heading-title elementor-size-default\">Controlling Versions<\/h4>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-a13551f elementor-widget elementor-widget-text-editor\" data-id=\"a13551f\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p>In theory, we now have control over the Jenkins version and the plugin versions. For convenience reasons, people tend to set many versions to &#8220;latest&#8221;, both the Dockerfile (base image) and in the plugin.txt. Keep in mind that you can do this, but it means that a new build of the image can behave differently, if new versions are available.<br \/>If you want to guarantee that the image is exactly the same, freeze the version numbers and only bump them every half a year or so, in a controlled way. Luckily, evaluating different versions is trivial:<\/p><ul><li>adapt versions<\/li><li>build image<\/li><li>run a container and test<\/li><li>if it works as expected<\/li><\/ul><p>\u00a0<\/p><p>The part that we don&#8217;t control at this point is the configuration of our Jenkins controller. We solve this by using JCasC &#8211; Jenkins Configuration as Code.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-c3ee013 elementor-widget elementor-widget-spacer\" data-id=\"c3ee013\" data-element_type=\"widget\" data-widget_type=\"spacer.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<div class=\"elementor-spacer\">\n\t\t\t<div class=\"elementor-spacer-inner\"><\/div>\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-27e953f elementor-widget elementor-widget-heading\" data-id=\"27e953f\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h3 class=\"elementor-heading-title elementor-size-default\">Step 2 - Jenkins Configuration-as-Code (JCasC)<\/h3>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-82bf7fb elementor-widget elementor-widget-heading\" data-id=\"82bf7fb\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h4 class=\"elementor-heading-title elementor-size-default\">What is it good for?<\/h4>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-6cdaf06 elementor-widget elementor-widget-text-editor\" data-id=\"6cdaf06\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p>The &#8220;configuration-as-code&#8221; paradigm became pretty popular in recent years. When configuring applications at runtime, you&#8217;re relying on backups in case a previous state needs to be restored. But backups usually include everything &#8211; data and configuration.<br \/>Configuration-as-Code sounds more complex than it is. Usually &#8211; and this is also the case for Jenkins &#8211; the configuration state is described in a YAML, XML or JSON file.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-167ee6e elementor-widget elementor-widget-heading\" data-id=\"167ee6e\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h4 class=\"elementor-heading-title elementor-size-default\">How to use it?<\/h4>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-88cf8b4 elementor-widget elementor-widget-text-editor\" data-id=\"88cf8b4\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p>In order to use this in with Jenkins, you need the Jenkins Configuration-as-Code plugin (you can add configuration-as-code:latest to the plugins.txt, so it&#8217;s included in your custom image). By default, the plugin expects a file called jenkins.yaml in the JENKINS_HOME directory.<\/p><p><span style=\"color: var( --e-global-color-text ); font-family: var( --e-global-typography-text-font-family ), Sans-serif; font-size: var( --e-global-typography-text-font-size ); font-style: var( --e-global-typography-text-font-style ); font-weight: var( --e-global-typography-text-font-weight ); letter-spacing: var( --e-global-typography-text-letter-spacing ); text-transform: var( --e-global-typography-text-text-transform ); background-color: var( --e-global-color-8c64e01 );\">So how do we create this file? In case other teams in your company already use it, they may have a template for you to start from, but if not? Do we need to learn, how each of the available configuration options is described? Luckily, we don&#8217;t.<\/span><\/p><p>The main task of the configuration-as-code plugin is to parse the configuration file and apply the respective options but it also provides the opposite transformation step: you can use the Jenkins web UI to configure any Jenkins settings (including plugin settings) and export a jenkins.yaml that represents the current state of your Jenkins controller. Just be sure to remove any options that you don&#8217;t need or are unsure about. We want the config file to only list the things where we override the defaults. This way, the file stays readable and maintainable. Later, we will add it to a git repository.<\/p><p>There are a bunch of step-by-step guides available on the web. I can recommend <a href=\"https:\/\/www.digitalocean.com\/community\/tutorials\/how-to-automate-jenkins-setup-with-docker-and-jenkins-configuration-as-code\" target=\"_blank\" rel=\"noopener\">Daniel and Kathryn&#8217;s guide<\/a> on the topic.<\/p><p>Our list of files, that controls the Jenkins controller now looks like this:<\/p><ol><li>A <strong>Dockerfile<\/strong> to describe how our custom Jenkins controller image is created<\/li><li>A <strong>plugin.txt<\/strong> file that lists the plugins and versions that we want to use<\/li><li>A <strong>jenkins.yaml<\/strong> file with the configuration options that deviate from the defaults<\/li><\/ol>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-bffb02f elementor-widget elementor-widget-image\" data-id=\"bffb02f\" data-element_type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<a href=\"https:\/\/www.btc-embedded.com\/wp-content\/uploads\/2022\/09\/blog_1.png\" data-elementor-open-lightbox=\"yes\" data-elementor-lightbox-title=\"blog_1\" data-e-action-hash=\"#elementor-action%3Aaction%3Dlightbox%26settings%3DeyJpZCI6MTA4ODksInVybCI6Imh0dHBzOlwvXC93d3cuYnRjLWVtYmVkZGVkLmNvbVwvd3AtY29udGVudFwvdXBsb2Fkc1wvMjAyMlwvMDlcL2Jsb2dfMS5wbmcifQ%3D%3D\">\n\t\t\t\t\t\t\t<img fetchpriority=\"high\" decoding=\"async\" width=\"800\" height=\"462\" src=\"https:\/\/www.btc-embedded.com\/wp-content\/uploads\/2022\/09\/blog_1.png\" class=\"attachment-large size-large wp-image-10889\" alt=\"\" srcset=\"https:\/\/www.btc-embedded.com\/wp-content\/uploads\/2022\/09\/blog_1.png 3602w, https:\/\/www.btc-embedded.com\/wp-content\/uploads\/2022\/09\/blog_1-768x443.png 768w, https:\/\/www.btc-embedded.com\/wp-content\/uploads\/2022\/09\/blog_1-1536x887.png 1536w, https:\/\/www.btc-embedded.com\/wp-content\/uploads\/2022\/09\/blog_1-2048x1183.png 2048w\" sizes=\"(max-width: 800px) 100vw, 800px\" \/>\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-93c295f elementor-widget elementor-widget-text-editor\" data-id=\"93c295f\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\tSo far we&#8217;ve only concerned ourselves with the Jenkins controller. But this is just a part of the picture. Say hello to dynamic agents!\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-8f3060b elementor-widget elementor-widget-heading\" data-id=\"8f3060b\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h3 class=\"elementor-heading-title elementor-size-default\">Step 3 - Dynamic Jenkins agents<\/h3>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-725734a elementor-widget elementor-widget-text-editor\" data-id=\"725734a\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p>We&#8217;ve established how and why to run Jenkins in docker but the Jenkins controller is only a part of the whole picture. Any material work should not run on the Jenkins controller but should be performed by agents.<\/p><p>We&#8217;ve already established that we want to avoid preparing static servers \/ virtual machines, as they are hard to maintain and don&#8217;t scale. So why not also use docker to provide agents running in containers?<\/p><p>They can be created on demand and destroyed after use and ensure a reproducible state. The workload that our Jenkins setup supports can then easily scale up while the resource consumption is kept at a minimal when no jobs are running.<\/p><p>In order to achieve this, we need to configure a &#8220;Docker Cloud&#8221; in the &#8220;Manage nodes and clouds&#8221;-section of Jenkins (requires the docker plugin to be installed: we&#8217;ve add &#8220;docker:latest&#8221; to cover this)<\/p><ul><li>Provide connection details to access a docker host<ul><li>Option A: Make the docker socket available via network by changing the docker daemon configuration (<a href=\"https:\/\/medium.com\/trabe\/using-docker-engine-api-securely-584e0882158e\" target=\"_blank\" rel=\"noopener\">step-by-step guide from Mart\u00edn Lamas<\/a>)<\/li><li>Option B: Use <a href=\"https:\/\/hub.docker.com\/r\/alpine\/socat\" target=\"_blank\" rel=\"noopener\">Socat<\/a> to make the docker engine available via network<\/li><li>I&#8217;ve had troubles with option A on my Mac and used option B<\/li><\/ul><\/li><li>Prepare a docker image for jenkins agents and select it in the agent-templates section<ul><li>In order to be able to start docker containers from the dockerized Jenkins agent, I use an image that includes the docker static binary<\/li><li>The Jenkins Pipeline integration for docker can then call docker commands on our dockerized agents which will use the host&#8217;s docker engine and effectively spin up the container as a sibling to the Jenkins agent container<\/li><li>By adding them to the same docker network, http connections (e.g. for REST API calls) are also possible between the Jenkins agent and any sibling containers<\/li><li>There&#8217;s a great\u00a0<a href=\"https:\/\/anthony-f-tannous.medium.com\/setup-a-jenkins-local-devops-environment-using-docker-and-wsl2-c74ca47db9be\" target=\"_blank\" rel=\"noopener\">step-by-step guide by Tony Tannous<\/a> &#8211; in chapter 3 he explains this step-by-step<a href=\"https:\/\/anthony-f-tannous.medium.com\/setup-a-jenkins-local-devops-environment-using-docker-and-wsl2-c74ca47db9be\" target=\"_blank\" rel=\"noopener\"><br \/><\/a>about this\u00a0<\/li><\/ul><\/li><li>Prepare docker images with the required software<ul><li>In our domain, these are tools for model-based development code generation and test like <a href=\"https:\/\/www.btc-embedded.com\/test_environments\/mathworks-simulink\/\">Matlab Simulink<\/a>, <a href=\"https:\/\/www.btc-embedded.com\/test_environments\/dspace-targetlink\/\">dSpace TargetLink<\/a> and <a href=\"https:\/\/www.btc-embedded.com\/btc-embeddedplatform\/\">BTC EmbeddedPlatform<\/a><\/li><li>Providing these images allows your Pipeline to work on naked environments, as long as they support docker containers<\/li><\/ul><\/li><\/ul><p>By configuring the Docker Cloud to be used &#8220;as much as possible&#8221;, any pipelines that we start from now on can use these dynamic agents. The relevant parts of my jenkins.yaml file now look like this:<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-262f2bb elementor-widget elementor-widget-code-highlight\" data-id=\"262f2bb\" data-element_type=\"widget\" data-widget_type=\"code-highlight.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<div class=\"prismjs-default copy-to-clipboard \">\n\t\t\t<pre data-line=\"\" class=\"highlight-height language-bash line-numbers\">\n\t\t\t\t<code readonly=\"true\" class=\"language-bash\">\n\t\t\t\t\t<xmp>jenkins:\n  numExecutors: 0 # disable executions on master\n    \n  clouds:\n    - docker:\n        dockerApi:\n          dockerHost:\n            uri: \"tcp:\/\/192.168.19.112:2375\"\n        name: \"docker\"\n        templates:\n        - connector: \n            attach:\n              user: \"root\"\n          dockerTemplateBase:\n            image: \"jenkins\/agent:custom\" # agent with docker cli binaries\n            mounts: # access to the host's docker engine allows to mount sibling containers\n            - \"type=bind,source=\/var\/run\/docker.sock,destination=\/var\/run\/docker.sock\"\n            network: \"jdn\"\n          labelString: \"linux\"\n          name: \"linux\"\n          remoteFs: \"\/home\/jenkins\/agent\"\n  \n  tool:\n    git:\n      installations:\n      - home: \"git\"\n        name: \"Default\"\n        \nunclassified:\n  location:\n    url: \"http:\/\/192.168.0.20:8080\/\"<\/xmp>\n\t\t\t\t<\/code>\n\t\t\t<\/pre>\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-fd0c198 elementor-widget elementor-widget-text-editor\" data-id=\"fd0c198\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\tWe&#8217;ve also added a couple of items to our list of configuration files:\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-cce8834 elementor-widget elementor-widget-image\" data-id=\"cce8834\" data-element_type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<a href=\"https:\/\/www.btc-embedded.com\/wp-content\/uploads\/2022\/09\/blog_2.png\" data-elementor-open-lightbox=\"yes\" data-elementor-lightbox-title=\"blog_2\" data-e-action-hash=\"#elementor-action%3Aaction%3Dlightbox%26settings%3DeyJpZCI6MTA4OTIsInVybCI6Imh0dHBzOlwvXC93d3cuYnRjLWVtYmVkZGVkLmNvbVwvd3AtY29udGVudFwvdXBsb2Fkc1wvMjAyMlwvMDlcL2Jsb2dfMi5wbmcifQ%3D%3D\">\n\t\t\t\t\t\t\t<img decoding=\"async\" width=\"800\" height=\"462\" src=\"https:\/\/www.btc-embedded.com\/wp-content\/uploads\/2022\/09\/blog_2.png\" class=\"attachment-large size-large wp-image-10892\" alt=\"\" srcset=\"https:\/\/www.btc-embedded.com\/wp-content\/uploads\/2022\/09\/blog_2.png 3602w, https:\/\/www.btc-embedded.com\/wp-content\/uploads\/2022\/09\/blog_2-768x443.png 768w, https:\/\/www.btc-embedded.com\/wp-content\/uploads\/2022\/09\/blog_2-1536x887.png 1536w, https:\/\/www.btc-embedded.com\/wp-content\/uploads\/2022\/09\/blog_2-2048x1183.png 2048w\" sizes=\"(max-width: 800px) 100vw, 800px\" \/>\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-da6f93a elementor-widget elementor-widget-heading\" data-id=\"da6f93a\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h3 class=\"elementor-heading-title elementor-size-default\">Step 4 - Repo Discovery and Automatic Pipeline Creation<\/h3>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-fa217fc elementor-widget elementor-widget-text-editor\" data-id=\"fa217fc\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p>We now have a dockerized Jenkins controller, with configuration-as-code and dynamic agents that can run container defined in the pipeline (Jenkinsfile). Creating a job for a repository that contains a Jenkinsfile is pretty straight forward from this point on:<\/p><p>New Item &gt; Pipeline &gt; add Git URL &gt; Save &gt; Run<\/p><p>But what if we already have dozens of repositories and don&#8217;t want to manually manage creating the jobs for each one? For some SCM providers, you might find plugins that scan your repositories and do the job for you. However, I&#8217;ve always found this to be slightly messy, automatically creating multi-branch pipelines for all repositories.<\/p><p>I&#8217;ve chosen an alternative approach that works with any SCM provider that offers an API:<\/p><ul><li>Add the job-dsl plugin (add to plugins.txt)<\/li><li>Create a seed job<ul><li>Create repository templates<ul><li>Query relevant repositories from the SCM server<\/li><li>Create a job-dsl template for each repository<\/li><li>Create a webhook for each repository to trigger the Jenkins pipeline on certain events (e.g., git push)<\/li><\/ul><\/li><li>Auto-Create all jobs using the job dsl plugin on the created templates<\/li><\/ul><\/li><\/ul><p>This approach relies on the Job DSL plugin, that allows to automate job creation and update for Jenkins.\u00a0For further information on the general use of Job DSL, check out this <a href=\"https:\/\/www.digitalocean.com\/community\/tutorials\/how-to-automate-jenkins-job-configuration-using-job-dsl\" target=\"_blank\" rel=\"noopener\">tutorial from Daniel<\/a>.\u00a0<\/p><p>As mentioned above, this would work for any SCM provider \/ git service that offers an API to query some basic data about its repositories. I used Gogs (Go Git Service) that feels pretty similar to GitHub in many regards. I&#8217;ve implemented this for Gogs (<a href=\"https:\/\/github.com\/thabok\/gogs-repo-discovery\" target=\"_blank\" rel=\"noopener\">https:\/\/github.com\/thabok\/gogs-repo-discovery<\/a>), but the code is pretty specific and will need some adaptations to use it for another git service, different repository structures, etc.<\/p><p>Now, in addition to the Jenkins controller container and the socat container to enable tcp access of the docker engine, Gogs was now the third container that I relied on. In order to manage the different containers, I decided to use docker-compose. This way, I can start the complete stack (jenkins + socat + gogs) with a simple docker-compose up instead of having to start three containers individually while having to remember a bunch of settings for each one:<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-0f350a0 elementor-widget elementor-widget-code-highlight\" data-id=\"0f350a0\" data-element_type=\"widget\" data-widget_type=\"code-highlight.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<div class=\"prismjs-default copy-to-clipboard \">\n\t\t\t<pre data-line=\"3,12,21\" class=\"highlight-height language-bash line-numbers\">\n\t\t\t\t<code readonly=\"true\" class=\"language-bash\">\n\t\t\t\t\t<xmp>services:\n\n  jenkins_controller:\n    image: jenkins\/controller:custom\n    container_name: jenkins_controller\n    volumes:\n        - .\/jenkins_home:\/var\/jenkins_home\n    ports:\n        - \"8080:8080\"\n        - \"50000:50000\"\n\n  socat_bridge:\n    image: alpine\/socat\n    command: tcp-listen:2375,fork,reuseaddr unix-connect:\/var\/run\/docker.sock\n    container_name: socat_bridge\n    volumes:\n      - \/var\/run\/docker.sock:\/var\/run\/docker.sock\n    ports:\n      - 2375:2375\n\n   gogs_git_server:\n    image: gogs\/gogs\n    container_name: gogs_git_server\n    volumes:\n        - .\/gogs\/data:\/data\n    ports:\n        - \"10022:22\"\n        - \"3000:3000\"\n<\/xmp>\n\t\t\t\t<\/code>\n\t\t\t<\/pre>\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-7303e71 elementor-widget elementor-widget-heading\" data-id=\"7303e71\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h3 class=\"elementor-heading-title elementor-size-default\">Piecing it all together<\/h3>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-79c29f9 elementor-widget elementor-widget-text-editor\" data-id=\"79c29f9\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p>We&#8217;ve looked into many aspects of isolating our applications in containers and applying cofiguration-as-code, all with the goal to achieve a distributed setup where teams have ownership. By removing complex environment requirements we can leverage the benefits of scalable cloud resources.<\/p><p>After I applied these steps to my demo setup, it now looks like this and I can finally start up everything with a simple docker-compose up:<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-6cce638 elementor-widget elementor-widget-image\" data-id=\"6cce638\" data-element_type=\"widget\" data-widget_type=\"image.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<a href=\"https:\/\/www.btc-embedded.com\/wp-content\/uploads\/2022\/09\/blog_3.png\" data-elementor-open-lightbox=\"yes\" data-elementor-lightbox-title=\"blog_3\" data-e-action-hash=\"#elementor-action%3Aaction%3Dlightbox%26settings%3DeyJpZCI6MTA4OTUsInVybCI6Imh0dHBzOlwvXC93d3cuYnRjLWVtYmVkZGVkLmNvbVwvd3AtY29udGVudFwvdXBsb2Fkc1wvMjAyMlwvMDlcL2Jsb2dfMy5wbmcifQ%3D%3D\">\n\t\t\t\t\t\t\t<img decoding=\"async\" width=\"800\" height=\"462\" src=\"https:\/\/www.btc-embedded.com\/wp-content\/uploads\/2022\/09\/blog_3.png\" class=\"attachment-large size-large wp-image-10895\" alt=\"\" srcset=\"https:\/\/www.btc-embedded.com\/wp-content\/uploads\/2022\/09\/blog_3.png 3602w, https:\/\/www.btc-embedded.com\/wp-content\/uploads\/2022\/09\/blog_3-768x443.png 768w, https:\/\/www.btc-embedded.com\/wp-content\/uploads\/2022\/09\/blog_3-1536x887.png 1536w, https:\/\/www.btc-embedded.com\/wp-content\/uploads\/2022\/09\/blog_3-2048x1183.png 2048w\" sizes=\"(max-width: 800px) 100vw, 800px\" \/>\t\t\t\t\t\t\t\t<\/a>\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-53e1572 elementor-widget elementor-widget-spacer\" data-id=\"53e1572\" data-element_type=\"widget\" data-widget_type=\"spacer.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t<div class=\"elementor-spacer\">\n\t\t\t<div class=\"elementor-spacer-inner\"><\/div>\n\t\t<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-cf6fff1 elementor-widget elementor-widget-heading\" data-id=\"cf6fff1\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h2 class=\"elementor-heading-title elementor-size-default\">Final comments<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-ef2476b elementor-widget elementor-widget-text-editor\" data-id=\"ef2476b\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p>I&#8217;ve raised a lot of technical reasons to move applications and services into docker, apply configuration-as-code, etc. Ultimately, we want to solve a problem of having slow and unmanageable CI setups that require a lot of support from IT \/ infrastructure teams.<\/p><p>We solve it by decentralizing our CI and providing solutions per team rather than for the complete organization. This allows you to grant full control to development teams, lifting the weight off of central IT and increasing satisfaction on both sides.<\/p><p>By moving away from manual installations, big resource overheads that make scaling costly and setups, that require a hundred things in the environment to be perfectly prepared, we enable a shift towards team ownership. The isolated nature of containerized environments allows us to involve cloud resources for better scalability at controlled cost.<\/p><p>Development teams can now constantly improve their solutions, without waiting for hardware or the central IT team. Upgrading to a new version of a software tool can be done, simply by increasing the version number in the respective Dockerfile.<\/p><p>And finally: No &#8211; Jenkins is not dead. But the competition is modern, cloud native and is less chained by legacy projects that want to be supported.\u00a0<span style=\"color: var( --e-global-color-text ); font-family: var( --e-global-typography-text-font-family ), Sans-serif; font-size: var( --e-global-typography-text-font-size ); font-style: var( --e-global-typography-text-font-style ); font-weight: var( --e-global-typography-text-font-weight ); letter-spacing: var( --e-global-typography-text-letter-spacing ); text-transform: var( --e-global-typography-text-text-transform ); background-color: var( --e-global-color-8c64e01 );\">Luckily <a href=\"https:\/\/www.btc-embedded.com\/btc-embeddedplatform\/\">our products<\/a> come with a <a href=\"https:\/\/www.btc-embedded.com\/use_cases\/continuous-integration-cloud\/\">REST API<\/a> that allows to use them with any modern CI solution.\u00a0<\/span><span style=\"color: var( --e-global-color-text ); font-family: var( --e-global-typography-text-font-family ), Sans-serif; font-size: var( --e-global-typography-text-font-size ); font-style: var( --e-global-typography-text-font-style ); font-weight: var( --e-global-typography-text-font-weight ); letter-spacing: var( --e-global-typography-text-letter-spacing ); text-transform: var( --e-global-typography-text-text-transform ); background-color: var( --e-global-color-8c64e01 );\">I love working with Jenkins and it&#8217;s still the go-to CI solution at most of our customers, but it&#8217;s good to be prepared for a more varied CI landscape in the years to come.<\/span><\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-76a97cf elementor-widget elementor-widget-heading\" data-id=\"76a97cf\" data-element_type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t<h3 class=\"elementor-heading-title elementor-size-default\">Credits<\/h3>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-bcd18bb elementor-widget elementor-widget-text-editor\" data-id=\"bcd18bb\" data-element_type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t<div class=\"elementor-widget-container\">\n\t\t\t\t\t\t\t\t\t<p>My personal motivation for this article was to prepare a fully dockerized setup based on as much configuration as code as possible, to provide a flexible and recreatable environment to do demonstrations for customers. The memory of the talk &#8220;<a href=\"https:\/\/www.youtube.com\/watch?v=47D3H1BZi4o\" target=\"_blank\" rel=\"noopener\">Look Ma, no Hands!<\/a>&#8221; by Ewelina Wilkosz &amp; Nicolas De Loof and Tony Tannous&#8217; article on a <a href=\"https:\/\/anthony-f-tannous.medium.com\/setup-a-jenkins-local-devops-environment-using-docker-and-wsl2-c74ca47db9be\" target=\"_blank\" rel=\"noopener\">local docker-based DevOps environment<\/a> gave the initial spark.<\/p><p>Full list of references:<\/p><ul><li>Ewelina Wilkosz, Nicolas De Loof &#8211; <a href=\"https:\/\/www.youtube.com\/watch?v=47D3H1BZi4o\" target=\"_blank\" rel=\"noopener\">Look Ma, no Hands!<\/a><\/li><li>Tony Tannous &#8211; <a href=\"https:\/\/anthony-f-tannous.medium.com\/setup-a-jenkins-local-devops-environment-using-docker-and-wsl2-c74ca47db9be\" target=\"_blank\" rel=\"noopener\">Setup a Jenkins Local DevOps Environment using Docker and WSL2<\/a><\/li><li>Daniel Li, Kathryn Hancox &#8211; <a href=\"https:\/\/www.digitalocean.com\/community\/tutorials\/how-to-automate-jenkins-setup-with-docker-and-jenkins-configuration-as-code\" target=\"_blank\" rel=\"noopener\">How To Automate Jenkins Setup with Docker and Jenkins Configuration as Code<\/a><\/li><li>Daniel Li, Kathryn Hancox &#8211; <a href=\"https:\/\/www.digitalocean.com\/community\/tutorials\/how-to-automate-jenkins-job-configuration-using-job-dsl\" target=\"_blank\" rel=\"noopener\">How To Automate Jenkins Job Configuration Using Job DSL<\/a><\/li><li>Mart\u00edn Lamas &#8211; <a href=\"https:\/\/medium.com\/trabe\/using-docker-engine-api-securely-584e0882158e\" target=\"_blank\" rel=\"noopener\">Using Docker Engine API securely<\/a><\/li><li><a href=\"https:\/\/hub.docker.com\/r\/alpine\/socat\" target=\"_blank\" rel=\"noopener\">Socat on Dockerhub<\/a><\/li><\/ul>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<\/div>\n\t\t","protected":false},"excerpt":{"rendered":"<p>Introduction During the last six years I have worked wi [&hellip;]<\/p>\n","protected":false},"author":7,"featured_media":11236,"comment_status":"open","ping_status":"closed","sticky":false,"template":"elementor_theme","format":"standard","meta":{"_acf_changed":false,"inline_featured_image":false,"footnotes":""},"categories":[1],"tags":[55,56,54],"product":[],"use_cases":[],"class_list":["post-10179","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-uncategorized","tag-ci-cd","tag-cloud","tag-jenkins"],"acf":[],"_links":{"self":[{"href":"https:\/\/www.btc-embedded.com\/zh-hans\/wp-json\/wp\/v2\/posts\/10179","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.btc-embedded.com\/zh-hans\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.btc-embedded.com\/zh-hans\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.btc-embedded.com\/zh-hans\/wp-json\/wp\/v2\/users\/7"}],"replies":[{"embeddable":true,"href":"https:\/\/www.btc-embedded.com\/zh-hans\/wp-json\/wp\/v2\/comments?post=10179"}],"version-history":[{"count":0,"href":"https:\/\/www.btc-embedded.com\/zh-hans\/wp-json\/wp\/v2\/posts\/10179\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.btc-embedded.com\/zh-hans\/wp-json\/wp\/v2\/media\/11236"}],"wp:attachment":[{"href":"https:\/\/www.btc-embedded.com\/zh-hans\/wp-json\/wp\/v2\/media?parent=10179"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.btc-embedded.com\/zh-hans\/wp-json\/wp\/v2\/categories?post=10179"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.btc-embedded.com\/zh-hans\/wp-json\/wp\/v2\/tags?post=10179"},{"taxonomy":"product","embeddable":true,"href":"https:\/\/www.btc-embedded.com\/zh-hans\/wp-json\/wp\/v2\/product?post=10179"},{"taxonomy":"use_cases","embeddable":true,"href":"https:\/\/www.btc-embedded.com\/zh-hans\/wp-json\/wp\/v2\/use_cases?post=10179"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}