Kubernetes CI/CD with Jenkins, Buildkit, and Helm (K3s)

What You Will Build

• GitHub push triggers Jenkins.
• Only changed services build and deploy.
• Buildkit builds and pushes container images.
• Helm deploys to K3s with dynamic hostnames.
• Merged branches are cleaned up automatically.

Prerequisites

• Jenkins with Kubernetes plugin enabled.
• Global Pipeline Library configured in Jenkins.
• K3s or Kubernetes cluster with Helm installed.
• Container registry like `registry.com`.
• Ingress controller and a base domain.

Step 0: Configure Jenkins Global Library

In Jenkins, go to Manage Jenkins → Configure System → Global Pipeline Libraries and add a library named `jenkins-global-pipeline`. Set the default version to your main branch and enable “Load implicitly” if you want to avoid repeating the `@Library` line in Jenkinsfiles.

Files in This Setup

Folder layout used in the Jenkins library and Helm chart.

Jump to file sections:kub.groovyK3sEnv.groovyk3sDeploy.groovyChart.yamlvalues.yamldeployment.yamlservice.yamlingress.yaml

Step 1: Entry Point in vars/kub.groovy

This is the pipeline entry. It sets up the environment, creates parallel stages per service, runs audit, build, deploy, and cleanup.

import k3s.k3sDeploy
import k3s.K3sEnv

def call(Map appConfig = [:], List deployemnts = []) {

    def parallelStages = [:]

    // MAIN START
    properties([disableConcurrentBuilds()])

    node {
        // Initialize our unified Environment class
        def k3sEnv = new K3sEnv(this)

        stage("Detect Changes & Setup Env") {
            // Function to configure environment variables and source code
            k3sEnv.initEnv()
            
            // Example of adding a custom value:
            // k3sEnv.setEnv('MY_CUSTOM_VAR', 'my-value')
        }

        deployemnts.each { app ->
            app.each { name, config ->
                def servicePath = config.path

                parallelStages[servicePath] = {
                    if (k3sEnv.shouldRunService(servicePath)) {
                        def deployer = new k3sDeploy(this, appConfig, name, config, servicePath)
                        def deployName = k3sEnv.getEnv('DEPLOY_NAME')
                        
                        stage("Npm Audit ${servicePath}") {
                            deployer.npmAudit()
                        }

                        stage("Build & Push ${servicePath}") {
                            deployer.buildAndPush()
                        }
                        
                        if (deployName == 'dev' || deployName == 'sprint') {
                            stage("Deploy ${servicePath}") {
                                deployer.deploy()
                            }
                        } else {
                            echo "Skipping Deploy for ${servicePath}: Deploy Name is not 'dev' or 'sprint' (Current: ${deployName})"
                        }
                        
                        if (k3sEnv.getEnv('IS_MERGE') == true || k3sEnv.getEnv('IS_MERGE') == 'true') {
                            stage("Cleanup Merged Source ${servicePath}") {
                                deployer.deleteMergedDeploy()
                            }
                        }
                    }
                }
            }
        }

        if (parallelStages) {
            parallel parallelStages
        } else {
            echo "No deployments provided."
        }
    }
}

How it works: builds run in parallel for each service path. Deploy only runs for `dev` and `sprint` branches. Cleanup runs on merge.

Step 2: Environment Logic in src/k3s/K3sEnv.groovy

This file builds the entire runtime context. It detects changed files, extracts repo/org, sets registry and domain, and derives deploy ID from branch names.

package k3s

class K3sEnv implements Serializable {
    def script
    Map envMap = [:]

    K3sEnv(script) {
        this.script = script
    }

    // Function to run the logic previously found in kub.groovy
    def initEnv() {
        script.checkout(script.scm)

        // Get changed files from last commit
        def changedFilesStr = script.sh(
            script: "git diff --name-only HEAD~1 HEAD || true",
            returnStdout: true
        ).trim()
        
        def changedFiles = changedFilesStr ? changedFilesStr.split("\n") as List : []
        setEnv('CHANGED_FILES', changedFiles)

        // Get remote configs to extract repository metadata
        def url = script.scm.getUserRemoteConfigs()[0].getUrl()
        
        // REPO NAME
        def repoName = url.tokenize('/.')[-2]
        setEnv('REPO_NAME', repoName)
        
        // ORG NAME
        def gitOrg = url.tokenize('/.')[-3]
        setEnv('GIT_ORG', gitOrg)
        
        // REGISTRY URL
        setEnv('REGISTRY_URL', "registry.com")
        // BASE DOMAIN
        setEnv('BASE_DOMAIN', "deploy.example.com")

        // IDENTIFY DEPLOY ID FROM BRANCH
        def branchName = script.env.BRANCH_NAME ?: 'latest'
        
        // CHECK IF MERGE REQUEST
        def isMerge = false
        def sourceBranch = ""
        def targetBranch = branchName
        
        // Typical git merge commit messages look like: "Merge branch 'feature-x' into 'dev'"
        def commitMsg = script.sh(script: "git log -1 --pretty=%B || true", returnStdout: true).trim()

        script.echo "===== COMMIT MESSAGE ===="
        script.echo "Commit Message: ${commitMsg}"
        script.echo "========================"
        
        // Simple regex to match "Merge branch 'X' into 'Y'" or similar standard merge commits
        def mergeMatcher = (commitMsg =~ /Merge branch '([^']+)' into '([^']+)'/)
        if (mergeMatcher.find()) {
            isMerge = true
            sourceBranch = mergeMatcher.group(1)
            targetBranch = mergeMatcher.group(2)
            script.echo "=== MERGE DETECTED ==="
            script.echo "Branch '${sourceBranch}' merged into branch '${targetBranch}'"
            script.echo "======================"
        } else {
            // Check alternative format "Merge pull request #123 from org/branch"
            def prMatcher = (commitMsg =~ /Merge pull request #[0-9]+ from [^\/]+\/([^\s]+)/)
            if (prMatcher.find()) {
                isMerge = true
                sourceBranch = prMatcher.group(1)
                script.echo "=== PR MERGE DETECTED ==="
                script.echo "Branch '${sourceBranch}' merged into branch '${targetBranch}'"
                script.echo "========================="
            }
        }
        
        setEnv('IS_MERGE', isMerge)
        setEnv('SOURCE_BRANCH', sourceBranch)
        
        // Parse source branch details (the branch that was merged)
        def sourceBranchDetails = getBranchDetails(sourceBranch)
        setEnv('SOURCE_DEPLOY_ID', sourceBranchDetails.deployId)
        setEnv('SOURCE_DEPLOY_NAME', sourceBranchDetails.deployName)
        
        // Parse current branch details
        def currentBranchDetails = getBranchDetails(branchName)
        def deployId = currentBranchDetails.deployId
        setEnv('DEPLOY_ID', deployId)
        setEnv('DEPLOY_NAME', currentBranchDetails.deployName)
        if (currentBranchDetails.sprintNumber) {
            setEnv('SPRINT_NUMBER', currentBranchDetails.sprintNumber)
        }
        
        // Parse target branch details (useful if it's a merge)
        def targetBranchDetails = getBranchDetails(targetBranch)
        setEnv('TARGET_BRANCH', targetBranch)
        setEnv('TARGET_DEPLOY_ID', targetBranchDetails.deployId)
        setEnv('TARGET_DEPLOY_NAME', targetBranchDetails.deployName)

        script.echo "REPO NAME: ${repoName}"
        script.echo "ORG NAME: ${gitOrg}"
        script.echo "BRANCH: ${branchName} | DEPLOY ID: ${deployId}"
        script.echo "Changed files: ${changedFiles}"
    }

    // A simple method to add any custom variable
    def setEnv(String key, def value) {
        this.envMap[key] = value
        // Optional: Keep Jenkins native env in sync so standard plugins can also read it
        if (value instanceof String || value instanceof GString || value instanceof Boolean || value instanceof Integer) {
            script.env[key] = value.toString()
        }
    }

    // Retrieve the variable from class storage or natively from script.env
    def getEnv(String key) {
        return this.envMap[key] ?: script.env[key]
    }

    // Function to check if a specific service should be built/deployed
    def shouldRunService(String servicePath) {
        def branchName = script.env.BRANCH_NAME ?: 'latest'
        
        // Ensure we only build for permitted branches
        if (!(branchName == 'dev' || branchName == 'prod' || branchName.startsWith('sprint-'))) {
            script.echo "Skipping ${servicePath}: Branch '${branchName}' is not dev, prod, or sprint."
            return false
        }

        // Check if build was triggered manually or is the first build
        def isManual = false
        if (script.currentBuild && script.currentBuild.getBuildCauses()) {
            isManual = script.currentBuild.getBuildCauses().any { it._class == 'hudson.model.Cause$UserIdCause' }
        }
        
        def isFirstBuild = script.env.BUILD_NUMBER == '1'
        
        // Ensure exact path matching by expecting a trailing slash (e.g. 'api/') if checking directories
        // or just using exactly matches or startsWith. Depending on git diff output:
        def pathToCheck = servicePath.endsWith('/') ? servicePath : "${servicePath}/"
        def changedFiles = getEnv('CHANGED_FILES') as List ?: []
        def hasChanges = changedFiles.any { it.startsWith(pathToCheck) }
        
        if (isManual || isFirstBuild || hasChanges) {
            script.echo "Service ${servicePath} matches build conditions. (Manual: ${isManual}, FirstBuild: ${isFirstBuild}, Changed: ${hasChanges})"
            return true
        } else {
            script.echo "Skipping ${servicePath}: No changes detected, not triggered manually, and not the first build."
            return false
        }
    }
    // Helper to extract Deploy ID and Deploy Name
    def getBranchDetails(String branch) {
        def deployId = branch
        def deployName = branch
        def sprintNumber = null
        
        if (branch.startsWith('sprint-')) {
            deployName = 'sprint'
            def parts = branch.split('-')
            if (parts.length > 1 && parts[1].matches("\\d+")) {
                deployId = parts[1]
                sprintNumber = parts[1]
            }
        }
        
        return [
            deployId: deployId,
            deployName: deployName,
            sprintNumber: sprintNumber
        ]
    }
}

Replace `registry.com` and `deploy.example.com` with your real registry and base domain before using in production.

Step 3: Build and Deploy in src/k3s/k3sDeploy.groovy

This class handles npm audit, Buildkit image build + push, Helm deployment, and cleanup on merge.

package k3s

class k3sDeploy implements Serializable {
    def script
    Map appConfig
    String name
    Map config
    String servicePath
    String envJsonData = ""

    k3sDeploy(script, Map appConfig, String name, Map config, String servicePath) {
        this.script = script
        this.appConfig = appConfig
        this.name = name
        this.config = config
        this.servicePath = servicePath
    }

    private def getDomain(sl1, sl2, sl3) {
        def baseDomain = script.env.BASE_DOMAIN ?: "deploy.example.com"
        return "${sl1}.${sl2}.${sl3}.${baseDomain}"
    }

    def start() {
        script.echo "Starting deployment process for ${servicePath}..."
    }

    def checkLint() {
        script.echo "Running lint checks for ${servicePath}..."
    }

    def npmAudit() {
        script.podTemplate(
            agentContainer: 'npm-audit',
            agentInjection: true,                            
            containers: [
                script.containerTemplate(name: 'node', image: "node:${config.node_version}-alpine", ttyEnabled: true, privileged: true)
            ]
        ) {
            script.node(script.POD_LABEL) {
                script.checkout(script.scm)
                script.dir(servicePath) {
                    if (script.fileExists('package.json')) {
                        script.container('node') {
                            script.sh 'ls -la'                                            
                            // adding || true to prevent pipeline from failing if audit fails, or maintain old behavior
                            script.sh 'npm audit --audit-level=high'
                        }
                    } else {
                        script.echo "Skipping Npm Audit: No package.json found in ${servicePath}"
                    }
                }
            }
        }  
    }

    def buildAndPush() {
        script.podTemplate(
            containers: [
                script.containerTemplate(name: 'buildkit', image: 'moby/buildkit:rootless', command: 'sleep', args: '99d', ttyEnabled: true, privileged: true)
            ],
            volumes: [
                script.secretVolume(secretName: 'runner-docker-config', mountPath: '/home/user/.docker')
            ]
        ) {
            script.node(script.POD_LABEL) {
                script.checkout(script.scm)
                script.dir(servicePath) {
                    if (script.fileExists('Dockerfile')) {
                        script.container('buildkit') {
                            def gitOrg = script.env.GIT_ORG ?: 'ORG_NAME'
                            def repoName = script.env.REPO_NAME ?: 'REPO_NAME'                            
                            def deployId = script.env.DEPLOY_ID
                            def imageName = "${script.env.REGISTRY_URL}/${gitOrg}/${repoName}-${servicePath}-${deployId}:latest"

                            // Safe access to appConfig property
                            def apiPath = appConfig?.apiPath ?: ""
                            def API_URL = "https://${getDomain(apiPath, deployId, repoName)}"                              
                            
                            // Save default.json across stages using class property
                            if (script.fileExists("env/default.json")) {
                                this.envJsonData = script.sh(
                                    script: "cat env/default.json | tr -d '[:space:]'", 
                                    returnStdout: true
                                ).trim()
                            }

                            def buildArgsStr = ""
                            if (apiPath && deployId != "prod" && deployId != "master") {
                                buildArgsStr += "--opt build-arg:REACT_APP_BASE_URL=${API_URL} "
                                buildArgsStr += "--opt build-arg:API_BASE_URL=${API_URL} "
                            }

                            script.sh """#!/bin/sh
                                # Handle docker config json mapping
                                if [ -f /home/user/.docker/.dockerconfigjson ]; then
                                    mkdir -p /tmp/.docker
                                    cp /home/user/.docker/.dockerconfigjson /tmp/.docker/config.json
                                    export DOCKER_CONFIG=/tmp/.docker
                                else
                                    export DOCKER_CONFIG=/home/user/.docker
                                fi
                                
                                buildctl-daemonless.sh build \\
                                    --frontend dockerfile.v0 \\
                                    --local context=. \\
                                    --local dockerfile=. \\
                                    --output type=image,name=${imageName},push=true \\
                                    ${buildArgsStr}
                            """
                        }
                    } else {
                        script.echo "Skipping Build & Push: No Dockerfile found in ${servicePath}"
                    }
                }
            }
        }
    }

    def deploy() {
        script.podTemplate(
            containers: [
                script.containerTemplate(name: 'helm', image: 'alpine/helm:3.14.3', command: 'sleep', args: '99d', ttyEnabled: true)
            ],
            volumes: [
                script.secretVolume(secretName: 'runner-docker-config', mountPath: '/home/user/.docker')
            ]
        ) {
            script.node(script.POD_LABEL) {
                script.container('helm') {
                    def gitOrg = script.env.GIT_ORG ?: 'ORG_NAME'
                    def repoName = script.env.REPO_NAME ?: 'REPO_NAME'                    
                    def deployId = script.env.DEPLOY_ID
                    def imageName = "${script.env.REGISTRY_URL}/${gitOrg}/${repoName}-${servicePath}-${deployId}"

                    def service_name = "${repoName}-${servicePath}-${deployId}"
                    def url = getDomain(servicePath, deployId, repoName)

                    script.sh """#!/bin/sh
                       # Setup helm registry credentials from the mounted docker config secret
                       mkdir -p ~/.config/helm/registry
                       if [ -f /home/user/.docker/.dockerconfigjson ]; then
                           cp /home/user/.docker/.dockerconfigjson ~/.config/helm/registry/config.json
                       fi

                       ENV_FLAG=""
                       # Retrieve envJsonData class property populated in buildAndPush
                       if [ -n '${this.envJsonData}' ]; then
                           echo '${this.envJsonData}' | tr -d '[:space:]' > ${service_name}-env.json
                           ENV_FLAG="--set-file env=${service_name}-env.json"
                       fi
                       
                       PERSISTENCE_FLAG=""
                       if [ -n '${config.volume}' ]; then
                           PERSISTENCE_FLAG="--set persistence.enabled=true --set persistence.mountPath='${config.volume}'"
                       fi

                       helm upgrade --install ${service_name} oci://${script.env.REGISTRY_URL}/helm/${name.toLowerCase()} \\
                       --set fullnameOverride=${service_name} \\
                       --set ingress.host=${url} \\
                       --set image.repository=${imageName} \\
                       --set image.tag=latest \\
                       \$ENV_FLAG \\
                       \$PERSISTENCE_FLAG \\
                       --namespace ms-dev
                    """
                }
            }
        }
    }

    def deleteMergedDeploy() {
        def isMerge = script.env.IS_MERGE == 'true' || script.env.IS_MERGE == true
        if (!isMerge) {
            script.echo "Skipping Merge Cleanup: Not a merge event."
            return
        }

        def sourceBranch = script.env.SOURCE_BRANCH
        if (!sourceBranch) {
            script.echo "Skipping Merge Cleanup: SOURCE_BRANCH is empty."
            return
        }

        // Get the Deploy ID that would have been used for the source branch
        // We reuse the pre-calculated value from K3sEnv
        def sourceDeployId = script.env.SOURCE_DEPLOY_ID ?: sourceBranch

        script.podTemplate(
            containers: [
                script.containerTemplate(name: 'helm', image: 'alpine/helm:3.14.3', command: 'sleep', args: '99d', ttyEnabled: true)
            ],
            volumes: [
                script.secretVolume(secretName: 'runner-docker-config', mountPath: '/home/user/.docker')
            ]
        ) {
            script.node(script.POD_LABEL) {
                script.container('helm') {
                    def repoName = script.env.REPO_NAME ?: 'REPO_NAME'
                    def service_name = "${repoName}-${servicePath}-${sourceDeployId}"

                    script.echo "Attempting to cleanup merged deployment: ${service_name} (from branch ${sourceBranch})"

                    script.sh """#!/bin/sh
                       # Setup helm registry credentials from the mounted docker config secret
                       mkdir -p ~/.config/helm/registry
                       if [ -f /home/user/.docker/.dockerconfigjson ]; then
                           cp /home/user/.docker/.dockerconfigjson ~/.config/helm/registry/config.json
                       fi

                       # Run helm uninstall
                       helm uninstall ${service_name} --namespace ms-dev || echo "Deployment ${service_name} not found or already deleted."
                    """
                }
            }
        }
    }
}

How it works: image tags include repo + service + deployId, so each branch gets an isolated deployment and URL.

Step 4: Helm Chart Metadata (Chart.yaml)

apiVersion: v2
name: nextjs
description: A Helm chart for deploying generic NextJS fullstack applications
type: application
version: 0.1.0
appVersion: "1.0.0"

# UPDATE HELM REPO: helm registry login registry.com --username USERNAME --password PASSWORD
# PUSH HELM CHART: helm package . && helm push nextjs-0.1.0.tgz oci://registry.com/helm

Step 5: Helm Values (values.yaml)

These values are what the pipeline overrides at deploy time.

fullnameOverride: "next-js-test"
namespace: "ms-dev"

replicaCount: 1

image:
  repository: registry.com/test
  tag: latest

service:
  port: 3000

ingress:
  enabled: true
  host: test.dev.example.com

Step 6: Deployment Template (deployment.yaml)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: "{{ include \"nextjs.fullname\" . }}"
  namespace: "{{ .Values.namespace }}"
spec:
  replicas: {{ .Values.replicaCount }}
  selector:
    matchLabels:
      app: "{{ include \"nextjs.fullname\" . }}"
  template:
    metadata:
      annotations:
        rollout: "{{ now | unixEpoch }}"
      labels:
        app: "{{ include \"nextjs.fullname\" . }}"
    spec:
      imagePullSecrets:
        - name: msdev-k303-registry-secret
      containers:
        - name: nextjs
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
          imagePullPolicy: Always
          ports:
            - containerPort: {{ .Values.service.port }}

Step 7: Service Template (service.yaml)

apiVersion: v1
kind: Service
metadata:
  name: "{{ include \"nextjs.fullname\" . }}"
  namespace: "{{ .Values.namespace }}"
spec:
  type: ClusterIP
  selector:
    app: "{{ include \"nextjs.fullname\" . }}"
  ports:
    - port: {{ .Values.service.port }}
      targetPort: {{ .Values.service.port }}

Step 8: Ingress Template (ingress.yaml)

{{- if .Values.ingress.enabled -}}
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: "{{ include \"nextjs.fullname\" . }}"
  namespace: "{{ .Values.namespace }}"
  annotations:
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/proxy-body-size: "2G"
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - {{ .Values.ingress.host | quote }}
      secretName: "{{ include \"nextjs.fullname\" . }}-tls"
  rules:
    - host: {{ .Values.ingress.host | quote }}
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: "{{ include \"nextjs.fullname\" . }}"
                port:
                  number: {{ .Values.service.port }}
{{- end }}

Step 9: Jenkinsfile Example

Each app repo only needs a simple Jenkinsfile to call the shared library.

@Library('jenkins-global-pipeline') _

def appConfig = [apiPath: \"api\"]
def deployments = [
  [web: [path: \"web\", node_version: \"20\", volume: \"/data\"]],
  [api: [path: \"api\", node_version: \"20\"]]
]

kub(appConfig, deployments)

Step 10: Setup Checklist

• Configure the Jenkins Global Pipeline Library.
• Create Kubernetes secret for registry pull.
• Configure `runner-docker-config` secret for Buildkit.
• Ensure Helm registry login works for your OCI chart.
• Set base domain and registry URL for your environment.