JayeX - Cloud Native CI/CD Built On Kubernetes

Introduction

Fri, 13 Mar 2020 15:03:05 +0000

Understanding MLOps

When we talk about MLOps, what we are referring to is best defined as the extension of the DevOps methodology to include Machine Learning and Data Science assets as first class citizens within the best known methods of DevOps. The aim is to demystify Machine Learning and re-integrate the delivery of technology solutions as a seamless process of product development and deployment.

The Continuous Delivery Foundation hosts a collaborative Roadmap for MLOps and JayeX is committed to implementing features aligned to that model.

Using Machine Learning Quickstarts

Fri, 13 Mar 2020 15:03:05 +0000

The JayeX MLOps Quickstarts Library provides template projects to make it quick and easy to set up everything you need to get started with a building a new ML-based asset.

Each quickstart project comprises two repositories, one which contains your training script and a second which takes the final model you have trained and wraps it as a RESTful service for deployment into your overall solution.

Getting started

You can create an instance of a project using the command:

Finding a Machine Learning Quickstart

Fri, 13 Mar 2020 15:03:05 +0000

This directory is intended to help you find your way around the JayeX MLOps Quickstarts Library and get you up and running rapidly with a template project based around the class of Machine Learning approach you wish to work with and the language and framework you prefer.

The directory is divided by target programming language (Python only at this stage, but with additional quickstarts to follow in other languages) and then by ML framework.

CLI

Mon, 01 Jan 0001 00:00:00 +0000

CLI

To upgrade the jx CLI run:

jx upgrade cli

To upgrade jx subcommand plugins run:

jx upgrade plugins

Note

If you encounter this error:

ERROR: failed to load plugin <plugin-name>: can't find latest version of plugin: {"message":"Not Found","documentation_url":"https://docs.github.com/rest/reference/repos#get-the-latest-release"}

Then most probably it’s an old plugin which is not maintained, and can be removed from ~/.jx3/plugins/bin folder.

The jx CLI version used to upgrade to is derived from the JayeX version stream.

CLI

Mon, 01 Jan 0001 00:00:00 +0000

For those who like command lines you can view and watch most things via the jx command line.

You can download 3.x of jx from here: https://github.com/jenkins-x/jx/releases

Browse the command line commands along with the plugin commands

Many things in JayeX are exposed as custom resources so that you can also use kubectl to interact with the JayeX.

to view environments

kubectl get env

to view preview environments

kubectl get preview

to view build logs try:

jx pipeline log

Enhancement Process

Mon, 01 Jan 0001 00:00:00 +0000

Enhancements

The enhancements repository contains design proposals for Jenkins X enhancements.

All proposals are welcome, please follow this process:

Raise an issue on this repo describing at high level what the enhancement looks to achieve and whether you would like any help with the proposal.
If you would like some early collaboration from the Jenkins X community you might want to start with Google Docs for a faster feedback cycle.
Open a pull request containing markdown giving some context, describing the problem the enhancement aims to solve along with possible solutions.

Suggest changes to documentation

Mon, 01 Jan 0001 00:00:00 +0000

Did you spot a typo or feel something’s missing from a certain page? You can still contribute your suggestions without having to go through setting everything up locally as explained on Contribute to Documentation.

The process requires only three steps:

Click the “Edit this page” link on the right
Make your suggested changes
Create a Pull Request

We’ll go through each step here. Keep in mind though, that if you’re looking to contribute larger amounts of changes, pages, or sections, it’s best to test things out locally first (using the process described on Contribute to Documentation) instead of using this method.

What is JayeX?

Mon, 01 Jan 0001 00:00:00 +0000

JayeX automates and accelerates Continuous Integration and Continuous Delivery for developers on the cloud, so they can focus on building awesome software.

Embracing popular open source projects JayeX automates the setup and management to provide an integrated Cloud Native solution teams can use to develop better software faster and more reliably than traditional non cloud solutions.

Open Source projects that JayeX integrates with:

Kubernetes target platform JayeX is installed onto, optionally deploy and run applications built with JayeX
Tekton Cloud Native pipeline orchestration
Kuberhealthy Periodic health checks of the systems
Grafana [optional] Centralised logs and Observability
Jenkins [optional] traditional pipeline orchestration
Nexus [optional] artifact repository

At a high level JayeX can be split into a few areas:

Working with GPUs

Fri, 13 Mar 2020 15:03:05 +0000

To use CUDA to accelerate your ML training and services, you first need to set up your Kubernetes cluster to add some physical GPU resources to your nodes.

To do this, typically you will need to request an allocation of GPU resources from your Cloud provider and then configure an additional Node Pool to provision a set of Nodes such that each Node has access to at least one physical GPU card.

API Documentation

Mon, 01 Jan 0001 00:00:00 +0000

JayeX has two types of API documentation: Kubernetes Custom Resource Documentation and Godoc. Both types are generated from the codegen binary which is part of the jx repository.

Setup your development environment

It’s best to make changes to the JayeX code on your local machine. Follow the development guide to get set up.

Writing custom resource documentation

The custom resource documentation is mostly generated from the comments on the go structs that define the custom resources, with the introductory content and structure injected.

Managing Data

Fri, 13 Mar 2020 15:03:05 +0000

In most scenarios, you will be expecting to remotely access data hosted elsewhere on your network and can manage this in code as part of your training scripts and service implementations. There are however a couple of situations in which JayeX can help you to manage certain types of data.

Handling data in Buckets

If you are working with data in the form of arbitrary files, you can transfer these to your training environment via a Storage Bucket in your Cloud project.

Accelerate

Mon, 01 Jan 0001 00:00:00 +0000

JayeX is a reimagined CI/CD implementation for the Cloud which is heavily influenced by the State of DevOps reports and more recently the Accelerate book from Nicole Forsgren, Jez Humble and Gene Kim.

Years of gathering data from real world teams and organizations which has been analyzed by inspiring thought leaders and data scientists from the DevOps world. The Accelerate book recommends a number of capabilities that JayeX is implementing so users gain the scientifically proven benefits, out of the box. We’ve started documenting the capabilities that are available today and will continue as more become available.

Cluster

Mon, 01 Jan 0001 00:00:00 +0000

Cluster

Automatic upgrades

To enable automatic upgrades of your cluster you need to modify your jx-requirements.yml file in your development git cluster repository:

apiVersion: core.jenkins-x.io/v4beta1
kind: Requirements
spec:
  autoUpdate:
    enabled: true
    schedule: "0 0 * * *"
    autoMerge: true

Once you commit and push that change and your boot job has completed you should have a CronJob running at the above schedule (every night at midnight) creating a Pull Request to upgrade your cluster’s versions of charts and images.

Dashboard

Mon, 01 Jan 0001 00:00:00 +0000

It’s a common requirement to want to view pipelines running, their logs or to be able to click on a Pull Request on your git provider and view the pipeline log for that commit on that Pull Request.

To implement this we include the jx-pipelines-visualizer application inside JayeX.

This provides a simple read only web UI for viewing pipelines and pipeline logs and its linked automatically with any Pull Request you create on repositories managed by JayeX.

Demo

Mon, 01 Jan 0001 00:00:00 +0000

Demo

The following demo walks you through how to setup TLS and DNS:

Getting Started

Mon, 01 Jan 0001 00:00:00 +0000

Make sure you have got the jx 3.x binary and you have installed version 3 before proceeding.

Adding Jenkins Servers into JayeX

You can use JayeX to install one or more Jenkins servers by running the following command in a git clone of your dev cluster git repository:

jx gitops jenkins add --name myjenkins

That will add a new helmfile/myjenkins/helmfile.yaml file for the jenkins charts along with helmfile/myjenkins/values.yaml file that can be used to configure the jenkins helm chart configuration values.

Step by Step setup

Mon, 01 Jan 0001 00:00:00 +0000

We’ll go through each step below, but here’s what you need to get started with Git:

The git command line interface installed locally
a GitHub account
a local working copy of the code

Install Git on your system

Git is a version control system to track the changes of source code.

You will need to have Git installed on your computer to contribute to JayeX development. Teaching Git is outside the scope of the JayeX docs, but if you’re looking for an excellent reference to learn the basics of Git, we recommend the Git book if you are not sure where to begin.

Pull Request Reviews

Mon, 01 Jan 0001 00:00:00 +0000

The pull request is the main place we gate changes in JayeX. It’s here we:

run the tests (make test-slow-integration)
execute end to end tests against static jenkins and Tekton
check the change for linting issues
check the code for formatting issues
validate that all generated code (mocks, kubernetes clients, kubernetes openapi structs) are up to date
validate that we can generate the apidoc
validate that your commits are conventional

Our philosophy is to automate the checks as much as possible - there are some that must still be done by a human but we plan to reduce those as far as possible.

Version Everything

Mon, 01 Jan 0001 00:00:00 +0000

Always version everything explicitly in git and never use latest or SNAPSHOT images or artifacts!

The reason is you never know when a latest or SNAPSHOT image or artifact will upgrade in any container/pod/process/step in any environment. Its basically random. So not versioning things means random failures.

If you explicitly version things in all repositories things are stable and repeatable.

If you are worried about upgrading versions of things over time then look at either tools like dependabot or the updatebot plugin which we use extensively throughout JayeX to upgrade versions of libraries, images, binaries etc.

Documentation Style Guide

Mon, 01 Jan 0001 00:00:00 +0000

Documentation writer style guide

The JayeX Documentation Special Interest Group (SIG) has established some guidelines for docs contribution. The following are not meant to be exhaustive, but are simple style and technical tips to help conform to the existing voice and technical writing techniques used in our existing docs content.

Writer voice and tone

Pages should have titles, and should be descriptive enough on its own that readers (and web search crawlers) can pick up the title without too much missing context. Proper nouns should be capitalized in titles but nothing else (for example, write “JayeX tips and tricks” instead of “JayeX Tips And Tricks”)

Lighthouse Dashboard

Mon, 01 Jan 0001 00:00:00 +0000

Lighthouse is the JayeX component responsible for all interactions between your Git hosting provider (GitHub, Gitlab, …) and your Kubernetes cluster. Amongst other things, it:

receive and process webhook events
trigger pipelines
handle automatic merging of Pull Requests

Lighthouse comes with an optional Web UI: lighthouse-webui-plugin - see the admin guide to install it.

This UI gives you a read-only view of:

the webhook events received and processed by Lighthouse: git push, PR events, comments, …
the Lighthouse Jobs (pipelines) created, and their states - with a direct link to the pipeline visualizer to see the logs
the Lighthouse Merge Status and History: which PRs have been automatically merged by Lighthouse

This UI will help you understand what Lighthouse is doing, and in case of issue (pipeline not triggered, PR not being automatically merged, …) it will help you find the root cause.

Background

Mon, 01 Jan 0001 00:00:00 +0000

To achieve this we will use a couple of open source projects to help enable automated DNS for your applications.

For this guide we are going to assume you own a domain called foo.io which is managed by Google Cloud DNS, if it is not see configure cloud dns to manage a domain. A similar procedure is described for an external registrar, see the Azure section.

A common requirement for domains is to have production services accessed using a parent / apex domain for example:

Continuous Integration of JX itself

Mon, 01 Jan 0001 00:00:00 +0000

You may be wondering how JayeX introduce changes to JayeX. Of course, JayeX is built using JayeX itself! That means that new changes to the project go through a CI process, and are built and tested using pipelines that run on a JayeX Kubernetes cluster.

Pipelines

A Pull Request in the jx repository will automatically trigger some jobs to do CI. The jobs are triggered by Lighthouse, and we can configure which jobs to execute. The jobs with always_run configured to be true, will be run when the PR is opened. All jobs (independently of having always_run set to true or false) can be manually triggered writing a comment in the PR. The comment needed to trigger the job is also in the configuration, in the trigger key. For example, to trigger the pr tests manually, you may write a new comment in the PR containing “/test pr”, and the pr job will be triggered.

Infrastructure

Mon, 01 Jan 0001 00:00:00 +0000

Infrastructure

If you have used one of the JayeX Terraform Git repositories to create and manage your cloud resources then you have two options.

Terraform manual apply

If you are running the Terraform apply commands yourself then from your infrastructure Git repository run:

export TF_VAR_jx_bot_username=[your bot username]
export TF_VAR_jx_bot_token=[your bot token]
terraform get -update
terraform plan
terraform apply

Terraform Cloud

If you are using Terraform Cloud then from your infrastructure Git repository run:

Jenkinsfile support

Mon, 01 Jan 0001 00:00:00 +0000

When importing a project jx project import looks for a Jenkinsfile in the source code.

If there is no Jenkinsfile then the wizard assumes you wish to proceed with automated CI/CD pipelines based on tekton and imports it in the usual JayeX way. You also get to confirm the kind of pipeline catalog and language you wish to use for the automated CI/CD - so its easy to import any workload whether its a library, a binary, a container image, a helm chart or a fully blown microservice for automated kubernetes based CI/CD.

Pull Dependent Versions

Mon, 01 Jan 0001 00:00:00 +0000

When you Version Everything you need a way to pull dependent versions into your git repositories.

Using the pull model means that you periodically create Pull Requests on your git repositories to upgrade dependencies.

One solution is to use a tool tool like dependabot.

Or you can run periodic jobs via the updatebot plugin to create pull requests.

e.g. we automatically pull upgrades of chart versions into our Version Stream via this github action

References

Mon, 01 Jan 0001 00:00:00 +0000

The following sections contains other information that’s helpful when working with Hugo and the JayeX site; you don’t necessary need to go through this if this is your first time.

Search by Algolia/DocSearch

We’re using DocSearch by Algolia to power the internal search.

The script and local config are declared in /layouts/partials/scripts.html
Styling is included via /layouts/partials/head-css.html
The configuration is managed in the Algolia console.

Markdown Syntax Reference

Code examples

Hugo uses goldmark parser to handle markdown and markdown related configuration and chroma to handle syntax highlighting. Chroma supports the following languages. All pages on the JayeX docs use the typical triple-back-tick markdown syntax.

Triaging issues

Mon, 01 Jan 0001 00:00:00 +0000

The main issue tracker for the JayeX project is https://github.com/jenkins-x/jx/issues. This aims to capture issues, ideas and development work. If in doubt please raise an issue and a JayeX team member will look to triage it as soon as possible.

As JayeX is using prow from the Kubernetes ecosystem we figured we’d take their lead in triaging a large number of issues to aid and encourage contributions. We are reusing the style of labels including colours in an attempt to create familiarity across open source projects and reduce the barrier to contributing.

Vault

Mon, 01 Jan 0001 00:00:00 +0000

Installing Vault

Internal vault (vault created as part of JayeX installation)

If you are using Terraform with one of the Cloud Providers then your Vault will be installed automatically via Terraform.

Otherwise please see the On-Premises Vault Install Guide

whichever apporoach take you should have:

Kubernetes External Secrets is installed to populate Secrets from vault
the vault operator is installed for operating vault
a vault instance is created in the jx-vault namespace

You can wait for the vault-0 pod in namespace jx-vault to be ready via jx secret vault wait command:

Contribution workflow

Mon, 01 Jan 0001 00:00:00 +0000

Once you’ve completed the initial steps to get started, you can begin to make changes and add new content.

At a high level, your workflow will likely look something like this:

Create a new branch for you work
Start the Hugo server to preview your changes (updates the site live)
Make changes/add new content
Commit and push your changes to your fork of jx-docs
Raise a Pull Request (PR) to have your changes merged into the main jx-docs repo
Wait for and then participate in a review of your changes
- might involve making adjustments or adding a bit more
See your changes go live on the JayeX site

We’ll go though each of the steps below in more detail

Push Version Changes

Mon, 01 Jan 0001 00:00:00 +0000

When you Version Everything you need a way to pull dependent versions into your git repositories.

Using the push model means that as you create a new versioned release of an artifact (library, binary, image, chart or whatever) you generate the necessary Pull Requests in your downstream repositories to upgrade to use this version.

We do this in JayeX using the updatebot plugin as part of a release pipeline. The go-plugin release pipelines pipelines have a promote-release which uses the updatebot plugin to promote the new version to other git repositories. Here are some examples:

LTS

Mon, 01 Jan 0001 00:00:00 +0000

WARNING: The LTS is not updated, so we don't recommend using it

JayeX uses version streams as a quality gate when promoting plugins, charts, cli packages, container images etc. This results in a release of JayeX.

The default version stream for JayeX 3.x is https://github.com/jenkins-x/jx3-versions. The JayeX own infrastructure runs a number of end to end BDD tests for different base install options which covers the core of JayeX. Provided these tests pass on from a Pull Request it will be merged and users can upgrade their CLI and Cluster bringing in the versions that make that release.

From 3.0 Alpha

Mon, 01 Jan 0001 00:00:00 +0000

The 3.0 beta is almost ready, so we’ve rolled out the new jx-requirements.yml file format which uses API versioning and removes lots of old deprecated content.

Your cluster git repository will be upgraded automatically when you upgrade your cluster via the jx gitops upgrade command inside a git clone of your cluster git repository.

NOTE once you upgrade your cluster you will also need to make sure all the quickstarts and imported projects are using the latest pipelines from the catalog to ensure the container images are at least at 3.0 beta level.

Lens

Mon, 01 Jan 0001 00:00:00 +0000

Warning

Lens v6 or higher is not supported.

Lens is a desktop console for Kubernetes which runs locally on your laptop as a desktop application.

Install Lens

Download the Lens distribution for your computer then run it or follow the getting started guide

You can check out the catalog documentation for how to connect to a kubernetes cluster.

Install the JayeX extension

Lens supports extensions to support custom UIs for different extensions to Kubernetes such as JayeX.

Changes

Mon, 01 Jan 0001 00:00:00 +0000

You may also find the Roadmap and Maturity Matrix documents useful:

Breaking Changes

we have noticed a regression in the auto upgrade jobs which causes them to fail with an error like this:

updatebot WARNING: no $GIT_SECRET_MOUNT_PATH environment variable set                      
updatebot error: failed to setup git: failed to clone the cluster git URL: failed to clone cluster git repository https://github.com/myorg/myrepo.git: failed to clone cluster git repo https://github.com/
updatebot fatal: could not read Username for 'https://github.com': No such device or address

you can fix this by following the workarounds on this issue which should get you past this and working auto upgrade jobs again

Projects

Mon, 01 Jan 0001 00:00:00 +0000

JayeX stands on the shoulders of many open source giants…

Command Line Tools

helm a package manager for kubernetes
helmfile a tool for installing, upgrading and configuring helm charts
kaniko creates container images on kubernetes using the familiar dockerfile
kapp is a tool for applying changes to kubernetes resources safely with dependency tracking
kpt provides tools for sharing and configuring YAML files across git repositories
kubectl a command line tool for interacting with kubernetes clusters
kustomize a tool for configuring and modifying kubernetes resources
tekton cli the CLI tool for working with tekton pipelines
terraform a tool managing infrastructure as code

Microservices

CNCF build packs for packaging source code as container images. Or you can use kaniko which uses a familiar dockerfile to package container images
cert-manager for managing certificates for TLS and DNS
external-dns for managing certificates for TLS and DNS
knative a framework for building auto scaling (to zero) serverless style applications on kubernetes
kuberhealthy for health reporting of Kubernetes itself, JayeX and other microservices
kubernetes external secrets for managing secrets via Hashicorp Vault or the cloud native secret managers on Alibaba, Amazon, Azure, Google etc
jenkins the most popular build automation server which can be setup via GitOps with JayeX and a fellow CDF founding project
jenkinsfile runner a way of running a jenkins pipeline in a container
lighthouse our strategic solution for webhooks and ChatOps for multiple git providers
push-wave for automatically performing rolling upgrades when secrets are rotated in your secret store
tekton pipelines cloud native pipeline engine for kubernetes and a fellow CDF founding project
tekton catalog a catalog of reusable tekton pipelines which can be used easily with Jenkins
vault a secret store
vault operator an operator for installing and unsealing vault

Event Calendar

Mon, 01 Jan 0001 00:00:00 +0000

Add Events or Make Changes

Please create a documentation issue using the link on the right hand menu, or click here.

You should then be at a page for creating an issue on the jayex-io/jx-docs repo, with the issue title as “Event Calendar”. Please describe your event, with meeting links, and the time in UTC. Thank you!

Maturity Level Matrix

Mon, 01 Jan 0001 00:00:00 +0000

This maturity matrix describes the tentative plan post JayeX v3.0 GA release. This matrix will continue to evolve as we progress further.

Version Stream

Mon, 01 Jan 0001 00:00:00 +0000

JayeX is made up of a large number of command line packages, images and helm charts, some of which are released by the JayeX community and others come from the wider open source ecosystem.

To improve the stability of JayeX when lots of packages and charts are changing all the time we have introduced the JayeX Version Stream

How it works

The version stream is stored in the jenkins-x/jx3-versions git repository and stores the stable version of all packages and charts used by JayeX.

OpenShift CodeReady Containers

Mon, 01 Jan 0001 00:00:00 +0000

This guide will walk you though how to setup JayeX on your laptop using OpenShift 4.x with CodeReady Containers

Prerequisites

download OpenShift 4.x with CodeReady Containers
once you have the crc binary setup the amount of memory and disk:

crc config set cpus 6
crc config set memory 11264

crc start

once your cluster boots up you can setup your environment…

eval $(crc oc-env)

You can copy/paste the oc login -u kubeadmin login command….

Benefits

Mon, 01 Jan 0001 00:00:00 +0000

We can use vanilla tools like helm 3, helmfile, kustomize, kpt to install, update or delete charts in any namespace without needing helm 2.x or tiller or custom code to manage helm template.
- We can avoid all the complexities of the jx step helm apply logic we used in JayeX 2.x
- Instead we can replace this with vanilla helmfile to allow optional templating of values.yaml files when using helm
The new Getting Started approach is much simpler, easier to configure and customise and is cleanly integrated with tools like Terraform and works well with different cloud infrastructure platforms.
- The default install/upgrade pipelines check in all the generated kubernetes resources and custom resources as YAML so its easy to understand
- You can read more about the git layout here
  - The config-root/cluster folder contains all the global cluster level resources like ClusterRole, Namespace or Custom Resources
  - The config-root/namespaces/jx folder contains all the namespaced resources in the jx namespace
- This makes it easy to use flexible apply logic in different boot Jobs with different RBAC (or a system admin could apply the cluster level resources for you by hand) - to make it easier to install JayeX on more locked down and restricted clusters
We use Kubernetes External Secrets to provide a single way to manage secrets which supports the following back end systems:
- Alibaba Cloud KMS Secret Manager
- AWS Secrets Manager
- Azure Key Vault
- GCP Secret Manager
- Hashicorp Vault
It opens the door to a flexible multi-cluster support so that every cluster can be managed in the same canonical GitOps approach from a single git repository
The new getting started approach runs the boot pipeline as a Job inside the Kubernetes cluster. This ensures consistency in tooling used and also improves security by avoiding having the secrets on a developers laptop.
- The only thing you run on your local machine when installing JayeX is installing the git operator which is a simple helm chart.
Everything is now an app. So if you want to remove our nginx-ingress chart and replace it with another ingress solution (knative / istio / gloo / ambassador / linkerd or whatever) just go ahead and use the apps commands to add/remove apps and have boot manage everything in a consistent way
- e.g. here’s an example of removing chartmusem and nexus and replacing it with bucketrepo via a single simple yaml change.
You can install an app in a specific namespace if you wish
- This also opens the door to using boot to setup multi-team installations where multiple teams use different namespaces but share services in the same cluster
The cluster GitOps repository is simpler and easier to keep in sync/rebase/merge with the upstream git repositories.
- We use kpt to do that for us
- We now include the version stream inside your GitOps repository too inside the versionStream directory after installation so that all the information about your installation is inside a single git repository so its simpler to test changes & ensure consistency.
We can avoid composite charts to simplfiy configuration and upgrades
We no longer use exposecontroller, instead use regular helm configuration to create Ingress resources and override domain names
secret handling is currently much simpler using Kubernetes External Secrets for any secrets in any namespace or cluster for your own apps or for those used by JayeX.

Git

Mon, 01 Jan 0001 00:00:00 +0000

JayeX supports a number of different Git providers. You can specify the Git provider you wish to use and the organisation to use for the Git providers for each environment in your jx-requirements.yml

GitHub

This is the default Git provider if you don’t specify one.

cluster:
  environmentGitOwner: myorg
  provider: gke
environments:
- key: dev
- key: staging
- key: production

GitHub Enterprise

The configuration is similar to the above but you need to specify the URL of the gitServer (if it differs from https://github.com) and gitKind: github

JayeX Security

Mon, 01 Jan 0001 00:00:00 +0000

The JayeX project takes security seriously. We make every possible effort to ensure users can adequately secure their automation infrastructure. To that end, we work with JayeX platform and app developers, as well as security researchers, to fix security vulnerabilities in JayeX in a timely manner, and to improve the security of JayeX in general.

Supported Versions

Version	Supported
3.x	✅

Reporting a Vulnerability

If you find a vulnerability in JayeX, please report it in https://github.com/jenkins-x/jx/security/advisories. Please do not report security issues in the normal github issue tracker.

Azure

Mon, 01 Jan 0001 00:00:00 +0000

DNS zone creation

Create a common resource group dedicated to all your DNS zones:

$ az group create --name rg-dns --location westeurope
{
  "id": "/subscriptions/49721339-fe83-4562-afec-783c3f00c06f/resourceGroups/rg-dns",
  "location": "westeurope",
  "managedBy": null,
  "name": "rg-dns",
  "properties": {
    "provisioningState": "Succeeded"
  },
  "tags": null,
  "type": "Microsoft.Resources/resourceGroups"
}

Create an Azure DNS zone with the name of your domain:

$ az network dns zone create -g rg-dns -n foo.io
{
  "etag": "00000002-0000-0000-e2d9-80ef0df0d601",
  "id": "/subscriptions/49721339-fe83-4562-afec-783c3f00c06f/resourceGroups/rg-dns/providers/Microsoft.Network/dnszones/foo.io",
  "location": "global",
  "maxNumberOfRecordSets": 10000,
  "name": "foo.io",
  "nameServers": [
    "ns1-05.azure-dns.com.",
    "ns2-05.azure-dns.net.",
    "ns3-05.azure-dns.org.",
    "ns4-05.azure-dns.info."
  ],
  "numberOfRecordSets": 2,
  "registrationVirtualNetworks": null,
  "resolutionVirtualNetworks": null,
  "resourceGroup": "rg-dns",
  "tags": {},
  "type": "Microsoft.Network/dnszones",
  "zoneType": "Public"
}

Domain DNS servers configuration

In your registrar admin panel, find the DNS servers section of the domain you want to use and replace the default ones by those from the first step:

Comparison

Mon, 01 Jan 0001 00:00:00 +0000

This document outlines the similarities and differences of the 3.x approach for those who are aware of jx boot with helm 2 in 2.x of JayeX.

Similarities between 2.x and 3.x

Just like classic boot with the jenkins-x-boot-config git repository, this new helmfile solution supports:

you can install and upgrade JayeX via GitOps
you can reuse helm charts from the internet, local charts or charts built by JayeX in any environment
a YAML file is used to store all the charts that are applied during install/upgrade

Differences with in 3.x

we support any permutation of tools such as: helm 3, helmfile, kustomize and/or kpt to create the kubernetes resources
in 3.x the installation/upgrade of JayeX is run inside the kubernetes cluster via a Job rather than on a developers laptop which helps with consistency and security.
in 3.x we use a single git repository for each cluster; which can manage as many teams/namespaces as you like within the cluster
- so any local environments like Staging which reside in the same kubernetes cluster are defined in the same git repository in 3.x - whereas in 2.x we used a separate git repository for Dev, Staging and Production when sharing the same cluster.
- if Dev, Preprod and Production environments are in separate kubernetes clusters then those will have a git repository each.
any helm chart can be deployed in any namespace (previously we used a single namespace for all charts in the env/requirements.yaml)
instead of using env/requirements.yaml we now use a simple and more powerful helmfile.yaml file which is similar but supports:
- we can specify a namespace on any chart
- we can add extra values files to use with the chart to override the helm values.yaml files
instead of copying lots of env/$appName/values*.yaml files into the boot config like we do in these folders such as the lighthouse/values.tmpl.yaml we can instead default all of these from the version stream at apps/jenkins-x/lighthouse - which means the boot config git repository is much simpler, we can share more configuration with the version stream and it avoids lots of git merge/rebase issues.
adding and removing apps in your GitOps repository causes those resources to be properly installed or uninstalled
- you can also review exactly what kubernetes resources will change on the Pull Request
we no longer use a composite chart for env/Chart.yaml and instead deploy each chart independently
- this means that each chart has its own unique version number you can see in the helmfile.yaml file
we have done away with the complexity of jenkins-x-platform (a composite chart containing logs of dependencies like jenkins + chartmuseum + nexus etc) so that each chart can be added/removed independently or swapped out with a different version/distribution

Removing complexity and magic

Removing complexity out of JayeX and reusing other solutions wherever possible. JayeX 2.x was tightly coupled to helm 2 for example. There were jx CLI steps that wrapped helm commands when installing applications into the cluster which injected secrets from an internal Vault and ultimately made it very confusing for users and maintainers.

Cluster Recovery Demo

Mon, 01 Jan 0001 00:00:00 +0000

This section describes cluster recovery situations for when things go bad. It can also be used to reguary recreate clusters, this is something the JayeX project does itself as we prefer to treat our clusters as cattle and not pets, giving confidence that we can restore services at any time.

Disclaimer: there may well be better approaches so if you know of better ways please contribute and help improve the experience. There are some manual steps below that we know to work but expect we can improve.

Configure Azure Service Principal

Mon, 01 Jan 0001 00:00:00 +0000

Azure has a notion of a Service Principal which is a service acount. This doc will demonstrate how to set up an Azure service principal that can be used by Terraform to execute JayeX Azure Module

💡 This doc has been designed to assist in performing the demonstration through copying and pasting each block of code into a shell terminal.

To execute the commands listed in your local bash shell will require the Azure CLI and the JQ command-line JSON processor.

Configure Google Cloud DNS to manage my domain

Mon, 01 Jan 0001 00:00:00 +0000

This guide will describe how to purchase a domain and configure GCP to manage it with Cloud DNS

Setup below is a cut down version of the original docs located here

Go to Google Domains and purchase a domain.

Create a DNS zone which will contain the managed DNS records.

gcloud dns managed-zones create "foo-io" \
    --dns-name "foo.io." \
    --description "Automatically managed zone by kubernetes.io/external-dns" \
    --project foo

Make a note of the nameservers that were assigned to your new zone.

Default domain

Mon, 01 Jan 0001 00:00:00 +0000

If you use the default configuration your ingress.domain will be empty. When you first install JayeX it will discover the LoadBalancer Service from nginx in the nginx namespace and resolve that to an external IP address. Then it will use that IP address as a domain with .nip.io as the suffix.

So you will see your jx-requirements.yml file looking something like:

ingress:
  domain: 1.2.3.4.nip.io
  externalDNS: false
  tls:
    email: ""
    enabled: false
    production: false

where 1.2.3.4 is your external IP address of your nginx LoadBalancer service.

Delete only jx

Mon, 01 Jan 0001 00:00:00 +0000

Use this if you want to only delete JayeX from your cluster.

kubectl delete -R -f config-root/namespaces
kubectl delete -R -f config-root/cluster

If you imported repositories into JayeX using jx project import or jx project quickstart, then you also need to delete the webhooks manually from the repositories.

If you installed jx on minikube/kind/k3s (local installations), refer to the docs of those platforms on how to uninstall the cluster.

Enable WebHooks

Mon, 01 Jan 0001 00:00:00 +0000

If your cluster is not accessible on the internet and you can’t open a firewall to allow services like GitHub to access your ingress then you will need to enable webhooks as follows:

install and setup ngrok
setup a webhook tunnel to your laptop find your hook host name:

kubectl get ing -n jx

copy the hook host name into…

ngrok http http://yourHookHost

copy the following YAML to a file: helmfiles/jx/jxboot-helmfile-resources-values.yaml

ingress:
  customHosts:
    hook: "abcdef1234.ngrok.io"

modify the hook: line in your helmfiles/jx/jxboot-helmfile-resources-values.yaml file to use your personal ngrok domain name of the form abcdef1234.ngrok.io

From 2.x

Mon, 01 Jan 0001 00:00:00 +0000

If you wish to migrate from older versions of JayeX to v3 we recommend spinning up a new cluster for v3 and then incrementally moving projects from the old cluster to the new cluster.

This might sound heavy handed but the issue with working with multiple versions of a number of charts like lighthouse and tekton is that they use cluster scoped resources (CRDs and clusterrole resources for example) which can compete with each other. e.g. installing a new version of a chart may well break an old installation.

Configure Google Service Account

Mon, 01 Jan 0001 00:00:00 +0000

This doc will demonstrate how to set up a Google service account that can be used by Terraform to execute JayeX GKE Module

💡 This doc has been designed to assist in performing the demonstration through copying and pasting each block of code into a shell terminal.

To execute the commands listed in your local bash shell will require the Google gcloud tool and Cloud SDK and the JQ command-line JSON processor.

To execute the commands listed in Google Cloud Shell does not require any additiobnal software installed.

General

Mon, 01 Jan 0001 00:00:00 +0000

What is the directory layout?

To understand the directory layout see this document and the file reference

How do I add an Environment

With v3 everything is done via GitOps - so if in doubt the answer is to modify git.

You can create new environments by adding to the environments: section of jx-requirements.yml

How do I change promotion for my app?

Usually when you import a repository or create a quickstart they inherit the default environments for promotion. It is common to share the same environments across all of your microservices.

General

Mon, 01 Jan 0001 00:00:00 +0000

Where do I raise issues?

One of the challenges with JayeX 3.x is the source code is spread across a number of organisations and repositories since its highly decoupled into many plugins and microservices so it can be confusing

If you know the specific plugin causing an issue, say jx-preview then just raise the issue there in the issue tracker.

Otherwise use the issue tracker for JayeX 3.x and we can triage as required.

Git

Mon, 01 Jan 0001 00:00:00 +0000

Each kubernetes cluster has a git repository so that all the kubernetes resources in all namespaces can be managed by GitOps. Each cluster may also have a separate infrastructure git repository (e.g. for Terraform) to define the cloud resources (buckets, IAM roles, kubernetes cluster, node pools, VPNs, firewalls etc).

So if you use a Multi Cluster Setup you have git repository per cluster to define the kubernetes resources in all the namespaces in that cluster.

GitOps Recommendations

Mon, 01 Jan 0001 00:00:00 +0000

One of the awesome things about Jenkins is you can use it to do anything in any way you like. e.g. you can add/configure any Jenkins server via the UI which then modifies the state on disk. This can make things harder to manage at scale; tracking who changes what and to diagnose issues and perform backup and restore etc.

We have been using the GitOps approach to managing things in production for the last few years on the JayeX project and it has lots of benefits:

GitOps

Mon, 01 Jan 0001 00:00:00 +0000

The idea of GitOps is to treat everything as source code, including your infrastructure, and check it into git and version it.

So rather than connecting to a machine in your cluster, you make changes via Pull Requests in git. This means:

every change is audited so you can see who changed what, when and why
its easy to revert changes to your infrastructure if things go bad
teams can easily review changes to work more effectively as a team, share knowledge and get better feedback

Challenges

Secrets

One of the biggest challenges with GitOps is dealing with secrets.

GitOps

Mon, 01 Jan 0001 00:00:00 +0000

The GitOps pattern means you use source code and git repositories for both your applications, your infrastructure and environments too.

In the DevOps space we have been using git to version infrastructure configuration with many tools like terraform and before that tools like Ansible.

With kubernetes you can connect to a cluster and modify it via kubectl apply or helm install`.

However with the GitOps pattern developers don’t modify kubernetes directly; instead they propose changes to a git repository via a Pull Request which when it gets approved and merged causes the kubernetes cluster to be modified via some kind of operator such as the JayeX git operator

Google

Mon, 01 Jan 0001 00:00:00 +0000

Prerequisites

cluster created using JayeX GCP Terraform getting started
own a domain and have GCP manage it, configure cloud dns to manage a domain
latest JayeX CLI, Infrastructure and Cluster git repository updates upgrade

Cloud Infrastructure

First we will configure the cloud infrastructure requirements:

a GCP Service Account with the dns.admin role, see here for more information
a managed cloud dns zone, see here for more information

To satisfy these requirements go to your infrastructure repository (contains Terraform main.tf) and add to your values.auto.tfvars the following:

Install

Mon, 01 Jan 0001 00:00:00 +0000

Check you are using a cluster Git URL

If using Terraform make sure the values.auto.tfvars file contains a jx_git_url = value that points to a cluster git repo that contains a helmfile.yaml and a folder ./helmfiles. A common mistake is users set the jx_git_url to the infrastructure repo instead.

After a successful bootjob, these are few things to look for:

Secrets have been generated in the secret backend.
A webhook has been created in the cluster git repo.

Issues with secret generation

In some cases, the webhook generation will fail with a message :

jx gitops webhook update --warn-on-fail
Error: failed to find hmac token from secret:

This is normally because the secret generation failed during the boot job.

JayeX Office Hours - Archives

Mon, 01 Jan 0001 00:00:00 +0000

Multi-Cluster

Mon, 01 Jan 0001 00:00:00 +0000

We recommend using separate clusters for your Preprod and Production environments. This lets you completely isolate your environments which improves security.

Setting up multi cluster

For remote environments (e.g. Preprod and Production) you typically won’t need lots of the development tools such as:

Lighthouse
Tekton
Webhooks
Nexus / Bucketrepo

And install only services to run and expose your applications, e.g.:

Nginx-ingress
Cert-manager
kubernetes external secrets for populating Secrets from your secret store (vault or cloud provider secret manager)
push-wave for automatically performing rolling upgrades when secrets are rotated in your secret store

Create a repository

Create a git repository for your remote cluster using a cut down repository you might want to start with:

okteto

Mon, 01 Jan 0001 00:00:00 +0000

If you install okteto it has a command you can use to spin up a development container inside a deployment so you can iteratively rebuild an application.

Then if your application is deployed to your staging environment you can cd into a git clone of your application and run…

# switch to the staging namespace
jx ns jx-staging

okteto up

You will now have a development container setup you can perform incremental builds

Pipelines

Mon, 01 Jan 0001 00:00:00 +0000

The following are the links to the various configuration file formats:

Tekton resources:

A Task is made up of Steps which each support all of the properties you can modify on a kubernetes Container

Lighthouse

Lighthouse TriggerConfig:

presubmits for triggering pipelines on Pull Request
postsubmits for triggering pipelines on a push to a branch (e.g. releasing)

Also check out the lighthouse pipeline configuration docs

JayeX Roadmap

Mon, 01 Jan 0001 00:00:00 +0000

The JayeX roadmap can be found on GitHub.

Please join our TOC meetings and Slack channels to ask questions and drive input on the future of the project! The best way to get a change into JayeX is to write a pull request with the change to the appropriete. If a change requires changes to multiple repositories you should start with creating a PR in the JayeX enhancements repo documenting the propsed change.

Source Repositories

Mon, 01 Jan 0001 00:00:00 +0000

Source Control Management (SCM) Repositories which have been configured to run CI/CD pipelines using JayeX are represented as Source Repositories. Source Repositories are a kubernetes custom resource (CR) scoped to a namespace and created in the development namespace/environment.

A source repository includes information about the SCM provider, Organization, Repository name and clone URLs.

A simple example of a source repository is shown below

apiVersion: jenkins.io/v1
kind: SourceRepository
metadata:
  labels:
    gitops.jenkins-x.io/pipeline: namespaces
    owner: jenkins-x
    provider: github
    repository: jx
  name: jenkins-x-jx
  namespace: jx
spec:
  httpCloneURL: https://github.com/jenkins-x/jx.git
  org: jenkins-x
  provider: https://github.com
  providerKind: github
  providerName: github
  repo: jx
  url: https://github.com/jenkins-x/jx

This shows a source repository named jenkins-x-jx in the namespace jx. It represents a repository named jx in the jenkins-x organization which is hosted in github.

Vault

Mon, 01 Jan 0001 00:00:00 +0000

NOTE that in the following instructions it is left to the user to manage, backup and restore the vault installation once it has been installed.

For production workloads we recommend you use a cloud provider secret store or Vault as a service. Managing on-premises vault instances is undifferentiated heavy lifting that should be outsourced to a cloud provider if you can.

Prerequisites

The prerequisites are the same as regular on-premises kubernetes around having a kubernetes cluster with ingress and storage

Multi-Cluster Example

Mon, 01 Jan 0001 00:00:00 +0000

This is an example on how to build a multi-cluster environment having two separate cluster repos (i.e. ‘dev’ and ‘prod’). The steps will include building a GKE/GSM/DSN environment from scratch for both environments, and deploying projects to staging and the remote production. It will use DNS (jx3rocks.com), TLS, Let’s Encrypt certificates. This example is intended for an audience already familiar with JayeX operability and focuses on an example of actual commands used to build a multi-cluster environment. Additional information regarding using Google as the provider for this example can be found under Google Cloud Platform Prequisites.

Pipelines As Code

Mon, 01 Jan 0001 00:00:00 +0000

Define your Continuous Integration and Continuous Delivery Pipelines as code and keep them in git with your application code.

In JayeX you can see the Source Layout for more detail using standard Tekton Pipelines YAML for defining pipelines in a cloud native self-contained way.

One downside with Pipelines As Code is that you can end up copy/pasting lots of YAML into lots of repositories which can become tricky to both maintain and to keep on recent images and configurations.

Trunk Based Development

Mon, 01 Jan 0001 00:00:00 +0000

This approach is essentially:

use a single branch for releases
work in small batches
use Continuous Integration to frequently merge changes to the main branch

It is described in the Accelerate book.

The idea is to avoid long term feature branches which have a cost in terms of maintenance and being the opposite of Continuous Integration.

For more background see the Trunk Based Development site which has lots of examples and approaches.

Platform Observability

Mon, 01 Jan 0001 00:00:00 +0000

This guide will help you install, configure and use an observability stack in your JayeX cluster.

We selected the Grafana stack because:

it’s open-source
it has support for logs, metrics, and distributed traces
it has a low memory footprint, and a great Kubernetes integration

Installation

Please follow the usual getting started guide for boot and helm 3 first.

The first step is to edit your main helmfile.yaml file located in the root directory of your development environment git repository, so that it references the helmfiles/jx-observability/helmfile.yaml file, such as:

Automate Continuous Delivery

Mon, 01 Jan 0001 00:00:00 +0000

As described in the Accelerate book, high performing teams automate their Continuous Delivery.

This lets your team focus on delivering business value for your customers rather than writing pipelines by hand.

If you use JayeX for your environments and to create projects you will have completely automated Continuous Delivery using Pipelines As Code, Preview Environments and Trunk Based Development

Continuous Delivery Indicators

Mon, 01 Jan 0001 00:00:00 +0000

This guide will help you install, configure and use the Continuous Delivery Indicators addon in your JayeX cluster.

This addon will automatically:

collect metrics from cluster events and git events: pull requests, pipelines, releases, deployments, …
store them in a PostgreSQL database
expose Grafana dashboards to visualize Continuous Delivery Indicators

This is an example of a pre-defined dashboard to visualize Continuous Delivery Indicators for a single repository (application):

How it works

Mon, 01 Jan 0001 00:00:00 +0000

How it works

The GitOps repository templates contain the source code, scripts and docs to help you get your cloud resources created (e.g. a kubernetes cluster and maybe buckets and/or a secret manager).

Once you have created the GitOps repository from one of the available templates and followed the instructions to set up your infrastructure you install the git operator via the jx admin operator command:

    jx admin operator

That command essentially installs the git operator chart, passing in the git URL, username and token to run the boot process.

Charts and resources

Mon, 01 Jan 0001 00:00:00 +0000

How do I add a chart?

To add a new chart add to the helmfiles/mynamespace/helmfile.yaml file follow the add chart guide.

How do I customise an App in an Environment?

We use helmfile to configure helm charts whether its in the dev, staging or production environment and whether you are using multiple clusters.

See how to customise a chart

For a given namespace called ns there is a folder in the cluster git repository at:

Demo

Mon, 01 Jan 0001 00:00:00 +0000

Demo

The following demo walks through how to trigger tekton pipelines and create custom pipelines:

Using

Mon, 01 Jan 0001 00:00:00 +0000

How do I list the apps that have been deployed?

There is a handy HTML report in your cluster dev git repository at docs/README.md which lists all the charts deployed in every namespace with their version.

You can see the helm charts that are installed along with their version, namespaces and any configuration values by looking at the releases section of your helmfile.yaml and helmfile/*/helmfile.yaml files in your cluster git repository.

Registries

Mon, 01 Jan 0001 00:00:00 +0000

How do I use a custom container registry?

What container registry is used by default depends on the which cloud provider you use. There are some things to consider when migrating from container registry to artifact registry in Google GCP.

To allow a pipeline to be able to push to a container registry you can add this secret…

kubectl create secret generic container-registry-auth  \
  --from-literal=url=myserver.com \
  --from-literal=username=myuser \
  --from-literal=password=mypwd

This will then take effect the next time a commit merges on your cluster git repository e.g. next time you upgrade your cluster.

Repositories

Mon, 01 Jan 0001 00:00:00 +0000

How do I customise a Scheduler?

The .jx/gitops/source-config.yaml file in your dev cluster git repository lets you configure the name of the scheduler to use for all repositories, for a group of repositories or for an individual repository.

The schedular name is then resolved to be either a file versionStream/schedulers/$name.yaml or schedulers/$name.yaml.

So if you want to create your own Scheduler you could copy the default in-repo based scheduler versionStream/schedulers/in-repo.yaml to schedulers/myname.yaml and then modify it to suit - then associate myname with whatever repositories you wish to use this scheduler for.

Bot Token

Mon, 01 Jan 0001 00:00:00 +0000

Commons problems with Bot Tokens

Make sure the bot user has access to the organisation and / or git repositories that it needs.

tip you can accept an organisation and repo GitHub invitation using

jx admin invitation

Make sure the bot token has the correct permissions to perform the action you require

Cloud Native

Mon, 01 Jan 0001 00:00:00 +0000

One of the Accelerate recommendations is around using the cloud well; letting developers use the cloud to solve software problems.

Here are a few of the lessons we have learnt about using the cloud well.

Prefer cloud over kubernetes

You can deploy a database via a helm chart in your kubernetes cluster. Or you can configure your cloud provider to create a managed database offering.

You can deploy, say, vault as helm charts inside your kubernetes cluster. Or you can use your cloud providers secret store solution such as:

Configuration

Mon, 01 Jan 0001 00:00:00 +0000

In your development cluster git repository the jx-requirements.yml file is used to define which are the default environments are used for promotion on your repositories.

The default configuration for environments looks something like this:

apiVersion: core.jenkins-x.io/v4beta1
kind: Requirements
spec:
  ...
  environments:
  - key: dev
  - key: staging
  - key: production

which defaults to meaning that Staging and Production are namespaces (jx-staging and jx-production) in the local cluster. Staging will use Auto promotion and Production will use Manual (more on that later).

Delete jx and the cluster

Mon, 01 Jan 0001 00:00:00 +0000

Use this if you created the kubernetes cluster using the AWS/GCP/Azure terraform module. This will also work, if you installed JayeX on an existing AWS EKS cluster.

Always examine the plan of the terraform destroy command before approving it, once destroyed there is no going back.

In case of AWS, manually delete the network load balancer (NLB) created by the nginx helm chart from your aws account.

Editing

Mon, 01 Jan 0001 00:00:00 +0000

Source layout

If you upgrade your cluster to the latest version stream then you will find if you create a new quickstart that:

.lighthouse/jenkins-x directory contains the default CI/CD pipelines for JayeX with these files:
- triggers.yaml to define the lighthouse TriggerConfig which defines the ChatOps and triggering configuration via a spec field which defines presubmits and postsubmits (i.e. Pull Request and Release triggers).
- pullrequest.yaml defines the Pull Request pipeline using a Tekton Task, Pipeline or PipelineRun
- release.yaml defines the Release pipeline using a Tekton Tekton Task, Pipeline or PipelineRun
jenkins-x.yml files are no longer used by default in new quickstarts instead we use the above. Note if you have projects using jenkins-x.yml files they are still supported if you import them into v3 or you can use this tool to migrate them to tekton pipelines

Environments

Mon, 01 Jan 0001 00:00:00 +0000

Environment is a kubernetes custom resource (CR) in JayeX where the application code lives. Examples of environment include development, testing, staging and production. Environments are scoped to kubernetes namespaces and essentially extend a namespace with additional JayeX related metadata.

After installing JayeX, a development environment is created. This is the environment where all JayeX related resources like lighthouse, build controller, nexus, chart museum are installed. Dev environment by default is in the jx namespace, but it is configurable. This is also the environment where all your pipelines will run.

Files

Mon, 01 Jan 0001 00:00:00 +0000

JayeX uses GitOps and so has a number of different source files with declarative schemas.

You may also find the git layout document useful.

Files in any repository

File	Schema	Description
`.jx/settings.yaml`	Settings	Optional file to override any settings from the development cluster git repository; such as the chart repository, container registry or environments to promote to. Usually the Requirements in the dev cluster git repository are used.
`.jx/updatebot.yaml`	Updatebot	Describes the settings when using the jx updatebot plugin to promote versions of a repository to different git repositories using different strategies
`.lighthouse/*/triggers.yaml`	TriggerConfig	Defines the pipeline triggers in lighthouse which start tekton pipelines in response to git or ChatOps activity
`.lighthouse//.yaml`	PipelineRun	Tekton pipelines to define pull request pipelines (or `presubmits`) and release pipelines (`postsubmits`). For more detail check out the pipeline reference
`charts`	helm chart	Defines the kubernetes resources to be installed in a kubernetes cluster
`Dockerfile`	Dockerfile	Defines how to create a container image for repositories that create images
`OWNERS`	OWNERS	this lighthouse configuration file defines who can review and approve files
`OWNERS_ALIASES`	OWNERS_ALIASES	this lighthouse configuration file defines aliases for who can review and approve files
`preview/helmfile.yaml`	helmfile	The helmfile defines the helm charts, version and value mappings to be installed in a preview environment

Files in a dev cluster repository

The following files are only used in a development cluster git repository

Golang upgrades

Mon, 01 Jan 0001 00:00:00 +0000

JayeX codebase uses Golang/Go. New versions of golang comes every 6 months, and golang supports only 2 versions. It is highly desireable to use the latest and greatest version of golang for JayeX as newer versions have performance improvements and get regular security updates.

To update golang version in JayeX follow these steps

Open an issue in jx repository announcing the upgrade.
Upgrade the goreleaser image - https://github.com/jenkins-x/jx-goreleaser-image
Change the go version in the pipeline catalog repository. See this PR for which files to change.
Update the catalog version in the version stream See this PR for which files to change.
Update the library packages first (please follow this order)
Once these packages are upgraded, start the upgrade for the plugins and main jx repository
Follow these steps to upgrade the version
- Change the version in go.mod file
- Upgrade any direct dependencies to other JayeX packages that you see in go.mod: If you for example find the line github.com/jenkins-x/jx-logging/v3 v3.0.7 you run go get -u github.com/jenkins-x/jx-logging/v3
- Run go mod tidy and ensure that it does not error out
- Change GO_VERSION in makefile to the version you are upgrading to.
- Check if there are any references to an older version and replace those.
- Verify that
  - the build works by running make build
  - tests are working, by running make test
  - linting checks are working by running golangci-lint run
    - Use the same version that is used by JayeX. At the time of writing that is 1.58; you can see what image version is referenced in the pipeline.
    - If there is a configuration file for golangci-lint in the repository (.golangci.yaml) it likely needs to be updated. See for example https://github.com/jenkins-x/jx-kube-client/blob/main/.golangci.yml for reference.

GSoC

Mon, 01 Jan 0001 00:00:00 +0000

JayeX has participated in the Google Summer of code (GSoC) before.

This is the process that JayeX will follow when applying for GSoC (timelines are subject to change, check the timeline section in the gsoc webpage).

Publish a blog post announcing JayeX interest in applying to GSoC when the application is announced by Google
- this normally happens in November
Set a deadline when all project ideas are submitted for internal review by JayeX TOC
- It’s a good idea to set the deadline internally for project list submission to February 1. This is normally when the application opens for organizations. The application remain open only for 2 weeks, so it’s better to have a list beforehand.
Decide internally on a list of projects and the mentors that are interested.
- Projects with multiple mentors are more likely to be shortlisted
Once the application is announced by Google, publish a blog post with the project ideas. A sample is available here
Submit the application
If selected by GSoC (results are announced in March), publish a follow up blog post.
- This blog post is expected to help the GSoC applicants understand more about JayeX and what we are looking for in an ideal contributor. Check the follow up blog from 2022 to get an idea.
- It helps the contributors and the mentors if a project template is provided.
After the project proposal deadline (April), decide on a list of contributors and rank the project proposals (the one at the top is the strongest proposal)
Once Google announces the list of accepted proposals for JayeX (not all proposals ranked will get selected), mentors should discuss the project details (and time commitments) with the contributors.

Jenkins Server Example

Mon, 01 Jan 0001 00:00:00 +0000

This is an example on building a JX3 Google/GSM environment from scratch which includes a Jenkins server and the import of a pipeline. It will utilize DNS (jx3rocks.com), TLS, Let’s Encrypt certificates, and additional Jenkins plugins and installer resources. It is inteded for an audience already familiar with JayeX operability and focuses on an example of actual commands required to build and operate the environment. Additional information regarding using Google as the provider for this example can be found under Google Cloud Platform Prequisites.

ksync

Mon, 01 Jan 0001 00:00:00 +0000

You can use ksync to synchronise your local source code with a running pod which you can then use to perform incremental builds inside kubernetes.

kubectl run -ti --image maven maven-build-pod bash
ksync create -n jx --pod maven-build-pod $(pwd) /
ksync watch

# then in your build pod run `mvn clean install` or `mvn spring-boot:run` or whatever

Pipelines

Mon, 01 Jan 0001 00:00:00 +0000

How do I diagnose a step in a pipeline?

If you are wondering what image, command, environment variables are being used in a step in the pipeline you can look at the pipelinerun or the related pod(s). If you use your favourite tool for looking in kubernetes (kubectl, k9s, …) and list the pipelineruns in the jx namespace you will proably be able to identify the one you are interested.

Preview Environments

Mon, 01 Jan 0001 00:00:00 +0000

If you are using Trunk Based Development you want to keep the main branch clean and ready to release at all times so you can quickly get important bug fixes out.

But how can you shift left and do more testing on changes before you are ready to merge to the main branch?

The idea of a Preview Environment is to spin up a temporary environment to deploy the code from a Pull Request before the change is merged to get fast feedback from your team on the change and to shift left and get more testing done before you agree to merge the change.

Repository

Mon, 01 Jan 0001 00:00:00 +0000

JayeX lets you configure different artifact repositories. We use artifact repositories to:

store artifacts from some kinds of build (e.g. Java builds tend to deploy jars, pom.xml files and tarballs)
act as a Maven proxy to cache maven dependencies when using java/maven builds
implement a chart repository for releasing helm charts

Nexus

By default if you don’t make any explicit configuration then JayeX uses:

Nexus as an artifact repository to store artifacts (e.g. Java jars, pom.xml files, tarballs or npm modules etc)
ChartMuseum as a repository of charts

You can explicitly configure nexus via the following jx-requirements.yml file:

Secrets

Mon, 01 Jan 0001 00:00:00 +0000

How do I add a new Secret?

See how to add a new Secret

How do I edit a secret?

Secrets are stored in external secret storage like Vault or your cloud providers secret store.

So you can edit them in the underlying secret store directly (e.g. with your cloud providers CLI or the vault CLI).

Or you can use jx secret edit command to do this for you.

The interactive mode is a nice way to find the secret and property you want to edit, then edit…

Slack

Mon, 01 Jan 0001 00:00:00 +0000

Many of us use chat to keep in touch with the developers and tools we work with. Slack is becoming popular in commercial settings and is already used throughout the Kubernetes open source ecosystem: including the JayeX project slack channels

To get slack notifications of pipelines you can use the jx-slack plugin.

Creating the slack app

Before you can install the jx-slack plugin you need to create a Slack app.

create a new slack app and fill in the details of the application name and associate it with the slack workspace you wish to use
navigate to the Features / OAuth & Permissions page on the slack app site

add the Scope chat:write to your bot so it can post messages to your slack workspace
find your Bot User OAuth Access Token which should start with xoxb- you will need it later…
invite the slack app you have created into whatever channels you want it to notify. e.g. inside the channel you can type @ and start typing the slack app name to send it a message which will get Slack to prompt you to invite the bot user to the room.

Installing

To install the jx-slack plugin add the following to your helmfiles/jx/helmfile.yaml file in your dev cluster git repository in the releases: section:

Storage

Mon, 01 Jan 0001 00:00:00 +0000

By storage we mean cloud storage (e.g. bucket storage for logs). Any chart you add via helmfile which uses Persistent Volumes instead of cloud storage will be up to you to backup and manage.

If you are using a cloud platform and terraform then your storage will be configured OOTB.

If you are on-premises and have installed something like minio on your cluster then you can configure the storage section of your jx-requirements.yml file to map to minio bucket URLs

Previews

Mon, 01 Jan 0001 00:00:00 +0000

When do Preview Environments get removed?

We have a background garbage collection job which removes Preview Environments after the Pull Request is closed/merged. You can run it any time you like via the jx preview gc command

jx preview gc

You can also view the current previews via jx preview get:

jx preview get

and delete a preview by choosing one to delete via jx preview destroy:

jx preview destroy

How do I access the preview namespace or URL?

After the jx preview create step in a pull request pipeline you can access a number of preview environment variables.

Vault

Mon, 01 Jan 0001 00:00:00 +0000

After an upgrade the boot job is waiting for vault in jx-vault

In the alpha we used to install vault via the helmfile/secret-infra/helmfile.yaml file and install vault into the secret-infra namespace.

A production installation of vault requires cloud resources such as a key ring, crypto key and bucket.

So to make it easier to manage vault properly with cloud resources and to simplify the operation of JayeX (so that the secret store can be used on an empty cluster before we boot anything in the boot process) we have moved the installation of vault into terraform. (e.g for GKE)

ChatOps

Mon, 01 Jan 0001 00:00:00 +0000

What is ChatOps?

We use the phrase ChatOps to mean operating code changes, pipelines and GitOps promotion via chat. More specifically this usually is done via commenting on Pull Requests on your git providers website; though in the future this could be via Slack or web consoles too.

What are the benefits of ChatOps?

ChatOps helps developers collaborate on Pull Requests and speeds up merging of Pull Requests. We want to be able to merge changes as quickly as possible into master so that we continuously integrate code which minimises the downsides of long term feature branching and merge hell.

Cluster

Mon, 01 Jan 0001 00:00:00 +0000

Next we will configure the cluster requirements:

Install external-dns - Kubernetes controller which watches for new Kubernetes Ingress resources and creates A records in Google Cloud DNS which will propagate globally across the internet
Install cert-manager - Kubernetes controller which watches for requests to ask Let’s Encrypt to issue a new wildcard TLS certificate for your domain and will manage this including renewals

Cert-manager will use the cluster issuer to request a TLS certificate. A Kubernetes secret will be automatically created and contain the TLS cert. The nginx controllers in the nginx namespace will use this secret in the jx namespace for the default SSL certificate which will automatically enable TLS for all applications in your cluster.

Multiple domains

Mon, 01 Jan 0001 00:00:00 +0000

Example: Use different domains for production and staging environments

Add acme chart to helmfiles/jx-production/helmfile.yaml

- chart: jx3/acme
  version: 0.0.19
  condition: jxRequirementsIngressTLS.enabled
  name: acme-jx
  namespace: jx-production
  values:
  - jx-values.yaml
  - acme-values.yaml

Create a file helmfiles/jx-production/acme-values.yaml

issuer:
  enabled: true
  cluster: false

Do the same for other environments, e.g. staging. Make sure the correct namespace is set.

Add ingress and tls configuration to the environments in jx-requirements.yaml

  environments:
    - ingress:
        domain: staging.foo.io
        externalDNS: true
        tls:
          email: admin@foo.com
          enabled: true
          production: true
          secretName: tls-staging-foo-io-p
      key: staging
    - ingress:
        domain: foo.io
        externalDNS: true
        tls:
          email: admin@foo.com
          enabled: true
          production: true
          secretName: tls-foo-io-p
      key: production

Make sure to use the correct secretName, as it’s being generated (production or staging) by the acme chart.

Cluster Rebuild Example

Mon, 01 Jan 0001 00:00:00 +0000

This is an example on how to treat your clusters as cattle and not pets by rebuilding a JX3/GKE environment and restoring to it’s previous configuration. The steps will include building a JX3 Google/GSM environment from scratch and deploying projects to staging and production. It will utilize DNS (jx3rocks.com), TLS, Let’s Encrypt certificates. The example includes the recovery of the helm chart repository which is currently implemented with ChartMuseum . This example is inteded for an audience already familiar with JayeX operability and focuses on an example of actual commands used to build and maintain a recoverable environment. Additional information regarding using Google as the provider for this example can be found under Google Cloud Platform Prequisites.

FAQ

Mon, 01 Jan 0001 00:00:00 +0000

What if I have a chartmuseum with charts running using nip.io?

It is best to comment out your JayeX chartmuseum repository and charts from your helmfile until your new domain and ingress is working. Then uncomment and make sure you update the chartmuseum URL to your new one.

What if I use a subdomain with an apex domain in a different GCP project?

When using a subdomain Terraform will create a managed zone in GCP, add the recordsets to your parent / apex domain.

JayeX infrastructure

Mon, 01 Jan 0001 00:00:00 +0000

JayeX runs it’s infrastructure in Google Cloud Platform (GCP). We have 2 GCP projects

jenkinsxio
jenkins-x-bdd

jenkinsxio

This is where the main infrastructure runs (the jobs triggered when a pull request is opened in any repository). Some details about the infrastructure are shown below

GKE kubernetes version 1.21
Spot instances to keep costs low
- n1-standard-2 with boot size 100 GB
Cluster autoscaler scales the nodes between 1 and 5.
We use terraform, an infrastructure as Code tool to manage the cluster using the google-jx module.

jenkins-x-bdd

This project is only used for bdd testing. Test clusters are created here when a pull request is opened in the jx3-versions repository.

Pipeline activites

Mon, 01 Jan 0001 00:00:00 +0000

JayeX creates pipeline activites for jobs. It’s a kubernetes custom resource (CR) which is scoped to a namespace.

Pipeline activities are made up of steps. A step can have three kinds

Stage
Preview
Promote

Each type of step has a name, status, start and end timestamp. A pipeline activity can have steps of different kinds (for example a pipeline activity can have a stage, preview and promote step) Let’s look at each of these kinds in detail.

Prefer Cloud Over Kubernetes

Mon, 01 Jan 0001 00:00:00 +0000

You can deploy a database via a helm chart in your kubernetes cluster. Or you can configure your cloud provider to create a managed database offering.

You can deploy, say, vault as helm charts inside your kubernetes cluster. Or you can use your cloud providers secret store solution such as:

Alibaba Cloud KMS Secret Manager
Amazon Secret Manager
Azure Key Vault
GCP Secret Manager

However installing a chart in kubernetes is a very different thing to having a fully managed service which considers things like:

Secrets

Mon, 01 Jan 0001 00:00:00 +0000

Commons problems with Secrets

Things to check:

the general health

jx health status -A

the boot job

jx admin logs

and make sure in the logs jx secret populate command is successful.

check the external secrets deployment and logs

kubectl get pods -n secret-infra
kubectl logs deploy/kubernetes-external-secrets -n secret-infra

check the status of the external secret using kubectl

kubectl get es -n [the namespace the secret / external secret is in]

skaffold

Mon, 01 Jan 0001 00:00:00 +0000

See the skaffold site on how to use skaffold to perform localy builds of a project.

Essentially you do the following:

install skaffold
cd into a git clone of your application
run

skaffold init

then to run your application

skaffold run --tail

Variables

Mon, 01 Jan 0001 00:00:00 +0000

Environment Variables

The following tekton parameters and environment variables are available inside the pipeline catalog. They are populated by lighthouse

Variable	Description
BUILD_ID	a unique long number for this build
JOB_NAME	the name of the build which matches the name in the `presubmit` or `postsubmit` in your lighthouse `triggers.yaml`
JOB_SPEC	is of the form `type:presumit` or `type:postsubmit` so you know what kind of job you are inside
PULL_BASE_REF	the base branch name. e.g. `master` or `main`
PULL_BASE_SHA	the base git SHA being built
REPO_NAME	the name of the git repository
REPO_OWNER	the owner of the git repository (a user or organisation)
REPO_URL	the git URL to clone the repository being built

Pull Requests presubmit also have the following values:

Map Cloud IAM to Kubernetes Service Accounts

Mon, 01 Jan 0001 00:00:00 +0000

Cloud providers make it relatively easy to map kubernetes Service Account resources to cloud IAM roles and accounts which avoids having to expose secrets to access cloud resources which simplifies things and makes your applications more secure.

On AWS use IAM roles for service accounts (IRSA)

On GCP use Workload Identity (WLI)

In both cases this maps cloud IAM roles to kubernetes ServiceAccount resources using annotations.

This means that you don’t have to populate your kubernetes cluster with cluster-admin style cloud IAM secrets - which makes your system more secure and reduces the possibility of accidentally exposing a secret.

Ingress

Mon, 01 Jan 0001 00:00:00 +0000

How do I configure the ingress domain in Dev, Staging or Production?

With the new helm 3 based boot every environment uses boot - so there is a single way to configure anything whether its in the dev, staging or production environment and whether or not you are using multiple clusters.

You can override the domain name for use in all apps within an environment by modifying the jx-requirements.yml in the git repository for the dev, staging or production environment.

Add sbom generation step to release pipelines

Mon, 01 Jan 0001 00:00:00 +0000

Introduction

JayeX started to support supply chain security through different steps. Till now we support:

SBOM generation
Uploading SBOMs to the release page and to our container-registry

How we do it?

There are two types of artifacts we deal with:

Zipped binaries

For .tar.gz artifacts, we use goreleaser to generate the artifacts themselves and to generate SBOMs for them. (Both get released on the release page on github of the repository). See Assets at the release-page for example.

Catalog

Mon, 01 Jan 0001 00:00:00 +0000

As we create more and more software we tend to get an explosion in the number of git repositories and microservices. Each repository needs automated CI and CD; but how do we manage the hundreds of pipelines we need - while also making it easy to share pipelines across repositories and allowing each repository to customize when required?

JayeX solves this as follows:

the pipelines, tasks and steps are defined via Tekton YAML letting you use any tekton tooling such as IDE completion and validation
we support an image: uses:sourceURI notation that lets you inherit steps from a git repository without having to copy/paste the source code aross repositories.

For example if you create a new quickstart pipeline may look like this (slightly condensed)…

Deployment lifecycle

Mon, 01 Jan 0001 00:00:00 +0000

Ever wondered how the code you commit to your source control repository ends up deployed in production by JayeX? Well, this document is an attempt to answer that question.

Please go over documentation regarding source repositories, environments and pipeline activities before reading this section.

After you have installed JayeX in a kubernetes cluster and created a quickstart or imported an existing repository, you should see a folder structure which resembles this:

├── charts
│   └── app
│       ├── Chart.yaml
│       ├── .helmignore
│       ├── Kptfile
│       ├── Makefile
│       ├── README.md
│       ├── templates
│       │   ├── canary.yaml
│       │   ├── deployment.yaml
│       │   ├── _helpers.tpl
│       │   ├── hpa.yaml
│       │   ├── ingress.yaml
│       │   ├── ksvc.yaml
│       │   ├── NOTES.txt
│       │   ├── sa.yaml
│       │   └── service.yaml
│       └── values.yaml
├── Dockerfile
├── .gitignore
├── go.mod
├── go.sum
├── .lighthouse
│   └── jenkins-x
│       ├── pullrequest.yaml
│       ├── release.yaml
│       └── triggers.yaml
├── main.go
├── Makefile
├── OWNERS
├── OWNERS_ALIASES
└── preview
    ├── helmfile.yaml
    ├── Kptfile
    └── values.yaml.gotmpl

The JayeX pipeline files are all located in the .lighthouse/jenkins-x folder. Inside this folder, you should see these files:

localizer

Mon, 01 Jan 0001 00:00:00 +0000

localizer is a lightweight go lang based rewrite of telepresencse. See the blog for more detail

It basically lets you run local processes on your laptop right inside your IDE which are then connected into the kubernetes cluster via networking magic.

Using localizer

install localizer
run localizer:

sudo localizer

run/debug your application locally.
then to expose your service into a namespace as a service name run something like this::

localizer expose $namespace/$svc --map 80:8080

when you have finished debugging, return things to normal via:

Source

Mon, 01 Jan 0001 00:00:00 +0000

JayeX is built on the shoulders of open source giants and also has lots of different source repositories to make various things from CLI tools, docker images and helm charts

This page lists the main organisations and repositories.

Organisations

jenkins-x the main organisation for source code
jenkins-x-plugins contains plugins to JayeX 3.x. See the extension guide for details
jenkins-x-charts the main helm charts we distribute
jenkins-x-images contains some custom docker image builds
jenkins-x-quickstarts the quickstart projects used by create quickstart
jenkins-x-test-projects test projects we use in test cases
jx3-gitops-repositories the quickstart repositories for creating new infrastructure and installations on different cloud providers

Repositories

Here we’ll call out of some of the main repositories in the above organisations:

Troubleshooting

Mon, 01 Jan 0001 00:00:00 +0000

Why did my quickstart / import not work?

If you are not able to create quickstarts or import projects its most probably webhooks not being setup correctly.

When the jx project import or jx project quickstart runs it creates a Pull Request on your dev cluster repository. This should trigger a webhook on your git provider which should then trigger a Pipeline (via lighthouse webhooks). The pipeline should then create a second commit on the pull request to configure your repository which then should get labelled and auto-merge.

Webhooks

Mon, 01 Jan 0001 00:00:00 +0000

Webhooks are used to trigger lighthouse when you:

merge commits to your main branch
open a Pull Request
push code to a Pull Request branch
comment on a Pull Request to trigger ChatOps

If webhooks are not working in your cluster then you won’t get ChatOps or pipelines triggered.

Requirements

For webhooks to work you must have a working Ingress for the hook endpoint from lighthouse

The ingress defaults to using the domain name specified in ingress.domain in your jx-requirements.yml file.

Debugging

Mon, 01 Jan 0001 00:00:00 +0000

Warning

TaskRun breakpoint functionality is no longer supported since Tekton 0.29.0 upgrades in 3.2.298. For more info see Kubernetes 1.22 - Breaking change!.

Here is a demo which shows how to debug pipelines:

Prerequisites

Make sure your cluster is upgraded to the latest version stream.

If you intend to use the jx in the below examples make sure you upgrade the CLI too

Enable a breakpoint

To enable a breakpoint you can:

ChatOps

Mon, 01 Jan 0001 00:00:00 +0000

You can type the following ChatOps commands on Pull Requests as comments:

ChatOps Commands

ChatOps Command	Description
`/lgtm`	This PR looks good to me - this command can be from anyone with access to the repo who is in the `OWNERS` file
`/approve`	This PR can be merged - must be someone in the repo `OWNERS` file
`/test this`	Run the default test pipeline context for this PR
`/test (context)`	Run a specific test pipeline context by name
`/retest`	Rerun any failed test pipeline contexts for this PR
`/override (context)`	Override a failed pipeline context
`/hold`	Set this PR to not automerge even if it has been set `lgtm` and approved
`/hold cancel`	remove the `hold` label from the PR, allowing automerge
`/assign (user)`	assign the PR to the given (`user`)
`/unassign (user)`	remove the `user` as assignee
`/cc (user)`	add the given `user` as a reviewer for the PR
`/uncc (user)`	remove the `user` as a reviewer
`/ok-to-test`	If a `user` without write access to the repo opens a PR, the PR will not be built automatically. It receives the `needs-ok-to-test` label, until a user with rights enters `/ok-to-test`, at which point it gets be built

Previews

Mon, 01 Jan 0001 00:00:00 +0000

JayeX lets you spin up Preview Environments for your Pull Requests so you can get fast feedback before changes are merged to the main branch. This gives you faster feedback for your changes before they are merged and released and allows you to avoid having human approval inside your release pipeline to speed up delivery of changes merged to master.

When the Preview Environment is up and running JayeX will comment on your Pull Request with a link so in one click your team members can try out the preview!

Extending

Mon, 01 Jan 0001 00:00:00 +0000

JayeX has a number of extension points you can use to extend the CI/CD platform to suit your needs:

Charts

Helm charts are the standard way to package applications for kubernetes.

It’s easy to use GitOps to add charts to any of your clusters and customize them however you need.

You can also easily add one or more kubernetes resources to a cluster via a source layout chart

Plugins

The jx command line in version 3 is build on plugins.

jx help

Mon, 01 Jan 0001 00:00:00 +0000

You can browse the Command Line Reference online or you can use the jx CLI directly to get help:

jx --help

To get help about a particular plugin, such as gitops you can type:

jx gitops --help

Quickstart/Import

Mon, 01 Jan 0001 00:00:00 +0000

After a successful installation of JayeX, the first thing to do is to either create a quickstart or import an existing repository into JayeX. They are both very similar, so the steps mentioned here can be applied for both.

Import/Quickstart involves the following steps:

Cloning of the dev/cluster git repository
Selecting and copying catalog folder into imported repository
Pull Request in the cluster git repository
Webhook creation in the imported repository
Addition of the repository name to the config configmap
Source repository creation in the jx namespace

Issues with cloning dev repository

If you encounter an issue of this form:

Upgrading

Mon, 01 Jan 0001 00:00:00 +0000

Converting older pipelines

If you have older pipelines in your git repository created with an older version of JayeX you can convert them to the new concise image: uses:sourceURI syntax via the jx pipeline convert command:

jx pipeline convert

The pipelines should be modified and if you have a Kptfile it will be removed.

If you have any old jenkins-x.yml files, those will be converted across to the new .lighthouse tekton notation too.

Version Stream Manual Migrations

Mon, 01 Jan 0001 00:00:00 +0000

Version Stream Manual Migrations

The sections below document manual migration steps required that occur from time to time where an automated upgrade is not possible or practical

v0.0.969

NB: These steps are only relevant if you are using Hashicorp Vault as secret storage within jx. Ignore for all other secret stores

v0.0.969 introduced a change to remove Hashicorp Vault from jx control and become an explicit cluster dependency installed either via Terraform (for cloud environments) or Helm (for on-premises). The reasoning behind this is to reduce cyclic complexity within jx where the platform requires secrets and the jx platform is also responsible for provisioning where to store these secrets.

Ngrok

Mon, 01 Jan 0001 00:00:00 +0000

Follow these steps to set up ngrok for local JayeX installs.

Download and install ngrok.
Log into ngrok account and get the ngrok auth token (Authenticated sessions can run for unlimited time, unauthenticated sessions expire in 1.5 hours):
Create a ngrok config file at ~/.config/ngrok/ngrok.yml with the following content (replace <ngrok-auth-token> with the auth token from the ngrok account):

authtoken: <ngrok-auth-token>
tunnels:
  hook:
    proto: http
    addr: 8080
    schemes:
      - http
  ui:
    proto: http
    addr: 9090
    schemes:
      - http
version: "2"
region: us

Verify that the config is correct using:

ngrok config check

Run this in a new terminal window/tab:

ngrok start --all

Once the command succeeds, you should see something like this:

ngrok                                                                                                                                                                                       (Ctrl+C to quit)

Hello World! https://ngrok.com/next-generation

Session Status                online
Account                       user123 (Plan: Free)
Update                        update available (version 3.0.6, Ctrl-U to update)
Version                       3.0.5
Region                        United States (us)
Latency                       25ms
Web Interface                 http://127.0.0.1:4040
Forwarding                    http://XXXX.ngrok.io -> http://localhost:8080
Forwarding                    http://YYYY.ngrok.io -> http://localhost:9090

Connections                   ttl     opn     rt1     rt5     p50     p90
                              0       0       0.00    0.00    0.00    0.00

JayeX Most Valuable Contributor 2026 🏆

Wed, 08 Apr 2026 00:00:00 +0000

The Continuous Delivery Foundation gives awards to recognize all the work that makes this community and the progress of Continuous Delivery possible.

Vote on someone for the award for JayeX Most Valuable Contributor 2026

This award recognizes excellence of technical contributions to JayeX. This individual commits key pieces to projects and, more importantly, contributes in a way that benefits the project neutrally as a whole.

You vote by someone by adding a thumbs on the comment with their name in this issue. Voting is now open and will close on Tuesday, May 5, 2026.

Jenkins X changes name to JayeX!

Thu, 19 Mar 2026 00:00:00 +0000

Background

During the early days of Jenkins X the Jenkins part of the name made sense for two reasons: Jenkins was used as the main pipeline engine and the development was done within CloudBees, the company that also is the main developer of Jenkins.

Why change?

Nowadays the Jenkins X name is more of a burden than boon marketing wise: It attracts people interested in Jenkins, who will dissapointed. It detracts people that are looking for an alternative to Jenkins, who might actually find what this project offers attractive.

JayeX Most Valuable Contributor 2026 🏆

Thu, 05 Mar 2026 00:00:00 +0000

The Continuous Delivery Foundation gives awards to recognize all the work that makes this community and the progress of Continuous Delivery possible.

Nominate someone to the award for JayeX Most Valuable Contributor 2026

You nominate people by adding a comment with their name in this issue. Nominations are now open and will close on Thursday, April 2, 2026

Migrate to Google Artifact Registry

Mon, 06 May 2024 00:00:00 +0000

Google has announced that container registry will be shut down some time after March 18, 2025. For GKE clusters created with version 1.12.0 or later of terraform-google-jx it’s unlikely that anything needs to be done, but for older clusters you should upgrade your cluster while considering our advice regarding migration from container registry to artifact registry.

If you are using a Google Service Account to run terraform you need to add the role requirement roles/artifactregistry.admin. See our guide regarding Google Service Account for details.

Improve your changelogs

Wed, 24 May 2023 00:00:00 +0000

Background

A standard part of the Jenkins X pipelines since a long time is the execution of jx changelog create that takes the commit messages between the release currently being created and the previous one and creates a change log from these. The change log is then stored as a release note in GitHub or other git provider.

During the last year some improvements have landed in various Jenkins X components to improve the changelogs and their usefulness. So I’ll take this opportunity to describe these improvements and also in general give hints to how to get useful changelogs.

Reconcile with kpt live apply

Thu, 09 Mar 2023 00:00:00 +0000

Since the dawn of Jenkins X 3 the default last step of reconciling the state of the files in your cluster repository to your cluster has been to execute kubectl apply. You can find more details about this here.

There are some drawbacks with kubectl apply though. The one that made me start looking for alternatives was that if you remove a resource from your cluster repository it may not be removed from your cluster. The way deletion works with kubectl apply is that it is handed the option --prune which will remove resources that are not in the manifests. Except that it doesn’t always work as expected. It will only remove certain kinds of resources defined in kubectl. In my case I removed an HorizontalPodAutoscaler from my cluster repository, but it wasn’t removed from my cluster.

Foreign aliases

Thu, 09 Feb 2023 00:00:00 +0000

Background

In an organisation with many repositories and developers that are frequently shifting the maintenance of OWNERS and OWNERS_ALIASES files can be tedious. In the passing year a couple of functionalities has been added to help with this.

Foreign aliases

To avoid maintaining the OWNERS_ALIASES file in many repositories you can now refer to the OWNERS_ALIASES file in another repository. In the Jenkins X project we have the main OWNERS_ALIASES file in the jx-community repository. So in the jx repository the OWNERS_ALIASES file only looks like this:

Project ideas for Google Summer of Code 2023 ☀️

Mon, 06 Feb 2023 00:00:00 +0000

We have put together some project ideas as part of our application to participate in the Google Summer of Code 2023 program.

1. CD events integration with Jenkins X

Description

The cdEvents project standardises the way systems talk to each other, which enables Interoperability between systems so they speak a common language through the cdEvents spec in the event. Creating a capability in Jenkins X that can receive and sent a cdEvent would benefit the project and the DevOps ecosystem in general, by stopping glue code used to integrate systems and power innovation by letting end users swap out tools with no effort.

GSoC 2022 Final Report: Building Jenkins X UI

Sun, 13 Nov 2022 00:00:00 +0000

Jenkins X New UI

It is a web application built with Golang for the backend and Sveltekit for the frontend, both of which are built together and used in the same container. To function properly, it must be installed as a helm chart with Jenkins X CRDs.

🌟 It has light and dark themes.

Why need a new UI?

A good UI is essential for a CI/CD tool, as not everyone is familiar with the CLI. The current UI (jx-pipeline-visualizer) is a read-only UI, the user can view the logs of PipelineActivity but neither can start nor stop the pipeline.

GSoC 2022 Final Report: Improving Supply Chain Security

Tue, 08 Nov 2022 00:00:00 +0000

Project Description

Supply chain security is a rising concern in the current software era. Securing the software supply chain encompasses vulnerability remediation and the implementation of controls throughout the software development process. Due to massive increase in attacks on software supply chain and the diversity of its types, Jenkins X has to make efforts to ensure that the build process is secure. As part of securing Jenkins X installation by default I worked on both securing our own components and enabling our users to use these features in their build and release steps..

Hacktoberfest 2022

Mon, 03 Oct 2022 00:00:00 +0000

We are excited to announce that Jenkins X will be participating in Hacktoberfest again this year! Hacktoberfest is a month-long global celebration of open source software.

All backgrounds and skill levels are encouraged to participate in Hacktoberfest and join a global community of open source contributors.

Learn more about Hacktoberfest and sign up here.

Contribute to Jenkins X

We welcome your contributions to the Jenkins X project!

Issues labelled “hacktoberfest” generally indicate good first issues. However, all pull requests will count towards your Hacktoberfest challenge.

Introduction to Software Bill Of Materials

Sun, 24 Jul 2022 00:00:00 +0000

Introduction

Before going through Software Bill Of Materials (SBOMs), we need to set the ground for a rising concern in the software industry which is Software Supply Chain Security. Like traditional industries, deploying a piece of a software artifact goes through multiple stages composed of collecting source code components, libraries, tools, and processes used in those stages.

Software Bill Of Materials Formats

Sun, 24 Jul 2022 00:00:00 +0000

Prerequisite

If you don’t understand what is Software Bill of Materials (SBOM), please read this blog post first.

Different SBOM formats comparison

The National Telecommunications and Information Administration (NTIA) in the U.S. defined minimum requirements for SBOM formats:

Identifying the supplier of the software component.
Identifying the details about the version of the component.
Including unique identifiers for the component like cryptographic hash functions.
Including the relationships between all dependencies inside the component.
Including a timestamp of when and by whom the SBOM report was created or last modified

In this section, we discuss different kinds and formats for SBOM standards and make a brief comparison between them. Three commonly used standards achieved the NTIA minimum requirements for SBOM generation and each one results in a different final SBOM document.

Software Bill Of Materials generation tools

Sun, 24 Jul 2022 00:00:00 +0000

Prerequisite

Before you read this, you have to understand what are SBOMs and what are different formats of SBOMs

Different SBOM generation tools comparison

If you got this far, you already realize the importance of SBOM generation, and also it should meet certain requirements to achieve its purpose. Due to various requirements depending on what standard you’re following, there has to be a way to automatically generate different output formats for different standards. Also, it has to be suited for ci/cd solutions to keep up with the increasing number of releases for each organization.

GSoC 2022 Community Bonding Period with Jenkins X

Tue, 12 Jul 2022 00:00:00 +0000

Introduction

Hello everyone, I am Rajat Gupta, pursuing my bachelor’s in Information Technology. In 2022, I have been selected as a student developer in Google Summer of Code under Jenkins X. We will be building a new UI for Jenkins X. I got this news on May 20th, as I received an email from google.

How I started

The technologies needed were Golang, Kubernetes, and GitOps. I used golang only once before, while linting Jenkins X codebase, I only used Kubernetes once before while setting up a k3s cluster to run Jenkins X pipelines. These tasks were necessary to do for all GSoC participants. Apart from that, I was a total beginner.

Kubernetes 1.22 - Breaking change!

Fri, 22 Apr 2022 00:00:00 +0000

To allow Jenkins X to support Kubernetes 1.22, we had to update our version of Tekton. This updated version of Tekton contains breaking changes that has consequences if you made your own custom Jenkins X pipelines.

To make sure that your custom pipelines continue to work after this upgrade, you must edit the resource settings in your pipelines. Otherwise your pipelines will most likely not be able to start at all, or if they do, consume a lot of resources.

Google Summer of Code 2022 project proposal template ☀️

Tue, 05 Apr 2022 00:00:00 +0000

Proposal template

1. Contact details

Full name:
Country:
Time zone:
Email:
Github ID:
Personal blog (optional):
Twitter/LinkedIn/others:

2. GSOC information

Have you participated in the Google Summer of Code previously? Please describe your experience.
Are you applying to any other organizations this year? If so, please list them.
How many hours will you devote to your GSoC project each week? Do you have any other commitments during the summer?
Have you ever contributed code to Jenkins X? If you have, post the Pull Request (PR) links (It’s ok if they have not been merged/approved yet)
Do you plan on contributing to the Jenkins X project after GSoC is finished?
Were you able to install Jenkins X locally on your laptop?
- If no, please explain why (Unsupported architecture, missing documentation, time constraints etc)?
- If yes, how was the installation experience (missing documentation, complexity etc)?
Have you used kubernetes in the past? Please provide a brief description and if possible links to the work.
Have you used golang in the past? Please provide a brief description and if possible some code samples.
(If you are interested in the UI project) Have you worked with a frontend framework/library in the past? Please provide a brief description and if possible some code samples.

3. Project idea

Repeat this section for as many project ideas as you want (Go in the order most likely to least likely).

Project proposal for Google Season of Docs 2022 📄

Wed, 23 Mar 2022 00:00:00 +0000

We have put together a project proposal as part of our application to participate in the Google Season of Docs 2022 program.

Project proposal

Better documentation for real world use cases - Jenkins X

About us

Jenkins X provides automated CI/CD for Kubernetes with Preview Environments on Pull Requests using Cloud Native pipelines from Tekton.

Problem

End users of Jenkins X are unable to find information on how to use Jenkins X for real world use cases which includes:
- Production best practices
- Scanning images in Jenkins X pipelines
- How to make your app use different configuration/secrets for each environment
- Observability for your apps, logging, metrics, tracing
- Tests (Integration, e2e)
- Artifact management
- Multi cluster
- Integration with other tools in the cloud native sector

Project scope

Audit the current Jenkins X documentation and create a friction log
Use the friction log as a guide for understanding the gaps in the documentation
Interact with the Jenkins X community to create a list of top use cases
- Create a survey
- Look at the github issues and slack messages
Work with the volunteers to incorporate those changes into the documentation
Establish a feedback loop (the target customers are the end users of Jenkins X) to improve the documentation

Measuring project success

Decrease in the number of github issues raised by end users on documentation related to common real world use cases
Short 1-2 minute videos on common tasks that end users of Jenkins X would like to perform.

Recommended skills

Must have:
- Experience with github
- Communication skills to work with the Jenkins X community
- Creating online surveys
Nice to have:
- Basic experience with docker compose (local development scripts use compose)
- Good video editing skills.
- Some knowledge working with the documentation of other tools in the kubernetes ecosystem

Mentors/Volunteers

Timeline

The project will take roughly 6 months to complete. Once the technical writer is selected, we will do an orientation to bring him up to speed. This is the timeline we have in mind, but we are flexible.

Google Summer of Code 2022 ☀️

Sat, 12 Mar 2022 00:00:00 +0000

Project proposal template can be found here.

We are very happy to announce that Jenkins X has been selected to participate in the Google Summer of Code (GSoC) 2022!

If you are new to GSoC and want to learn more about it, check out their new site. You can find the list of project ideas from Jenkins X here. If you are interested in applying to Jenkins X, please check the timeline. Apply as early as possible.

Project ideas for Google Summer of Code 2022 ☀️

Sun, 20 Feb 2022 00:00:00 +0000

Note

Jenkins X has been accepted into the Google Summer of Code 2022 program, take a look at the follow up blog.

Project proposal template can be found here.

We have put together some project ideas as part of our application to participate in the Google Summer of Code 2022 program.

1. Cloud events integration with Jenkins X

Description

The only way to trigger jobs/workflows in Jenkins X at the moment is by listening to events from Source Control Management (SCM) providers like github, gitlab, bitbucket, however it would be nice to listen to other event sources and trigger jobs/pipelines in Jenkins X. One interesting application would be to trigger some Jenkins X job in response to some alerting event (pagerduty, opsgenie). As a start we should focus on (emitting and listening to) cloudevents which define a common format for events produced from different sources. This will also help make Jenkins X compatible with other platforms.

Jenkins X Survey Result Details

Tue, 15 Feb 2022 00:00:00 +0000

For more information on the Jenkins X survey see Survey results Some highlights from the free text answers.

What do you enjoy the most with Jenkins X?

KUBERNETES

Cloud-native feel, “you know Kubernetes, you know how to use JX”, plugins-based architecture
Complete CI/CD platform in a k8s cluster

GIT/CHAT/OPS/CONFIGURABILITY

Configuration as code, auto generation of k8s files, flexibility
easy to create preview/dev/staging/prod environments.
modularity and ease to extend/change

EASY

Everything works out of the box.
Very intuitive
Lightweight and easy to manage.

COMMUNITY

The community is incredible. I really like how everything works together.
love the community.

VARIOUS

scalability and how fast it is!
Integration with different secret backends is easy. Also love the community.
It’s opinionated
No vendor lock in

What do you believe should be improved with Jenkins X?

ON PREMISE

run it against a local running cluster to test changes to Jenkins-X before updated in GitHub.

Jenkins X Survey Results

Tue, 15 Feb 2022 00:00:00 +0000

The Jenkins X survey was active for four weeks and closed on February 11 2022. We received lots of valuable insights into how people are using Jenkins X. We need more contributors in the Jenkins X community, so if you feel strongly about how Jenkins X should evolve, your best bet is to dive in and get your hands dirty:) Some highlights of the free text responses we got are collected here

January 2022 updates from the JX community

Wed, 02 Feb 2022 00:00:00 +0000

Happy new year 2022!

This monthly blog post series is an attempt to showcase all the incredible work being done by the Jenkins X community to the wider audience.

Lot of exciting features, bug fixes and documentation improvements were made.

Community effort

We restarted the office hours this month (https://jayex.io/community/#office-hours). Drop by to say hello, we are a friendly group!
First Jenkins X survey was also created this month (https://jayex.io/blog/2022/01/21/survey-1-2022/). We have extended the deadline by 2 weeks (Feb 11, 2022 midnight UTC), so fill it out if you have not yet. We will use this for the roadmap moving forward.
Monthly blog post update series to keep up with all the amazing progress.

Features

jx-plugins/jx-gitops:
- Cron job to delete old boot jobs
- Option to Keep n boot jobs older than default age
jenkins-x/lighthouse:
- Support for using gitlab nested repositories
- Allow running lighthouse with cluster scoped permissions
- Add a flag to keep polling releases until commit status is successful
- Do not include tekton roles when tekton engine is disabled
jx-plugins/jx-pipeline:
- Use pager to aid in visualizing long pipeline logs
jenkins-x-terraform/terraform-jx-azure
- Support provisioning spot instances in the azure terraform jx module
jx-plugins/jx-verify:
- Support specifying label for jx verify install
jenkins-x-plugins/jx-registry:
- Support adding ECR registry policy using jx-registry
jenkins-x-charts/jxboot-helmfile-resources:
- Make all storage locations available as envrironment variables

Bug fixes

GoogleContainerTools/kaniko (upstream fix - outside jx codebase):
- Kaniko in jx pipelines can now push to ACR (Azure Container Registry)
jenkins-x/terraform-aws-eks-jx:
- Remove deprecated jx v2 keys from requirements configmap
- Fix issue with on-demand billing mode of dynamodb
jenkins-x-plugins/jx-promote:
- Fix local chart check when using cloud buckets to store helm charts
jenkins-x-plugins/jx-project:
- Fix jx project rendering invalid chart.yaml files on import for custom packs in catalog

Documentation improvement

Guides on configuring Azure service principle and GCP service account for terraform users (https://jayex.io/v3/admin/platforms/azure/svc_principal/ and https://jayex.io/v3/admin/platforms/google/svc_acct/)
New section showing current Jenkins X users (https://jayex.io/#users)
New section on learning resources for Jenkins X v3 (https://jayex.io/v3/learning-resources/)
Improved documentation on setting up Jenkins X v3 on AWS EKS (https://jayex.io/v3/admin/platforms/eks/)
Improved guide on installing and configuring Jenkins X on k3s (https://jayex.io/v3/admin/platforms/k3s/)
Fixed content getting clipped in mobile view

Plumbing/Quality improvements

jenkins-x/jx:
- Disable running github actions on forks
jenkins-x/jx-docs:
- Upgrade docsy submodules
- Upgrade hugo docker image
- Migrate jx-docs to new pipeline format
jenkins-x/jx-gitops:
- upgrade kubectl version for kpt
- upgrade helm and helmfile version

Huge thanks to all the contributors for their hardwork!

Jenkins X Survey

Fri, 21 Jan 2022 00:00:00 +0000

We have made a short survey where we try to gain insight into how people experience Jenkins X. This is meant to be used as guidelines going forward so we can be more focused on what areas to improve.

All contributions are welcome, if you are just browsing or have used it for years, we want to know you all!

We have extended the survey period and it will now be open untill Tuesday February 11 2022 midnight UTC. All submissions are anonymous and the results will be published. So don’t hesitate, fill in the Jenkins X survey today

Incident: Kaniko and ACR

Tue, 28 Dec 2021 00:00:00 +0000

We’ve recently had an issue with one of our packages come to light. We wanted to talk through the resolution steps we’re going to put into place.

So what happened?

Azure users started reporting seeing the following error within the build step:

error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly, and try again: checking push permission for "xyz.azurecr.io/myorg/myrepo:0.0.1": resolving authorization for xyz.azurecr.io failed: error getting credentials - err: exec: "docker-credential-acr-env": executable file not found in $PATH

This seemed to be indicating to an authorization issues with the terraform module. Other users were seemingly unaffected by this issue.

Hacktoberfest conclusion 2021

Mon, 22 Nov 2021 00:00:00 +0000

Hacktoberfest 2021 is over, and we got quite a few contributions from the open source community.

Contributions included various document improvements, adding jira as an issue tracker for generating changelogs and adding initial support for external vault!

The top contributors to Jenkins X in hacktoberfest 2021 were:

Marten Svantesson
Christopher vig
James Rawlings
Hays Clark
Anatoli Babenia

We would also like to thank all the contributors who participated and made it a success. The strength of Jenkins X lies in it’s vast community, and we hope to see many more major contributions from them in the near and far future.

Hacktoberfest 2021

Wed, 06 Oct 2021 00:00:00 +0000

We are excited to announce that Jenkins X will be participating in Hacktoberfest again this year! Hacktoberfest is a month-long global celebration of open source software.

All backgrounds and skill levels are encouraged to participate in Hacktoberfest and join a global community of open source contributors.

Learn more about Hacktoberfest and sign up here.

Contribute to Jenkins X docs

We welcome your contributions to the Jenkins X documentation project!

Issues labelled “hacktoberfest” generally indicate good first issues. However, all pull requests will count towards your Hacktoberfest challenge.

Moving Jenkins X v2 artifacts

Thu, 26 Aug 2021 00:00:00 +0000

ACTION REQUIRED

TL;DR - Jenkins X specific helm repositories and container registries hosted on GCP have been moved to GitHub. This will mainly affect jx v2 users but there is expected to be a small impact on v3 users too. Below describes the steps we believe are needed to keep Jenkins X installations working as normal but there will be some action needed.

Why the disruption?

When Jenkins X first started we made heavy use of GCP’s services for hosting the cloud infrastructure needed by users to install and run Jenkins X. This was great as we could use the same IAM to push and maintain content from our own hosted build infrastructure and ensure we were validating the same experience of using cloud provider hosted services wherever possible. As Jenkins X grew in popularity the cloud costs began to increase with the pricing model from GCP, specifically the networking costs of cross continent egress.

How to debug your Tekton pipelines

Wed, 18 Aug 2021 00:00:00 +0000

TaskRun breakpoint functionality is no longer supported since Tekton 0.29.0 upgrades in 3.2.298

For more info see Kubernetes 1.22 - Breaking change!.

Tekton recently introduced a debug feature when you create TaskRun resources so that steps can be paused at a breakpoint until told to move forwards so that you can diagnose why pipeline steps fail.

The latest Tekton release only supports breakpoints on TaskRun resources but there is a Pull Request #4145 to add support also to debugging PipelineRun resources as well. If you are reading this please add your thumbs up emoji feedback to the PR #4145

How to use GitOps and Kubernetes External Secrets for better audit and security

Tue, 17 Aug 2021 00:00:00 +0000

So GitOps is a cool approach to managing kubernetes resources in a cluster, by checking in the source code for:

the kubernetes YAMLs
details of the helm charts you want to install along with any configuration
kustomize scripts.

Then everything is versioned and audited; you know who changed what, when and why. If a change breaks things, just revert via git like any other source code change.

You can then add pipelines to verify changes in the Pull Requests result in valid kubernetes YAML etc.

Jenkins X 3 and Argo CD

Mon, 28 Jun 2021 00:00:00 +0000

There have been a number of requests from the Jenkins X community to use Argo CD for the last mile deployment phase of their continuous delivery pipelines. This blog explains some of the advantages of using Jenkins X and Argo CD all together.

What’s included?

Jenkins X for Cloud Infrastructure using Terraform, core installation management with GitOps, external secret management, ingress controller, quickstarts, automated CI + CD pipelines, ChatOps
Tekton for pipeline orchestration
Argo CD for end users application deployments (not the main installation) Argo CD provides a declarative GitOps approach to deploying Kubernetes based applications. There is a GUI which helps construct the deployment definition (in the form of an Application custom resource) which offers a number of configuration options, as well and providing insight into your clusters applications.

You might be wondering why you would want to use BOTH Jenkins X and Argo CD together. Jenkins X aims to embrace OSS, where possible providing a nice UX to integrate with other projects and help provide better solutions for building, developing and running software on Kubernetes. Jenkins X indeed does have a git operator that applies Kubernetes YAML from a Git repository but there are some differences:

Continuous microservices with databases in Jenkins X

Fri, 25 Jun 2021 00:00:00 +0000

A common question we get asked on the Jenkins X project is how to get started creating microservices that use databases with automated CI/CD with GitOps Promotion and Preview Environments.

To make things a little easier to get started we’ve created a new node-postgresql quickstart.

Before you start

If you are using the cloud then we prefer cloud over kubernetes for things like databases, storage, ingress and secret managers so please try use your clouds managed databases if you can.

Jenkins X at cdCon

Tue, 22 Jun 2021 00:00:00 +0000

cdCon 2021 is about to start with lots of great sessions.

Here’ a list of the Jenkins X related sessions:

Tuesday, June 22 GitOps Summit

Best Practices for Secret Management with GitOps - Kara de la Marck, CloudBees
- GitOps uses Git as the “single source of truth” for declarative infrastructure and enables developers to manage infrastructure with the same Git-based workflows they use to manage a codebase. Having all configuration files version-controlled by Git has many advantages, but best practices for securely managing secrets with GitOps remain contested. Join us in this presentation about GitOps and Secret Management. Attendees will learn about different approaches to secret management with GitOps, the issues involved, and the secret management solutions offered by various tools and platforms. We will discuss the pros and cons of Vault, SOPS, offerings by public cloud providers, and more.

Wednesday, June 23

MLOps with Jenkins-X: Production-ready Machine Learning by Terry Cox
- Explore ways to treat Machine Learning assets as first class citizens within a DevOps process as Jenkins-X MLOps Lead, Terry Cox demonstrates how to automate your training and release pipeline in Cloud environments, using the library of ML template projects provided with Jenkins-X.
Enabling a DevOps Mindset at Scale in an Enterprise by Jimmy McNamara & Nick Penston, Fidelity Investments
- Talk through the cultural enablers to create a strong DevOps culture within large organisations. Nick and Jimmy will talk through the cultural enablers to support large numbers of very talented and ambitious cloud engineers. Touching on strategies supporting strong communication and talent development for cloud engineers. Key themes: Talent Development Enabling DevOps culture Harnessing the voice of the developer/cloud engineer
BoF Session: Jenkins X - James Strachan & James Rawlings, Cloudbees by James Strachan & James Rawlings
- This BoF will be an open discussion on anything related to Jenkins X, automating Continuous Delivery, Kubernetes, cloud and how to Accelerate delivering value to your customers

Thursday, June 24

How Jenkins X is Integrating Observability from the Inside, and the Benefits for its Users by Vincent Behar, Dailymotion

Don't use docker, use kubernetes

Mon, 17 May 2021 00:00:00 +0000

Are you developing software that’s intended to run on kubernetes? If so we recommend not to use docker on your laptop.

Docker on Windows/MacOS helps you run a VM that can then run linux containers easily. But why bother?

We highly recommend just use a development kubernetes cluster - build and run your containers there instead then you’re closer to a production like environment.

Why use kubernetes instead of docker?

why test on a completely different VM and container orchestrator than production? It’s better to test on a similar environment to where you are really going to deploy your code
test your kubernetes yaml / helm chart and associated configuration at the same time as you run your containers helps you catch mistakes earlier:
- it’s not just about running the container image; it’s about lots of other things too like networking, configuration, secrets, storage/volumes, cloud infrastructure, service mesh, liveness/readiness/startup probes - so why not test all of those things rather than just the image?
some corporate environments don’t let you run VMs on your laptop anyway so running docker locally isn’t an option

How to get kubernetes?

First you’ll need a kubernetes cluster.

Jenkins X 3 - May 2021 LTS

Wed, 12 May 2021 00:00:00 +0000

May 2001 LTS release is now available!

LTS is a slower cadence version stream which contains a verified set of releases and configurations that have been used by teams tracking the bleeding edge Jenkins X.

Included in this release:

Protect Pipeline Visualiser with OAuth2 how to docs
Terraform repositories are now protected by the Jenkins X version stream
external-dns helm chart upgrade to v5.0.0
Reduce external secrets polling of cloud services to reduce cloud costs
[Jenkins] for users using Jenkins the Tekton Client plugin is now installed by default
Stackdriver format logging enabled when using GKE and services that use jx-logging library. If you enable the Stackdriver API in GCP you will get well formatted logs and alerts via Stackdriver.
Jenkins X Grafana dashboards updates with Lighthouse telemetry

Jenkins X 3.x GA is here!

Thu, 15 Apr 2021 00:00:00 +0000

I’m super excited to announce the 3.0 GA (General Availability) release of Jenkins X!

Jenkins X automates your CI/CD on kubernetes to help you accelerate:

Automated CI/CD pipelines lets you focus on your actually application code while Jenkins X automatically creates battle tested Tekton CI/CD pipelines for your project which are managed via GitOps so that its super easy to keep your pipelines up to date across your repositories or to upgrade or override pipelines or steps for specific repositories.
Automatic promotion of versioned artifacts via GitOps through your Environments such as Staging, Pre-production and Production whether they are running in the same kubernetes cluster or you are using multiple clusters for your environments
Preview Environments lets you propose code changes via Pull Requests and have a Preview Environment automatically created, running your code in kubernetes to get fast feedback from your team before agreeing to merge changes to the main branch
ChatOps comment on Pull Requests to give feedback, approve/hold changes, trigger optional pipelines for additional testing and other ChatOps commands

Demo

Here’s a demo of how to develop code with Jenkins X

Jenkins X 3 - April 2021 LTS

Mon, 12 Apr 2021 00:00:00 +0000

This is the second LTS release for Jenkins X 3.x.

LTS is a slower cadence version stream which contains a verified set of releases and configurations that have been used by teams tracking the bleeding edge Jenkins X.

Initially when we decided to maintain an LTS version stream we thought we’d aim for monthly releases however this second release comes two months after the first. This has given us more chances to run fixes and chart upgrades on Jenkins X own infrastructure to verify stability.

Traces for your pipelines

Thu, 08 Apr 2021 00:00:00 +0000

Now that Jenkins X has solid integration with Grafana for its observability, it’s time to start building fun things!

And the first one is tracing for all your pipelines:

With it, you can easily see the timings of all your pipelines, stages, and steps. This is great to inspect a “slow” pipeline and quickly see the slower steps.

We are using OpenTelemetry to generate a “logical” view of the pipeline, with 1 trace per pipeline and 1 span for each stage and step.

Jenkins X v3: now with built-in observability

Thu, 01 Apr 2021 00:00:00 +0000

As a Continuous Delivery platform, Jenkins X has a central part in your infrastructure. If it becomes unstable or unusable, it will impact the whole software delivery of your organization.

This is why observability is a critical topic for Jenkins X, and work has started to get observability built-in for Jenkins X v3:

Platform Observability: visualize logs and metrics for everything running in the Kubernetes cluster: Jenkins X’s own components - Tekton, Lighthouse, cert-manager, … - but also your own applications, that will be deployed either in preview environments or in the staging/prod environments.
Continuous Delivery Indicators: visualize pull requests, pipelines, releases, and deployments metrics, collected from cluster events and git events.

We’re using Grafana as the central visualization component: the main entry point from which you can get a complete overview of both your application’s lifecycle - development, build, tests, releases, deployments, runtime - and your Continuous Delivery platform.

Scaling Preview Environments with Osiris

Thu, 01 Apr 2021 00:00:00 +0000

One of Jenkins X’s core features is the preview environments: temporary environments created automatically for each Pull Requests, to deploy your application and its dependencies. You can then use this preview environment to run integration tests, or manually use/test your application.

This is all great until you have more and more applications, each with a few dependencies (postgresql, mongodb, …) and a few opened pull requests at any time. This means that you’ll get more and more pods running in your Kubernetes cluster, in addition to Jenkins X’s own components, your build pipelines, and of course your staging and production applications - unless you are using multi-cluster. The result is that you’ll need more nodes or bigger nodes. Which means more money.

cdCon 2021 - Call for Jenkins X Proposals

Thu, 25 Feb 2021 00:00:00 +0000

Hear ye! Hear ye! Jenkins X Community,

cdCon 2021 (the Continuous Delivery Foundation’s annual flagship event) is happening June 23-24 and its call for papers is open!

This is your chance to share what you’ve been doing with Jenkins X. Are you building something cool? Using it to solve real-world problems? Are you making things fast? Secure? Or maybe you’re a contributor and want to share what’s new. In all cases, we want to hear from you!

GitOps your cloud native pipelines

Thu, 25 Feb 2021 00:00:00 +0000

Tekton pipelines are cloud native and are designed from the ground up for kubernetes and the cloud:

there’s no single point of failure and the pipelines are elastically scalable
each pipeline is completely declarative and self defined
each pipeline executes independently of any others
pipelines are orchestrated via the sophisticated kubernetes scheduler:
- can use pipeline specific metadata for resource limits and node selectors: memory, CPU, machine type (GPU, windows/macOS/linux etc)
its easy to associate pipelines with Cloud IAM roles to avoid you having to upload cluster admin secrets to your public CI service which really helps security and helps reduce accidental bitcoin mining on your cloud account

In a previous blog we talked about how you can accelerate your use of tekton with Jenkins X.

Jenkins X 3 - February 2021 LTS

Mon, 01 Feb 2021 00:00:00 +0000

This is the first LTS release for Jenkins X 3.x. We are still in the Beta release and the leadup to GA includes ensuring the process for LTS monthly releases is validated and working well. This first releases focuses on:

community feedback following the Beta release
general helm chart upgrades
improved developer UX when editing Tekton pipelines here
support for in-repo and shared Tekton pipeline libraries including git URI support, e.g.

uses:jenkins-x/jx3-pipeline-catalog/packs/javascript/.lighthouse/jenkins-x/pullrequest.yaml@v1.2.3`

more documentation and examples can be found here

Jenkins X 3.x walkthroughs

Tue, 26 Jan 2021 00:00:00 +0000

Jenkins X 3.x is now looking ahead towards a GA release, with that we are producing walkthroughs for key areas to help users not only get started but get the most out of Jenkins X.

To kick this off we are going to start with 9 videos that we’ll follow up with more dedicated blogs over the coming weeks. The complete playlist can be found here however the blog below gives a more context for each one.

New features in the pipelines visualizer UI

Mon, 18 Jan 2021 00:00:00 +0000

The Jenkins X Pipelines Visualizer UI has recently received a number of new features, so let’s do a little tour of these new features!

Pipeline View

When viewing a pipeline, the biggest new feature is the collapsed logs. No more hundreds - or thousands - of log lines, we now group the logs per-container (step), which are collapsed by default. Along with the status of the step and its duration, so it’s easier to go to the interesting part of the logs.

Jenkins X 3.x beta is here!

Wed, 09 Dec 2020 00:00:00 +0000

I’m super excited to announce the 3.0 beta of Jenkins X! Christmas has come early this year!

the main documentation of the changes are:

the new architecture with modular plugins and improved extension points
what has changed since 3.x started
how 3.x compares to 2.x

but here’s a brief summary of the differences:

User Changes

As a user the high level UX of Jenkins X is similar:

automated Continuous Delivery pipelines for using tekton for your repositories with automatic promotion between your environments
pull requests on your repositories create separate Preview Environments where your team can review your changes and give fast feedback before your changes are approved and merged into the main trunk.

New features

we now default to vanilla tekton YAML for defining pipelines while accelerating your tekton with tekton catalog
we include an open source dashboard for visualising pipelines and logs which you can invoke via:

jx dash

Platform Changes

we now use helm (3.x) and helmfile along with optionally kustomize in a GitOps style to define and configure both Jenkins X itself, your tools and applications in any namespace
support multi cluster out of the box so you can keep Staging and Production in separate clusters to your development cluster where your pipelines run, you create and release immutable container images and other artifacts.
to setup or upgrade Jenkins X we use terraform to setup your cloud resources on Azure, Amazon or Google while also supporting on-premises, minkube and OpenShift - see the Admin Guides for more detail
- the actual installation of kubernetes resources takes place using the git operator so it runs reliably inside the cluster itself
we default to using Kubernetes External Secrets to manage all secrets for Jenkins X itself, development tools and your applications too.
- This means we can support various secret backends such as Alibaba Cloud KMS Secret Manager, Amazon Secret Manager, Azure Key Vault, Hashicorp Vault or GCP Secret Manager
- It also means we can then check in all kubernetes resources and custom resources directly into git (apart from Kubernetes Secrets) so that it super easy to version, review and reason about your kubernetes resources in a GitOps way.
built in TLS and DNS support along with Heath reporting and visualising via kuberhealthy
we now have an LTS distribution which lets you switch to a much more slower cadence of releases of Jenkins X

We have been using Jenkins X 3.x in production now for many months (for CI/CD of all of the 3.x codebase and continuously upgrading our cluster in the standard way and it’s been much simpler and easier to use, operate and configure.

Jenkins X 3.x - beta is close!

Fri, 04 Dec 2020 00:00:00 +0000

It has been ‘all hands on deck’ in recent months with the focus on Jenkins X 3 alpha. First off a huge thankyou to everyone involved. The OSS community spirit has really shone through what has been a very difficult year for everyone. Knowing that people from all over the world come and help each other, share banter and work at all hours of the day to help build out a true open source cloud native continuous delivery solution for developers - it’s quite fantastic to see and amazing to be apart of.

Accelerate your Tekton with Jenkins X

Wed, 11 Nov 2020 00:00:00 +0000

WARNING: This post refers to the unmaintained Jenkins X version 1 or 2

This post is outdated and is kept only out of historical interest. Some broken links have been removed.

One of the goals of Jenkins X has always been to help accelerate and automate Continuous Delivery so that developers can focus on delivering value to their customers; either by creating that new microservice or adding features to an existing project and not writing and managing pipelines.

Hacktoberfest

Wed, 23 Sep 2020 00:00:00 +0000

We are excited to announce that Jenkins X will be participating in Hacktoberfest again this year! Hacktoberfest is a month-long global celebration of open source software.

From October 1 to October 31, submit four pull requests to qualify for the limited edition Hacktoberfest shirt. All backgrounds and skill levels are encouraged to participate in Hacktoberfest and join a global community of open source contributors.

Learn more about Hacktoberfest and sign up here.

New UI to visualize your pipelines and logs

Wed, 23 Sep 2020 00:00:00 +0000

Welcome to the Jenkins X Pipelines Visualizer: a new open-source read-only UI for Jenkins X, with a very specific goal and scope: visualize the pipelines and logs.

This project was started at Dailymotion and quickly shared with the Jenkins X community.

Why a new UI?

There is already the Octant-based UI, so why a new UI?

The main reason is that Octant “is an application and is intended as a single client tool and at this time there are no plans to support hosted versions of Octant” - see this thread on the Octant github repository for more information and details.

Jenkins X Talks at CDCon

Wed, 16 Sep 2020 00:00:00 +0000

There will be six Jenkins X talks given by various speakers at the first-ever CDCon on October 7-8. The two-day virtual event, hosted by the Continuous Delivery Foundation, will focus on improving the world’s capacity to deliver software with security and speed.

Wednesday, October 7

A CI/CD Framework for Production Machine Learning at Massive Scale (using Jenkins X and Seldon Core) Speaker: Alejandro Saucedo, Seldon Time: 1:15 PM PDT

Welcome to Jenkins X 3.x alpha!

Wed, 16 Sep 2020 00:00:00 +0000

We are very pleased to announce the alpha release of Jenkins X version 3

You can read more about the overview of the architecture and components here.

This release has lots of benefits over version 2 is much easier to use, understand and manage. It is more flexible and simpler to configure for different infrastructures and cloud providers.

The new release works well with any combination of helm 3, helmfile, kpt and/or kustomize.

Octant: the OSS UI for Jenkins X

Thu, 06 Aug 2020 00:00:00 +0000

A common question we have heard in the community over the years is Is there an open source UI for Jenkins X?.

Well we now have an answer: its Octant using the octant-jx plugin.

Why Octant?

We love Octant because:

open source and very easy to extend with plugins in Go or TypeScript/JavaScript
lets you visualise and work with all kubernetes and custom resources across multiple clusters
thanks to octant-jx has awesome integration with Jenkins X components like apps, environments, pipelines, repositories etc.

Features

Longer term we’re planning on making most of the developer and operations features of Jenkins X available through the UI via octant-jx.

CJXD to upstream Jenkins X to the future

Wed, 29 Jul 2020 00:00:00 +0000

CJXD was first introduced at the end of July 2019 with the aim to provide a more stable version of Jenkins X that was under heavy development. The release cadence was reduced to one per month, the upgrade process well tested and a large focus on stability issues was taken on. It is fair to say that this was not only expected but absolutely needed by users of Jenkins X to service their own software delivery. A little over a year on we are now in a better place, following the improvements above means we are now able to focus all attention back towards the upstream Jenkins X and look towards where Jenkins X needs to go next. There is still much to do, areas which require attention in design, upgrades related to other projects and innovation to explore. With that, the need for CJXD over and above using upstream Jenkins X is reduced. CloudBees will stop releasing CJXD builds and focus purely upstream and it’s engineering efforts towards working with the OSS community on JX3 Alpha, Beta, GA and beyond.

Consolidate the use of Apps / Addons - Coding Phase 2

Mon, 27 Jul 2020 00:00:00 +0000

The coding phase 2 of Google Summer of Code will end this week, so I’d like to share my coding phase 2 work.

The addons have been migrated to apps on coding phase 1, but these apps have not been hosted to OSS, so during the coding phase 2, Jenkins X team build OSS cluster to host my apps:

you can use jx add app command to install the above apps, I also made a number of PR:

Consolidate the use of Apps / Addons - Coding Phase 1

Sun, 12 Jul 2020 00:00:00 +0000

The coding phase 1 of Google Summer of Code ended last week, I’m working on Consolidate the use of Apps / Addons, so I’d like to share my coding phase 1 work.

In the project, I mainly migrate addons to apps, the way is easier for improve jx apps and addons.

During the time, I’ve migrated all addons to apps via Helm chart way.

The following is my make apps:

Current, we only move to jx-app-kubeless to the jenkins-x-apps repository, other apps haven’t been moved to the jenkins-x-apps repository.

Jenkins X & Open Innovation

Wed, 08 Jul 2020 00:00:00 +0000

Jenkins X burst onto the CI/CD scene a couple of years ago - reimagining CI/CD on Kubernetes with pipeline automation, GitOps built-in and and preview environments to help teams collaborate and accelerate their software delivery at any scale. It has been evolving at a breakneck pace, keeping up with the changes in the wider Kubernetes ecosystem.

With the release of Helm 3, Jenkins X Labs nurtured an early experiment to adopt it. On May 15th, the primary sponsor of Jenkins X, CloudBees, shared a post talking about the future direction of Helm 3 and Jenkins X.

CloudBees contributions to Jenkins X - June 2020

Fri, 26 Jun 2020 00:00:00 +0000

For everyone here at Cloudbees, June has been a really productive month and the release 11 of CJXD comes with a set of new available features and fixes for upstream Jenkins X, as well as a new version of the UI that we are really excited to release!

Regarding Jenkins X new features:

Thanks to Hardy Ferentschik, you can now bring your own Vault to your Jenkins X cluster. It will allow you to use an already existing Vault instance to store your Jenkins X secrets, checkout the documentation here. He also improved the existing documentation to help you access and manage your secrets.

My story with Jenkins X GSoC - Consolidate the use of Apps / Addons

Thu, 18 Jun 2020 22:55:25 +0800

I’m Zixuan, and I’m very happy to be recognized by the Jenkins and Jenkins X teams in Google Summer of Code 2020.

I’m working on Consolidate the use of Apps / Addons.

Excellent mentor team

Jenkins and Jenkins X has an excellent team of mentors, everyone is very kind and super knowledgeable.

James Strachan
Kara de la Marck
markyjackson
Nikhil Da Rocha
Oleg Nenashev
Oscar Medina
Sahil Kalra
Sladyn

Work output

I have been working on this project for some time. During this time, I have migrated some addons to apps.

Lighthouse Goes GA!

Mon, 08 Jun 2020 00:00:00 +0000

In May of this year, Jenkins X switched from using Prow as its default webhook handler to using Lighthouse. This is the result of almost a year of work, and provides some significant improvements for users of Jenkins X. You can find more information on the differences between Prow and Lighthouse here.

Most notably, Jenkins X now properly supports GitHub Enterprise as well as github.com, with preview support for GitLab and BitBucket Server. Prow only supports github.com, so until now, Jenkins X users have had to jump through hoops or accept significant limitations to their workflows in order to use any other SCM (source control management) providers. Lighthouse is tested against every provider we list support for, including the preview support for GitLab and BitBucket Server. While some functionality may not behave exactly the same on all providers, the core functionality of ChatOps should work on all. Lighthouse also has a smaller footprint in terms of resources and pods used in your Kubernetes cluster than Prow.

An update on helm 3 and JXL

Fri, 15 May 2020 00:00:00 +0000

What’s next for JXL and Helm 3?

A few months back, we unveiled JXL to the Jenkins X community and framed it as a new thing, Jenkins X Labs, that would build out experimental features and test potential new things. Some would stick and go upstream into Jenkins X, others would be killed off as interesting but unsuccessful experiments.

This was a new thing for Jenkins X, trying to better separate the experimentation loop from the increasingly stable and relied-upon upstream. And, to a degree it worked great - the JXL workaround helm 3 was awesome, and got a lot of excitement.

CloudBees contributions to Jenkins X - April 2020

Wed, 29 Apr 2020 00:00:00 +0000

Despite the challenges of this particular time in the world it amazes me how members of the Jenkins X community continue to rise up and generously contribute to the project. This month’s CJXD release is with no lack of upstream community contributions, thus I’m kicking off this blog with a very sincere THANK YOU to the contributors.

A number of folks here at CloudBees continue to focus on Jenkins X stability and supportability. In parallel, other folks are focused on new features and expanding support for additional platforms. In an effort to help us continue these efforts, we completed the plan to remove code for Jenkins static masters in favor of going full-on with Tekton pipelines. In addition, a number of unsupported commands scheduled for deprecation have now been removed from the code.

Season of Docs 2020 📄

Tue, 28 Apr 2020 11:44:25 +0100

WARNING: This post refers to the unmaintained Jenkins X version 1 or 2

This post is outdated and is kept only out of historical interest. Some broken links have been removed.

Google Season of Docs fosters collaboration between open source projects and technical writers. This program brings the open source and technical writing communities together to improve documentation and increase awareness of open source. During the program, technical writers are given support in contributing to open source, information about the technical project to which they are contributing, and a stipend.

CloudBees contributions to Jenkins X - March 2020

Wed, 25 Mar 2020 00:00:00 +0000

March has been a really busy month for us working on Jenkins X, several exciting projects are now well underway.

Recently at office hours we discussed a proposal to improve how we provision the cloud resources (storage buckets, service accounts & cryptographic keys) that Jenkins X requires. The idea behind this enhancement is that some companies may require the infrastructure to be provisioned by a user (someone in operations perhaps) that requires a different set of permissions than the user that runs jx boot to install Jenkins X. Previously this happened when a user ran jx boot. But from community feedback we learned that some users would like this process split. To help with this we are in the process of developing Terraform modules that you can use to create the necessary resources. Work on this has been moving really quickly and we have been focusing on support for GKE, EKS & AKS. We are really excited to see the awesome contributions from Heba Elayoty of Microsoft on adding support for Azure Kubernetes Service in Jenkins X. If you do not use Terraform and you have another way of provisioning infrastructure don’t worry, we’ll be providing lots of information in the docs on exactly what you will need to create before booting Jenkins X. For anyone that would like more information or would like to contribute to the Terraform modules you will find the git repository for GKE here and EKS here.

Google Summer of Code 2020 ☀️

Mon, 23 Mar 2020 11:26:18 +0000

Jenkins X is incredibly happy to participate in Google Summer of Code 2020!

Jenkins X has been open source from its inception and many of the founders and core contributors to Jenkins X have been deeply involved in open source for years. We love open source for many reasons, including how it enables rapid innovation and collaboration among remote, distributed teams and communities. For these reasons and more, we are very happy to participate in Google Summer of Code.

Jenkins X ❤ Tekton

Wed, 11 Mar 2020 00:00:00 +0000

Jenkins X is committing fully to Tekton as its pipeline execution engine. We are convinced that this is the right choice for Jenkins X, as a cloud-native CI/CD platform on Kubernetes, and for our users.

This means we are formally deprecating - and will be removing - traditional Jenkins static masters support inside Jenkins X. We are excited about the new way forward, which we’ll discuss further below, but it’s important to be clear about what this means for current users. If you are already using Jenkins X with Tekton-based pipelines, then nothing will change for you, and you do not need to change anything. If you’re running a traditional Jenkinsfile on Jenkins X, then you have three choices:

Walkthrough: Setting up Jenkins X on a Kubernetes cluster

Tue, 10 Mar 2020 00:00:00 +0000

In this tutorial, we’ll walk you through setting up Jenkins X on a Kubernetes cluster hosted on GKE. We will use GitHub as our Git provider.

The purpose of this tutorial is to give a detailed step-by-step walkthrough of setting up Jenkins X.

Setup used in this tutorial

GitHub as our Git provider and a GitHub user account, eg MarckK
A Google Cloud Platform (GCP) account with the ability to provision kubernetes resources / create kubernetes clusters

Google Secrets Manager Kubernetes controller

Thu, 05 Mar 2020 00:00:00 +0000

In Jenkins X Labs we have been working with Google Secrets Manager (which is currently in Beta so may still change). It is an extremely nice experience and if you also use Google Container Engine (GKE), Google’s managed Kubernetes service then we can make use of another cool feature, Workload Identity to automatically access secrets in their hosted service using a Kubernetes service account.

One of the initial challenges we had was we didn’t see an obvious way to automatically create Kubernetes secrets that include the data stored from Secrets Manager. So we created a little standalone controller that runs in a Kubernetes cluster and watches for Kubernetes secrets. When one is added or updated with a particular annotation the controller will access the secret using its ID in Secrets Manager and update the Kubernetes secret with the value.

JayeX Office Hours - March 5th 2020

Thu, 05 Mar 2020 00:00:00 +0000

Announcements:

JayeX Labs

Demos:

jxl boot
k8s controller for Google Secret Manager

Jenkins X Labs

Fri, 28 Feb 2020 00:00:00 +0000

Jenkins X has entered it’s third year and as always we are looking for big ways things can be improved. Briefly reflecting on the last two years:

first was about initial adoption and finding out what developers were both looking for and needed when moving to Kubernetes. We started off using a Static Jenkins server for example to orchestrate builds and other open source projects that folks would be familiar with, to aid their journey to the cloud. We quickly realised that the Jenkins X community wanted to leverage more of their cloud capabilities and so we began to swap in cloud native implementations like Tekton and Prow, as well as better integration with managed cloud services like storage, dns, IAM etc.

CloudBees contributions to Jenkins X - February 2020

Wed, 26 Feb 2020 00:00:00 +0000

2020 is fully underway and Jenkins X is feeling the benefits from a lot of hard work from the tail end of last year. There is now support on EKS for Jenkins X! Here are the docs:

https://jayex.io/docs/getting-started/setup/boot/clouds/amazon/

They are still being improved so we’d love feedback in the Slack channel or issues (PRs would be even better ;)) on the website repository to help improve. By the way a Pull Request on the docs repo results in a preview environment of the website, so you can see your changes and ask for feedback in true Jenkins X form.

3 Jenkins X talks at FOSDEM 2020

Fri, 14 Feb 2020 00:00:00 +0000

FOSDEM 2020 was fantastic! We were thrilled to be there, answering questions about Jenkins X at the Jenkins X - CDF - Jenkins booth.

This was the first year there was a CI/CD devroom at FOSDEM. Among the excellent talks, three focused on Jenkins X in their demos:

Choosing The Right Deployment Strategy, by Viktor Farcic
Progressive Delivery: Continuous Delivery the Right Way, by Carlos Sanchez
A Practical CI/CD Framework for Machine Learning at Massive Scale, by Alejandro Saucedo

Choosing The Right Deployment Strategy

by Viktor Farcic

JayeX Office Hours - Jan 23rd 2020

Thu, 23 Jan 2020 00:00:00 +0000

Overview of jx boot apps proposal

Demo of early idea around converting a declaritive Jenkins pipeline used with JayeX to jenkins-x.yml

fun movie trailer style video for JayeX - Oscar Medina

Asking and Finding help - Outreachy

Wed, 22 Jan 2020 00:00:00 +0000

Neha Gupta is adding support for Kustomize in Jenkins X, to enable Kubernetes native configuration management, while participating in Outreachy from December 2019 to March 2020.

Outreachy open-source contribution for applicants — Asking/Finding help

This blog might be helpful for beginners who are fear-stricken or I would say hesitant to ASK, to get lost in the new world while trying to understand any open source project, fear of asking questions that may sound stupid later on or are very obvious! First of all.. Relax!

Happy Second Birthday Jenkins X!

Tue, 14 Jan 2020 00:00:00 +0000

The Jenkins X project started the beginning of 2019 by celebrating its first birthday on the 14th January, a big event for any open source project, and we have just celebrated our 2nd - hooray!

image by Ashley McNamara, creative commons license

Two Years of Jenkins X!

Jenkins X has evolved from a vision of how CI/CD could be reimagined in a cloud native world, to a fast-moving, innovative, rapidly maturing open source project.

Outreachy - Motivation to apply!

Fri, 10 Jan 2020 00:00:00 +0000

Neha Gupta is adding support for Kustomize in Jenkins X, to enable Kubernetes native configuration management, while participating in Outreachy from December 2019 to March 2020.

Motivation to apply to Outreachy :

In my graduation class of fifty people we were three girls struggling to set up our space and comfort with the weird reactions we got from fellow students for trying to understand technology. When my professor asked us to make an autonomous drone. I couldn’t make one, I was shattered, until a friend from computer science batch helped me make one. He showed me some of the cool apps he made, that sparked an interest, and I started building apps, realising that computer science is beyond just coding, it’s more about solving real life problems.

FOSDEM 2020

Thu, 09 Jan 2020 00:00:00 +0000

FOSDEM is a free event for software developers to meet, share ideas, and collaborate.

FOSDEM

FOSDEM (Free and Open source Software Developers’ European Meeting) is a non-commercial, volunteer-organized European event centered on free and open source software development. It is aimed at developers and anyone interested in the free and open source software movement. FOSDEM will take place on 1 and 2 of February in Brussels, Belgium. See the event site for more details.

Welcome Neha Gupta to Jenkins X as our Outreachy intern

Thu, 28 Nov 2019 00:00:00 +0000

With sponsorship from the CDF (Continuous Delivery Foundation), Jenkins X is participating in the Outreachy mentorship program. This is the first time that Jenkins X will be participating in Outreachy and we are fortunate to be working with such a phenomenal organisation. You can read more about our decision to participate in Outreachy here.

We’re excited to announce that an intern has been selected! Neha Gupta will be working with us to create a cloud storage backed Helm repository for Jenkins X to replace ChartMuseum.

Contribute to Jenkins X at our Hackathon in Lisbon on 2 December

Fri, 08 Nov 2019 00:00:00 +0000

We’re having a Jenkins X hackathon in Lisbon on December 2! The hackathon is free to attend and will be a joint hackathon with the Jenkins project.

Please note this hackathon takes place the day before the Jenkins X Contributor Summit, which is also free to attend and we would love to see you at both.

These events are being organised for the community in the days prior to DevOps World | Jenkins World Lisbon. We realised many Jenkins X and Jenkins community members will be attending the conference and wanted to create additional opportunities for community members to meet and collaborate.

Join Us at the Jenkins X Contributor Summit Lisbon on 3 December

Fri, 18 Oct 2019 00:00:00 +0000

Attendance at the Jenkins X Contributor Summit is free, and a DevOps World | Jenkins World conference pass is not necessary, but please RSVP if you are going to attend.

The Jenkins X Contributor Summit is an opportunity for current and future Jenkins X contributors to come together to discuss the project, learn and collaborate. There will be discussions on the latest exciting developments in Jenkins X and time to contribute to the project. We welcome new contributors: this is a great chance to learn more about the project, find out how to contribute and meet the core contributors!

Introduction to Jenkins X @ Jenkins World | DevOps World Lisbon

Wed, 16 Oct 2019 12:20:42 -0700

In today’s technology landscape, people find it difficult to get up to speed with technologies for many reasons. One reason that comes to mind is time. Another may be that some folks are just not able to read a tutorial or blog post and be successful at ramping up with a specific technology.

So these individuals need either a bootcamp or to sit and watch live demos and have someone explain concepts. It is for these reasons that Oscar and Andrew will be introducing Jenkins X to people that might have heard of it, but have not yet had a chance to play with it.

CFP: CI/CD Devroom at FOSDEM 2020

Wed, 09 Oct 2019 00:00:00 +0000

FOSDEM is a free event for software developers to meet, share ideas, and collaborate.

For the first time, there will be a Continuous Integration and Continuous Deployment (CI/CD) devroom at FOSDEM! The CI/CD devroom will take place on 2nd of February 2020 in Brussels, Belgium.

FOSDEM

FOSDEM (Free and Open source Software Developers’ European Meeting) is a non-commercial, volunteer-organized European event centered on free and open source software development.

It is aimed at developers and anyone interested in the free and open source software movement. FOSDEM will take place on 1st and 2nd of February in Brussels, Belgium. See the event site for more details.

Outreachy

Tue, 01 Oct 2019 00:00:00 +0000

Jenkins X is happy to announce our participation in the Outreachy program running from December 2019 to March 2020.

Outreachy helps people from groups underrepresented in free and open source software get involved by providing “a supportive community for beginning to contribute any time throughout the year and offer focused internship opportunities twice a year with a number of free software organizations.” The internships are for three-months, during which mentees work on specific projects.

Contribute to Jenkins X as part of Hacktoberfest

Fri, 27 Sep 2019 00:00:00 +0000

Hacktoberfest

Hacktoberfest is a month-long global celebration of open source software.

We’d love for you to contribute to the Jenkins X project!

All pull requests will count towards your Hacktoberfest challenge. Issues labelled “hacktoberfest” generally indicate good first issues. Jenkins X welcomes contributors to both:

Jenkins X Key Takeaways from Jenkins World 2019

Tue, 03 Sep 2019 00:00:00 +0000

Jenkins X was the star of the show at DevOps World | Jenkins World 2019. In this article I will share with you the dozen key things I learned about this exciting new cloud native CI/CD tool. Beyond its capabilities as a CI/CD tool, Jenkins X also provides an excellent example of how to architect a cloud native application on Kubernetes.

Jenkins X is a completely new CI/CD system that shares little but its name with the existing Jenkins. Jenkins X incorporates the best practices from the State of DevOps reports and the seminal book, Accelerate by Nicole Forsgren, Jez Humble and Gene Kim.

A New Logo for Jenkins X

Wed, 31 Jul 2019 07:36:00 +0200

Back in March 2018, the Jenkins X project burst onto the scene as the Jenkins counterpart for automated CI/CD for Kubernetes. As part of that launch featured the logo: a variation of the Jenkins logo, featuring a pipe-smoking ship captain with Kubernetes logo on his cap.

In software, we like to say that naming is hard - because it is. Another thing that is also hard is trying to capture the essence of a project in a logo. Logos pack a lot of meaning into a small space. Icons, such as the Jenkins logo, establish a strong emotional connection with many developers. So with that in mind we always listened closely to feedback on the new logo and how people perceive the project as a result of it.

Increasing CI/CD Pipeline Observability in Jenkins X

Mon, 29 Jul 2019 10:44:40 -0700

Increasing CI/CD Pipeline Observability, implement tracing

Credit: [Minds Eye Creative](https://www.mindseyecreative.ca/) | [DevOps Days Toronto](https://devopsdays.org/events/2018-toronto/program/clay-smith/)

Overview

You might have heard of Observability given that folks have been talking about this for a while now. Sure, you might think it is just the latest tech buzzword. However, the practice has been around for a long time now.

Observability is certainly relevant today given the Microservices architectures, distributed systems, and the characteristics of modern applications being deployed at a faster pace by leveraging CI/CD pipelines to Kubernetes, in this case using Jenkins X. Indeed old practices of setting up monitoring after an app is deployed, are no longer acceptable.

JayeX Office Hours - July 25, 2019

Thu, 25 Jul 2019 00:00:00 +0000

Announcements:

New JayeX Logo!! - James Rawlings

Demos:

meta pipeline - Hardy Ferentschik
BDD Tests report generation - Daniel Gozalo

Bonus:

fun movie trailer style video for JayeX - Oscar Medina

Using Jenkins X To Define And Run Serverless Deployments

Tue, 23 Jul 2019 12:20:42 -0200

Using Jenkins X To Define And Run Serverless Deployments

What is Serverless Computing?

To understand serverless computing, one needs to understand the challenges we are facing with more “traditional” types of deployments of our applications. A long time ago, most of us were deploying our apps directly to servers. We had to decide the size (memory and CPU) of the nodes where our applications would run, we had to create those servers, and we had to maintain them. The situation improved with the emergence of cloud computing. We still had to do all those things, but now those tasks were much easier due to the simplicity of the APIs and services cloud vendors gave us. Suddenly, we had (a perception of) infinite resources and all we had to do is run a command, and a few minutes later the servers (VMs) we needed would materialize. Things become much easier and faster. But, that did not remove the tasks of creating and maintaining servers. Instead, that made them more straightforward. Concepts like immutability become mainstream as well. As a result, we got much-needed reliability, reduced drastically lean time, and started to rip benefits of elasticity.

DevOps World - Jenkins World 2019 San Francisco: Lunch Time Demos

Thu, 18 Jul 2019 12:20:42 -0200

If you’re looking for more opportunities to learn Jenkins and Jenkins X during the lunch hours while at DevOps World | Jenkins World 2019 San Francisco, come join us at the Jenkins and Jenkins X Community Booth!

If you don’t yet have your pass for DevOps World | Jenkins World 2019 San Francisco, and don’t want to miss out on the fun, you can get yours using JWFOSS for a 30% discount.

Join Us at the Jenkins X Contributor Summit San Francisco, Monday August 12, 2019

Tue, 16 Jul 2019 10:20:42 -0200

Register here to participate in the Jenkins X Contributor Summit in San Francisco. Attendance at the Jenkins X Contributor Summit is free, and a DevOps World | Jenkins World conference pass is not necessary, but please RSVP if you are going to attend.

JayeX Office Hours - June 27, 2019

Thu, 27 Jun 2019 00:00:00 +0000

WARNING: This post refers to the unmaintained Jenkins X version 1 or 2

This post is outdated and is kept only out of historical interest. Some broken links have been removed.

Demos:

pipeline scheduler configuration - Dave Conde and Pete Muir
Jenkins X Boot - James Strachan

Announcement:

Jenkins X Contributor Summit + Jenkins X at DevOps World | Jenkins World - Andrew Bayer and Tracy Miranda
- Contributor Summit is a free event before DevOps World | Jenkins World
- Register your interest in speaking, demoing or attending

Installing Jenkins X on Alibaba Cloud Container Service

Mon, 17 Jun 2019 07:36:00 +0200

Installing Jenkins X on Alibaba Cloud Container Service

Installing Jenkins X in Alibaba Kubernetes Container Service

Jenkins X has just added initial support to install in Alibaba Container Service, its Kubernetes offering.

The following instructions allow installation in a managed Kubernetes cluster in any region outside of mainland China, where more configuration is needed avoid using Google services blocked by chinese authorities (Docker images in GCR). There is a pending issue for that.

DevOps World-Jenkins World 2019 San Francisco: Agenda is Live

Thu, 06 Jun 2019 12:20:42 -0700

DevOps World | Jenkins World is the largest Jenkins gathering of the year. From August 12 - 15, 2019 the event will take place at the Moscone West Center in San Francisco. The event boasts 100+ sessions, and will offer training, hands-on workshops, onsite certification, contributor summit and much more. Conference attendees can expect to be inspired while learning the latest innovations from industry leaders. Attendees will learn the value that digital transformation has in delivering software more efficiently, more quickly and with higher quality.

JayeX Office Hours - May 30, 2019

Thu, 30 May 2019 00:00:00 +0000

Agenda

JayeX and machine learning - Terry Cox
walkthrough of new code coverage

JayeX Office Hours - May 3, 2019

Fri, 03 May 2019 00:00:00 +0000

Agenda

bucketrepo as a lightweight replacement for Nexus (https://github.com/jenkins-x/bucketrepo) - Cosmin Cojocar
latest JayeX Apps demo - Pete Muir https://www.youtube.com/watch?v=MqdMynweoUU&feature=youtu.be and https://www.youtube.com/watch?v=csRftjH1uXI&feature=youtu.be

JayeX Office Hours - April 19, 2019

Fri, 19 Apr 2019 00:00:00 +0000

WARNING: This post refers to the unmaintained Jenkins X version 1 or 2

This post is outdated and is kept only out of historical interest. Some broken links have been removed.

Agenda

Update on multi cluster - James Strachan
Update on Dogfooding JayeX - James Rawlings
Update on monitoring - Gareth Evans
Update on external dns - Cai Cooper
Update on multiple git providers for Prow - Will Refvem

Demos:

Multi-cluster - James Strachan
Installing apps using GitOps, with private chart repos - Pete Muir
Creating optional questions for app install - Dave Conde
Jx upgrade app - Pete Muir

Jenkins Community Awards

Wed, 17 Apr 2019 12:20:42 -0700

It is time for the 3rd annual, DevOps World | Jenkins World 2019 Community Awards. These nominations are indicative of the excellent work Jenkins community members are doing for the betterment of Jenkins.

This will be the third year we are commemorating community members who have shown excellence through commitment, creative thinking, and contributions to continue making Jenkins a great open source community.

The award categories include:

Most Valuable Contributor - This award is presented to the Jenkins contributor who has helped move the Jenkins project forward the most through their invaluable feature contributions, bug fixes or plugin development efforts.
Jenkins Security MVP - This award is presented to the individual most consistently providing excellent security reports or who helped secure Jenkins by fixing security issues.
Most Valuable Advocate - This award is presented to an individual who has helped advocate for Jenkins through the organization of their local Jenkins Area Meetup.
Most Innovative Jenkins X Implementation - Presented to an innovative user who has embraced Jenkins X.

Nominate yourself or someone today! Winners will be announced at DevOps World | Jenkins World San Francisco 2019 on August 12-15. The deadline for nomination is Tuesday, April 30th.

JayeX Office Hours - April 4, 2019

Thu, 04 Apr 2019 00:00:00 +0000

Agenda

Rebasing & Batching pull requests - Pete Muir & James Rawlings
Provisioning JayeX K8s Clusters Using Infrastructure as Code (Terraform) - Oscar Medina
Knative Server - James Strachan
Multi-select questions for jx add app - Pete Muir
jx add app against local filesystem - Pete Muir

JayeX Office Hours - March 21, 2019

Thu, 21 Mar 2019 00:00:00 +0000

Agenda

Overview and demo of DevEnv - Cai Cooper
CLI plugins - Pete Muir
CloudBees Docs Team (Ashton, Kim, Tammy, John) - want to know how they can help the community with JayeX docs :-)

Jenkins X joins the CDF

Sat, 09 Mar 2019 07:36:00 +0200

We are thrilled to announce that Jenkins X will be joining the Continuous Delivery Foundation as one of the founding projects. The Continuous Delivery Foundation (CDF) is a brand new sub-foundation of the Linux Foundation and will be dedicated to advancing the practice of continuous delivery and nurturing an ecosystem of interoperable tools for software delivery.

Jenkins X is just over a year old but has been growing rapidly as the CI/CD solution for modern cloud applications on Kubernetes. Jenkins X automates CI+CD for Kubernetes using the best of breed OSS tools such as Jenkins, Tekton, Prow, Skaffold, Kaniko and Helm. The CDF will be a sibling foundation to the Cloud Native Computing Foundation (CNCF) which hosts Kubernetes amongst others. CDF will have its first event, CDF Summit, on May 20th alongside KubeCon Barcelona. We always love to work closely with other communities, and this will continue at scale within the CDF.

JayeX Office Hours - February 21, 2019

Thu, 21 Feb 2019 00:00:00 +0000

WARNING: This post refers to the unmaintained Jenkins X version 1 or 2

This post is outdated and is kept only out of historical interest. Some broken links have been removed.

Agenda

Jenkins X Pipelines - Andrew Bayer
Container Tests - Gareth Evans

Jenkins X Pipelines integrated with Tekton

Tue, 19 Feb 2019 10:36:00 +0200

WARNING: This post refers to the unmaintained Jenkins X version 1 or 2

This post is outdated and is kept only out of historical interest. Some broken links have been removed.

Introducing the new Jenkins X Pipeline Engine

The Jenkins X team and CloudBees are excited to announce some changes that we’ve been working on and are jntroducing the new Jenkins X Pipeline Engine.

Today, there are two ways that you can run your Jenkins pipelines in Jenkins X.

Happy 1st Birthday Jenkins X

Mon, 14 Jan 2019 07:36:00 +0200

Time flies when you are delivering software. It has been just over a year since the initial commit went into Github for Jenkins X, the CI/CD solution for cloud native. In just 12 months, Jenkins X progress has been breathtaking, so let’s celebrate by taking a look at all that has been accomplished in this short space of time.

One of the main founders of Jenkins X is James Strachan of CloudBees, who is known for building many popular open source software projects such as Apache Groovy, Apache Camel and fabric8. Notable these projects also have enduring open source software communities around them. With all this experience it is safe to say James simply gets developers and what they want, need and care about.

Serverless Jenkins

Sat, 20 Oct 2018 07:36:00 +0200

WARNING: This post refers to the unmaintained Jenkins X version 1 or 2

This post is outdated and is kept only out of historical interest. Some broken links have been removed.

Serverless Jenkins

James Rawlings has just published an excellent article on Serverless Jenkins with Jenkins X that is well worth a read.

It describes how you can use it via the --prow feature flag when using either jx create cluster

Helm 2.x without Tiller

Wed, 03 Oct 2018 07:36:00 +0200

WARNING: This post refers to the unmaintained Jenkins X version 1 or 2

This post is outdated and is kept only out of historical interest. Some broken links have been removed.

How to avoid Tiller

Helm 2.x uses a server side component called tiller. Typically Tiller is installed in a global namespace; or you can install a Tiller in each namespace and configure your helm CLI to talk to the right tiller in the right namespace.

Jenkins X at Jenkins World

Thu, 06 Sep 2018 08:36:00 +0200

Jenkins X at JenkinsWorld

Jenkins World has combined with DevOps world to become the premier DevOps event in your calendar this year - and there really isn’t much time left if you still have to register. The event is happening at San Francisco 16-19 September, and Nice,France 22-25 October. To get a discount on entry - you are invited to use the dscount code: JWRDAVIES

Some of the Jenkins X team, including James Strachan, James Rawlings, Rob Davies and Pete Muir will be available if you want some extra help with the Jenkins X project - we are cheap too - beer tokens gratefully received.

Jenkins X does KubeCon

Sat, 05 May 2018 10:36:00 +0200

Jenkins X at KubeCon

Here is the video of my talk: Jenkins X: Easy CI/CD for Kubernetes at KubeCon Europe 2018.

You can also get the slides from this talk here

You can also watch all the other KubeCon videos here

You can also register your interest in the CloudBees Kubernetes product that I gave a quick sneak peek at towards the end of the talk.

Introducing Jenkins X

Mon, 19 Mar 2018 18:36:00 +0200

Introducing Jenkins X

There’s a new blog post on jenkins.io which introduces Jenkins X and gives an overview of what it, what we’re doing and why:

Introducing Jenkins X: a CI/CD solution for modern cloud applications on Kubernetes

Please check it out and let us know what you think

JEP 400 submitted

Fri, 09 Mar 2018 18:36:00 +0200

JEP 400 Submitted for Jenkins X

We have submitted JEP 400 to propose Jenkins X as a sub project in the Jenkins Foundation.

Please check out JEP 400 and let us know what you think on the Jenkins dev list.

Mon, 01 Jan 0001 00:00:00 +0000

Contributing Rich Event Data

To make upcoming JayeX events show up more prominently in Google search and map results, we mark up our event information in structured data.

More information here: https://developers.google.com/search/docs/data-types/event

Sample Event Data

Here is a multi-day event for DevOps World - Jenkins World with an offers URL for the registration page:

<div class="event-wrapper" itemscope itemtype="http://schema.org/Event">
  <meta itemprop="image" content="https://jayex.io/images/community/events/2019-DWJW-JAM_banner-600x338.jpg" />
  <meta itemprop="description" content="DevOps World - Jenkins World is the largest gathering of Jenkins users in the world, including Jenkins experts, continuous delivery practitioners, and companies offering complementary technologies to Jenkins." />
  <meta class="event-date" itemprop="startDate" content="2019-12-03" />
  <meta class="event-date" itemprop="endDate" content="2019-12-05" />
  <meta class="event-title" itemprop="name" content="DevOps World - Jenkins World" />
  <div class="event-venue" itemprop="location" itemscope itemtype="http://schema.org/Place">
      <meta itemprop="name" content="Lisbon Congress Center" />
      <div class="address" itemprop="address" itemscope itemtype="http://schema.org/PostalAddress">
          <meta itemprop="streetAddress" content="Praça das Indústrias 1" />
          <meta itemprop="postalCode" content="1300-307" />
          <meta itemprop="addressLocality" content="Lisboa" />
          <meta itemprop="addressCountry" content="Portugal" />
      </div>
  </div>
  <div itemprop="offers" itemscope itemtype="http://schema.org/Offer">
    <meta itemprop="priceCurrency" content="EUR" />
    <meta itemprop="url" content="https://www.cloudbees.com/devops-world" />
  </div>
</div>

Here is a one-day event with start and end times. Google assumes the timezone of the provided location, so if the event is online be sure to add timezone info like 2020-02-02T09:00:00+02:00 (for Brussels UTC+2) if no location is specified. Here we use isAccessibleForFree to denote a free event.