Flux – User Guides

Flux: Ways of structuring your repositories

Mon, 01 Jan 0001 00:00:00 +0000

This guide walks you through several approaches of organizing repositories for a smooth GitOps experience with Flux.

Monorepo

In a monorepo approach you would store all your Kubernetes manifests in a single Git repository. The various environments specific configs are all stored in the same branch (e.g. main).

Repository structure

Example structure (kustomize overlays):

├── apps
│ ├── base
│ ├── production
│ └── staging
├── infrastructure
│ ├── base
│ ├── production
│ └── staging
└── clusters
 ├── production
 └── staging

Each cluster state is defined in a dedicated dir e.g. clusters/production where the specific apps and infrastructure overlays are referenced.

The separation between apps and infrastructure makes it possible to define the order in which a cluster is reconciled, e.g. first the cluster addons and other Kubernetes controllers, then the applications.

A complete example of this approach can be found at flux2-kustomize-helm-example.

Delivery management

In trunk-based development, the changes are made in small batches and are merged into the main branch often. Besides main, branches are short-lived, once a pull request is merged, the branch gets deleted.

New app release can be automatically delivered to staging using Flux’s image updates to Git. For production, you may choose to manually approve app version bumps by configuring Flux to push the changes to a new branch from which you can create a pull request. You can limit the impact of an issue that escaped staging testing by using Flagger’s canary releases.

Changes to infrastructure can be disruptive and could cause cluster-wide outages. These config changes can be made in steps, first you merge a change to staging, when the cluster is successfully reconciled, and the new cluster state passes conformance tests, you then promote the change to production. The promotion process is gated by PR reviews and end-to-end testing.

Repo per environment

This approach is similar to the monorepo, with some notable differences:

In the monorepo approach, all team members can read the production config since Git is not designed to restrict access to certain files in a repository. Access control can be provided by tools like GitHub or GitLab, as they provide additional layers of security and permission management on top of Git. Having a separate repository for production means that you can grant access to a subset of team members while allowing everyone to clone staging and open pull requests.
Promoting changes from one environment to another can be more time-consuming especially for infrastructure changes that can’t be automated with Flux image updates.
When using the same repository for all environments, unintentional changes to production are harder to spot, especially for large pull requests. Having a dedicated production repository, limits the scope of changes and makes the review process less error prone.

Repo per team

Assuming your organization has a dedicated platform admin team that provides Kubernetes as-a-service for other teams.

The platform admin team is responsible for:

Setting up the staging and production environments.
Maintains the cluster addon-ons and other cluster-wide resources (CRDs, controllers, admission webhooks, etc).
Onboards the dev teams repositories using Flux’s GitRepository custom resources.
Configures how the dev teams repositories are reconciled on each cluster using Flux’s Kustomization custom resources.

The dev teams are responsible for:

Setting up the apps definitions (Kubernetes deployments, Helm releases).
Configures how the apps are reconciled on each environment (Kustomize overlays, Helm values).
Manages the apps promotion between environments using Flux’s automated image updates to Git.

Repository structure

Platform admin repository example (kustomize overlays):

├── teams
│ ├── team1
│ ├── team2
├── infrastructure
│ ├── base
│ ├── production
│ └── staging
└── clusters
 ├── production
 └── staging

Dev team repository example (kustomize overlays):

└── apps
 ├── base
 ├── production
 └── staging

A complete example of this approach can be found at flux2-multi-tenancy.

Delivery

The delivery process is similar to the monorepo one. The main difference is the separation of concerns, the platform admin team handles the change management of the infrastructure, but delegates the apps delivery to the dev teams.

Repo per app

It is common to use the same repository to store both the application source code and its deployment manifests. The deployment manifests from an app repo can serve as the base config for both the monorepo and the repo-per-team approaches.

Instead of duplicating the deployment manifests between the app repo and the cluster(s) config repo, the config repo can hold a pointer to the app manifests.

Inside the config repo you can define a GitRepository that tells Flux to clone the app repo inside the cluster, then with a Kustomization, you can tell Flux which directory holds the app manifests and how to patch them based on the target environment.

Another option is to bundle the app manifests into a Helm chart and publish it to a Helm repository. In the config repo you can define the HelmRepository and create an app HelmRelease for each target environment.

Repository structure

App repository plain Kubernetes manifests example:

├── src
└── deploy
 └── manifests

Delivery example (stored in config repo):

apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
 name: app
spec:
 url: https://<host>/<org>/app
 ref:
 semver: "1.x"
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
 name: app
spec:
 sourceRef:
 kind: GitRepository
 name: app
 path: ./deploy/manifests
 patches:
 - patch: |
 apiVersion: apps/v1
 kind: Deployment
 metadata:
 name: app
 spec:
 replicas: 2
 target:
 kind: Deployment
 name: app

App repository Kustomize overlays example:

├── src
└── deploy
 ├── base
 ├── production
 └── staging

Delivery example (stored in config repo):

apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
 name: app
spec:
 url: https://<host>/<org>/app
 ref:
 branch: main
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
 name: app
spec:
 sourceRef:
 kind: GitRepository
 name: app
 path: ./deploy/production

App repository Helm chart example:

├── src
└── chart
 ├── templates
 ├── values.yaml
 └── values-prod.yaml

Delivery example (stored in config repo):

apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
 name: apps
spec:
 url: https://<host>/<org>/charts
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
 name: app
spec:
 chart:
 spec:
 chart: app
 version: "1.x"
 sourceRef:
 kind: HelmRepository
 name: apps
 values:
 replicas: 2

Flux: Manage Helm Releases

Mon, 01 Jan 0001 00:00:00 +0000

The helm-controller allows you to declaratively manage Helm chart releases with Kubernetes manifests. It makes use of the artifacts produced by the source-controller from HelmRepository, GitRepository, Bucket and HelmChart resources. The helm-controller is part of the default toolkit installation.

Prerequisites

To follow this guide you’ll need a Kubernetes cluster with the GitOps toolkit controllers installed on it. Please see the get started guide or the installation guide.

Define a chart source

To be able to release a Helm chart, the source that contains the chart (either a HelmRepository, GitRepository, or Bucket) has to be known first to the source-controller, so that the HelmRelease can reference to it.

Helm repository

Helm repositories are the recommended source to retrieve Helm charts from, as they are lightweight in processing and make it possible to configure a semantic version selector for the chart version that should be released.

They can be declared by creating a HelmRepository resource.

Helm HTTP/S repository

The source-controller will fetch the Helm repository index for this resource on an interval and expose it as an artifact:

apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
 name: podinfo
 namespace: default
spec:
 interval: 1m
 url: https://stefanprodan.github.io/podinfo

The interval defines at which interval the Helm repository index is fetched, and should be at least 1m. Setting this to a higher value means newer chart versions will be detected at a slower pace, a push-based fetch can be introduced using webhook receivers

The url can be any HTTP/S Helm repository URL.

Authentication

HTTP/S basic and TLS authentication can be configured for private Helm repositories. See the HelmRepository CRD docs for more details.

Helm repository authentication with credentials

In order to use a private Helm repository, you may need to provide the credentials.

For HTTP/S repositories, the credentials can be provided as a secret reference with basic authentication.

apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
 name: podinfo
 namespace: default
spec:
 interval: 1m
 url: https://stefanprodan.github.io/podinfo
 secretRef:
 name: example-user
---
apiVersion: v1
kind: Secret
metadata:
 name: example-user
 namespace: default
stringData:
 username: example
 password: "123456"

Git repository

Charts from Git repositories can be released by declaring a GitRepository, the source-controller will fetch the contents of the repository on an interval and expose it as an artifact.

The source-controller can build and expose Helm charts as artifacts from the contents of the GitRepository artifact (more about this later on in the guide).

An example GitRepository:

apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
 name: podinfo
 namespace: flux-system
spec:
 interval: 1m
 url: https://github.com/stefanprodan/podinfo
 ref:
 branch: master
 ignore: |
 # exclude all
 /*
 # include charts directory
 !/charts/

The interval defines at which interval the Git repository contents are fetched, and should be at least 1m. Setting this to a higher value means newer chart versions will be detected at a slower pace, a push-based fetch can be introduced using webhook receivers

The url can be any HTTP/S or SSH address (the latter requiring authentication).

The ref defines the checkout strategy, and is set to follow the master branch in the above example. For other strategies like tags or commits, see the GitRepository CRD docs.

The ignore defines file and folder exclusion for the artifact produced, and follows the .gitignore pattern format. The above example only includes the charts directory of the repository and omits all other files.

Authentication

HTTP/S basic and SSH authentication can be configured for private Git repositories. See the GitRepository CRD docs for more details.

Cloud Storage

It is inadvisable while still possible to use a Bucket as a source for a HelmRelease, as the source-controller will download the whole storage bucket at each sync. The bucket can easily become very large if there are frequent releases of multiple charts that are stored in the same bucket.

A better option is to use an OCI registry for chart storage.

OCI repository

Helm charts stored in an OCI registry, can be retrieved by declaring an OCIRepository.

apiVersion: source.toolkit.fluxcd.io/v1
kind: OCIRepository
metadata:
 name: podinfo
spec:
 interval: 5m0s
 url: oci://ghcr.io/stefanprodan/charts/podinfo
 layerSelector:
 mediaType: "application/vnd.cncf.helm.chart.content.v1.tar+gzip"
 operation: copy
 ref:
 semver: "^6.9.0"

The source-controller will fetch the Helm chart from the OCI registry namespace on an interval and expose it as an artifact.

The interval defines the interval at which the OCI repository contents are fetched, and should be at least 1m. Setting this to a higher value means newer chart versions will be detected at a slower pace, a push-based fetch can be introduced using webhook receivers.

The url has to point to a registry repository and start with prefix oci://.

The layerSelector has to be set to select the Helm chart content layer.

The ref defines the checkout strategy, and can be one of tag, digest or semver. When using semver, an optional semverFilter can be provided to filter the tags. See the OCIRepository CRD docs for more details.

Authentication

HTTP/S authentication and contextual login can be configured for private OCI registries. See the OCIRepository CRD docs for more details.

Define a Helm release

To release a Helm chart, a HelmRelease resource has to be created. The HelmRelease resources can either reference an existing OCIRepository or Helmchart resource, or it creates a new HelmChart resource and manages it.

Using a chart template

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
 name: podinfo
 namespace: default
spec:
 interval: 10m
 chart:
 spec:
 chart: <name|path>
 version: '4.0.x'
 sourceRef:
 kind: <HelmRepository|GitRepository|Bucket>
 name: podinfo
 namespace: flux-system
 interval: 10m
 values:
 replicaCount: 2

The .chart.spec values are used by the helm-controller as a template to create a new HelmChart resource in the same namespace as the .sourceRef. The source-controller will then look up the chart in the artifact of the referenced source, and either fetch the chart for a HelmRepository, or build it from a GitRepository or Bucket. It will then make it available as a HelmChart artifact to be used by the helm-controller.

The .chart.spec.chart can either contain:

The name of the chart as made available by the HelmRepository (without any aliases), for example: podinfo
The relative path the chart can be found at in the GitRepository or Bucket, for example: ./charts/podinfo
The relative path the chart package can be found at in the GitRepository or Bucket, for example: ./charts/podinfo-1.2.3.tgz

The .chart.spec.version can be a fixed semver, or any semver range (i.e. >=4.0.0 <5.0.0). It is only taken into account for HelmRelease resources that reference a HelmRepository source.

Advanced configuration

The HelmRelease offers an extensive set of configurable flags for finer grain control over how Helm actions are performed. See the HelmRelease CRD docs for more details.

Using a chart reference

It is possible to reference a chart directly from an OCIRepository:

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
 name: podinfo
spec:
 chartRef:
 kind: OCIRepository
 name: podinfo
 namespace: flux-system
 interval: 10m
 values:
 replicaCount: 2

Or a HelmChart:

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
 name: podinfo
spec:
 chartRef:
 kind: HelmChart
 name: podinfo
 namespace: flux-system
 interval: 10m
 values:
 replicaCount: 2

The .chartRef field is used to reference a OCIRepository or HelmChart resource. The helm-controller will then look up the chart in the artifact of the referenced source, and fetch it directly.

The pros of using a chart reference are:

The chart is fetched directly from the source, without the need to create a HelmChart resource for the specific HelmRelease. This can reduce the number of resources in the cluster.
In the case of a OCIRepository, the fact that it is possible to pin to a specific tag or digest makes it easier to enforce a specific change, and is more flexible overall.

Note: When switching from a .chart.spec to a .chartRef, the old HelmChart resource is garbage collected by the helm-controller.

Reacting immediately to changes in referenced Secrets and ConfigMaps

To trigger a Helm release upgrade when changes occur in referenced Secrets or ConfigMaps, you can set the following label on the Secret or ConfigMap:

metadata:
 labels:
 reconcile.fluxcd.io/watch: Enabled

An alternative to labeling every Secret or ConfigMap is setting the --watch-configs-label-selector=owner!=helm flag in helm-controller, which allows watching all Secrets and ConfigMaps except for Helm storage Secrets.

Note: An upgrade will be triggered for an event on a referenced Secret/ConfigMap even if it’s marked as optional in the .spec.valuesFrom field, including deletion events.

Refer to values in Secret generated with Kustomize and SOPS

It is possible to use Kustomize Secret generator to trigger a Helm release upgrade every time the encrypted secret values change.

A SOPS configuration for your cluster is required first. Follow the Manage Kubernetes secrets with Mozilla SOPS guide. The details of configuring SOPS are out of scope for this entry.

Once you have SOPS configured, create a kustomizeconfig.yaml for Kustomize to be able to patch Secrets referenced in HelmRelease manifests:

nameReference:
- kind: Secret
 version: v1
 fieldSpecs:
 - path: spec/valuesFrom/name
 kind: HelmRelease

Create a HelmRelease definition that references a Secret:

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
 name: podinfo
 namespace: podinfo
spec:
 interval: 10m
 releaseName: podinfo
 chart:
 spec:
 chart: podinfo
 sourceRef:
 kind: HelmRepository
 name: podinfo
 valuesFrom:
 - kind: Secret
 name: podinfo-values

Ensure that this HelmRelease will be applied from the same Flux Kustomization that will decrypt the values secret. The encrypted secret data must be housed in the same Flux Kustomization path as the HelmRelease in order to allow Kustomize’s Secret generator function to compose them together with the Kustomize configuration.

Create a kustomization.yaml that generates the Secret:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: podinfo
resources:
 - namespace.yaml
 - repository.yaml
 - release.yaml
secretGenerator:
 - name: podinfo-values
 files:
 - values.yaml=my-values.enc.yaml
configurations:
 - kustomizeconfig.yaml

Now HelmRelease values can come from an encrypted secret! Read on below to prepare the secret file and apply it to the cluster in a Flux Kustomization.

An alternative to relying on the Secret name change described here for triggering a reconciliation is reacting immediately to changes in the Secret through the controller watch.

Encrypting a `values.yaml` with the SOPS CLI

To prepare the encrypted secret, please note there are some divergences from encrypting a well-formed Kubernetes Secret with metadata and versioning.

Run this command to encrypt to the output file that was named in kustomization.yaml above:

$ sops -e --input-type=yaml --output-type=yaml values.yaml > my-values.enc.yaml

Commit the my-values.enc.yaml file and discard the temp file, being sure not to accidentally commit secrets to the repository. Read on for more information.

If users have followed the SOPS guide then we likely added a creation_rules entry telling the sops CLI how to handle encrypting all YAML secrets, like:

# ./apps/sensitive/.sops.yaml
creation_rules:
 - path_regex: ".*\\.yaml"
 encrypted_regex: ^(data|stringData)$
 pgp: KEY_ID_ASDF1234

This selects only the data or stringData fields for encryption, leaving all the metadata unencrypted, which is the proper way to handle normal, well-formed Secrets according to Flux’s SOPS guide.

Since a values file is not a well-formed Secret that rule would fail to encrypt the secret. Instead, here, we add a more specific rule which is listed first, so that our values.yaml file does not get captured by the *.yaml rule.

# ./apps/sensitive/.sops.yaml
creation_rules:
 - path_regex: .*values.yaml$
 pgp: KEY_ID_ASDF1234
 - path_regex: ".*\\.yaml"
 encrypted_regex: ^(data|stringData)$
 pgp: KEY_ID_ASDF1234

The filename was chosen to not match with *.enc or *.encrypted so that sops CLI never infers a binary input type, to avoid that decryption with sops my-values.enc.yaml for editing in-place would also fail.

Be careful not to mix secrets with other non-sensitive data. A note that Helm stores its values as Kubernetes Secrets internally; if users can read secrets in the context then they can recover decrypted values using helm get values.

Refer to values inside the chart

It is possible to replace the values.yaml with a different file present inside the Helm chart.

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
 name: mongodb
 namespace: mongodb
spec:
 interval: 10m
 chart:
 spec:
 chart: mongodb
 sourceRef:
 kind: HelmRepository
 name: bitnami
 valuesFiles:
 - values.yaml
 - values-production.yaml
 values:
 replicaCount: 5

If one or many of the spec.chart.spec.valuesFiles don’t exist inside the chart, helm-controller will not be able to fetch the chart. That default behavior can be overridden by setting .spec.chart.spec.ignoreMissingValuesFiles, so a missing file named in valuesFiles will not cause any failure.

To determine why the HelmChart fails to produce an artifact, you can inspect the status with:

$ kubectl get helmcharts --all-namespaces
NAME READY STATUS
mongodb False failed to locate override values file: values-production.yaml

Configure notifications

The default toolkit installation configures the helm-controller to broadcast events to the notification-controller.

To receive the events as notifications, a Provider needs to be setup first as described in the notifications guide. Once you have set up the Provider, create a new Alert resource in the flux-system to start receiving notifications about the Helm release:

apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Alert
metadata:
 name: helm-podinfo
 namespace: flux-system
spec:
 providerRef:
 name: slack
 eventSeverity: info
 eventSources:
 - kind: HelmRepository
 name: podinfo
 - kind: HelmChart
 name: default-podinfo
 - kind: HelmRelease
 name: podinfo
 namespace: default

Configure webhook receivers

When using semver ranges for Helm releases, you may want to trigger an update as soon as a new chart version is published to your Helm repository. In order to notify source-controller about a chart update, you can setup webhook receivers.

First generate a random string and create a secret with a token field:

TOKEN=$(head -c 12 /dev/urandom | shasum | cut -d ' ' -f1)
echo $TOKEN

kubectl -n flux-system create secret generic webhook-token \
--from-literal=token=$TOKEN

When using Harbor as your Helm repository, you can define a receiver with:

apiVersion: notification.toolkit.fluxcd.io/v1
kind: Receiver
metadata:
 name: helm-podinfo
 namespace: flux-system
spec:
 type: harbor
 secretRef:
 name: webhook-token
 resources:
 - kind: HelmRepository
 name: podinfo

The notification-controller generates a unique URL using the provided token and the receiver name/namespace.

Find the URL with:

$ kubectl -n flux-system get receiver/helm-podinfo

NAME READY STATUS
helm-podinfo True Receiver initialised with URL: /hook/bed6d00b5555b1603e1f59b94d7fdbca58089cb5663633fb83f2815dc626d92b

Endpoint URL: compose the address using the receiver LB and the generated URL http://<LoadBalancerAddress>/<ReceiverURL>
Auth Header: use the token string

With the above settings, when you upload a chart, the following happens:

Harbor sends the chart push event to the receiver address
Notification controller validates the authenticity of the payload using the auth header
Source controller is notified about the changes
Source controller pulls the changes into the cluster and updates the HelmChart version
Helm controller is notified about the version change and upgrades the release

Note

Besides Harbor, you can define receivers for GitHub, GitLab, Bitbucket and any other system that supports webhooks e.g. Jenkins, CircleCI, etc. See the Receiver CRD docs for more details.

Release when a source revision changes for Git and Cloud Storage

It is possible to create a new chart artifact when a Source’s revision has changed, but the version in the Chart.yml has not been bumped, for GitRepository and Bucket sources.

apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmChart
metadata:
 name: podinfo
 namespace: default
spec:
 chart: ./charts/podinfo
 sourceRef:
 name: podinfo
 kind: <GitRepository|Bucket>
 interval: 10m
 reconcileStrategy: Revision

Flux: Setup Webhook Receivers

Mon, 01 Jan 0001 00:00:00 +0000

Flux is by design pull-based. In order to notify the Flux controllers about changes in Git or Helm repositories, you can setup webhooks and trigger a cluster reconciliation every time a source changes. Using webhook receivers make pull-based pipelines as responsive as push-based pipelines.

Prerequisites

To follow this guide you’ll need a Kubernetes cluster with the GitOps toolkit controllers installed on it. Please see the get started guide or the installation guide.

The notification controller can handle events coming from external systems (GitHub, GitLab, Bitbucket, Harbor, Jenkins, etc) and notify the GitOps toolkit controllers about source changes. The notification controller is part of the default toolkit installation.

Expose the webhook receiver

In order to receive Git push or Helm chart upload events, you’ll have to expose the webhook receiver endpoint outside of your Kubernetes cluster on a public address.

The notification controller handles webhook requests on port 9292 and comes with a Kubernetes Service named webhook-receiver that maps the port to 80 inside the cluster.

The webhook receiver port can be used to create a Kubernetes LoadBalancer Service, Ingress or HTTPRoute (Gateway API).

Using a LoadBalancer

Create a Service of type LoadBalancer in the flux-system namespace pointing to the notification-controller pods on port 9292:

apiVersion: v1
kind: Service
metadata:
 name: receiver
 namespace: flux-system
spec:
 type: LoadBalancer
 selector:
 app: notification-controller
 ports:
 - name: http
 port: 80
 protocol: TCP
 targetPort: 9292

Wait for Kubernetes to assign a public address with:

watch kubectl -n flux-system get svc/receiver

Using an Ingress

Create an Ingress in the flux-system namespace pointing to the Kubernetes service named webhook-receiveron port 80:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
 name: webhook-receiver
 namespace: flux-system
spec:
 rules:
 - host: webhook-receiver.example.com
 http:
 paths:
 - pathType: Prefix
 path: /
 backend:
 service:
 name: webhook-receiver
 port:
 number: 80

Add any necessary annotations for your ingress controller and cert-manager to provide for encryption. Full configuration of cert-manager issuers, ingress controllers, and TLS are beyond the scope of this documentation.

However, one common issue caused by Flux’s default network policy securing the flux-system namespace is that unexpected traffic to any pod in that namespace is prevented. The HTTP-01 ACME challenge ingress is blocked from receiving cert-manager traffic.

An example of a policy that permits the traffic from only namespaces with a matching cert-manager label into the challenge pod follows:

kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
 name: allow-cert-manager-resolver
 namespace: "flux-system"
spec:
 podSelector:
 matchLabels:
 acme.cert-manager.io/http01-solver: "true"
 ingress:
 - from:
 - namespaceSelector:
 matchLabels:
 app.kubernetes.io/instance: cert-manager

Your environment’s network policy details may vary. Please take care that the policy is appropriate for the security posture of your environment.

Using a HTTPRoute (Gateway API)

Create a HTTPRoute in the flux-system namespace pointing to the Kubernetes service named webhook-receiver on port 80:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
 name: webhook-receiver
 namespace: flux-system
spec:
 parentRefs:
 - group: gateway.networking.k8s.io
 kind: Gateway
 name: internet-gateway
 namespace: gateway-namespace
 hostnames:
 - "webhook-receiver.example.com"
 rules:
 - matches:
 - path:
 type: PathPrefix
 value: /
 backendRefs:
 - name: webhook-receiver
 namespace: flux-system
 port: 80

Note the parentRefs section must be updated to match your Gateway configuration.

If the Gateway proxy is running in the cluster, the Flux network policy allow-webhooks will allow the traffic to the notification-controller pods without any further configuration.

Define a Git repository

Create a Git source pointing to a GitHub repository that you have control over:

apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
 name: webapp
 namespace: flux-system
spec:
 interval: 60m
 url: https://github.com/<GH-ORG>/<GH-REPO>
 ref:
 branch: master

Authentication

SSH or token based authentication can be configured for private repositories. See the GitRepository CRD docs for more details.

Define a Git repository receiver

First generate a random string and create a secret with a token field:

TOKEN=$(head -c 12 /dev/urandom | shasum | cut -d ' ' -f1)
echo $TOKEN

kubectl -n flux-system create secret generic webhook-token \
--from-literal=token=$TOKEN

Create a receiver for GitHub and specify the GitRepository object:

apiVersion: notification.toolkit.fluxcd.io/v1
kind: Receiver
metadata:
 name: webapp
 namespace: flux-system
spec:
 type: github
 events:
 - "ping"
 - "push"
 secretRef:
 name: webhook-token
 resources:
 - kind: GitRepository
 name: webapp

Webhooks can also be used to trigger ImageRepository and OCIRepository resources to reconcile immediately when the GitHub repositories publish new images to them.

The Receiver is configured as follows: the package event replaces push, and for the webhook configuration select the GitHub webhook “Package” event in the list marked “Let me select individual events.”

apiVersion: notification.toolkit.fluxcd.io/v1
kind: Receiver
metadata:
 name: webapp-image
 namespace: flux-system
spec:
 type: github
 events:
 - "ping"
 - "package"
 secretRef:
 name: webhook-token
 resources:
 - kind: ImageRepository
 name: webapp

Receivers should reconcile source kinds, not the appliers downstream of them. When any Flux image or source kind detects a new artifact revision, Flux will automatically notify Kustomization, HelmRelease, or ImageUpdateAutomation resources downstream of those without any further configuration.

Other receiver

Besides GitHub, you can define receivers for GitLab, Bitbucket, Harbor and any other system that supports webhooks e.g. Jenkins, CircleCI, etc. See the Receiver CRD docs for more details.

The notification controller generates a unique URL using the provided token and the receiver name/namespace.

Find the URL with:

$ kubectl -n flux-system get receiver/webapp

NAME READY STATUS
webapp True Receiver initialised with URL: /hook/bed6d00b5555b1603e1f59b94d7fdbca58089cb5663633fb83f2815dc626d92b

On GitHub, navigate to your repository and click on the “Add webhook” button under “Settings/Webhooks”. Fill the form with:

Payload URL: compose the address using the receiver LB and the generated URL http://<LoadBalancerAddress>/<ReceiverURL>
Secret: use the token string

With the above settings, when you push a commit to the repository, the following happens:

GitHub sends the Git push event to the receiver address
Notification controller validates the authenticity of the payload using HMAC
Source controller is notified about the changes
Source controller pulls the changes into the cluster and updates the GitRepository revision
Kustomize controller is notified about the revision change
Kustomize controller reconciles all the Kustomizations that reference the GitRepository object

Flux: Manage Kubernetes secrets with SOPS

Mon, 01 Jan 0001 00:00:00 +0000

In order to store secrets safely in a public or private Git repository, you can use SOPS CLI to encrypt Kubernetes secrets with OpenPGP, AWS KMS, GCP KMS and Azure Key Vault.

Prerequisites

To follow this guide you’ll need a Kubernetes cluster with the GitOps toolkit controllers installed on it. Please see the get started guide or the installation guide.

Install gnupg and SOPS:

brew install gnupg sops

Generate a GPG key

Generate a GPG/OpenPGP key with no passphrase (%no-protection):

export KEY_NAME="cluster0.yourdomain.com"
export KEY_COMMENT="flux secrets"

gpg --batch --full-generate-key <<EOF
%no-protection
Key-Type: 1
Key-Length: 4096
Subkey-Type: 1
Subkey-Length: 4096
Expire-Date: 0
Name-Comment: ${KEY_COMMENT}
Name-Real: ${KEY_NAME}
EOF

The above configuration creates an rsa4096 key that does not expire. For a full list of options to consider for your environment, see Unattended GPG key generation.

Retrieve the GPG key fingerprint (second row of the sec column):

gpg --list-secret-keys "${KEY_NAME}"

sec rsa4096 2020-09-06 [SC]
 1F3D1CED2F865F5E59CA564553241F147E7C5FA4

Store the key fingerprint as an environment variable:

export KEY_FP=1F3D1CED2F865F5E59CA564553241F147E7C5FA4

Export the public and private keypair from your local GPG keyring and create a Kubernetes secret named sops-gpg in the flux-system namespace:

gpg --export-secret-keys --armor "${KEY_FP}" |
kubectl create secret generic sops-gpg \
--namespace=flux-system \
--from-file=sops.asc=/dev/stdin

It’s a good idea to back up this secret-key/K8s-Secret with a password manager or offline storage. Also consider deleting the secret decryption key from your machine:

gpg --delete-secret-keys "${KEY_FP}"

Configure in-cluster secrets decryption

flux create source git my-secrets \
--url=https://github.com/my-org/my-secrets \
--branch=main

Create a kustomization for reconciling the secrets on the cluster:

flux create kustomization my-secrets \
--source=my-secrets \
--path=./clusters/cluster0 \
--prune=true \
--interval=10m \
--decryption-provider=sops \
--decryption-secret=sops-gpg

Note that the sops-gpg can contain more than one key, SOPS will try to decrypt the secrets by iterating over all the private keys until it finds one that works.

Optional: Export the public key into the Git directory

Commit the public key to the repository so that team members who clone the repo can encrypt new files:

gpg --export --armor "${KEY_FP}" > ./clusters/cluster0/.sops.pub.asc

Check the file contents to ensure it’s the public key before adding it to the repo and committing.

git add ./clusters/cluster0/.sops.pub.asc
git commit -am 'Share GPG public key for secrets generation'

Team members can then import this key when they pull the Git repository:

gpg --import ./clusters/cluster0/.sops.pub.asc

The public key is sufficient for creating brand new files. The secret key is required for decrypting and editing existing files because SOPS computes a MAC on all values. When using solely the public key to add or remove a field, the whole file should be deleted and recreated.

Configure the Git directory for encryption

Write a SOPS config file to the specific cluster or namespace directory used to store encrypted objects with this particular GPG key’s fingerprint.

cat <<EOF > ./clusters/cluster0/.sops.yaml
creation_rules:
 - path_regex: .*.yaml
 encrypted_regex: ^(data|stringData)$
 pgp: ${KEY_FP}
EOF

This config applies recursively to all sub-directories. Multiple directories can use separate SOPS configs. Contributors using the sops CLI to create and encrypt files won’t have to worry about specifying the proper key for the target cluster or namespace.

encrypted_regex helps encrypt the data and stringData fields for Secrets. You may wish to add other fields if you are encrypting other types of Objects.

Hint

Note that you should encrypt only the data or stringData section. Encrypting the Kubernetes secret metadata, kind or apiVersion is not supported by kustomize-controller.

Encrypting secrets using OpenPGP

Generate a Kubernetes secret manifest with kubectl:

kubectl -n default create secret generic basic-auth \
--from-literal=user=admin \
--from-literal=password=change-me \
--dry-run=client \
-o yaml > basic-auth.yaml

Encrypt the secret with SOPS using your GPG key:

sops --encrypt --in-place basic-auth.yaml

You can now commit the encrypted secret to your Git repository.

Hint

Note that you shouldn’t apply the encrypted secrets onto the cluster with kubectl. SOPS encrypted secrets are designed to be consumed by kustomize-controller.

Encrypting secrets using age

age is a simple, modern alternative to OpenPGP. It’s recommended to use age over OpenPGP, if possible.

Encrypting with age follows the same workflow than PGP.

Generate an age key with age using age-keygen:

$ age-keygen -o age.agekey
Public key: age1helqcqsh9464r8chnwc2fzj8uv7vr5ntnsft0tn45v2xtz0hpfwq98cmsg

Create a secret with the age private key, the key name must end with .agekey to be detected as an age key:

cat age.agekey |
kubectl create secret generic sops-age \
--namespace=flux-system \
--from-file=age.agekey=/dev/stdin

Use sops and the age public key to encrypt a Kubernetes secret:

sops --age=age1helqcqsh9464r8chnwc2fzj8uv7vr5ntnsft0tn45v2xtz0hpfwq98cmsg \
--encrypt --encrypted-regex '^(data|stringData)$' --in-place basic-auth.yaml

And finally set the decryption secret in the Flux Kustomization to sops-age.

Encrypting secrets using HashiCorp Vault

HashiCorp Vault is an identity-based secrets and encryption management system.

Encrypting with HashiCorp Vault follows the same workflow as PGP & Age.

Export the VAULT_ADDR and VAULT_TOKEN environment variables to your shell, then use sops to encrypt a Kubernetes Secret (see HashiCorp Vault for more details on enabling the transit backend and sops).

Then use sops to encrypt a Kubernetes Secret:

export VAULT_ADDR=https://vault.example.com:8200
export VAULT_TOKEN=my-token
sops --hc-vault-transit $VAULT_ADDR/v1/sops/keys/my-encryption-key --encrypt \
--encrypted-regex '^(data|stringData)$' --in-place basic-auth.yaml

Create a secret the vault token, the key name must be sops.vault-token to be detected as a vault token:

echo $VAULT_TOKEN |
kubectl create secret generic sops-hcvault \
--namespace=flux-system \
--from-file=sops.vault-token=/dev/stdin

And finally set the decryption secret in the Flux Kustomization to sops-hcvault.

Encrypting secrets using various cloud providers

When using AWS/GCP KMS, you don’t have to include the gpg secretRef under spec.provider (you can skip the --decryption-secret flag when running flux create kustomization), instead you’ll have to bind an IAM Role with access to the KMS keys to the kustomize-controller service account of the flux-system namespace for kustomize-controller to be able to fetch keys from KMS.

AWS

See the SOPS guide to Encrypting Using AWS KMS.

See the AWS integrations docs for details on how to set up SOPS authentication for AWS KMS in kustomize-controller.

Azure

See the SOPS guide to Encrypting Using Azure Key Vault.

See the Azure integrations docs for details on how to set up SOPS authentication for Azure Key Vault in kustomize-controller.

GCP

See the SOPS guide to Encrypting Using GCP KMS.

See the GCP integrations docs for details on how to set up SOPS authentication for GCP KMS in kustomize-controller.

GitOps workflow

A cluster admin should create the Kubernetes secret with the PGP keys on each cluster and add the GitRepository/Kustomization manifests to the fleet repository.

Git repository manifest:

apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
 name: my-secrets
 namespace: flux-system
spec:
 interval: 1m
 url: https://github.com/my-org/my-secrets

Kustomization manifest:

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
 name: my-secrets
 namespace: flux-system
spec:
 interval: 10m0s
 sourceRef:
 kind: GitRepository
 name: my-secrets
 path: ./
 prune: true
 decryption:
 provider: sops
 secretRef:
 name: sops-gpg

Hint

You can generate the above manifests using flux create <kind> --export > manifest.yaml.

Assuming a team member wants to deploy an application that needs to connect to a database using a username and password, they’ll be doing the following:

create a Kubernetes Secret manifest locally with the db credentials e.g. db-auth.yaml
encrypt the secret data field with sops
create a Kubernetes Deployment manifest for the app e.g. app-deployment.yaml
add the Secret to the Deployment manifest as a volume mount or env var
commit the manifests db-auth.yaml and app-deployment.yaml to a Git repository that’s being synced by the GitOps toolkit controllers

Once the manifests have been pushed to the Git repository, the following happens:

source-controller pulls the changes from Git
kustomize-controller loads the GPG keys from the sops-pgp secret
kustomize-controller decrypts the Kubernetes secrets with SOPS and applies them on the cluster
kubelet creates the pods and mounts the secret as a volume or env variable inside the app container

SOPS encrypted_regex conflict

Security notice

The below example is injecting secret data into environment variable key-value pairs in the Pod spec, which is a bad security practice. It should be taken as an explanation of Flux’s behavior related to handling merging of secret data, not a security recommendation.

If your resource is encrypted it will be decrypted right before apply, but it may happen, that your patches will bring fields that match SOPS’ encrypted_regex expression and SOPS will fail during the decryption. Let’s say we have a simple resource.

apiVersion: v1
kind: Pod
metadata:
 name: pod
spec:
 containers:
 - name: main
 image: nginx:stable-alpine
 env:
 - name: ENC[AES256_GCM,data:...
 value: ENC[AES256_GCM,data:...
 resources:
 limits:
 memory: 50Mi
 cpu: 50m
sops:
 ...
 encrypted_regex: ^env$ # There it is
 ...

This Pod has every env list encrypted since we have encrypted_regex set during SOPS encryption. But next we have a patch like this.

apiVersion: v1
kind: Pod
metadata:
 name: pod
spec:
 containers:
 - name: patched
 image: nginx:stable-alpine
 env:
 - name: MainEnvValueIsEncrypted
 value: but this one is not

And as a result you will have.

apiVersion: v1
kind: Pod
metadata:
 name: pod
spec:
 containers:
 - name: main
 image: nginx:stable-alpine
 env:
 - name: ENC[AES256_GCM,data:...
 value: ENC[AES256_GCM,data:...
 resources:
 limits:
 memory: 50Mi
 cpu: 50m
 - name: patched
 image: nginx:stable-alpine
 env:
 - name: MainEnvValueIsEncrypted
 value: but this one is not
sops:
 ...
 encrypted_regex: ^env$ # There it is
 ...

At this point, Flux will call SOPS to decrypt the file and SOPS will try to decrypt all env keys, but container patched has this list in a plain text. SOPS will fail here.

Hint

Move all your secrets to patches and your resource will not require a decryption at the end, since patches are decrypted before.

Flux: Sealed Secrets

Mon, 01 Jan 0001 00:00:00 +0000

In order to store secrets safely in a public or private Git repository, you can use Bitnami’s sealed-secrets controller and encrypt your Kubernetes Secrets into SealedSecrets. The sealed secrets can be decrypted only by the controller running in your cluster and nobody else can obtain the original secret, even if they have access to the Git repository.

Prerequisites

To follow this guide you’ll need a Kubernetes cluster with the GitOps toolkit controllers installed on it. Please see the get started guide or the installation guide.

The sealed-secrets controller comes with a companion CLI tool called kubeseal. With kubeseal you can create SealedSecret custom resources in YAML format and store those in your Git repository.

Install the kubeseal CLI:

brew install kubeseal

For Linux or Windows you can download the kubeseal binary from GitHub.

Deploy sealed-secrets with a HelmRelease

You’ll be using helm-controller APIs to install the sealed-secrets controller from its Helm chart.

First you have to register the Helm repository where the sealed-secrets chart is published:

flux create source helm sealed-secrets \
--interval=1h \
--url=https://bitnami-labs.github.io/sealed-secrets

With interval we configure source-controller to download the Helm repository index every hour. If a newer version of sealed-secrets is published, source-controller will signal helm-controller that a new chart is available.

Create a Helm release that installs the latest version of sealed-secrets controller:

flux create helmrelease sealed-secrets \
--interval=1h \
--release-name=sealed-secrets-controller \
--target-namespace=flux-system \
--source=HelmRepository/sealed-secrets \
--chart=sealed-secrets \
--chart-version=">=1.15.0-0" \
--crds=CreateReplace

With chart version >=1.15.0-0 we configure helm-controller to automatically upgrade the release when a new chart version is fetched by source-controller.

At startup, the sealed-secrets controller generates a 4096-bit RSA key pair and persists the private and public keys as Kubernetes secrets in the flux-system namespace.

You can retrieve the public key with:

kubeseal --fetch-cert \
--controller-name=sealed-secrets-controller \
--controller-namespace=flux-system \
> pub-sealed-secrets.pem

The public key can be safely stored in Git, and can be used to encrypt secrets without direct access to the Kubernetes cluster.

Encrypt secrets

Generate a Kubernetes secret manifest with kubectl:

kubectl -n default create secret generic basic-auth \
--from-literal=user=admin \
--from-literal=password=change-me \
--dry-run=client \
-o yaml > basic-auth.yaml

Encrypt the secret with kubeseal:

kubeseal --format=yaml --cert=pub-sealed-secrets.pem \
< basic-auth.yaml > basic-auth-sealed.yaml

Delete the plain secret and apply the sealed one:

rm basic-auth.yaml
kubectl apply -f basic-auth-sealed.yaml

Verify that the sealed-secrets controller has created the basic-auth Kubernetes Secret:

$ kubectl -n default get secrets basic-auth

NAME TYPE DATA AGE
basic-auth Opaque 2 1m43s

GitOps workflow

A cluster admin should add the stable HelmRepository manifest and the sealed-secrets HelmRelease to the fleet repository.

Helm repository manifest:

apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
 name: sealed-secrets
 namespace: flux-system
spec:
 interval: 1h0m0s
 url: https://bitnami-labs.github.io/sealed-secrets

Helm release manifest:

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
 name: sealed-secrets
 namespace: flux-system
spec:
 chart:
 spec:
 chart: sealed-secrets
 sourceRef:
 kind: HelmRepository
 name: sealed-secrets
 version: ">=1.15.0-0"
 interval: 1h0m0s
 releaseName: sealed-secrets-controller
 targetNamespace: flux-system
 install:
 crds: Create
 upgrade:
 crds: CreateReplace

You can generate the above manifests using flux create <kind> --export > manifest.yaml.

Once the sealed-secrets controller is installed, the admin fetches the public key and shares it with the teams that operate on the fleet clusters via Git.

When a team member wants to create a Kubernetes Secret on a cluster, they use kubeseal and the public key corresponding to that cluster to generate a SealedSecret.

Assuming a team member wants to deploy an application that needs to connect to a database using a username and password, they’ll be doing the following:

create a Kubernetes Secret manifest locally with the db credentials e.g. db-auth.yaml
encrypt the secret with kubeseal as db-auth-sealed.yaml
delete the original secret file db-auth.yaml
create a Kubernetes Deployment manifest for the app e.g. app-deployment.yaml
add the Secret to the Deployment manifest as a volume mount or env var using the original name db-auth
commit the manifests db-auth-sealed.yaml and app-deployment.yaml to a Git repository that’s being synced by the GitOps toolkit controllers

Once the manifests have been pushed to the Git repository, the following happens:

source-controller pulls the changes from Git
kustomize-controller applies the SealedSecret and the Deployment manifests
sealed-secrets controller decrypts the SealedSecret and creates a Kubernetes Secret
kubelet creates the pods and mounts the secret as a volume or env variable inside the app container

Flux: Automate image updates to Git

Mon, 01 Jan 0001 00:00:00 +0000

This guide walks you through configuring container image scanning and deployment rollouts with Flux.

For a container image you can configure Flux to:

scan the container registry and fetch the image tags
select the latest tag based on the defined policy (semver, calver, regex)
replace the tag in Kubernetes manifests (YAML format)
checkout a branch, commit and push the changes to the remote Git repository
apply the changes in-cluster and rollout the container image

For production environments, this feature allows you to automatically deploy application patches (CVEs and bug fixes), and keep a record of all deployments in Git history.

Production CI/CD workflow

DEV: push a bug fix to the app repository
DEV: bump the patch version and release e.g. v1.0.1
CI: build and push a container image tagged as registry.domain/org/app:v1.0.1
CD: pull the latest image metadata from the app registry (Flux image scanning)
CD: update the image tag in the app manifest to v1.0.1 (Flux cluster to Git reconciliation)
CD: deploy v1.0.1 to production clusters (Flux Git to cluster reconciliation)

For staging environments, this features allow you to deploy the latest build of a branch, without having to manually edit the app deployment manifest in Git.

Staging CI/CD workflow

DEV: push code changes to the app repository main branch
CI: build and push a container image tagged as ${GIT_BRANCH}-${GIT_SHA:0:7}-$(date +%s)
CD: pull the latest image metadata from the app registry (Flux image scanning)
CD: update the image tag in the app manifest to main-2d3fcbd-1611906956 (Flux cluster to Git reconciliation)
CD: deploy main-2d3fcbd-1611906956 to staging clusters (Flux Git to cluster reconciliation)

Prerequisites

You will need a Kubernetes cluster version 1.16 or newer and kubectl version 1.18. For a quick local test, you can use Kubernetes kind. Any other Kubernetes setup will work as well.

In order to follow the guide you’ll need a GitHub account and a personal access token that can create repositories (check all permissions under repo).

Export your GitHub personal access token and username:

export GITHUB_TOKEN=<your-token>
export GITHUB_USER=<your-username>

Install Flux

Enable image automation components

If you bootstrapped Flux before, you need to add --components-extra=image-reflector-controller,image-automation-controller to your bootstrapping routine as image automation components are not installed by default. Please note that you need to delete the flux-system secret before rerunning bootstrap to rotate the deploy key.

Install Flux with the image automation components:

flux bootstrap github \
 --components-extra=image-reflector-controller,image-automation-controller \
 --owner=$GITHUB_USER \
 --repository=flux-image-updates \
 --branch=main \
 --path=clusters/my-cluster \
 --read-write-key \
 --personal

The bootstrap command creates a repository if one doesn’t exist, and commits the manifests for the Flux components to the default branch at the specified path. It then configures the target cluster to synchronize with the specified path inside the repository.

GitLab and other Git platforms

You can install Flux and bootstrap repositories hosted on GitLab, BitBucket, Azure DevOps and any other Git provider that support SSH or token-based authentication. When using SSH, make sure the deploy key is configured with write access --read-write-key. Please see the installation guide for more details.

Deploy a demo app

We’ll be using a tiny webapp called podinfo to showcase the image update feature.

Clone your repository with:

git clone https://github.com/$GITHUB_USER/flux-image-updates
cd flux-image-updates

Create a deployment for podinfo inside clusters/my-cluster:

cat <<EOF > ./clusters/my-cluster/podinfo-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
 name: podinfo
 namespace: default
spec:
 selector:
 matchLabels:
 app: podinfo
 template:
 metadata:
 labels:
 app: podinfo
 spec:
 containers:
 - name: podinfod
 image: ghcr.io/stefanprodan/podinfo:5.0.0
 imagePullPolicy: IfNotPresent
 ports:
 - name: http
 containerPort: 9898
 protocol: TCP
EOF

Commit and push changes to main branch:

git add -A && \
git commit -m "add podinfo deployment" && \
git push origin main

Tell Flux to pull and apply the changes or wait one minute for Flux to detect the changes on its own:

flux reconcile kustomization flux-system --with-source

Print the podinfo image deployed on your cluster:

$ kubectl get deployment/podinfo -oyaml | grep 'image:'
image: ghcr.io/stefanprodan/podinfo:5.0.0

Configure image scanning

Create an ImageRepository to tell Flux which container registry to scan for new tags:

flux create image repository podinfo \
--image=ghcr.io/stefanprodan/podinfo \
--interval=5m \
--export > ./clusters/my-cluster/podinfo-registry.yaml

The above command generates the following manifest:

apiVersion: image.toolkit.fluxcd.io/v1
kind: ImageRepository
metadata:
 name: podinfo
 namespace: flux-system
spec:
 image: ghcr.io/stefanprodan/podinfo
 interval: 5m

For private images, you can create a Kubernetes secret in the same namespace as the ImageRepository with kubectl create secret docker-registry. Then you can configure Flux to use the credentials by referencing the Kubernetes secret in the ImageRepository:

kind: ImageRepository
spec:
 secretRef:
 name: regcred

Storing secrets in Git

Note that if you want to store the image pull secret in Git, you can encrypt the manifest with Mozilla SOPS or Sealed Secrets.

Create an ImagePolicy to tell Flux which semver range to use when filtering tags:

flux create image policy podinfo \
--image-ref=podinfo \
--select-semver=5.0.x \
--export > ./clusters/my-cluster/podinfo-policy.yaml

The above command generates the following manifest:

apiVersion: image.toolkit.fluxcd.io/v1
kind: ImagePolicy
metadata:
 name: podinfo
 namespace: flux-system
spec:
 imageRepositoryRef:
 name: podinfo
 policy:
 semver:
 range: 5.0.x

A semver range that includes stable releases can be defined with 1.0.x (patch versions only) or >=1.0.0 <2.0.0 (minor and patch versions). If you want to include pre-release e.g. 1.0.0-rc.1, you can define a range like: ^1.x-0 or >1.0.0-rc <2.0.0-rc.

Other policy examples

For policies that make use of CalVer, build IDs or alphabetical sorting, have a look at the examples.

Commit and push changes to main branch:

git add -A && \
git commit -m "add podinfo image scan" && \
git push origin main

Tell Flux to pull and apply changes:

flux reconcile kustomization flux-system --with-source

Wait for Flux to fetch the image tag list from GitHub container registry:

$ flux get image repository podinfo
NAME LAST SCAN SUSPENDED READY MESSAGE
podinfo 2020-12-13T17:51:48+02:00 False True successful scan: found 13 tags

For debugging purposes, to see a sample of the tags scanned by the ImageRepository, a list of ten latest tags scanned by the ImageRepository resource can be seen in the status of the resource:

$ kubectl -n flux-system describe imagerepositories podinfo
...
Status:
 Canonical Image Name: ghcr.io/stefanprodan/podinfo
 Last Scan Result:
 Latest Tags:
 latest
 6.3.3
 6.3.2
 6.3.1
 6.3.0
 6.2.3
 6.2.2
 6.2.1
 6.2.0
 6.1.8
 Scan Time: 2020-12-13T17:51:48Z
 Tag Count: 13
 Observed Exclusion List:
 ^.*\.sig$
 Observed Generation: 2
...

Find which image tag matches the policy semver range with:

$ flux get image policy podinfo
NAME LATEST IMAGE READY MESSAGE
podinfo ghcr.io/stefanprodan/podinfo:5.0.3 True Latest image tag for 'ghcr.io/stefanprodan/podinfo' resolved to 5.0.3

Configure image updates

Edit the podinfo-deployment.yaml and add a marker to tell Flux which policy to use when updating the container image:

spec:
 containers:
 - name: podinfod
 image: ghcr.io/stefanprodan/podinfo:5.0.0 # {"$imagepolicy": "flux-system:podinfo"}

Create an ImageUpdateAutomation to tell Flux which Git repository to write image updates to:

flux create image update flux-system \
--interval=30m \
--git-repo-ref=flux-system \
--git-repo-path="./clusters/my-cluster" \
--checkout-branch=main \
--push-branch=main \
--author-name=fluxcdbot \
--author-email=fluxcdbot@users.noreply.github.com \
--commit-template="{{range .Changed.Changes}}{{print .OldValue}} -> {{println .NewValue}}{{end}}" \
--export > ./clusters/my-cluster/flux-system-automation.yaml

The above command generates the following manifest:

apiVersion: image.toolkit.fluxcd.io/v1
kind: ImageUpdateAutomation
metadata:
 name: flux-system
 namespace: flux-system
spec:
 interval: 30m
 sourceRef:
 kind: GitRepository
 name: flux-system
 git:
 checkout:
 ref:
 branch: main
 commit:
 author:
 email: fluxcdbot@users.noreply.github.com
 name: fluxcdbot
 messageTemplate: '{{range .Changed.Changes}}{{print .OldValue}} -> {{println .NewValue}}{{end}}'
 push:
 branch: main
 update:
 path: ./clusters/my-cluster
 strategy: Setters

Commit and push changes to main branch:

git add -A && \
git commit -m "add image updates automation" && \
git push origin main

Note that the ImageUpdateAutomation runs all the policies found in its namespace at the specified interval.

Tell Flux to pull and apply changes:

flux reconcile kustomization flux-system --with-source

In a couple of seconds, Flux will push a commit to your repository with the latest image tag that matches the podinfo policy:

$ git pull && cat clusters/my-cluster/podinfo-deployment.yaml | grep "image:"
image: ghcr.io/stefanprodan/podinfo:5.0.3 # {"$imagepolicy": "flux-system:podinfo"}

Wait for Flux to apply the latest commit on the cluster and verify that podinfo was updated to 5.0.3:

$ watch "kubectl get deployment/podinfo -oyaml | grep 'image:'"
image: ghcr.io/stefanprodan/podinfo:5.0.3

You can check the status of the image automation objects with:

flux get images all --all-namespaces

Configure image update for custom resources

Besides Kubernetes native kinds (Deployment, StatefulSet, DaemonSet, CronJob), Flux can be used to patch image references in any Kubernetes custom resource stored in Git.

The image policy marker format is:

{"$imagepolicy": "<policy-namespace>:<policy-name>"}
{"$imagepolicy": "<policy-namespace>:<policy-name>:tag"}
{"$imagepolicy": "<policy-namespace>:<policy-name>:name"}
{"$imagepolicy": "<policy-namespace>:<policy-name>:digest"}

Note: The digest is only made available for ImagePolicy resources that have spec.digestReflectionPolicy set to IfNotPresent or Always. When set to Never (the default), the digest will not be reflected in the ImagePolicy status.

These markers are placed inline in the target YAML, as a comment. The “Setter” strategy refers to kyaml setters which Flux can find and replace during reconciliation, when directed to do so by an ImageUpdateAutomation like the one above.

Here are some examples of using this marker in a variety of Kubernetes resources.

HelmRelease example:

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
 name: podinfo
 namespace: default
spec:
 values:
 image:
 repository: ghcr.io/stefanprodan/podinfo # {"$imagepolicy": "flux-system:podinfo:name"}
 tag: 5.0.0 # {"$imagepolicy": "flux-system:podinfo:tag"}
 digest: sha256:ec0119616bb8be9199575c05bfc23a6bf0fbdb0690ee15834e7b43bc3f4f6017 # {"$imagepolicy": "flux-system:podinfo:digest"}

Tekton Task example:

apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
 name: golang
 namespace: default
spec:
 steps:
 - name: golang
 image: docker.io/golang:1.15.6 # {"$imagepolicy": "flux-system:golang"}

Flux Kustomization example:

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
 name: podinfo
 namespace: default
spec:
 images:
 - name: ghcr.io/stefanprodan/podinfo
 newName: ghcr.io/stefanprodan/podinfo # {"$imagepolicy": "flux-system:podinfo:name"}
 newTag: 5.0.0 # {"$imagepolicy": "flux-system:podinfo:tag"}

Kustomize config (kustomization.yaml) example:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- deployment.yaml
images:
- name: ghcr.io/stefanprodan/podinfo
 newName: ghcr.io/stefanprodan/podinfo # {"$imagepolicy": "flux-system:podinfo:name"}
 newTag: 5.0.0 # {"$imagepolicy": "flux-system:podinfo:tag"}

Push updates to a different branch

With .spec.git.push.branch you can configure Flux to push the image updates to different branch than the one used for checkout. If the specified branch doesn’t exist, Flux will create it for you.

kind: ImageUpdateAutomation
metadata:
 name: flux-system
spec:
 git:
 checkout:
 ref:
 branch: main
 push:
 branch: flux-image-updates

You can use CI automation e.g. GitHub Actions such as Flux’s GitHub Actions Auto PR example to open a pull request against the checkout branch.

This way you can manually approve the image updates before they are applied on your clusters.

Configure the commit message

The .spec.git.commit.messageTemplate field is a string which is used as a template for the commit message.

The message template is a Go text template that lets you range over the objects and images e.g.:

kind: ImageUpdateAutomation
metadata:
 name: flux-system
spec:
 git:
 commit:
 messageTemplate: |
 Automated image update

 Automation name: {{ .AutomationObject }}

 Files:
 {{ range $filename, $_ := .Changed.FileChanges -}}
 - {{ $filename }}
 {{ end -}}

 Objects:
 {{ range $resource, $changes := .Changed.Objects -}}
 - {{ $resource.Kind }} {{ $resource.Name }}
 Changes:
 {{- range $_, $change := $changes }}
 - {{ $change.OldValue }} -> {{ $change.NewValue }}
 {{ end -}}
 {{ end -}}
 author:
 email: fluxcdbot@users.noreply.github.com
 name: fluxcdbot

Trigger image updates with webhooks

You may want to trigger a deployment as soon as a new image tag is pushed to your container registry. In order to notify the image-reflector-controller about new images, you can setup webhook receivers.

First generate a random string and create a secret with a token field:

TOKEN=$(head -c 12 /dev/urandom | shasum | cut -d ' ' -f1)
echo $TOKEN

kubectl -n flux-system create secret generic webhook-token \ 
--from-literal=token=$TOKEN

Define a receiver for DockerHub:

apiVersion: notification.toolkit.fluxcd.io/v1
kind: Receiver
metadata:
 name: podinfo
 namespace: flux-system
spec:
 type: dockerhub
 secretRef:
 name: webhook-token
 resources:
 - kind: ImageRepository
 name: podinfo

The notification-controller generates a unique URL using the provided token and the receiver name/namespace.

Find the URL with:

$ kubectl -n flux-system get receiver/podinfo

NAME READY STATUS
podinfo True Receiver initialised with URL: /hook/bed6d00b5555b1603e1f59b94d7fdbca58089cb5663633fb83f2815dc626d92b

Log in to DockerHub web interface, go to your image registry Settings and select Webhooks. Fill the form “Webhook URL” by composing the address using the receiver LB and the generated URL http://<LoadBalancerAddress>/<ReceiverURL>.

Other receivers

Besides DockerHub, you can define receivers for Harbor, Quay, Nexus, GCR, and any other system that supports webhooks e.g. GitHub Actions, Jenkins, CircleCI, etc. See the Receiver CRD docs for more details.

Incident management

Suspend automation

During an incident you may wish to stop Flux from pushing image updates to Git.

You can suspend the image automation directly in-cluster:

flux suspend image update flux-system

Or by editing the ImageUpdateAutomation manifest in Git:

kind: ImageUpdateAutomation
metadata:
 name: flux-system
 namespace: flux-system
spec:
 suspend: true

Once the incident is resolved, you can resume automation with:

flux resume image update flux-system

If you wish to pause the automation for a particular image only, you can suspend/resume the image scanning:

flux suspend image repository podinfo

Revert image updates

Assuming you’ve configured Flux to update an app to its latest stable version:

flux create image policy podinfo \
--image-ref=podinfo \
--select-semver=">=5.0.0"

If the latest version e.g. 5.0.1 causes an incident in production, you can tell Flux to revert the image tag to a previous version e.g. 5.0.0 with:

flux create image policy podinfo \
--image-ref=podinfo \
--select-semver=5.0.0

Or by changing the semver range in Git:

kind: ImagePolicy
metadata:
 name: podinfo
 namespace: flux-system
spec:
 policy:
 semver:
 range: 5.0.0

Based on the above configuration, Flux will patch the podinfo deployment manifest in Git and roll out 5.0.0 in-cluster.

When a new version is available e.g. 5.0.2, you can update the policy once more and tell Flux to consider only versions greater than 5.0.1:

flux create image policy podinfo \
--image-ref=podinfo \
--select-semver=">5.0.1"

ImageRepository cloud providers authentication

Two methods are available for authenticating container registers as ImageRepository resources in Flux:

Automated authentication mechanisms (where the controller retrieves the credentials itself and is only available for the three major cloud providers), or
Secret-based authentication where the user creates a Kubernetes secret with the credentials.

Native authentication mechanisms have been implemented in Flux for the three major cloud providers, but they have to be set in the individual ImageRepository resources. Please see individual sections on how to do this.

ImagePolicy Examples

Select the latest main branch build tagged as ${GIT_BRANCH}-${GIT_SHA:0:7}-$(date +%s) (numerical):

kind: ImagePolicy
spec:
 filterTags:
 pattern: '^main-[a-fA-F0-9]+-(?P<ts>.*)'
 extract: '$ts'
 policy:
 numerical:
 order: asc

A more strict filter would be ^main-[a-fA-F0-9]+-(?P<ts>[1-9][0-9]*). Before applying policies in-cluster, you can validate your filters using regex101.com.

Select the latest stable version (semver):

kind: ImagePolicy
spec:
 policy:
 semver:
 range: '>=1.0.0'

Select the latest stable patch version in the 1.x range (semver):

kind: ImagePolicy
spec:
 policy:
 semver:
 range: '>=1.0.0 <2.0.0'

Select the latest version including pre-releases (semver):

kind: ImagePolicy
spec:
 policy:
 semver:
 range: '>=1.0.0-0'

Select the latest release candidate (semver):

kind: ImagePolicy
spec:
 filterTags:
 pattern: '.*-rc.*'
 policy:
 semver:
 range: '^1.x-0'

Select the latest version by extracting the semver from tags in the format <semver>-<distro> e.g. Redis on DockerHub:

kind: ImagePolicy
spec:
 filterTags:
 pattern: '^(?P<semver>[0-9]*\.[0-9]*\.[0-9]*)-(alpine[0-9]*\.[0-9]*)'
 extract: '$semver'
 policy:
 semver:
 range: '>=1.0.0'

Select the latest release tagged as RELEASE.<RFC3339-TIMESTAMP> e.g. Minio (alphabetical):

kind: ImagePolicy
spec:
 filterTags:
 pattern: '^RELEASE\.(?P<timestamp>.*)Z$'
 extract: '$timestamp'
 policy:
 alphabetical:
 order: asc

Digest pinning

Following `latest` tags

In some cases you may want to use the latest tag for some container images. A recommended GitOps way to do this is pinning the image digest after the tag in the image reference in your Git repository. For example:

image: ghcr.io/stefanprodan/podinfo:latest@sha256:ec0119616bb8be9199575c05bfc23a6bf0fbdb0690ee15834e7b43bc3f4f6017

This gives you control over the image version that is deployed in your cluster, simply restarting pods will no longer pull the latest image automatically and change the underlying software without you knowing. This also makes the updates traceable in your Git history.

With the right ImagePolicy configuration Flux is capable of polling image digests for a fixed tag. To configure Flux to watch the digest for latest, you can use an ImagePolicy like the following:

apiVersion: image.toolkit.fluxcd.io/v1
kind: ImagePolicy
metadata:
 name: podinfo
 namespace: flux-system
spec:
 imageRepositoryRef:
 name: podinfo
 filterTags:
 pattern: '^latest$'
 policy:
 alphabetical: {}
 digestReflectionPolicy: Always
 interval: 10m

The trick is filtering a single tag, the one you want to follow, and setting digestReflectionPolicy to Always, which will tell Flux to fetch the digest for that tag according to the specified interval, once every ten minutes in the example above.

Then in the manifest where you want to use the latest tag, you can do the following:

apiVersion: apps/v1
kind: Deployment
metadata:
 name: podinfo
 namespace: default
spec:
 selector:
 matchLabels:
 app: podinfo
 template:
 metadata:
 labels:
 app: podinfo
 spec:
 containers:
 - name: podinfo
 image: ghcr.io/stefanprodan/podinfo:latest@sha256:ec0119616bb8be9199575c05bfc23a6bf0fbdb0690ee15834e7b43bc3f4f6017 # {"$imagepolicy": "flux-system:podinfo"}

Or if you have a HelmRelease where repository, tag and digest are separate values:

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
 name: podinfo
 namespace: default
spec:
 values:
 image:
 repository: ghcr.io/stefanprodan/podinfo # {"$imagepolicy": "flux-system:podinfo:name"}
 tag: latest  # {"$imagepolicy": "flux-system:podinfo:tag"}
 digest: sha256:ec0119616bb8be9199575c05bfc23a6bf0fbdb0690ee15834e7b43bc3f4f6017 # {"$imagepolicy": "flux-system:podinfo:digest"}

Achieving immutability for tags

Another common use case is preventing images from ever changing after a new tag is released. If later for whatever reason this tag is pushed with a different image, i.e. a different digest, and you don’t want it to be updated e.g. due to the reasons of how you test and ship your software, you can set digestReflectionPolicy to IfNotPresent in the ImagePolicy:

apiVersion: image.toolkit.fluxcd.io/v1
kind: ImagePolicy
metadata:
 name: podinfo
 namespace: flux-system
spec:
 imageRepositoryRef:
 name: podinfo
 policy:
 semver:
 range: 5.0.x
 digestReflectionPolicy: IfNotPresent

In this case the digest will only be updated in the ImagePolicy status if a new tag is elected as latest, in which case both the tag and the digest will be updated together. Setting an interval in this case is not allowed.

Flux: How to make sortable image tags to use with automation

Mon, 01 Jan 0001 00:00:00 +0000

Flux does not support selecting the latest image by build time. Obtaining the build time needs the container config for each image, and fetching that is subject to strict rate limiting by image registries (e.g., by DockerHub).

This guide explains how to construct image tags so that the most recent image has the tag that comes last in alphabetical or numerical order. The technique suggested is to put a timestamp or serial number in each image tag.

Formats and alternatives

The important properties for sorting are that the parts of the timestamp go from most significant to least (e.g., the year down to the second). For numbers it is best to use numerical order, since this will work with values of different width (e.g., ‘12’ sorts after ‘2’).

Image tags are often shown in user interfaces, so readability matters. Here is an example of a readable timestamp that will sort well:

$ # date and time (remember ':' is not allowed in a tag)
$ date +%F.%H%M%S
2021-01-28.133158

You can use a timestamp that sorts as a number, like Unix time:

$ # seconds since Jan 1 1970
$ date +%s
1611840548

Alternatively, you can use a serial number as part of the tag. Some CI platforms will provide a build number in an environment variable, but that may not be reliable to use as a serial number – check the platform documentation. For example, GitHub makes availabe the variable github.run_number which can be used as a reliable ever increasing serial number.

A commit count can be a reasonable stand-in for a serial number, if you build an image per commit and you don’t rewrite the branch in question:

$ # commits in branch
$ git rev-list --count HEAD
1504

Beware: this will not give a useful number if you have a shallow clone.

Other things to include in the image tag

It is also handy to quickly trace an image to the branch and commit of its source code. Including the branch also means you can filter for images from a particular branch.

A useful tag format is

<branch>-<sha1>-<timestamp>

The branch and tag will usually be made available in a CI platform as environment variables. See

Example of a build process with timestamp tagging

Here is an example of a GitHub Actions job that creates a “build ID” with the git branch, SHA1, and a timestamp, and uses it as a tag when building an image:

jobs:
 build-push:
 env:
 IMAGE: org/my-app
 runs-on: ubuntu-latest
 steps:

 - name: Generate build ID
 id: prep
 run: |
 branch=${GITHUB_REF##*/}
 sha=${GITHUB_SHA::8}
 ts=$(date +%s)
 echo "::set-output name=BUILD_ID::${branch}-${sha}-${ts}"

 # These are prerequisites for the docker build step
 - name: Set up QEMU
 uses: docker/setup-qemu-action@v2
 - name: Set up Docker Buildx
 uses: docker/setup-buildx-action@v2
 - name: Login to DockerHub
 uses: docker/login-action@v2
 with:
 username: ${{ secrets.DOCKERHUB_USERNAME }}
 password: ${{ secrets.DOCKERHUB_TOKEN }}

 - name: Build and publish container image with tag
 uses: docker/build-push-action@v4
 with:
 push: true
 context: .
 file: ./Dockerfile
 tags: |
 ${{ env.IMAGE }}:${{ steps.prep.outputs.BUILD_ID }}

Alternative example utilizing github.run_number

Here is another example example of a GitHub Actions job which tags images using GitHub action’s built in run_number and the git SHA1:

jobs:
 build-push:
 env:
 IMAGE: org/my-app
 runs-on: ubuntu-latest
 steps:
 # These are prerequisites for the docker build step
 - name: Set up QEMU
 uses: docker/setup-qemu-action@v2
 - name: Set up Docker Buildx
 uses: docker/setup-buildx-action@v2
 - name: Login to DockerHub
 uses: docker/login-action@v2
 with:
 username: ${{ secrets.DOCKERHUB_USERNAME }}
 password: ${{ secrets.DOCKERHUB_TOKEN }}

 - name: Build and publish container image with tag
 uses: docker/build-push-action@v4
 with:
 push: true
 context: .
 file: ./Dockerfile
 tags: |
 ${{ env.IMAGE }}:${{ github.sha }}-${{ github.run_number }}

Using in an `ImagePolicy` object

When creating an ImagePolicy object, you will need to extract just the timestamp part of the tag, using the tagFilter field. You can filter for a particular branch to restrict images to only those built from that branch.

Here is an example that filters for only images built from main branch, and selects the most recent according to a timestamp (created with date +%s) or according to the run number (github.run_number for example):

apiVersion: image.toolkit.fluxcd.io/v1
kind: ImagePolicy
metadata:
 name: image-repo-policy
 namespace: flux-system
spec:
 imageRepositoryRef:
 name: image-repo
 filterTags:
 ## use "pattern: '[a-f0-9]+-(?P<ts>[0-9]+)'" if you copied the workflow example using github.run_number
 pattern: '^main-[a-f0-9]+-(?P<ts>[0-9]+)'
 extract: '$ts'
 policy:
 numerical:
 order: asc

If you don’t care about the branch, that part can be a wildcard in the pattern:

apiVersion: image.toolkit.fluxcd.io/v1
kind: ImagePolicy
metadata:
 name: image-repo-policy
 namespace: flux-system
spec:
 imageRepositoryRef:
 name: image-repo
 filterTags:
 pattern: '^.+-[a-f0-9]+-(?P<ts>[0-9]+)'
 extract: '$ts'
 policy:
 numerical:
 order: asc

Flux – User Guides

Flux: Ways of structuring your repositories

Monorepo

Repository structure

Delivery management

Repo per environment

Repo per team

Repository structure

Delivery

Repo per app

Repository structure

Flux: Manage Helm Releases

Prerequisites

Define a chart source

Helm repository

Helm HTTP/S repository

Authentication

Helm repository authentication with credentials

Git repository

Authentication

Cloud Storage

OCI repository

Authentication

Define a Helm release

Using a chart template

Advanced configuration

Using a chart reference

Reacting immediately to changes in referenced Secrets and ConfigMaps

Refer to values in Secret generated with Kustomize and SOPS

Encrypting a values.yaml with the SOPS CLI

Refer to values inside the chart

Configure notifications

Configure webhook receivers

Note

Release when a source revision changes for Git and Cloud Storage

Flux: Setup Webhook Receivers

Prerequisites

Expose the webhook receiver

Using a LoadBalancer

Using an Ingress

Using a HTTPRoute (Gateway API)

Define a Git repository

Authentication

Define a Git repository receiver

Other receiver

Flux: Manage Kubernetes secrets with SOPS

Prerequisites

Generate a GPG key

Configure in-cluster secrets decryption

Optional: Export the public key into the Git directory

Configure the Git directory for encryption

Hint

Encrypting secrets using OpenPGP

Hint

Encrypting secrets using age

Encrypting secrets using HashiCorp Vault

Encrypting secrets using various cloud providers

AWS

Azure

GCP

GitOps workflow

Hint

SOPS encrypted_regex conflict

Security notice

Hint

Flux: Sealed Secrets

Prerequisites

Deploy sealed-secrets with a HelmRelease

Encrypt secrets

GitOps workflow

Flux: Automate image updates to Git

Prerequisites

Install Flux

Enable image automation components

GitLab and other Git platforms

Deploy a demo app

Configure image scanning

Storing secrets in Git

Other policy examples

Configure image updates

Encrypting a `values.yaml` with the SOPS CLI

Following `latest` tags

Using in an `ImagePolicy` object