Flux – Blog

Blog: Stairway to GitOps: Scaling Flux at Morgan Stanley

Sun, 15 Mar 2026 12:00:00 +0000

One of the things we love most about this community is hearing how you take Flux and run with it - truly solving problems for teams at scale. At our inaugural FluxCon NA, Tiffany Wang and Simon Bourassa from Morgan Stanley gave us a glimpse of their Flux environment.

Their talk, “Stairway to GitOps,” walked us through a five-year journey from push-based pipelines to a self-service GitOps platform managing over 500 clusters. Hearing the core principles of Flux - Lean, Performant, Extensible, and Secure - validated by end-users at this scale matters a lot to us as maintainers. We think their lessons are worth sharing with all of you.

Matheus Pimenta cracking a joke with the Flux team together at FluxCon NA 2025 - (Moments with all of these people in-person are rare!)

The Early Days: Pushing Limits

Like many teams, Morgan Stanley started with traditional push-based CI/CD pipelines. App teams used tools like Helm to push manifests directly to clusters. While functional for initial deployments, challenges emerged as they scaled. Familiar pain points crept in:

Configuration Drift: Without an agent continuously reconciling state, clusters drifted from the source of truth in Git. Manual changes and failed deployments left systems in an unknown state.
Fragile Recovery: Cluster rebuilds required heavy coordination. The platform team could restore infrastructure, but application teams had to manually redeploy their workloads. (Not a great place to be at 2 AM in another team’s timezone)

At “Step 0” of their Stairway to GitOps, they realized they needed to decouple delivery from the pipeline and embrace continuous reconciliation.

Step 1: Security and Self-Service

In a highly regulated financial environment, security isn’t optional. The team chose Flux to fit their strict multi-tenancy model.

Morgan Stanley leveraged Flux’s service account impersonation and native Kubernetes RBAC to enforce least-privilege access. Controllers reconciling manifests for one team had zero visibility into another team’s resources. Granular, secure multi-tenancy is a first priority part of Flux’s design, so this is the golden path, but implementing it always involves deciding what teams get what permissions, and they put in that work.

To streamline adoption, they built a self-service onboarding platform. Instead of requiring developers to manage low-level Kubernetes details, they created tooling that:

Automated entitlement checks and change control processes.
Registered services in their CMDB.
“Primed” the target namespace with the necessary Flux GitRepository and Kustomization resources.
Scaffolded a ready-to-use application repository.

This approach demonstrates Flux’s extensibility. Flux can serve as the glue between systems. Developers interact with their normal tooling, while company specific systems like CMDB’s (which likely predate Kubernetes adoption at all) integrate smoothly into the GitOps flow.

Step 2: Operating at Scale

As adoption grew, so did the deployment footprint. Tiffany shared some numbers from their environment:

“And now we have over 500 clusters, over 2,000 nodes, over 100,000 containers, and tens of thousands of Flux resources.” (13:34)

Operating at this magnitude brings challenges around performance. The team shared how they tuned Flux to handle this load without overwhelming the Kubernetes control plane.

Tuning for Performance

With tens of thousands of resources reconciling, the team started some performance tuning. Their focus areas:

Reconciliation Intervals: They increased their platform defaults, tuning intervals to balance responsiveness with load.
Controller Concurrency: By adjusting the --concurrent flags on Flux controllers, they increased how many reconciliations could happen in parallel.
Resource Management: They monitored and adjusted resource limits for Flux components to ensure reliability under sustained load.

We put a lot of thought into making these knobs available. Flux should run well on a Raspberry Pi and on a fleet of 500 clusters alike. The platform team taking ownership of Flux’s runtime in this manner shows operational excellence.

Moving from Git to S3

The team also moved from a self-hosted Git provider to S3 buckets as the source of truth for their clusters. Driven by high availability and compliance requirements, they built a mechanism to push artifacts from CI to S3. Because Flux’s Source Controller supports various sources - Git, Helm repositories, OCI Repositories, and S3-compatible buckets - this transition was possible. The GitOps Toolkit architecture makes this kind of swap straightforward. You change the source layer but keep the delivery pipeline.

Step 3: Observability and Feedback Loops

Managing 500 clusters requires effective observability. The team built a centralized Grafana dashboard providing a unified view of their fleet.

They extended the open-source Flux dashboards with custom metrics from kube-state-metrics, tailored to their developers’ needs. At a glance, they could see which reconciliations were failing and why.

They also closed the developer experience loop by integrating Flux’s Notification Controller - sending success and failure notifications directly to the pipelines and tools developers were already using.

Looking Ahead

The team also shared what’s next on their roadmap:

Flux Sharding: Exploring sharding Flux controllers to distribute load across multiple instances within a cluster.
OCI Artifacts: Considering OCI artifacts as the primary source of truth, aligning with the “Git-less GitOps” model for improved performance and security.
Progressive Delivery: Planning to adopt Flagger for canary and blue-green deployments, helping de-risk releases.

It’s cool to see a team that’s been running Flux for five years still finding new ways to push it further. This is a sophisticated environment, and these improvements could win some performance and improve their developer experience further.

Watch the Full Talk

For the full story, including the architectural decisions and lessons learned, watch the recording:

Thank you to Tiffany, Simon, and the team at Morgan Stanley for sharing their journey so openly. Stories like theirs remind us why we build Flux - what we build for the Raspberry Pi’s in our closets at home is the same software that is so widely deployed all around us at enterprise scale. We can’t help but wonder what wild stories we’ll hear from you all next week at FluxCon and KubeCon!

Join Us at FluxCon Europe 2026

Inspired by Morgan Stanley’s infra? Come connect with the community and learn from teams running Flux in production. FluxCon Europe is happening on March 23, 2026 at RAI Amsterdam, co-located with KubeCon. Speakers from KLM, NatWest Group, Orange, and more will be sharing their Flux stories.

We’d love to see you there – come say hi! 🙂 We’ll also be in the Project Pavilion all week. Catch up with us at fluxcd.io/kubecon 👋

Blog: Announcing Flux 2.8 GA

Tue, 24 Feb 2026 11:00:00 +0000

We are thrilled to announce the release of Flux v2.8.0! In this post, we highlight some of the new features and improvements included in this release.

Highlights

Flux v2.8 comes with Helm v4 support, bringing server-side apply and enhanced health checking to Helm releases. Big thanks to the Helm maintainers for their work on Helm v4 and for collaborating with us to ensure a smooth integration with Flux!

In this release, we have also introduced several new features to the Flux controllers:

Reduced the mean time to recovery of Flux-managed applications
Readiness evaluation of Helm-managed objects with CEL expressions
ArtifactGenerator support for extracting and modifying Helm charts
Support for commenting on Pull Requests directly from Flux notifications
Custom SSA apply stages for ordering resource application in kustomize-controller
Automatic GitHub App installation ID lookup from the repository owner
Support for Cosign v3 for verifying OCI artifacts and container images

In ecosystem news, there is a new release of Flux Operator that comes with a dedicated Flux Web UI and new providers for preview environments.

Helm v4 Support

Flux now ships with Helm v4. This brings two major improvements to how Flux manages Helm releases: server-side apply and kstatus-based health checking.

With server-side apply, the API server takes ownership of merging fields, rather than the client. This means fewer conflicts when multiple controllers or tools manage overlapping resources, and more accurate drift detection out of the box.

Health checking now defaults to kstatus, the same library used by the kustomize-controller. Instead of relying on Helm’s legacy readiness logic, Flux can now understand the actual rollout status of Deployments, StatefulSets, Jobs, and other resources — including custom resources that follow standard status conventions. For teams that rely on custom readiness logic, Flux now supports CEL-based health check expressions on HelmReleases, giving you the same flexibility already available in the Kustomization API.

Both server-side apply and kstatus health checking are the new defaults. Because Helm persists the apply method in its release storage, existing HelmReleases will continue to use client-side apply until explicitly opted in. Health checking, on the other hand, will switch to kstatus for all HelmReleases. For teams that prefer Helm v3’s behavior across the board, the UseHelm3Defaults feature gate restores the previous defaults.

Finally, HelmReleases now track an inventory of managed resources in .status.inventory, giving you full visibility into what Flux has deployed — useful for debugging, auditing, and building tooling on top of Flux.

Faster recovery from failed deployments

A common pain point with GitOps is the wait time after pushing a fix for a broken deployment. When a release fails health checks, Flux would previously wait for the full timeout before acting — even if a new revision was already available. This delay directly impacts the mean time to recovery (MTTR).

In Flux 2.7, the kustomize-controller introduced the CancelHealthCheckOnNewRevision feature gate, allowing ongoing health checks to be canceled when a new source revision is detected. With Flux 2.8, this capability has been extended to helm-controller and expanded to react to more kinds of changes:

Changes in the resource spec (e.g. path, patches, images, values)
Changes in referenced ConfigMaps and Secrets (var substitutions, SOPS decryption keys, Kubeconfig)
Reconciliations triggered manually with flux reconcile or via notification-controller receivers

In all these cases, Flux cancels the ongoing health checks and immediately starts reconciling the new state. Instead of waiting several minutes for a failing release to time out, the fix is picked up as soon as it lands.

For observability, a new HealthCheckCanceled reason is added to the Ready condition when this happens.

This feature gate is opt-in for now, and we plan to enable it by default once the implementation is stable across both controllers.

Note

When enabling CancelHealthCheckOnNewRevision for helm-controller, enabling DefaultToRetryOnFailure together is recommended. HelmReleases are more prone to get stuck after the cancellation when using the default retry configuration (no retries).

Ecosystem News

Flux Operator Web UI

At KubeCon Atlanta 2025, the Flux maintainers from ControlPlane gave a sneak peek of the new Flux Web UI, which is now available in the latest release of Flux Operator.

The Flux Web UI provides a modern and user-friendly interface for managing and monitoring your Flux-managed clusters. It offers a comprehensive view of your GitOps resources, including:

Cluster dashboard with sync statistics and overall system health
Deep-dive views for ResourceSets, HelmReleases and Kustomizations
Workload monitoring from deployment rollouts to pod statuses
Powerful search and filtering
Favorites for quick access to critical resources
SSO support via OIDC & Kubernetes RBAC for multi-tenant clusters
GitOps Graphs for visualizing the delivery pipeline
GitOps Actions guarded by RBAC for manual interventions and incident response

Get started by installing the latest version of Flux Operator and following the Flux Web UI documentation.

Preview environments

Flux Operator’s ResourceSet API makes it easy to deploy ephemeral preview environments from GitHub Pull Requests and GitLab Merge Requests. With Flux 2.8, closing the feedback loop on these environments is now much simpler thanks to new notification-controller provider types: githubpullrequestcomment, gitlabmergerequestcomment and giteapullrequestcomment.

Previously, posting deployment status on a Pull Request required setting up a githubdispatch provider and a GitHub Actions workflow to parse the event payload and post a comment. With the new providers, notification-controller posts and updates comments directly on the PR or MR page — no CI workflow needed. Comments are automatically deduplicated, so the PR stays clean with a single status comment that gets updated on each deployment.

In addition, commit status reporting now works with any Flux API — not just Kustomizations and GitRepositories. This means HelmReleases deployed for preview environments can now report their status as commit checks on the PR, giving developers immediate visibility into whether their changes deployed successfully.

To annotate your Flux resources for these providers, use the standard event metadata keys:

event.toolkit.fluxcd.io/change_request — identifies the PR/MR number for comment providers
event.toolkit.fluxcd.io/commit — identifies the commit SHA for commit status providers

For complete setup guides, see the Flux Operator documentation:

Supported Versions

Flux v2.5 has reached end-of-life and is no longer supported.

Flux v2.8 supports the following Kubernetes versions:

Distribution	Versions
Kubernetes	1.33, 1.34, 1.35
OpenShift	4.20

Enterprise support Note that the CNCF Flux project offers support only for the latest three minor versions of Kubernetes. Backwards compatibility with older versions of Kubernetes and OpenShift is offered by vendors such as ControlPlane that provide enterprise support for Flux.

Upgrade Procedure

Note that in Flux v2.8, the following APIs have reached end-of-life and have been removed from the CRDs:

source.toolkit.fluxcd.io/v1beta2
kustomize.toolkit.fluxcd.io/v1beta2
helm.toolkit.fluxcd.io/v2beta2

Before upgrading to Flux v2.8, make sure to migrate all your resources to the stable APIs using the flux migrate command.

Upgrade Procedure for Flux v2.8

We have published a dedicated step-by-step upgrade guide, please follow the instructions from Upgrade Procedure for Flux v2.7+.

Over and out

If you have any questions or simply just like what you read and want to get involved, here are a few good ways to reach us:

Join our upcoming dev meetings.
Talk to us in the #flux channel on CNCF Slack.
Join the planning discussions.
Follow Flux on Twitter, or join the Flux LinkedIn group.

Blog: Announcing Flux 2.7 GA

Tue, 30 Sep 2025 06:00:00 +0000

We are thrilled to announce the release of Flux v2.7.0! In this post, we highlight some of the new features and improvements included in this release.

Highlights

Flux v2.7 marks the General Availability (GA) of the image update automation features and comes with new APIs ExternalArtifact and ArtifactGenerator for advanced source composition and decomposition patterns.

In this release, we have also introduced several new features to the Flux controllers, including watching for changes in ConfigMaps and Secrets references, extended readiness evaluation of dependencies with CEL expressions, and support for OpenTelemetry tracing for Flux Kustomization and HelmRelease reconciliation.

In ecosystem news, there is a new release of Flux Operator that comes with in-cluster image update automation features, that can be used for GitLess GitOps workflows.

General availability of Image Update Automation

This release marks the General Availability (GA) of Flux Image Automation APIs and controllers. The image-reflector-controller and image-automation-controller work together to update Kubernetes manifests in Git repositories when new container images are available in container registries.

The following APIs have been promoted to stable v1:

The ImagePolicy API now supports the .spec.suspend field to pause and resume the policy evaluation.

The ImageUpdateAutomation API gains support for Git sparse checkout. To enable this optimization, the image-automation-controller can be configured with the --feature-gates=GitSparseCheckout=true flag.

In addition, the image-automation-controller can now be configured to use Kubernetes Workload Identity for authenticating with AzureDevOps repositories.

Breaking changes:

The image-reflector-controller autologin flags which were deprecated since 2023 are now removed. Users should set ImageRepository.spec.provider to the appropriate cloud provider for their container registry.
The ImageUpdateAutomation commit template fields .Updated and .Changed.ImageResult which were deprecated since 2024 are now removed. Users should migrate to:
- .Changed.FileChanges for detailed change tracking
- .Changed.Objects for object-level changes
- .Changed.Changes for a flat list of changes

Watching for changes in ConfigMaps and Secrets

Starting with Flux v2.7, the kustomize-controller, helm-controller and notification-controller gain support for reacting to changes in ConfigMaps and Secrets references.

The following references are now watched for changes:

Kustomization.spec.postBuild.substituteFrom
Kustomization.spec.decryption.secretRef
Kustomization.spec.kubeConfig.secretRef
Kustomization.spec.kubeConfig.configMapRef
HelmRelease.spec.valuesFrom
HelmRelease.spec.kubeConfig.secretRef
HelmRelease.spec.kubeConfig.configMapRef
Receiver.spec.secretRef

When a referenced ConfigMap or Secret changes, the controller will immediately trigger a reconciliation if the referenced object is labelled with reconcile.fluxcd.io/watch: Enabled.

To enable the watching of all referenced objects without the need to label them, the controllers can be configured with the --watch-configs-label-selector=owner!=helm flag.

Workload Identity Authentication for Remote Clusters

Starting with Flux v2.7, you can configure workload identity at the object level in the Kustomization and HelmRelease resources to authenticate with cloud providers when running Flux in the hub-and-spoke model.

This feature allows cluster admins to use cloud identities on the hub cluster to configure Flux authentication to spoke clusters, without the need to create and manage static kubeconfig Secrets.

For more details on how to configure secret-less authentication to remote clusters, please refer to the following guides:

Object-level Workload Identity

In Flux v2.7, we have completed the integration of Kubernetes Workload Identity at the object level for all Flux APIs that support authentication with cloud providers.

This includes the following resources:

Bucket.spec.serviceAccountName for authenticating with AWS S3, Azure Blob Storage and Google Cloud Storage.
GitRepository.spec.serviceAccountName for authenticating with Azure DevOps.
OCIRepository.spec.serviceAccountName for authenticating with AWS ECR, Azure Container Registry and Google Artifact Registry.
ImageRepository.spec.serviceAccountName for authenticating with AWS ECR, Azure Container Registry and Google Artifact Registry.
Kustomization.spec.decryption.serviceAccountName for authenticating with AWS KMS, Azure Key Vault and Google KMS.
Kustomization.spec.kubeConfig.configMapRef.name for authenticating with remote clusters on AWS EKS, Azure AKS and Google GKE.
HelmRelease.spec.kubeConfig.configMapRef.name for authenticating with remote clusters on AWS EKS, Azure AKS and Google GKE.
Provider.spec.serviceAccountName for authenticating with Azure DevOps, Azure Event Hub and Google Pub/Sub.

For more details on how to configure object-level workload identity for Flux, see the following docs:

OpenTelemetry Tracing

Starting with Flux v2.7, users can enable OpenTelemetry tracing for Flux reconciliations by configuring a Provider of type otel:

apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
 name: jaeger
 namespace: flux-system
spec:
 type: otel
 address: http://jaeger-collector.jaeger:4318/v1/traces

The notification-controller converts Flux events into OTEL spans with proper trace relationships based on the Flux object hierarchy. Source objects (GitRepository, HelmChart, OCIRepository, Bucket) create root spans, while Kustomization and HelmRelease objects create child spans within the same trace. Each span includes event metadata as attributes and uses the alert name and namespace as the service identifier.

For more details on how to configure OpenTelemetry tracing for Flux, please refer to the notification-controller documentation.

Controller Improvements

The GitRepository API gains support for mTLS in GitHub App authentication.
The Kustomization API now supports CEL expressions for extended readiness evaluation of dependencies.
The Kustomization API gains a new field .spec.ignoreMissingComponents for ignoring missing Kustomize components in the source.
The kustomize-controller now supports global SOPS decryption for Age keys, allowing centralized management of decryption keys.
The kustomize-controller can be configured to cancel ongoing health checks when a new source revision is detected with the --feature-gates=CancelHealthCheckOnNewRevision=true flag.
The HelmRelease API now supports CEL expressions for extended readiness evaluation of dependencies.
The HelmRelease API gains a new strategy called RetryOnFailure for better handling of release failures.
The Provider API now supports setting proxy via spec.proxySecretRef and mTLS via spec.certSecretRef.
The Provider API has been extended with support for Zulip and OpenTelemetry tracing.

CLI Improvements

The flux bootstrap and flux install commands now support the --components-extra=source-watcher flag to enable the new source-watcher component.
A new flux migrate command has been added to migrate Flux resources stored in Kubernetes etcd to their latest API version.
The flux debug command gains a new --show-history flag to display the reconciliation history of Flux objects.
The flux diff command now handles the kustomize.toolkit.fluxcd.io/force: Enabled annotation.
The flux create hr command gains a new --storage-namespace flag for changing the namespace of Helm storage objects.
New commands were added for ImagePolicy resources:
- flux reconcile image policy
- flux suspend image policy
- flux resume image policy
New commands were added for ArtifactGenerator resources:
- flux get artifact generator
- flux export artifact generator
- flux tree artifact generator
- flux events --for ArtifactGenerator/<name>

Artifact Generators

Flux v2.7 comes with a new component that can be enabled at bootstrap time with the --components-extra=source-watcher flag.

The source-watcher controller implements the ArtifactGenerator API which allows Flux users to:

Compose multiple Flux sources (GitRepository, OCIRepository, Bucket) into a single deployable artifact
Decompose monorepos into multiple independent artifacts with separate deployment lifecycles
Optimize reconciliation by only triggering updates when specific paths change
Structure complex deployments from distributed sources maintained by different teams

Multiple Source Composition

The ArtifactGenerator can be used to combine multiple sources into a single deployable artifact, for example, you can combine upstream Helm charts from OCI registries with your organization’s custom values and configuration overrides stored in Git:

apiVersion: source.extensions.fluxcd.io/v1beta1
kind: ArtifactGenerator
metadata:
 name: podinfo
 namespace: apps
spec:
 sources:
 - alias: chart
 kind: OCIRepository
 name: podinfo-chart
 - alias: repo
 kind: GitRepository
 name: podinfo-values
 artifacts:
 - name: podinfo-composite
 originRevision: "@chart"
 copy:
 - from: "@chart/"
 to: "@artifact/"
 - from: "@repo/charts/podinfo/values.yaml"
 to: "@artifact/podinfo/values.yaml"
 strategy: Overwrite
 - from: "@repo/charts/podinfo/values-prod.yaml"
 to: "@artifact/podinfo/values.yaml"
 strategy: Merge
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
 name: podinfo
 namespace: apps
spec:
 interval: 15m
 releaseName: podinfo
 chartRef:
 kind: ExternalArtifact
 name: podinfo-composite

Monorepo Decomposition

The ArtifactGenerator can be used to decompose a monorepo into multiple independent artifacts with separate deployment lifecycles. For example:

apiVersion: source.extensions.fluxcd.io/v1beta1
kind: ArtifactGenerator
metadata:
 name: app-decomposer
 namespace: apps
spec:
 sources:
 - alias: git
 kind: GitRepository
 name: monorepo
 artifacts:
 - name: frontend
 originRevision: "@git"
 copy:
 - from: "@git/deploy/frontend/**"
 to: "@artifact/"
 - name: backend
 originRevision: "@git"
 copy:
 - from: "@git/deploy/backend/**"
 to: "@artifact/"
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
 name: frontend-service
 namespace: apps
spec:
 interval: 15m
 prune: true
 sourceRef:
 kind: ExternalArtifact
 name: frontend
 path: ./
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
 name: backend-service
 namespace: apps
spec:
 interval: 15m
 prune: true
 sourceRef:
 kind: ExternalArtifact
 name: backend
 path: ./

Each service gets its own ExternalArtifact with an independent revision. Changes to deploy/backend/ only trigger the reconciliation of the backend-service Kustomization, leaving other services untouched.

For more details on how to use the ArtifactGenerator API, please refer to the source-watcher documentation.

Supported Versions

Flux v2.4 has reached end-of-life and is no longer supported.

Flux v2.7 supports the following Kubernetes versions:

Distribution	Versions
Kubernetes	1.32, 1.33, 1.34
OpenShift	4.19

Enterprise support Note that the CNCF Flux project offers support only for the latest three minor versions of Kubernetes. Backwards compatibility with older versions of Kubernetes and OpenShift is offered by vendors such as ControlPlane that provide enterprise support for Flux.

Upgrade Procedure

Note that in Flux v2.7, the following APIs have reached end-of-life and have been removed from the CRDs:

source.toolkit.fluxcd.io/v1beta1
kustomize.toolkit.fluxcd.io/v1beta1
helm.toolkit.fluxcd.io/v2beta1
image.toolkit.fluxcd.io/v1beta1
notification.toolkit.fluxcd.io/v1beta1

Before upgrading to Flux v2.7, make sure to migrate all your resources to the stable APIs using the flux migrate command.

Upgrade Procedure for Flux v2.7+

We have published a dedicated step-by-step upgrade guide, please follow the instructions from Upgrade Procedure for Flux v2.7+.

Over and out

If you have any questions or simply just like what you read and want to get involved, here are a few good ways to reach us:

Join our upcoming dev meetings.
Talk to us in the #flux channel on CNCF Slack.
Join the planning discussions.
Follow Flux on Twitter, or join the Flux LinkedIn group.

Blog: Time-based deployments with Flux Operator

Mon, 07 Jul 2025 12:00:00 +0000

We are thrilled to announce time-based deployments, a feature long-awaited by Flux users, in Flux Operator v0.23.0!

Organizations using Flux for GitOps deployments frequently require sophisticated control over when changes are applied to production systems, particularly in regulated industries or critical business environments. Key requirements include adhering to Change Advisory Board (CAB) approval windows, enforcing “No Deploy Fridays” policies, and restricting deployments during peak business hours to ensure service stability.

Maintenance windows become critical when managing helm upgrades, where teams need to skip reconciliation unless the current time falls within a specified interval. In regulated environments like medical device companies, automated deployments must be controlled to prevent unexpected disruptions during critical operational periods. Large telecommunications providers and ISVs managing multiple client clusters need gating mechanisms to control application rollouts, allowing tenants to consume platform updates when ready.

In this post, we show how to use time-based deployment with Flux Operator ResourceSets.

How it works

The Flux Operator ResourceSet API allows defining bundles of Flux objects by templating a set of resources with inputs provided by the ResourceSetInputProvider API.

The ResourceSetInputProvider API allows pulling inputs from external sources, such as GitHub pull requests, branches and tags. For example, on the reconciliation of a ResourceSetInputProvider of type GitHubTag, the operator will list the tags of a GitHub repository, filter them according to a semver range, and export a set of inputs for each matching tag in the ResourceSetInputProvider .status.exportedInputs field. For example:

status:
 exportedInputs:
 - id: "48955639"
 tag: "6.0.4"
 sha: 11cf36d83818e64aaa60d523ab6438258ebb6009

Starting with Flux Operator v0.23.0, the ResourceSetInputProvider API now has the field .spec.schedule, which allows defining a cron-based schedule for the reconciliation of the ResourceSetInputProvider. For example:

spec:
 schedule:
 # Every day-of-week from Monday through Thursday
 # between 10:00 to 16:00
 - cron: "0 10 * * 1-4"
 timeZone: "Europe/London"
 window: "6h"
 # Every Friday from 10:00 to 13:00
 - cron: "0 10 * * 5"
 timeZone: "Europe/London"
 window: "3h"

With this configuration, reconciliations of the ResourceSetInputProvider object would only be allowed to run within the specified time windows. When the window is active, the reconciliation happens normally, according to the interval defined in the fluxcd.controlplane.io/reconcileEvery annotation.

A complete example

Define a ResourceSetInputProvider: This provider will scan a Git branch or tag for changes and export the commit SHA as an input.
Configure schedule: The provider will have a reconciliation schedule that defines when it should check for changes in the Git repository.
Define a ResourceSet: The ResourceSet will use the inputs from the provider to create a GitRepository and Kustomization that deploys the application at the specified commit SHA.

ResourceSetInputProvider Definition

Assuming the Kubernetes deployment manifests for an application are stored in a Git repository, you can define a input provider that scans a branch for changes and exports the commit SHA:

apiVersion: fluxcd.controlplane.io/v1
kind: ResourceSetInputProvider
metadata:
 name: my-app-main
 namespace: apps
 labels:
 app.kubernetes.io/name: my-app
 annotations:
 fluxcd.controlplane.io/reconcileEvery: "10m"
 fluxcd.controlplane.io/reconcileTimeout: "1m"
spec:
 schedule:
 - cron: "0 8 * * 1-5"
 timeZone: "Europe/London"
 window: 8h
 type: GitHubBranch # or GitLabBranch
 url: https://github.com/my-org/my-app
 secretRef:
 name: gh-app-auth
 filter:
 includeBranch: "^main$"
 defaultValues:
 env: "production"

For when Git tags are used to version the application, you can define an input provider that scans the Git tags and exports the latest tag according to a semantic versioning:

apiVersion: fluxcd.controlplane.io/v1
kind: ResourceSetInputProvider
metadata:
 name: my-app-release
 namespace: apps
 labels:
 app.kubernetes.io/name: my-app
 annotations:
 fluxcd.controlplane.io/reconcileEvery: "10m"
 fluxcd.controlplane.io/reconcileTimeout: "1m"
spec:
 schedule:
 - cron: "0 8 * * 1-5"
 timeZone: "Europe/London"
 window: 8h
 type: GitHubTag # or GitLabTag
 url: https://github.com/my-org/my-app
 secretRef:
 name: gh-auth
 filter:
 semver: ">=1.0.0"
 limit: 1

ResourceSet Definition

The exported inputs can then be used in a ResourceSet to deploy the application using the commit SHA from the input provider:

apiVersion: fluxcd.controlplane.io/v1
kind: ResourceSet
metadata:
 name: my-app
 namespace: apps
spec:
 inputsFrom:
 - kind: ResourceSetInputProvider
 selector:
 matchLabels:
 app.kubernetes.io/name: my-app
 resources:
 - apiVersion: source.toolkit.fluxcd.io/v1
 kind: GitRepository
 metadata:
 name: my-app
 namespace: << inputs.provider.namespace >>
 spec:
 interval: 12h
 url: https://github.com/my-org/my-app
 ref:
 commit: << inputs.sha >>
 secretRef:
 name: gh-auth
 sparseCheckout:
 - deploy
 - apiVersion: kustomize.toolkit.fluxcd.io/v1
 kind: Kustomization
 metadata:
 name: my-app
 namespace: << inputs.provider.namespace >>
 spec:
 interval: 30m
 retryInterval: 5m
 prune: true
 wait: true
 timeout: 5m
 sourceRef:
 kind: GitRepository
 name: my-app
 path: deploy/<< inputs.env >>

When the ResourceSetInputProvider runs according to its schedule, if it finds a new commit, the ResourceSet will be automatically updated with the new commit SHA which will trigger an application deployment for the new version.

Blog: FluxCon NA 2025

Thu, 05 Jun 2025 12:00:00 +0000

This year at KubeCon NA in Atlanta, we’ll be hosting the first-ever FluxCon! We’re so excited to have a space specifically for Flux end-users to share their stories. FluxCon will be taking place on November 10th, 2025.

We’ve added a number of new features to Flux and continue to stabilize API’s for a stable, scalable lifetime. FluxCon is a great way to learn about new use-cases, hear about Flux at scale, and connect with other Flux practitioners.

Call for Papers

Are you using Flux in your org? We’d love to hear your story. You can submit your talk proposal before June 30th, 2025.

New Use-Cases

Flux is used in retail stores, massive datacenters, trains, cell-towers, satellites, tractors, and so many more places. Practitioners choose Flux because it’s performant, flexible, and secure. Over the years, we’ve heard so many use-cases for Flux’s uniquely extensible continuous delivery API’s.

This year we’re advocating for Gitless GitOps, experimenting with AI-assisted GitOps, and showing object-level identity for external API’s. The ecosystem of projects innovating around Flux is also healthy. The Headlamp project builds UI tools for Flux users, and there are a number of vendors offering SaaS and support for Flux in their products. We can’t wait to hear what people are doing next at FluxCon.

Connect with the Community

The most important part of the Flux project is our community. Yes, our software is beautiful, and simple, and incredibly principled, but we build Flux purely for the love of our community of practitioners. We created GitOps to change the way we work, and meeting each other and sharing our stories is the only way to do so.

You can expect to meet other users, talk directly with Flux maintainers (represented by multiple companies), and hear more about how we’re moving GitOps forward.

Please consider this a warm invitation to come join us at FluxCon this November. We’ll see you there!

Blog: Announcing Flux 2.6 GA

Thu, 29 May 2025 12:00:00 +0000

We are thrilled to announce the release of Flux v2.6.0! In this post, we will highlight some of the new features and improvements included in this release.

Highlights

Flux v2.6 marks the General Availability (GA) of the Flux Open Container Initiative (OCI) Artifacts features. The OCI artifacts support was first introduced in 2022, and since then we’ve been evolving Flux towards a Gitless GitOps model. In this model, the Flux controllers are fully decoupled from Git, relying solely on container registries as the source of truth for the desired state of Kubernetes clusters.

In the last couple of years, the OCI feature-set has matured, and we’ve seen major financial institutions and enterprises adopting Flux and OCI as their preferred way of managing production deployments. To see it in action, you can check the reference architecture guide made by ControlPlane on how highly regulated industries can securely implement Gitless GitOps with Flux and OCI.

In this release, we have also introduced several new features to the Flux controllers, including digest pinning in image automation, object-level workload identity for container registries and KMS services authentication, and various improvements to notifications.

In ecosystem news, there is a new release of Flux Operator that comes with a Model Context Protocol (MCP) implementation for allowing AI assistants to interact with Flux. For more details on the Flux MCP Server, see the AI-Assisted GitOps blog post.

General availability of Flux OCI Artifacts

This release marks the General Availability (GA) of Flux OCIRepository API, which allows storing the desired state of Kubernetes clusters in OCI container registries.

The OCIRepository v1 API comes with new features including:

Support for Object-Level Workload Identity, which allows Flux to use different cloud identities for accessing container registries on multi-tenant clusters.
Caching of registry credentials for cloud providers, which allows Flux to reuse the OIDC tokens for subsequent requests to the same registry, reducing the number of authentication requests.

The OCIRepository v1 API is backward compatible with the previous v1beta2 API, users can upgrade by changing the apiVersion in the YAML files that contain OCIRepository definitions from source.toolkit.fluxcd.io/v1beta2 to source.toolkit.fluxcd.io/v1.

The Flux CLI commands for working with OCI artifacts have been promoted to stable:

flux build artifact
flux push artifact
flux pull artifact
flux tag artifact
flux diff artifact
flux list artifacts

The Flux custom media types used for OCI artifacts produced by the Flux CLI are now stable:

config media type application/vnd.cncf.flux.config.v1+json
content media type application/vnd.cncf.flux.content.v1.tar+gzip

Breaking changes

Prior to v2.6.0, the OCIRepository and ImageRepository APIs allowed the spec.provider field to be set to a value that did not necessarily match the repository URL. In these cases the controllers would simply ignore the spec.provider, not configuring OIDC authentication for the repository.

For example, the repository public.ecr.aws/aws-controllers-k8s never matched Flux’s regular expression for the aws provider, but the controller would still allow the spec.provider to be set to aws in this case and would simply ignore it. This specific configuration would work correctly because this particular repository is public and does not require authentication.

Similarly, a private repository that did not match any of Flux’s validations for the three container registry providers (aws, azure, gcp) would also work with the spec.provider set to one of these values, as long as it was also configured with one of the spec.secretRef or spec.serviceAccountName fields for using image pull secrets. In these cases, the controller would simply ignore the spec.provider and use the image pull secret instead.

Starting with v2.6.0, Flux is fixing this behavior. The repository URL must now match the provider set in spec.provider, otherwise the controller will reject the configuration and return an error. For automatic OIDC authentication, the spec.provider must be set to one of the three container registry providers (aws, azure, gcp). For public repositories or authentication using image pull secrets, the spec.provider must not be set, or set to generic. These configuration instructions were explicit in the Flux docs since many releases, but are only now in v2.6.0 being strictly enforced by the controllers.

Image Automation Digest Pinning

In Flux v2.6, the image automation has been enhanced to support digest pinning for container images. This feature allows users to configure the ImagePolicy to track the latest digest of a container image, and the ImageUpdateAutomation to update the manifests in the Git repository with the new digest.

The ImagePolicy can now be configured to select the latest image digest with .spec.digestReflectionPolicy set to Always. Once a policy is set to track the latest digest, the manifests in the Git repository will be updated with digest references in the format <registry>/<name>:<tag>@<digest>.

A new marker has been introduced to allow setting the digest in custom resources where repository, tag and digest are separate values:

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
 name: my-app
 namespace: apps
spec:
 values:
 image:
 repository: docker.io/my-org/my-app # {"$imagepolicy": "flux-system:my-app:name"}
 tag: latest  # {"$imagepolicy": "flux-system:my-app:tag"}
 digest: sha256:ec0119... # {"$imagepolicy": "flux-system:my-app:digest"}

For more details on how to configure image automation digest pinning, see the following guide.

Object-level Workload Identity

Starting with Flux v2.6, you can configure workload identity at the object level in the Kustomization API for SOPS decryption with KMS services, and in the OCIRepository and ImageRepository APIs for accessing container registries.

This feature allows cluster admins to use different cloud identities on multi-tenant clusters. Instead of relying on static Secrets that require manual rotation, you can now assign cloud identities per tenant by leveraging Kubernetes Workload Identity.

To use this feature, cluster admins have to enable the feature gate ObjectLevelWorkloadIdentity which is opt-in from Flux v2.6.

For more details on how to configure object-level workload identity for Flux, see the following docs:

GitHub App Authentication

In Flux v2.6, we have completed the integration of GitHub App authentication for Git repositories. This feature was introduced in Flux v2.5, and it is now fully supported across all Flux APIs.

The GitHub App authentication tokens are now cached by the Flux controllers and reused for subsequent requests for the duration of the token lifetime.

The notification-controller has also been updated to support GitHub App authentication when updating Git commit statuses and for triggering GitHub Actions workflows.

Notifications Improvements

Starting with Flux v2.6, users can customize the Git commit status identifier in the notifications sent to Git providers by using Common Expression Language (CEL) expressions.

apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
 name: github-status
 namespace: flux-system
spec:
 type: github
 address: https://github.com/my-gh-org/my-gh-repo
 secretRef:
 name: github-app-auth
 commitStatusExpr: "(event.involvedObject.kind + '/' + event.involvedObject.name + '/' + event.metadata.clusterName)"

Customizing the commit status ID is particularly useful when using a monorepo for a fleet of Kubernetes clusters, as it allows you to differentiate the commit statuses for each cluster.

Other improvements include:

The notification-controller can now use Azure Workload Identity when sending notifications to Azure Event Hub.
The github and githubdispatch providers now support authenticating with a GitHub App.

Controller Improvements

The GitRepository v1 API now supports sparse checkout by setting a list of directories in the .spec.sparseCheckout field. This allows for optimizing the amount of data fetched from the Git repository.
The GitRepository v1 API gains supports mTLS authentication for HTTPS Git repositories.
The Kustomization v1 API now supports the value WaitForTermination for the .spec.deletionPolicy field. This instructs the controller to wait for the deletion of all resources managed by the Kustomization before allowing the Kustomization itself to be deleted.
The helm-controller v1.3.0 comes with a new feature gate called DisableChartDigestTracking, which allows disabling appending the digest of OCI Helm charts to the chart version. This is useful for charts that do not follow Helm’s recommendation of using the app version instead of the chart version as a label in the manifests.

Supported Versions

Flux v2.3 has reached end-of-life and is no longer supported.

Flux v2.6 supports the following Kubernetes versions:

Distribution	Versions
Kubernetes	1.31, 1.32, 1.33
OpenShift	4.18

Enterprise support Note that the CNCF Flux project offers support only for the latest three minor versions of Kubernetes. Backwards compatibility with older versions of Kubernetes and OpenShift is offered by vendors such as ControlPlane that provide enterprise support for Flux.

Over and out

If you have any questions or simply just like what you read and want to get involved, here are a few good ways to reach us:

Join our upcoming dev meetings.
Talk to us in the #flux channel on CNCF Slack.
Join the planning discussions.
Follow Flux on Twitter, or join the Flux LinkedIn group.

Blog: AI-Assisted GitOps with Flux Operator MCP Server

Wed, 14 May 2025 12:00:00 +0000

In this blog post, we introduce the Flux MCP Server, a new component of the Flux Operator project that connects AI assistants directly to your Kubernetes clusters, enabling seamless interaction through natural language. It serves as a bridge between AI tools and your GitOps pipelines, allowing you to analyze the cluster state, troubleshoot deployment issues, and perform operations using conversational prompts.

Bringing AI to GitOps

The GitOps movement started with the Flux community back in 2016, and since then, it has gained immense popularity in the Kubernetes ecosystem as a way to manage infrastructure and application deployments declaratively. But as the GitOps pipelines grow in complexity, so does the cognitive load required to troubleshoot issues, understand resource relationships, and perform routine operations.

That’s where the Flux MCP Server comes in. By connecting AI assistants to Kubernetes clusters and the desired state in Git, it allows operators to:

Debug GitOps pipelines end-to-end from Flux resources to application logs
Get accurate root cause analysis for failed deployments
Compare Flux configurations and Kubernetes resources between clusters
Visualize Flux dependencies with diagrams generated from the cluster state
Instruct Flux to perform operations using conversational prompts
Get up-to-date information and recommendations using the latest Flux official docs

How It Works

The Flux MCP Server implements the Model Context Protocol (MCP), providing purpose-built tools that allow AI assistants to interact with your clusters. When you ask a question or make a request, the AI model uses these tools to gather information, analyze configurations, and even perform operations based on your instructions.

The AI assistants leveraging the Flux MCP Server can trace issues from high-level GitOps resources like ResourceSets, HelmReleases, and Kustomizations all the way down to Kubernetes deployments and pod logs.

In addition, the MCP Server enables the AI to search the Flux documentation and provide accurate, up-to-date guidance based on the latest features and best practices, rather than relying solely on its training data.

Getting Started

Setting up the Flux MCP Server is straightforward. The server is written in Go and statically compiled as a single binary with no external dependencies.

You can install it using Homebrew:

brew install controlplaneio-fluxcd/tap/flux-operator-mcp

Alternatively, you can download pre-built binaries for Linux, macOS, and Windows, for more details refer to the installation guide.

Once installed, you can configure your AI assistant to use the Flux MCP Server. For Claude, Cursor, Windsurf, or GitHub Copilot add the following configuration to the MCP settings:

{
 "flux-operator-mcp":{
 "command":"flux-operator-mcp",
 "args":["serve"],
 "env":{
 "KUBECONFIG":"/path/to/.kube/config"
 }
 }
}

Make sure to replace /path/to/.kube/config with the absolute path to your kubeconfig file.

Setting Up AI Instructions

For the best experience with the Flux MCP Server, it’s crucial to provide your AI assistant with proper instructions on how to interact with Kubernetes clusters and the Flux resources. These instructions help the AI understand the context and make appropriate tool calls.

The Flux MCP Server comes with a set of predefined instructions that you can copy from the instructions.md file.

It’s recommended to enhance these instructions with information specific to your clusters, such as:

Kubernetes distribution details (EKS, GKE, AKS, etc.)
Cloud-specific services integrated with your clusters
Types of applications deployed
Secret management approaches

For detailed guidance on how to configure these instructions with different AI assistants, refer to the AI Instructions section of the documentation.

Practical Applications

Let’s look at some practical ways the Flux MCP Server can enhance your GitOps experience:

1. Quick Health Assessment

Instead of running multiple kubectl and flux commands to check the status of your GitOps pipeline, you can simply ask:

Analyze the Flux installation in my current cluster and report the status of all components and ResourceSets.

The AI assistant will gather information about your Flux Operator installation, controllers, and managed resources, providing a comprehensive health assessment.

2. GitOps Pipeline Visualization

Understanding the relationships between various GitOps resources can be challenging. The Flux MCP Server makes it easy:

List the Flux Kustomizations and draw a Mermaid diagram for the depends on relationship.

The AI will generate a visual representation of your GitOps pipeline, showing the dependency relationships between Flux Kustomizations and helping you understand the deployment order and potential bottlenecks.

3. Cross-Cluster Comparisons

When managing multiple environments, comparing configurations can be tedious. With Flux MCP Server:

Compare the podinfo HelmRelease between production and staging clusters.

The AI will switch contexts, gather the relevant information, and highlight the differences between the two environments.

3. Root Cause Analysis

When deployments fail, finding the root cause can involve digging through multiple resources and logs:

Perform a root cause analysis of the last failed Helm release in the frontend namespace.

The AI assistant will trace through dependencies, check resource statuses, analyze logs, and provide a detailed explanation of what went wrong and how to fix it.

4. GitOps Operations

You can even perform GitOps operations directly through natural language:

Resume all the suspended Flux resources in the current cluster and verify their status.

The AI will identify suspended resources, resume them, and report on the results.

5. Kubernetes Operations

The Flux MCP Server enables complex Kubernetes operations with simple instructions:

Create a namespace called test, then copy the podinfo Helm release and its source. Change the Helm values for ingress to test.podinfo.com

The AI will generate and apply the necessary Kubernetes resources, handling the details of creating namespaces, cloning Helm releases, and modifying configuration values - all through a single conversational request.

Security Considerations

As with any tool that interacts with your clusters, security should be a top priority. The Flux MCP Server includes several security features to ensure safe operations:

Operates with your existing kubeconfig permissions
Supports service account impersonation for limited access
Masks sensitive information in Kubernetes Secret values
Provides a read-only mode for observation without affecting the cluster state

For more details on security settings, please refer to the configuration guide.

The Future of AI-Assisted GitOps

The Flux MCP Server is currently an experimental feature, and it’s being actively developed based on user feedback and real-world use cases.

We plan to enhance the server with the following features in future releases:

Integration with Kubernetes metrics-server and other observability tools
Improved the documentation search capabilities
More advanced troubleshooting capabilities
Support for staged rollout/rollback of apps across clusters

All feedback is welcome, please reach out to us on GitHub Discussions.

Blog: GitHub App bootstrap with Flux Operator

Mon, 14 Apr 2025 12:00:00 +0000

In this blog post, we will showcase how Flux Operator can be used to bootstrap Kubernetes clusters using the GitHub App authentication method introduced in Flux 2.5.0.

Prior to Flux 2.5.0, the GitHub repository authentication methods were based on using a secret that is tied to a GitHub user, be it a personal access token (PAT) or an SSH deploy key. When the user leaves the organization, the GitHub deploy keys are revoked resulting in Flux losing access to all repositories. To restore access, the cluster administrators have to generate new GitHub deploy keys tied to a different user and rotate the secret in all clusters.

To avoid this situation, the recommendation was for organizations to create a dedicated GitHub user for Flux, but this is also not ideal since an extra user affects billing. The login credentials and MFA code have to be stored in an external secret management system like 1Password, increasing the complexity of the cluster bootstrap process.

Starting with Flux 2.5.0, the GitHub App authentication method allows organizations to create a GitHub App with access to the repositories from where Flux syncs the desired state of Kubernetes clusters. Instead of using the credentials of a GitHub user, Flux running on the clusters will use the GitHub App private key to authenticate with the GitHub API, acquiring a short-lived access token to perform Git operations.

Flux Operator

The Flux Operator offers an alternative to the Flux CLI bootstrap procedure. It removes the operational burden of managing Flux across fleets of clusters by fully automating the installation, configuration, and upgrade of the Flux controllers based on a declarative API called FluxInstance.

The FluxInstance custom resource defines the desired state of the Flux components and allows the configuration of the cluster state syncing from Git repositories, OCI artifacts and S3-compatible storage.

When using a GitHub repository as the source of truth, the Flux instance can be configured to use the GitHub App authentication method by referencing a Kubernetes secret that contains the app ID, the installation ID and the private key of the GitHub App.

What follows is a step-by-step guide on how to install the Flux Operator and bootstrap a cluster using the GitHub App authentication.

Bootstrap using Flux Operator and Helm

First, install the Flux Operator using the Helm chart:

helm install flux-operator oci://ghcr.io/controlplaneio-fluxcd/charts/flux-operator \
 --namespace flux-system \
 --create-namespace

Next, create a GitHub App secret using the flux CLI (see docs on how to create a GitHub App here):

flux create secret githubapp flux-system \
 --app-id=1 \
 --app-installation-id=2 \
 --app-private-key=./path/to/private-key-file.pem

Finally, bootstrap the cluster by creating a FluxInstance custom resource in the flux-system namespace:

apiVersion: fluxcd.controlplane.io/v1
kind: FluxInstance
metadata:
 name: flux
 namespace: flux-system
spec:
 distribution:
 version: "2.x"
 registry: "ghcr.io/fluxcd"
 components:
 - source-controller
 - kustomize-controller
 - helm-controller
 - notification-controller
 - image-reflector-controller
 - image-automation-controller
 cluster:
 type: kubernetes
 multitenant: false
 networkPolicy: true
 domain: "cluster.local"
 sync:
 kind: GitRepository
 provider: github
 url: "https://github.com/my-org/my-fleet.git"
 ref: "refs/heads/main"
 path: "clusters/my-cluster"
 pullSecret: "flux-system"

When the FluxInstance is applied on the cluster, the operator will automatically deploy the Flux controllers and configure them to sync the cluster state from the specified repository using GitHub App authentication. Similarly to the Flux CLI bootstrap, the operator generates a Flux GitRepository and Kustomization named flux-system that points to the clusters/my-cluster path inside the Git repository.

The Flux instance can be customized in various ways including multi-tenancy lockdown, sharding, horizontal and vertical scaling, persistent storage, and fine-tuning the Flux controllers with Kustomize patches. For more information on the available options, please refer to the Flux Operator documentation.

Bootstrap using Flux Operator and Terraform

Alternatively, you can use Terraform or OpenTofu to install the Flux Operator and the FluxInstance. A Terraform example is available in the Flux Operator repository.

The command for applying this Terraform example with a GitHub App would be the following:

export GITHUB_APP_PEM=`cat path/to/app.private-key.pem`

terraform apply \
 -var flux_version="2.x" \
 -var flux_registry="ghcr.io/fluxcd" \
 -var github_app_id="1" \
 -var github_app_installation_id="2" \
 -var github_app_pem="$GITHUB_APP_PEM" \
 -var git_url="https://github.com/my-org/my-fleet.git" \
 -var git_ref="refs/heads/main" \
 -var git_path="clusters/production"

GitHub App Docs

After installing your GitHub App in your organization you can find the installation ID like this:

Go to the Organization settings
Click on ‘GitHub Apps’ under ‘Third-party Access’
If there are multiple GitHub apps, choose your App and click on ‘Configure’
Once your GitHub App is selected check the URL for obtaining ‘GitHub App Installation ID’

The URL looks like this:

https://github.com/organizations/<Organization-name>/settings/installations/<ID>

Conclusion

Using the GitHub App authentication method with Flux Operator offers a more secure way of bootstrapping Flux on Kubernetes clusters, as it eliminates the need for managing GitHub user credentials and deploy keys. This approach ensures that Flux can continue to operate seamlessly even when users leave the organization or change their access permissions.

Migrating clusters that have been bootstrapped with the Flux CLI to the Flux Operator is a straightforward process. For more information on how to do this, please refer to the Flux Operator bootstrap migration guide.

Blog: Announcing Flux 2.5 GA

Thu, 20 Feb 2025 12:00:00 +0000

We are thrilled to announce the release of Flux v2.5.0! In this post, we will highlight some of the new features and improvements included in this release.

Highlights

Flux v2.5 marks a significant milestone in the project’s evolution, we have integrated Common Expression Language (CEL) with the Flux controllers to enable long-awaited features such as custom health checks and webhook receiver filters. Moreover, we have added support for GitHub App authentication, custom event metadata for notifications and Flux CLI helpers for troubleshooting Flux resources.

In ecosystem news, the Flux Operator v0.14 release brings one of the most requested features: deploy app code and/or config changes made in a GitHub Pull Request or GitLab Merge Request to an ephemeral environment for testing and validation.

The Flux Operator has the ability to create, update and delete application instances on-demand based on the ResourceSet definitions and Pull/Merge Requests state.

For more details on how to use the ephemeral environments feature, see the following guides:

Health Checks for Custom Resources

In this release, we have extended the Flux Kustomization API with support for defining custom health checks using Common Expression Language (CEL). The health checks are used to verify the readiness of the resources managed by Flux and are a key feature for ensuring that the desired state of the cluster is achieved.

While Flux performs a series of built-in health checks for Kubernetes core resources, the new feature allows users to teach Flux how to check the health of Kubernetes custom resources. This is particularly useful for custom resources that do not subscribe to the Kubernetes API conventions or for resources that require additional logic to determine if they reached the desired state.

A common use case for custom health checks is to verify the status of Cluster objects reconciled by the Cluster API controllers. When Flux is used to manage a fleet of Kubernetes clusters, the health checks can be used to ensure that the clusters are ready before deploying cluster addons and applications.

Example of a Kustomization with a custom health check for Cluster API:

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
 name: prod-clusters
 namespace: infra
spec:
 interval: 30m
 retryInterval: 5m
 prune: true
 sourceRef:
 kind: GitRepository
 name: fleet
 path: "./production"
 timeout: 15m
 wait: true
 healthCheckExprs:
 - apiVersion: cluster.x-k8s.io/v1beta1
 kind: Cluster
 failed: "status.conditions.filter(e, e.type == 'Ready').all(e, e.status == 'False')"
 current: "status.conditions.filter(e, e.type == 'Ready').all(e, e.status == 'True')"

The above example configures Flux to wait for all the Cluster objects to reach the Ready state before proceeding with the reconciliation of other Kustomizations that have a dependsOn relationship defined for the prod-clusters.

We have published a health check library that contains CEL expressions for popular custom resources. The library is community-maintained, and we encourage users to contribute new health checks.

Other kustomize-controller improvements include:

Fine-grained control of garbage collection with .spec.deletionPolicy.
SOPS support for decryption of Kubernetes secrets generated by Kustomize components.

GitHub App Authentication for Git Repositories

Starting with Flux v2.5, you can configure source-controller and image-automation-controller to authenticate against GitHub repositories using a GitHub App installation.

Instead of relying on personal access tokens or SSH keys that require manual rotation, you can now configure Flux to authenticate against GitHub repositories using an identity that is not tied to a specific user account.

We have added a new command to the Flux CLI that can be used to create the Kubernetes Secret required for the GitHub App authentication.

flux create secret githubapp github-auth \
 --app-id=1 \
 --app-installation-id=2 \
 --app-private-key=~/private-key.pem

The Kubernetes Secret generated by the above command can be referenced in a GitRepository and ImageUpdateAutomation with .spec.secretRef.name.

For more details on how to configure the GitHub App authentication, see the GitRepository API documentation.

Custom event metadata for notifications

Starting with Flux v2.5, users can enrich the metadata of the events sent by the notification-controller by adding annotations on the Flux Kustomization and HelmRelease resources. The metadata is included in the notifications sent to the configured providers, such as Slack, Microsoft Teams, etc., and can be used to provide additional context about a particular application or environment.

One highly requested feature was the ability to include the image tag in the notifications send when Flux image automation updates the container image tag in HelmRelease values.

Example of a HelmRelease with custom event metadata containing the image tag:

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
 name: my-app
 namespace: apps
 annotations:
 event.toolkit.fluxcd.io/image: docker.io/org/my-app:1.0.0 # {"$imagepolicy": "apps:my-app"}
spec:
 chart:
 spec:
 chart: my-app
 sourceRef:
 kind: HelmRepository
 name: podinfo
 values:
 image:
 tag: 1.0.0 # {"$imagepolicy": "apps:my-app:tag"}

When the image automation updates the my-app HelmRelease with a new image tag e.g. 1.0.1, the notification sent after the Helm release upgrade will include image: docker.io/org/my-app:1.0.1 in message body.

For more details on how to configure custom event metadata, see the Alert API documentation.

Other notifications improvements include:

The notification-controller is now capable of updating Git commit statuses from events about Kustomizations that consume OCIRepositories.
The Receiver API now supports filtering the declared resources that match a given Common Expression Language (CEL) expression.

CLI Improvements

To help users troubleshoot Flux, we’ve added a new flux debug command the following subcommands:

flux debug kustomization --show-vars used to inspect the final variables values by merging the Flux Kustomization inline vars with the vars coming from Kubernetes ConfigMaps/Secrets.
flux debug helmrelease --show-values used to inspect the final Helm values by merging the HelmRelease inline values with the values coming from Kubernetes ConfigMaps/Secrets.

Note that these commands will print sensitive information if Kubernetes Secrets are referenced in the Flux Kustomization or HelmRelease resources.

Other CLI improvements include:

A new command was added, flux create secret githubapp that can be used to generate a Kubernetes Secret for GitHub App authentication.
The flux create source git command now supports the --provider=github flag to configure GitHub App authentication for Git repositories.

Supported Versions

Flux v2.2 has reached end-of-life and is no longer supported.

Flux v2.5 supports the following Kubernetes versions:

Distribution	Versions
Kubernetes	1.30, 1.31, 1.32
OpenShift	4.17

Enterprise support

Note that the CNCF Flux project offers support only for the latest three minor versions of Kubernetes.

Backwards compatibility with older versions of Kubernetes and OpenShift is offered by vendors such as ControlPlane that provide enterprise support for Flux.

Over and out

If you have any questions, or simply just like what you read and want to get involved, here are a few good ways to reach us:

Join our upcoming dev meetings.
Join the Flux mailing list and let us know what you need help with.
Talk to us in the #flux channel on CNCF Slack.
Join the planning discussions.
Follow Flux on Twitter, or join the Flux LinkedIn group.

Blog: Announcing Flux 2.4 GA

Mon, 30 Sep 2024 12:00:00 +0000

We are thrilled to announce the release of Flux v2.4.0! In this post, we will highlight some of the new features and improvements included in this release.

General availability of Flux S3-compatible Source API

This release marks the General Availability (GA) of Flux Bucket API which allows storing the desired state of Kubernetes clusters in S3-compatible storage services such as Amazon S3, Azure Blob Storage, Google Cloud Storage, Alibaba Cloud, MinIO, and others.

The Bucket v1 API comes with new features including: proxy support, mTLS and custom STS configuration for AWS S3 and MinIO LDAP authentication.

New fields in the source.toollkit.fluxcd.io/v1 API:

.spec.proxySecretRef allows configuring HTTP/S Proxy authentication for the S3-compatible storage service.
.spec.certSecretRef allows custom TLS client certificate and CA for secure communication with the S3-compatible storage service.
.spec.sts allows custom STS configuration for AWS S3 and MinIO LDAP authentication.

For more details, please see the Bucket documentation.

To upgrade, make sure the new CRDs and controllers are deployed, and then change the manifests in Git:

Set apiVersion: source.toolkit.fluxcd.io/v1 in the YAML files that contain Bucket definitions.
Commit, push and reconcile the API version changes.

Bumping the APIs version in manifests can be done gradually. It is advised to not delay this procedure as the deprecated versions will be removed after 6 months.

Azure DevOps OIDC Authentication

Starting with Flux v2.4, you can configure source-controller and image-automation-controller to authenticate against Azure DevOps repositories using AKS Workload Identity.

Instead of using Azure personal access tokens or SSH keys that require manual rotation, you can now use OIDC tokens to authenticate against Azure DevOps repositories by leveraging Kubernetes Workload Identity.

For more details on how to configure the Azure DevOps OIDC authentication, see the GitRepository API documentation.

Controller Improvements

The OCIRepository v1beta2 API gains support for proxy configuration thus allowing dedicated HTTP/S Proxy authentication on multi-tenant Kubernetes clusters.
The HelmRelease v2 API gains support for disabling JSON schema validation of the Helm release values during installation and upgrade. And allows adopting existing Kubernetes resources during Helm release installation.
The notification-controller allows transitioning the Microsoft Teams alerting from the deprecated Office 365 connector to MS Workflows and the Adaptive Card format.
The Flux Kustomization and HelmRelease APIs now support defining dependencies between resources managed by different controller shards.

CLI Improvements

A new command was added, flux create secret proxy that can be used to generate a Kubernetes Secret for HTTP/S Proxy authentication referenced by Bucket, GitRepository and OCIRepository.
The flux create source git command now supports the --provider=azure flag to configure OIDC authentication for Azure DevOps repositories.
The flux diff kustomization command now supports the --recursive flag to recursively diff encountered Kustomizations.
On Windows, the Flux CLI can now be installed using the WinGet tool by running winget install -e --id FluxCD.Flux.

Supported Versions

Flux v2.1 has reached end-of-life and is no longer supported.

Flux v2.4 supports the following Kubernetes versions:

Distribution	Versions
Kubernetes	1.29, 1.30, 1.31
OpenShift	4.16

Enterprise support

Note that the CNCF Flux project offers support only for the latest three minor versions of Kubernetes.

Backwards compatibility with older versions of Kubernetes and OpenShift is offered by vendors such as ControlPlane that provide enterprise support for Flux.

Flux Operator and OpenShift Compatibility

Flux can be installed on Red Hat OpenShift cluster directly from OperatorHub or by using the Flux Operator Helm chart.

The Flux Operator is an open-source project developed by ControlPlane that offers an alternative to the Flux Bootstrap procedure, it removes the operational burden of managing Flux across fleets of clusters by fully automating the installation, configuration, and upgrade of the Flux controllers based on a declarative API.

The operator simplifies the configuration of Flux multi-tenancy lockdown, sharding, horizontal and vertical scaling, persistent storage, and allows fine-tuning the Flux controllers with Kustomize patches. The operator streamlines the transition from Git as the delivery mechanism for the cluster desired state to OCI artifacts and S3-compatible storage.

After installing the Flux Operator on OpenShift, you can deploy the Flux controllers using the FluxInstance custom resource e.g.:

apiVersion: fluxcd.controlplane.io/v1
kind: FluxInstance
metadata:
 name: flux
spec:
 distribution:
 version: "2.x"
 registry: "ghcr.io/fluxcd"
 artifact: "oci://ghcr.io/controlplaneio-fluxcd/flux-operator-manifests:latest"
 components:
 - source-controller
 - kustomize-controller
 - helm-controller
 - notification-controller
 - image-reflector-controller
 - image-automation-controller
 cluster:
 type: openshift
 multitenant: true
 networkPolicy: true
 domain: "cluster.local"
 sharding:
 shards: [ "shard1", "shard2" ]
 sync:
 kind: OCIRepository
 url: "oci://ghcr.io/my-org/my-fleet-manifests"
 ref: "latest"
 path: "clusters/my-cluster"
 pullSecret: "flux-system"

For more details on how to use configure Flux using the operator, please see the Flux Operator documentation.

Over and out

If you have any questions, or simply just like what you read and want to get involved, here are a few good ways to reach us:

Join our upcoming dev meetings.
Join the Flux mailing list and let us know what you need help with.
Talk to us in the #flux channel on CNCF Slack.
Join the planning discussions.
Follow Flux on Twitter, or join the Flux LinkedIn group.

Blog: Announcing Flux 2.3 GA

Mon, 13 May 2024 12:00:00 +0000

We are thrilled to announce the release of Flux v2.3.0! In this post, we will highlight some of the new features and improvements included in this release.

General availability of Flux Helm features and APIs

This release marks a significant milestone for the Flux project, after almost four years of development, the helm-controller and the Helm related APIs have reached general availability.

The following Kubernetes CRDs have been promoted to GA:

HelmRelease - helm.toolkit.fluxcd.io/v2
HelmChart - source.toolkit.fluxcd.io/v1
HelmRepository - source.toolkit.fluxcd.io/v1

The Helm features and APIs have been battle-tested by the community in production and are now considered stable. Future changes to the Helm APIs will be made in a backwards compatible manner, and we will continue to support and maintain them for the foreseeable future.

Enhanced Helm OCI support

The HelmRelease v2 API comes with a new field .spec.chartRef that adds support for referencing OCIRepository and HelmChart objects in a HelmRelease. When using .spec.chartRef instead of .spec.chart, the controller allows the reuse of a Helm chart version across multiple HelmRelease resources.

Starting with this version, the recommended way of referencing Helm charts stored in container registries is through OCIRepository.

Using OCIRepository objects instead of HelmRepository improves the controller’s performance and simplifies the debugging process. The OCIRepository provides more flexibility in managing Helm charts, as it allows targeting a Helm chart version by tag, semver or OCI digest pinning. If a chart version gets overwritten in the container registry, the controller will detect the change in the upstream OCI digest and reconcile the HelmRelease resources accordingly.

Example of a HelmRelease referencing an OCIRepository:

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
 name: metrics-server
spec:
 interval: 10m
 chartRef:
 kind: OCIRepository
 name: metrics-server
 driftDetection:
 mode: enabled
 values:
 apiService:
 create: true
---
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: OCIRepository
metadata:
 name: metrics-server
spec:
 interval: 12h
 layerSelector:
 mediaType: "application/vnd.cncf.helm.chart.content.v1.tar+gzip"
 operation: copy
 url: oci://docker.io/bitnamicharts/metrics-server
 ref:
 semver: ">=7.0.0"

Improved observability of Helm releases

By popular demand, the helm-controller now emits Kubernetes events annotated with the Helm chart appVersion in addition to the version info. When configuring alerts for Helm releases, the appVersion is now available as a field in the alert metadata and is displayed in the notification messages. The appVersion field is also included in the HelmRelease status, and in the gotk_resource_info Prometheus metrics.

When using an OCIRepository as the HelmRelease chart source, the controller will also include the OCI digest of the Helm chart artifact in the Kubernetes events and the HelmRelease status.

Benchmark results

To measure the real world impact of the helm-controller GA, we have set up benchmarks that measure Mean Time To Production (MTTP). The MTTP benchmark measures the time it takes for Flux to deploy application changes into production. Below are the results of the benchmark that ran on a GitHub hosted runner (Ubuntu, 16 cores):

Objects	Type	Flux component	Duration	Max Memory
100	HelmChart	source-controller	25s	40Mi
100	HelmRelease	helm-controller	28s	190Mi
500	HelmChart	source-controller	45s	68Mi
500	HelmRelease	helm-controller	2m45s	250Mi
1000	HelmChart	source-controller	1m30s	110Mi
1000	HelmRelease	helm-controller	8m1s	490Mi

Compared to Flux v2.2, in this version the memory consumption of the helm-controller has improved a lot, especially when the cluster has hundreds of CRDs registered. In Flux v2.2, helm-controller on Kubernetes v1.28 runs out of memory with only 100 CRDs registered. Whereas, in Flux v2.3 on Kubernetes v1.29, it can handle 500+ CRDs without issues. Given these results, it is recommended to upgrade the Kubernetes control plane to v1.29 and Flux to v2.3.

Image update automation improvements

The ImageUpdateAutomation API has been promoted to v1beta2 and the image-automation-controller has been refactored to enhance the reconciliation process.

The v1beta2 API comes with a new template model that can be used to customize the commit message when the controller updates the image references in the Git repository. The commit template supports old and new values for the changes made to the files containing the policy markers. In addition, the commit message is included in the Kubernetes events emitted by the controller, offering better visibility into the automation process.

The ImageUpdateAutomation API now supports selecting ImagePolicies using label selectors in the new field .spec.policySelector.

Migration to v1beta2 template model

To migrate to the v1beta2 API, update the apiVersion field in the ImageUpdateAutomation resources to image.toolkit.fluxcd.io/v1beta2, and modify the messageTemplate to use the Changed template data.

Example template:

apiVersion: image.toolkit.fluxcd.io/v1beta2
kind: ImageUpdateAutomation
metadata:
 name: <automation-name>
spec:
 git:
 commit:
 messageTemplate: |-
 Automated image update

 Changes:
 {{ range .Changed.Changes -}}
 - {{ .OldValue }} -> {{ .NewValue }}
 {{ end -}}

 Files:
 {{ range $filename, $_ := .Changed.FileChanges -}}
 - {{ $filename }}
 {{ end -}}

Example generated commit message:

Automated image update

Changes:
- docker.io/nginx:1.25.4 -> docker.io/nginx:1.25.5
- docker.io/org/app:1.0.0 -> docker.io/org/app:1.0.1

Files:
- apps/my-app/deployment.yaml

For more examples and details, see the ImageUpdateAutomation documentation.

Signatures verification with Notation

The Flux source-controller now supports verifying the authenticity of OCI artifacts signed with Notation (CNCF Notary project).

To enable Notation signature verification, please see the following documentation:

In addition, the Flux CLI now supports generating Kubernetes secrets with Notation trust policies, using the flux create secret notation command.

Big thanks to Microsoft for contributing to the development of this feature!

Terraform provider improvements

The Flux Terraform provider has undergone a major refactoring and now supports air-gapped bootstrap, drift detection and correction for Flux components, and the ability to upgrade and restore the Flux controllers in-cluster. Starting with this release, the provider is fully compatible with OpenTofu.

The provider documentation has been updated with examples and detailed usage instructions.

New maintainer

We are very happy to announce that Steven Wade has joined the Flux project as a maintainer of the Terraform provider. Steven has been a long-time contributor to the Flux project and we are excited to have him on board!

Controllers improvements

The Flux Kustomization API gains two optional fields .spec.namePrefix and .spec.nameSuffix that can be used to specify a prefix and suffix to be added to the names of all managed resources.
The kustomize-controller now supports the --feature-gates=StrictPostBuildSubstitutions=true flag, when enabled the post-build substitutions will fail if a variable without a default value is declared in files but is missing from the input vars.
The notification-controller Receiver API has been extended to support CDEvents.
The OCIRepository API has been extended with support for semver filtering.
The HelmChart API v1 comes with a new optional field .spec.ignoreMissingValuesFiles.

CLI improvements

The bootstrap capabilities have been extended to support Oracle VBS repositories.
The bootstrap procedure for Azure DevOps repositories has been update with support for SSH RSA SHA-2 keys.
The flux bootstrap command gains a new flag --ssh-hostkey-algos that can be used to specify the host key algorithms to be used for SSH connections.
The flux bootstrap and flux install commands now support the --registry-creds flag that can be used for generating an image pull secret for container images stored in private registries.
A new command was added, flux envsubst that can be used to replicate the behavior of the Flux Kustomization post-build substitutions.
The flux create source oci command now supports the --verify-subject and --verify-issuer for cosign keyless verification.
New commands were added for managing HelmChart objects: flux create|delete|export source chart.

Breaking changes and deprecations

Deprecated fields have been removed from the HelmRelease v2 API:

.spec.chart.spec.valuesFile replaced by .spec.chart.spec.valuesFiles
.spec.postRenderers.kustomize.patchesJson6902 replaced by .spec.postRenderers.kustomize.patches
.spec.postRenderers.kustomize.patchesStrategicMerge replaced by .spec.postRenderers.kustomize.patches
.status.lastAppliedRevision replaced by .status.history.chartVersion

The following APIs have been deprecated and will be removed in a future release:

HelmRelease v2beta2 and v2beta1
HelmChart v1beta2 and v1beta1
HelmRepository v1beta2 and v1beta1
ImageUpdateAutomation v1beta1

Supported versions

Flux v2.0 has reached end-of-life and is no longer supported.

Flux v2.3 supports the following Kubernetes versions:

Distribution	Versions
Kubernetes	1.28, 1.29, 1.30
OpenShift	4.15

Flux v2.3 is the first release end-to-end tested on OpenShift. Big thanks to Replicated for sponsoring the Flux project with on-demand OpenShift clusters. For more information on how to bootstrap Flux on OpenShift, see the OpenShift installation guide.

Enterprise support

Note that the CNCF Flux project offers support only for the latest three minor versions of Kubernetes.

Backwards compatibility with older versions of Kubernetes and OpenShift is offered by vendors such as ControlPlane that provide enterprise support for Flux.

Installing or upgrading Flux

To install Flux, take a look at our installation and get started guides.

To upgrade Flux from v2.x to v2.3.0, either rerun flux bootstrap or use the Flux GitHub Action.

To upgrade the APIs in the manifests stored in Git:

Before upgrading, ensure that the HelmRelease v2beta2 YAML manifests are not using deprecated fields. Search for valuesFile and replace it with valuesFiles, replace patchesJson6902 and patchesStrategicMerge with patches.
Commit and push the changes to the Git repository, then wait for Flux to reconcile the changes.
Upgrade the controllers and CRDs on the cluster using Flux v2.3 release.
Update the apiVersion field of the HelmRelease resources to helm.toolkit.fluxcd.io/v2.
Update the apiVersion field of the HelmRepository resources to source.toolkit.fluxcd.io/v1.
Update the apiVersion field of the ImageUpdateAutomation resources to image.toolkit.fluxcd.io/v1beta2.
Commit and push the changes to the Git repository.

Bumping the APIs version in manifests can be done gradually. It is advised to not delay this procedure as the deprecated versions will be removed after 6 months.

What’s next for Flux?

The next milestone for the Flux project is v2.4, which is planned for Q3 2024 and will focus on the image automation APIs and S3-compatible storage APIs. For more details on the upcoming features and improvements, see the Flux project roadmap.

After the introduction of OCI Artifacts in 2022, we had a recurring ask from users about improving the UX of running Flux fully decoupled from Git. In response, we made a proposal for a flux bootstrap oci command and a new Terraform/OpenTofu provider that relies on container registries as the unified data storage for the desired state of Kubernetes clusters. The RFC can be found at fluxcd/flux2#4749 and we welcome feedback from the community.

Over and out

If you have any questions, or simply just like what you read and want to get involved, here are a few good ways to reach us:

Join our upcoming dev meetings.
Join the Flux mailing list and let us know what you need help with.
Talk to us in the #flux channel on CNCF Slack.
Join the planning discussions.
Follow Flux on Twitter, or join the Flux LinkedIn group.

Blog: Flux project gains New Corporate Support and Ecosystem in 2024

Tue, 19 Mar 2024 08:00:00 +0000

The CNCF graduated Flux project is proud to announce that it will receive enhanced support from dedicated companies in 2024. These organizations are committed to the ongoing maintenance and development of Flux GitOps tools. At KubeCon EU 2024 in Paris, the Flux project has keynote highlights, sessions, and a booth.

Vendors and clouds are stepping up contributions

Major vendors are ramping up their ecosystem involvement in Flux moving forward. GitLab announced its continued support for Flux and working with partners. In early 2023, GitLab integrated Flux with its agent for Kubernetes offering as the recommended GitOps solution.

Similarly, Flux continues to be the GitOps engine for cloud vendors such as Microsoft and AWS. Lachie Evenson at Microsoft affirms that “Flux is the engine that powers several GitOps experiences on Azure as well as in our customer’s environments’ and that Microsoft is committed to upstream contributions.

Flux is the engine that powers several GitOps experiences on Azure as well as in our customer’s environments. We will continue to invest in Flux through upstream contributions for the long-term health and support of the project, and downstream partnerships to help customers

Lachie Evenson, Principal PDM Manager – Cloud Native Ecosystem, Microsoft

Both Azure and AWS use Flux to streamline Kubernetes cluster and application management, adopting GitOps principles for enhanced automation, security, and reliability in cloud-native application deployments for Azure Arc Kubernetes and EKS-Anywhere.

New Enterprise adopters: Silva project, Cisco, Tchibo, and more

Flux’s significant benefits show in its widespread adoption by large-scale enterprises across various sectors, including telecommunications and financial services. The new Sylva project showcases the critical role of Flux for its complex telecom cloud native environments. Sylva streamlines the management of Kubernetes workload clusters, specifically designed to deploy Containerized Network Functions (CNF) provided by both external CNF vendors and telecom operators’ in-house services. As Orange VP of Software Engineering affirms, Flux’s security, modularity, resilience, and community all contribute to how it is the GitOps framework of choice.

We are really happy to observe new big tech shops bringing their support to Flux. It should give to everyone the confidence to keep committing and investing. The level of quality and security, and the modularity of Flux have been prime reasons for our decision to use it two years ago to automate the deployment and lifecycle management of network functions. Our Linux Foundation Sylva-based cloud native infrastructure relies on the strength and resilience of the Flux community. Flux’s technology and community make it the GitOps framework in our Telco networks.

Philippe Ensarguet, VP Software Engineering, Orange

Other long-time Flux adopters who have added themselves to the public list recently include Cisco and German retailer, Tchibo.

New Support and Ecosystem: Hirings and Value-Add Extensions

While several ecosystem companies are coordinating to step up as maintainers, contributors, and supporters of Flux, ControlPlane was first to hire Stefan Prodan (Flux project maintainer and architect as well as Flagger creator) and Soulé Ba (core maintainer) to continue their contributions. As ControlPlane CEO, Andrew Martin, has reinforced their commitment to help maintain the project, which is part of their offering of an enterprise-grade distribution of Flux, including support services for critical system.

ControlPlane is delighted to continue supporting the Flux project for all users, and to provide organisations utilising Flux with access to a hardened, FIPS-compliant, enterprise-grade distribution of Flux.

Andrew Martin, CEO, ControlPlane

In addition, companies such as Aviator, OpsMX, OpsWorks Group, OSO and Teracloud began providing support for Flux among their offerings.

At KubeCon EU 2024 in Paris, these and other companies in the ecosystem will meet at a Birds of a Feather meeting to kick off further commitments to add value to Flux’s extensibility and ecosystem reach.

Cloud Native Computing Foundation Exemplar Project

At this juncture, the Flux project is a strong graduated project within the Cloud Native Computing Foundation with continued recognition from CNCF CTO, Chris Aniszczyk, who states how Flux exemplifies the strength and resilience of the community. Flux reached General Availability in December 2023 and blew through its second security audit with the CNCF with no CVEs. The project’s published benchmarks results demonstrate why cloud vendors and enterprises alike have been trusting Flux with their needs for scale.

I am glad to see the continued support across the open source cloud native community for Flux. I encourage other organizations to get involved as there’s never a bad time to contribute to an open source project you depend on. Also, this is a great example of the strength and resilience of our community and we look forward to Flux’s continued evolution and growth.

Chris Aniszczyk, CTO, CNCF

If you’re at KubeCon in Paris this week, visit the Flux project booth and the many sessions, including Stefan’s maintainer talk on the Flux roadmap at the event.

Blog: Introducing Capacitor, a general purpose UI for Flux

Tue, 27 Feb 2024 18:00:00 +0000

Flux has been one of the most popular gitops tools available for years. Yet, it only existed as a CLI tool until now. Capacitor is a GUI that acts as a dashboard for Flux where you can get quick overview about your Flux resources and application deployments to debug issues quickly.

A word from Laszlo, the maintainer of Capacitor

“Hello Flux blog,

Long time Flux user here, although I haven’t been very active in the community so far. The maintainers have been doing an amazing job with Flux. We’re standing on the shoulders of giants 🙌.

There is an odd fact though: there was no de facto Flux GUI until now. How come? I thought we could make one, hence we made Capacitor.

Why? Because it is not easy to observe Kustomization and HelmRelease states in the cluster. Even with tools that show Custom Resources, it is not obvious to make the connection between application deployments and Flux resources.

The goal with Capacitor is to create the right context for developers to debug their deployments. Whether the error is related to Flux or not.

We hope you’re going to find the tool useful.”

Use cases

Commandless Flux observation

The GUI substitutes for interacting with Flux resources and runtime via flux CLI commands.

Connecting application deployments with Flux resources

Application deployments show which Flux Kustomization or HelmRelease deployed them.

With a click of a button you can jump to the Flux resource and check the reconsiliation state.

Application deployment debugging

Application deployments have controls to perform routine tasks, like checking logs, describing deployments, pods, configmaps.

With these controls Capacitor can become your daily driver for your Kubernetes dashboarding needs.

What’s supported?

Flux resources:

Kustomization
HelmRelease
GitRepository
OCIRepositories
Buckets

Kubernetes resources:

Deployment
Pod
Service
Ingress
Configmap
Secret

Who made Capacitor?

Capacitor is an open-source project backed by Gimlet, a team that creates a Flux-based IDP.

Gimlet is our opinionated project, Capacitor is our un-opinionated take.

How to get started?

Capacitor doesn’t come with Flux natively, you’ll need to set it up separately.

Deploy the latest Capacitor release in the flux-system namespace by adding the following manifests to your Flux repository:

---
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: OCIRepository
metadata:
 name: capacitor
 namespace: flux-system
spec:
 interval: 12h
 url: oci://ghcr.io/gimlet-io/capacitor-manifests
 ref:
 semver: ">=0.1.0"
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
 name: capacitor
 namespace: flux-system
spec:
 targetNamespace: flux-system
 interval: 1h
 retryInterval: 2m
 timeout: 5m
 wait: true
 prune: true
 path: "./"
 sourceRef:
 kind: OCIRepository
 name: capacitor

Note that Flux will check for Capacitor releases every 12 hours and will automatically deploy the new version if it is available.

Access Capacitor UI with port-forwarding:

kubectl -n flux-system port-forward svc/capacitor 9000:9000

Where is the project hosted?

It is hosted on Github: gimlet-io/capacitor

Blog: Announcing Flux 2.2 GA

Tue, 12 Dec 2023 15:00:00 +0000

We are thrilled to announce the release of Flux v2.2.0! In this post, we will highlight some of the new features and improvements included in this release, with the primary theme being the many changes made to the helm-controller.

This new release will also be demoed by Priyanka “Pinky” Ravi and Max Werner on Monday, December 18. To attend this demo and ask any questions, you can register here.

Important things first: API changes

This release is accompanied by a series of (backwards compatible) API changes and introductions. Please refer to the release notes for a comprehensive list, and make sure to read them before updating your Flux installation.

Enhanced `HelmRelease` reconciliation model

The reconciliation model of the helm-controller has been rewritten to be able to better determine the state a Helm release is in, to then decide what Helm action should be performed to reach the desired state.

Effectively, this means that the controller is now capable of continuing where it left off, and to run Helm tests as soon as they are enabled without a Helm upgrade having to take place first.

In addition, it now takes note of releases while they are happening, instead of making observations afterward. Ensuring that when performing a rollback remediation, the version we revert to is always exactly the same as the one previously released by the controller. In cases where it is uncertain about state, it will always decide to (reattempt to) perform a Helm upgrade.

This also allows it with certainty to only count release attempts that did cause a mutation to the Helm storage as failures towards retry attempts, improving continuity due to it retrying instantly instead of remediating first.

Improved observability of Helm releases

An additional thing the enhanced reconciliation model allowed us to work on is making improvements to how we report state back to you, as a user.

The improvements range from the introduction of Reconciling and Stalled Condition types to become kstatus compatible, to an enriched overview of Helm releases up to the previous successful release in the Status, and more informative Kubernetes Event and Condition messages.

Events:
 Type Reason Age From Message
 ---- ------ ---- ---- -------
 Normal HelmChartCreated 25s helm-controller Created HelmChart/demo/demo-podinfo with SourceRef 'HelmRepository/demo/podinfo'
 Normal InstallSucceeded 20s helm-controller Helm install succeeded for release demo/podinfo.v1 with chart podinfo@6.5.3
 Normal TestSucceeded 12s helm-controller Helm test succeeded for release demo/podinfo.v1 with chart podinfo@6.5.3: 3 test hooks completed successfully

For more details around these changes, refer to the Status section in the HelmRelease v2beta2 specification.

Recovery from `pending-*` Helm release state

A much-reported issue was the helm-controller being unable to recover from another operation (install/upgrade/rollback) is in progress errors, which could occur when the controller Pod was forcefully killed. From this release on, the controller will recover from such errors by unlocking the Helm release from a pending-* to a failed state, and retrying it with a Helm upgrade.

Helm Release drift detection and correction

Around April we launched cluster state drift detection and correction for Helm releases as an experimental feature. At that time, it could only be enabled using a controller global feature flag, making it impractical to use at scale due to the wide variability in charts and unpredictability of the effects on some Helm charts.

For charts with lifecycle hooks, or cluster resources like Horizontal/Vertical Pod Autoscalers for which controllers may write updates back into their own spec, those updates would always be considered as drift by the helm-controller unless the resource would be ignored in full.

To address the above pain points, Helm drift detection can now be enabled on the HelmRelease itself, while also allowing you to ignore specific fields using JSON Pointers:

spec:
 driftDetection:
 mode: enabled
 ignore:
 - paths: ["/spec/replicas"]
 target:
 kind: Deployment

Using these settings, any drift detected will now be corrected by recreating and patching the Kubernetes objects (instead of doing a Helm upgrade) while changes to the .spec.replicas fields for Deployments will be ignored.

For more information, refer to the drift detection section in the HelmRelease v2beta2 specifiation.

Forcing and retrying Helm releases

Another much-reported issue was the impractical steps one had to take to recover from “retries exhausted” errors. To instruct the helm-controller to retry installing or upgrading a Helm release when it is out of retries, you can now either:

Instruct it to reset the failure counts, allowing it to retry the number of times as configured in the remediation strategy
```
flux reconcile helmrelease <release> --reset
```
Instruct it to force a one-off Helm install or upgrade
```
flux reconcile helmrelease <release> --force
```

For in-depth explanations about these new command options, refer to the “resetting remediation retries” and “forcing a release” sections in the HelmRelease v2beta2 specification.

Benchmark results

To measure the real world impact of the helm-controller overhaul, we have set up benchmarks that measure Mean Time To Production (MTTP). The MTTP benchmark measures the time it takes for Flux to deploy application changes into production. Below are the results of the benchmark that ran on a GitHub hosted runner (Ubuntu, 16 cores):

Objects	Type	Flux component	Duration	Max Memory
100	OCIRepository	source-controller	25s	38Mi
100	Kustomization	kustomize-controller	27s	32Mi
100	HelmChart	source-controller	25s	40Mi
100	HelmRelease	helm-controller	31s	140Mi
500	OCIRepository	source-controller	45s	65Mi
500	Kustomization	kustomize-controller	2m2s	72Mi
500	HelmChart	source-controller	45s	68Mi
500	HelmRelease	helm-controller	2m55s	350Mi
1000	OCIRepository	source-controller	1m30s	67Mi
1000	Kustomization	kustomize-controller	4m15s	112Mi
1000	HelmChart	source-controller	1m30s	110Mi
1000	HelmRelease	helm-controller	8m2s	620Mi

The benchmark uses a single application ( podinfo) for all tests with intervals set to 60m. The results may change when deploying Flux objects with a different configuration.

For more information about the benchmark setup and how you can run them on your machine, check out the fluxcd/flux-benchmark repository.

Breaking changes to Kustomizations

All Flux components have been updated from Kustomize v5.0.3 to v5.3.0.

You should be aware that this update comes with a breaking change in Kustomize, as components are now applied after generators. If you use Kustomize components or .spec.components in Kustomizations along with generators, then please make necessary changes before upgrading to avoid any undesirable behavior. For more information, see the relevant Kustomize issue.

Other notable changes

flux install and flux bootstrap now have guardrails to protect users from destructive operations.
Gitea support has been added to flux bootstrap. To bootstrap Flux onto a cluster using Gitea as the Git provider, run flux bootstrap gitea --repository <repo> --owner <owner>.
The OIDC issuer and identity subject can now be verified for images signed using Cosign. Refer to the HelmChart and OCIRepository specifications for more information.
Prefix based file filtering support has been added to the Bucket API for generic, aws and gcp providers.
Support for insecure (non-TLS HTTP) container registries has been added to the ImageRepository and HelmRepository APIs.
The Flux alerting capabilities have been extended with NATS and Bitbucket Server & Data Center support.

Installing or upgrading Flux

To install Flux, take a look at our installation and get started guides.

To upgrade Flux from v2.x to v2.2.0, either rerun flux bootstrap or use the Flux GitHub Action.

To upgrade the APIs, make sure the new Custom Resource Definitions and controllers are deployed, and then change the manifests in Git:

Set apiVersion: helm.toolkit.fluxcd.io/v2beta2 in the YAML files that contain HelmRelease definitions.
Set apiVersion: notification.toolkit.fluxcd.io/v1beta3 in the YAML files that contain Alert and Provider definitions.
Commit, push and reconcile the API version changes.

Bumping the APIs version in manifests can be done gradually. It is advised to not delay this procedure as the deprecated versions will be removed after 6 months.

Over and out

If you have any questions, or simply just like what you read and want to get involved. Here are a few good ways to reach us:

Join our upcoming dev meetings.
Join the Flux mailing list and let us know what you need help with.
Talk to us in the #flux channel on CNCF Slack.
Join the planning discussions.
Follow Flux on Twitter, or join the Flux LinkedIn group.

Blog: Second Flux Security Audit has concluded

Thu, 09 Nov 2023 00:00:00 +0000

Precisely 2 years after performing our first security Audit, we had the chance to put Flux through a second audit this year, again facilitated by the CNCF and the Open Source Technology Improvement Fund. Trail of Bits partnered with us this time to make Flux even more secure. Flux passed the “General Availability” milestone earlier this year and the focus was on the features shipped in the Flux GA release.

The Flux maintainers and community are very grateful for the work put into this by everyone and the opportunity to grow and improve as a project. Thanks to Trail of Bits, notably Maciej Domański, Sam Alws, Sam Greenup and Jeff Braswell, who have always been extremely responsive during the process.

No new CVEs

Good news first: No new CVEs have been published for Flux in response to this second audit. Trail of Bits highlight that they found Flux was “well structured and generally written defensively” and the “audit uncovered only low- and informational-severity findings”, 10 in total. 8 of the discovered issues have been fixed as of publication of this announcement. From the remaining two issues to be fixed, one is in the process of being resolved and for the other one we have decided to accept the very low risk due to reasons mentioned in the report.

The assessment was kicked off with a list of 23 questions to be answered, circling around potential data leaks, security documentation, access control or denial of service vulnerabilities. Since the focus was on the GA components, the following parts of Flux have been put under scrutiny:

source-controller
kustomize-controller
notification-controller
Flux CLI
The pkg library, and git/gogit/fs in particular

Details on the discovered issues

You will find the full report here. The following table shows all the findings together with links to the pull requests fixing them:

Issue	Severity	Fix
1: SetExpiration does not set the expiration for the given key	low	source-controller#1185
2: Inappropriate string trimming function	informational	notification-controller#590
3: Go’s default HTTP client uses a shared value that can be modified by other components	low	flux2#4182
4: Unhandled error value	informational	flux2#4181
5: Potential implicit memory aliasing in for loops	informational	source-controller#1257, notification-controller#627, flux2#4329
6: Directories created via os.MkdirAll are not checked for permissions	informational	n/a
7: Directories and files created with overly lenient permissions	informational	pkg#663, pkg#681, source-controller#1276, kustomize-controller#1005, flux2#4380
8: No restriction on minimum SSH RSA public key bit size	informational	flux2#4177
9: Flux macOS release binary susceptible to dylib injection	low	in progress
10: Path traversal in SecureJoin implementation	undetermined	pkg#650, go-git/go-billy#31, go-git/go-billy#34

In addition to the pull requests linked above we also enabled security and quality CI checks through CodeQL via flux2#4121 to prevent any avoidable regressions.

Conclusion and next steps

From our perspective as Flux maintainers, 2 years feel like a lifetime. We added lots of new features and fixed even more bugs in that timeframe. That’s why we are particularly grateful that CNCF and OSTIF gave us the opportunity to let a team of security experts assess Flux another time. We are proud of having been able to learn from the first assessment and kept on making Flux more and more secure over these past 2 years, leading to only low- and informational-severity security findings within the GA components of Flux.

Our next milestone is the general availability of Flux’s Helm features and the subsequent general availability of the remaining Flux components. If you are interested in contributing to this, we are very much looking forward to working with you. We welcome contributions in helping resolve issues of the road, additional comments on our security posture and also welcome contributions in the form of extending our fuzzing infrastructure. Finally, if you have any additional security feedback, please come and talk to us.

Again we would like to thank the Cloud Native Computing Foundation for sponsoring the audit, the Open Source Technology Improvement Fund for the coordination and Trail of Bits for the careful review and advice during the audit period.

We are happy and proud to be part of this community!

Blog: Announcing Flux 2.1 GA

Mon, 04 Sep 2023 00:00:00 +0000

New releases

We are happy to announce the latest GA releases for Flux and Flagger.

Flux v2.1.0

This new release comes with lots of new features, fixes, restructured documentation and performance improvements. Everyone is encouraged to upgrade for the best experience.

The Flux APIs were extended with new opt-in features in a backwards-compatible manner.

The Flux Git capabilities have been improved with support for Git push options, Git refspec, Gerrit, HTTP/S and SOCKS5 proxies.

In case you missed it, Flux reached General Availability in June. You can read the announcement here.

You can now check the end-of-life(EOL) dates and support information for different Flux versions at https://endoflife.date/flux.

Features

The GitRepository API has a new field .spec.proxySecretRef that is used for specifying proxy configuration to use for all remote Git operations related to the particular object.
The.spec.verify.mode field of the GitRepository API now accepts one of the following values HEAD, Tag, TagAndHEAD. These values are used to specify how the Git tags and commits are verified.
The server-side apply behaviour in the kustomize-controller has been extended with two extra policies: IfNotPresent and Ignore. These policies are specified with the kustomize.toolkit.fluxcd.io/ssa annotation on the resource manifest. The IfNotPresent policy is useful to have Flux create an object that will later be managed by another controller.
Support for sending notifications to DataDog.
The ImageUpdateAutomation API has two new optional fields - .spec.git.push.refspec and .spec.git.push.options for to specify a refspec and push options that will be used when pushing commits upstream.

Fixes and improvements

Here is a short list of features and improvements in this release:

A new flag --concurrent-ssa has been introduced in the kustomize-controller to set the number of concurrent server-side operations that will be performed by the controller per object. This increases speed when reconciling Kustomization with a considerable amount of objects.
Performance improvement when loading helm repositories with large indexes (up to 80% memory reduction).
The load distribution has been improved when reconciling Flux objects in parallel to reduce CPU and memory spikes.
The Installation and Monitoring sections of the Flux documentation have been restructured to make navigation and locating guides easier. We are always open to receiving feedback on how we can improve the documentation.

Deprecation

All APIs that accept TLS data have been modified to support Kubernetes TLS style secrets. The keys caFile, certFile and keyFile have been deprecated. For more details about the TLS changes please see the Kubernetes TLS Secrets section.
⚠️ Breaking changes: This release comes with breaking changes to the Flux monitoring stack (Prom+Grafana). The stack now leverages the kube-state-metrics Custom Resource State metrics to report some Flux resource metrics. This will allow users to extend the Flux metrics with custom metadata. The monitoring configuration in the fluxcd/flux2 repository is now deprecated and will be removed in a future release. The new monitoring configuration is located at fluxcd/flux2-monitoring-example. Please see the new monitoring guide https://fluxcd.io/flux/monitoring for more information.

Upgrade

To upgrade Flux from v0.x to v2.1.0 please follow the Flux GA upgrade procedure.

To Upgrade Flux from v2.0.x to v2.1.0 either by rerunning bootstrap or by using the Flux GitHub Action.

You can take a look at the changelog for the full list of changes.

❣️Big thanks to all the Flux contributors who helped us with this release!

Flux Grafana Dashboards

The Flux monitoring stack comes with two dashboards for easy visualization of Flux controllers and resource metrics. You can follow this link to learn how to set it up.

Flagger v1.33.0

This release fixes bugs related to the Canary lifecycle. The confirm-traffic-increase webhook is no longer called if the Canary is in the WaitingPromotion phase. Furthermore, a bug which caused downtime when initializing the Canary deployment has been fixed. Also, a bug in the request-duration metric for Traefik which assumed the result to be in milliseconds instead of seconds has been addressed.

The loadtester now also supports running kubectl commands.

Please see the changelog for the full changes.

Community News

This section highlights additions to our community - new contributors, project members, maintainers or adopters.

New adopters

We are very pleased to announce that the following adopters of Flux have come forward and added themselves to our website:

Zeit Online: a German-language platform for demanding online journalism and reader discussions with level.
Sonatype: a developer-friendly full-spectrum software supply chain management platform helps organizations and software developers.
Prophesee: a company using sensor design and AI algorithms to develop computer vision systems.
Infolegale: a legal information platform to monitor company solvency.
Eco Vadis: a collaborative platform that allows companies to assess the environmental and social performance of their suppliers.

If you have not already done so, use the instructions here or give us a ping and we will help to add you. Not only is it great for us to get to know and welcome you to our community. It also gives the team a big boost in morale to know where in the world Flux is used everywhere.

New Contributors

Shoutout to all our new contributors:

Thanks to all of our old and new contributors, and reach out if you’d like to become one as well.

People writing/talking about Flux

We love it when you all write about Flux and share your experience, write how-tos on integrating Flux with other pieces of software or other things. Give us a shout-out and we will link it from this section!

How to Build a Self-Service Platform on Upbound: Day 1

Our friends at Upbound wrote a great blog post on how you can use the power of Flux and Crossplane to drive control plane interactions and configure your control plane for GitOps Flows.

Canary deployment with Flagger and Istio on Devtron

Rupin Solanki describes how to leverage Flagger and Istio, to automate the canary release process, ensure seamless traffic shifting and real-time application health monitoring.

Events

It’s important to keep you up to date with events in Flux and provide simple ways to see our work in action and chat with our engineers.

Recent Events

In August here are a couple of talks we would like to highlight.

Cloud Native Islamabad - Harnessing the Power of GitOps with Flux

Flux maintainer, Stefan Prodan spoke at Cloud Native Islamabad on Harnessing the Power of GitOps with Flux. It is packed with a informed introduction to the concept of GitOps and a demo of Flux and the Weave GitOps UI! Click on the video below to watch it.

Upcoming Events

We are happy to announce that we have a number of events coming up. Tune in to learn more about Flux and GitOps best practices, get to know the team and join our community.

If you wish to speak at GitOpsCon EU, reach out to us to collaborate on proposals on a range of topics related to Kubernetes. We are happy to provide our writing expertise to your proposal and to collaborate on ideas. The CFP deadline is October 4, so kindly contact tamao@weave.works ASAP if you’re interested. The conference will take place virtually on the 5th - 6th of December.

CNCF On-Demand Webinar

Flux Maintainer, Kingdon B will be giving a talk titled How to start building a self-service infrastructure platform on Kubernetes on the 14th of September. It’s going to be packed with knowledge on how to use Backstage and GitOps. Register here.

Project meetings and Bug Scrub

Our Flux Bug Scrubs still are happening on a weekly basis and remain one of the best ways to get involved in Flux. They are a friendly and welcoming way to learn more about contributing and how Flux is organised as a project.

2023-09-05 22:00 UTC, 00:00 CEST The Flux Bug Scrub (AEST)
2023-09-06 12:00 UTC, 14:00 CEST The Flux Bug Scrub
2023-09-07 15:00 UTC, 17:00 CEST CNCF Flux Project Meeting (late)
2023-09-13 12:00 UTC, 14:00 CEST CNCF Flux Project Meeting (early)
2023-09-14 17:00 UTC, 19:00 CEST The Flux Bug Scrub
2023-09-19 22:00 UTC, 00:00 CEST The Flux Bug Scrub (AEST)

We are flexible with subjects and often go with the interests of the group or of the presenter. If you want to come and join us in either capacity, just show up or if you have questions, reach out to Kingdon B on Slack.

Flux Ecosystem

Terraform-controller

The ecosystem is buzzing with news about the licensing changes to Hashicorp’s open-source projects including Terraform. Weaveworks has released a statement on this and the impact on the tf-controller.

VS Code GitOps Extension

Significant performance upgrades and code refactoring has been introduced with VS Code GitOps Tools extension version 0.25.0. Previously cluster metadata was loaded using kubectl get commands. Now, a new javascript client is also used which permits faster loading and real-time watching of cluster resources. kubectl proxy is executed in the background for the new client. Rendering of resource treeviews has been reworked to minimise data reloading, to maintain collapsible state and to allow visualising resource errors grouped by namespaces. Timeout settings were added and bad cluster connections should no longer slow down Clusters treeview rendering.

UI refinements and bug fixes for the new client are ongoing. The most up-to-date UI features can be previewed by selecting “Install Pre-Release Version” in the VS Code Extension Browser.

Flux Fun Fact!

Did you know … 🔒 Flux is designed with security in mind: Pull vs. Push, least amount of privileges, adherence to Kubernetes security policies and tight integration with security tools and best-practices. Read more about our security considerations.

Over and out

If you like what you read and would like to get involved, here are a few good ways to do that:

Join our upcoming dev meetings.
Join the Flux mailing list and let us know what you’d like to see.
Talk to us in the #flux channel on CNCF Slack.
Join the planning discussions.
And if you are completely new to Flux, take a look at our Get Started guide and give us feedback.
Social media: Follow Flux on Twitter, join the discussion in the Flux LinkedIn group.
We are looking forward to working with you.

❤️ Your Flux maintainer, Somtochi Onyekwere, and project member, Tamao Nakahara.

Blog: Announcing Flux 2.0 GA

Thu, 20 Jul 2023 00:00:00 +0000

Flux 2.0 and General Availability!

On July 5, 2023, Flux reached a major landmark with Flux 2.0 and the general availability of its GitOps components! Flux has continued to grow during its incredible journey. Its early iteration was built at Weaveworks for their own needs and for a previous SaaS product built on Flux. Flux led to Weaveworks CEO Alexis Richardson to coin the term, GitOps, which has taken the world by storm with a CNCF Working Group, GitOpsCon, GitOps Days, and several GitOps community groups. Moreover, leaders such as Kubernetes co-creators Brendan Burns and Joe Beda have stated how GitOps is a natural evolution of Kubernetes itself.

“GitOps practices and Flux has elevated our engineering: code infra as software, eliminate human intervention, accelerate lead time for changes - without compromising security requirements.”

Tahir Raza, Staff Engineer - Cloud & Platform Engineering at Best Buy

Intending to be the best GitOps tool available, the Flux project has evolved into a mature and trustworthy software. During its evolution, Flux has accomplished several goals such as low resource consumption by adopting a microservices architecture, safe multi-tenancy through its security-first design and support for bleeding edge innovation by having first-class support for technologies like OCI and Cosign.

We are proud to see that Flux is one of the few CNCF graduated projects and the GitOps tool that companies such as Microsoft, AWS, GitLab, D2iQ,q and more trust to deliver GitOps to their customers.

“Safaricom PLC provides mobile telephony, mobile money transfer, consumer electronics, e-commerce, cloud computing, data, music streaming, and fiber optic services to the Kenyan Market predominantly and to the wider East Africa. So, Flux has been an essential part of critical areas such as deployment frequency, standardization, and security, among other GitOps capabilities that help us to be competitive. We are excited about Flux 2.0 and the project’s continued maturity.”

Winnie Gakuru, DevSecOps Engineer II at Safaricom PLC

Flux General Availability

“EKS Anywhere has been providing GitOps capabilities with Flux to our happy enterprise customers. We’ve been testing Flux 2.0 since our EKS-A v0.16.0 release and it has been solid. Flux, as a CNCF Graduated project and now with its GitOps components at GA, has been reliable and enterprise grade so that we can deliver the best experience to the customers who depend on our quality of product.”

Joey Wang, Senior Software Engineer at Amazon Web Services

What does General Availability (GA) mean for you as a Flux user:

“Flux is often my go-to technology choice for building multi-cluster and even multi-region deployment patterns. It helps me enable teams in evolving their applications from one cluster to many with consistent and repeatable config.”

Bryan Oliver, Principal Engineer at Thoughtworks

This signifies that the APIs that have achieved GA (Generally Available) status are now considered stable and can be used with confidence in production environments. They offer backward compatibility, ensuring that existing implementations will continue to function as expected. Flux encompasses various APIs, but not all of them have attained GA status yet.

The APIs that have reached GA include:

GitRepository: This API facilitates pulling configurations from Git repositories.
Kustomization: It enables the application and synchronization of Kubernetes manifests defined in Git.
Receiver API: This API triggers the reconciliation of Flux Custom Resources using webhooks.

It is important to note that these GA APIs will not undergo backwards-incompatible changes unless accompanied by a major version update and appropriate advance announcements As for the remaining Flux APIs, they will undergo further development and enhancements before being promoted to GA status at a later stage.

“GitLab picked Flux for its official GitOps integration within our GitLab agent for Kubernetes. Flux’s maturation and reliability have continued to show as we’ve tested Flux 2.0 in development. Now with Flux’s GA, we can continue to build the best user experience for our enterprise customers on solid foundations.”

Viktor Nagy, Senior Product Manager, Environments group at GitLab

Releases

“It has been a fantastic journey of rebuilding the original Flux into a microservices architecture, adding Flagger as a subproject, getting validated as a graduated project in the CNCF, and now reaching GA with Flux 2.0. I am grateful to work with great teams, maintainers, contributors, and partners, and then to see major enterprises and cloud providers relying on Flux to start or mature their Kubernetes journey. Keeping great company with users (the likes of Amazon AWS, D2iQ, Microsoft Azure, VMware, Weaveworks, GitLab, Volvo, SAP, Xenit and many more) keeps me motivated for the future innovations and growth for Flux.”

Stefan Prodan, Principal Developer Experience Engineer at Weaveworks, Flux maintainer and Flagger creator

Release Cadence: Flux will have at least three minor releases in a year following the Kubernetes release cadence. The release will happen roughly two weeks after a new Kubernetes release. The two weeks timeline can be adjusted if more time is needed for testing compatibility with the new Kubernetes version.

API Versioning: The Flux project follows the semver standard for versioning. Release candidates are marked as x.y.z-rc.a (e.g v1.0.0-rc.3) and stable releases are marked as x.y.z.

Support: Flux will support the last three minor release versions of a major release and the previous major release version for a year after its release. A newly released Flux version offers support for Kubernetes N-2 minor versions.

CVE Backport: We will backport bug fixes and security fixes to the last three minor releases as patch releases. Users are advised to run the latest patch release of a given minor release.

For more details on the release procedure, take a look at https://fluxcd.io/flux/releases.

“Xenit is proud to be contributors and maintainers of the Flux project, which is the GitOps tool of choice for enterprises and cloud providers such as Volvo, GitLab, Microsoft, and AWS. We are particularly proud to be part of Flux’s major milestones: not only graduating in the Cloud Native Computing Foundation some months ago, but now getting Flux to 2.0 and general availability. We enjoy being part of the Flux community and look forward to the next stages of this growing community.”

Simon Gottschlag, CTO at Xenit

How to get started?

Watch our CNCF webinar on Flux 2.0, which has an intro to GitOps for newcomers and Flux 2.0-specific updates for existing users.
Need extra support for Flux and Flagger? Check out the Flux support page and this August 2 webinar on Flux 2.0-specific support.

❤️ Your Flux maintainer, Somtochi Onyekwere, and project member, Tamao Nakahara.

Blog: May 2023 Update

Tue, 06 Jun 2023 20:30:00 +0000

May was packed with exciting stories from Flux users, newly updated Flux adopters, contributors, contributions and a new GA release candidate! Also, don’t miss future Flux Bug Scrubs using ChatGPT.

Flux technology things to know

Three more Flux release candidates! Many improvements - Please test

On our path to GA, we released v2.0.0-rc5, the fifth release candidate for the 2.0.0 release. It includes many fixes, so you are very much encouraged to upgrade to this latest version - even though it carries “RC” in its version number, it is the most stable Flux release to date. Users are advised to upgrade from v0.41 and older versions to v2.0.0-rc.5 as soon as possible.

Fixes and improvements

Starting with this version, source-controller, kustomize-controller and helm-controller pods are marked as system-cluster-critical.
The Alert v1beta2 API has two new optional fields. .spec.inclusionList for fine-grained control over events filtering (notification-controller) and .spec.metadata that allows users to enrich the alerts with information about the cluster name, region, environment, etc.
New command flux reconcile source chart for pulling Helm OCI charts on-demand from container registries (CLI).
Support annotated Git tags with .spec.ref.name in GitRepository (source-controller).
The deprecated field .status.url was removed from the Receiver v1 API (notification-controller).
Add support for commit signing using OpenPGP keys with passphrases (image-automation-controller).
Fix bootstrap for BitBucket Server (CLI).
Fix secrets decryption when using Azure Key Vault (kustomize-controller).
Fix drift detection for renamed HelmReleases (helm-controller).
Improve performance when handling webhook receivers (notification-controller).
Improve the detection of values changes for HelmReleases by stable sorting them by key (helm-controller)
Update cosign to v2 (source-controller)
Support for Helm 3.12.0 and Kustomize v5.0.3.

To upgrade from v0.x to v2.0.0-rc.5, please see the procedure documented in RC.1.

⚠️ Note that Kubernetes 1.27.0 contains a regression bug that affects Flux, it is recommended to upgrade Kubernetes to 1.27.1 or newer. The upgrade to Kustomize v5 also contains breaking changes, please consult their CHANGELOG for more details.

Big thanks to all the Flux contributors that helped us with this release!

Security news

All components have been updated to patch vulnerabilities in Docker (CVE-2023-28840, CVE-2023-28841, CVE-2023-28842) and Sigstore (CVE-2023-30551, CVE-2023-33199).

Flagger 1.31.0

This release adds support for Linkerd 2.13. Furthermore, a bug which led the confirm-rollout webhook to be executed at every step of the Canary instead of only being executed before the canary deployment is scaled up, has been fixed.

⚠️ This release contains some breaking changes for the Linkerd integration. Please see the CHANGELOG on how to upgrade.

News from Flux users & the Community!

Newly posted Flux adopters!

Blablacar: a long distance carpooling platform that connects drivers with empty seats and passengers to share travel costs.
Nuvme: a consulting firm specializing in cloud application modernization.
TTMzero: a RegTech company that assists financial players with pre and post-trade digitization.

Thanks to Horacio Granillo ( @hgranillo), Peter König ( @konigpeter), and Julien Haumont ( @jhaumont) for taking the time to make these additions to the Flux adopters list!

If you have not already done so, use the instructions here or give us a ping and we will help to add you. Not only is it great for us to get to know and welcome you to our community, it also gives the team a big boost in morale to see Flux being used across the world.

Flux members, contributors, and maintainers!

Priyanka Ravi joins as Flux Project Member

We are very happy that Priyanka “Pinky” Ravi joined us as a Flux Project Member.

Over the past years, Pinky spoke at conferences, meetups and elsewhere, demoing Flux, discussing use-cases and discussing what’s new. If you want to have a look at some of her talks, check out our resources section.

Thanks a lot for everything you have done - we are happy to have you in our team!

Matheus Pimenta joins as a Flux Project Member

We are very happy to have Matheus Pimenta as a Flux Project Member. Matheus has been very active in the Flux community. He has been opening issues, participating in discussions and raising pull requests especially in the notification-controller.

Thanks a lot for everything you have done - we are happy to have you in our team!

Tamao Nakahara joins as Flux Project Member

Tamao has been actively assisting with managing the Flux community and organizing efforts around getting Flux represented at various conferences. She is the lead organizer of GitOps Days.

Tamao has done so much for the Flux project. We are happy to welcome her to the team.

Sanskar Jaiswal becomes a Core Maintainer

Sanskar has been making major code contributions to Flux for a while and is already a Flagger maintainer. He has been instrumental in getting the improving the git implementation in Flux and a host of other features.

Thanks for all your contributions to Flux! This is well-deserved.

Mehak Saeed selected for Flux’s Season of Docs

We are excited to welcome Mehak Saheed who would be working to improve Flux’s documentation during this year’s Google Season of Docs. Mehak is a technical writer with over six years of experience and has worked on documentation for projects such as cert-manager and Unfurl.

We look forward to the great work she’ll do!

If you wish to speak at KubCon NA, reach out to us to collaborate on proposals on a range of topics related to Kuberentes. We are happy to provide our writing expertise to your proposal and to collaborate on ideas. The CFP deadline is June 18, so kindly contact tamao@weave.works ASAP if you’re interested. The conference is from 6th-9th November in Chicago.

Use Cases from Flux users at GitOpsCon / Open Source Summit 2023 in May!

Flux users, contributors, and maintainers spoke at the 2-day co-located event, GitOpsCon-CDCon, as well as at the 3-day core conference, Open Source Summit NA 2023, during the week of May 8-12, 2023 in Vancouver, Canada. See below for more talks from the conference from contributors and maintainers. Here are highlighted talks from Flux users:

Keynote Session: GitOps as an Evolution of Kubernetes - Flux user and Kubernetes co-creator, Brendan Burns, Corporate Vice President
Multitenancy - Build Vs. “Buy”: Zcaler’s Journey - Flux users Neeta Rathi & Josh Carlisle, ZScaler
Managing Software Upgrades with a kpt, GitLab and Flux Workflow in a Telecom Context - Flux user, Peter Wörndle, Ericsson
Flux at the Point of Change - Using the K8s Golang SDK and the Flux Api to Automatically Fix and Deploy CVEs in Your Base Images - Flux user, Bryan Oliver, Thoughtworks, Inc.
Kubernetes Quick Wins and Migration Best Practices: RingCentral Example - Flux user, Ivan Anisimov, RingCentral
Deliver a Multicloud Application with Flux and Carvel - Flux ecosystem user, Peter Tran, VMware
High-Security, Zero-Connectivity & Air-Gapped Clouds: Delivering Complex Software with the Open Component Model & Flux - Flux user, Dan Small, SAP & Mohamed Ahmed, Weaveworks
Delivering Secure & Compliant Software Components with the Open Component Model & GitOps - Flux user, Dan Small, SAP SE
Extending Observability to the Application Lifecycle with ArgoCD, Flux and Keptn - Flux users Ana Margarita Medina, Lightstep & Adam Gardner, Dynatrace

DevOps Days Medellin, Colombia

David Caballero gave a talk this month on Flux and shared slides and other resources in the CNCF Flux slack. Check it out!

Talks on Flux+GitLab, Flux+ARM64, Flux+Terraform, Flux+VS Code, Flux+WASM and more from GitOpsCon-CDCon / Open Source Summit 2023

Here are additional talks from GitOpsCon-CDCon and Open Source Summit NA 2023, during the week of May 8-12, 2023 in Vancouver, Canada.

Talk summaries in The New Stack:

Kingdon’s talk on WASM
GitOps principles quoting Pinky’s GitOpsCon keynote panel

Talks by Flux contributors and maintainers include:

GitOpsCon Keynote panel featuring Flux contributor, Priyanka “Pinky” Ravi, Weaveworks
GitLab + Flux! - Priyanka “Pinky” Ravi, Weaveworks & Flux user, Viktor Nagy, GitLab
GitOps Sustainability with Flux and arm64 (full version)- Tamao Nakahara, Weaveworks & Liz Fong-Jones, Honeycomb
Microservices and WASM, Are We There Yet? - Flux user, Will Christensen, Defense Unicorns & Kingdon Barrett, Weaveworks
Automate with Terraform + Flux + EKS: Level Up Your Deployments - Flux contributor, Priyanka “Pinky” Ravi, Weaveworks
Exotic Runtime Targets: Ruby and Wasm on Kubernetes and GitOps Delivery Pipelines (15-min version) - Flux maintainer, Kingdon Barrett, Weaveworks
VS Code+Flux: Dev-Driven Automated Deployments Like a Cloud Native Pro (Even if You’re a Beginner) - Flux ecosystem contributor, Juozas Gaigalas, Weaveworks
Level Up Your Deployments: Automate with Terraform + Flux - Flux contributor, Priyanka “Pinky” Ravi, Weaveworks
Platform Engineering Done Right: Safe, Secure, & Scalable Multi-Tenant GitOps - Flux ecosystem contributor, Juozas Gaigalas, Weaveworks
Exotic Runtime Targets: Ruby and Wasm on Kubernetes and GitOps Delivery Pipelines (40-min version) - Flux maintainer, Kingdon Barrett, Weaveworks
Lightning Talk: GitOps Sustainability with Flux and arm64 (5-min version) - Flux contributor, Tamao Nakahara, Weaveworks
Community Diversity and Inclusion as Business Metric (and Not Just a Feel-Good Tactic) - Flux contributor, Tamao Nakahara, Weaveworks

Upcoming Events

Flux project meetings and Flux Bug Scrub+ChatGPT!

Our June 27 and 28 bug scrubs will involve using ChatGPT Experiment with us and we’ll learn together! Join the Weave Online User Group for updates.

The next dates are going to be:

2023-06-07 12:00 UTC, 19:00 CEST CNCF Flux Project Meeting (early)
2023-06-08 17:00 UTC, 19:00 CEST The Flux Bug Scrub
2023-06-13 22:00 UTC, 00:00 CEST The Flux Bug Scrub (AEST)
2023-06-14 12:00 UTC, 14:00 CEST The Flux Bug Scrub
2023-06-15 15:00 UTC, 17:00 CEST CNCF Flux Project Meeting (late)
2023-06-21 12:00 UTC, 19:00 CEST CNCF Flux Project Meeting (early)
2023-06-22 17:00 UTC, 19:00 CEST The Flux Bug Scrub
2023-06-27 22:00 UTC, 14:00 CEST The Flux Bug Scrub (AEST): playing with ChatGPT!
2023-06-28 12:00 UTC, 00:00 CEST The Flux Bug Scrub: playing with ChatGPT!
2023-06-29 15:00 UTC, 17:00 CEST CNCF Flux Project Meeting (late)

Flux Fun Fact!

Did you know … 🔩 Flux works with your existing tools: Flux works with your Git providers (GitHub, GitLab, Bitbucket, can even use s3-compatible buckets as a source), all major container registries, and all CI workflow providers. GitLab also announced that Flux is their GitOps tool of choice, so you’ll see even more synergy this year!

Over and out

If you like what you read and would like to get involved, here are a few good ways to do that:

Join our upcoming dev meetings.
Join the Flux mailing list and let us know what you’d like to see.
Talk to us in the #flux channel on CNCF Slack.
Join the planning discussions.
And if you are completely new to Flux, take a look at our Get Started guide and give us feedback.
Social media: Follow Flux on Twitter, join the discussion in the Flux LinkedIn group.
We are looking forward to working with you.

❤️ Your Flux maintainer, Somtochi Onyekwere, and project member, Tamao Nakahara.

Blog: April 2023 Update

Tue, 02 May 2023 06:30:00 +0000

As the Flux family of projects and its communities are growing, we strive to inform you each month about what has already landed, new possibilities which are available for integration, and where you can get involved. Read our last update here.

It’s the beginning of May 2023 - let’s recap together what happened in April - it has been a lot!

News in the Flux family

Flux v2.0.0 release candidate

This is the first release candidate of Flux v2.0 GA 🎉.

Users are encouraged to upgrade for the best experience. We also very much welcome feedback!

Flux v2.0.0-rc.1 comes with the promotion of the GitOps related APIs to v1 and adds horizontal scaling & sharding capabilities to Flux controllers.

In addition, RC.1 comes with support for auth with Azure Workload Identity when pulling OCI artifacts from ACR and when decrypting secret with Azure Vault. Also, Bootstrap for GitLab was extended with support for generating GitLab Deploy Tokens.

Big thanks to all the Flux contributors that helped us with this release!

And a special shoutout to the GitLab team for their first contribution to Flux!

This release brings API changes we want to highlight here:

GitRepository v1
Kustomization v1
Receiverv1

The GitRepository kind was promoted from v1beta2 to v1 (GA) and deprecated fields were removed. The v1 API is backwards compatible with v1beta2, except for the following:

the deprecated field .spec.gitImplementation was removed
the unused field .spec.accessFrom was removed
the deprecated field .status.contentConfigChecksum was removed
the deprecated field .status.artifact.checksum was removed
the .status.url was removed in favor of the absolute .status.artifact.url

The Kustomization kind was promoted from v1beta2 to v1 (GA) and deprecated fields were removed. A new optional field .spec.commonMetadata was added to the API for setting labels and/or annotations to all resources part of a Kustomization. The v1 API is backwards compatible with v1beta2, except for the following:

the deprecated field .spec.validation was removed
the deprecated field .spec.patchesStrategicMerge was removed (replaced by .spec.patches)
the deprecated field .spec.patchesJson6902was removed (replaced by.spec.patches`)

The Receiver kind was promoted from v1beta2 to v1 (GA). The v1 API now supports triggering the reconciliation of multiple resources using .spec.resources.matchLabels. The v1 API is backwards compatible with v1beta2, no fields were removed.

To upgrade Flux from v0.x to v2.0.0-rc-1 you can either rerun flux bootstrap or use the Flux GitHub Action.

To upgrade the APIs from v1beta2, after deploying the new CRDs and controllers, change the manifests in Git:

set apiVersion: source.toolkit.fluxcd.io/v1 in the YAML files that contain GitRepository definitions and remove the deprecated fields if any
set apiVersion: kustomize.toolkit.fluxcd.io/v1 in the YAML files that contain Kustomization definitions and remove the deprecated fields if any
set apiVersion: notification.toolkit.fluxcd.io/v1 in the YAML files that contain Receiver definitions

Bumping the APIs version in manifests can be done gradually. It is advised to not delay this procedure as the beta versions will be removed after 6 months.

⚠️ Note that this release updates the major version of the Flux Go Module to v2. Please update your go.mod to require github.com/fluxcd/flux2/v2, see pkg.go.dev for the documentation of the module.

New Documentation

Flagger: Bug fix release 1.30.0 hits the streets

This release fixes a bug related to the lack of updates to the generated object’s metadata according to the metadata specified in spec.service.apex. Furthermore, a bug where labels were wrongfully copied over from the canary deployment to primary deployment when no value was provided for --include-label-prefix has been fixed. This release also makes Flagger compatible with Flux’s helm-controller drift detection.

Flux Ecosystem

Weave GitOps

Weave GitOps has recently released two new versions, v0.21.2 and v0.22.0, bringing various enhancements and bug fixes to the community.

In v0.21.2, the release includes client-side apply for better interactivity, removal of runs in non-session mode, custom SVGs for navigation icons, health checks in the UI, and more. Alongside these enhancements, bug fixes include resolving dashboard reconciliation issues and URL checking regex.

In v0.22.0, enhancements include group claim support for strings, OIDC prefix support for impersonation, additional health checks, and support for .sourceignore for GitOps Run. Bug fixes address concurrent ID token refreshing, clean-up process issues, and vulnerabilities in the YAML NPM package.

Weave GitOps Enterprise has introduced v0.21.2 and v0.22.0, offering new features and improvements. In v0.21.2, users can view GitOpsSets on leaf clusters in the UI, experience a fixed bug related to GitOpsSets not updating ConfigMaps, and utilize the “View Open Pull Requests” button to select any GitRepository. Enhancements include updating the GoToOpenPullRequest button and extending unwatch cluster logic for better resource management. The UI now has a sync external secret button on the secret details page.

In v0.22.0, the new Explorer backend has been introduced, providing better scalability for Weave GitOps Enterprise. The Explorer now supports Flux sources, and the Applications UI and Sources UI can be configured to use the Explorer backend for an improved user experience.

GitOpsSets offer enhanced templating for numbers and object chunks, and cluster bootstraps now sync secrets without waiting for ControlPlane readiness. The Explorer collector utilizes impersonation, and a feature flag has been added for replacing Applications and Sources with the query service backend. Bug fixes include addressing Git authentication checks, non-deterministic GitRepository template application, and improved support for “View Open PRs” in different URL formats.

Documentation updates include instructions for configuring Weave GitOps Enterprise to create PRs in Azure DevOps and user guides for raw templates and chart paths. In addition, updates cover secrets management, using private Helm repositories, and frontend development process improvements.

You might be interested in our recent blog post about how to use Weave GitOps as your Flux UI as well.

Terraform-controller

The team has recently released Terraform Controller v0.15.0-rc.1 which supports Flux v2.0.0-rc.1. This update brings significant improvements and moves us closer to the Flux GA.

⚠️Important Note:⚠️ With this release, there are breaking changes to be aware of:

Terraform Controller now uses API version v1alpha2, deprecating v1alpha1.
This version is not compatible with Flux v2 v0.41.x and earlier versions.

Flux Subsystem for Argo

The team has recently shared a sneak preview of the new version of Flamingo, a powerful drop-in extension for Argo CD that seamlessly integrates Flux as a GitOps engine in any Argo CD environments.

Now with the ability to switch between Argo CD UI and Weave GitOps (the UI for Flux), Flamingo aims to take DevOps and GitOps user experiences to the next level with this integration.

If the video is not displayed, view the video here.

You might be interested in this blog post on the Weaveworks blog about Flamingo.

New additions to the Flux Ecosystem

AWS Labs introduced their new project awslabs/aws-cloudformation-controller-for-flux. It is a Flux controller for managing AWS CloudFormation stacks and helps you to store CloudFormation templates in a git repository and automatically sync template changes to CloudFormation stacks in your AWS account with Flux.

Check out the demo and example.

Recent & Upcoming Events

It’s important to keep you up to date with new features and developments in Flux and provide simple ways to see our work in action and chat with our engineers.

cdCon + GitOpsCon North America 2023

cdCon + GitOpsCon NA 2023 is only a few days away. It will happen May 8-9 in Vancouver, Canada. Of course Team Flux will be there to talk about all things GitOps!

Here’s what we put in our calendar:

OSS Summit North America 2023

Open Source Summit NA 2023 is coming up May 10-12 in Vancouver, Canada. It plays host great number of sub-conferences in many of which you will see Flux goodness happening.

Here are a few that we are looking forward to:

Recent Events (ICYMI) 📺

We feel blessed to have such a big community of users, contributors and integrators and so many are happy to talk about their experiences. In April here are a couple of talks we would like to highlight.

CloudNativeCon / KubeCon EU 2023

CloudNativeCon / KubeCon is the most important event for us, as it’s such a great venue to meet contributors, friends, end-users and folks who are generally interested. It was a very busy event and luckily Team Flux was there as a big group, so we were able to respond to all requests.

We kicked off the event with the Flux Project Meeting, which saw 4 hours of updates from the maintainers, lots of time for Q&A, story telling and a good opportunity to get to know each other.

Next up was the CNCF Graduated Projects Update, here is the link to the timestamp where we provided the Flux update.

Many folks were looking forward to hear how we envision Flux is used in an OCI world. Luckily Hidde and Stefan gave a talk about it:

At KubeCon EU, @hiddeco and myself, we’ve talked about @fluxcd beyond Git and how Flux OCI artifacts can streamline #Kubernetes continuous delivery.

Check out the recording on YouTubehttps://t.co/HhOJSpTmzq
— Stefan Prodan (@stefanprodan) May 2, 2023

We thank the Cloud Native Computing Foundation for setting up a Graduation Celebration for Argo and Flux, the two GitOps solutions which graduated around the same time! Cupcake time for everyone!

Last up was a great panel which featured Priyanka Ravi, Weaveworks; Christian Hernandez, Red Hat; Filip Jansson, Strålfors; Roberth Strand, Amesto Fortytwo; Leigh Capili, VMware.

They all talked about “How GitOps Changed Our Lives & Can Change Yours Too!”. Priyanka “Pinky”, Leigh and Roberth are long-time friends of Flux.

And thanks a lot to the Cloud Native Photo Crew, who took these pictures:

Upcoming Events 📆

We are happy to announce that we have a number of events coming up in May - tune in to learn more about Flux and GitOps best practices, get to know the team and join our community.

Flux Bug Scrub

The next dates are going to be:

In other news

Michael Fornaro joins Flux as a Project Member

We are pleased to announce that Michael Fornaro has joined Flux as a project member. Michael has been heavily involved in the Flux community, offering valuable assistance and support through the Slack #flux channels and participating in Flux Bug Scrub sessions.

In collaboration with Kingdon, Michael is working to expand the Bug Scrub initiative, recently launching the first AEST session to accommodate members in Eastern Europe, India, Southeast Asia, and other regions including Australia.

Michael is the founder of Raspbernetes and co-founder in K8s@Home, both of which are organizations that focus on learning and supporting Kubernetes at home. The community has a strong presence on GitHub and Discord, where Michael has been a valuable contributor.

People writing/talking about Flux

Grafana Operator Blog: Install Grafana-operator using Flux and Kustomize

The grafana-operator team have recently started to ship their Kustomize manifests using OCI with the help of Flux artifact. As a part of this, they have written a small blog on how to install grafana-operator using Flux and how to manage grafana dashboards as code.

News from the Website and our Docs

Flux Adopters shout-out

We are very pleased to announce that the following adopters of Flux have come forward and added themselves to our website: Alluvial, Orange, Kiln, Tchibo.

More docs and website news

We are constantly improving our documentation and website - here are a couple of small things we landed recently:

Internal documentation which explains how to use certain parts of the website.
Updated our announcements for KubeCon EU 2023 and Google Season of Docs 2023 to support the events better!
Updates to the docs to move graduated APIs to v1.
New documentation: Sharding Cheatsheet.
New additions to our resources page.
Lots of fixes and improvements all over the place.

Thanks a lot to these folks who contributed to docs and website: Stefan Prodan, Max Jonas Werner, Daniel Favour, Hidde Beydals, Claire Liguori, David Blaisonneau, Eddie Zaneski, Jan Christoph Ebersbach, Mehdi Bechiri, Romain Guichard, Sanskar Jaiswal, Stacey Potter, Tim Rohwedder, harshitasao, lehnerj.

Flux Project Facts

We are very proud of what we have put together. We want to reiterate some Flux facts - they are sort of our mission statement with Flux.

🤝 Flux provides GitOps for both apps or infrastructure. Flux and Flagger deploy apps with canaries, feature flags, and A/B rollouts. Flux can also manage any Kubernetes resource. Infrastructure and workload dependency management is built-in.
🤖 Just push to Git and Flux does the rest. Flux enables application deployment (CD) and (with the help of Flagger) progressive delivery (PD) through automatic reconciliation. Flux can even push back to Git for you with automated container image updates to Git (image scanning and patching).
🔩 Flux works with your existing tools: Flux works with your Git providers (GitHub, GitLab, Bitbucket, can even use s3-compatible buckets as a source), all major container registries, fully integrates with OCI and all CI workflow providers.
🔒 Flux is designed with security in mind: Pull vs. Push, least amount of privileges, adherence to Kubernetes security policies and tight integration with security tools and best-practices. Read more about our security considerations.
☸️ Flux works with any Kubernetes and all common Kubernetes tooling: Kustomize, Helm, RBAC, and policy-driven validation (OPA, Kyverno, admission controllers) so it simply falls into place.
🤹 Flux does Multi-Tenancy (and “Multi-everything”): Flux uses true Kubernetes RBAC via impersonation and supports multiple Git repositories. Multi-cluster infrastructure and apps work out of the box with Cluster API: Flux can use one Kubernetes cluster to manage apps in either the same or other clusters, spin up additional clusters themselves, and manage clusters including lifecycle and fleets.
✨ Dashboards love Flux: No matter if you use one of the Flux UIs or a hosted cloud offering from your cloud vendor, Flux has a thriving ecosystem of integrations and products built on top of it and all have great dashboards for you.
📞 Flux alerts and notifies: Flux provides health assessments, alerting to external systems and external events handling. Just “git push”, and get notified on Slack and other chat systems.
👍 Users trust Flux: Flux is a CNCF Graduated project and was categorised as “Adopt” on the CNCF CI/CD Tech Radar (alongside Helm).
💖 Flux has a lovely community that is very easy to work with! We welcome contributors of any kind. The components of Flux are on Kubernetes core controller-runtime, so anyone can contribute and its functionality can be extended very easily.

Over and out

If you like what you read and would like to get involved, here are a few good ways to do that:

Join our upcoming dev meetings on 2023-05-04 or 2023-05-10.
Talk to us in the #flux channel on CNCF Slack
Join the planning discussions
And if you are completely new to Flux, take a look at our Get Started guide and give us feedback
Social media: Follow Flux on Twitter, join the discussion in the Flux LinkedIn group.

We are looking forward to working with you.

Blog: How to use Weave GitOps as your Flux UI

Tue, 04 Apr 2023 08:30:00 +0000

Here comes the newest blog post in our ecosystem category. One of the key reasons to rewrite Flux was to break up the former monolith solution into separate controllers which provide distinct parts of the functionality. This allows users to pick just the parts they need, and integrators to very easily build on top of Flux’s APIs. Today we have a very active Flux Ecosystem - we very much welcome this to happen and see it as an indicator of success.

An introduction to Weave GitOps

Today we would like to talk about Weave GitOps. It has been built out in the open for about a year and brings among other things one of the most requested additions to Flux: a UI.

With Weave GitOps you

manage and view applications all in one place
easily see your continuous deployments and what is being produced via GitOps
sync your latest git commits directly from the UI
leverage Kubernetes RBAC to control permissions in the dashboard
quickly see the health of your reconciliation deployment runtime

The Weave GitOps team works very closely together with the Flux Community - many engineers on both teams are actually colleagues.

In addition to the UI Weave GitOps provides a frictionless way to get up to speed with your GitOps experience: GitOps Run. All you need to get started is a cluster and the Weave GitOps CLI. Everything else, including Flux and the Weave GitOps Dashboard will be set up automatically for you.

GitOps Run actually does more than the setup. You see changes sync almost in real time instead of the normal loop, where everything goes through a PR process, enabling you to iterate very quickly without sacrificing the GitOps pattern. The moment you are happy with the changes you create a PR just as usual. It’s the best of both worlds.

Watch this short video to see the beauty and ease of use: set up

If you are a Terraform user, you will love that the terraform-controller is integrated by default and your terraform resources will show up in the dashboard as well.

Getting Started

Using GitOps Run as shown in the video above is the easiest way to get set up. Period.

Here is an example of how to get an app deployment set up using GitOps (powered by Flux), including the dashboard.

brew install fluxcd/tap/flux
Head to podinfo and create a fork with the name podinfo-gitops-run.

Clone locally and change into the directory

export GITHUB_USER=<your github username>
# you can ignore these two commands if you already created and
# cloned your repository
git clone git@github.com:$GITHUB_USER/podinfo-gitops-run.git
cd podinfo-gitops-run

Now run the gitops command with --no-session as it’s a single user cluster which we want to use in direct mode. The port-forward points at the podinfo pod we will create later on.
```
gitops beta run ./podinfo --no-session \
--port-forward namespace=dev,resource=svc/backend,port=9898:9898
```
The other arguments denote a directory where the manifests are going to be stored and we set up port-forwarding for the application we are about to install.
During the installation process, Flux will be installed if it isn’t and you will now be asked if you want to install the GitOps dashboard. Answer yes and set a password.

Note: If you do not set a password, you won’t be able to login to the GitOps UI 😱.

Shortly after you should be able to open the dashboard. The username is admin and the password will be the one you set above.
If you check the contents of the podinfo directory, you will notice a kustomization.yaml file. Edit the resources element to list "../deploy/overlays/dev" as well. It should like below:
```
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
name: dev-podinfo
resources: [
 "../deploy/overlays/dev"
]
```

If you save the file, podinfo will be deployed and able to access it at http://localhost:9898.

There’s more: if you Ctrl-C the running “gitops” process in the terminal, you will be asked if you want to change the deployment to be in “GitOps mode”, this means that the manifests for the cluster definition and dashboard will be added as well and pushed to GitHub for you.

As you can see, Weave GitOps takes care of a lot of the repetitive tasks and heavy lifting. A beautiful way to get set up and know that Flux is doing everything behind the scenes for you.

Is there more?

There is an Enterprise version of Weave GitOps as well, so if you need professional support for everything mentioned above, you will be covered. In addition to that, you get advanced features, such as templates and GitOpsSets - these are what will enable you to create a self-service for application teams.

The Weave GitOps team is very friendly and are always happy to help and receive feedback. Just join them in the #weave-gitops channel on the Weave Users Slack.

Come and talk to us

If you have feedback to this story, let us know on Slack or on social media and if you have a story to tell yourself, come find us as well - you can also hit us up on the fluxcd.io website repository. We want to report more stories from our ecosystem and Flux success stories. Thanks in advance for reaching out!

Blog: March 2023 Update

Mon, 03 Apr 2023 08:30:00 +0000

It’s the beginning of April 2023 - let’s recap together what happened in March - it has been a lot!

News in the Flux family

We have released Flux v0.41 with new features and improvements. As always, everyone is encouraged to upgrade for the best experience.

Here is a short-list of features and improvements in this release:

Experimental support of drift detection of Helm releases compared to cluster-state.
Improved handling of SIGTERM signals received by the helm-controller, which will now terminate running Helm install or upgrade actions, instead of potentially leaving them in a pending state.
Opt-in OOM watcher in helm-controller to handle graceful termination of the controller before it is forcefully killed by Kubernetes’ OOM killer.
Kubernetes client and Custom Resource Definition life-cycle improvements to reduce the memory consumption of the helm-controller, with observed reductions up to 50%.
Opt-in allowance of DNS lookups during the rendering of Helm templates in the helm-controller via feature gate.
Optional disabling of the cache of the status poller used to determine the health of the resources applied by the kustomize-controller. This may improve memory usage on large scale clusters at the cost of more direct API calls.
Changes to the logging of all controllers to ensure Kubernetes components like the discovery client use the configured logging format.
New flux events command to display Kubernetes events for Flux resources, including the events of a referenced resource.
Custom annotations can now be set with flux push using --annotations.

It’s important to us to document all the new features, so here goes a list of new articles and how-tos:

Cheatsheet: Enable Helm drift detection
Cheatsheet: Enable Helm near OOM detection
Cheatsheet: Allow Helm DNS lookups
Controller: New helm-controller feature gates and options
Controller: New kustomize-controller feature gate
Spec: HelmRelease drift detection

Big thanks to all the Flux contributors that helped us with this release!

Flux Ecosystem

Weave GitOps

Weave GitOps Enterprise v0.19.0 brings a host of new features to help the Flux community streamline their workflows and improve overall efficiency. The GitOpsSets GUI makes it easier to manage applications across a fleet of clusters, while additional generators like cluster, gitRepository, and apiClient offer enhanced functionality and customization. Weave GitOps Enterprise now supports raw templating for greater flexibility, and the Sandbox environments provide real-time visibility and testing capabilities for Kubernetes infrastructure.

New additions to the PolicyConfig UI simplify policy management, and the SOPS Secrets features enable seamless GPG and AGE key management, making it easier than ever to create encrypted secrets. Experience the benefits of adopting Weave GitOps by leveraging these powerful new features for Kubernetes environments.

Terraform-controller

The team has been working on the new release of Terraform controller. Bug fixes related to the GitOps dependency management have been landed in the main branch. So please feel free to try it out.

The team has also been working closely with the Flux team to ensure that Terraform controller will support everything in the coming Flux GA.

Flux Subsystem for Argo

Flamingo, the Flux Subsystem for Argo, for ArgoCD 2.6 and Flux v0.41, has been released. Flamingo is the only tool that combines the best two GitOps technologies together. Flux and ArgoCD are both CNCF graduate projects.

This new Flamingo version includes support for the following:

Flux v0.41 which comes with many features and enhancements,
Pack of new user interface features from ArgoCD 2.6

Chanwit Kaekwasi, the main developer of Flamingo, is looking for feedback. If you use Flamingo and want to chat about how you use it, where you would like it to go or just want to give some feedback, please find him on the #flux channel on CNCF Slack. Thanks a lot in advance!

VS Code GitOps Extension

Version 0.24.0 of the VS Code extension was released. This version introduces a new feature for the users of Weave GitOps Enterprise: Templates. Using Templates users can create complex GitOps configurations, workflows and pipelines with a simple UI. Templates must be enabled in the VS Code settings to be available. The README file includes further information.

The team is continuing to work on UI and performance improvements. In the 0.24 release, the Sources and Workloads views are now grouped by Namespaces and their details are presented in a consistent way.

New additions to the Flux Ecosystem

We are very happy to see Timoni joining the Flux Ecosystem.

Timoni is a package manager for Kubernetes, powered by CUE and inspired by Helm.

The Timoni project strives to improve the UX of authoring Kubernetes configs. Instead of mingling Go templates with YAML like Helm, or layering YAML on top of each-other like Kustomize, Timoni relies on cuelang’s type safety, code generation and data validation features to offer a better experience of creating, packaging and delivering apps to Kubernetes.

Timoni can be used together with Flux to create a GitOps delivery pipeline for Timoni’s module instances.

In my quest to write less #Kubernetes YAML this year, I've been hacking on a new tool called Timoni.

Timoni is a package manager for Kubernetes, powered by @cue_lang and inspired by @HelmPack, Homebrew and Docker compose.https://t.co/xrGCN5pZGY
— Stefan Prodan (@stefanprodan) March 31, 2023

Recent & Upcoming Events

It’s important to keep you up to date with new features and developments in Flux and provide simple ways to see our work in action and chat with our engineers.

Recent Events (ICYMI) 📺

We feel blessed to have such a big community of users, contributors and integrators and so many are happy to talk about their experiences. In March here are a couple of talks we would like to highlight.

Kubernetes co-founder Brendan Burns and Flux maintainer Stefan Prodan recently gave a CNCF talk on #ubernetes in 2023.

Monolith to Microservices with Bite-Sized Kubernetes.

Cloud Native Live: Automating Kubernetes Deployments.

Here is a list of additional videos and topics we really enjoyed - please let us know if we missed anything of interest and we will make sure to mention it in the next post!

Upcoming Events 📆

We are happy to announce that we have a number of events coming up in April - tune in to learn more about Flux and GitOps best practices, get to know the team and join our community.

CloudNativeCon / KubeCon EU 2023

We are very excited! From Tuesday, April 18 through Friday, April 21 it is CloudNativeCon / KubeCon EU 2023 in Amsterdam.

At the time of writing all the tickets have been sold out, so if you managed to get one, we look forward to seeing you there! Let’s dig into what’s happening over all of the days, because it is a lot.

Here is the link to our mini-website to keep you up to date at all times.

Tuesday, April 18

This is where CloudNativeCon starts and we are happy to have representation at a number of Day-0 events.

08:00: OpenGitOps Project Meeting in the Auditorium Center.
11:55: Cloud Native Telco Day Panel: Looking Ahead to the Future with Project Sylva, Energy Efficiency & Telco Cloud Platforms. Panelists are Niki Manoledaki (Weaveworks), Philippe Ensarguet (Orange Business Services), Gergely Csatári (Nokia), Tim Irnich (SUSE) in Hall 7, Room A.

In the afternoon, starting at 13:00 (1pm), we start our Flux Project Gathering in Room G108 of the Auditorium Center. A lot of Flux maintainers and contributors will be there to chat with and we have a nice programme prepared for you.

1:00 pm Meet and Greet
1:15 pm Welcome and Overview of Flux activities at KubeCon
1:30 pm Intro to GitOps and Flux + Q&A (Priyanka “Pinky” Ravi)
2:00 pm What’s New with Flux! Overview (Flux team)
- Flux GA Release is coming up! What does that mean for you?
- Graduation and other updates!
- Contributing to Flux
3:00 pm Q&A time
3:15 pm Break
3:30 pm Flux & Other Tools
- Terraform (Pinky)
- Vault (Pinky)
- Helm (Hidde Beydals)
- VS Code (Juozas Gaigalas)
- Secrets & Sops (Hidde)
- Cosign (Stefan)
- Plus lots of time for Q&A!
4:45 pm Closing: Thanks and final questions

Wednesday, April 19

10:30 - 21:00 (9pm): Meet the Flux team at our booth at the Project Pavilion
at Project Pavilion, Kiosk 3
11:00 - 11:35: A CI/CD Platform in the Palm of Your Hand
Claudia Beresford, Weaveworks at Elicium Building, D201-202
11:55 - 12:30: Flux Beyond Git: Harnessing the Power of OCI
Stefan Prodan & Hidde Beydals, Weaveworks at Forum Center, E103-E104
18:00 (6pm) - 20:00 (8pm): Meet the Flux team at our booth for the Booth Crawl
at Project Pavilion, Kiosk 3

Update: previously the OCI talk was scheduled to happen at 17:25. 11:55 is correct.

Thursday, April 20

10:30 - 17:30 (5.30pm): Meet the Flux team at our booth at the Project Pavilion
at Project Pavilion, Kiosk 3
16:00 - 16:30 (4.30pm): Flux Project Graduation Celebration
in Hall 5

Friday, April 21

10:30 - 14:30 (2.30pm): Meet the Flux team at our booth at the Project Pavilion
at Project Pavilion, Kiosk 3
14:55 (2.55pm) - 15:30 (5.30pm): How GitOps Changed Our Lives & Can Change Yours Too! at Auditorium Center, G104-105
Priyanka Ravi, Weaveworks
Christian Hernandez, Red Hat
Filip Jansson, Strålfors
Roberth Strand, Amesto Fortytwo
Leigh Capili, VMware
16:00 (4pm) - 16:35 (4.35 pm): Self-Service GitOps at a Regulated Enterprise
Erick Bourgeois (Freelance) at Auditorium Center, G104-105
16:00 (4pm) - 16:35 (4.35 pm): A Look Under the Hood of CNCF Security Audits
Adam Korczynski & David Korczynski, Ada Logics at Auditorium Center, Auditorium + Balcony

Update: previously the time for the panel was 16:55, it has been moved to 14:55.

Flux Bug Scrub

The Flux Bug Scrub has undertaken a slightly new format, where we’ll be spending more time working on our own code, and even building some new software under the aegis of the “flux-community” organization! We’ll still have our familiar spreadsheet every week, but we will be conversely spending less time per-meeting on curating the list of issues across the FluxCD org than we have been.

The Bug Scrub community is small but growing; we need your feedback to make this effort a blockbuster hit (which you can provide asynchronously, please remember to @ KingdonB if you are interested, whether you can or cannot make the meeting time!)

Since we’ll be spending more time on code at the meeting, we are also evaluating the possibility of a new, more inclusive “Late Late” Bug Scrub, which would aim to be more accessible to people in the Asia/Pacific time zones, Australia, as well as those in Eastern Europe, Middle East, and Africa. We hope that there are some folks who might not have been able to make the earlier times, who could benefit from this new addition. If you are in these time zones which are not covered now, it’ll actually be “Early Work Hours” for you if we’ve calibrated this correctly. (Good morning, Internet…)

The addition of another meeting at a new time of day, is intended to provide a little extra time for the curation of issues to go along with our new format change, and also to balance the scales somewhat better across all regions. Now, we just need your feedback and RSVP to decide exactly what time this new meeting should be! Please check out the FluxCD calendar for an indication of when it has been scheduled, or ping KingdonB in the Flux channel on CNCF slack if you have a strong opinion on the matter!

The next dates are going to be:

In other news

People writing/talking about Flux

blakyaks.com: AKS & Flux via Terraform

Flux Adopters BlakYaks wrote this nice article in which they explain how to bring Flux to AKS via Terraform. In closing they say:

Hopefully by now you’ve got an idea of why we’re big advocates of GitOps workflows and, in particular, Flux.

The convention-based approach allows us to build complex application deployments with ease, and by leveraging source control best practices such as pull requests and branch policy we can quickly integrate deployments into our existing CI/CD toolchains in a secure and scalable manner.

News from the Website and our Docs

Flux Adopters shout-out

We are very pleased to announce that the following adopters of Flux have come forward and added themselves to our website: BlakYaks, Enliven Systems, Kratix, MyTaxi, ScaleAq, Szerzi and TrueLayer

More docs and website news

We are constantly improving our documentation and website - here are a couple of small things we landed recently:

Update of fluxcd.io landing page: we removed the terminal as it had been outdated and slowed down the loading time. We also simplified our messaging about Flux features and mission so it’s easier to understand at a glance.
We restructured our use of shortblocks, so the code structure is more straight-forward.
We applied for Google Season of Docs 2023. If you are interested in the initiative, go and check out the link.
We added information about to enable Helm drift detection and how to allow Helm DNS lookups.

And on top of that countless fixes, small improvements and updates as always. Thanks a lot to these folks who contributed to docs and website: Stefan Prodan, Hidde Beydals, Arhell, Max Jonas Werner, Andreas Olsen Gulla, Craig Hurt, Gangstead, Jake, KwongTN, Matteo Martellini, Metin OSMAN, Sanskar Jaiswal, Timo Furrer and zoltan.zvara.

Flux Project Facts

We are very proud of what we have put together. We want to reiterate some Flux facts - they are sort of our mission statement with Flux.

🤝 Flux provides GitOps for both apps or infrastructure. Flux and Flagger deploy apps with canaries, feature flags, and A/B rollouts. Flux can also manage any Kubernetes resource. Infrastructure and workload dependency management is built-in.
🤖 Just push to Git and Flux does the rest. Flux enables application deployment (CD) and (with the help of Flagger) progressive delivery (PD) through automatic reconciliation. Flux can even push back to Git for you with automated container image updates to Git (image scanning and patching).
🔩 Flux works with your existing tools: Flux works with your Git providers (GitHub, GitLab, Bitbucket, can even use s3-compatible buckets as a source), all major container registries, fully integrates with OCI and all CI workflow providers.
🔒 Flux is designed with security in mind: Pull vs. Push, least amount of privileges, adherence to Kubernetes security policies and tight integration with security tools and best-practices. Read more about our security considerations.
☸️ Flux works with any Kubernetes and all common Kubernetes tooling: Kustomize, Helm, RBAC, and policy-driven validation (OPA, Kyverno, admission controllers) so it simply falls into place.
🤹 Flux does Multi-Tenancy (and “Multi-everything”): Flux uses true Kubernetes RBAC via impersonation and supports multiple Git repositories. Multi-cluster infrastructure and apps work out of the box with Cluster API: Flux can use one Kubernetes cluster to manage apps in either the same or other clusters, spin up additional clusters themselves, and manage clusters including lifecycle and fleets.
✨ Dashboards love Flux: No matter if you use one of the Flux UIs or a hosted cloud offering from your cloud vendor, Flux has a thriving ecosystem of integrations and products built on top of it and all have great dashboards for you.
📞 Flux alerts and notifies: Flux provides health assessments, alerting to external systems and external events handling. Just “git push”, and get notified on Slack and other chat systems.
👍 Users trust Flux: Flux is a CNCF Graduated project and was categorised as “Adopt” on the CNCF CI/CD Tech Radar (alongside Helm).
💖 Flux has a lovely community that is very easy to work with! We welcome contributors of any kind. The components of Flux are on Kubernetes core controller-runtime, so anyone can contribute and its functionality can be extended very easily.

Over and out

If you like what you read and would like to get involved, here are a few good ways to do that:

Join our upcoming dev meetings on 2023-04-06 or 2023-04-12.
Talk to us in the #flux channel on CNCF Slack
Join the planning discussions
And if you are completely new to Flux, take a look at our Get Started guide and give us feedback
Social media: Follow Flux on Twitter, join the discussion in the Flux LinkedIn group.

We are looking forward to working with you.

Blog: February 2023 Update

Wed, 01 Mar 2023 07:30:00 +0000

It’s the beginning of March 2023 - let’s recap together what happened in February - it has been a lot!

News in the Flux family

Two Flux minor releases hit the streets

Last month gave us two minor releases of Flux. Here’s what you can look forward to on your next upgrade. As always: Users are encouraged to upgrade for the best experience.

v0.40: ImageRepository and ImagePolicy promote to v1beta2

Flux v0.40 brings a number of features and improvements:

The GitRepository API has a new optional field .spec.ref.name for specifying a Git Reference. This allows Flux to reconcile resources from GitHub Pull Requests (refs/pull/<id>/head) and GitLab Merge Requests (refs/merge-requests/<id>/head).
RFC-0005 (source revision format) and RFC-0003 (custom OCI media types) have been fully rolled out.
The ImageRepository and ImagePolicy APIs have been promoted to v1beta2.
The image-reflector-controller autologin flags have been deprecated, please see the migration instructions to v1beta2.
Allow specifying the cloud provider contextual login for container registries with ImageRepository.spec.provider.
Improve observability of ImageRepository by showing the latest scanned tags under .status.lastScanResult.latestTags.
Improve observability of ImagePolicy by reporting the current and previous image tag in status and events.
The Kubernetes builtin cluster roles: view, edit and admin have been extended to allow access to Flux custom resources.
Print a report of Flux custom resources and the amount of cumulative storage used for each source type with flux stats -A.

To read up on the details of the above, you might want to check out these pieces of documentation:

v0.39: better security support, improved performance and observability

Flux v0.39 includes these highlights:

Starting with this version, the Flux controllers come with SBOMs and SLSA Provenance Attestations embedded in their container images.
The Flux Terraform Provider has a new resource for bootstrapping Flux, without depending on third-party Terraform providers, that allows customising the controllers at install time. Users are encouraged to migrate to this new resource and provide feedback.
The Flux CLI is now included in Wolfi OS, the Linux (Un)distro designed for securing the software supply chain. The Chainguard team and Wolfi maintainers are shipping updates for the Flux package on a regular basis.

Features and improvements include:

Recreate immutable resources (e.g. Kubernetes Jobs) by annotating or labeling them with kustomize.toolkit.fluxcd.io/force: enabled.
Support for HTTPS bearer token authentication for Git repositories.
Improve memory usage by disabling the caching of Secret and ConfigMap resources in all controllers.
Better observability with progressive status updates for Sources (Git, OCI, Helm, S3 Buckets).
Allow extracting the OCI artifact SHA256 digest for Cosign with flux push artifact -o json.
Track CRDs managed by Flux, flux trace and flux tree will show which HelmRelease deployed which CRDs.
Allow the Flux GitHub Action to use a GitHub token when checking for updates to avoid rate limiting.

Documentation

Big thanks to all the Flux contributors that helped us with this release!

Security news

We extended our Security Docs to show more examples for verifying SBOMs. Now the newly introduced SLSA Provenance Attestation feature is documented as well.

Flagger 1.29.0 brings support for template variables for analysis metrics

A canary analysis metric can reference a set of custom variables with .spec.analysis.metrics[].templateVariables. For more info see the docs. Furthermore, a bug related to Canary releases with session affinity has been fixed.

Improvements & Fixes

Allow custom affinities for flagger deployment in helm chart
Add namespace to namespaced resources in helm chart
modify release workflow to publish rc images
build: Enable SBOM and SLSA Provenance
Add support for custom variables in metric templates
docs(readme.md): add additional tutorial
use regex to match against headers in istio

Flux Ecosystem

Weave GitOps

The latest release includes enhancements, improvements, bug fixes, and documentation updates to enhance Weave GitOps’ overall functionality and user experience.

Enhancements in this version include improved detection of the OSS dashboard and the addition of imagePolicy details. The get-session logs feature has also been enhanced to support pod logs, filters, and return logging sources. A new optional tooltip has been added to the Timestamp component, and the formatting of log message timestamps in the log UI has been improved.

UI enhancements in this version aim to improve the overall user experience of Weave GitOps. Access properties on undefined ImageAutomation objects can now be handled, and an issue where graph nodes hopped around has been fixed. A text search has been added to table URLs, and undefined icon types can now be handled.

The Helm reloading strategy has been fixed, and the chart spec has been updated with values.yaml to address reloading issues.

Terraform-controller

The latest release of TF-Controller, version v0.14.0, introduces several new features and many bug fixes. Notably, the release offers first-class support for Terraform Cloud with the spec.cloud field. This enhancement allows Weave GitOps Enterprise users to leverage GitOps Templates with Terraform Cloud as a backend for their Terraform resources, opening up a world of possibilities for GitOps workflows.

In addition to Terraform Cloud support, the update upgrades Flux to v0.40.0 and Terraform to v1.3.9, with bug fixes including improved AWS package documentation, and missing inventory entries.

The new release also offers multi-arch image support, customizable controller log encoding, and the option to configure Kube API QPS and Burst. The Terraform apply stage now features a parallelism option for even more customization.

Users are highly recommended to upgrade to TF-Controller v0.14.0 to take advantage of these improvements. For any feedback or questions, please reach out to the team on the GitHub repository.

Flux Subsystem for Argo

The team has recently updated Flamingo by rebasing it onto the upstream ArgoCD versions v2.3.17, v2.4.23, and 2.5.11. This update has been made in response to the recent vulnerability CVE-2023-23947, and the team strongly recommends that all users update their systems as soon as possible.

Updated Flamingo images are:

v2.3.17-fl.3-main-bc5b4abb
v2.4.23-fl.3-main-bc5b4abb
v2.5.11-fl.3-main-bc5b4abb

VS Code GitOps Extension

Version 0.23.0 of the vscode-gitops-tools extension was released. This version introduces a new webview for configuring GitRepository, HelmRepository, OCIRepository, Bucket and Kustomization resources. Extension context (right-click) file and folder actions now work with multiple open repositories in the expected way.

Pulumi Kubernetes Operator

Michel Bridgen wrote a blog post about how to combine Pulumi with Flux using the Pulumi Kubernetes Operator, which extends the reach of both Flux and Pulumi.

What you can look forward to in the next release of the operator is that - based on Paolo’s work on git-go - the operator (and Pulumi itself) will be able work with Azure DevOps.

Recent & Upcoming Events

It’s important to keep you up to date with new features and developments in Flux and provide simple ways to see our work in action and chat with our engineers.

Recent Events (ICYMI) 📺

We feel blessed to have such a big community of users, contributors and integrators and so many are happy to talk about their experiences. In February here are a couple of talks we would like to highlight.

Here is a list of additional videos and topics we really enjoyed - please let us know if we missed anything of interest and we will make sure to mention it in the next post!

Upcoming Events 📆

We are happy to announce that we have a number of events coming up in March - tune in to learn more about Flux and GitOps best practices, get to know the team and join our community.

2nd March: GitOps Testing in Kubernetes with Flux & Testkube
7th March: GitOps: Automatic Deployments and Updates with Flux w/ Julian Hennig
9th March: CNCF On Demand- Microservices and Kubernetes
15th March: CNCF Live Stream - Automating Kubernetes Deployments
16th March: CNCF On Demand- Kubernetes in 2023 w/ Stefan Prodan & Brendan Burns
23rd March: Microsoft Live Stream: Automating Kubernetes Deployments

Flux Bug Scrub

The next dates are going to be:

In other news

Gitlab adopts Flux for GitOps

Have some great news about @fluxcd and @gitlab. In todays dev meeting, @nagyviktor shared with us that GitLab has chosen Flux to offer #GitOps to their users. The Flux team is super excited about collaborating with GitLab 🎉 https://t.co/3f9tHJbhnX
— Stefan Prodan (@stefanprodan) January 26, 2023

We are incredibly pleased that GitLab chose to move forward with Flux for the GitOps capabilities in their project. In the past weeks, members of the GitLab team joined our Dev meetings where it became clearer what needs to happen next. This is another great recognition of the versatility and great feature set of Flux and we very much look forward to the collaboration.

Please check out the announcement on the GitLab blog, which links to all the individual discussions and development epics where you can track the progress of the integration.

People writing/talking about Flux

TrueLayer: Flux2 migration: how we dropped our CPU usage by nearly 40x

We love hearing end-user success stories, particularly to learn how a migration went well. Surya Pandian wrote up the entire experience in the blog post and comes to this conclusion:

With our original Flux setup, we were running one pod per GitOps, and with 40 teams, that required a lot of cash and CPU. But with this setup, we run just one flux GitOps agent for an entire cluster. In total, one flux GitOps agent manages over 40 GitRepoCRDResources and 240 FluxKustomizeCRDResources.

Our migration to Flux2 has paved the way for a config-managed setup. Not only did this drastically reduce costs, but it also made Flux reconciliations faster and reduced CPU usage by almost 40x.

As the sun sets on Flux1, migrating to Flux2 may sound like a daunting task. But with the right migration plan, engineering teams can reap the benefits.

Flux + Testkube: GitOps Testing is here

Abdallah Abedraba describes how to set up Flux with Testkube in the blog post in an easy to follow step-by-step fashion. The takeaway is:

Once fully realized - using GitOps for testing of Kubernetes applications as described above provides a powerful alternative to a more traditional approach where orchestration is tied to your current CI/CD tooling and not closely aligned with the lifecycle of Kubernetes applications.

This tutorial uses Postman collections for testing an API, but you can bring your a whole suite of tests with you to Testkube.

News from the Website and our Docs

Flux Adopters shout-out

We are very pleased to announce that the following adopters of Flux have come forward and added themselves to our website: B1 Systems GmbH and Wildlife Studios.

More docs and website news

We are constantly improving our documentation and website - here are a couple of small things we landed recently:

Bootstrapping: here’s how to disable Kubernetes cluster role aggregations
Update image-updates guide to reflect the new API version and recent use of flags, extend examples.
We updated the docs to reflect current Flux version and fixed typos and readability pieces in many many places.
We updated our Security Docs.

Thanks a lot to these folks who contributed to docs and website: Ben Bodenmiller, Stefan Prodan, Stefan Bodenmiller, Michael Bridgen, Hidde Beydals, Sunny, Kingdon Barrett, Mac Chaffee, Ronan, Sanskar Jaiswal, zipizapclouds.

Flux Project Facts

We are very proud of what we have put together. We want to reiterate some Flux facts - they are sort of our mission statement with Flux.

🤝 Flux provides GitOps for both apps or infrastructure. Flux and Flagger deploy apps with canaries, feature flags, and A/B rollouts. Flux can also manage any Kubernetes resource. Infrastructure and workload dependency management is built-in.
🤖 Just push to Git and Flux does the rest. Flux enables application deployment (CD) and (with the help of Flagger) progressive delivery (PD) through automatic reconciliation. Flux can even push back to Git for you with automated container image updates to Git (image scanning and patching).
🔩 Flux works with your existing tools: Flux works with your Git providers (GitHub, GitLab, Bitbucket, can even use s3-compatible buckets as a source), all major container registries, fully integrates with OCI and all CI workflow providers.
🔒 Flux is designed with security in mind: Pull vs. Push, least amount of privileges, adherence to Kubernetes security policies and tight integration with security tools and best-practices. Read more about our security considerations.
☸️ Flux works with any Kubernetes and all common Kubernetes tooling: Kustomize, Helm, RBAC, and policy-driven validation (OPA, Kyverno, admission controllers) so it simply falls into place.
🤹 Flux does Multi-Tenancy (and “Multi-everything”): Flux uses true Kubernetes RBAC via impersonation and supports multiple Git repositories. Multi-cluster infrastructure and apps work out of the box with Cluster API: Flux can use one Kubernetes cluster to manage apps in either the same or other clusters, spin up additional clusters themselves, and manage clusters including lifecycle and fleets.
✨ Dashboards love Flux: No matter if you use one of the Flux UIs or a hosted cloud offering from your cloud vendor, Flux has a thriving ecosystem of integrations and products built on top of it and all have great dashboards for you.
📞 Flux alerts and notifies: Flux provides health assessments, alerting to external systems and external events handling. Just “git push”, and get notified on Slack and other chat systems.
👍 Users trust Flux: Flux is a CNCF Graduated project and was categorised as “Adopt” on the CNCF CI/CD Tech Radar (alongside Helm).
💖 Flux has a lovely community that is very easy to work with! We welcome contributors of any kind. The components of Flux are on Kubernetes core controller-runtime, so anyone can contribute and its functionality can be extended very easily.

Over and out

If you like what you read and would like to get involved, here are a few good ways to do that:

Join our upcoming dev meetings on 2023-03-09 or 2023-03-15.
Talk to us in the #flux channel on CNCF Slack
Join the planning discussions
And if you are completely new to Flux, take a look at our Get Started guide and give us feedback
Social media: Follow Flux on Twitter, join the discussion in the Flux LinkedIn group.

We are looking forward to working with you.

Blog: How Flux and Pulumi give each other superpowers

Tue, 14 Feb 2023 08:30:00 +0000

Pulumi is an “Infrastructure as Code” tool that lets you specify your infrastructure as programs written in JavaScript, Python, Java, Go, .NET languages, or YAML. The Pulumi Kubernetes operator drives Pulumi from Kubernetes, so you can maintain your infrastructure by pushing commits to git and letting automation take it from there.

Recently, we added support to the operator for using Flux sources. This is a great addition to the operator, but it’s not the only way Flux and Pulumi can work together.

Below I’m going to talk about how Flux and Pulumi can be combined, and the superpowers each grants to the other.

Adding OCI support and supply chain security to Pulumi

The support for Flux sources in the Pulumi operator gives you more ways to make your Pulumi programs available to the operator. Notably, you can now package your program as an OCI image and push it to an image registry, before using it with the operator. This might be appealing if you have deployment pipelines to generate Kubernetes YAML (e.g., from Cue) or other data, before using it in a program. It’s also convenient when you’re on a platform like AWS, GCP or Azure, because these have OCI registries with useful operational features like caching, security scanning, and so on.

But I think an even better reason for using Flux is that it can verify your sources. If you use a Flux source in a Pulumi stack, you can better secure your supply chain. When you’re using an OCI repository source for example, Flux will check Cosign signatures on each image for you, and refuse to update a source that does not have a valid signature.

A more subtle security benefit of Flux sources is context-based authorization. For example, in AWS the Flux controller can take advantage of workload identity to gain access to an ECR container registry containing your sources, so that you don’t have to explicitly manage credentials.

Using Pulumi to extend the reach of Flux

With Pulumi you are not restricted to Kubernetes resources – you can create any resource defined in a provider, e.g., within AWS, GCP, and Azure, among many such platforms and services. You can write a Pulumi program in YAML, and you can declare a YAML program as a Kubernetes custom resource, making the whole chain amenable to Kubernetes Resource Model (KRM)-oriented tooling, including Flux.

For example, here’s Kubernetes YAMLs for creating an AWS EC2 instance, and a security group, with Pulumi:

---
# This is a program for creating the EC2 instance and security group
apiVersion: pulumi.com/v1
kind: Program
metadata:
 name: ec2-instance
program:
 resources:
 group:
 type: aws:ec2:SecurityGroup
 properties:
 description: Enable HTTP access
 ingress:
 - protocol: tcp
 fromPort: 80
 toPort: 80
 cidrBlocks: ["0.0.0.0/0"]
 server:
 type: aws:ec2:Instance
 properties:
 ami: ami-6869aa05
 instanceType: t2.micro
 vpcSecurityGroupIds: ${group.name}
 outputs:
 publicIp: ${server.publicIp}
 publicDns: ${server.publicDns}
---
# This stack tells the operator how to run the program
apiVersion: pulumi.com/v1
kind: Stack
metadata:
 name: dev-ec2-instance
spec:
 stack: squaremo/ec2-instance/dev
 programRef:
 name: ec2-instance
 destroyOnFinalize: true
 config:
 aws:region: us-east-1

You can commit these in files in git, and have them synced by Flux. Then the Pulumi operator will take over, create the infrastructure as it’s declared, and mark the Stack object as ready. As far as Flux knows, these are just regular Kubernetes resources – but now you can bring your whole infrastructure under control!

Using Flux to simplify Kubernetes for Pulumi

Pulumi makes a huge variety of infrastructure accessible by just writing programs. In the specific context of Kubernetes, many folks will find KRM-oriented tooling and YAML files easier. Flux’s Kustomize and Helm controllers work alongside its source controller, but happily they also work in harmony with the Pulumi operator, and you can mix and match to suit yourself.

For example, you might find it useful to write all your Pulumi Program and Stack YAMLs as files in a directory for Flux to sync, rather than trying to create them in Pulumi code (or – horror – applying them by hand).

Getting started with Flux and the Pulumi operator

If you are already invested in Pulumi, it would make sense to bootstrap Flux, using Pulumi. You can use the Flux provider for Pulumi from your Pulumi program, and gain verified sources and effortless YAML syncing.

And, vice versa – if you are starting with Flux and want to expand its reach with Pulumi, you can bootstrap the Pulumi operator, using Flux, by syncing the deployment manifests in the operator’s GitHub repo:

---
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: GitRepository
metadata:
 name: pulumi-operator
 namespace: flux-system
spec:
 interval: 5m0s
 ref:
 semver: "1.11.x"
 url: https://github.com/pulumi/pulumi-kubernetes-operator
---
apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
 name: deploy-pulumi-operator
 namespace: flux-system
spec:
 interval: 5m0s
 path: ./deploy/yaml
 prune: true
 sourceRef:
 kind: GitRepository
 name: pulumi-operator

Visit Pulumi’s “Get Started” portal to learn more about what you can accomplish with Pulumi.

Blog: January 2023 Update

Wed, 01 Feb 2023 08:30:00 +0000

Now it’s the beginning of February 2023 - let’s recap together what happened in December and January - it has been a lot!

News in the Flux family

Flux 0.38 brings performance improvements and new features

We have released Flux v0.38. Users are encouraged to upgrade for the best experience. Here is a short summary of its features and improvements:

Graduation of Notification APIs to v1beta2, to upgrade please see the release notes.
Support for defining Kustomize components with Kustomization.spec.components.
Support for piping multi-doc YAMLs when publishing OCI artifacts with kustomize build . | flux push artifact --path=-.
Support for Gitea commit status updates with Provider.spec.type set to gitea.
Improve the memory usage of helm-controller by disabling the caching of Secret and ConfigMap resources.
Update the Helm SDK to v3.10.3 (fix for Helm CVEs).
All code references to libgit2 were removed, and the GitRepository.spec.gitImplementation field is no longer being honored.

The official example repository was refactored. The new version comes with the following improvements:

Make the example compatible with ARM64 Kubernetes clusters.
Add Weave GitOps Helm release to showcase the Flux UI.
Replace the ingress-nginx Bitnami chart with the official one that contains multi-arch container images.
Add cert-manager Helm release to showcase how to install CRDs and custom resources using dependsOn.
Add Let’s Encrypt ClusterIssuer to showcase how to patch resources in production with Flux Kustomization.
Add the flux-system overlay to showcase how to configure Flux at bootstrap time.

♥ Big thanks to all the Flux contributors that helped us with this release!

Security news

Flux 0.39, the upcoming release, will come with SBOMs and SLSA Provenance attached to all the controllers container images. In addition, all controller images will be updated to Alpine 3.17 (which contains CVE fixes for OS packages).

Starting with 0.39, the Flux controllers should consume less memory on busy clusters due to the disabling of Secret and ConfigMap caching.

Flagger 1.27 and 1.28 add support for APISIX and different autoscaling configs

1.28 comes with support for setting a different autoscaling configuration for the primary workload. The .spec.autoscalerRef.primaryScalerReplicas is useful in the situation where the user does not want to scale the canary workload to the exact same size as the primary, especially when opting for a canary deployment pattern where only a small portion of traffic is routed to the canary workload pods.

1.27 comes with support for Apache APISIX. For more details see the tutorial.

Flux Ecosystem

Flux Subsystem for Argo

Flamingo is a tool that combines Flux and Argo CD to provide the best of both worlds for implementing GitOps on Kubernetes clusters. With Flamingo, you can:

Automate the deployment of your applications to Kubernetes clusters and benefit from the improved collaboration and deployment speed and reliability that GitOps offers.
Enjoy a seamless and integrated experience for managing deployments, with the automation capabilities of Flux embedded inside the user-friendly interface of Argo CD.
Take advantage of additional features and capabilities that are not available in either Flux or Argo CD individually, such as the robust Helm support from Flux, Flux OCI Repository, Weave GitOps Terraform Controller for Infrastructure as Code, Weave Policy Engine, or Argo CD ApplicationSet for Flux-managed resources.

In recent releases, the team updated Flamingo to support Flux v0.38 and Argo CD v2.5.7, v2.4.19 and v2.3.13. Please note that Argo CD v2.2 will not be supported and updated by Flamingo anymore.

Flux	Argo CD	Image
v0.38	v2.5	v2.5.9-fl.3-main-14aff24e
v0.38	v2.4	v2.4.21-fl.3-main-14aff24e
v0.38	v2.3	v2.3.15-fl.3-main-14aff24e
v0.37	v2.2	v2.2.16-fl.3-main-2bba0ae6

Terraform-controller

The tf-controller team is currently working on getting the new release v0.14 out. They are updating the Terraform binary to version 1.3.7 and the Flux tool to version 0.38. Additionally, they are fixing the Helm chart and enabling the parallelism option for the apply stage. They are currently at release candidate v0.14.0-rc.2 with the new Helm chart version 0.10.0. Please stay tuned for further updates.

Weave GitOps

Besides a huge amount of general small improvements, the team has fixed two security vulnerabilities ( 1, 2) and made GitOps Run much more secure along the way. If you’re using a version older than 0.12.0 you are highly encouraged to upgrade.

Also with GitOps Run you can now open the deployed application’s Web UI by simply hitting a key on your keyboard. GitOps Run sets up the port-forwarding and opens up a browser window for you.

As always lots of improvements went into Weave GitOps’ Web UI so make sure to take a look.

On the Weave GitOps Enterprise side you can now automatically create Pipelines from GitOpsTemplates, the Terraform UI has been improved to allow for a more detailed view into a Terraform inventory and support for observing and managing Secrets has landed in its initial incarnation.

Recent & Upcoming Events

It’s important to keep you up to date with new features and developments in Flux and provide simple ways to see our work in action and chat with our engineers.

Recent Events (ICYMI) 📺

We feel blessed to have such a big community of users, contributors and integrators and so many are happy to talk about their experiences. In December and January here are a couple of talks we would like to highlight.

HashiCorp User Group Luxembourg: GitOps your Terraform Configurations

Flux Terraform Controller is a controller for Flux to reconcile Terraform configurations in the GitOps way with the power of Flux and Terraform, Terraform Controller allows you to GitOps-ify your infrastructure, and your application resources, in the Kubernetes and Terraform universe.

Flux Terraform Controller ensures what you’ve defined in your Terraform configurations is what’s always running and available. Flux continuously looks for changes and reconciles with the desired state. Take advantage of all the benefits of GitOps: streamlined and secure deployments, quicker time to market, and more time to concentrate on app development!

Flux’s Security & Scalability with OCI & Helm (Part 2) with Kingdon Barrett

With Flux, you can distribute and reconcile Kubernetes configuration packaged as OCI artifacts. Instead of connecting Flux to a Git repository where the application desired state is defined, you can connect Flux to a container registry where you’ll push the application deploy manifests, right next to the application container images.

During this session Kingdon Barrett, OSS Engineer at Weaveworks & Flux Maintainer, shows you how to quickly create scalable and Cosign-verified GitOps configurations with Flux using the same process with two demo environments: one will be a Kustomize Environment and the other a Helm-based environment.

Flux Security & Scalability using VS Code GitOps Extension

Recently Flux has released two new features (OCI and Cosign) for scalable and secure GitOps. Juozas Gaigalas, a Developer Experience Engineer at Weaveworks, will demonstrate how developers and platform engineers can quickly create scalable and Cosign-verified GitOps configurations using VS Code GitOps Tools extension. New and experienced Flux users can learn about Flux’s OCI and Cosign support through this demo. Join us!

Here is a list of additional videos and topics we really enjoyed - please let us know if we missed anything of interest and we will make sure to mention it in the next post!

Upcoming Events 📆

We are happy to announce that we have a number of events coming up in February - tune in to learn more about Flux and GitOps best practices, get to know the team and join our community.

Flux Bug Scrub

The next dates are going to be:

In other news

Conference Call For Papers

Conferences are all about the people. It’s also more fun to present together. You get to share collective experience and be more entertaining as a duo!

Two upcoming call for paper deadlines are the following

CFP until 2023-02-05, SustainabilityCon

May 10 – 12, 2023 | Vancouver, Canada Join the community of developers, technologists, sustainability leaders and anyone working on technological solutions to decarbonize the global economy, mitigate and address the impacts of climate change, and build a more sustainable future. SustainabilityCon provides a forum to drive open source innovation in energy efficiency and interoperability and clean development practices within industries ranging from manufacturing to agriculture and beyond through collaboration and learning within the community.
CFP until 2023-02-10, GitOpsCon

May 8 – 9, 2023 | Vancouver, Canada

cdCon + GitOpsCon is designed to foster collaboration, discussion, and knowledge sharing by bringing two communities together. It’s the best place for vendors and end users to collaborate in shaping the future of GitOps and Continuous Delivery (CD).

Talk to Niki Manoledaki for SustainabilityCon and in general to Vanessa Abankwah and Stacey Potter if you want to present anything Flagger, Flux, GitOps related at any of the events with us!

Soulé Ba joins Flux Core Maintainers

Soulé Ba has been working on Flux for a long while. Already a maintainer of Flux’s go-git-providers, he didn’t stop there but was involved in a lot of the RFC planning process of many features and contributed code and fixes for a long long time.

The Flux community is grateful to have you. Well deserved becoming a Core maintainer now, Soulé!

Your Community Team

We have been working on filling up the speakers calendar for the next weeks and organising proposals for the upcoming CFP deadlines for the next conferences. If you are interested in speaking about Flux and GitOps, please reach out to us!

Next up we are going to look into making our Community page more interesting and useful. We are also going to apply for Google Season of Docs. If you have input or ideas and would like to get involved, talk to us on Slack!

People writing/talking about Flux

Bill Doerrfeld: Introduction to Flux (containerjournal.com)

Read more in this article about Flux, where Bill interviewed Priyanka “Pinky” Ravi about what’s new in Flux. It’s a nice introduction to Flux.

GitOps has become a chosen strategy for releasing and deploying cloud-native microservices. The goal of GitOps, a term coined by Alexis Richardson, CEO of Weaveworks, in 2017, is to “make operations automatic for the whole system based on a model of the system which was living outside the system.” And propelling the GitOps practice is Flux, an open source tool that provides GitOps for apps and infrastructure.

In late 2022, Flux became the 18th project to reach graduation status with the Cloud Native Computing Foundation (CNCF). Earlier this year, downloads of the Flux container image surpassed a staggering one billion.

Max Strübing: Automatic deployment updates with Flux (D2iQ Engineering Blog)

We were very pleased to see this blog post from our friends at D2iQ. Do go check it out, particularly if you are new to Flux. Max takes a how-to approach to explaining automatic deployment updates with Flux and explains why this is generally a good idea:

You can deploy fast, easily and often by simply pushing to a repository

You can run a git revert if you messed up your environment and everything is like it was before

This means you can easily roll back to every state of your application or infrastructure

Not everyone needs access to the actual infrastructure environment, access to the git repository is enough to manage the infrastructure

Self-documenting infrastructure: you do not need to ssh into a server and look around running services or explore all resources on a Kubernetes cluster

Easy to create a demo environment by replicating the repository or creating a second deploy target

News from the Website and our Docs

Flux Adopters shout-out

We are very pleased to announce that the following adopters of Flux have come forward and added themselves to our website: DoneOps and Riley.

More docs and website news

We are constantly improving our documentation and website - here are a couple of small things we landed recently:

The Flux landing page is shorter and less overwhelming now. This was achieved by moving the adopters logos into a horizontal scroll band, dropping some old content and there will be more to come here.
Flagger docs were update to the latest.
Flux Bootstrap: cheatsheet for how to Persistent storage for Flux internal artifacts
Our FAQ now has entries about how to safely rename a Flux Kustomization and how to set local overrides to a Helm chart. As it’s one of the very common FAQs: We also mention the different Flux UIs a lot more prominently now!
Flux GCP docs were updated.
We improved the Flux Support page to be even clearer about how to get Support for Flux, no matter if it’s professionally or for community support.
We renived a lot of unnecessary website build code; now a lot of the dynamic content is generated straight from YAML through Hugo Data Templates. This makes the website build process a lot more stable and we have less build scripts to maintain!
Update to latest hugo plus docsy and gallery themes.

Thanks a lot to these folks who contributed to docs and website: Stefan Prodan, Arhell, Aurel Canciu, Hidde Beydals, Sanskar Jaiswal, h20220026, Paulo Gomes, Stacey Potter, Johannes Wienke, Jonathan Meyers, Kingdon Barrett, Lassi Pölönen, Max Jonas Werner, Nate, Scott Rigby, Sunny, Tarunbot, h20220025, surya.

Flux Project Facts

We are very proud of what we have put together. We want to reiterate some Flux facts - they are sort of our mission statement with Flux.

🤝 Flux provides GitOps for both apps or infrastructure. Flux and Flagger deploy apps with canaries, feature flags, and A/B rollouts. Flux can also manage any Kubernetes resource. Infrastructure and workload dependency management is built-in.
🤖 Just push to Git and Flux does the rest. Flux enables application deployment (CD) and (with the help of Flagger) progressive delivery (PD) through automatic reconciliation. Flux can even push back to Git for you with automated container image updates to Git (image scanning and patching).
🔩 Flux works with your existing tools: Flux works with your Git providers (GitHub, GitLab, Bitbucket, can even use s3-compatible buckets as a source), all major container registries, and all CI workflow providers.
🔒 Flux is designed with security in mind: Pull vs. Push, least amount of privileges, adherence to Kubernetes security policies and tight integration with security tools and best-practices. Read more about our security considerations.
☸️ Flux works with any Kubernetes and all common Kubernetes tooling: Kustomize, Helm, RBAC, and policy-driven validation (OPA, Kyverno, admission controllers) so it simply falls into place.
🤹 Flux does Multi-Tenancy (and “Multi-everything”): Flux uses true Kubernetes RBAC via impersonation and supports multiple Git repositories. Multi-cluster infrastructure and apps work out of the box with Cluster API: Flux can use one Kubernetes cluster to manage apps in either the same or other clusters, spin up additional clusters themselves, and manage clusters including lifecycle and fleets.
📞 Flux alerts and notifies: Flux provides health assessments, alerting to external systems and external events handling. Just “git push”, and get notified on Slack and other chat systems.
👍 Users trust Flux: Flux is a CNCF Graduated project and was categorised as “Adopt” on the CNCF CI/CD Tech Radar (alongside Helm).
💖 Flux has a lovely community that is very easy to work with! We welcome contributors of any kind. The components of Flux are on Kubernetes core controller-runtime, so anyone can contribute and its functionality can be extended very easily.

Over and out

If you like what you read and would like to get involved, here are a few good ways to do that:

Join our upcoming dev meetings on 2023-02-09 or 2023-02-15.
Talk to us in the #flux channel on CNCF Slack
Join the planning discussions
And if you are completely new to Flux, take a look at our Get Started guide and give us feedback
Social media: Follow Flux on Twitter, join the discussion in the Flux LinkedIn group.

We are looking forward to working with you.

Blog: November 2022 Update

Mon, 05 Dec 2022 12:30:00 +0000

It’s the beginning of December 2022 - let’s recap together what happened in November - it has been a lot!

News in the Flux family

Flux has graduated

It’s been quite the journey, and it wouldn’t have been possible without everybody’s help in our community. We made it! Flux is now officially a CNCF Graduated project. Here are some news pieces you might want to check out:

Please help us share the good news - it’s the moment of recognition and endorsement many have still been waiting for!

Also please join us for our celebratory Flux Graduation AMA sessions:

December 7, 12:00 UTC with Flux maintainers: Daniel, Max, Philip, Sanskar, Stefan, Somtochi
December 8, 18:00 UTC with Flux maintainers: Kingdon, Paulo, Somtochi, Soulé

Next Flux release brings consolidated Git implementation

The Flux development team keeps on innovating. The latest release is Flux v0.37 and as always we encourage you all to upgrade for the best experience.

The biggest change is that the gitImplementation field of GitRepository by source-controller and image-automation-controller is now deprecated. Flux will effectively always use go-git. This now supports all Git servers, including Azure DevOps and AWS CodeCommit, which previously were only supported by libgit2. This is a big improvement and will help us focus on making Flux work great with just one git implementation.

Here is our shortlist of features and improvements in the release:

Support for bootstrapping Azure DevOps and AWS CodeCommit repositories using flux bootstrap git.
Support cloning of Git v2 protocol (Azure DevOps and AWS CodeCommit) for go-git Git provider.
Support force-pushing ImageUpdateAutomation repositories.
Allow a dry-run of flux build kustomization with --dry-run and --kustomization-file ./path/to/local/my-app.yaml. Using these flags, variable substitutions from Secrets and ConfigMaps are skipped, and no connection to the cluster is made.
Use signed OCI Helm chart for kube-prometheus-stack.

Check out these new pieces of documentation:

Guide: AWS CodeCommit bootstrap
Guide: Azure DevOps bootstrap

💖 Big thanks to all the Flux contributors that helped us with this release!

Flux Roadmap Updates

Here’s an update from the Flux roadmap - we are rushing forward towards GA!

Starting with release v0.37, we started solidifying all required changes for the Bootstrap GA milestone targeted to Q1 2023. That release should include all major changes from a Git perspective that we want to ship for GA. Please make sure you upgrade as soon as possible and provide us with feedback, so we can work on it before the GA release.

Upcoming in the next release is a new feature for Image Automation Controller: GitShallowClones. You can already check it out in the recently published release candidate. If you are interested, you can reach out via the PR or on Slack: https://github.com/fluxcd/image-automation-controller/pull/463

Security news

To benefit from our strong OCI integration, you might want to take a look at our latest blog post about how to verify the integrity of Helm charts stored as OCI artifacts.

To help you tighten security, the Kubernetes community has released the security-profiles-operator project. We are very pleased that it now comes with an AppArmor profile for Flux.

Flagger 1.25 and 1.26 update to newest Gateway API

Flagger 1.26.0 comes with support Kubernetes Gateway API v1beta1. For more details see the Gateway API Progressive Delivery tutorial. Please note that starting with this version, the Gateway API v1alpha2 is considered deprecated and will be removed from Flagger after 6 months.

Flagger 1.25.0 introduces a new deployment strategy combining Canary releases with session affinity for Istio. Check out the tutorial here. Furthermore, it contains a regression fix regarding metadata in alerts introduced in #1275.

Flux Ecosystem

Flux Subsystem for Argo

The team upgraded Flux Subsystem for Argo aka Flamingo to support Flux v0.37 and Argo CD v2.5.3, v2.4.17, v2.3.11 and v2.2.16.

Terraform-controller

The team has released Weave TF-controller v0.13.1 and recently updated its Helm chart to v0.9.3. In this version, the team started shipping the AWS Package for TF-controller. The AWS Package is an OCI Image which contains a set of Terraform primitive modules that you can use out-of-the-box to provision your Terraform resources by describing them as YAML.Please visit the package repository for more information: https://github.com/tf-controller/aws-primitive-modules.

Weave GitOps

GitOps Run continues to be enhanced as an easy way to get started with Flux and GitOps, and now includes yaml validation for both Flux and core Kubernetes resources. The Weave GitOps UI for Flux is now able to support multiple instances of Flux on the same cluster, for when resource isolation strategies are in place, so you can see the health of all controllers in the Flux Runtime view.

Then in the Enterprise edition of Weave GitOps, the Pipelines feature is now enabled by default to help you automatically promote applications through a series of environments, and GitOpsTemplates continue to be enhanced as a generic self-service capability for building out an Internal Developer Platform.

VS Code GitOps Extension

In its latest pre-release of the extension a “Configure GitOps” workflow was introduced. It features a new unified user interface for creating Source and Workload and for attaching Workloads to Sources. It supports both Generic Flux and Azure Flux (Arc/AKS) cluster modes. In Azure mode, FluxConfig resources are created automatically (this can be disabled if the user wants Generic mode compatibility). Currently this feature is in the Extension Marketplace pre-release channel and supports GitRepository and Kustomization resources.

If you want a user-friendly UI for working with every type of Source and Workflow please check out this pre-release and give the team feedback!

Recent & Upcoming Events

It’s important to keep you up to date with new features and developments in Flux and provide simple ways to see our work in action and chat with our engineers.

Recent Events (ICYMI) 📺

We feel blessed to have such a big community of users, contributors and integrators and so many are happy to talk about their experiences. In November here are a couple of talks we would like to highlight.

Playlist: Flux at Prometheus Day, GitOpsCon, & KubeCon North America 2022

This playlist is a curated compilation of all Flux related talks from KubeCon / CloudNativeCon NA 2022 (Detroit) as well as the respective co-located events, Prometheus Day and GitOpsCon. We’ve also included a list of the individual videos below.

Prometheus Day North America 2022

Automate Your SLO Validation with Prometheus & Flagger - Sanskar Jaiswal & Kingdon Patrick Barrett

GitOpsCon North America 2022

KubeCon North America 2022

Here is a list of additional videos and topics we really enjoyed - please let us know if we missed anything of interest and we will make sure to mention it in the next post!

Upcoming Events 📆

We are happy to announce that we have a number of events coming up in December- tune in to learn more about Flux and GitOps best practices, get to know the team and join our community.

Dec 13: Implementing Flux for Scale with Soft Multi-tenancy

The Flux project continues in active development with the addition of OCI configuration planned in the GA roadmap. Another Flux advancement has been the creation of the new VSCode Extension which provides a convenient interface to Flux that can help reduce friction moving between editor and terminal, alleviating the headache of context switching overloading developer focus.

Flux maintainer Kingdon Barrett will demonstrate the pre-release of Flux’s new OCI features and a convenient way to access them while they remain in pre-release so you can provide the feedback that is needed by Flux maintainers to make this feature a success!

Flux Bug Scrub

The next dates are going to be:

In other news

Your Community Team

The Flux Community Team has been busy this month. We wrapped-up everything related to KubeCon, prepared the announcement of Flux Graduation and wrote this summary.

We would love your help, so if you are interested in joining a small team which handles Community and Communications of Flux, please join our meetings and introduce yourself!

People writing/talking about Flux

Josh Carlisle wrote this blog post as a decision making help for people who are new to Cloud Native. He says

I came away with Flux offering some easier onboarding and bootstrapping

and

I found Flux to better align with things that were important to me

Thanks for the shout-out!

News from the Website and our Docs

Flux Adopters shout-out

We are very pleased to announce that the following adopters of Flux have come forward and added themselves to our website: Amesto Fortytwo, DataGalaxy, Divistant, DKB Deutsche Kreditbank AG, Housing Anywhere, synyx.

More docs and website news

We are constantly improving our documentation and website - here are a couple of small things we landed recently:

Following the deprecation of Flux Legacy, we have removed the Flux Legacy docs and highlighted migration videos and other helpful content
To make it easier to participate, we now show the upcoming event on the landing page
Show adopters logos in horizontal scroll band
Updated Flagger docs to 1.25.0
Updated AWS CodeCommit docs
Updated Azure docs
Added GitOpsCon talk videos
Many other improvements and fixes

Thanks a lot to these folks who contributed to docs and website: Stefan Prodan, Arhell, Vanessa Abankwah, David Harris, Sanskar Jaiswal, Batuhan Apaydın, Max Jonas Werner, André Kesser, Marko Petrovic, Matthieu Dufourneaud, Paul Lockaby, Paulo Gomes, Piotr Sobieszczański, Roberth Strand, Tarun Rajpurohit, husni6, surya.

Flux Project Facts

We are very proud of what we have put together. We want to reiterate some Flux facts - they are sort of our mission statement with Flux.

🤝 Flux provides GitOps for both apps or infrastructure. Flux and Flagger deploy apps with canaries, feature flags, and A/B rollouts. Flux can also manage any Kubernetes resource. Infrastructure and workload dependency management is built-in.
🤖 Just push to Git and Flux does the rest. Flux enables application deployment (CD) and (with the help of Flagger) progressive delivery (PD) through automatic reconciliation. Flux can even push back to Git for you with automated container image updates to Git (image scanning and patching).
🔩 Flux works with your existing tools: Flux works with your Git providers (GitHub, GitLab, Bitbucket, can even use s3-compatible buckets as a source), all major container registries, and all CI workflow providers.
🔒 Flux is designed with security in mind: Pull vs. Push, least amount of privileges, adherence to Kubernetes security policies and tight integration with security tools and best-practices. Read more about our security considerations.
☸️ Flux works with any Kubernetes and all common Kubernetes tooling: Kustomize, Helm, RBAC, and policy-driven validation (OPA, Kyverno, admission controllers) so it simply falls into place.
🤹 Flux does Multi-Tenancy (and “Multi-everything”): Flux uses true Kubernetes RBAC via impersonation and supports multiple Git repositories. Multi-cluster infrastructure and apps work out of the box with Cluster API: Flux can use one Kubernetes cluster to manage apps in either the same or other clusters, spin up additional clusters themselves, and manage clusters including lifecycle and fleets.
📞 Flux alerts and notifies: Flux provides health assessments, alerting to external systems and external events handling. Just “git push”, and get notified on Slack and other chat systems.
👍 Users trust Flux: Flux is a CNCF Graduated project and was categorised as “Adopt” on the CNCF CI/CD Tech Radar (alongside Helm).
💖 Flux has a lovely community that is very easy to work with! We welcome contributors of any kind. The components of Flux are on Kubernetes core controller-runtime, so anyone can contribute and its functionality can be extended very easily.

Over and out

If you like what you read and would like to get involved, here are a few good ways to do that:

Join our upcoming dev meetings on 2022-12-07 or 2022-12-15.
Talk to us in the #flux channel on CNCF Slack
Join the planning discussions
And if you are completely new to Flux, take a look at our Get Started guide and give us feedback
Social media: Follow Flux on Twitter, join the discussion in the Flux LinkedIn group.

We are looking forward to working with you.

Blog: Flux is a CNCF Graduated project

Wed, 30 Nov 2022 09:00:00 +0000

Flux has graduated

Today is a very exciting day for the Flux community! Flux is now a graduated project in the Cloud Native Computing Foundation and joining the ranks of Kubernetes, Helm, Prometheus and others in this category.

Flux History

We all worked very hard to make this happen - it is another important milestone in the Flux success story. Started in July 2016, engineers at Weaveworks built the first version of Flux to guarantee predictable deployments internally. This was way before Kubernetes had won the Cloud Native market.

In the coming years at Weaveworks, the learnings with Flux helped to establish and refine the principles of GitOps. Flux was integrated ever more closely with Kubernetes, and later on Helm and Kustomize. It also grew a community and an ecosystem. In 2018, Flagger was born, a Flux companion that made progressive delivery a natural extension of GitOps.

When Weaveworks donated Flux and Flagger to the CNCF, we already saw large-scale adoption growing and cloud vendors making the Flux suite core of their offerings to provide GitOps functionality.

This was also the point where we decided to rewrite Flux from scratch, using modern tooling such as controller-runtime and as a set of targeted controllers, which made Flux development a lot more straight-forward. In the past weeks we archived Legacy Flux and are very close to making Flux v2 GA. Watch this space for the announcement!

“We created Flux as open source from the beginning, in order to work out in the open. It was very gratifying therefore, and far from inevitable, that a loyal and mutually supportive community grew around it. Making that happen takes a lot of empathy and patience from all involved – so, thank you everyone, for carrying Flux ever further.”

– Michael Bridgen, co-creator of Flux

Flux’s home: the CNCF

Today is a great time to look back at our time in the CNCF. We wouldn’t be where we are today without the services and help of people at the CNCF. It wasn’t just the great benefits and infrastructure we enjoy as a project, but also the careful guidance and collaboration of CNCF groups such as the TOC, TAG Security / TAG Contributor Experience and all the adjacent project communities which also live at the CNCF. We also would like to thank our TOC sponsor, Matt Farina, who helped us navigate this process and encouraged us to take Flux even further!

“I feel humbled and honored to be part of the Flux & Flagger team for the past five years. With the help of our community, we have come a long way since Flux inception and the start of the GitOps movement. Today, Flux is an established continuous delivery solution for Kubernetes, trusted by organisations around the world and backed by vendors like Amazon AWS, D2iQ, Microsoft Azure, VMware, Weaveworks and others that offer Flux to their users. The Flux team is very grateful to the cloud-native community and CNCF who supported us over the years and made Flux what it is today”

– Stefan Prodan, Flux maintainer and creator of Flagger

During the Graduation process, we particularly reflected on security and governance. We threat-modelled the Flux components, which resulted in documented security best practice. We will continue to educate our user community on how to use Flux securely. Today both Flux and Flagger are 100% compliant with the CLO monitor, which is the highest score amongst graduated CNCF projects. We streamlined our security processes, and have regular conversations with security professionals from CNCF tag-security. Soon we are going to undergo a second security audit for an external validation of all the great work we have done over the last few years.

We are incredibly proud of what we have achieved and what we have given to the wider ecosystem. GitOps is close to becoming the de-facto standard. Cloud vendors offer GitOps capabilities to their customers these days, a lot of this is based on Flux as a technology and the learnings we made until today. We are extremely pleased to have this huge ecosystem built on top of and around Flux, including recent Flux UIs!

Next up: Flux going GA

The 2.0.0 release of Flux is drawing near as well!

While Flux has been production ready for quite some time, we have an extremely strict backwards compatibility policy and take major versions very seriously.

The Flux community was working on a number of concurrent projects at the same time: qualifying for Graduation, refactoring the controllers to standardise the internal APIs, stabilizing the use of APIs of e.g. Helm and Git, integrating OCI artifacts and Cosign verification fully into Flux, and more. All of these workstreams were happening at the same time. To make it clear to everyone what a GA release for Flux would look like, we’ve updated our GA roadmap. There are many important details here for those following the 2.0.0 release, but one thing that is very important to us is further stabilizing Flux and its APIs so it will be even easier for new community members to contribute and build on top of Flux!

💖 Huge thank-you

You are all rock stars! 🤩 Continued thanks to everyone of our Flux community members who have, in ways small and large, contributed to the success of Flux!

If you want to celebrate with us or are now more curious about Flux, please join us at our Flux Graduation Ask-Us-Anything sessions:

December 7, 12:00 UTC with Flux maintainers: Daniel, Max, Philip, Sanskar, Stefan, Somtochi
December 8, 18:00 UTC with Flux maintainers: Kingdon, Paulo, Somtochi, Soulé

We are looking forward to seeing you and getting to know you there!

Blog: Verify the integrity of the Helm Charts stored in OCI-compliant registries as OCI artifacts

Mon, 14 Nov 2022 10:30:00 +0000

Cosign integration was one of the most important features we shipped in the Flux v0.35 release. After that, we wrote a blog post which explains how to use the feature with OCIRepository resources which enables fetching OCI artifacts from container registries. If you haven’t read it yet, we highly encourage you to go and check it out first.

Flux v0.36.0 allows you to prove the authenticity of HelmChart resources with the help of the cosign integration. Here we will demonstrate how to use the cosign integration to verify the integrity of the Helm charts stored in OCI-compliant registries as OCI artifacts.

Not at #kubecon so I had time to prepare Flux and Flagger releases. Flagger v1.24 comes with signed releases & OCI Helm charts. Flux v0.36 adds support for verifying Helm charts with Cosign. https://t.co/6MYwzfMA3W
— Stefan Prodan (@stefanprodan) October 27, 2022

Starting with Helm v3.8.0, Helm supports the OCI registry as a one of the storage option for Helm charts as an alternative to Helm repositories. The Helm CLI can push and pull Helm charts to and from OCI-compliant registries.

Note: Prior to Helm v3.8.0, OCI support was experimental. To use it there, you need to enable the feature by setting the HELM_EXPERIMENTAL_OCI environment variable to 1.

As we store Helm charts in OCI-compliant registries as OCI artifacts, we can now use the cosign integration to sign and verify them. Also, thanks to Flux, you can reconcile resources such as plain-text Kubernetes YAML manifests, Terraform modules, etc. from OCI-compliant registries with the help of OCIRepository resources. You can achieve the same thing for Helm charts with HelmRepository resources. This means that you can store Helm charts in OCI-compliant registries as OCI artifacts and use Flux to reconcile them like the following:

---
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: HelmRepository
metadata:
 name: podinfo
 namespace: default
spec:
 type: "oci"
 interval: 5m0s
 url: oci://ghcr.io/stefanprodan/charts

Note:

Here you can review the complete registries lists that support the OCI artifact specification: OCI-Conformant Products.

You will notice that when you open the list, DockerHub is not included into the list but it will be added soon because they recently announced OCI Artifacts support, and you can read more about it from here.

Let’s jump right into the details of how we can actually use it.

We will deploy Prometheus by using its community Helm charts stored as OCI artifacts in OCI registry. Recently, Prometheus’ community started to publish their Helm charts to OCI registries and sign them with cosign using the keyless approach, you can learn more the process here. Then we are going to verify it with cosign and configure Flux to verify the Helm chart’s signatures before they are downloaded and reconciled. As the Prometheus community signed their Helm Charts without providing a key pair, we do not need to specify any key in the HelmChart resource’ provider.cosign spec to enable keyless verification for Flux.

For the sake of simplicity, we’ve deployed Prometheus alone but if you want to learn more about installing the Prometheus stack including Grafana, Alertmanager, etc., please refer to the official Flux page that can help you to do that.

You need three things to complete this demo;

cosign CLI
- https://docs.sigstore.dev/cosign/installation/
A Kubernetes cluster
- https://kind.sigs.k8s.io/#installation-and-usage
Flux CLI
- https://fluxcd.io/flux/cmd/

Let’s start by creating a simple Kubernetes cluster:

kind create cluster

Use the Flux CLI to do pre-flight checks:

$ flux check --pre
► checking prerequisites
✔ Kubernetes 1.25.3 >=1.20.6-0
✔ prerequisites checks passed

If the checks are successful, you can install Flux on the cluster.

Let’s install Flux on it - if you need to use other options, check out the installation page.

export GITHUB_USER=developer-guy
export GITHUB_TOKEN=$GITHUB_TOKEN

flux bootstrap github \
 --owner=$GITHUB_USER \
 --repository=flux-cosign-helm-oci-demo \
 --branch=main \
 --path=./clusters/my-cluster \
 --personal

Note: Don’t forget to change the values with your own details!

As we stick to GitOps practices, we only create files that contain the HelmRepository and HelmRelease resources. After committing and pushing those changes into the upstream repository, Flux will watch for changes and use them as source-of-truth for the configuration:

git clone git@github.com:developer-guy/flux-cosign-helm-oci-demo.git
cd flux-cosign-helm-oci-demo

Let’s create the HelmRepository resource first:

flux create source helm prometheus-community \
 --url=oci://ghcr.io/prometheus-community/charts \
 --interval=10m \
 --export > ./clusters/my-cluster/prometheus-community-helmrepository.yaml

Now, let’s move on with creating the HelmRelease resource:

flux create helmrelease prometheus \
 --source=HelmRepository/prometheus-community \
 --chart=prometheus \
 --interval=10m \
 --release-name prometheus \
 --target-namespace=monitoring \
 --create-target-namespace \
 --export > ./clusters/my-cluster/prometheus-helmrelease.yaml

and run the following command to add the verify section to the HelmRelease resource’ .spec.chart.spec section to enable keylesss verification:

yq e '.spec.chart.spec|=({"verify": { "provider": "cosign" } } +.)' ./clusters/my-cluster/prometheus-helmrelease.yaml

This command above will add the following part to the HelmRelease resource’s .spec.chart.spec section:

verify:
 provider: cosign

then, commit and push the changes:

git commit -m "Add prometheus HelmRelease and HelmRepository resources"
git push

After a couple of seconds, Flux will have applied these changes. Now let’s check the status of them:

Or, you can trigger the reconciliation immediately by running the simple command: flux reconcile source git flux-system

For the HelmRepository resource:

$ flux get sources helm
NAME REVISION SUSPENDED READY MESSAGE
prometheus-community False True Helm repository is ready

For the HelmRelease resource:

$ flux get helmreleases
NAME REVISION SUSPENDED READY MESSAGE
prometheus 15.18.0 False True Release reconciliation succeeded

If everything is fine, you can check the pods in the monitoring namespace:

$ kubectl get pods -n monitoring
NAME READY STATUS RESTARTS AGE
prometheus-alertmanager-54b7d7cf45-2b7zf 2/2 Running 0 115s
prometheus-kube-state-metrics-67f68d64bb-vlmvd 1/1 Running 0 115s
prometheus-node-exporter-46gm6 1/1 Running 0 115s
prometheus-pushgateway-596cd99697-t79zt 1/1 Running 0 115s
prometheus-server-c458cf6f9-nvstw 2/2 Running 0 115s

Great! Now, you have installed Prometheus with Flux by using the Helm chart stored in the OCI registry and verified it with cosign.

We can assume that the Helm chart’s signature is verified as we let it be deployed in a cluster but let’s do have double check and check the status of the HelmRelease to see whether the verification is successful or not:

$ kubectl get helmcharts -n flux-system flux-system-prometheus -ojsonpath='{.status.conditions[?(@.type=="SourceVerified")]}'
{"lastTransitionTime":"2022-11-09T13:27:38Z","message":"verified signature of version 15.18.0","observedGeneration":1,"reason":"Succeeded","status":"True","type":"SourceVerified"}

That’s super cool! Because Flux is going to add a condition to the HelmChart resource’s status section to show the verification status. If the verification is successful, it will add a condition like the one above.

DIY (Do it yourself) Approach

The Prometheus community Helm charts only serve as an example. Here is how you can do the same thing with your own Helm charts.

Create a Helm chart
Package the Helm chart as .tar.gz file
Login to the OCI-compliant registry that you want to use to store your Helm chart
Push the Helm chart as OCI artifact

Let’s create a sample directory that will contain our Helm chart:

mkdir -p helm-oci-demo
cd helm-oci demo

Now package the Helm chart as .tar.gz file:

 helm create nginx

Let’s package the Helm chart as .tar.gz file:

helm package nginx

Now, let’s login to the OCI-compliant registry that you want to use to store your Helm chart. In this example, we’ll be using the GitHub Container Registry (ghcr.io):

echo $GHCR_PAT | helm registry login ghcr.io -u $USER --password-stdin

Note: Don’t forget to change the values with your own details!

At this point, we are ready to push the Helm chart as OCI artifact:

$ helm push nginx-0.1.0.tgz oci://ghcr.io/$USER
Pushed: ghcr.io/developer-guy/nginx:0.1.0
Digest: sha256:21f92cbd63ab495d8fc44d54dabc4815c88d37697b3f8b757ca8e51ef178a2e7

So, the Helm chart is pushed to the OCI registry. It’s time to sign it with cosign. As cosign recommends we should always sign images based on their digests (@sha256:) rather than a tag. So, we should grab the digest from the command output above which is 21f92cbd63ab495d8fc44d54dabc4815c88d37697b3f8b757ca8e51ef178a2e7 in this case, and use that digest while signing the image:

As we saw the keyless approach before, let’s try the key-based approach this time. To do that, we should create public/private key pairs first:

cosign generate-key-pair

This command will generate two files, a cosign.pub which is a publickey and cosign.key which is a private key pair and store them in the current directory directory.

Now, let’s sign the image with the private key:

cosign sign --key cosign.key ghcr.io/$USER/nginx@21f92cbd63ab495d8fc44d54dabc4815c88d37697b3f8b757ca8e51ef178a2e7

Cool! Now we have signed the image with the private key. Let’s check the signature:

cosign verify --key cosign.key ghcr.io/$USER/nginx@21f92cbd63ab495d8fc44d54dabc4815c88d37697b3f8b757ca8e51ef178a2e7

Yay! It’s verified. But in order to make the public key accessible by Flux, we need to create a Kubernetes secret to store the public key:

kubectl -n flux-system create secret generic cosign-pub \
 --from-file=cosign.pub=cosign.pub

Now, we can use it in our Flux configuration. The rest of the steps are the same as the previous section. For the sake of simplicity, we won’t repeat them here other than the HelmRepository and HelmRelease resources’ creation steps.

Let’s create the HelmRepository resource first:

flux create source helm $USER-charts\
 --url=oci://ghcr.io/$USER \
 --interval=10m \
 --export > ./clusters/my-cluster/nginx-helmrepository.yaml

Let’s move on with creating the HelmRelease resource:

flux create helmrelease nginx \
 --source=HelmRepository/$USER-charts \
 --chart=nginx \
 --interval=10m \
 --release-name nginx \
 --target-namespace=default \
 --export > ./clusters/my-cluster/nginx-helmrelease.yaml

Don’t forget to run the following command to add the verify section to the HelmRelease resource’ .spec.chart.spec section to enable verification:

yq e '.spec.chart.spec|=({"verify": { "provider": "cosign", "secretRef": { "name": "cosign-pub" } } } +.)' ./clusters/my-cluster/nginx-helmrelease.yaml

This command above will add the following part to the HelmRelease resource’s .spec.chart.spec section:

verify:
 provider: cosign
 secretRef:
 name: cosign-pub

That’s all you need to do folks!

Congratulations! You have successfully signed your Helm chart with cosign with key-based approach and used it with Flux.

Blog: October 2022 Update

Tue, 01 Nov 2022 15:30:00 +0000

It’s the beginning of November 2022 - let’s recap together what happened in October - it has been a lot!

News in the Flux family

Flux v0.36 adds support for verifying Helm charts with Cosign

Team Flux has released Flux 0.36 which continues the integration of OCI features further into Flux. Here is a list of features and improvements that were added in the last release:

Verify OCI Helm charts signed by Cosign (including keyless) with HelmChart.spec.verify.
Allow publishing a single YAML file to OCI with flux push artifact <URL> --path=deploy/install.yaml.
Detect changes to local files before pushing to OCI with flux diff artifact <URL> --path=<local files>.
New Alert Provider type named generic-hmac for authenticating the webhook requests coming from notification-controller.
The Kustomization.status.conditions have been aligned with Kubernetes standard conditions and kstatus.
The kustomize-controller memory usage was reduced by 90% when performing artifact operations.

For this release we also added new documentation to our site:

Please upgrade for the best experience.

Security news

Keeping the Flux Community up to date on new Security features and ways to keep their organisations and clusters secure is important to us. We are very happy that Flux project member Batuhan Apaydın took the time to write this blog post about proving the authenticity of OCI artifacts. Please take a look at to get practical advice on how to make use of this.

What’s more? CLOMonitor is a service which checks open source project repositories to verify they meet project health best practices. With the last Flux release, we have hit 100% compliance with Linux Foundation security best practices.

We would also like to high five Alexander Block, a member of our community and maintainer of kluctl: he reported the last Flux CVE (CVE-2022-39272) Improper use of metav1.Duration allows for Denial of Service. Thanks a lot for helping out with this!

Flagger v1.24 comes with signed releases & OCI Helm charts

Starting with Flagger 1.24.0, the Flagger release artifacts are published to GitHub Container Registry, and they are signed with Cosign and GitHub ODIC.

OCI artifacts:

ghcr.io/fluxcd/flagger:<version>: multi-arch container images
ghcr.io/fluxcd/flagger-manifest:<version>: Kubernetes manifests
ghcr.io/fluxcd/charts/flagger:<version>: Helm charts

To verify an OCI artifact with Cosign:

export COSIGN_EXPERIMENTAL=1
cosign verify ghcr.io/fluxcd/flagger:1.24.0
cosign verify ghcr.io/fluxcd/flagger-manifests:1.24.0
cosign verify ghcr.io/fluxcd/charts/flagger:1.24.0

To deploy Flagger from its OCI artifacts the GitOps way, please see the Flux installation guide.

The previous release, Flagger 1.23.0 added support for Slack bot token authentication.

Flux Legacy reaches End-Of-Life

As discussed in the last monthly update for the Flux project, we retired Flux v1 and Helm Operator on November 1st. The projects will no longer be supported and were archived on GitHub.

Please look into migrating to Flux v2 as soon as possible.

If you still need migration help, there are still free migration workshops, or reach out for paid support to one of the companies listed here.

Flux Ecosystem

Weave GitOps

The Weave GitOps team continues to iterate and just released v0.10.1 of Weave GitOps.

With the release of v0.10.0 they are excited to announce the beta launch of a new tool called GitOps Run. GitOps can be challenging for the everyday developer to work with and it can create some friction, especially for developers who are less familiar with Kubernetes or Flux. The purpose of GitOps Run is to remove the complexity for developers so that platform operators can create developer environments easily, and application developers can benefit from GitOps and focus on writing code. Basically, they set up a live reconciliation loop between your cluster and local working directory of choice. Any changes made to your local working directory will automatically be pulled onto the cluster so you can iterate quickly. When you are done you can turn off GitOps Run and your cluster will go back to the previous state. This tool is incredibly useful with the VSCode GitOps extension.

You can either toggle GitOps Run to allow changes directly on the cluster or choose a sandbox option as well. The team is definitely looking for feedback on this exciting new feature so please don’t hesitate to engage and submit feature requests. Check out an overview and quick getting started video here.

The team continues to make improvements to the GitOps Dashboard as well. You are now able to inspect the YAML of all objects within the application as well as being able to navigate to objects via the various graph views. We have also added support for alerts and providers.

Terraform Controller

The Weave GitOps team has been hard at work on the next version of the tf-controller and just released Weave TF-controller v0.13.0 this week.

First-class YAML Support (tech preview)

A notable feature in this version is the first-class YAML support for Terraform. A Terraform object in v0.13.0+ allows you to better configure your Terraform resources via YAMLs, without introducing any extra CRDs to your cluster. Together with a new generator, Tofu-Jet will now be able to ship pre-generated primitive Terraform modules for all major cloud providers. The team shipped the alpha version of AWS package in this release. Tofu-Jet generator will be open-sourced later by the end of this year.

A primitive Terraform module is a module that only contains a single primitive resource like, aws_iam_role or aws_iam_policy. With this concept, we would be able to use Terraform without writing Terraform codes and make it more GitOps-friendly at the same time.

New Features and Bug Fixing

Implement webhooks for Terraform stages
Add use case examples
Add .spec.workspace field
Add the default value to workspace
Implement spec.values and map it to Terraform HCL
Add docs for preflight checks
Implement Helm-like template for Terraform files
Add runner Dockerfile for Azure
Upgrade Golang to v1.19
Bundle an alpha version AWS Package
Fix e2e
Implement init containers support on the runner pod
Implement spec.dependsOn and watch for the output secret changes
Implement templating for input references
Fix the check of dependencies by taking the output secret into account
Add tests for the spec.dependsOn feature
Change templating delimiter to ${{ }}
Add labels to tfstate via the K8s backend so that we can group them by the labels
Fix dependency in the finalizer
Add an ability to Helm chart for creating service accounts in each namespace
Parameterize AWS package in chart
Add trace logging
Fix runner service account template not returning multiple docs
Implement replan to avoid double planning
Add SHA and version information to the binaries

Weave GitOps Enterprise

Weave GitOps Enterprise continues to improve with numerous features including all of those mentioned in the OSS version above. They have released v0.10.1. First, you can now view terraform resources from the UI, plus sync and suspend resources like Kustomizations, HelmReleases, and Sources.

The team has also launched their pipeline feature which will enable you to set up environments for helm charts and track the chart versions across dev, staging, and production (or however you decide to define your environment stages).

Policy sets have been added as well so you can now state whether policies should just be treated as non-blocking (audit) or blocking (admission). This means you can easily configure your various policies to request the team to fix their code, either to future-proof it, or for the fix to be included before changes can actually be applied to the clusters.

Finally, the team has been working hard to open up templates to all types of objects within the platform. In the past, templates were isolated to only CAPI providers so you could easily self service clusters. From Weave GitOps Enterprise you are now able to create templates for any yaml objects so you can self-serve anything from new microservices to cloud infrastructure all driven by GitOps and the power of Flux.

Flux Subsystem for Argo

Flux Subsystem for Argo (aka Flamingo) is the safe migration path for Argo CD to Flux and Weave GitOps. A Flamingo image is the drop-in replacement of the equivalent version of Argo CD. You can safely run workloads reconciled by Flux and Argo CD on the same clusters.

The team has upgraded Flamingo to support Flux v0.36 and Argo CD v2.5. Not only the v2.5 support, this train of releases also include Flamingo for v2.2 - v2.4 too.

Here’s the updated support matrix

Flux	Argo CD	Flamingo Image
v0.36	v2.5	v2.5.0-fl.3-main-2bba0ae6
v0.36	v2.4	v2.4.15-fl.3-main-2bba0ae6
v0.36	v2.3	v2.3.10-fl.3-main-2bba0ae6
v0.36	v2.2	v2.2.15-fl.3-main-2bba0ae6

VS Code GitOps Extension

A new “Configure GitOps” workflow is available in the pre-release of the extension. The workflow introduces a new unified user interface for creating Source and Workload and for attaching Workloads to Sources. It supports both Generic Flux and Azure Flux (Arc/AKS) cluster modes. In Azure mode, FluxConfig resources are created automatically (this can be disabled if the user wants Generic mode compatibility). Currently this feature is in the Extension Marketplace pre-release channel and supports GitRepository and Kustomization resources. Final release will be available early in November and will provide an user-friendly UI for working with every type of Source and Workflow.

New additions to the Flux Ecosystem

We are very pleased to announce the following new members of the Flux Ecosystem. We feel blessed to have a lively and active community like this!

First up is the Datadog Agent for Flux: it runs on your hosts and collects events and metrics from hosts and sends them to Datadog, where you can analyze your monitoring and performance data.

KubeVela is next on the list, which now integrates Flux as well for Helm Chart delivery and GitOps, and provides multi-cluster capabilities.

We could have some kind of Halloween reference here, but GitOps zombies is actually a tool for finding Kubernetes resources which are not managed via GitOps. Go check it out.

And last but not least, here is the Pulumi Kubernetes Operator, which runs Pulumi programs, and can fetch them via Flux sources.

Really nice to see Pulumi adopting @fluxcd for source management. Pulumi programs can now be packaged as OCI artifacts with the Flux CLI and signed with Cosign. Before the Pulumi operator runs them, Flux pulls the artifacts in-cluster and verifies their signatures 🔐🚀 https://t.co/Z9KGipWij4
— Stefan Prodan (@stefanprodan) October 24, 2022

Recent & Upcoming Events

It’s important to keep you up to date with new features and developments in Flux and provide simple ways to see our work in action and chat with our engineers.

Recent Events (ICYMI) 📺

Thanks to all the Flux community members who are happy to talk about their experiences. In October there was obviously KubeCon (which we will have a separate blog post about), here is one talk already which we would like to highlight.

GitOps with Flux and OCI Registries - Soulé Ba & Scott Rigby, Weaveworks

Please let us know if we missed anything of interest and we will make sure to mention it in the next post!

Upcoming Events 📆

We are happy to announce that we have a number of events coming up in November - tune in to learn more about Flux and GitOps best practices, get to know the team and join our community.

ATO 2022 Get Started with Kubernetes & GitOps Workshop (Nov 1)

For those that are new to Kubernetes, don’t fret! Justin will give a brief overview of Kubernetes core concepts, features, architecture, and key components to ensure you have a necessary understanding of the Kubernetes ecosystem so that you can follow along with the rest of this hands-on workshop.

HashiCorp User Group Luxembourg (virtual) (Nov 30)

Flux Terraform Controller is a controller for Flux to reconcile Terraform configurations in the GitOps way with the power of Flux and Terraform, Terraform Controller allows you to GitOps-ify your infrastructure, and your application resources, in the Kubernetes and Terraform universe.

Flux Bug Scrub

The next dates are going to be:

2022-11-02 12:00 UTC, 14:00 CEST
2022-11-10 18:00 UTC
2022-11-16 13:00 UTC

In other news

Your Community Team

The Flux Community Team started its own set of meetings as an experiment for the next 3 months. Here we want to discuss everything that’s important for the Flux community, such as organisation of events, advocacy, getting more people involved in the community and more.

This month we had our first two meetings. Check out the meeting notes which include the meeting recordings to see what was discussed in detail.

A few themes we are looking into as a group are:

Document and refine processes and tools to make it a lot easier to be involved
Highlight events and meetings to our community better
Make things like our social and editorial calendars public so people can feed into it more easily and it becomes more of a team effort

Please join us for the next meeting - instructions and agenda can be found here.

People writing/talking about Flux

News from the Website and our Docs

Flux Adopters shout-out

We are very pleased to announce that the following adopters of Flux have come forward and added themselves to our website: Cyera, Syneki and University of Bordeaux.

More docs and website news

We are constantly improving our documentation and website - here are a couple of things we landed recently:

Update themes and move to using them as Hugo modules. This simplified our setup quite a bit.
Generate resources section from YAML.
Deemphasise Legacy Flux in our docs.
Updates to the frontpage to make events easier to find.
Update Flagger docs to 1.24. New guide to set up Flagger on a Kubernetes cluster the GitOps way.
And lots of other updates and improvements.

Thanks a lot to these folks who contributed to docs and website: Stefan Prodan, Batuhan Apaydın, Mohamed F. Ahmed, Arhell, FG, Hidde Beydals, Michael Bridgen, Santosh Kaluskar, Jasmin Müller, Kingdon Barrett, Martin PAUCOT, Raffael Sahli, Shalom Yerushalmy, Steve Wilkerson and ebCrypto.

Flux Project Facts

We are very proud of what we have put together. We want to reiterate some Flux facts - they are sort of our mission statement with Flux.

🤝 Flux provides GitOps for both apps or infrastructure. Flux and Flagger deploy apps with canaries, feature flags, and A/B rollouts. Flux can also manage any Kubernetes resource. Infrastructure and workload dependency management is built-in.
🤖 Just push to Git and Flux does the rest. Flux enables application deployment (CD) and (with the help of Flagger) progressive delivery (PD) through automatic reconciliation. Flux can even push back to Git for you with automated container image updates to Git (image scanning and patching).
🔩 Flux works with your existing tools: Flux works with your Git providers (GitHub, GitLab, Bitbucket, can even use s3-compatible buckets as a source), all major container registries, and all CI workflow providers.
🔒 Flux is designed with security in mind: Pull vs. Push, least amount of privileges, adherence to Kubernetes security policies and tight integration with security tools and best-practices. Read more about our security considerations.
☸️ Flux works with any Kubernetes and all common Kubernetes tooling: Kustomize, Helm, RBAC, and policy-driven validation (OPA, Kyverno, admission controllers) so it simply falls into place.
🤹 Flux does Multi-Tenancy (and “Multi-everything”): Flux uses true Kubernetes RBAC via impersonation and supports multiple Git repositories. Multi-cluster infrastructure and apps work out of the box with Cluster API: Flux can use one Kubernetes cluster to manage apps in either the same or other clusters, spin up additional clusters themselves, and manage clusters including lifecycle and fleets.
📞 Flux alerts and notifies: Flux provides health assessments, alerting to external systems and external events handling. Just “git push”, and get notified on Slack and other chat systems.
👍 Users trust Flux: Flux is a CNCF Incubating project and was categorised as "Adopt" on the CNCF CI/CD Tech Radar (alongside Helm).
💖 Flux has a lovely community that is very easy to work with! We welcome contributors of any kind. The components of Flux are on Kubernetes core controller-runtime, so anyone can contribute and its functionality can be extended very easily.

Over and out

If you like what you read and would like to get involved, here are a few good ways to do that:

Join our upcoming dev meetings on 2022-11-03 or 2022-11-11.
Talk to us in the #flux channel on CNCF Slack
Join the planning discussions
And if you are completely new to Flux, take a look at our Get Started guide and give us feedback
Social media: Follow Flux on Twitter, join the discussion in the Flux LinkedIn group.

We are looking forward to working with you.

Blog: CNCF Talk: Increased security and scalability with OCI

Wed, 26 Oct 2022 13:20:00 +0000

Integrating OCI into Flux was one of the most-requested features of all times. We listened to your feedback and in the past couple of releases, OCI was integrated more deeply into Flux. Here is a brief summary of what landed when:

v0.31 (Jun 2022): Support for Helm repositories of type OCI
v0.32 (Aug 2022): Kubernetes manifests, Kustomize overlays and Terraform code as OCI artifacts
v0.33 (Aug 2022): More configurability of OCI settings
v0.34 (Sep 2022): More flexibility when interacting with OCI artifacts/repositories
v0.35 (Sep 2022): verify OCI artifacts signed by cosign
v0.36 (Oct 2022): verify OCI helm charts signed by cosign plus lots of new tooling to interact with OCI using the Flux CLI

To bring you up to speed with what’s possible, Max Jonas Werner, Flux Core Maintainer and Senior Software Engineer at Weaveworks, gave a talk in the CNCF Online Programme series to give some background and do a practical demo.

First off, Max explained the core GitOps concepts and gave an overview of the architecture of Flux. In the next step, he dived into how Docker and others created the Open Containers Initiative (OCI) which is a part of the Linux Foundation.

One of the key points Max is making is that we went through a transformation from Docker containers to generic application and configuration containers. More and more OCI is becoming an application delivery format.

OCI registries (which implement the distribution spec) are a commodity in the cloud space. This means that it’s very easy to get enhanced scalability this way, because pulling an OCI image is much less resource-intensive compared to a full or shallow Git clone. Additionally, high available registries are available everywhere.

It also provides many ways to secure your infrastructure. Flux leverages Kubernetes workload identity and IAM when pulling OCI artifacts from managed registries. So no more key management, no more SSH keys to generate, no more proprietary API usage for token generation. You use the same mechanism that is used for pulling container images. You might also want to check out this post about verifying authenticity of artifacts with cosign.

Max spends more than half of his presentation time for the demo, so you get a good idea of how to use these new features and integrate them into your setup.

Check out the video here:

Thanks a lot Max for taking the time to walk us through this!

Start your journey and start using Flux’s OCI features today.

Blog: Flux proudly representing at KubeCon 2022 NA

Wed, 19 Oct 2022 15:30:00 +0000

Everybody in Team Flux is busy with the run-up to Flux GA, but – with the help of the great folks in our Community and Ecosystem – we also managed to put together quite a number of talks, tutorials and sessions for you at KubeCon / CloudNativeCon!

We are happy to announce that we will be at GitOpsCon and KubeCon.

We are only a few days away from the event and we look forward to meeting you there, or seeing you in the online sessions! Meet us at our booth in-person at the Project Pavilion during KubeCon, see demos, ask questions and get involved in our community!

To make this all easier, we put together this mini-site for you: Flux @ KubeCon mini site. Go ahead and bookmark it now - there you have all the Flux things in one place.

Monday, October 24 (Flux Project Meeting at KubeCon)

13:00 - 17:00 Flux Project Meeting Room 335, Level 300

We will have talks/demos from beginner to advanced, including but not limited to: Flux basics, what’s new with Flux including OCI support, VS Code, Terraform Controller, Cosign, Helm, & Flagger, and of course you can ask Maintainers all your questions.

Tuesday, October 25 (GitOpsCon)

9:45 - 10:15 GitOpsCon: How to Achieve (Actual) GitOps with Terraform and Flux

Priyanka "Pinky" Ravi (Weaveworks)

9:45 - 10:15 GitOpsCon: Toward Full Adoption of GitOps and Best Practices at RingCentral

Tamao Nakahara (Weaveworks) and Ivan Anisimov (RingCentral)

11:10 - 11:40 GitOpsCon: Simplifying Edge Deployments Using EMCO and GitOps

Igor DC & Adarsh Vincent Chittilappilly (Intel)

11:40 - 12:10 Prometheus Days: Automate your SLO validation with Prometheus & Flagger

Sanskar Jaiswal & Kingdon Barrett (Weaveworks)

12:00 - 12:10 GitOpsCon: Why Do We Do This? The Heart of GitOps

Leigh Capili (VMware)

13:10 - 13:20 GitOpsCon: Green(ing) CI/CD: A Sustainability Journey with GitOps

Niki Manoledaki (Weaveworks)

13:40 - 14:10 GitOpsCon: Complete DR of Workloads, PVs and CSI Snapshots via Flux and Vault OSS

Kingdon Barrett (Weaveworks)

14:15 - 14:45 GitOpsCon: GitOps with Flux and OCI Registries

Soulé Ba & Scott Rigby (Weaveworks)

14:15 - 14:45 GitOpsCon: Pixie + Flux, VSCode, GitOps Observability from Top to Bottom

Somtochi Onyekwere (Weaveworks)

Wednesday, October 26 (KubeCon)

14:30 - 16:00 KubeCon: Tutorial: So You Want To Develop a Cluster API Provider

Anusha Hegde & Winnie Kwon & Sedef Savas (VMware), Richard Case (Weaveworks),

Avishay Traeger (Red Hat)

15:25 - 16:00 KubeCon: Flagger, Linkerd, And Gateway API: Oh My!

Jason Morgan (Linkerd) & Sanskar Jaiswal (Weaveworks)

15:25 - 16:00 KubeCon: Tutorial: How To Write a Reconciler Using K8s Controller-Runtime!

Scott Rigby, Somtochi Onyekwere, Niki Manoledaki & Soulé Ba (Weaveworks),

Amine Hilaly (Amazon Web Services)

Thursday, October 27 (KubeCon)

11:00 - 11:35 KubeCon: Learn About Helm And Its Ecosystem

Andrew Block & Karena Angell (Red Hat), Matt Farina (SUSE) Scott Rigby (Weaveworks)

14:30 - 15:05 KubeCon: Flux Maturity, Feature, and Contrib Update

Somtochi Onyekwere & Kingdon Barrett (Weaveworks)

Friday, October 28 (KubeCon)

11:00 - 12:30 KubeCon: Flux ContribFest

Room 410 B

Conclusion

Again: Here is the Flux @ KubeCon mini site. Go ahead and bookmark it now - there you have all the Flux things in one place.

Thanks to everyone who helped to organise and prepare the event and talks! Good luck and safe travels to everyone - have a lot of fun! See you soon!

Blog: Prove the Authenticity of OCI Artifacts

Mon, 17 Oct 2022 12:30:00 +0000

Software supply chain attacks are one of the most critical risks threatening today’s software and have begun to collapse like a dark cloud over the software industry. For the Flux family of projects we are taking precautions against these threats. Apart from implementing security features and best practices, it is important to us to educate our users. You can find all Flux’s security articles here. Today we will talk about a new security feature.

Let’s start with a brief historical explanation of how we got to this point. It all started with the following sentence:

Flux should be able to distribute and reconcile Kubernetes configuration packaged as OCI artifacts.

RFC-0003: Flux OCI support for Kubernetes manifests.

From then on, the Flux community worked hard and brought this feature with Flux v0.32. So with that, you can store and distribute various sources such as Kubernetes manifests, Kustomize overlays, and Terraform modules as OCI (Open Container Initiative) artifacts with Flux CLI and tell Flux to reconcile your sources that are stored in OCI Artifacts, and Flux will do that for you. 🕺🏻

But this only covered the first stage of the entire implementation. There is more than that. ☝️

One of the most exciting features of this RFC is the verification of artifacts. But why, what is it, is it really necessary or just a hype thing? This is a long topic that we need to discuss. Suppose you store the cluster desired state as OCI artifacts in a container registry. How can you be one hundred percent sure that the resources that Flux reconciles are the same as the resources that you’ve pushed to the OCI registry? This is where the verification of artifacts comes into play. But, how can we do that? 🤔

Thanks to the Sigstore community we have a great set of services and tools for signing and verifying authenticity. One of the tools is cosign which can be used for container signing, verification, and storage in an OCI registry. We will use it to verify the authenticity of the OCI Artifacts in Flux. Starting with v0.35, Flux comes with support for verifying OCI artifacts signed with Sigstore Cosign. Documentation for setting it up can be found here.

Let’s jump right into the details of how we can actually use it.

We will deploy cert-manager by storing its manifests in OCI registry packaged as an OCI Artifacts, using the Flux CLI. Then we are going to sign it with cosign and configure Flux to verify the artifacts’ signatures before they are downloaded and reconciled.

You need three things to complete this demo;

cosign CLI
- https://docs.sigstore.dev/cosign/installation/
A Kubernetes cluster
- https://kind.sigs.k8s.io/#installation-and-usage
Flux CLI
- https://fluxcd.io/flux/cmd/

Let’s start by creating a simple Kubernetes cluster:

kind create cluster

Let’s install Flux on it - if you need to use other options, check out the installation page.

export GITHUB_USER=developer-guy
export GITHUB_TOKEN=$GITHUB_TOKEN

flux bootstrap github \
 --owner=$GITHUB_USER \
 --repository=flux-cosign-demo \
 --branch=main \
 --path=./clusters/my-cluster \
 --personal

⚠️ Note: Don’t forget to change the values with your own details!

First we download the cert-manager install manifests from GitHub:

curl -sSLO https://github.com/cert-manager/cert-manager/releases/download/v1.9.1/cert-manager.yaml

https://github.com/cert-manager/cert-manager/releases/tag/v1.9.1

Next we push the manifests to GitHub container registry with Flux CLI:

mkdir -p ./manifests
 cp cert-manager.yaml ./manifests
$ flux push artifact oci://ghcr.io/$GITHUB_USER/manifests/cert-manager:v1.9.1 \
 --path="./manifests" \
 --source="https://github.com/cert-manager/cert-manager.git" \
 --revision="v1.9.1/4486c01f726f17d2790a8a563ae6bc6e98465505"
► pushing artifact to ghcr.io/developer-guy/manifests/cert-manager:v1.9.1
✔ artifact successfully pushed to ghcr.io/developer-guy/manifests/cert-manager@sha256:d1fb0442865148a4e9b4c3431c71d8e44af56c3eb658ea495c5ec48d48c6638b

Before signing the OCI artifact with Cosign, we need to create a set of key pairs, a public and private one:

cosign generate-key-pair

This command above outputs two files to disk: cosign.pub and cosign.key. The cosign.pub file is the public key and the cosign.key file is the private key. You can use the cosign.pub file to verify the container image and the cosign.key file to sign the container image.

To let Flux to verify the signature of the OCI artifact, we should create a secret that contains the public key::

kubectl -n flux-system create secret generic cosign-pub \
 --from-file=cosign.pub=cosign.pub

Now, let’s sign it:

$ cosign sign --key cosign.key ghcr.io/$GITHUB_USER/manifests/cert-manager:v1.9.1
Enter password for private key:
Pushing signature to: ghcr.io/developer-guy/manifests/cert-manager

As we stick into the GitOps practices, we should create a file that contains the OCIRepository resource, then commit and push those changes into the upstream repository that Flux watches for changes:

git clone git@github.com:developer-guy/flux-cosign-demo.git
cd flux-cosign-demo

Let’s create a secret with the GitHub token:

$ flux create secret oci ghcr-auth \
 --url=ghcr.io \
 --username=${GITHUB_USER} \
 --password=${GITHUB_TOKEN}
► oci secret 'ghcr-auth' created in 'flux-system' namespace

Configure Flux to pull the cert-manager artifact, verify its signature and apply its contents:

cat << EOF | tee ./clusters/my-cluster/cert-manager-sync.yaml
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: OCIRepository
metadata:
 name: cert-manager
 namespace: flux-system
spec:
 interval: 5m
 url: oci://ghcr.io/${GITHUB_USER}/manifests/cert-manager
 ref:
 semver: "*"
 secretRef:
 name: ghcr-auth
 verify:
 provider: cosign
 secretRef:
 name: cosign-pub
---
apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
 name: cert-manager
 namespace: flux-system
spec:
 interval: 1h
 timeout: 5m
 sourceRef:
 kind: OCIRepository
 name: cert-manager
 path: "."
 prune: true
 wait: true
EOF

Let’s commit and push these changes:

git add .
git commit -m"Add cert-manager OCIRepository and Kustomization"
git push

After couple of seconds for Flux will have applied these changes. Now let’s check the status of them:

Or, you can trigger the reconcilation immediately by running the simple command: flux reconcile source git flux-system

For Kustomization resources:

$ flux get kustomizations
NAME REVISION SUSPENDED READY MESSAGE
cert-manager v1.9.1/d1fb0442865148a4e9b4c3431c71d8e44af56c3eb658ea495c5ec48d48c6638b False True Applied revision: v1.9.1/d1fb0442865148a4e9b4c3431c71d8e44af56c3eb658ea495c5ec48d48c6638b
flux-system main/14f1e66 False True Applied revision: main/14f1e66

For OCIRepository resources:

$ flux get sources oci
NAME REVISION SUSPENDED READY MESSAGE
cert-manager v1.9.1/d1fb0442865148a4e9b4c3431c71d8e44af56c3eb658ea495c5ec48d48c6638b False True stored artifact for digest 'v1.9.1/d1fb0442865148a4e9b4c3431c71d8e44af56c3eb658ea495c5ec48d48c6638b'

If you see the status of the OCIRepository is True, it means that Flux has successfully verified the signature of the container image. Because Flux adds a condition with the following attributes to the OCIRepository’s .status.conditions:

type: SourceVerified
status: “True”
reason: Succeeded

If the verification fails, Flux will set the SourceVerified status to False and will not fetch the artifact contents from the registry. If you see the status of the Kustomization is True, it means that Flux has successfully applied the manifests that are stored in the container image.

Let’s check the status of the cert-manager deployment:

$ kubectl get pods --namespace cert-manager
NAME READY STATUS RESTARTS AGE
cert-manager-cainjector-857ff8f7cb-l469h 1/1 Running 0 76s
cert-manager-d58554549-9fbgj 1/1 Running 0 76s
cert-manager-webhook-76fdf7c485-9v82g 1/1 Running 0 76s

Furthermore

As we can store Helm Charts in OCI registries with the release of Helm v3.8.0 which means that we can also sign them with cosign and verify them with Flux. The Flux community is already working on it and want to add support for verifying the Helm charts stored in OCI registries as OCI Artifacts in the next releases of Flux. You can follow the progress of this feature in the following issue: fluxcd/source-controller#914.

The Sigstore community is aware of the risks and the toil of managing public/private key pairs, so cosign offers another mode for signing and verification called Keyless, which do not require managing any keys manually. Flux also supports that. If you omit the .verify.secretRef field, Flux will try to verify the signature using the Keyless mode. It’s worth mentioning keyless verification is an experimental feature, using custom root CAs or self-hosted Rekor instances are currently not supported.

Blog: September 2022 Update

Tue, 04 Oct 2022 11:30:00 +0000

It’s the beginning of October 2022 - let’s recap together what happened in September - it has been a lot!

News in the Flux family

Flux v0.34.0 and 0.35.0 bring OCI improvements

Flux v0.35 and Flux v0.34 landed in September. They bring tons of improvements, especially in the area of OCI. We encourage everyone to upgrade for the best experience.

Please note: there are breaking changes: The Flux controller logs have been aligned with the Kubernetes structured logging. For more details on the new logging structure please see: fluxcd/flux2#3051.

Here is a quick summary of what you can look forward to in terms of features and improvements:

Verify OCI artifacts signed by Cosign (including keyless - currently still experimental and only supporting GCP and GHCR) with OCIRepository.spec.verify. Note this supports contextual login, but not insecure registries.
Allow pulling Helm charts dependencies from HTTPS repositories with mixed self-signed TLS and public CAs.
Allow pulling Helm charts from OCI artifacts stored at the root of AWS ECR.
Allow running bootstrap for insecure HTTP Git servers with flux bootstrap git --allow-insecure-http --token-auth.
Improve health checking for global objects such as ClusterClass, GatewayClass, StorageClass, etc.
The controllers and the Flux CLI are now built with Go 1.19.
Allow pulling artifacts from an in-cluster Docker Registry over plain HTTP with OCIRepository.spec.insecure.
Allow defining OCI sources for non-TLS container registries with flux create source oci --insecure.
Enable contextual login when publishing OCI artifacts from a Cloud VM using flux push artifact --provider=aws|azure|gcp.
Prioritise static credentials over OIDC providers when pulling OCI artifacts from container registries on multi-tenant cluster.
Reconcile Kubernetes Class types (ClusterClass, GatewayClass, StorageClass, etc) in a dedicated stage before any other custom resources like Clusters, Gateways, Volumes, etc.
When multiple SOPS providers are available, run the offline decryption methods first to avoid failures due to KMS unavailability.
Add finalizers to the notification API to properly record the reconciliation metrics for deleted resources.
Publish the Flux install manifests as OCI artifacts on GitHub and DockerHub container registries under fluxcd/flux-manifests.

For more information on OCI and Cosign support please see the Flux documentation.

It took us six months to debate, design and implement OCI support in Flux. Big thanks to all the Flux contributors that helped us reach this milestone!

Flux Legacy (v1) Retirement Plan

Thanks to so many of you who have been migrating to the latest Flux version, often in conversation with us. We appreciate your enthusiasm for the increased capabilities of Flux. In October 2020 we put Flux Legacy and Helm operator into maintenance mode (cf flux#3320 and helm-operator#546). Back then we promised to continue to support them for 6 months once we reached feature-parity across all former feature sets, and instead we have offered extended support for over a year.

We reached parity in March 2021 and announced stable APIs in July 2021. Since then we added OCI support and many other modern features to Flux v2. Thanks to you not only for migrating, but also for adding yourselves to the latest Flux adopters page! We really appreciate it. Your work has brought down the number of support requests for legacy Flux to 5% of all volume in the past year.

We will archive Flux Legacy in November this year. If you still need migration help, there are still free migration workshops, or reach out for paid support to one of the companies listed here.

Some recent prompts for this include:

Some of the Flux v1 dependencies are pinned to EOL versions, which cannot be upgraded without causing regressions or a cascading amount of changes to the codebase.
All Kubernetes dependencies are pinned within version v1.21. That version already reached end-of-life support upstream.

Thanks for joining us on this journey of building Flux. Please give Flux a star!

Flux Ecosystem

Flux Subsystem for Argo

The team is happy to announce that Flux Subsystem for Argo (FSA) is now on-par with ArgoCD regarding supported versions. FSA now provides all versions supported by ArgoCD. The project will provide security updates based on ArgoCD v2.2 and v2.3, and for the active ArgoCD version (currently v2.4), FSA will support them, starting from v2.4.12.

For Flux compatibility, FSA will be tested against every release of future Flux versions.

Weave GitOps

Weaveworks just released version v0.9.6 for Weave GitOps. There are a lot of great new features that have been released in the last month. First, it is continuing the trend of being a feature rich Flux UI by adding support for Flux Providers and Alerts. When you click on the user icon you are then taken to a screen that contains those objects. As a platform operator, you can easily understand where events are being sent.

On the kustomization and helm release detail pages there is now a tab to check your dependencies for those objects. The dependsOn is a powerful feature in Flux and now you can easily see these visualised within the application. We’re also making it easy to navigate from these graphs to relevant objects in a near future release.

In addition, the team added numerous improvements to gitops run our live coding environment. Now you can run the command against an empty folder and it will generate a kustomization.yaml file and give you a live connection between that working directory and the cluster you are connected to. The team is full steam ahead on the next set of features for the run experience.

Terraform Controller

The Weave GitOps team is continuing to improve our ecosystem of controllers with the latest release of the tf-controller v0.12.0.

The notable features in this release are: custom backend support, interop with Notification controller, and support human readable plan output in ConfigMap. This is all new:

Enable custom backends for Terraform
Support backendConfigsFrom for specifying backend configuration from Secrets
Add a parameter for specifying max gRPC message size, default to 4MB
Implement force-unlock for tfstate management
Fix the initialization status
Recording events to support Flux notification controller
Support specifying targets for plan and apply
Add node selector, affinity and tolerations for the runner pod
Add volume and volumeMounts for the runner pod
Add file mapping to map files from Secrets to home or workspace directory
Fix Plan prompt being overridden by the progressing message
Support storing human-readable plan output in a ConfigMap

Learn more at the following blog post “How to GitOps Your Terraform”, by Priyanka Ravi & Daniel Holbach.

Weave GitOps Enterprise

The Weave GitOps Enterprise continues to build on top of the OSS feature set with its latest v0.9.5 release. First, the team has added a new add application button with support for both Kustomizations and Helm Releases. This makes it super easy to add the relevant Flux primitives to get your applications loaded onto the cluster(s) of your choice.

Workspaces were added as well. This makes it super easy to manage multi-tenancy on Weave GitOps Enterprise. It is built on top of Flux’s tenancy model with a lot of extra flexibility and power. For example, all of your workspaces can be defined in one or more files. We then have a simple CLI command that will generate all of the necessary YAML for you. This includes advanced features such as policies to ensure full compliance within the tenant. You can define which repositories your users can use as well as which clusters applications can be deployed to. To learn more about this feature check out the documentation.

You can also now define pipelines and environments for Helm Charts. This will allow your application teams to see how things are rolled out across dev, staging, and production environments; or however you choose to define your environments. There will be a lot of continued efforts in this area so stay tuned.

Your engineering teams are able to see policy violations for applications across clusters. Policy sets can be used by platform operators to define in one place whether policies are for auditing purposes or should be blocked by the admission controller. The team built out a profile for making it easy to set up policy dashboards using the ELK stack. Platform operators now have greater flexibility when configuring the same policy with different values for different clusters.

VS Code GitOps Extension

A lot of great features have been added to the extension, most notably support for OCI and Azure. Please see the recent blog post in our ecosystem category for more details.

Recent & Upcoming Events

It’s important to keep you up to date with new features and developments in Flux and provide simple ways to see our work in action and chat with our engineers.

Recent Events (ICYMI) 📺

We feel blessed to have such a big community of users, contributors and integrators and so many are happy to talk about their experiences. In September here are a couple of talks we would like to highlight.

CNCF On-Demand Webinar (Sep 15): Flux increased security & scalability with OCI
Flux is trusted for its high levels of security, and new OCI support brings even greater GitOps security and scalability. Max Jonas Werner covers the benefits like more streamlined repo structure options and better ways to manage breaking changes in your app.
CNCF On-Demand Webinar (Sep 29) How to GitOps Your Terraform
Priyanka “Pinky” Ravi walks you through step-by-step how to manage Terraform resources the GitOps way, from provisioning to enforcement. Bring GitOps to infrastructure and application resources for hybrid automation, state enforcement, drift detection and more.

Here is a list of additional videos and topics we really enjoyed - please let us know if we missed anything of interest and we will make sure to mention it in the next post!

Upcoming Events 📆

KubeCon

We are happy to announce that we will be at GitOpsCon and KubeCon in October! Visit our booth in-person at the Project Pavilion during KubeCon and the full schedule is below (and on our Flux @ KubeCon mini site. See you soon!

Monday, October 24 (Flux Project Meeting at KubeCon)

13:00 - 17:00 Flux Project Meeting Room 335, Level 300

We’ll have talks/demos from beginner to advanced, including but not limited to: Flux basics, what’s new with Flux including OCI support, VS Code, Terraform Controller, Cosign, Helm, & Flagger, and of course you can ask Maintainers all your questions.

Tuesday, October 25 (GitOpsCon)

9:45 - 10:15 GitOpsCon: How to Achieve (Actual) GitOps with Terraform and Flux

Priyanka "Pinky" Ravi (Weaveworks) and Roberth Stand (Crayon Group)

9:45 - 10:15 GitOpsCon: Toward Full Adoption of GitOps and Best Practices at RingCentral

Tamao Nakahara (Weaveworks) and Ivan Anisimov (RingCentral)

11:10 - 11:40 GitOpsCon: Simplifying Edge Deployments Using EMCO and GitOps

Igor DC & Adarsh Vincent Chittilappilly (Intel)

11:40 - 12:10 Prometheus Days: Automate your SLO validation with Prometheus & Flagger

Sanskar Jaiswal & Kingdon Barrett (Weaveworks)

12:00 - 12:10 GitOpsCon: Why Do We Do This? The Heart of GitOps

Leigh Capili (VMware)

13:10 - 13:20 GitOpsCon: Green(ing) CI/CD: A Sustainability Journey with GitOps

Niki Manoledaki (Weaveworks)

13:40 - 14:10 GitOpsCon: Complete DR of Workloads, PVs and CSI Snapshots via Flux and Vault OSS

Kingdon Barrett (Weaveworks)

14:15 - 14:45 GitOpsCon: GitOps with Flux and OCI Registries

Soulé Ba & Scott Rigby (Weaveworks)

14:15 - 14:45 GitOpsCon: Pixie + Flux, VSCode, GitOps Observability from Top to Bottom

Somtochi Onyekwere (Weaveworks)

Wednesday, October 26 (KubeCon)

14:30 - 16:00 KubeCon: Tutorial: So You Want To Develop a Cluster API Provider

Anusha Hegde & Winnie Kwon & Sedef Savas (VMware), Richard Case (Weaveworks),

Avishay Traeger (Red Hat)

15:25 - 16:00 KubeCon: Flagger, Linkerd, And Gateway API: Oh My!

Jason Morgan (Linkerd) & Sanskar Jaiswal (Weaveworks)

15:25 - 16:00 KubeCon: Tutorial: How To Write a Reconciler Using K8s Controller-Runtime!

Scott Rigby, Somtochi Onyekwere, Niki Manoledaki & Soulé Ba (Weaveworks),

Amine Hilaly (Amazon Web Services)

Thursday, October 27 (KubeCon)

11:00 - 11:35 KubeCon: Learn About Helm And Its Ecosystem

Andrew Block & Karena Angell (Red Hat), Matt Farina (SUSE) Scott Rigby (Weaveworks)

Friday, October 28 (KubeCon)

11:00 - 12:30 KubeCon: Flux ContribFest

Room 410 B

16:55 - 17:30 KubeCon: Flux Maturity, Feature, and Contrib Update

Somtochi Onyekwere & Kingdon Barrett (Weaveworks)

Flux Bug Scrub

The next dates are going to be:

In other news

New Flux Project Members: Batuhan Apaydın and Rashed Kamal

We are very excited to be able to announce two new Flux project members this month.

Batuhan Apaydın, Senior Software Engineer at Trendyol, has been helping out quite a bit in the OCI discussions and wrote a blog post explaining how to manage Kyverno policies as OCI artifacts recently. We are very glad to have him in our community and there’s more OCI awesomeness and blog posts planned.

Rashed Kamal, Staff Engineer at VMware, joined us in September as well. His interests include OCI, where he contributed to the RFC too. On top of that he fixed a number of issues in Flux. Thanks for all of that and for being part of the team!

News from the Website and our Docs

Flux Adopters shout-out

We are very pleased to announce that the following adopters of Flux have come forward and added themselves to our website: NovaID.

More docs and website news

We are constantly improving our documentation and website - here are a couple of small things we landed recently:

We simplified the build process of the website. We are on a very recent version of the Docsy theme again!
Our Bootstrap Cheatsheet now contains instructions on how to enable notifications for third party controllers.
Flux End-To-End documentation was updated to reflect recent changes.
We added a lot of new videos to the Flux Resources page.
Many small improvements and fixes across the entire site and docs.

Thanks a lot to these folks who contributed to docs and website: Stefan Prodan, Kingdon Barrett, Arhell, Paulo Gomes, Max Jonas Werner, Vanessa Abankwah, Santosh Kaluskar, Batuhan Apaydın, Stacey Potter, Bang Nguyen, Sven Nebel, Aurel Canciu, David Harris, Gustaf Lindstedt, Simo Aleksandrov and annaken.

Flux Project Facts

We are very proud of what we have put together. We want to reiterate some Flux facts - they are sort of our mission statement with Flux.

🤝 Flux provides GitOps for both apps or infrastructure. Flux and Flagger deploy apps with canaries, feature flags, and A/B rollouts. Flux can also manage any Kubernetes resource. Infrastructure and workload dependency management is built-in.
🤖 Just push to Git and Flux does the rest. Flux enables application deployment (CD) and (with the help of Flagger) progressive delivery (PD) through automatic reconciliation. Flux can even push back to Git for you with automated container image updates to Git (image scanning and patching).
🔩 Flux works with your existing tools: Flux works with your Git providers (GitHub, GitLab, Bitbucket, can even use s3-compatible buckets as a source), all major container registries, and all CI workflow providers.
🔒 Flux is designed with security in mind: Pull vs. Push, least amount of privileges, adherence to Kubernetes security policies and tight integration with security tools and best-practices. Read more about our security considerations.
☸️ Flux works with any Kubernetes and all common Kubernetes tooling: Kustomize, Helm, RBAC, and policy-driven validation (OPA, Kyverno, admission controllers) so it simply falls into place.
🤹 Flux does Multi-Tenancy (and “Multi-everything”): Flux uses true Kubernetes RBAC via impersonation and supports multiple Git repositories. Multi-cluster infrastructure and apps work out of the box with Cluster API: Flux can use one Kubernetes cluster to manage apps in either the same or other clusters, spin up additional clusters themselves, and manage clusters including lifecycle and fleets.
📞 Flux alerts and notifies: Flux provides health assessments, alerting to external systems and external events handling. Just “git push”, and get notified on Slack and other chat systems.
👍 Users trust Flux: Flux is a CNCF Incubating project and was categorised as "Adopt" on the CNCF CI/CD Tech Radar (alongside Helm).
💖 Flux has a lovely community that is very easy to work with! We welcome contributors of any kind. The components of Flux are on Kubernetes core controller-runtime, so anyone can contribute and its functionality can be extended very easily.

Over and out

If you like what you read and would like to get involved, here are a few good ways to do that:

Join our upcoming dev meetings on 2022-10-06 or 2022-10-12.
Talk to us in the #flux channel on CNCF Slack
Join the planning discussions
And if you are completely new to Flux, take a look at our Get Started guide and give us feedback
Social media: Follow Flux on Twitter, join the discussion in the Flux LinkedIn group.

We are looking forward to working with you.

Blog: GitOps Without Leaving your IDE

Wed, 28 Sep 2022 11:00:00 +0000

Welcome to the second blog post in our Flux Ecosystem category! This time we are talking about one of the Flux UIs: it’s the VS Code GitOps Extension.

If you already use VS Code, this extension will be straight up your alley: it provides an intuitive way to manage, troubleshoot and operate your Kubernetes environment following the GitOps operating model, accelerating your development lifecycle and simplifying your continuous delivery pipelines. Of course it uses Flux under the hood.

Getting Started

Installing it: It’s in the Visual Studio Code Marketplace, so if you search for it in VS Code, it’s just a click of the Install button away.

Additionally, you will need to

Optionally, if available, the extension will make use of the az tool (for Azure clusters) and docker as well.

With that out of the way, let’s get going and take the extension for a spin.

Drive everything from your IDE

Once you launch VS Code, you should see available clusters listed in the Clusters section of the GitOps extension. Now you can easily interact with the resources in each of the clusters. This makes it very straightforward to make changes in your manifests, commit and observe changes in the clusters without leaving your IDE.

The extension was designed so that you always have access to the immediate tasks and events. Turning a manifest into a Kustomization or Source? Right-click on the YAML file. View repositories, clusters, sources, kustomizations, etc. - you see them at the first glance. View GitOps Output panel with CLI command traces for diagnostics, cluster and components versions, Flux controller logs, everything you might need to debug. Enable/disable GitOps cluster operations with just a click. Reconcile Sources and Workloads demand, and much more - links to most-needed docs included.

Constantly evolving

The extension is rapidly growing new features. In 0.21.0, the team added OCI support which is supported natively in Flux. If you would like to see a video demo of this, check out this talk done by Annie Talvasto and Kingdon Barrett in the CNCF Webinar series.

In 0.22.0 basic support for Azure AKS/Arc was added. Future releases will add a new beginner-friendly UI workflow for creating complete Flux configurations using both generic Flux Source (Git, OCI, Bucket, HelmRepository) and Workload (Kustomization, HelmRelease) resources as well as Azure FluxConfig resources.

The team has begun work to transition the extension implementation from shell out commands to Javascript APIs for Kubernetes and Azure. Once that is complete, extension responsiveness and performance will improve dramatically.

Join the community

The team behind the extension loves feedback. If you like what you see, please star the extension on GitHub or leave an issue if something is missing, or send a PR if you can.

All feedback is very welcome!

Blog: How to GitOps Your Terraform

Wed, 14 Sep 2022 11:00:00 +0000

This is the first blog post in a series where we want to shine a light on projects in the Flux Ecosystem. This time it’s going to be the Terraform Controller.

If you use Terraform, you might think of it as “Infrastructure as Code” and to be separate from the concept of GitOps. Quite often we have seen debates about “Infrastructure as Code vs. GitOps”. The Terraform Controller reconciles these two worlds and lets you take advantage of the benefits of GitOps for existing Terraform resources: one source of truth, one single pane of glass and drift detection among them.

You might have resorted to using pipelines or manual deployments up until now. In this blog post we are going to show how to have your Terraform resources managed the GitOps way. Without having to convert your code at all!

What is the Terraform Controller?

The Terraform Controller is a Flux controller that can manage your Terraform resources. Although Flux runs on Kubernetes, whatever you are using Terraform for, the Flux controller can manage it. It has several features including the ability to do manual approvals or auto-approve Terraform plans, and the Terraform outputs can be set as a Kubernetes secret. It is also integrated with Terraform Cloud and Terraform Enterprise.

The benefits of using the Terraform Controller is that you are able to take advantage of GitOps for existing Terraform resources. There is drift detection of Terraform resources and it can be used as a glue for Terraform resources and Kubernetes workloads.

Terraform Controller is very versatile because it offers different modes of operation and many features which give you the integration points and control you need. Primarily it supports these use-cases:

GitOps Automation Model: Here you GitOps your Terraform resources from the provision steps to the enforcement steps, like for example a whole EKS cluster.
Hybrid GitOps Automation Model: Here you GitOps parts of your existing infrastructure resources. For example, you have an existing EKS cluster. You can choose to GitOps only its nodegroup, or its security group.

Building on this, you can make use of these features if you have a TFSTATE file:

State Enforcement: Use GitOps to enforce it, without changing anything else.
Drift Detection: Use GitOps just for drift detection, so you can decide to do things later when a drift occurs.

And there’s more: Multi-Tenancy, Plan and Manual Approve and more features on the roadmap.

Now let’s move on to how to integrate it practically!

GitOpsing your Terraform

Prerequisites

Obviously you will need a Kubernetes cluster and Flux installed. Terraform Controller will require at least Flux 0.32, which in turn needs at least Kubernetes version 1.20.6. Either use flux install or flux bootstrap as explained in the Flux documentation.

Installation

Now you need to install Terraform Controller. There are many ways to do it, check out the installation docs for more information.

One very easy way to do it is to add this HelmRelease to your bootstrap repository.

Tying in your Terraform resources

And here is where all the beauty of Terraform Controller comes in - it does all the hard work for you. All you will need to do to is

Define the source of your Terraform resources
Enable GitOps Automation

Define source

So let’s go ahead, here we define a Source controller’s source (you can pick any of GitRepository, Bucket, OCIRepository). A GitRepository entry could look like this:

apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: GitRepository
metadata:
 name: helloworld
 namespace: flux-system
spec:
 interval: 30s
 url: https://github.com/tf-controller/helloworld
 ref:
 branch: main

The GitOps Automation mode

The GitOps automation mode could be enabled by setting .spec.approvePlan=auto. In this mode, Terraform resources will be planned, and automatically applied for you. Here is a simple example you can just copy and paste.

apiVersion: infra.contrib.fluxcd.io/v1alpha1
kind: Terraform
metadata:
 name: helloworld
 namespace: flux-system
spec:
 interval: 1m
 approvePlan: "auto"
 path: ./
 sourceRef:
 kind: GitRepository
 name: helloworld
 namespace: flux-system

Note: If you have a kustomization.yaml file (which is the case in the basic flux bootstrap use-case), make sure you add the file(s) the above manifest portions are in into the resources list.

Once you commit this to Git, you should see Terraform Controller pick this up quickly. One way to confirm is:

kubectl -n flux-system get terraforms.infra.contrib.fluxcd.io
NAME READY STATUS AGE
helloworld True No drift:
main/d9c5cc348e555526ea563fb82fc901e37de4d732 1m

Simple, wasn’t it?

What else is there?

The Terraform Controller team has been hard at work and made sure that many of the common use-cases are supported. Above we covered the automation mode, some teams might want more control, so there’s a “plan and manual apply” mode as well. You can configure it as well to just do “drift detection only”.

And there’s more, you can disable drift detection, use it with AWS EKS IRSA, interact with Terraform (set variables, manage terraform state), there’s health checks and lots of other flexibility. OCI fans will love to hear that it supports OCI Artifacts as Source as well.

It is also integrated with Terraform Cloud and Terraform Enterprise.

In past weeks the performance of the Terraform Controller has been improved significantly as well. Now the controller is greatly scalable to reconcile and provision a high volume of Terraform modules concurrently. The team has recently tested the controller with 1,500 Terraform modules.

In the most recent release (v0.12.0) new features are: custom backend support, interoperability with Flux’s Notification Controller, and supporting human-readable plan output in ConfigMap.

And there’s more to come, check out the team’s roadmap. While you are checking it out, please give feedback as well. If you are missing something, if you like it, if you want to contribute - the team is eager to hear from you.

Want to learn more?

Priyanka “Pinky” Ravi, Developer Experience Engineer at Weaveworks gave a great introduction to the Terraform Controller a couple of weeks ago. Take a look to dive into some of the finer details of this!

You are lucky, there is more to come! Pinky will give an on-demand webinar as part of the CNCF series. You can sign up for it here. It will become available on Sep 29th, 2022.

Title: How to GitOps your Terraform!

Presenter: Priyanka “Pinky” Ravi

Link: Registration here.

Blog: August 2022 Update

Mon, 05 Sep 2022 11:30:00 +0000

It’s the beginning of September 2022 - let’s recap together what happened in August - it has been a lot!

News in the Flux family

New Flux releases add OCI support and better integration with cloud services

August saw two big releases of Flux: v0.33 and v0.32. Let’s go through the major changes one by one.

Enable contextual login to container registries when pulling Helm charts from Amazon Elastic Container Registry, Azure Container Registry and Google Artifact Registry using HelmRepository.spec.provider.
Select which layer contains the Kubernetes configs by specifying a matching OCI media type using OCIRepository.spec.layerSelector.
Authenticate to Azure Blob storage with SAS tokens using Bucket.spec.secretRef.
Allow filtering OCI artifacts by semver and regex when listing artifact with flux list artifacts.
Allow excluding local files and directories when building and publishing artifacts with flux push artifact.
New Flux CLI commands flux push|pull|tag artifact for publishing OCI Artifacts to container registries.
New source type OCIRepository for fetching OCI artifacts from container registries.
Resolve Helm dependencies from OCI for charts defined in Git.

The big news was of course that we added support for distributing Kubernetes manifests, Kustomize overlays and Terraform code as OCI artifacts. For more information on OCI support please see the Flux documentation.

Big thanks to the Flux contributors that helped us along the way. It took us almost 4 months, from the first RFC version to shipping OCI support today. And a special thanks to Rashed and the whole VMware Tanzu team for the excellent collaboration!

No more pesky secrets to authenticate against Azure, AWS and GCP container registries when deploying Helm charts with @fluxcd. Starting with v0.33, Flux leverages #Kubernetes workload identity and IAM when pulling OCI artifacts from managed registries. https://t.co/V9dbT6orrP pic.twitter.com/N7EB4D0Is8
— Stefan Prodan (@stefanprodan) August 31, 2022

I'm super excited to announce that @fluxcd support for distributing #Kubernetes manifests, Kustomize overlays and Terraform code as OCI artifacts has finally shipped in v0.32. https://t.co/144HY6LUTy
— Stefan Prodan (@stefanprodan) August 11, 2022

Security news

We are continuously putting effort into the security story of Flux. One cornerstone of this is fuzzing of all code. As promised, we started transitioning our fuzz tests to the native Go implementation.

We are happy to say that we managed to contribute back to Google’s oss-fuzz improving Go Native Fuzz implementation as well during this effort ( patch 1, patch 2).

Flagger 1.22.2

Flagger 1.22.2 received a patch release as well during August. It fixes a bug related to scaling up the canary deployment when a reference to an auto-scaler is specified.

Furthermore, it contains updates to packages used by the project, including updates to Helm and grpc-health-probe used in the load-tester.

A number of CVEs originating from its dependencies were fixed as well.

Flux Ecosystem

Flux Subsystem for Argo

Flux added OCIRepository as a new kind of Source in its recent release. The new version of Flux Subsystem for Argo (FSA) brings these good bits of Flux to Argo CD. The team has also recently upgraded FSA to Argo CD v2.2.12 to contain recent security bug fixes again. This version of Flux Subsystem for Argo requires Flux v0.32.0 to install.

Terraform-controller

The team has released TF-controller v0.11 which now supports Flux OCIRepository. To use Flux OCIRepository with TF-controller, you’re required to upgrade Flux to v0.32+.

In addition to the new OCIRepository support, the TF-controller team is glad to announce that the performance of TF-controller has been improved significantly. Now the controller is greatly scalable to reconcile and provision high volumes of Terraform modules concurrently. The team has recently tested the controller with 1,500 Terraform modules.

Weave GitOps

The team at Weaveworks is continuing to invest in Applications first! They’ve focused this quarter on building and improving the primitives that make up Weave GitOps. Their aim is to make it easy for platform operators to simplify adoption of Kubernetes and Cloud Native in general across their engineering organization. An easy to use platform that is extensible and safe for organizations to meet their needs.

The OSS team released v0.9.4. There are a lot of iterative improvements in the app such as the ability to pause and resume multiple sources or automation objects from the UI. In addition, there are a bunch of tiny UI and visual improvements. Getting started is now simpler due to a new gitops create dashboard command for producing the HelmRelease and HelmRepository objects. Plus, some foundational improvements for gitops run.

On the enterprise side they are wrapping up workspaces including the GUI, that gives you a single pane of glass what applications and policies belong to which tenant! That makes governance for Platform teams easy and enables Application teams to operate efficiently in safe boundaries. In addition, they have a new add application experience that makes it easy to use Kustomizations and Helm Charts via their UI. Now you have a single simple flow to add your workloads/applications independently if it’s k8s manifest in a Git Repository or Helm Charts. Look for an upcoming release (v0.9.4) in the next week for these two items.

VS Code GitOps Extension

Anyone who loves the GitOps Extension for VS Code should update to the latest version. Among other things it just received a number of security fixes. Find the relevant details on its advisories page.

Recent & Upcoming Events

It’s important to keep you up to date with new features and developments in Flux and provide simple ways to see our work in action and chat with our engineers.

Recent Events (ICYMI) 📺

We feel blessed to have such a big community of users, contributors and integrators and so many are happy to talk about their experiences. In August here are a couple of talks we would like to highlight.

CNCF Livestream with Kingdon Barrett: VSCode and Flux: Testing the new OCI Repository feature

The Flux project continues in active development with the addition of OCI configuration planned in the GA roadmap. Another Flux advancement has been the creation of the new VSCode Extension which provides a convenient interface to Flux that can help reduce friction moving between editor and terminal, alleviating the headache of context switching overloading developer focus. Flux maintainer Kingdon Barrett demonstrates Flux’s new OCI features and a convenient way to access them.

Upcoming Events 📆

We are happy to announce that we have a number of events coming up in September - tune in to learn more about Flux and GitOps best practices, get to know the team and join our community.

Sep 15 CNCF on-demand webinar: Flux Increased Security & Scalability with OCI

Flux is trusted for its high levels of security, and new OCI support brings even greater GitOps security and scalability. Max will cover the benefits like more streamlined repo structure options and better ways to manage breaking changes in your app.
Sep 29 CNCF on-demand webinar: How to GitOps Your Terraform

Pinky will walk you through step-by-step how to manage Terraform resources the GitOps way, from provisioning to enforcement. Bring GitOps to infrastructure and application resources for hybrid automation, state enforcement, drift detection and more.

Flux Bug Scrub

The next dates are going to be:

In other news

News from the Website and our Docs

Flux Adopters shout-out

We are very pleased to announce that the following adopters of Flux have come forward and added themselves to our website: Embark Studios and NexHealth.

More docs and website news

We are constantly improving our documentation and website - here are a couple of small things we landed recently:

New security docs on Secrets Management and Contextual Authorization.
New blog post: Managing Kyverno Policies as OCI Artifacts with OCIRepository Sources
Cheatsheet news
- OCI Artifacts
- Bootstrap: Git repository access via SOCKS5 ssh proxy
- Bootstrap: Enable notifications for third party controllers
Flux’s Work Well With section: find out how to make Flux work with your favourite other OSS software
Lots of new videos from GitOpsCon / KubeCon on our resources page
Various updates to the Flux Roadmap to indicate what needs to be done for the Flux GA release
Move to a fluxcd.io/<project> kind of structure. Add a project picker in the main navbar. Updates of Flux Legacy docs to 1.4.4, Flagger docs to 1.22.2.
Updates of Docsy theme and dependencies. Prevent click-jacking of the site.

Thanks a lot to these folks who contributed to docs and website: Stefan Prodan, Paulo Gomes, Arhell, Kingdon Barrett, Max Jonas Werner, Santosh Kaluskar, David Harris, Sunny, Aurel Canciu, Benny and annaken.

New Flux Project Member: Leigh Capili

We are proud to announce a new project member in the Flux project. Leigh Capili, Staff Developer Advocate at VMware, has been contributing to Flux for a long time already. If you check out his application, he has left a trail of fixes and improvements across almost all of our projects.

What we would like to specifically call out as well, is the countless talks he has done about Flux and GitOps. Take a look at the Flux Resources page to learn more. Three of our current favourites are:

Be like Leigh: If you have contributed to Flux and are interested in joining the Flux project as a member, please take a look at our governance documentation for this.

Flux Project Facts

We are very proud of what we have put together. We want to reiterate some Flux facts - they are sort of our mission statement with Flux.

🤝 Flux provides GitOps for both apps or infrastructure. Flux and Flagger deploy apps with canaries, feature flags, and A/B rollouts. Flux can also manage any Kubernetes resource. Infrastructure and workload dependency management is built-in.
🤖 Just push to Git and Flux does the rest. Flux enables application deployment (CD) and (with the help of Flagger) progressive delivery (PD) through automatic reconciliation. Flux can even push back to Git for you with automated container image updates to Git (image scanning and patching).
🔩 Flux works with your existing tools: Flux works with your Git providers (GitHub, GitLab, Bitbucket, can even use s3-compatible buckets as a source), all major container registries, and all CI workflow providers.
🔒 Flux is designed with security in mind: Pull vs. Push, least amount of privileges, adherence to Kubernetes security policies and tight integration with security tools and best-practices. Read more about our security considerations.
☸️ Flux works with any Kubernetes and all common Kubernetes tooling: Kustomize, Helm, RBAC, and policy-driven validation (OPA, Kyverno, admission controllers) so it simply falls into place.
🤹 Flux does Multi-Tenancy (and “Multi-everything”): Flux uses true Kubernetes RBAC via impersonation and supports multiple Git repositories. Multi-cluster infrastructure and apps work out of the box with Cluster API: Flux can use one Kubernetes cluster to manage apps in either the same or other clusters, spin up additional clusters themselves, and manage clusters including lifecycle and fleets.
📞 Flux alerts and notifies: Flux provides health assessments, alerting to external systems and external events handling. Just “git push”, and get notified on Slack and other chat systems.
👍 Users trust Flux: Flux is a CNCF Incubating project and was categorised as "Adopt" on the CNCF CI/CD Tech Radar (alongside Helm).
💖 Flux has a lovely community that is very easy to work with! We welcome contributors of any kind. The components of Flux are on Kubernetes core controller-runtime, so anyone can contribute and its functionality can be extended very easily.

Over and out

If you like what you read and would like to get involved, here are a few good ways to do that:

Join our upcoming dev meetings on 2022-09-08 or 2022-09-14.
Talk to us in the #flux channel on CNCF Slack
Join the planning discussions
And if you are completely new to Flux, take a look at our Get Started guide and give us feedback
Social media: Follow Flux on Twitter, join the discussion in the Flux LinkedIn group.

We are looking forward to working with you.

Blog: Managing Kyverno Policies as OCI Artifacts with OCIRepository Sources

Thu, 01 Sep 2022 11:30:00 +0000

The Flux team has released a new version of Flux v0.32 that includes fantastic features. One of them is OCI Repositories feature that allows us to store and distribute a wide variety of sources such as Kubernetes manifests, Kustomize overlays, and Terraform modules as OCI (Open Container Initiative) artifacts. Furthermore, the Flux team got us even more excited because they are planning to verify the authenticity of the OCI artifacts before they get applied into Kubernetes by integrating Cosign, which is one of the most significant projects from the @projectsigstore community that help us to sign and verify OCI images, blobs, etc. please see the issue to get more details about the plan.

⚠️ Note: You can read the RFC of this feature here.

I'm super excited to announce that @fluxcd support for distributing #Kubernetes manifests, Kustomize overlays and Terraform code as OCI artifacts has finally shipped in v0.32. https://t.co/144HY6LUTy
— Stefan Prodan (@stefanprodan) August 11, 2022

Today’s blog post is all about a quick tour of this feature and will give you a real-world example of it to show you how you can leverage this feature to manage Kyverno policies as OCI Artifacts. It is worth saying that this topic has been discussed for a while in the Kyverno community, too. There is an ongoing issue about packaging and distributing Kyverno policies as OCI Artifacts through its CLI. Also, there is a chance to move that logic into Kyverno’s core.

But for those who might not be familiar enough with OCI artifacts (including me), it’s worth explaining what the OCI Artifacts are before jumping into the details. OCI Artifacts gives you the power of storing and distributing other types of data (nearly anything), such as Kubernetes deployment files, Helm Charts, and CNAB, in addition to container images via OCI registries. And today, we’ll be using this feature for Kyverno policies. To be more precise, OCI Artifacts are not a new specification, format, or API. It just utilizes the existent OCI manifest and OCI index definitions. Hence, we can quickly start using the same client tooling, such as a crane, skopeo, etc., and distribute them using OCI registries, thanks to the OCI distribution-spec. Because OCI Artifacts does not change anything related to the specs, it only expands them to give people (artifact authors) power to define their content types. It is more like a generic definition for determining what can be stored in an OCI registry and consumed by clients.

The Flux CLI generates a single layer OCI image for storing things. As you can use some other tools to generate an OCI image with multiple layers in it, you can use the Layer Selection feature that Flux provides to select the layers you want to use in the OCI image. If the layer selector matches more than one layer, the first layer matching the specified media type will be used. Note that Flux requires that the OCI layer is compressed in the tar+gzip format.

Today, we’ll leverage the OCI Repositories feature to apply Kyverno policies stored in an OCI registry into the Kubernetes cluster.

First, we need to install Flux CLI, please see the installation page for more details.

Next, we should have a Kubernetes cluster running. We’ll be using KinD for this purpose.

kind create cluster

Once the cluster has been provisioned successfully, we need to install Flux components into it by simply running the command below:

$ flux bootstrap github \
 --owner=developer-guy \
 --repository=flux-kyverno-policies \
 --path=clusters/local \
 --personal

⚠️ Note: Don’t forget to change the values with your own details!

This command will install Flux and create necessary files for us and push them into the repository.

Next, we should install Kyverno by using a GitOps approach with Flux. In order to do that, we use the following resources:

---
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: GitRepository
metadata:
 name: kyverno-controller
 namespace: flux-system
spec:
 interval: 30m
 url: https://github.com/kyverno/kyverno
 ignore: |
 /*
 !/config/
 ref:
 semver: "1.x"
---
apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
 name: kyverno-controller
 namespace: flux-system
spec:
 interval: 30m
 sourceRef:
 kind: GitRepository
 name: kyverno-controller
 serviceAccountName: kustomize-controller
 path: ./config/release
 prune: true
 wait: true
 timeout: 5m

Do not forget to check whether everything works fine before moving into the next steps:

$ flux get kustomizations kyverno-controller
NAME REVISION SUSPENDED READY MESSAGE
kyverno-controller v1.7.3/f2b63ce False True Applied revision: v1.7.3/f2b63ce

Now, we are ready to create an OCI image to store my Kyverno policies.

⚠️ You can find all the code examples in GitHub.

In order to do that, we will clone our repository that holds the Kyverno policies and create an OCI artifact to store them.

⚠️ We are expecting that some other team like DevSecOps will be responsible for maintaining and publishing the policies to our registry.

$ git clone https://github.com/developer-guy/my-kyverno-policies.git
$ cd my-kyverno-policies
$ flux push artifact oci://ghcr.io/developer-guy/policies:v1.0.0 \
 --path="." \
 --source="$(git config --get remote.origin.url)" \
 --revision="$(git branch --show-current)/$(git rev-parse HEAD)"
► pushing artifact to ghcr.io/developer-guy/policies:v1.0.0
✔ artifact successfully pushed to ghcr.io/developer-guy/policies@sha256:56e853e3c5c02139c840b7f5c89a02f63ede8dc498ed3925a52360032aa49e60

⚠️ Note: Don’t forget to change the values with your own details!

Last but not least, we need to create an OCIRepository resource that points to my OCI artifact:

---
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: OCIRepository
metadata:
 name: kyverno-policies
 namespace: flux-system
spec:
 interval: 5m
 url: oci://ghcr.io/developer-guy/policies
 ref:
 semver: "v1.x"
 secretRef:
 name: ghcr-auth
---
apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
kind: Kustomization
metadata:
 name: kyverno-policies
 namespace: flux-system
spec:
 sourceRef:
 kind: OCIRepository
 name: kyverno-policies
 interval: 60m
 retryInterval: 5m
 path: ./
 prune: true
 wait: true
 timeout: 2m
 dependsOn:
 - name: kyverno-controller
 patches: # enforce all policies
 - patch: |
 - op: replace
 path: /spec/validationFailureAction
 value: enforce
 target:
 kind: ClusterPolicy

I’d like to highlight some key points about the resources above. Here in OCIRepository resource, we are using SemVer to select the policies that we want to apply. .spec.ref is an optional field to specify the OCI reference to resolve and watch for changes. If not specified, the latest version of the repository will be used. You can reach out to the complete list of references supported in Flux, here is the link for you.

Also, in the Kustomization resource, we are using .spec.patches to apply patches to the policies that we want to enforce. We are using op: replace to replace the existing value of the field with the new one. path is the path to the field that we want to replace. value is the value of the field that we want to replace. To get more detail about the Patches, please see the link.

Last but not least, we are specifying an explicit dependencies for the Kustomization resource by using dependsOn keyword that ensures the Kyverno deployment is ready before applying the policies. This is important because Kyverno needs to be installed before applying the policies. Otherwise, the policies won’t be used because CRD (Custom Resource Definitions) won’t exist until Kyverno works. You can learn more about the dependencies of Kustomization resource, here.

Now, we can apply these manifests by committing and pushing them to the repository and letting Flux take care of the rest but still, one little step left that we need to do, which is authentication.

⚠️ Don’t forget, the authentication part is only needed when the OCI artifact is not publicly accessible. If your image has publicy available, you can skip that part.

You might notice a secretRef section in the OCIRepository resource. We should create this secret because Flux should be able to pull my container image. To do that, we should follow the documentation.

$ flux create secret oci ghcr-auth \
 --url=ghcr.io \
 --username=developer-guy \
 --password=${GITHUB_PAT}
► oci secret 'ghcr-auth' created in 'flux-system' namespace

Once everything is completed, you should be able to see the following output:

$ kubectl get clusterpolicies
NAME BACKGROUND ACTION READY
require-base-image true enforce true

This is what we expected to happen, whee!🕺🏻

This is an exciting policy, though, if you want to learn more about it, I wrote a blog post that explains what the base image concept refers to and how we can enforce policies related to them.

As you can see, this feature is quite promising and easy to use. I hope you enjoyed it, and please stay tuned because there are more features on the way you don’t want to miss.

Thanks for reading.

Blog: July 2022 Update

Tue, 02 Aug 2022 11:30:00 +0000

It’s the beginning of August 2022 - let’s recap together what happened in July - it has been a lot!

News in the Flux family

Next Flux release: OCI Helm improvements and consolidated Git implementations

The whole Flux team is busy working on the v0.32.x Flux release that’s planned for early August. A lot of our planned changes have already landed and what you can look forward to is: OCI for Kubernetes manifests and further enhancements to the OCI for Helm support that shipped already are also included. Support for Cosign will not be included in this release just yet, but will come later.

It’s not too late to provide early feedback for the OCI support, we still need more user engagement/feedback to guarantee this feature is ready for release.

We have planned on this release finally decommissioning our libgit2 Unmanaged Transport and replacing it with the new Managed Transport (it will no longer be experimental, now default!)

The upgrade to managed transport should be opaque and seamless for the end user. Hopefully Flux users will notice things are more stable, but no changes are needed in order to take advantage of this upgrade, other than simply upgrading.

Security news

When we started writing about Security in Flux, folks started asking us more questions about how to ensure their Flux deployments were secure. We are happy to announce that we documented Flux’s Security Best Practices on our website. It comes with a simple checklist that you can follow to ensure you implemented it. You can also go deeper and expand the text blocks to understand the rationale and backgrounds better.

Please let us know if you have any questions or feedback - we are happy to add to this section.

Flagger 1.22 brings KEDA Support

This Flagger release is a big one. It comes with support for KEDA ScaledObjects as an alternative to HPAs. KEDA is a CNCF Incubation project and is supported in e.g. Azure. Check out our tutorial to understand how to use it with Flagger.

Other improvements in the release are:

The .spec.service.appProtocol field can now be used to specify the appProtocol of the services that Flagger generates.
A bug related to the Contour prometheus query for when service name is overwritten along with a bug related to Contour HTTPProxy annotations have been fixed.
The installation guide for Alibaba ServiceMesh has been updated.

Read the full list of improvements and fixes in the 1.22.0 and 1.22.1 changelog entries.

Flux Ecosystem

Flux Subsystem for Argo

The team released Flux Subsystem for Argo by rebasing it to Argo CD v2.2.11, which contains many serious security fixes. They verified that this version of FSA worked with recent versions of Flux, including Flux v2 0.31.4.

Terraform-controller

The authors had been identifying performance bottlenecks in the TF controller. Now with the bottlenecks identified, they have been able to start rewriting the certification rotation component to improve the performance of the controller. The performance improvement is expected to land by the mid of August.

Their most recent release 0.10.0 contains the following improvements:

Add support for Terraform Enterprise
Implement resource inventory
Improve security to make the images work with Weave GitOps Enterprise
Re-implement certificate rotator
Correct IRSA docs
Update Kubernetes libraries to v0.24.3, go-restful to fix CVE-2022-1996
Add pprof to the /debug/pprof endpoint
Fix race condition to make sure that gRPC client and the runner use the same TLS

VS Code GitOps Extension

In our last monthly updates we talked about the GitOps Extension for VS Code that is based on top of Flux. If you always wanted to see it in action to be able to understand what it can do for you, check out our recent blog post which contains the VSCode Extension Demo from GitOps Days.

Weave GitOps

The team is working towards a new release of Weave GitOps OSS. They’ve made some quality of life improvements in our latest release v0.9.1. I'm so glad you asked. This is a CLI command in Weave GitOps OSS that will make it simpler to get started with Flux and GitOps. In addition, it enables live feedback while configuring your cluster. They are aiming for simplicity for those that are new to Kubernetes and GitOps. They are looking for beta testers so if you know anyone that might be interested then please have them sign up here.

On the Enterprise side they are getting close to enhance and extend the flux tenant model, providing the user with capabilities to create tenants from a declarative yaml that can be versioned. Enabling platform teams to create isolated tenants with boundaries. Define allowed sources, targets. RBAC and policy with a single tool.

Azure GitOps

Azure GitOps now supports Flux v2 in Azure Kubernetes Service (AKS) and Azure Arc-enabled Kubernetes (Arc K8s) clusters ( blog post). Azure lets customers use the same managed Flux service for their cluster configuration and application deployment across all their clusters – Azure, on-premises, multi-cloud. The Azure team works closely with Weaveworks to improve upstream Flux (e.g., multi-tenancy) and continues the partnership.

New additions to the Flux Ecosystem

We redesigned our Ecosystem page! Up until recently we simply listed tools, extensions and integrations that either simplified using Flux in various contexts or extended its functionality.

What was missing was the great work a lot of companies have done to bring GitOps to their users in the form of products and services. We now show a list of these and logos for those who approved the use of logos. If you are in the market for a complete GitOps solution, go check it out!

Another big topic in our user community is the one of UIs. We now added a section with screenshots to give you a good idea of what your options are and how they can simplify your workflow.

We realise that some ecosystem entries might be missing - if you find one, please send a PR, we want this page to grow!

Recent & Upcoming Events

It’s important to keep you up to date with new features and developments in Flux and provide simple ways to see our work in action and chat with our engineers.

Recent Events (ICYMI) 📺

We feel blessed to have such a big community of users, contributors and integrators and so many are happy to talk about their experiences. In July here are a couple of talks we would like to highlight.

Check out the recent CNCF livestream with Kingdon Barrett and Priyanka Ravi, Enhance your GitOps Experience with Flux Tools & Extensions.

In addition to that we recently started discussing a number of great talks from last month’s GitOps Days in blog posts. Check out these posts - they contain a summary of the talks and show the videos as well:

Weaveworks Blog: GitOps Days 2022 recap: major clouds & vendors offering GitOps with Flux
CNCF Blog: How to apply GitOps to everything with Crossplane and Flux
CNCF Blog: Keep calm and trust A/B testing with Flux, Flagger, and Linkerd
CNCF Blog: GitOps with Flux at Safaricom

Please let us know if we missed anything of interest and we will make sure to mention it in the next post!

Upcoming Events 📆

We are happy to announce that we have a number of events coming up in August - tune in to learn more about Flux and GitOps best practices, get to know the team and join our community.

CNCF Livestream (Aug 17) with Kingdon Barrett

The Flux project continues in active development with the addition of OCI configuration planned in the GA roadmap. Another Flux advancement has been the creation of the new VSCode Extension which provides a convenient interface to Flux that can help reduce friction moving between editor and terminal, alleviating the headache of context switching overloading developer focus.

Flux maintainer Kingdon Barrett will demonstrate the pre-release of Flux's new OCI features and a convenient way to access them while they remain in pre-release so you can provide the feedback that is needed by Flux maintainers to make this feature a success!

Flux Bug Scrub

The next dates are going to be:

In other news

News from the Website and our Docs

Flux Adopters shout-out

We are very pleased to announce that the following adopters of Flux have come forward and added themselves to our website: Mintmesh and SenseLabs.

More docs and website news

We added a Troubleshooting cheatsheet! This has been a request from our community for a long time and we would love to hear your feedback! What do you and your team use for incidents? Is it playbooks? What would you expect in Flux docs for managing incidents and troubleshooting?

We are constantly improving our documentation and website - here are a couple of small things we landed recently:

New use-case: GitHub Actions Basic App Builder:
This guide shows how to configure GitHub Actions to build an image for each new commit pushed on a branch, for PRs, or for tags in the most basic way that Flux’s automation can work with and making some considerations for both dev and production.
A single GitHub Actions workflow is presented with a few variations but one simple theme: Flux’s only firm requirement for integrating with CI is for the CI to build and push an image. So this document shows how to do just that.
We expanded our documentation on Azure to include Using Helm OCI with Azure Container Registry.
Flagger news! We updated the docs on our website to match the newest version of Flagger (1.22). This adds a tutorial for how to do Canary analysis with KEDA SealedObjects. In addition to that the install guides were updated, in particular the instructions for setting up Flagger on Alibaba ServiceMesh was simplified quite a bit.
We updated the resources section on the fluxcd.io landing page to show updated content with more breadth across the Flux space.
We updated to a more recent version of the Docsy theme, which allowed us to drop some of our own customisations. With this we also updated to the new version of the Algolia API - this should give you better search results as well.
And lots of other small improvements.

Thanks a lot to these folks who contributed to docs and website: Paulo Gomes, Kingdon Barrett, Ihor Sychevskyi, Max Jonas Werner, Santosh Kaluskar, Stefan Prodan, Hidde Beydals, Jonathan Innis, Soulé Ba, Stacey Potter, @chengleqi and @kirankldevops.

Archival of Flux Web UI

The fluxcd/webui project was archived. It was in active development from November 2020 to June 2021, but unfortunately it could not be kept alive. This is why we felt the need to point users to the following alternatives for UIs for Flux instead.

Weaveworks offers a free and open source GUI for Flux under the weave-gitops project.

You can install the Weave GitOps UI using a Flux HelmRelease, please see the Weave GitOps documentation for more details.
The Flux community maintains a series of Grafana dashboards for monitoring Flux.

See the monitoring section of the Flux documentation for how to install Flux's Grafana dashboards.

New Flux Project Member: Ihor Sychevskyi

We are very pleased to welcome Ihor Sychevskyi as a project member into the Flux family. Over the past months Ihor has been busy improving our website in many many places. A lot of small UI glitches all over the place fell into this category and if you view fluxcd.io on mobile the site is getting better all the time!

Be like Ihor: If you have contributed to Flux and are interested in joining the Flux project as a member, please take a look at our governance documentation for this.

Flux Project Facts

We are very proud of what we have put together. We want to reiterate some Flux facts - they are sort of our mission statement with Flux.

🤝 Flux provides GitOps for both apps or infrastructure. Flux and Flagger deploy apps with canaries, feature flags, and A/B rollouts. Flux can also manage any Kubernetes resource. Infrastructure and workload dependency management is built-in.
🤖 Just push to Git and Flux does the rest. Flux enables application deployment (CD) and (with the help of Flagger) progressive delivery (PD) through automatic reconciliation. Flux can even push back to Git for you with automated container image updates to Git (image scanning and patching).
🔩 Flux works with your existing tools: Flux works with your Git providers (GitHub, GitLab, Bitbucket, can even use s3-compatible buckets as a source), all major container registries, and all CI workflow providers.
🔒 Flux is designed with security in mind: Pull vs. Push, least amount of privileges, adherence to Kubernetes security policies and tight integration with security tools and best-practices. Read more about our security considerations.
☸️ Flux works with any Kubernetes and all common Kubernetes tooling: Kustomize, Helm, RBAC, and policy-driven validation (OPA, Kyverno, admission controllers) so it simply falls into place.
🤹 Flux does Multi-Tenancy (and “Multi-everything”): Flux uses true Kubernetes RBAC via impersonation and supports multiple Git repositories. Multi-cluster infrastructure and apps work out of the box with Cluster API: Flux can use one Kubernetes cluster to manage apps in either the same or other clusters, spin up additional clusters themselves, and manage clusters including lifecycle and fleets.
📞 Flux alerts and notifies: Flux provides health assessments, alerting to external systems and external events handling. Just “git push”, and get notified on Slack and other chat systems.
👍 Users trust Flux: Flux is a CNCF Incubating project and was categorised as "Adopt" on the CNCF CI/CD Tech Radar (alongside Helm).
💖 Flux has a lovely community that is very easy to work with! We welcome contributors of any kind. The components of Flux are on Kubernetes core controller-runtime, so anyone can contribute and its functionality can be extended very easily.

Over and out

If you like what you read and would like to get involved, here are a few good ways to do that:

Join our upcoming dev meetings on 2022-08-03 or 2022-08-11.
Talk to us in the #flux channel on CNCF Slack
Join the planning discussions
And if you are completely new to Flux, take a look at our Get Started guide and give us feedback
Social media: Follow Flux on Twitter, join the discussion in the Flux LinkedIn group.

We are looking forward to working with you.

Blog: GitOps Days - VSCode Extension Demo

Mon, 11 Jul 2022 14:30:00 +0000

Helping to close out GitOps Days 2022, Kingdon Barrett, OSS Engineer at Weaveworks, Flux Maintainer, and maintainer of the Weaveworks GitOps Tools extension for VSCode presented the new Flux extension. Kingdon showed the new extension and how it helps minimize context switching, keeping you in your editor where you can be most productive!

The presentation starts down in the trenches after Kingdon launched the new VSCode Extension Marketplace page and soft-launched the extension’s availability to install from the store the night before, and fully launching the marketplace entry during the conference only a few hours earlier!

The VSCode extension is still considered a prerelease even though it’s been available for some time and is in the store now. Since the extension was first launched outside of the marketplace its development was hampered with low usage and low discoverability. He wanted to avoid launching with glaring usability issues and ensure that integration with Flux was a tight fit. Ultimately several key usability issues have been addressed since the alpha extension was demonstrated in an earlier state, and he decided that GitOpsDays was going to be a great time to formally launch the extension in the store!

The addition of a Flux status widget in the VSCode editor makes monitoring the changes as Flux automatically deploys without leaving the comfort of your editor window a total snap.

When Flux detects an issue in your manifest and the deployment fails, the editor extension shines most brightly as you can see the error and the condition status in a mouse-over hover panel above the resources that are having issues.

This quick demonstration gives an overview of the GitOps extension for Flux, and also what it’s like to use the extension to help recover when things have gone wrong. What kind of live demo is it if nothing went wrong? (Hint…an unrealistic one, as something always goes wrong!) These tools make recovery fast and ensure you can do it without a heavy context switch out of the editor and into monitoring dashboards or terminal CLI debugging land.

If you want to try it out, just search “GitOps” or “FluxCD” in the extension marketplace! The new VSCode extension is available right now, in the marketplace, no compiler necessary.

If you’re already familiar with the VSCode Kubernetes extension then you’ll be happy to know the configuration of both are identical, as the Flux VSCode extension just uses your KUBECONFIG there is nothing else to configure, so you can start using the extension to help manage your workloads and avoid unnecessary context switching in your day to day!

Here’s the video in its entirety if you’d like to watch from start to finish:

Next Steps

The GitOps Days team will be publishing more blog posts along with videos from the event to the GitOps Days 2022 Playlist, so stay tuned for more as they become available. And don’t forget to subscribe to the YouTube channel!

Blog: June 2022 Update

Mon, 04 Jul 2022 13:30:00 +0000

It’s the beginning of July 2022 - let’s recap together what happened in June - it has been a lot!

News in the Flux family

A lot of work culminated in the 0.31 release series, where we landed at Flux v0.31.3. You can look forward to the the following set of important fixes and documentation improvements:

✨ Flux releases v0.31

We’ve released Flux v0.31. This release comes with new features and improvements.

🚀 New features

Pull Helm charts from container registries by configuring Helm repositories with type: oci.
For more information please see the Helm OCI documentation
Trigger GitHub Actions workflows from Flux by configuring alerting providers with type: githubdispatch.
For more information please see the GitHub dispatch provider documentation.

📔 New guides

🤖 New improvements and fixes

Starting with this version, all Flux controllers conform to the Kubernetes API Priority and Fairness.
Add support for configuring the authentication to AWS KMS, Azure Key Vault and Google Cloud KMS on multi-tenant clusters.
The Git reconciliation has been made more efficient by adding support for no-op clones that should reduce the outbound traffic substantially.
The libgit2 managed transport feature has been enabled by default to improve the Azure DevOps and AWS CodeCommit Git operations.
Fix an issue where the token used for Helm operations would go stale if it was provided using a Bound Service Account Token Volume.
Update the controllers and CLI dependencies to Kubernetes v1.24, Kustomize v4.5.5 and Helm v3.9.0.
Fix caching issue in registry client (source-controller)
Fix repository url error for Helm OCI (source-controller)
Fix semver sorting for Helm OCI charts (source-controller)
Fix service account impersonation when using target namespace (helm-controller)
Validate that the image name does not contain tags (image-reflector-controller)
Fix libgit2 SSH host key verification (source-controller & image-automation-controller)
Fix authentication when using Gitlab via HTTP/S (source-controller & image-automation-controller)

Thanks to everyone who contributed to this release. 🤗

Flux Ecosystem

Since the rewrite of Flux as a set of targeted controllers, we believe it has become a lot easier to extend Flux to whatever you need it to do. If you check out the Flux Ecosystem page you can see a lot of very useful extensions, products and tools you might find useful.

In this section of our monthly update, let’s go through what happened in the ecosystem.

Terraform-controller

The team around terraform-controlle started the new development cycle towards v0.10.0. They introduced a new feature that supports resource inventory inside a Terraform object so that other controllers like an external drift detector, or a cost estimator would be able to leverage it.

For the coming v0.10.0 release, they will focus on performance improvements so that the controller will handle large numbers of objects better.

Weave GitOps

The GitOps Dashboard is continuing to evolve and we have added a bunch of new features with the release of v0.9.0. You can now pause and resume automations and sources within the UI. The team also added a new yaml tab to each object page so you can see the full detail of the object on the cluster.

They have also added support for displaying custom metadata. It is super easy to use and enables you to put relevant information such as a description of the object or hyperlinks to metrics dashboards.

They have also improved the detail and graph views in the application. They were only able to show a subset of kubernetes objects that were created by Kustomizations and Helm Releases. You can now get a full view of all of the objects that were created.

The team is turning their attention to a new feature at the moment and are looking for people willing to participate as early beta users. They are building a new feature in Weave GitOps that will change the way you can interact with Kubernetes as you build out your system. The idea is to reduce friction as much as possible and get live feedback. Once you are done then you will be able to easily transition the workload management over to GitOps via Flux.

They are excited about this feature and would appreciate people that are willing to test early versions so they build the best possible solution that solves problems. If you are interested please sign up here and they will reach out to you via email when they are ready to start the beta test.

New additions to the Flux Ecosystem

We are very pleased to recognise KubeVela as part of the Flux Ecosystem: it integrates Flux well for Helm Chart delivery and GitOps, and provides multi-cluster capabilities.

Recent & Upcoming Events

It’s important to keep you up to date with new features and developments in Flux and provide simple ways to see our work in action and chat with our engineers.

Upcoming Events

Flux maintainer Somtochi Onyekwere at KCD Africa 2022 - Virtual

Somtochi Onyekwere has been contributing to Kubernetes and Flux for a long while already. We are very grateful to have her as part of the Flux maintainers team.

At the keynote at Kubernetes Community Days Africa 2022 - Virtual, she will be speaking about her experience going from contributor to project maintainer.

Join her and all the other speakers on July 7 & 8. Register here.

Recent Events (ICYMI) 📺

GitOps Days 2022

GitOps Days was a big celebration of everything we achieved as the Flux community in the past years. It was a big get-together of its maintainers, GitOps practitioners, cloud service vendors and our big community to talk about everything that’s possible today.

If you check out the schedule on its website you get a good idea of the high quality talks and workshops that happened there.

If you should have missed it, don’t despair - the GitOps Days team is working on publishing separate videos and dedicated videos very soon. In the meantime you can still hit the “Register” button on the website to get a link to the recordings of the two days!

Thanks to everyone who attended and organised the event - we had a fabulous time!

Flux Bug Scrub

The next dates are going to be:

We really enjoyed this demo of the k3d git server recently. It’s a local Git server that runs outside of Kubernetes, to support offline dev in a realistic but also simple way that does not depend on GitHub or other hosted services. If folks have other Flux-related topics and want a friendly audience to present for interest and feedback, we are always open to ideas and will host, come pitch us with your Flux talks while we iterate weekly issue queue hygiene!

In other news

People writing/talking about Flux

Alexander Block: Multiple Environments with Flux and Kluctl

Alexander Block is not only the author of Kluctl, but has joined us as a Flux contributor as well. If you are new to Kluctl, it says on its website that

Kluctl is the missing glue to put together large Kubernetes deployments.

It allows you to declare and manage small, large, simple and/or complex multi-env and multi-cluster deployments.

Kluctl does not have cluster-side dependencies and works out of the box.

Check out the new blog post to get an idea how Kluctl helps you wire up multiple environments.

News from the Website and our Docs

Flux Adopters shout-out

We are very pleased to announce that the following adopters of Flux have come forward and added themselves to our website: Autops, MediaMarktSaturn, J.B. Hunt, QuickTable, SenseLabs and TraefikLabs.

More docs and website news

We are constantly improving our documentation and website - here are a couple of small things we landed recently:

Add section for OCI Helm repositories
Add documentation for how to run jobs with Flux.
Add post-deployment jobs and repo structure.
Add "Karmada + Flux" user guide.
Add --ssh-hostkey-algos to image-automation-controller docs.
helm gh actions guide: Exclude events related to dependencies check.
Add guide: Promote Flux Helm Releases with GitHub Actions.
Monitoring: Add Loki and Flux logs to guide.
Update azure docs on mozilla sops.
Migration: Add links for flux v1 uninstall.
Roadmap: Add OCI items for GA.
Ecosystem page: We finally got around to adding Weave GitOps.
Build: Update docsy and hugo. Make builds more robust.

Thanks a lot to these folks who contributed to docs and website: Stefan Prodan, Ihor Sychevskyi, Ed Briggler, Max Jonas Werner, Paulo Gomes, xiexiong, Chrliey Haley, Hidde Beydals, Jianbo Sun, Kevin Fritz, LXM, Philip Laine, Poor12, Somtochi Onyekwere, Soulé Ba, Vincent Palmer, Vincent Van der Kussen, Wiliam Brode, netthier.

In particular we would like to thank Ihor Sychevskyi again who took on fixing small UI glitches all over the place - especially on mobile the site should work a lot better now!

Flux Project Facts

We are very proud of what we have put together. We want to reiterate some Flux facts - they are sort of our mission statement with Flux.

🤝 Flux provides GitOps for both apps or infrastructure. Flux and Flagger deploy apps with canaries, feature flags, and A/B rollouts. Flux can also manage any Kubernetes resource. Infrastructure and workload dependency management is built-in.
🤖 Just push to Git and Flux does the rest. Flux enables application deployment (CD) and (with the help of Flagger) progressive delivery (PD) through automatic reconciliation. Flux can even push back to Git for you with automated container image updates to Git (image scanning and patching).
🔩 Flux works with your existing tools: Flux works with your Git providers (GitHub, GitLab, Bitbucket, can even use s3-compatible buckets as a source), all major container registries, and all CI workflow providers.
🔒 Flux is designed with security in mind: Pull vs. Push, least amount of privileges, adherence to Kubernetes security policies and tight integration with security tools and best-practices. Read more about our security considerations.
☸️ Flux works with any Kubernetes and all common Kubernetes tooling: Kustomize, Helm, RBAC, and policy-driven validation (OPA, Kyverno, admission controllers) so it simply falls into place.
🤹 Flux does Multi-Tenancy (and “Multi-everything”): Flux uses true Kubernetes RBAC via impersonation and supports multiple Git repositories. Multi-cluster infrastructure and apps work out of the box with Cluster API: Flux can use one Kubernetes cluster to manage apps in either the same or other clusters, spin up additional clusters themselves, and manage clusters including lifecycle and fleets.
📞 Flux alerts and notifies: Flux provides health assessments, alerting to external systems and external events handling. Just “git push”, and get notified on Slack and other chat systems.
👍 Users trust Flux: Flux is a CNCF Incubating project and was categorised as "Adopt" on the CNCF CI/CD Tech Radar (alongside Helm).
💖 Flux has a lovely community that is very easy to work with! We welcome contributors of any kind. The components of Flux are on Kubernetes core controller-runtime, so anyone can contribute and its functionality can be extended very easily.

Over and out

If you like what you read and would like to get involved, here are a few good ways to do that:

Join our upcoming dev meetings on 2022-07-06 or 2022-07-14.
Talk to us in the #flux channel on CNCF Slack
Join the planning discussions
And if you are completely new to Flux, take a look at our Get Started guide and give us feedback
Social media: Follow Flux on Twitter, join the discussion in the Flux LinkedIn group.

We are looking forward to working with you.

Blog: GitOps Days 2022

Fri, 03 Jun 2022 14:30:00 +0000

The Flux Community is proud to play a central part in the upcoming GitOps Days event. In a free two-day event we will join in and all learn from Flux maintainers, adopter use-cases, end-user stories and integrators who use Flux to power their GitOps offerings.

The event is a testament to what we as a community have accomplished together, how GitOps as a best-practise has evolved and to the general growth of the ecosystem.

We hope you will join us, as the event is a lot of fun, a great way to get to know people and get inspired.

GitOps Days 2022 is a free 2-day online event on June 8-9, 2022.

This is THE event for your GitOps journey! Getting started? Taking GitOps to the next level? We’ll cover all of the steps for your success!

The event will run from ~7:00 am Pacific Time to ~3:00 pm PT each day as a free online event.

✨✨ Register now to reserve your spot to receive updates to the schedule and speakers. ✨✨

Join the conversation! Chat with the speakers and other attendees!

Invite yourself at https://slack.weave.works and hang out with us at #gitopsdays

Event Schedule

The event is packed with great speakers and talks, here is an excerpt to whet your appetite.

Day 1 will bring you the fantastic mix of a GitOps hands-on tutorial, keynotes, demos how GitOps products have been successfully based on top of Flux and other news from our ecosystem.

Hands-on Tutorial: Intro to Kubernetes, GitOps, and Observability – Tiffany Wang (Weaveworks) & Joaquin Rodriguez (Microsoft)
Keynote: CNCF Ecosystem & GitOps as a Natural Evolution of Kubernetes – Taylor Dolezal (CNCF)
Keynote: GitOps is the Way to Overcome the Scaling Wall by Introducing New Operational Models? An Enterprise Market Perspective – Philippe Ensarguet (Orange)
Keynote: GitOps’ Wealth of Opportunity – Mae Large (VMware)
Keynote: GitOps Business Benefits at State Farm – Rayce Brossette (State Farm)
GitOps with Weave GitOps – James Wilson (Weaveworks)
GitOps in Microsoft Azure with Flux – Chris Sanders & Jonathan Innis (Microsoft)
GitOps with Amazon EKS Anywhere – Chandler Hoisington & Joey Wang (AWS)
GitOps with VMware Tanzu Application Platform – Leigh Capili (VMware)
How D2iQ Operates Flux – Deepak Goel & Max Jonas Werner (D2iQ)
GitOps with Flux on OpenShift – Andrew Block (Red Hat)
Reconcile Terraform Resources the GitOps Way – Priyanka “Pinky” Ravi (Weaveworks) & Roberth Strand (Crayon)
Securing Kubernetes Secrets for GitOps with HashiCorp Vault – Rosemary Wang (HashiCorp)
Happier Helming with GitOps and Flux – Scott Rigby (Weaveworks, Flux & Helm Maintainer)
Introducing Flux Visual Studio Code Extension in Public Beta – Kingdon Barrett (Weaveworks)

On Day 2 we will dive deeper. Particularly security and policy management will be hot topics. And if you are interested in how other organisations implemented GitOps, check out the wealth of user success stories.

Flux & Flagger Deep Dives – Philip Laine (Xenit) & Stefan Prodan (Weaveworks)
Crayon’s Cloud Native Journey – Roberth Strand (Crayon)
GitOps & Progressive Delivery with Flux, Flagger, and Istio – Marco Amador (Anova)
GitOps with Flux at Safaricom – Winnie Gakuru (Safaricom)
Key Learnings from Migrating from Flux1 to Flux2 – Josh Callis (OSO.sh)
GitOps, A Slightly Realistic Situation on Kubernetes with Flux – Laurent Grangeau (Google) & Ludovic Piot (theGarageBandOfIT)
GitOps + Security – Anaïs Urlichs (Aqua Security)
Applying GitOps to Everything with Flux + Crossplane – Viktor Farcic (Upbound)
GitOps and Flux at State Farm (Technical Deep Dive) – Rayce Brossette & Perry Wu (State Farm)
Multi-tenancy Best Practices Using the Cloud Native Ecosystem: ZScaler’s GitOps Journey – Josh Carlisle & Neeta Rathi (ZScaler)
GitOps at RingCentral with Flux & Flagger – Ivan Anisimov (RingCentral)
Flagger and Linkerd revisited: how I learned to stop worrying and use A/B testing – Jason Morgan (Buoyant)
GitOps and Flux Scaled to Hundreds of Developers – Bryan Oliver (Independent) & Kingdon Barrett (Weaveworks)
Policy Management & GitOps – Tony Chong (Weaveworks / Magalix)

Leading through the programme are our MCs Tamao Nakahara and Vanessa Abankwah from Weaveworks’ DX team. Daniel Holbach aka DJ Desired State will play his newest favorites in the breaks.

Please make sure you check the actual schedule on https://www.gitopsdays.com/ as it might be subject to change.

We are very much looking forward to the event and hope you’ll be there with us too! Register today!

#flexyourflux

The #flexyourflux campaign we started for KubeCon is still ongoing. Only here you can still win a 1h-long 1-on-1 meeting with Flux Core Maintainer Stefan Prodan.

Get your limited edition @fluxcd T-shirts at @KubeCon_ EU only! In person at Valencia only!#flexyourflux with our quiz at pick up your shirt at the Flux booth! https://t.co/BHxJxeYhRq @kubernetesio #GitOps pic.twitter.com/HWD2Uru0PX
— Tamao Nakahara - DevRelCon July 18-19🎉 (@mewzherder) May 17, 2022

The t-shirts unfortunately are out of stock, but the meeting with Stefan is still up for grabs! We will draw the lucky winners live at the GitOps Days event.

Blog: May 2022 Update

Wed, 01 Jun 2022 12:30:00 +0000

It’s the beginning of June 2022 - let’s recap together what happened in May - it has been a lot!

News in the Flux family

Flux v0.30 release

The latest Flux release is the v0.30 release series. It comes with new features and improvements. Users are encouraged to upgrade for the best experience. Note: v0.29.0 included breaking changes.

🚀 Features and improvements

Support for disabling remote bases in Kustomize overlays: this release adds support to the kustomize-controller for disallowing remote bases in Kustomize overlays using --no-remote-bases=true (default: false). When this flag is enabled on the controller, all resources must refer to local files included in the Source Artifact, meaning only the Flux Sources can affect the cluster-state. Users are advised to enable it on production systems for security and performance reasons.
Support for defining a KubeConfig Secret data key: both Kustomization and HelmRelease resources do now accept a .spec.kubeConfig.SecretRef.key definition. When the value is specified, the KubeConfig JSON is retrieved from this data key in the referred Secret, instead of the defaults (value or value.yaml).
Support for defining a ServiceAccountName in ImageRepository objects: the ImageRepository object does now accept a .spec.serviceAccountName definition. When specified, the image pull secrets attached to the ServiceAccount are used to authenticate towards the registry.

🎁 Link to release page

🔒 Flux Security announcement

We published three CVEs today which affect Flux versions earlier than v0.29.0. We recommend updating your Flux system at your earliest convenience.

More information on the advisories can be found in our security policy page.

To get some additional background on the advisories and what steps we are taking to make Flux more secure, check out our blog post about the advisories as well.

Upcoming Flux Release

The next Flux release is just a few days out. Here is in a nutshell what you can look forward to - but there’ll be more!

OCI Helm chart support as described in RFC-0002 will become available. But at time of writing, has two caveats:
- Chart dependencies from OCI repositories are not supported. #722
- Custom CA certificates are not supported. #723
GitRepository reconciliation will be more efficient when checking out repositories using branches or tags by added support for no-op clones.
The libgit2 managed transport will be moved out of experimental mode, and is the new default.

Make sure you watch our Slack and Twitter to get the update. Give us a star and watch for releases maybe as well.

Flagger 1.21.0 brings lots of improvements

This release comes with an option to disable cross-namespace references to Kubernetes custom resources such as AlertProviders and MetricProviders. When running Flagger on multi-tenant environments it is advised to set the -no-cross-namespace-refs=true flag.

In addition, this version enables Flagger to target Istio and Kuma multi-cluster setups. When installing Flagger with Helm, the service mesh control plane kubeconfig secret can be specified using --set controlplane.kubeconfig.secretName.

Flux Ecosystem

We have a lot of updates from the Flux Ecosystem and love how everything keeps on growing! If you are interested in more news from Flux integration, make sure you register for GitOps Days at 8-9 June - a lot of engineers and companies will be talking about their work and how you can benefit from it.

Flux Subsystem for Argo

Flux Subsystem for Argo was upgraded to support Argo CD v2.2.9, and welcomed Kingdon Barrett as a new maintainer for the project.

Terraform-controller

terraform-controller v0.9.5 was released which contains new features such as support for Runner Pod’s metadata, support environment variables for Runner Pod so that you can set proxy for Terraform binary with HTTPS_PROXY for example. This release also included many bug fixes.

Weave GitOps

The Weave GitOps team released v0.8.1 for Weave GitOps. This release is an iteration on top of our prior release. We have fixed a lot of bugs and made UI enhancements based on feedback from the community. For example, you are able to reconcile Flux objects directly from the UI. We have a lot of great features planned over the next couple months. Please do not hesitate to drop in some feature requests.

New additions to the Flux Ecosystem

We are thrilled to see the Flux Ecosystem growing on a continuous basis. The most recent additions to our Flux Ecosystem page are flux-kluctl-controller and gardener-extension-shoot-flux.

kluctl/flux-kluctl-controller is a Flux controller for managing Kluctl deployments. Its website explains kluctl as follows

Kluctl is the missing glue to put together large Kubernetes deployments.

It allows you to declare and manage small, large, simple and/or complex multi-env and multi-cluster deployments.

Kluctl does not have cluster-side dependencies and works out of the box.

23technologies/gardener-extension-shoot-flux is a new integration with Flux. Gardener implements the automated management and operation of Kubernetes clusters as a service. With this extension fresh clusters will be reconciled to the state defined in the Git repository by the Flux controller.

Recent & Upcoming Events

It’s important to keep you up to date with new features and developments in Flux and provide simple ways to see our work in action and chat with our engineers.

Recent Events (ICYMI) 📺

We feel blessed to have such a big community of users, contributors and integrators and so many are happy to talk about their experiences. In May here are a couple of talks we would like to highlight.

Last month was all about KubeCon and there were lots of great sessions we enjoyed and recommend watching. It might be best if you just head to our KubeCon Re-Cap blog post and take it from there!

Here is a list of additional videos and topics we really enjoyed - please let us know if we missed anything of interest and we will make sure to mention it in the next post!

📺 GitOps with Flux on AKS @AzureKubernetesService (Amsterdam) Meetup - Kingdon Barrett (Weaveworks) & Jonathan Innis (Microsoft)

📺 GitOps: Core Concepts & How to Structure Your Repos - Scott Rigby & Priyanka Ravi (Weaveworks)

📺 DevOpsDays Birmingham AL: GitOps and Flux scaled to 100s of Developers - Bryan Oliver & Kingdon Barrett (Weaveworks)

📺 DOK (Data On Kubernetes) #127: Flux for Helm Users! With Scott Rigby (Weaveworks)

📺 Community Office Hours: Injecting Secrets from HashiCorp Vault into Flux - Priyanka Ravi (Weaveworks) & Rosemary Wang (Hashicorp)

📺 Reconcile Terraform Resources the GitOps Way - Priyanka Ravi (Weaveworks)

📺 GitOps (Flux) Extension for VS Code - Kingdon Barrett (Weaveworks)

#flexyourflux

The #flexyourflux campaign we started for KubeCon is still ongoing. Until GitOps Days (see below) you can still win a 1h-long 1-on-1 meeting with Flux Core Maintainer Stefan Prodan.

Get your limited edition @fluxcd T-shirts at @KubeCon_ EU only! In person at Valencia only!#flexyourflux with our quiz at pick up your shirt at the Flux booth! https://t.co/BHxJxeYhRq @kubernetesio #GitOps pic.twitter.com/HWD2Uru0PX
— Tamao Nakahara - DevRelCon July 18-19🎉 (@mewzherder) May 17, 2022

We will draw the lucky winners live at the GitOps Days event (8-9 June).

Upcoming Events 📆

We are happy to announce that we have a number of events coming up in June - tune in to learn more about Flux and GitOps best practices, get to know the team and join our community.

Flux Bug Scrub

The next dates are going to be:

GitOps Days 2022

GitOps Days 2022 is a free 2-day online event on June 8-9, 2022 with Flux center stage!

This is THE event for your GitOps journey! Getting started? Taking GitOps to the next level? We’ll cover all of the steps for your success!

The event will run from ~9:00 am PT to ~3:00 pm PT each day as a free online event.

✨✨ Register now to reserve your spot to receive updates to the schedule and speakers. Join the conversation! Chat with the speakers and other attendees! Invite yourself at https://slack.weave.works and hang out with us at #gitopsdays

Talks and tutorials on how to get started with Kubernetes and GitOps
Talks from Flux users about their use cases
How to do GitOps securely
Platforms that offer GitOps: Microsoft Arc Kubernetes, AWS Anywhere, Weave GitOps, D2iQ Kubernetes Platform, and more! all using Flux!
Flux in the CNCF and the GitOps Ecosystem
Flux support and Integrations: Flux + Helm, Terraform, HashiCorp Vault, Jenkins, OpenShift, Visual Studio Code, and much much more!
Technical deep dives with Flux maintainers
Speakers from Orange, RingCentral, and more just added

In other news

People writing/talking about Flux

News from the Website and our Docs

Flux Adopters shout-out

We are very pleased to announce that the following adopters of Flux have come forward and added themselves to our website: Tietoevry, Grafana Labs, Aily Labs, SisID, FHE3, Qualifio, Axel Springer SE, Cookpad.

If you are like us, you really enjoy hearing adopter use case stories. At Gitops Days, there will be loads of those, so join us 8-9 June.

More docs and website news

We are constantly improving our documentation and website - here are a couple of small things we landed recently:

By updating to the latest hugo and docsy, we were able to drop some of our custom code to show e.g. tabs in our documentation.
We added a gallery shortcode to be able to show a collection of pictures nicely.
New docs for
- Enable Helm repositories caching
- Locking down multi-tenant clusters by disabling Kustomize remote bases
- Deploy key rotation
- How to disable cross namespace references
- How to bootstrap Flux on GCP GKE with Cloud Source repositories
New videos added to our Flux Resources page.

Thanks a lot to these folks who contributed to docs and website: Stefan Prodan, Ihor Sychevskyi, Matt J WIlliams, Paulo Gomes, Alexander Block, Andreas Loholt, Axel Fontana, Cosmin Banciu, Christian Berendt, Jiri Tyr, Julien Duchesne, Martin Weber, Max Jonas Werner, Steven Koeberich, as09.

In particular we would like to thank Ihor Sychevskyi who recently took on fixing small UI glitches all over the place - especially on mobile the site should work a lot better now!

New Project Member: Stacey Potter

Stacey Potter

We are very happy to announce that Stacey Potter joined us as a Flux Project Member.

Stacey has helped the Flux team out a great deal by organising a lot of Flux-related events like GitOps Days, Weave Online User Groups, adding videos to the Flux Resources page and our YouTube playlist, and coordinating with the team on our Project presence for KubeCon events. She’s such a pleasure to work with and we owe quite a bit of Flux’s success to the stages she created for our speakers.

As a side-note: we updated the Flux Governance recently to make it even clearer that we love all kinds of contributions, be they code or not. We hope that many more of you will follow this path.

@FluxCD welcomes contributors of all kinds, for realz!

🥳Today I joined as an official Project Member - without a single line of code written.🤩 Thx to all the Flux Fam, esp @dholbach @makkes for sponsoring me.

If I can do it, you can too! Join us!https://t.co/RO6CbSKiBK
— Stacey Potter (@stacey_potter) May 25, 2022

New Flagger Maintainer: Sanskar Jaiswal

Sanskar Jaiswal

Sanskar Jaiswal has been working on Flux and Flagger for quite a while now. One of his major contributions was to add Gateway API support to Flagger. We are very pleased to let you know that he joined the ranks of Flagger maintainers now.

Flux Project Facts

We are very proud of what we have put together. We want to reiterate some Flux facts - they are sort of our mission statement with Flux.

🤝 Flux provides GitOps for both apps or infrastructure. Flux and Flagger deploy apps with canaries, feature flags, and A/B rollouts. Flux can also manage any Kubernetes resource. Infrastructure and workload dependency management is built-in.
🤖 Just push to Git and Flux does the rest. Flux enables application deployment (CD) and (with the help of Flagger) progressive delivery (PD) through automatic reconciliation. Flux can even push back to Git for you with automated container image updates to Git (image scanning and patching).
🔩 Flux works with your existing tools: Flux works with your Git providers (GitHub, GitLab, Bitbucket, can even use s3-compatible buckets as a source), all major container registries, and all CI workflow providers.
🔒 Flux is designed with security in mind: Pull vs. Push, least amount of privileges, adherence to Kubernetes security policies and tight integration with security tools and best-practices. Read more about our security considerations.
☸️ Flux works with any Kubernetes and all common Kubernetes tooling: Kustomize, Helm, RBAC, and policy-driven validation (OPA, Kyverno, admission controllers) so it simply falls into place.
🤹 Flux does Multi-Tenancy (and “Multi-everything”): Flux uses true Kubernetes RBAC via impersonation and supports multiple Git repositories. Multi-cluster infrastructure and apps work out of the box with Cluster API: Flux can use one Kubernetes cluster to manage apps in either the same or other clusters, spin up additional clusters themselves, and manage clusters including lifecycle and fleets.
📞 Flux alerts and notifies: Flux provides health assessments, alerting to external systems and external events handling. Just “git push”, and get notified on Slack and other chat systems.
👍 Users trust Flux: Flux is a CNCF Incubating project and was categorised as "Adopt" on the CNCF CI/CD Tech Radar (alongside Helm).
💖 Flux has a lovely community that is very easy to work with! We welcome contributors of any kind. The components of Flux are on Kubernetes core controller-runtime, so anyone can contribute and its functionality can be extended very easily.

Over and out

If you like what you read and would like to get involved, here are a few good ways to do that:

Join our upcoming dev meetings on 2022-06-02 or 2022-06-08.
Talk to us in the #flux channel on CNCF Slack
Join the planning discussions
And if you are completely new to Flux, take a look at our Get Started guide and give us feedback
Social media: Follow Flux on Twitter, join the discussion in the Flux LinkedIn group.

We are looking forward to working with you.

Blog: KubeCon EU 2022 Wrap-Up

Wed, 25 May 2022 10:00:00 +0000

It was KubeCon + CloudNativeCon EU 2022 last week and if you weren’t able to attend, this post provides you with everything you need to know about Flux and GitOps that happened there. The schedule was packed with case studies, development updates and many new friendships formed at our booths and in the hallway track.

Here we go #KubeConEU 🎉 Flux office hours starts at 1PM room 2H. Come join us 🤗 pic.twitter.com/RfcTioi58Q
— Stefan Prodan (@stefanprodan) May 16, 2022

Monday

On Monday we kicked off the day for Flux with a Project Meeting. We are grateful we had this opportunity through the CNCF to offer a 4 hour event available to any interested community members. Stefan Prodan, Leigh Capili and Priyanka Ravi led various talks and provided a great and very diverse overview of what’s happening in Flux and adjacent tooling these days and how to best take advantage of it.

What a great kick-off to a great week of Flux and #GitOps at @kubecon_ - thanks @stefanprodan @r6by @capileigh @PinkyyRavi and Vanessa for leading the 4h Flux Project Meeting! 😍 pic.twitter.com/6YdxRNHNRM
— Flux Project (@fluxcd) May 16, 2022

Thanks as well Vanessa Abankwah for pulling all the strings in the background!

The Cloud Native Telco Day was happening as well on Monday and Philippe Ensarguet, CTO at Orange Business, live-tweeted some of the learnings there. We are very pleased to know that the GitOps setup of Swisscom is based on Flux. Deutsche Telekom is continuing their voyage on Das Schiff together with Flux as well.

Tuesday - GitOpsCon

Who would have thought a couple of years ago that one day we would have an entire conference day just about GitOps - this year it was two simultaneous tracks no less. We want to thank the Open GitOps group for organising this and inviting great speakers from across all of the GitOps space!

Taken from the CNCF flickr account.

Let’s kick the day at #kubecon with opening session cooked by @r6by, @todaywasawesome, and @ChrisShort sharing what #GitOps is, and how to get it right🚀
Special thoughts to my mate @monadic who defined this initial idea of #operation by pull request👏#GitOpsCon pic.twitter.com/5znl1txRuj
— Philippe Ensarguet (@P_Ensarguet) May 17, 2022

Here is our own small selection of favourites. If you want to see all the talks from GitOpsCon, take a look at the YouTube channel of the Cloud Native Computing Foundation - there are loads more.

Crossing the Divide: How GitOps Brought AppDev & Platform Teams Together! - Priyanka "Pinky" Ravi, Weaveworks
Michael Irwin on using Flux for multi-tenancy: Creating A Landlord for Multi-tenant K8s Using Flux, Gatekeeper, Helm, and Friends - Michael Irwin
AppsFlyer talks about Flux at the core of their huge infra:
- GitOps Everything!? We Sure Can!, Ayelet de-Roos, AppsFlyer
- We Have Always Done It This Way! Now Let’s Try Something Completely Different - Eliran Bivas
Max’ talk about Flux for multi-cluster envs: Managing Thousands of Clusters and Their Workloads with Flux - Max Jonas Werner, D2iQ
Environment promotion (Form3): Solving Environment Promotion with Flux - Sam Tavakoli & Adelina Simion, Form3
Lightning Talks:
- Secret decryption: Lightning Talk: Hiding in Plain Sight - How Flux Decrypts Secrets - Somtochi Onyekwere, Weaveworks
- Progressive delivery: Lightning Talk: GitOps and Progressive Delivery with Flagger, Istio and Flux - Marco Amador, Anova
- GitOps, A Slightly Realistic Situation on Kubernetes with Flux - Laurent Grangeau, Google & Ludovic Piot, theGarageBandOfIT

Wednesday - Friday - KubeCon

Wednesday through Friday was the main event, with the big keynotes, talks on many different tracks and a big booth space. We are happy we had such a great team at the Flux booth because ours was massively frequented and our team gave a huge amount of demos, answered questions and were there to hang out with.

In addition to the physical booth in Valencia, we had a virtual booth as well, where Kingdon Barrett held our weekly Bug Scrub event and we gave a number of lightning talks as well.

Lightning talks at the virtual booth

First up was Sanskar Jaiswal, Software Engineer at Weaveworks, who recently became Flagger maintainer and contributed Gateway API support. Watch the demo here:

We were happy to have Rosemary Wang, Developer Advocate at HashiCorp, there who walked us through securing secrets in Flux by using Vault:

#flexyourflux

Another contributing factor to the amount of people coming to our booth was the #flexyourflux campaign, where you could

Get a nice “flex your flux” t-shirt for answering a couple of questions about Flux
Win an opportunity to have a 1-on-1 1h long meeting with Stefan Prodan, Flux core maintainer

Get your limited edition @fluxcd T-shirts at @KubeCon_ EU only! In person at Valencia only!#flexyourflux with our quiz at pick up your shirt at the Flux booth! https://t.co/BHxJxeYhRq @kubernetesio #GitOps pic.twitter.com/HWD2Uru0PX
— Tamao Nakahara - DevRelCon July 18-19🎉 (@mewzherder) May 17, 2022

The t-shirts were only available in person, but the meeting with Stefan you can still win, simply enter by filling out https://bit.ly/flexyourflux. We will draw the winners live at GitOps Days (see below). Good luck to all participants!

Talks you might have missed

Getting Started with Flux and GitOps

Tiffany Wang from Weaveworks and Joaquin Rodriguez led through a 1.5h hands-on tutorial called “ Intro to Kubernetes, GitOps and Observability”. The idea was to offer newcomers a quick way to experience Kubernetes and its natural evolutionary developments: GitOps and Observability. Attendees were able to use and experience the benefits of Kubernetes that impact reliability, velocity, security, and more. The session covered key concepts and practices, as well as offer attendees a way to experience the commands in real-time. The tutorial covers: kubectl, K9s, Metrics (Prometheus), Dashboards (Grafana), Logging (Fluent Bit), GitOps (Flux).

The feedback we heard from people on the ground was that they had a blast. If you missed it: good news - it’ll be happening at GitOps Days as well!

The Flux Deep Dive

Stefan Prodan delivered the Flux Deep Dive session - this time focused on security aspects.

"Flux Security Deep Dive" by @stefanprodan #Flux @fluxcd #KubeCon #CloudNativeCon pic.twitter.com/rGQ5jWdFpm
— Nico Meisenzahl ☁️☕️ (@nmeisenzahl) May 18, 2022

There was a lot to be learnt and since security has been such a big focus for the entire team since the rewrite of Flux, also a lot to catch up on.

Stefan will give his talk at GitOps Days too (see below).

Flux Virtual Office Hours

Flux Maintainers Paulo Gomes & Kingdon Barrett hosted the Flux Virtual Office Hours where they covered the latest Flux features, what’s coming soon, and an intro to debugging the controllers for new contributors. Check out the replay here:

What’s more

Obviously the hallway track is one of the key events at KubeCon. It’s where you find new friends, learn and make new plans with other community members. There was a lot of this at the booth, at lunch and at the evening events.

In addition to that, many new users and community members find their way to Slack, our Twitter and LinkedIn group. We are also especially pleased that some new adopters added themselves to our website - remember that’s one of the safest ways to make Flux maintainers happy. 🥰

We expect that more talk videos are going to get to us in the next days and we will make sure to mention them all on our resources page and in our monthly updates.

Outlook: GitOps Days

If you had a FOMO experience over the last week, we are happy to let you know that GitOps Days are coming up! A free, two days event with lots of great talks, a lot of fun and lots to catch up with in case you missed seeing talks at KubeCon.

GitOps Days!
June 8-9, 2022

This is THE event for your GitOps Journey! Getting started? Taking GitOps to the next level? We’ll cover all of the steps for your success!

Come hear from speakers like Taylor Dolezal (CNCF), Anaïs Urlichs (Aqua Security, CNCF Ambassador), Viktor Farcic (Upbound/Crossplane), Mae Large (VMware), Rosemary Wang (HashiCorp), Jason Morgan (Buoyant/Linkerd), and so many more!

Big Thank You

This wouldn’t have been possible without the support of many many people.

People speaking: Stefan Prodan, Somtochi Onyekwere, Max Jonas Werner, Priyanka Ravi, Tiffany Wang, Paulo Gomes, Scott Rigby, Kingdon Barrett, Sanskar Jaiswal and others.

Folks organising, writing, fact-checking and supporting the booth: Vanessa Abankwah, Stacey Potter, Tamao Nakahara, Juozas Gaigalas and others.

And of course many people from adjacent communities, adopter companies and of course the CNCF.

Sorry if we missed to mention anyone.

And in closing out, here is a short selection of official KubeCon photos, all around Flux and GitOps:

Blog: Flux at KubeCon EU 2022

Thu, 12 May 2022 07:30:00 +0000

We are so excited! KubeCon / CloudNativeCon Europe 2022 is happening 16-20 May 2022 in Valencia, Spain. Team Flux is going to be there and we are looking forward to meeting you in-person or in our virtual sessions, meetings and co-located events. 🎉

Update: 2022-05-16: We updated the details of the project meeting.

If you can’t make it to KubeCon

Many of us won’t be able to attend in-person this time around. Don’t worry, we have you covered! There will be lots of opportunities to meet Flux people online, see talks, discuss and learn from each other.

There’s just one thing you need to do to join: visit our Flux at KubeCon site and bookmark it. It lists all the talks and all events - everything Flux-related in one place.

We are also going to have a Virtual Booth. This will allow you to talk to Flux people at all official booth times, we will have short talks, watch some of the talks together and bring like-minded people together. We can’t wait to meet you there.

Visit our Flux at KubeCon site and bookmark it now!

🎉 Flex your Flux knowledge and win prizes! 🎉

If you are at KubeCon, this is your chance to get a limited edition T-shirt (while supplies last) and a chance to win a meeting with a Flux or Flagger maintainer!

Essentially what you need to do is

Visit the "Flux Booth in the Project Pavilion"
You will be given a Flux question, if you answer correctly, you will win a T-shirt
If you answer an advanced question you will win a T-Shirt + an 1-hour meeting with a Flux maintainer (winners announced after KubeCon).
Make sure you tweet using #flexyourflux

If you are entering virtually make sure you participate online by tweeting questions will be posted during KubeCon so keep an eye out for them !!!

Visit the Flux booth (we are number 6) in person for more info and follow #flexyourflux from the @fluxcd Twitter handle.

At KubeCon, watch out for these folks

There will be lots more of Team Flux in the crowds: contributors, advocates, friends, implementers and folks in adjacent communities we work a lot with. Find them at our physical booth or in all of the sessions listed below.

All right, how about these folks above? They will all give sessions, MC the events and coordinate all the things.

Priyanka “Pinky” Ravi has been speaking about her experience as a DevOps Engineer a lot lately. She gathered a lot of experience working at a large insurance institution and helped them transition to Flux and GitOps. Since then she has been a recurring guest at Weave Online User Group, GitOps Days and elsewhere.
Stefan Prodan is the creator of Flagger and has been a core maintainer of Flux since Weaveworks donated the project to CNCF. He is a maintainer of SMI and he's been part of Team Flux for 4+ years already. He is also the owner of a very cute dog.
Scott Rigby is a maintainer of Flux and Helm, and one of the chairs of OpenGitOps. He knows the ecosystem very well, contributed a lot to all these projects and was one of the driving forces behind Flux Governance.
Philips Laine is a core maintainer of Flux as well and among other things founded the terraform-provider-flux project.
Vanessa Abankwah is pulling all the strings behind the scenes. Not only did she organise all of our presence at KubeCon, but lots and lots of meetups and events where Team Flux was present.
Max Jonas Werner is a core maintainer of Flux as well and makes sure that Flux works great in D2iQ’s DKP platform. Apart from that he is interested in landing OCI support in Flux and loads more.

Everyone is looking forward to meeting you! 💖

This is what is happening at KubeCon

Monday, 16 May

13:00 - 17:00 (Room 2H - Event Center): Flux Project Meeting: We will kick off the Flux get-togethers and festivities with an in-person meeting for all Flux users, contributors, maintainers and generally interested folks. This will be an opportunity to get to know each other, have a chat and see what people’s interests are. ( Sign up here.) Contact people on the ground are: Scott Rigby, Priyanka Ravi and Stefan Prodan.

Schedule:

13:00 Meet and greet
13:30 Intro to GitOps and Flux lightning talk + Q&A with Pinky
14:00 Flux’s Top 10 Features with Stefan Prodan + Q&A/office hours
15:00 Open topics. We can do office hours or mix and match with lightning talks on Flux + Helm, Flux + Terraform, Flux + HashiCorp Vault, Flux + Visual Studio Code, etc. Whatever people want.

Click here to register ( here) for the Flux Project Meeting. (Free, in-person, no special ticket required)

Details Flux Project Meeting

Monday, May 16, 13:00 - 17:00 CEST

Room 2H | Event Center

Space is limited Please note: we will not have any live streaming, recordings, or any virtual component available for this meeting.

Tuesday 17 May - GitOpsCon

Lots and lots of talks about GitOps in general and Flux in particular, here’s a short selection of what to look forward to:

What is GitOps and How to Get It Right - Dan Garfield (Codefresh); Chris Short (AWS) & Scott Rigby (Weaveworks) (9:00 - 9:35)
Hiding in Plain Sight - How Flux Decrypts Secrets - Somtochi Onyekwere (Weaveworks) (11:05 - 11:15)
Taming the Thundering Gitops Herd with Update Policies - Joaquim Rocha & Iago López Galeiras (Microsoft) (11:35 - 11:45)
GitOps and Progressive Delivery with Flagger, Istio and Flux - Marco Amador (Anova) (13:20-13:30)
Creating A Landlord for Multi-tenant K8s Using Flux, Gatekeeper, Helm, and Friends - Michael Irwin (Docker) (13:35-14:05)
GitOps, A Slightly Realistic Situation on Kubernetes with Flux - Laurent Grangeau (Google) & Ludovic Piot (theGarageBandOfIT) (14:10 - 14:40)
Solving Environment Promotion with Flux - Sam Tavakoli & Adelina Simion (Form3) (14:10 - 14:40)
Managing Thousands of Clusters and Their Workloads with Flux - Max Jonas Werner (D2iQ) (14:55 - 15:25)
Crossing the Divide: How GitOps Brought AppDev & Platform Teams Together! - Russ Palmer (State Farm) & Priyanka ‘Pinky’ Ravi (Weaveworks) (15.30 - 16:00)
GitOps Everything!? We Sure Can!, AppsFlyer (15:30 - 16:00)
Lightning Talk: Addressing Log4Shell with Software Supply Chains - Duane DeCapite (VMware) (18:04 - 18:09)

Wednesday 18 May - Friday May 20 - KubeCon

Over these three days we are going to be at the Flux booth (both virtually and on the ground), so come over for a chat. We are planning loads of talks, demos and ample time to have a chat, get to know everyone, ask questions and have great new ideas together!

On top of that, here is a list of talks, workshops and sessions during those days:

Wed 18: Flux Security Deep Dive - Stefan Prodan (Weaveworks) (11:55 - 12:30)
Wed 18: Intro to Kubernetes, GitOps, and Observability Hands-On Tutorial - Johee Chung (Microsoft) & Tiffany Wang (Weaveworks) (11:00 - 12:30)
Wed 18: Flux Bug Scrub - Kingdon Barrett (13:00 - 14:00)
Wed 18: A New Generation of Trusted GitOps for Mixed K8s and Non-K8s End Users - Alexis & Vasu Chandrasekhara (SAP) (15:25 - 16:00)
Thu 19: GitOps to Automate the Setup, Management and Extension a K8s Cluster - Kim Schlesinger (DigitalOcean) (11:00 - 12:30)
Thu 19: Flux Project Office Hour - Paulo Gomes (Weaveworks) (13:30 - 14:15)
Fri 20: Observing Fastly’s Network at Scale Thanks to K8s and the Strimzi Operator - Fernando Crespo & Daniel Caballero, (Fastly) (11:00 - 11:35)
Fri 20: Simplifying Service Mesh Operations with Flux and Flagger - Mitch Connors (Google) & Stefan Prodan (Weaveworks) (14:55 - 15:30)

Please note: all of the above might be subject to change. Please double-check the schedule beforehand. Please reach out to Vanessa Abankwah or Daniel Holbach on Slack if you have questions.

We very much look forward to seeing you there! 💖

What’s to come after KubeCon

GitOps Days!
June 8-9, 2022

Don’t miss this free, 2-day online event on June 8-9 (2 weeks after KubeCon). This is THE event for your GitOps Journey! Getting started? Taking GitOps to the next level? We’ll cover all of the steps for your success!

Schedule:

Talks and tutorials on how to get started with Kubernetes and GitOps
Talks from Flux users about their use cases
How to do GitOps securely
Platforms that offer GitOps: Microsoft Arc Kubernetes, AWS Anywhere, Weave GitOps, D2iQ Kubernetes Platform, and more! all using Flux!
Flux in the CNCF and the GitOps Ecosystem
Flux support and Integrations: Flux + Helm, Terraform, HashiCorp Vault, Jenkins, OpenShift, Visual Studio Code, and much much more!
Technical deep dives with Flux maintainers

Blog: May 2022 Security Announcement

Tue, 10 May 2022 08:30:00 +0000

tl;dr

The Flux Team has found three security vulnerabilities in Flux, and we strongly advise you to upgrade your clusters as soon as you can.

CVE	Advisory	Severity	Affected versions
CVE-2022-24817	Improper kubeconfig validation allows arbitrary code execution	Critical	`< 0.29.0 >= v0.1.0`
CVE-2022-24877	Improper path handling in Kustomization files allows path traversal	Critical	`< v0.29.0`
CVE-2022-24878	Improper path handling in Kustomization files allows for denial of service	High	`< v0.29.0 >= v0.19.0`

Breaking changes to be aware of in the upgrade process: 0.29, 0.28, 0.27, 0.26, 0.24 - 0.21.

If you cannot immediately update or are hard pressed for time and need a work-around for now, please see the CVE advisories linked above for more information.

Some Background

Last week the Flux Security Team disclosed three new vulnerabilities which affect v0.28 and older versions, and have a greater impact in multi-tenancy deployments.

The reason why the impact is greater in multi-tenancy deployments is due to the way that Flux/GitOps works. Flux tends to operate like a cluster admin, having permissions to apply any changes to a cluster, regardless of their scope - at namespace or cluster levels. Users having access to a source repository, or simply having access to create/alter Flux objects within a cluster can instruct Flux to apply such changes, which in single tenancy effectively means that such users have cluster admin permissions to the target clusters. The caveat being that using Flux you can add additional security controls between the user and the target cluster, for example, before each change is merged into a repository Pull Requests must be created requiring peer reviews. The Open GitOps community started defining and codifying this further - have a look at this blog post if you want to know more.

In multi-tenant environments, users with similar permissions can only affect part of a cluster, or an isolated cluster in a group of clusters. Therefore, if a user can gain escalated privileges (or deny service) at a Flux level, it will have an impact way larger than on single-tenant clusters, as the users can impact more than just themselves.

At time of writing, we are unaware of any public exploits in the wild, and therefore have no reason to believe such vulnerabilities have been actively exploited.

The advisories in detail

CVE-2022-24817 - Kubeconfig Validation

At the beginning of the Flux2’s journey we implemented a feature to apply state to remote clusters. It enables users to have a management cluster in which Flux is installed, which then applies changes to other clusters, making it ideal for some multi-tenancy scenarios in which high isolation across tenants is needed.

The connection between the management cluster and the target clusters is done by referencing a Kubernetes secret containing a kubeconfig, which is set at the spec.KubeConfig field in either Kustomization or HelmRelease objects.

One interesting thing about kubeconfigs is that they are quite extensible. You can for example define an executable which will then be automatically called by kubectl every time it requires a new on-demand access token. An example of this in action is aws-iam-authenticator, which enables AWS users to authenticate against AWS and then use the returning JWT tokens to access their EKS clusters. Once the token expires, the process happens again. All that is managed by kubectl behind the scenes without user intervention.

The problem here is that first, the use of executables in kubeconfigs was enabled by default. Meaning that a malicious tenant would be able to craft a malicious kubeconfig which could lead to privilege escalation within the cluster.

As a solution, we decided to disable this feature by default. It can still be enabled at a cluster level via a new flag --insecure-kubeconfig-exec being sent to the controller binary.

For cluster admins considering this feature, we also recommend the use of AppArmor and SELinux profiles to enforce at Kernel level what binaries could be executed.

CVE-2022-24877 - Kustomization Path Traversal

Flux allows users to lean on Kustomize features to make their lives easier as they go on about declaring the state of their clusters. Some of those features could result in sensitive data from the pod filesystem to be exposed into the target cluster, which could lead to a malicious tenant being privy to anything sensitive that may exist in the controller’s filesystem or attached volumes (e.g. token).

The mitigation for the path traversal was to create stronger bounds, enforcing that all kustomize operations happen within such bounds or result in error.

CVE-2022-24878 - Kustomization Denial of Service

Whilst working on the mitigation of the previous CVE, we have noticed that in some scenarios a specially crafted kustomization.yaml could lead to the kustomize-controller to enter into an endless loop, and finally crash.

For single-tenant clusters, this would mean that an user may make a mistake and the controller stops working, potentially resulting in future reconciliations not being applied. For multi-tenant clusters, depending on the deployment model, a tenant could cause a disruption to affect not only itself but also the other tenants and potentially even the management cluster.

The solution mitigating this vulnerability was to further improve our validation and ensure that such scenarios are not processed in the first place.

Inspecting your Flux system version

To check if your system is currently vulnerable, run flux version --context=my-cluster with the --context set to the cluster you want to inspect. This will report the current Flux binary and controller versions.

Vulnerable Flux system

To find out if your system could be vulnerable, simply find out the version of Flux. Here it’s important to check you are running all of these:

flux < v0.29.0
helm-controller < v0.19.0
kustomize-controller < v0.24.0.

You can do this like so:

$ flux version
flux: v0.22.1
helm-controller: v0.12.2
kustomize-controller: v0.17.0
notification-controller: v0.18.1
source-controller: v0.17.2

Updating your vulnerable system

💥 If you find the controllers versioned within the range mentioned above, follow the upgrade procedure for your system. For flux bootstrap, this can be done by running the command again with the same arguments as used during install.

⚠️ Please note that if you are upgrading from below one of the versions in the following list, there are breaking changes and, pre- and/or post-upgrade notes you need to take into account: 0.29, 0.28, 0.27, 0.26, 0.24 - 0.21.

Up-to-date Flux system

An up to date Flux system should at least have versions listed below:

flux >= v0.29.0
helm-controller >= v0.19.0
kustomize-controller >= v0.24.0

So in practice the output could look like this:

$ flux version
flux version
flux: v0.30.2
helm-controller: v0.21.0
kustomize-controller: v0.25.0
notification-controller: v0.23.5
source-controller: v0.24.4

We encourage all users to keep Flux up-to-date. We offer a GitHub Action with which you can automate the Flux upgrades in a GitOps manner, without having to connect from CI to the cluster’s API, as Flux is capable of upgrading itself from Git.

Flux Security more generally speaking

It is no secret that the Flux community has been investing in security for a long time. Early on the Flux journey we re-architectured the codebase to avoid shelling out to binaries to decrease the likelihood of code execution vulnerabilities. The story about our Git integration we wrote down here.

As Security is such a central pillar of Flux, we are keen to write about it and tell you how you can benefit from all the individual features and improvements we worked on, e.g. SBOMs, CI Checks, Branch Protection, restricted pod security standard and more. Since the beginning we worked hard to ensure that we ship code that does what it needs to do, even when that means having to rewrite parts of upstream dependencies.

When we had our first security audit last year, the results were quite reassuring as most of the findings were quite small, with the exception of a RCE in kustomize-controller, which speaks to the security improvements we have been investing on, and how they are resulting in better development practices.

Another recommendation from the auditors was to implement and follow a stricter and more elaborate RFC process, which is what we did. They also recommended we get in touch with other security teams or auditors for getting feedback on a refined and more general multi-tenancy proposal - which we also did as mentioned below.

What’s next for Flux

The way we shaped the Flux Roadmap for GA was

Feature party with Flux Legacy
Stable APIs (this was after the controller refactoring which consolidated functionality in fluxcd/pkg)
Straightforward multi-tenancy implementation
GA release

Here is the status quo regarding multi-tenancy:

Flux2 supports multi-tenancy, and users have been using it in production for some time now.

The documentation around the subject covers a bootstrap example to help users kick start their multi-tenancy deployments. And also how to implement control plane isolation with the multi-tenancy-lockdown.

What's next

In summary, the documentation needs expanding to better inform users around the security risks of multi-tenancy and the recommended deployment models for their specific isolation/security requirements.

There are proposed changes that would further improve Flux in multi-tenancy environments, by for example enabling tenants to share resources amongst themselves. Such changes must be progressed once the security impact of such changes have been assessed.

To help us get this right, we are engaging with the CNCF TAG Security. This is the upstream group where key contributors and experts of the CNCF Landscape assemble and define security best practices across all the individual Cloud Native projects. We are asking them for an independent security review and recommendations, particularly around multi-tenancy.

If you want to join the conversation, we are all ears. Please refer to the open RFC documents and have your say there. We definitely want to get this right for everyone.

In addition to that we are working hard to round up features, improve their performance, security and overall stability.

If you want to follow all the other GA related work, we explained how to do that here and if you would like to participate in any of the discussions, come and find us on Slack or in our regular meetings. We are always looking forward to growing Team Flux and the closer we get to GA, it’s getting even more important to have all voices heard.

Blog: April 2022 update

Tue, 03 May 2022 08:30:00 +0000

It’s the beginning of May 2022 - let’s recap together what happened in April - it has been a lot!

Update: Earlier versions of this post referred to the pre-KubeCon Bug Bash. Unfortunately we had to cancel our participation.

News in the Flux family

Latest Flux release series is 0.29

This is the latest and greatest, but before we get into the list of great features and improvements, let’s take a look at the breaking changes beforehand:

From this release on, the RUNTIME_NAMESPACE environment variable is no longer taken into account to configure the advertised HTTP/S address of the storage. Instead, variable substitution must be used, as described in the changelog entry for v0.5.2.
Use of file-based KubeConfig options are now permanently disabled (e.g. TLSClientConfig.CAFile, TLSClientConfig.KeyFile, TLSClientConfig.CertFile and BearerTokenFile). The drive behind the change was to discourage insecure practices of mounting Kubernetes tokens inside the controller’s container file system.
Use of TLSClientConfig.Insecure in KubeConfig file is disabled by default, but can be enabled at controller level with the flag --insecure-kubeconfig-tls.
Use of ExecProvider in KubeConfig file is now disabled by default, but can be enabled at controller level with the flag --insecure-kubeconfig-exec.

With that out of the way, here are the highlights of the release:

Notification Improvements

A new notification is now emitted to identify recovery from failures. It is triggered when a failed reconciliation is followed by a successful one.

In-memory cache for HelmRepository

An opt-in in-memory cache for HelmRepository that addresses issues where the index file is loaded and unmarshalled in concurrent reconciliation resulting in a heavy memory footprint. It can be configured using the flags: --helm-cache-max-size, --helm-cache-ttl, --helm-cache-purge-interval.

Configurable retention of Source Artifacts

Garbage Collection is enabled by default, and now its retention options are configurable with the flags: --artifact-retention-ttl (default: 60s) and --artifact-retention-records (default: 2). They define the minimum time to live and the maximum amount of artifacts to survive a collection.

Configurable Key Exchange Algorithms for SSH connections

Using the flag --ssh-kex-algos. Note this applies to the go-git gitImplementation or the libgit2 gitImplementation but only when Managed Transport is being used.

Configurable Exponential Back-off retry settings

With the new flags: --min-retry-delay (default: 750ms) and --max-retry-delay (default: 15min). Previously the defaults were set to 5ms and 1000s, which in some cases impaired the controller’s ability to self-heal (e.g. retrying failing SSH connections).

Experimental managed transport for libgit2 Git implementation

Now has self-healing capabilities, to recover from failure when long-running connections become stale.

SOPS refactored and optimized

Including various improvements and extended code coverage. Age identities are now imported once and reused multiple times, optimizing CPU and memory usage between decryption operations.

Helm chart directory loader improvements

Introduction of a secure directory loader which improves the handling of Helm charts paths.

For a more detailed list of changes in the series, please refer to the change logs of 0.29.0, 0.29.1, 0.29.2, 0.29.3, 0.29.4, and 0.29.5.

Flagger 1.20.0

This release comes with improvements to the AppMesh, Contour and Istio integrations.

Improvements

AppMesh: Add annotation to enable Envoy access logs #1156
Contour: Update the httproxy API and enable RetryOn #1164
Istio: Add destination port when port discovery and delegation are true #1145
Metrics: Add canary analysis result as Prometheus metrics #1148

Fixes

Fix canary rollback behaviour #1171
Shorten the metric analysis cycle after confirm promotion gate is open #1139
Fix unit of time in the Istio Grafana dashboard #1162
Fix the service toggle condition in the podinfo helm chart #1146

Flux Ecosystem

Flux Subsystem for Argo

In the latest release, we have added checkboxes to enable Flux Subsystem in the Argo CD UI. We also have a tutorial to use TF-controller with Flux Subsystem for Argo. With this you have an alternative option to Crossplane to manage infrastructure.

Terraform-controller

We have released TF-controller v0.9.4 which is a bug-fix release. We also added cloud cost estimation to our road map. Please feel free to give us feedback on how you would like this feature to be:

Issues here: https://github.com/weaveworks/tf-controller/issues, and
Discussions here: https://github.com/weaveworks/tf-controller/discussions

Weave GitOps

Weave GitOps is a powerful, open source extension to Flux, which provides insights into your deployments, and makes continuous delivery with GitOps easier to adopt and scale across your teams. You can easily install alongside an existing Flux setup, adding (or removing) Weave GitOps as a standard Helm resource.

The v0.8.0 release brings multi-namespace querying so you can see objects from across your cluster in the Web UI, several UI enhancements and bug fixes, as well as supporting the Source v1beta2 API - this breaking change means we now require Flux v0.29.0 or later.

Recent & Upcoming Events

It’s important to keep you up to date with new features and developments in Flux and provide simple ways to see our work in action and chat with our engineers.

Recent Events (ICYMI) 📺

Managing Thousands of Clusters and Their Workloads with Max Jonas Werner D2iQ uses Flux to automatically enable this experience in its products. Join Max for this hands-on session on multi-cluster management using GitOps.

CNCF on-demand webinar: Flux for Helm Users with Scott Rigby Scott Rigby, Flux & Helm Maintainer, takes you on a tour of Flux’s Helm Controller, shares the additional benefits Flux adds to Helm and then walks through a live demo of how to manage helm releases using Flux.

Women In GitOps Panel We celebrated international women’s day, GitOps Style. This event gathered female role models who innovate, challenge and embrace the world of GitOps. Inspirational women who have achieved great success within the sector and will share stories of their journey and explore the question why is it important to “Get on GitOps.”

Securing GitOps Debug Access with Flux, Pinniped, Dex, & GitHub with Leigh Capili In this live demo, Leigh will show how the incredibly flexible, open-source combo of Flux, Pinniped, and Dex can empower a team to leave a traceable solution during a production incident. He explores effective team debugging habits with Kubernetes and git.

Security: The Value of SBOMs with Dan Luhring (Anchore) During this session, Dan Luhring, OSS Engineering Manager at Anchore, dives into SBOMs - what they are, why you need them, some common use cases and how to get your pipeline ready for SBOM generation and verification using the Flux SBOM as an example.

OpenSource 101: WTF is GitOps & Why Should You Care? with Priyanka Ravi Pinky shares from personal experience why GitOps has been an essential part of achieving a best-in-class delivery and platform team, gives a brief overview of definitions, CNCF-based principles, and Flux’s capabilities: multi-tenancy, multi-cluster, (multi-everything!), for apps and infra, and more.

From Zero to GitOps Heroes with Mae Large, Russ Parmer, & Priyanka Ravi During this session Mae, Pinky, & Riss share key learnings from their early days of assessing GitOps as an idea and methodology to how it evolved into the de facto automated software change process in less than 1 year.

Upcoming Events 📆

We are happy to announce that we have a number of events coming up in May - tune in to learn more about Flux and GitOps best practices, get to know the team and join our community.

Flux Bug Scrub

The next dates are going to be:

KubeCon / CloudNativeCon Europe 2022 coming up

As every other project in the Cloud Natice space, we are very busy preparing everything for KubeCon / CloudNativeCon Europe 2022, which is going to be 16-20 May 2022 in Valencia, Spain (and virtual of course!).

We will post a separate announcement as soon as everything is confirmed, but we already want to inform you about what’s likely to happen, so you can plan accordingly or collaborate with us!

The Bug Bash

Unfortunately we will not be participating in the Bug Bash this KubeCon!

Despite earlier announcements claiming we would do this, we felt we could not do this well enough. If you were looking forward to this, we are sorry - but you know what: we still have the weekly Bug Scrub! Your weekly one-on-one mentoring to learn the ropes of working on Flux!

Monday, 16 May

13:00 - 17:00 (Room 2H - Event Center): Flux Project Meeting: We will kick off the Flux get-togethers and festivities with an in-person meeting for all Flux users, contributors, maintainers and generally interested folks. This will be an opportunity to get to know each other, have a chat, see what people’s interests are and to potentially start contributing. ( Sign up here.) Contact people on the ground are: Scott Rigby, Somtochi Onyekwere and Stefan Prodan.

Join Flux Maintainers Stefan Prodan, Somtochi Onyekwere & Scott Rigby for this Flux Project Meeting in-person at KubeCon EU on Monday, May 16 from 1pm - 5pm CEST

Click here to register ( here) for the Flux Project Meeting. Please note that you must be a KubeCon + CloudNativeCon Europe ( here) registrant in order to attend this meeting.

Details Flux Project Meeting Monday, May 16, 13:00 - 17:00 CEST Room 2H | Event Center

Space is limited Please note: we will not have any live streaming, recordings, or any virtual component available for this meeting.

Tuesday 17 May - GitOpsCon

Lots and lots of talks about GitOps in general and Flux in particular, here’s a short selection of what to look forward to:

What is GitOps and How to Get It Right - Dan Garfield (Codefresh); Chris Short (AWS) & Scott Rigby (Weaveworks) (9:00 - 9:35)
Hiding in Plain Sight - How Flux Decrypts Secrets - Somtochi Onyekwere (Weaveworks) (11:05 - 11:15)
Taming the Thundering Gitops Herd with Update Policies - Joaquim Rocha & Iago López Galeiras (Microsoft) (11:35 - 11:45)
GitOps and Progressive Delivery with Flagger, Istio and Flux - Marco Amador (Anova) (13:20-13:30)
Creating A Landlord for Multi-tenant K8s Using Flux, Gatekeeper, Helm, and Friends - Michael Irwin (Docker) (13:35-14:05)
GitOps, A Slightly Realistic Situation on Kubernetes with Flux - Laurent Grangeau (Google) & Ludovic Piot (theGarageBandOfIT) (14:10 - 14:40)
Solving Environment Promotion with Flux - Sam Tavakoli & Adelina Simion (Form3) (14:10 - 14:40)
Managing Thousands of Clusters and Their Workloads with Flux - Max Jonas Werner (D2iQ) (14:55 - 15:25)
Crossing the Divide: How GitOps Brought AppDev & Platform Teams Together! - Russ Palmer (State Farm) & Priyanka ‘Pinky’ Ravi (Weaveworks) (15.30 - 16:00)
GitOps Everything!? We Sure Can!, AppsFlyer (15:30 - 16:00)
Lightning Talk: Addressing Log4Shell with Software Supply Chains - Duane DeCapite (VMware) (18:04 - 18:09)

Wednesday 18 May - Friday May 20 - KubeCon

On top of that, here is a list of talks, workshops and sessions during those days:

Wed 18: Flux Security Deep Dive - Stefan Prodan (Weaveworks) (11:55 - 12:30)
Wed 18: Intro to Kubernetes, GitOps, and Observability Hands-On Tutorial - Johee Chung (Microsoft) & Tiffany Wang (Weaveworks) (11:00 - 12:30)
Wed 18: Flux Bug Scrub - Kingdon Barrett (13:00 - 14:00)
Wed 18: A New Generation of Trusted GitOps for Mixed K8s and Non-K8s End Users - Alexis & Vasu Chandrasekhara (SAP) (15:25 - 16:00)
Thu 19: GitOps to Automate the Setup, Management and Extension a K8s Cluster - Kim Schlesinger (DigitalOcean) (11:00 - 12:30)
Thu 19: Flux Project Office Hour - Paulo Gomes (Weaveworks) (13:30 - 14:15)
Fri 20: Observing Fastly’s Network at Scale Thanks to K8s and the Strimzi Operator - Fernando Crespo & Daniel Caballero, (Fastly) (11:00 - 11:35)
Fri 20: Simplifying Service Mesh Operations with Flux and Flagger - Mitch Connors (Google) & Stefan Prodan (Weaveworks) (14:55 - 15:30)

Please note: all of the above might be subject to change. Please double-check the schedule beforehand. Please reach out to Vanessa Abankwah or Daniel Holbach on Slack if you have questions or would like to participate in any of the above.

We very much look forward to seeing you there!

GitOps Days 2022

GitOps Days 2022 is a free 2-day online event on June 8-9, 2022.

This is THE event for your GitOps journey! Getting started? Taking GitOps to the next level? We’ll cover all of the steps for your success!

The event will run from 9:00 am PT to ~3:00 pm PT each day as a free online event.

✨✨ Register now to reserve your spot to receive updates to the schedule and speakers. ✨✨

Join the conversation! Chat with the speakers and other attendees! Invite yourself at https://slack.weave.works and hang out with us at #gitopsdays

What to expect?

Talks and tutorials on how to get started with Kubernetes and GitOps
Talks from Flux users about their use cases
How to do GitOps securely
Platforms that offer GitOps: Microsoft Arc Kubernetes, AWS Anywhere, Weave GitOps, D2iQ Kubernetes Platform, and more! all using Flux!
Flux in the CNCF and the GitOps Ecosystem
Flux support and Integrations: Flux + Helm, Terraform, HashiCorp Vault, Jenkins, OpenShift, Visual Studio Code, and much much more!
Technical deep dives with Flux maintainers
Music from DJ Desired State 🎶

In other news

People writing/talking about Flux

Manage Kubernetes Secrets for Flux with HashiCorp Vault

Rosemary Wang from HashiCorp wrote a great blog post about how to manage Kubernetes Secrets for Flux with HashiCorp Vault. The how-to is nicely written with a lot of detail and will take you through the steps to configure the Secrets Store CSI driver with HashiCorp Vault to securely inject secrets into Flux or other GitOps tools on Kubernetes.

We are looking forward to more collaboration together!

Full GitOps Tutorial: Getting started with Flux

This video is great for everyone who gets started, but also everyone who enjoys a story well-told.

In this video, Anais Urlichs covers

What is GitOps and how does Flux work
Flux installation
Managing Helm Charts with Flux
Managing Kubernetes Manifests with Flux
Setting up alerts with Flux

Anais also sat down wrote this all up in blog-post from.

How To Apply GitOps To Everything Using Crossplane And Flux

Viktor Farcic has done it again - check out this great video where he shows how to leverage the extensibility of Crossplane and Flux features to apply GitOps not only to applications running in Kubernetes but to everything (infrastructure, services, applications running anywhere, etc.)

Encrypted gitops secrets with flux and age

Major Hayden wrote a nice article about how to get encrypted gitops secrets with flux and age right.

Here you will learn how to store encrypted kubernetes secrets safely in your GitOps repository with easy-to-use age encryption. 🔐

Basic authentication with Traefik on kubernetes

Another post from Major Hayden! This time about Basic authentication with Traefik on kubernetes.

It’s nicely detailed and will take you through all the steps to keep prying eyes away from your sites behind Traefik with basic authentication. 🛃

Automated Canary Deployments with Rancher Fleet and Flagger

In this video, Lukonde Mwila will demonstrate how to execute automated canary deployments with Rancher Fleet and Flagger.

News from the Website and our Docs

Flux Adopters shout-out

We are very pleased to announce that the following adopters of Flux have come forward and added themselves to our website: Stackspin, Maersk and Rungway.

More docs and website news

We are constantly improving our documentation and website - here are a couple of small things we landed recently.

If you always wanted to join Team Flux and weren’t quite sure how, please read our blog post Contributing to Flux and say Hi on Slack!
Many mobile UI fixes!
Add flux-subsystem-argo/flamingo and weaveworks/vscode-gitops-tools to the Flux Ecosystem page.
New videos under Flux Resources! 😍
Various docs fixes.
And here is a big one: we moved all docs from https://flagger.app into https://fluxcd.io/flagger - this is part of a bigger move to subsume all of our documentation and web-presence into one place, so we won’t have to maintain too many pieces of infrastructure.
This has been on our to-do list since Flux became a CNCF Incubating project. Now that we are going for Graduation, we finally got around to doing it.

Thanks a lot to these folks who contributed to docs and website: Ihor Sychevskyi, Kingdon Barrett, Stefan Prodan, Endre Czirbesz, Maarten de Waard and Patrick Rodies.

In particular we would like to thank Ihor Sychevskyi who recently took on fixing small UI glitches all over the place - especially on mobile the site should work a lot better now!

Flux Project Facts

We are very proud of what we put together, here we want to reiterate some Flux facts - they are sort of our mission statement with Flux.

🤝 Flux provides GitOps for both apps or infrastructure. Flux and Flagger deploy apps with canaries, feature flags, and A/B rollouts. Flux can also manage any Kubernetes resource. Infrastructure and workload dependency management is built-in.
🤖 Just push to Git and Flux does the rest. Flux enables application deployment (CD) and (with the help of Flagger) progressive delivery (PD) through automatic reconciliation. Flux can even push back to Git for you with automated container image updates to Git (image scanning and patching).
🔩 Flux works with your existing tools: Flux works with your Git providers (GitHub, GitLab, Bitbucket, can even use s3-compatible buckets as a source), all major container registries, and all CI workflow providers.
🔒 Flux is designed with security in mind: Pull vs. Push, least amount of privileges, adherence to Kubernetes security policies and tight integration with security tools and best-practices. Read more about our security considerations.
☸️ Flux works with any Kubernetes and all common Kubernetes tooling: Kustomize, Helm, RBAC, and policy-driven validation (OPA, Kyverno, admission controllers) so it simply falls into place.
🤹 Flux does Multi-Tenancy (and “Multi-everything”): Flux uses true Kubernetes RBAC via impersonation and supports multiple Git repositories. Multi-cluster infrastructure and apps work out of the box with Cluster API: Flux can use one Kubernetes cluster to manage apps in either the same or other clusters, spin up additional clusters themselves, and manage clusters including lifecycle and fleets.
📞 Flux alerts and notifies: Flux provides health assessments, alerting to external systems and external events handling. Just “git push”, and get notified on Slack and other chat systems.
👍 Users trust Flux: Flux is a CNCF Incubating project and was categorised as "Adopt" on the CNCF CI/CD Tech Radar (alongside Helm).
💖 Flux has a lovely community that is very easy to work with! We welcome contributors of any kind. The components of Flux are on Kubernetes core controller-runtime, so anyone can contribute and its functionality can be extended very easily.

Over and out

If you like what you read and would like to get involved, here are a few good ways to do that:

Join our upcoming dev meetings on 2022-05-05 or 2022-05-11.
Talk to us in the #flux channel on CNCF Slack
Join the planning discussions
And if you are completely new to Flux, take a look at our Get Started guide and give us feedback
Social media: Follow Flux on Twitter, join the discussion in the Flux LinkedIn group.

We are looking forward to working with you.

Blog: Contributing to Flux

Wed, 13 Apr 2022 14:30:00 +0000

📈 Our Community is growing

We are very pleased to see that our community is growing, whichever metric you apply. Number of members in Slack or mailing list, newly added adopters or projects in our ecosystem - all numbers are going up. Even the length of our monthly updates is growing every month.

As many asked in the past months how to get involved in the Flux community, let’s walk you through some steps to join Team Flux!

Update: Earlier versions of this post referred to the pre-KubeCon Bug Bash. Unfortunately we had to cancel our participation.

First things first

The first thing to do is get in touch with us! The communication section on our website lists all the ways, but let’s drill into them in more detail.

A good way to reach us and talk to us in real-time is on CNCF Slack. Get an invite here, then proceed to the #flux channel.
We have public meetings on a weekly basis to talk about upcoming work and direction of the project. They are open to anyone and we very much want to welcome you to drop by, say Hi and introduce yourself. Here are the instructions.
To generally stay up to date, you also might want to check out our blog, our Twitter and/or Flux group on LinkedIn.

Please say Hi, let us know what you are interested in and how we can help you get started. From hereon it depends on what you would like to do. Let’s take a look at different ways to get involved.

Interacting with GitHub

All code for the Flux project is available under the fluxcd organisation in GitHub. So if you are interested in contributing to Flux, it is a good idea to learn how things are organised and potentially get involved with responding to GitHub issues.

The Bug Scrub

For 9 months now, Kingdon Barrett has been organising the Flux Bug Scrub:

For us, a great way to get started, is to learn more about Flux through direct experience, when e.g. trying to reproduce issues reported by other Flux users, and the general business of chopping wood and carrying water.

Each week we will have one or more contributors with triage access in attendance, and attempt to review each issue in the target group. (The targets will vary by week, or by event date.) A key goal is to figure out if a given issue can be reproduced, and report our findings. Some issues don't provide enough info to get us all the way to reproduction, so we may ask more questions and wait for the submitter to respond.

The great thing about this is: we run this on a weekly basis, so every week you have the opportunity to learn from experienced Flux maintainers in a friendly setting, so all you need to do is check our calendar and turn up.

The Bug Bash

Unfortunately we will not be participating in the Bug Bash this KubeCon!

Participating in Flux Development

If you are interested in Flux development, great! We are always looking for new people to get started. Come and talk to us on Slack and in our meetings as suggested above - that makes collaboration usually a lot easier.

All of Flux Development is happening in the open, the priorities for the next releases are documented publicly and we continuously tag issues as “good first issues” if you are looking for suggestions.

Our contributor docs will likely be helpful as well!

If you are interested in Security, you might have seen how much we have done to improve security lately. We are very interested in doing more - and are always open to new ideas!

Meeting in person

As stated above KubeCon / CloudNativeCon Europe 2022 is just around the corner. It is going to happen in Valencia, Spain 16 - 20 May and we will have many Flux maintainers and contributors at our booth and presenting talks and be available to talk to in other events. This is your chance to meet, have a chat and get to know the team.

There will be a separate announcement, but here already is a short summary of what to look forward to.

Monday, 16 May

13:00 - 17:00 (Room 2H - Event Center): Flux Project Meeting: We will kick off the Flux get-togethers and festivities with an in-person meeting for all Flux users, contributors, maintainers and generally interested folks. This will be an opportunity to get to know each other, have a chat, see what people’s interests are and to potentially start contributing. ( Sign up here.) Contact people on the ground are: Somtochi Onyekwere and Scott Rigby.

Tuesday 17 May - GitOpsCon

Lots and lots of talks about GitOps in general and Flux in particular, here’s a short selection of what to look forward to:

What is GitOps and How to Get It Right - Dan Garfield (Codefresh); Chris Short (AWS) & Scott Rigby (Weaveworks) (9:00 - 9:35)
Hiding in Plain Sight - How Flux Decrypts Secrets - Somtochi Onyekwere (Weaveworks) (11:05 - 11:15)
Taming the Thundering Gitops Herd with Update Policies - Joaquim Rocha & Iago López Galeiras (Microsoft) (11:35 - 11:45)
GitOps and Progressive Delivery with Flagger, Istio and Flux - Marco Amador (Anova) (13:20-13:30)
Creating A Landlord for Multi-tenant K8s Using Flux, Gatekeeper, Helm, and Friends - Michael Irwin (Docker) (13:35-14:05)
GitOps, A Slightly Realistic Situation on Kubernetes with Flux - Laurent Grangeau (Google) & Ludovic Piot (theGarageBandOfIT) (14:10 - 14:40)
Solving Environment Promotion with Flux - Sam Tavakoli & Adelina Simion (Form3) (14:10 - 14:40)
Managing Thousands of Clusters and Their Workloads with Flux - Max Jonas Werner (D2iQ) (14:55 - 15:25)
Crossing the Divide: How GitOps Brought AppDev & Platform Teams Together! - Russ Palmer (State Farm) & Priyanka ‘Pinky’ Ravi (Weaveworks) (15.30 - 16:00)
GitOps Everything!? We Sure Can!, AppsFlyer (15:30 - 16:00)
Lightning Talk: Addressing Log4Shell with Software Supply Chains - Duane DeCapite (VMware) (18:04 - 18:09)

Wednesday 18 May - Friday May 20 - KubeCon

On top of that, here is a list of talks, workshops and sessions during those days:

Wed 18: Flux Security Deep Dive - Stefan Prodan (Weaveworks) (11:55 - 12:30)
Wed 18: Intro to Kubernetes, GitOps, and Observability Hands-On Tutorial - Johee Chung (Microsoft) & Tiffany Wang (Weaveworks) (11:00 - 12:30)
Wed 18: A New Generation of Trusted GitOps for Mixed K8s and Non-K8s End Users - Alexis & Vasu Chandrasekhara (SAP) (15:25 - 16:00)
Thu 19: GitOps to Automate the Setup, Management and Extension a K8s Cluster - Kim Schlesinger (DigitalOcean) (11:00 - 12:30)
Thu 19: Flux Project Office Hour - Paulo Gomes (Weaveworks) (13:30 - 14:15)
Fri 20: Observing Fastly’s Network at Scale Thanks to K8s and the Strimzi Operator - Fernando Crespo & Daniel Caballero, (Fastly) (11:00 - 11:35)
Fri 20: Simplifying Service Mesh Operations with Flux and Flagger - Mitch Connors (Google) & Stefan Prodan (Weaveworks) (14:55 - 15:30)

If you can’t make it to KubeCon in person, we’ve got you covered as well. In addition to all of this, we are going to have a virtual booth with talks - will announce the schedule very soon.

Putting Flux to the Test

If you have already gathered some experience with Flux, you should be in a great position to help us further and move the project forward a lot. The term “Testing” might sound a bit dull to you, but there’s a lot that falls into this category, which is super important to us. You could for example:

verify bugs and make minimal test cases to help fix those issues
refine and expand on the existing unit and end-to-end tests
try Flux in new environments and figure out if it works, and if not, what it would take to make it work

As you can gather by now already - we’d love to join this effort!

Flux Advocacy

We all love GitOps and we have heard from many who use Flux that they integrated it into their machinery and afterwards “forgot about it” since it was “just doing its job”. That’s how we like it.

Still it is important to let people know and educate communities around you. So if you like blogging or giving talks, let us know and we can help promote your work, or collaborate on events or content.

Help answering questions

We receive questions and requests from our user base on Slack, also in Github Discussions and other places like e.g. Reddit as well. If you enjoy helping people, come and join the team. We are a friendly bunch and it can be quite satisfying to know you just helped somebody fix an issue, and potentially provided an onramp for them to join the community.

As part of this effort, we are always looking at improving our documentation, FAQ and how-tos. So making some of the learnt knowledge more generally available is of great use as well.

Flux Documentation and Website

https://fluxcd.io for many is the first impression of the project and documentation and blog posts the next interactions. We really want to make the project shine there, so if you have a knack for writing and/or organising content, come and talk to us. Web wizards are very welcome too! 💖

If you check out some of our last monthly update posts, you will get an idea of what we have been up to and how important it is to us.

Sorry, got no time - but still want to support

If you don’t have time for any of the above, that is entirely fine and understandable.

One good way how you can still help out, is by adding yourself to the Flux Adopters page, if your organisation is using it. We love hearing from folks using Flux projects. Contributors really love seeing where their software is being used and it gives our community a good idea of how wide-spread adoption really is.

The same goes for the Flux Ecosystem page. If you integrate with Flux, we want to hear from you!

⭐ And please star us on GitHub as well!

Conclusion

You might have heard the saying “It takes a village” before, and it is the same for Team Flux. The success of our project depends on many different people with many different skills. We love meeting you, we love helping you get started - just reach out and we are looking forward to meet you!

Talk to us

We love feedback, questions and ideas, so please let us know your personal use-cases today. Ask us if you have any questions and please

join our upcoming dev meetings
find us in the #flux channel on CNCF Slack
add yourself as an adopter if you haven’t already

See you around!

Blog: March 2022 Update

Mon, 04 Apr 2022 14:30:00 +0000

It’s the beginning of April 2022 - let’s recap together what happened in March - it has been a lot!

Update: Earlier versions of this post referred to the pre-KubeCon Bug Bash. Unfortunately we had to cancel our participation.

News in the Flux family

Source API getting more mature in Flux 0.28

The latest release of Flux is 0.28. One big focus was to graduate its Source API to v1beta2.

🤖 To upgrade and fully benefit from this, please follow the upgrade instructions.

This work had been a long time in the making, partly because of a larger refactoring effort, which we had reported about previously. The idea was to abstract reusable components and functionality into the fluxcd/pkg repository. While this is an ongoing effort, we are very happy with what we have learned so far and are convinced that we will get better test coverage this way and are providing external projects with a solid foundation to build on as well.

In this release we added new features and improvements across the board, here’s a quick list of our highlights:

Add the Git commit message (first 50 characters) to the events and alerts issued by GitRepository sources.
Improve performance for Helm repository index and chart download operations.
Improve observability for the Git, Helm and Bucket resources by providing explicit status conditions which conform to the Kubernetes kstatus conventions.
A new annotation ( kustomize.toolkit.fluxcd.io/ssa: merge) is available for allowing Flux to patch cluster addons such as CoreDNS.
Add Azure Blob Storage native support to Flux Bucket sources.
Add support for decrypting secrets with SOPS and Azure Key Vault on multi-tenant clusters.
Retry the Git operations on conflict errors to allow running bootstrap in-parallel for multiple clusters that target the same repository.
Add a new transport for libgit2 for improved reliability (experimental). We wrote about this in our last blog post as well.

Latest Flagger release comes with Gateway API support

We blogged about this separately already as it is such a big achievement for Team Flagger. With its recent 1.19 release, Flagger brings Gateway API support. This means native Progressive Delivery for all providers supported by the Gateway API project within Kubernetes. Be sure to check out the blog post to find out how to integrate this into your setups.

The Flux community is happy and proud that Flagger is part of our effort to bring GitOps solutions to the world.

Flux “Maintainers’ Focus” Project Board

Being clear about our priorities in Flux development was always important to us as a project. Discussing this regularly in weekly meetings to be able to get everyone’s input was one measure to do this. Updating our roadmap regularly was another. Monthly updates posted on all of Flux channels yet another.

As the development team around Flux grew and we had more work to be coordinated across Flux controllers with e.g. teams at cloud providers, bigger pieces of code refactoring, etc, we are now pleased to use GitHub’s new project boards for having a “Maintainer’s Focus” page which shows what’s bookmarked for the upcoming Flux releases - this might also be a good resource to check if you would like to get involved with Flux development and help out with one of the next releases.

On our way to Flux GA

A particular focus in our project management is GA, the big target we have been following ever since we started the rewrite of Flux. As you can see on the Flux Roadmap, we closed out the vast majority of items and last year we already announced that the Flux APIs will be stable from now on. So what’s left is to finish the refactoring for the remaining controllers, complete some parts of the documentation and some general tidying up. If you want more detail, or would like to help us to achieve this big milestone, you can follow the work here.

Security news

The latest addition to our blog series about Flux Security was a post called «Using Pod Security Standard "restricted"». Go check it out, as it you will learn more about Kubernetes’ pod security standard, seccomp and how we apply this in Flux to keep you safe.

The already mentioned blog post about our tight integration with Git APIs could also be of interest, as we discuss upcoming plans for integrating sha256 hash support.

Flux Ecosystem

What makes Flux great is its ecosystem. Tools and services which integrate seamlessly because that’s how the Cloud Native ecosystem works. We are celebrating all of this on the Flux Ecosystem page. (Please add yourself if your tool or integration isn’t listed yet.)

Renovate

Here are a couple of newcomers. Firstly, there’s Renovate, which is an Open Source tool to automate:

Detecting dependencies in a repository (Open Source and private/closed source)
Checking if there are dependency updates
Creating commits and Merge/Pull Requests to update dependencies
Showing the release notes

We are very pleased that the team at Renovate added a manager to integrate with Flux.

GitOps Visual Studio Code Extension

The Weaveworks GitOps Extension provides an intuitive way to manage, troubleshoot and operate your Kubernetes environment following the GitOps operating model, accelerating your development lifecycle and simplifying your continuous delivery pipelines.

Weaveworks GitOps Extension integrates with Kubernetes Tools, kubectl and flux for a consolidated and tightly integrated user experience.

And also this one about the Flux Gitops extension for VS Code: https://t.co/7VlRRyelOl #azure #gitops #flux #vscode
— Geert Baeke (@GeertBaeke) March 18, 2022

🚧 This extension is under active development and currently available as an alpha product.

Flux Subsystem for Argo

FSA (aka Flamingo) is the Flux Subsystem for Argo. Its container image can be used as a drop-in replacement for the equivalent ArgoCD version to visualise, and manage Flux workloads, alongside ArgoCD.

How does it work?

🚧 This project is currently available as a technology preview.

Terraform-controller

In some of our last issues we already reported about the terraform-controller hitting the streets. It’s a Flux controller which reconciles Terraform resources in the GitOps way. We received a short report from the team regarding their achievements of the first quarter of the year:

TF-controller v0.9.3 is considered the most stable release to date.
We reached 200 stars on GitHub, now at 211.
It's been 45 releases so far.
We re-factored it to the Controller/Runner architecture.
Standing on the shoulders of our giants (Flux), we successfully implemented the multi-tenancy feature in 2 months.
We cleared all Q1 roadmap with 68.2% test coverage.
We started seeing its adoption in public, from a Helm Controller user, for example.
We got its first promo video.
Chanwit Kaewkasi, Piaras Hoban and Tom Huang are the core team around it now!

Weave GitOps Core

The team around Weave GitOps has been busy and would love to hear your feedback. If you haven’t heard about it just yet, its GitHub says:

Weave GitOps enables an effective GitOps workflow for continuous delivery of applications into Kubernetes clusters. It is based on CNCF Flux, a leading GitOps engine.

The Flux community particularly loved the last sentence.

Getting started with it is very straight-forward. Please take up the offer of them and give feedback, they are building a very nice tool based on Flux!

Recent & Upcoming Events

It’s important to keep you up to date with new features and developments in Flux and provide simple ways to see our work in action and chat with our engineers.

Recent Events (ICYMI) 📺

Flux Maintainer Stefan Prodan at our friends of Tanzu Tuesday: Mar 15: Tanzu Tuesdays #89: GitOps with Flux on Kubernetes with Stefan Prodan
Flux contributor and VMware Tanzu Advocate Leigh Capili talking about a subject close to the heart of many - security and debugging 💖: Mar 16: Securing GitOps Debug Access with Flux, Pinniped, Dex, & GitHub - Leigh Capili
Weaveworks’ DX Engineer Priyanka Pinky and Anchore’s OSS Lead Dan Luhring dive deeper into security subjects here: Mar 24: Security: The Value of SBOMs with Dan Luhring & Priyanka Ravi
Want to hear from professionals who brought GitOps to 7000 devs in a heavily regulated industry? Mae Large, Pinky & Russ Palmer reflect on their work together: Mar 30: From Zero to GitOps Heros with Mae Large, Russ Parmer, Priyanka Ravi

Upcoming Events 📆

We are happy to announce that we have a number of events coming up in April - tune in to learn more about Flux and GitOps best practices, get to know the team and join our community.

April 7: GitOps with Flux on AKS with Kingdon Barrett & Jonathan Innis

Introduction to GitOps & Flux
You may have heard the term GitOps - it has become a bit of a buzzword, but it’s so much more! The benefits of GitOps are real - bringing better security, reliability, velocity and more! And the project that started it all was Flux - a CNCF Incubating project developed and later donated by Weaveworks (the GitOps company who coined the term).

GitOps in Microsoft Azure with Flux
To provide Kubernetes admins and app developers with the latest tooling for managing configuration and application deployment, Azure enables GitOps with Flux. In this session Jonathan Innis, Software Engineer II at Microsoft, will live demo how CNCF Flux is enabled in Azure Arc enabled Kubernetes and Azure Kubernetes Services and also give a sneak peek at implementation of Flux.

April 13: GitOps: Core Concepts & Ways of Structuring Your Repos

Whether you’re new to GitOps or a seasoned pro, this talk is for you! We'll start with the basics of how/where to get started, and then dive into one of the most asked GitOps questions: how to structure your repository!

During this talk, Scott & Pinky will review the Core Concepts of Flux including Git Sources, Reconciliation, Helm Releases, Kustomization, and Bootstrapping, to get you ramped up with how to think with a GitOps mindset! Then they’ll dive into and discuss considerations for and demo ways of structuring your repositories: monorepo, repo per environment, repo per team, or repo per app.

April 20: DoK Talks #131: Flux for Helm Users by Scott Rigby

Welcome Helm users! CNCF Flux has a best-in-class way to use Helm according to GitOps principles. For you, that means improved security, reliability, and velocity - no more being on the pager on the weekends or having painful troubleshooting or rollback when things go wrong.

Built on Kubernetes controller-runtime, Flux’s Helm Controller is an example of a mature software agent that uses Helm’s SDK to full effect.

Flux’s biggest addition to Helm is a structured declaration layer for your releases that automatically gets reconciled to your cluster based on your configured rules:

⭐️ The Helm client commands let you imperatively do things
⭐️ Flux Helm Custom Resources let you declare what you want the Helm SDK to do automatically.

In addition, Scott will show how to use Helm Charts to run reliable stateful workloads.

April 27: Reconcile Terraform Resources the GitOps Way with Jose Talavera

Some organisations depend heavily on their Terraform scripts because they are using multiple providers, have built wrappers around those providers, and might even be deploying their application code along with Terraform. Additionally, GitOps is in every IT roadmap, but unfortunately Terraform doesn’t have an easy way to reconcile its resources. This means that teams won't notice a sudden change in the running environment often with critical consequences.

What if teams could ensure that what they defined in the Terraform HCL code is what is always running and available? Flux can continuously look for changes on your Terraform resources and do reconciliation with the desired state. You can rest easy knowing that your deployments are always up to date with your desired state. This enables you to take advantage of all the benefits of GitOps: streamlined and secure deployments, quicker time to market, and more time to concentrate on app development!

Jose provides an in-depth look at TF-controller, a Flux-based controller to reconcile your Terraform resources the GitOps Way. Jose will share insights on the many benefits of TF-Controller, then demo a common use case implementation.

Flux Bug Scrub

The next dates are going to be:

KubeCon / CloudNativeCon Europe 2022 coming up

As every other project in the Cloud Native space, we are very busy preparing everything for KubeCon / CloudNativeCon Europe 2022, which is going to be 16-20 May 2022 in Valencia, Spain (and virtual of course!).

We will post a separate announcement as soon as everything is confirmed, but we already want to inform you about what’s likely to happen, so you can plan accordingly or collaborate with us!

The Bug Bash

Unfortunately we will not be participating in the Bug Bash this KubeCon!

Monday, 16 May

13:00 - 17:00 (Room 2H - Event Center): Flux Project Meeting: We will kick off the Flux get-togethers and festivities with an in-person meeting for all Flux users, contributors, maintainers and generally interested folks. This will be an opportunity to get to know each other, have a chat, see what people’s interests are and to potentially start contributing. ( Sign up here.) Contact people on the ground are: Somtochi Onyekwere and Scott Rigby.

Tuesday 17 May - GitOpsCon

Lots and lots of talks about GitOps in general and Flux in particular, here’s a short selection of what to look forward to:

What is GitOps and How to Get It Right - Dan Garfield (Codefresh); Chris Short (AWS) & Scott Rigby (Weaveworks) (9:00 - 9:35)
Hiding in Plain Sight - How Flux Decrypts Secrets - Somtochi Onyekwere (Weaveworks) (11:05 - 11:15)
Taming the Thundering Gitops Herd with Update Policies - Joaquim Rocha & Iago López Galeiras (Microsoft) (11:35 - 11:45)
GitOps and Progressive Delivery with Flagger, Istio and Flux - Marco Amador (Anova) (13:20-13:30)
Creating A Landlord for Multi-tenant K8s Using Flux, Gatekeeper, Helm, and Friends - Michael Irwin (Docker) (13:35-14:05)
GitOps, A Slightly Realistic Situation on Kubernetes with Flux - Laurent Grangeau (Google) & Ludovic Piot (theGarageBandOfIT) (14:10 - 14:40)
Solving Environment Promotion with Flux - Sam Tavakoli & Adelina Simion (Form3) (14:10 - 14:40)
Managing Thousands of Clusters and Their Workloads with Flux - Max Jonas Werner (D2iQ) (14:55 - 15:25)
Crossing the Divide: How GitOps Brought AppDev & Platform Teams Together! - Russ Palmer (State Farm) & Priyanka ‘Pinky’ Ravi (Weaveworks) (15.30 - 16:00)
GitOps Everything!? We Sure Can!, AppsFlyer (15:30 - 16:00)
Lightning Talk: Addressing Log4Shell with Software Supply Chains - Duane DeCapite (VMware) (18:04 - 18:09)

Wednesday 18 May - Friday May 20 - KubeCon

On top of that, here is a list of talks, workshops and sessions during those days:

Wed 18: Flux Security Deep Dive - Stefan Prodan (Weaveworks) (11:55 - 12:30)
Wed 18: Intro to Kubernetes, GitOps, and Observability Hands-On Tutorial - Johee Chung (Microsoft) & Tiffany Wang (Weaveworks) (11:00 - 12:30)
Wed 18: A New Generation of Trusted GitOps for Mixed K8s and Non-K8s End Users - Alexis & Vasu Chandrasekhara (SAP) (15:25 - 16:00)
Thu 19: GitOps to Automate the Setup, Management and Extension a K8s Cluster - Kim Schlesinger (DigitalOcean) (11:00 - 12:30)
Thu 19: Flux Project Office Hour - Paulo Gomes (Weaveworks) (13:30 - 14:15)
Fri 20: Observing Fastly’s Network at Scale Thanks to K8s and the Strimzi Operator - Fernando Crespo & Daniel Caballero, (Fastly) (11:00 - 11:35)
Fri 20: Simplifying Service Mesh Operations with Flux and Flagger - Mitch Connors (Google) & Stefan Prodan (Weaveworks) (14:55 - 15:30)

We very much look forward to seeing you there!

In other news

People writing/talking about Flux

Stefan Prodan on Flux, Flagger, and the Operator Pattern Applied to Non-Clustered Resources

In this podcast, Wesley Reisz talks to Stefan Prodan about Flux and Flagger–two tools built on top of Flux’s GitOps Toolkit. After discussing some of the architectural differences between Flux v1 and v2 and discussing some of the GitOps toolkit use cases, the two discuss the operator pattern on Kubernetes. They specifically spend time talking about the operator pattern, why developers may opt to build API’s on top of Kubernetes, and how the pattern can be used on non-clusters resources. The podcast wraps with a discussion on the work being down towards Flux v2’s push to GA.

A deep dive to Canary Deployments with Flagger, NGINX and Linkerd on Kubernetes

Chen wrote up a nice tutorial on using Flagger and has this to say about Flagger itself:

*Flagger is a progressive delivery tool that automates the release process for apps on Kubernetes. It can gradually shift traffic to the new version while measuring metrics and running conformance tests.

I prefer flagger because of two main points:

It integrates natively: it watches Deployment resources, while Argo uses its own CRD Rollout

It is highly extensible and comes with batteries included: it provides a load-tester to run basic, or complex scenarios*

GitOpsify Cloud Infrastructure with Crossplane and Flux

Check out this article by Piotr who dives into how to automate the provisioning of cloud resources via Crossplane and combine it with GitOps practices. At the end of it, you will have stopped using kubectl to manage resources, but rather delegate this to Flux using Git. GitOps for the win!

CNCF Live Webinar: From Pipelines to Supply Chains: Level up with Supply Chain Choreography

Cora Iberkleid and David Espejo at VMware talk about Cartographer. They say: The Kubernetes ecosystem has a rich set of solutions for various stages of CI/CD. Tools like Flux, Tekton, kpack, Knative, ArgoCD, and more each enable big steps forward in establishing a modern path to production. And yet, the teams and organizations that adopt these tools still struggle with complex, DIY snowflake pipelines. The challenge can be creating and maintaining imperative scripts; orchestrating the flow of information between tools; driving reusability; adopting GitOps practices; and enabling proper separation of concerns.

News from the Website and our Docs

Flux Adopters shout-out

We are very pleased to announce that the following adopters of Flux have come forward and added themselves to our website: Netrics, Syntasso, EmploymentHero, Anchore and Giant Swarm.

More docs and website news

We are constantly improving our documentation and website - here are a couple of small things we landed recently.

Documentation:

This was a big effort: The Source API documentation has been refactored to be more user-friendly. See the v1beta2 specification for: Git Repositories, Buckets and Helm Repositories.
Flux from End-to-End: This was a big part of work as well. It describes the flow of data through Flux, from End to End.
Cheatsheets: Various configurations of Flux controllers at install time are now available as a bootstrap cheatsheet.
We added new FAQ entries.
We added new resources to the site.

In terms of documentation, we are working on a bigger piece of navigation and information architecture refactoring. This was pointed out to us as piece of feedback from the CNCF TechDocs team. As the Flux project has grown over time, we appreciate this opportunity to restructure our docs to make them as easy to find as possible. Your feedback matters here, so if you could leave us a note with your impression on this PR, we would love to hear from you.

And finally on our blog, we added a tag cloud and a note to blog posts that are older than a year - we also typed up how to blog.

Thanks a lot to these folks who contributed to docs and website: Kingdon Barrett, Stefan Prodan, Stacey Potter, Hidde Beydals, Sebastian Bernheim, Ihor Sychevskyi, Colin Humphreys, Filip Sequeira, Jan Lauber, Marcus Noble, Morgan Christiansson, Satish Kumar Kardarkarai Mani, Tom Huang and Nguyen Duc Toan.

Flux Project Facts

We are very proud of what we have put together. We want to reiterate some Flux facts - they are sort of our mission statement with Flux.

🤝 Flux provides GitOps for both apps or infrastructure. Flux and Flagger deploy apps with canaries, feature flags, and A/B rollouts. Flux can also manage any Kubernetes resource. Infrastructure and workload dependency management is built-in.
🤖 Just push to Git and Flux does the rest. Flux enables application deployment (CD) and (with the help of Flagger) progressive delivery (PD) through automatic reconciliation. Flux can even push back to Git for you with automated container image updates to Git (image scanning and patching).
🔩 Flux works with your existing tools: Flux works with your Git providers (GitHub, GitLab, Bitbucket, can even use s3-compatible buckets as a source), all major container registries, and all CI workflow providers.
🔒 Flux is designed with security in mind: Pull vs. Push, least amount of privileges, adherence to Kubernetes security policies and tight integration with security tools and best-practices. Read more about our security considerations.
☸️ Flux works with any Kubernetes and all common Kubernetes tooling: Kustomize, Helm, RBAC, and policy-driven validation (OPA, Kyverno, admission controllers) so it simply falls into place.
🤹 Flux does Multi-Tenancy (and “Multi-everything”): Flux uses true Kubernetes RBAC via impersonation and supports multiple Git repositories. Multi-cluster infrastructure and apps work out of the box with Cluster API: Flux can use one Kubernetes cluster to manage apps in either the same or other clusters, spin up additional clusters themselves, and manage clusters including lifecycle and fleets.
📞 Flux alerts and notifies: Flux provides health assessments, alerting to external systems and external events handling. Just “git push”, and get notified on Slack and other chat systems.
👍 Users trust Flux: Flux is a CNCF Incubating project and was categorised as "Adopt" on the CNCF CI/CD Tech Radar (alongside Helm).
💖 Flux has a lovely community that is very easy to work with! We welcome contributors of any kind. The components of Flux are on Kubernetes core controller-runtime, so anyone can contribute and its functionality can be extended very easily.

Over and out

If you like what you read and would like to get involved, here are a few good ways to do that:

Join our upcoming dev meetings on 2022-04-07 or 2022-04-13.
Talk to us in the #flux channel on CNCF Slack
Join the planning discussions
And if you are completely new to Flux, take a look at our Get Started guide and give us feedback
Social media: Follow Flux on Twitter, join the discussion in the Flux LinkedIn group.

We look forward to working with you.

Blog: Flux puts the Git into GitOps

Fri, 25 Mar 2022 09:30:00 +0000

Ever since the rewrite of Flux as a set of focused controllers, it has become clearer what each of its functions and capabilities are. The aptly named controllers carry in their name what they are responsible for and which data or tooling they interact with, so that is, e.g. source, kustomize, image-automation, notification, helm, etc.

If you wanted to string a proof-of-concept for a GitOps tool together, a naïve solution could be to just shell out to various tools like curl, git, kubectl and helm. While that might feel intuitive at first (since it so closely resembles one’s manual workflow behind the keyboard) and might be quick and fundamentally functional, it comes at a big cost in the subsequent refinement stages: adequately catching errors, providing detailed status information, security considerations, mismatches between command line tools and infrastructure implementation, API and CLI versions, etc.

In the course of the last five years since the start of the Flux project, we have seen all of the above and more. Because other projects made those mistakes, or because we did.

Let’s drill a bit deeper into why we put so much effort into integrating as tightly with given tooling APIs and SDKs as possible.

Why we are not using the Git CLI

Without Git there is no GitOps, so we obviously want to support all Git providers, all the edge cases, all the different ways things can be set up and all the Git operations we need. Obvious interactions with Git are when we perform clone and push operations on remote Git repositories, for example.

Using a CLI for any code-path should be a last resort - if at all. It is a design principle for the Flux controllers not to do this. We avoid an entire class of vulnerabilities: command injection.

Another big reason back when we started working on source-controller for dropping the Git CLI was multi-tenancy. The Git CLI wants the SSH and PGP keys on disk while we wanted them to be loaded from memory to isolate the tenants secrets without having to write them on disk and risk being vulnerable to directory traversal attacks.

All in all we did choose not having to rely on the Git binary being present, instead we link statically against a known-good and sufficiently well-tested version. More on that below.

Why do we have support for multiple Git implementations

We started out using go-git for all Git operations, as it is an implementation of the Git protocol written entirely in Go. When we wanted to support Azure DevOps and saw that support for multi_ack and multi_ack_detailed wasn’t included in go-git, we started making use of git2go in addition. It is the Go bindings of the libgit2 library and has greater support for more complex capabilities in the git wire protocol, including git protocol version 2. Unfortunately git2go does not support shallow clones or Git submodules. The newly added support for commit signing with SSH keys is also not supported by our implementations at present.

While the above might sound like petty implementation details, we had to learn ourselves that each Git implementation has its own set of shortcomings. Things which “just work” in the Git CLI, any of the implementations get subtly wrong, as they work on the “plumbing” level of Git. In the end we chose to make gitImplementation a configurable setting.

Just to illustrate what happens when you try to do things just right, here’s a couple of pieces of work we needed to get done along the way:

we had to add support for e.g. verifying known_hosts files for SSH connections
when connecting over SSH, we received SHA1 and MD5 fingerprints of the returned public key from the server and not the key itself, which made known_hosts verification a little harder
changes on libgit2 would break the way that known hosts used to work
Making various kinds of SSH key types work, e.g. support for ECDSA* added as part of libgit2 >= 1.2.0, ED25519 as part of libgit2 >= 1.3.0.
we needed to start verifying PGP signatures

Tracking upstream developments in Git

As Git has become ubiquitous and virtually all of the world’s software development depends on Git, it still is in active development. Constant improvements affect features, efficiency, configurability and better security. Of course we want to pass this all down to our users: more efficient download makes a huge difference, support for Git Submodules enables new use-cases, support for more GPG verification or new SSH key formats adds additional security and when new features are rolled out in Git providers, we need to support them in Flux too.

Integrating these changes into Flux unfortunately isn’t as easy as it sounds. One part of the dependency chain for git2go goes:

flowchart LR libgit2 --> libssh2 --> OpenSSL

So that’s libgit2, libssh2 (to enable the SSH transport) and OpenSSL. As Linux vendors often take a very conservative approach to bringing new software releases to stable releases, we were unfortunately pushed to building these dependencies ourselves. In addition to that, these libraries have quite a few configuration options that can only be set at build time and unfortunately the openssl/libssh2 packages of different Linux distributions act in slightly different ways. This created a yet different problem: the versions we shipped on the containers could behave in different ways when we were developing on our Mac/Linux machines. This forced us to cross-compile statically built libraries that we can simply download at both development time or statically linking them into the final binary we create when releasing our controllers.

We decided to build the libraries for the AMD64, ARM64 and ARMv7 architectures and link them statically, which was a prerequisite for us to enable fuzzing for all Flux controllers. Getting this all up and running for every upstream release and making sure it’s all covered nicely with tests is a challenge and an area of work we invested quite a lot of time in.

What’s next in Git things?

libgit2 does not expose concepts to allow users to set timeouts for network operations, meaning that most git operations could hang indefinitely in specific circumstances. This would result in specific GitRepository objects getting stuck and stop updating until the controllers get restarted - Users reported this for both the image-automation and source controllers in the past 6 months.

We had a few challenges getting traction making changes upstream to fix similar issues, so to avoid delaying a fix or forking the dependency, we decided to add experimental support for Go managed transports, which means we can enforce that network operations won’t take more than a given amount of time to complete, but without requiring any changes upstream.

This is part of Flux 0.28 and can be enabled by adding an environment EXPERIMENTAL_GIT_TRANSPORT=true in both source and image-automation controllers.

This will give us more control over the transport with Go native transport using the libgit2 smart transport support. Read the source-controller changelog for more information.

If you want to enable this automatically, just add the following to your kustomization.yaml:

patches:
- patch: |
 - op: add
 path: /spec/template/spec/containers/0/env/0
 value:
 name: EXPERIMENTAL_GIT_TRANSPORT
 value: "true"
 target:
 kind: Deployment
 name: "(source-controller|image-automation-controller)"

Support for sha256 hashes: Git CLI supports it since 2.29, however all major Git service providers, such as GitLab and GitHub, are yet to make some progress on this front. Upstream, libgit2 is starting to pave the way for that support on v1.4.0, we will continue to watch this space so we can support Flux users as the industry moves on from SHA1.

After this strong focus on stability in the last months, we are now going to take a look at how we can optimise our git implementation to reduce resource consumption and network traffic across git reconciliations.

Conclusion

Flux doesn’t shell out to binaries like git, helm or kubectl, because we deem it too error-prone and we would miss big opportunities to bring you the best developer experience and the most accurate information on every step along the way. This way we offer more control to you. As you might have gathered from the amount of additional work this all means for us, we take the “Git” in “GitOps” very seriously.

Talk to us

We love feedback, questions and ideas, so please let us know your personal use-cases today. Ask us if you have any questions and please

join our upcoming dev meetings
find us in the #flux channel on CNCF Slack
add yourself as an adopter if you haven’t already

See you around!

Blog: Flagger adds Gateway API Support

Fri, 11 Mar 2022 13:30:00 +0000

The Flagger team is proud to bring you Kubernetes Gateway API support as part of the 1.19.0 release. Read here about why this is a significant development in Flagger and how you can make use of it.

What is Flagger?

Flagger is a progressive delivery tool that automates the release process for applications running on Kubernetes. It reduces the risk of introducing a new software version in production by gradually shifting traffic to the new version while measuring metrics and running conformance tests.

Flagger was designed to give developers confidence in automating production releases using delivery techniques such as:

Canary release (progressive traffic shifting)
A/B Testing (HTTP headers and cookies traffic routing)
Blue/Green (traffic switching and mirroring)

What is the Gateway API?

The announcement blog post defines its design principles as

Expressiveness - In addition to HTTP host/path matching and TLS, Gateway API can express capabilities like HTTP header manipulation, traffic weighting & mirroring, TCP/UDP routing, and other capabilities that were only possible in Ingress through custom annotations.

Role-oriented design - The API resource model reflects the separation of responsibilities that is common in routing and Kubernetes service networking.

Extensibility - The resources allow arbitrary configuration attachment at various layers within the API. This makes granular customization possible at the most appropriate places.

Flexible conformance - The Gateway API defines varying conformance levels - core (mandatory support), extended (portable if supported), and custom (no portability guarantee), known together as flexible conformance. This promotes a highly portable core API (like Ingress) that still gives flexibility for Gateway controller implementers.

Gateway API exposes a more general API than Ingress for proxying and you can use it for more protocols than just HTTP (although most implementations support just HTTP for now). It models more infrastructure components to provide better deployment and management options. There are three core components to the Gateway API:

GatewayClass: This lets us define which controller implementation we want to use.
Gateway: A Gateway resource is attached to a GatewayClass and has a 1:1 relationship with the actual load balancing infra. It lets us define a set of listeners, through which we can specify which Route resources to evaluate for routing, amongst other things.
HTTPRoute: This is a Route resource that is specific for HTTP requests. It defines routing rules such as filters, path and header matches, etc. and which services should the request be forwarded to.

How does this work in Flagger?

Flagger makes use of the fact that HTTPRoute allows users to define a weight related to each reference to a service inside a routing rule. These weights are used to determine which service should receive a request. For example, if we want to send 10% of our traffic to another service, we can define a HTTPRoute like:

apiVersion: gateway.networking.k8s.io/v1alpha2
kind: HTTPRoute
metadata:
 name: foo-route
spec:
 parentRefs:
 - name: example-gateway
 hostnames:
 - "foo.example.com"
 rules:
 - matches:
 - path:
 type: PathPrefix
 value: /login
 backendRefs:
 - name: foo-primary
 port: 8080
 weight: 90
 - name: foo-canary
 port: 8080
 weight: 10

This sends 10% of all requests coming to foo.example.com/login to the new service and the other 90% requests go to the stable service. You can read more about traffic splitting in Gateway API here.

Flagger fully automates the creation of HTTPRoutes with the appropriate header matches, path matches, etc and attaches the primary and canary service to the HTTPRoute. During the canary analysis, the weights related to both the services are adjusted accordingly.

If you want to get started right away, have a look at our tutorial, which shows you how to use Contour’s Gateway API implementation and Flagger to automate canary deployments. It won’t take long to follow, but will convey how powerful this integration is.

Flagger works with all implementations

With added support for Gateway API, Flagger now works with all implementations, which means that as of today these are natively supported: Contour, Emissary-Ingress, Google Kubernetes Engine, HAProxy Ingress, HashiCorp Consul, Istio, Kong and Traefik.

The Flagger team has successfully tested Contour and Istio using the v1beta2 Gateway API. Starting with Flagger v1.19, the Gateway API is part of our end-to-end test suite using the Contour implementation.

How metrics work

The Gateway API defines a common interface for traffic management, which saves us from doing anything vendor specific. But the metrics related to the traffic, still are specific to the Ingress/Service Mesh you’re using. Flagger lets you define a custom resource MetricTemplate, which runs queries against your metrics provider and calculates stats like error rate, latency, etc. For example, if you’re using Istio with Gateway API, the below MetricTemplate would calculate the error rate using Prometheus as a provider during a canary analysis:

apiVersion: flagger.app/v1beta1
kind: MetricTemplate
metadata:
 name: error-rate
 namespace: istio-system
spec:
 provider:
 type: prometheus
 address: http://prometheus.istio-system:9090
 query: |
 100 - sum(
 rate(
 istio_requests_total{
 reporter="source",
 destination_workload_namespace="{{ namespace }}",
 destination_workload=~"{{ target }}",
 response_code!~"5.*"
 }[{{ interval }}]
 )
 )
 /
 sum(
 rate(
 istio_requests_total{
 reporter="source",
 destination_workload_namespace="{{ namespace }}",
 destination_workload=~"{{ target }}"
 }[{{ interval }}]
 )
 ) * 100

Similarly the below MetricTemplate allows Flagger to compute the latency when using any Envoy based Ingress/Service Mesh:

apiVersion: flagger.app/v1beta1
kind: MetricTemplate
metadata:
 name: latency
 namespace: flagger-system
spec:
 provider:
 type: prometheus
 address: http://flagger-prometheus:9090
 query: |
 histogram_quantile(0.99,
 sum(
 rate(
 envoy_cluster_upstream_rq_time_bucket{
 envoy_cluster_name=~"{{ namespace }}_{{ target }}-canary_[0-9a-zA-Z-]+",
 }[{{ interval }}]
 )
 ) by (le)
 ) / 1000

Conclusion

The Gateway API is in alpha. As of 2022-03-11 its GitHub README says

The latest supported version is v1alpha2 as released by the v0.4.0 release of this project. This version of the API is expected to graduate to beta in the future with relatively minimal changes.

We as the Flux project will update the integration once the API becomes beta/stable.

Thanks a lot Sanskar Jaiswal for working on the implementation!

We are excited to bring this feature to you and we love feedback! Please let us know if you have feedback, questions or how you are going to use this!

Blog: Security: Using Pod Security Standard "restricted"

Wed, 09 Mar 2022 12:30:00 +0000

Next up in our blog series about Flux Security is how we moved to Pod Security Standard “restricted”, all the background info you need to know and how that makes things safer for you.

Since version 0.26 of Flux we are applying

[..] the restricted pod security standard to all controllers. In practice this means:

all Linux capabilities were dropped

the root filesystem was set to read-only

the seccomp profile was set to the runtime default

run as non-root was enabled

the filesystem group was set to 1337

the user and group ID was set to 65534

Flux also enables the Seccomp runtime default across all controllers. Why is this important? Well, the default seccomp profile blocks key system calls that can be used maliciously, for example to break out of the container isolation. The recently disclosed kernel vulnerability CVE-2022-0185 is a good example of that.

Pod Security Standards definition

Kubernetes defined three policies in its Pod Security Standards. They range from

Privileged: This does not place any restrictions on the workload at all. The idea being that this can be used for system- and infrastructure-level workloads which are managed by privileged and trusted users only.
Baseline: This policy comes with some restrictions. It aims to guard against known privilege escalations while still making it easy to adopt and use it by keeping a certain level of compatibility with most workloads.
Restricted: Inheriting all the restrictions from Baseline, it enforces additional limitations, thus follows hardening best practices by increasing the isolation levels the workload is exposed to.

We are very pleased that all Flux controllers were moved to Restricted, as that offers the highest level of security for you.

We recommend checking out the Upstream Kubernetes documentation on Pod Security Standards as it gives a generally good overview of all the security features enabled. In addition to that you can see which restrictions were added as part of which Kubernetes release, meaning that with every Kubernetes release, you will benefit from new Upstream Kubernetes security improvements automatically.

Note:

As of v1.24 Kubernetes still runs all workloads with seccomp in unconfined mode, in other words, disabled. On the other hand, Docker has seccomp enabled by default for years now.

There are discussions to change the Kubernetes default on v1.25, and have all workloads set to RuntimeDefault unless opted-out. This would be based on SeccompDefault feature gate being enabled from that version onwards.

Note: If you are an OpenShift user, you might run into this issue ( related upstream report). The work-around right now is to remove the seccomp profile as described in these instructions.

`seccomp` and `RuntimeDefault`

Seccomp is short for “Secure Computing”. It refers to a facility in the Linux kernel which can limit the number of system calls available to a given process. Right now there are around 300+ system calls available, e.g. read to read from a file descriptor or chmod to change the permissions of a file. The more syscalls you block, the more secure your application, as a rogue process will only be able to do what you specified.

In its first inception seccomp was introduced into Linux in 2005, to Docker in version 1.10 (Feb 2016) and to Kubernetes in version 1.3 (Jul 2016). So while the technology has been around for a while and you could handcraft your own seccomp profiles, the challenge has always been striking the right balance: if you are too generous in your filter, it won’t guard against malware effectively – if you are too strict, your application might not work.

All container runtimes come with a default seccomp profile. Docker Desktop for example blocks around 44 system calls. In Kubernetes you can enable the seccomp profile RuntimeDefault for your pod like so:

spec:
 securityContext:
 seccompProfile:
 type: RuntimeDefault

All Flux controllers have this implemented as well now!

By adopting both changes, we further restrict the permissions that Flux requires in order to operate. This, alongside other changes we are working on, translate in a decreased attack surface which may reduce the impact of eventual CVEs that may surface in our code base - or our supply chain.

Talk to us

We love feedback, questions and ideas, so please let us know your personal use-cases today. Ask us if you have any questions and please

join our upcoming dev meetings
find us in the #flux channel on CNCF Slack
add yourself as an adopter if you haven’t already

See you around!

Flux – Blog

Blog: Stairway to GitOps: Scaling Flux at Morgan Stanley

The Early Days: Pushing Limits

Step 1: Security and Self-Service

Step 2: Operating at Scale

Tuning for Performance

Moving from Git to S3

Step 3: Observability and Feedback Loops

Looking Ahead

Watch the Full Talk

Join Us at FluxCon Europe 2026

Blog: Announcing Flux 2.8 GA

Highlights

Helm v4 Support

Faster recovery from failed deployments

Note

Ecosystem News

Flux Operator Web UI

Preview environments

Supported Versions

Upgrade Procedure

Upgrade Procedure for Flux v2.8

Over and out

Blog: Announcing Flux 2.7 GA

Highlights

General availability of Image Update Automation

Watching for changes in ConfigMaps and Secrets

Workload Identity Authentication for Remote Clusters

Object-level Workload Identity

OpenTelemetry Tracing

Controller Improvements

CLI Improvements

Artifact Generators

Multiple Source Composition

Monorepo Decomposition

Supported Versions

Upgrade Procedure

Upgrade Procedure for Flux v2.7+

Over and out

Blog: Time-based deployments with Flux Operator

How it works

A complete example

ResourceSetInputProvider Definition

ResourceSet Definition

Further reading

Blog: FluxCon NA 2025

Call for Papers

New Use-Cases

Connect with the Community

Blog: Announcing Flux 2.6 GA

Highlights

General availability of Flux OCI Artifacts

Breaking changes

Image Automation Digest Pinning

Object-level Workload Identity

GitHub App Authentication

Notifications Improvements

Controller Improvements

Supported Versions

Over and out

Blog: AI-Assisted GitOps with Flux Operator MCP Server

Bringing AI to GitOps

How It Works

Getting Started

Setting Up AI Instructions

Practical Applications

1. Quick Health Assessment

2. GitOps Pipeline Visualization

3. Cross-Cluster Comparisons

3. Root Cause Analysis

4. GitOps Operations

5. Kubernetes Operations

Security Considerations

The Future of AI-Assisted GitOps

Blog: GitHub App bootstrap with Flux Operator

Flux Operator

Bootstrap using Flux Operator and Helm

Bootstrap using Flux Operator and Terraform

GitHub App Docs

Conclusion

Enhanced `HelmRelease` reconciliation model

Recovery from `pending-*` Helm release state