2025 By The Numbers​

Before we dive into the details, here's what the KubeVela community accomplished this year:

These numbers tell a story of a healthy, active open-source project—but the real story is in the features we shipped and the problems we solved.

The 1.10 Release Series: Maturity in Focus​

2025 was the year of the 1.10 release series, and our theme was clear: maturity and reliability. We listened to our users—platform engineers running KubeVela in production—and focused on the features that make their lives easier.

Kubernetes Compatibility​

We continue to invest in ensuring KubeVela works seamlessly with recent Kubernetes releases. Staying current with the Kubernetes ecosystem is critical for enterprise adoption, and we regularly update our dependencies, API versions, and test matrices to keep KubeVela on the leading edge.

StatefulSet Support​

One of our most requested features finally landed in v1.10.3: native StatefulSet support (#6638). Platform engineers can now define stateful workloads—databases, message queues, distributed caches—with the same elegance as stateless services. This opens up KubeVela to a whole new class of applications.

CueX Compiler for Templating​

Building on the CueX compiler framework originally developed for workflows, we expanded CueX support to component and trait templating (#6720). This extension unlocks new advanced capabilities and lays the groundwork for future templating improvements across KubeVela.

Dependency-Aware Workflows​

Issue #6852 was one of our most upvoted issues—users were confused about why component dependsOn wasn't respected in workflows. In v1.10.4, we fixed this. KubeVela now dynamically adds workflow dependencies between steps according to component dependencies.

This was a small change in code but a big change in user experience. It's the kind of improvement that makes you wonder, "Why didn't it always work this way?" And that's exactly the point—good software should be intuitive.

Enhanced Status Reporting​

Platform engineers told us they needed more visibility into what's happening with their applications. The existing health checks were helpful, but they wanted more. So we added Status Details—a new field that captures structured diagnostic information about your components and traits.

status:
  services:
    - name: my-service
      status:
        readyReplicas: "3"
        totalReplicas: "3"
        readyPercentage: "100"
        deploymentMode: "RollingUpdate"

This is the kind of information that makes debugging at 2 AM a little less painful.

Application Status Metrics​

Speaking of observability, we added new Prometheus metrics for application health:

• kubevela_application_health_status
• kubevela_application_phase
• kubevela_application_workflow_phase

Now you can set up alerts, build dashboards, and integrate KubeVela into your existing monitoring stack. We've kept these metrics low-cardinality to ensure they're cost-effective to collect and store.

Security Enhancements​

Security was a major theme in 2025. We shipped several improvements that make KubeVela safer for production use:

• securityContext and podSecurityContext traits (#6666): Configure pod and container security settings directly in your component definitions
• Definition permission validation (#6876): KubeVela now validates that your Definitions have the necessary permissions at application creation time, catching misconfigurations early
• Fail-fast CUE validation (#6774): Required parameters are now validated upfront, preventing deployments that would fail later

Supply Chain Security​

In an era where software supply chain attacks are increasingly common, we're proud to have shipped signed releases, SBOMs, and SLSA provenance for all KubeVela artifacts. When you pull a KubeVela container image, you can verify it came from us and see exactly what's inside.

Semantic Versioning for Definitions​

Platform engineers managing large definition libraries will appreciate the new semantic versioning support for Definitions (#6648). You can now version your ComponentDefinitions and TraitDefinitions, making it easier to evolve your platform APIs without breaking existing applications.

Looking Ahead: The Future of Application Delivery​

As we look beyond 2025, we find ourselves at an exciting inflection point. The landscape of software delivery is evolving rapidly, and we're paying close attention to the trends that will shape how developers and platform engineers work in the years to come.

The Rise of Intelligent Platforms​

The emergence of AI and large language models is transforming how we interact with complex systems. We're exploring how these capabilities could fundamentally change the KubeVela experience. Imagine describing what you need in natural language and having an intelligent system generate the right configuration, diagnose issues before they become problems, or suggest optimizations based on real-world patterns.

We're not building AI for the sake of AI—we're asking ourselves: What if deploying and operating applications could be as simple as having a conversation?

Lowering Barriers, Not Just Abstracting Complexity​

One consistent theme in our community feedback is the desire for simpler paths to productivity. Whether you're a developer who just wants to ship code or a platform engineer building golden paths for your organization, KubeVela should meet you where you are.

We're investing in new ways to author and share platform building blocks—approaches that leverage familiar tools and languages while maintaining the power and flexibility that KubeVela offers today.

Beyond Deployment: Intelligent Operations​

The future of application delivery isn't just about getting code to production—it's about running it well. We're thinking deeply about how KubeVela can provide proactive insights: cost awareness before you deploy, security posture at a glance, performance optimization suggestions based on actual usage patterns.

These are explorations, not promises. We're in the early stages of understanding how these capabilities could benefit our community. But we're genuinely excited about the possibilities, and we believe the intersection of platform engineering and intelligent tooling will define the next era of cloud-native development.

Stay tuned. We'll share more as these ideas take shape.

Community Highlights​

Open source is about people, and we've been fortunate to have an incredible community this year.

New Contributors​

We welcomed 33 unique contributors in 2025, from first-time contributors fixing typos to seasoned developers implementing major features. Every contribution matters, and we're grateful for each one.

Community Meetings​

Our monthly community meetings continue to be a highlight. Each month, we showcase new features, review upcoming milestones, and engage in Q&A. If you haven't joined one, we'd love to see you there. Check our community calendar for details.

KubeCon North America 2025​

KubeCon North America 2025 in Atlanta was a milestone moment for our community. We presented Democratizing Platform Engineering and Introducing the Component Contributor Architecture, sharing our vision for how KubeVela enables organizations to build platforms that empower every team to contribute—not just consume.

We also hosted a Project Pavilion booth throughout the conference, which gave us invaluable face time with the community. Platform engineers, developers, and operators from organizations of all sizes stopped by to share their experiences, challenges, and ideas for the future. These conversations are the lifeblood of open source—hearing directly from users about what's working and what needs improvement helps us build for the real world. If you visited us at the booth, thank you. Your feedback is already shaping our roadmap.

CNCF Incubation​

As a CNCF Incubating project, we continue to work closely with the foundation and the broader cloud-native ecosystem. We're committed to the open governance and vendor-neutral approach that makes CNCF projects trustworthy for enterprise adoption.

Lessons Learned​

Looking back at 2025, a few lessons stand out:

1. Listen to your users. The best features we shipped this year—StatefulSet support, dependency-aware workflows, enhanced status reporting, security traits—all came from user feedback. Keep those GitHub issues coming.

2. Boring is good. We didn't ship flashy new features every month. Instead, we focused on stability, compatibility, and reliability. For a project used in production, that's exactly what matters.

3. Documentation is a feature. We're the first to admit our documentation needs work. Simplifying the getting-started experience and making it easier for new users to succeed with KubeVela remains a priority for us.

Thank You​

To everyone who used KubeVela in 2025, filed an issue, submitted a PR, joined a community meeting, or just gave us a star on GitHub—thank you.

Building an open-source project is a marathon, not a sprint. We're in this for the long haul, and we're grateful to have you on this journey with us.

Get Involved​

Want to be part of the story? Here's how:

• Star us on GitHub: github.com/kubevela/kubevela
• Join our Slack: #kubevela on CNCF Slack
• Attend a community meeting: Every month, all are welcome
• Contribute: Check out our good first issues

Here's to making shipping applications more enjoyable. 🚀

The KubeVela Team

December 2025

This is the first blog post in a while but we are excited to reinvigorate the community and keep developers and users up to date with all things KubeVela. The team is committed to providing regular updates, news and information about upcoming events and features.

We recently released the latest v1.10.4 which includes a host of useful features to improve the maintainability of Applications deployed with Kubevela. This blog will provide a quick overview of the features and how you can get started with utilising them in your KubeVela setup.

While v1.10.4 follows semantic versioning as a patch release, we're excited to deliver several new opt-in features alongside the usual bug fixes and improvements.

These features are disabled by default (where applicable) to maintain backward compatibility, allowing users to adopt them at their own pace.

Kubernetes 1.31 Support​

The team have put huge efforts into completing the support for Kubernetes v1.31 across all KubeVela repositories.

Users can now confidently run KubeVela on Kubernetes 1.31 with all core functionality validated.

Resource Existence Validation​

KubeVela users can now validate that the resources being created by their definitions are present on the cluster ahead of deployment of the Application. This new feature will work across Component, Trait, Policy and WorkflowStep definitions to reduce runtime errors and improve the overall reliability of KubeVela deployments.

To get started, developers can set ValidateResourcesExist feature flag in their installations.

See PR 6932 for more information.

Native Cue in Health Fields​

Developers writing workload definitions can now add status attributes in native cue, rather than nested strings. Behind the scenes, the string and cue conversions happen automatically and include some new validations of the attribute types and the presence of required fields.

This feature is enabled by default; but you'll need to ensure to use the latest version of the KubeVela CLI to take advantage of cue formatting. String formatting is completely backwards compatible.

See PR 6859 for more information.

Enhanced Status Reporting​

Applications had two primary means to communicate status - the Health Check and Custom Message. Whilst these provide an insight into the overall health of the application and give a succinct human readable summary of current state - we had feedback from developers that they required a longer lived, more detailed way to capture information about managed resources.

Hence, we added the new Status Details field that can capture a map of any information developers feel is useful for reporting on, debugging or observing their components & traits.

It uses familiar cue syntax to extract information and has access to the full KubeVela context already accessible in Health Checks and Custom Messages.

For example, from the docs:

attributes: status: details: {
    readyReplicas: context.output.status.readyReplicas
    totalReplicas: context.output.spec.replicas
    readyPercentage: (context.output.status.readyReplicas / context.output.spec.replicas) * 100
    deploymentMode: context.output.spec.strategy.type
}

will provide users with the following in their Application status, allowing for better abstraction from the underlying resource.

status:
  services:
    - name: my-service
      status:
        readyReplicas: "3"
        totalReplicas: "3"
        readyPercentage: "100"
        deploymentMode: "RollingUpdate"

Better still - we've put work into unifying the various status evaluators. The fields captured in Status Details are made available in the Health Check and Custom Message (via context.status.details) so that logic can be better reused. See How Status Evaluation Works for full details of the changes.

See PR 6828 or the revised documentation for more information.

Application Status Metrics​

As part of the fix in PR 6824 we now ensures that Application health is continuously evaluated, even after the workflow succeeds.

To compliment this, we've added additional application level metrics so that operators can have simplified visibility into the status of their Applications.

These are the:

• kubevela_application_health_status{app_name, namespace}
• kubevela_application_phase{app_name, namespace}
• kubevela_application_workflow_phase{app_name, namespace}

We've made efforts to ensure these are low cardinality through numeric phase mapping - so operators should be able to collect and monitor these cost effectively in their monitoring solutions.

To add more depth to monitoring capabilities, we've also added structured logging when Application changes occur. These logs capture the Application state in a high detail of detail to supplement the new metrics.

The full structure of these logs can be found here

To get started, users can enable the EnableApplicationStatusMetrics feature gate to activate both metrics and the new structured logging.

See PR 6857 or the documentation for more information.

Dependency-Aware Workflows​

Issue 6852 raised the issue with the confusion of DependsOn within components, and DependsOn within workflows - and how workflows did not natively respect component dependencies.

KubeVela will now dynamically add workflow dependencies between steps according to the components dependencies; which is vastly simplified and much more in line with user expectations.

See PR 6854 for more information.

Colorised Logging Support​

We've introduced opt-in support for colorised logging to improve the developer debugging experience.

For local development, developers can use the new --dev-logs to enable this for their console output.

This feature is not recommended for production & CI usage and is disabled by default. When disabled, existing log functionality is preserved.

The log formatting enhancements include:

• Color-coded structured fields (cyan for keys, grey for values)
• Emphasized primary message text in bright white
• Enriched location information (file:method:line)
• Spring Boot-inspired formatting style

See PR 6931 for more information.

Full Release Details

Of course this blog doesn't capture all of the great work that went into this release. Our contributors have made amazing progress on fixing bugs, handling chore work, bolstering our test suites, providing feedback & issue reports, and much more.

So a huge thank you due to everyone who contributed during this (and all!) release cycles. The full details of the release can be found at v1.10.4

Give Us Your Feedback

We are excited for the community to get their hands on these new features and encourage any feedback or contributions to keep improving KubeVela!

Slack: #kubevela (English)

Every week we host a community meeting to showcase new features, review upcoming milestones, and engage in a Q&A. All are welcome!

In the spring of 2023, I was accepted as a CNCF student under the KubeVela project through LFX Mentorship. In this project, I am responsible for developing a CUE code and documentation generator based on Golang from scratch, laying the foundation for the infrastructure part of KubeVela's future extensibility.

What is LFX Mentorship?​

LFX Mentorship is a remote learning program that provides 12 weeks of learning opportunities for open source contributors. It is led by specific mentors (usually the maintainers of the projects) who help mentees contribute to the community and projects.

Many open-source organizations or foundations use LFX Mentorship to announce projects and recruit students for development and contributions. I focused on the cloud-native and noticed that CNCF started its spring projects on LFX in February 2023. I began to explore and apply for projects that interested me.

What is KubeVela?​

KubeVela is a modern software delivery and management control plane. The goal is to make deploying and operating applications across today's hybrid, multi-cloud environments easier, faster and more reliable.

KubeVela originated from the Open Application Model (OAM) model based on Kubernetes, jointly launched by Alibaba Cloud and Microsoft at the end of 2019. Through continuous evolution and iteration, it has incorporated a large amount of feedback and contributions from the open-source community (especially participants from Microsoft, ByteDance, 4Paradigm, Tencent, and Full Truck Alliance). In 2020, it officially met the open-source community under the name "KubeVela" at KubeCon North America.

The KubeVela project has been developing rapidly, and its community growth trend is as follows:

It is worth mentioning that in March, KubeVela was promoted to a CNCF incubation project, further proving the stability and flexibility of KubeVela in production environments.

Project Details​

Project Name: Support auto generation of CUE schema and docs from Go struct

Project Description: In KubeVela's provider system, we can use our defined Go functions in CUE schema. The Go providers usually have a parameter and return. Fields in Go providers are the same as fields in CUE schema, so it is possible and important to support automatic generation of CUE schemas and documents from Go structs.

Project Outcome: Auto-generators of CUE schemas and docs from Go structs, the capabilities should be wrapped in vela cli command.

Project Mentors: Fog Dong, Da Yin

Project Link: https://mentorship.lfx.linuxfoundation.org/project/85f61cae-02d7-4931-8d87-d3da3128060e

Application and Development​

When browsing through the project list, KubeVela quickly became one of my candidates. Before diving into the cloud-native, I had come across the KubeVela project and attempted to understand its concepts and working principles, but due to my limited expertise, I only scratched the surface. This time, if I could familiarize myself with KubeVela through a small entry point, it would be the best contribution path. Additionally, metaprogramming and code generation are important techniques in Golang, and I also wanted to participate in this project as an opportunity for practical experience.

The project involves the core part of KubeVela: CUE. This is the first concept I needed to understand. Through the KubeVela official documentation and CUE Issues, I realized that CUE is a language designed for configuration, with advantages compared to other languages in terms of programmability, automation, and integration with Golang. On the other hand, as KubeVela evolves, it continuously provides practical use cases and feedback for CUE.

After contacting the mentor, my initial goal was to create a demo as a showcase. The core part of the entire project lies in the conversion between Golang AST and CUE AST. I first found a snippet of code that I could learn from. After thoroughly understanding it, I extracted the core parts, made modifications and adaptations, and implemented the struct conversion for the demo.

By writing the demo, I gained a clearer understanding of the overall project targets. As the top-level language for users and platform developers, CUE needs to interact with Golang extensively, serving as an intermediary to connect and control cloud platforms. In many scenarios, CUE needs to maintain consistency with Golang code, or else there may be errors in the intermediate conversion. This process is time-consuming, labor-intensive and issues are only exposed at runtime, which can potentially impact the stability of production environments. The aim of this project is to solve this problem, making Golang code the single source of truth and ensuring overall configuration consistency through static code generation.

In the project description issue, the mentor provided an example of generating providers. Everything became clear: I divided the CUE Generator into three layers. The bottom layer is responsible for basic and core AST conversions. The middle layer reads specific Golang code, such as providers, policies, etc. , extracts information from the Golang code, and writes it into CUE files. The top layer exposes the generation capability as a CLI to users and developers, allowing them to quickly generate CUE and documentation. When supporting more different formats of CUE in the future, the underlying transformation capabilities can be easily reused.

After further communication with the mentor, I added support for struct tags and comments and summarized some ideas into the proposal. After a series of iterations and discussions, the project has taken shape, and I am honored to have been accepted as a mentee in the LFX Mentorship program.

Following the initial design and demo, the formal development process went relatively smoothly, with most of the communication focused on user experience and detailed design.

The first pull request (PR) received valuable reviews as it was not split into smaller parts, and it took 50 comments before it was finally merged. Since the initial code was written in a casual manner, I also focused on refactoring parts of the code to make it more clear and robust.

From the first PR in February to the fifteenth PR at the end of May, the project is essentially complete, and all the code has been merged into the main branch. It has also passed two mentor evaluations, and I am about to graduate from the first project of LFX Mentorship program.👏

Project Outcomes​

Over the past three months of development, the project has primarily produced three capabilities and two CLIs, with test coverage exceeding 90%.

The core capabilities of the project are located in the references/cuegen directory. It implements the basic functionality of converting Go AST to CUE AST and is accompanied by a README to provide developers with specific conversion rules. The code for the middle layer is placed in the references/cuegen/generators directory, and generators for the provider format have been implemented so far. The documentation generation component is located in references/docgen/provider.go.

The project has added two CLI subcommands, namely vela def gen-cue and vela def gen-doc. The former generates CUE files in the corresponding format from Go code, exposing the capabilities of the middle layer as a CLI, while the latter generates documentation for CUE.

Since vela def gen-cue only supports one file at a time, a shell script was written to enable batch generation by traversing directories: #6009

Taking the code snippet from kubevela/pkg/providers/kube as an example, let's perform the transformation and verification.

First, convert kube.go to kube.cue:

$ vela def gen-cue \
	-t provider \
	--types *k8s.io/apimachinery/pkg/apis/meta/v1/unstructured.Unstructured=ellipsis \
	--types *k8s.io/apimachinery/pkg/apis/meta/v1/unstructured.UnstructuredList=ellipsis \
	kube.go > kube.cue

Then, convert kube.cue to kube.md:

$ vela def gen-doc -t provider kube.cue > kube.md

The final result is as follows:

Future Outlook​

Although the expected outcomes of LFX Mentorship have been fully achieved, it is only the first step for cuegen, and its derivative work will also play an important role in the future development of KubeVela. For example, based on cuegen, we can automate the generation of policy rules that are currently manually maintained. We can migrate and validate the existing providers in kubevela/pkg. We can also develop scaffolding tools for user-defined providers, all of which rely on the capabilities of cuegen. These will be the key areas of my future work in the community.

In addition to the related work in the cuegen ecosystem, I will also delve into other aspects of KubeVela, such as gaining in-depth familiarity with OAM production practices and the user community, and exploring the possibilities of new features by reading the source code of Workflow component. I have also initiated an application to become a KubeVela Reviewer, aiming to contribute to the project's code quality control.

This is my first participation in the LFX Mentorship program, and throughout the three months of communication and collaboration, both mentors have provided me with a lot of help and guidance in terms of details and decision-making. We also conducted a complete demonstration of the functionality and discussed the future direction of the community through online meetings.

Open source is a process driven by interests and self-motivation. Developers can continuously improve themselves through their experiences in different communities and grow together with the community. Open source is about taking the first step with courage and trying to read the source code of projects that interest you. For students, participation in open source is primarily a learning process, and each step brings different rewards and insights. I am very grateful to have encountered the KubeVela community through LFX Mentorship, and I look forward to further deepening my involvement and contributions to the community in the future!

Now the question is, can we leverage ChatGPT to transform the way we deliver applications? With the integration of ChatGPT into DevOps workflows, we are witnessing the possible emergence of a new era of automation called PromptOps. This advancement in AIOps technology is revolutionizing the way businesses operate, allowing for faster and more efficient application delivery.

In this article, we will explore how to integrate ChatGPT into your DevOps workflow to deliver applications.

Integrate ChatGPT into Your DevOps Workflow​

When it comes to integrating ChatGPT into DevOps workflows, many developers are faced with the challenge of managing extra resources and writing complicated shells. However, there is a better way - KubeVela Workflow. This open-source cloud-native workflow project offers a streamlined solution that eliminates the need for pods or complex scripting.

In KubeVela Workflow, every step has a type that can be easily abstracted and reused. The step-type is programmed in CUE language, making it incredibly easy to customize and use atomic capabilities like a function call in every step. An important point to note is that with all these atomic capabilities, such as HTTP requests, it is possible to integrate ChatGPT in just 5 minutes by writing a new step.

Check out the Installation Guide to get started with KubeVela Workflow. The complete code of this chat-gpt step type is available at GitHub.

Now that we choose the right tool, let's see the capabilities of ChatGPT in delivery.

Case 1: Diagnose the resources​

It's quite common in the DevOps world to encounter problems like "I don't know why the pod is not running" or "I don't know why the service is not available". In this case, we can use ChatGPT to diagnose the resource.

For example, In our workflow, we can apply a Deployment with an invalid image in the first step. Since the deployment will never be ready, we can add a timeout in the step to ensure the workflow is not stuck in this step. Then, passing the unhealthy resources deployed in the first step to the second step, we can use the chat-gpt step type to diagnose the resource to determine the issue. Note that the second step is only executed if the first one fails.

The complete workflow is shown below:

apiVersion: core.oam.dev/v1alpha1
kind: WorkflowRun
metadata:
  name: chat-gpt-diagnose
  namespace: default
spec:
  workflowSpec:
    steps:
    # Apply an invalid deployment with a timeout
    - name: apply
      type: apply-deployment
      timeout: 3s
      properties:
        image: invalid
      # output the resource to the next step
      outputs:
        - name: resource
          valueFrom: output.value

    # Use chat-gpt to diagnose the resource
    - name: chat-diagnose
      # only execute this step if the `apply` step fails
      if: status.apply.failed
      type: chat-gpt
      # use the resource as inputs and pass it to prompt.content
      inputs:
        - from: resource
          parameterKey: prompt.content
      properties:
        token: 
          value: <your token>
        prompt:
          type: diagnose

Apply this Workflow and check the result, the first step will fail because of timeout. Then the second step will be executed and the result of chat-gpt will be shown in the log:

vela workflow logs chat-gpt-diagnose

Visualize in the dashboard​

If you want to visualize the process and the result in the dashboard, it's time to enable the [velaux](https://kubevela.io/docs/reference/addons/velaux#install) addon.

vela addon enable velaux

Copy all the steps in the above yaml to create a pipeline.

Run this pipeline, and you can check out the failed reason analyzed by ChatGPT in the logs of the second step.

Write the chat-gpt step from scratch​

How to write this chat-gpt step type? Is it simple for you to write a step type like this? Let's see how to complete this step type.

We can first define what this step type need from the user. That is: the users' token for ChatGPT, and the resource to diagnose. For some other parameters like the model or the request timeout, we can set the default value with * like below:

parameter: {
  token: value: string
	// +usage=the model name
	model: *"gpt-3.5-turbo" | string
	// +usage=the prompt to use
	prompt: {
		type:    *"diagnose" | string
		lang:    *"English" | string
		content: {...}
	}
	timeout: *"30s" | string
}

Let's complete this step type by writing the logic of the step. We can first import vela/op package in which we can use the op.#HTTPDo capability to send a request to the ChatGPT API. If the request fails, the step should be failed with op.#Fail. We can also set this step's log data with ChatGPT's answer. The complete step type is shown below:

// import packages
import (
	"vela/op"
	"encoding/json"
)

// this is the name of the step type
"chat-gpt": {
	description: "Send request to chat-gpt"
	type: "workflow-step"
}

// this is the logic of the step type
template: {
  // send http request to chat gpt
	http: op.#HTTPDo & {
		method: "POST"
		url:    "https://api.openai.com/v1/chat/completions"
		request: {
			timeout: parameter.timeout
			body:    json.Marshal({
				model: parameter.model
				messages: [{
					if parameter.prompt.type == "diagnose" {
						content: """
You are a professional kubernetes administrator.
Carefully read the provided information, being certain to spell out the diagnosis & reasoning, and don't skip any steps.
Answer in  \(parameter.prompt.lang).
---
\(json.Marshal(parameter.prompt.content))
---
What is wrong with this object and how to fix it?
"""
					}
					role: "user"
				}]
			})
			header: {
				"Content-Type":  "application/json"
				"Authorization": "Bearer \(parameter.token.value)"
			}
		}
	}

	response: json.Unmarshal(http.response.body)

	fail:     op.#Steps & {
		if http.response.statusCode >= 400 {
			requestFail: op.#Fail & {
				message: "\(http.response.statusCode): failed to request: \(response.error.message)"
			}
		}
	}
	result: response.choices[0].message.content
	log:    op.#Log & {
		data: result
	}
  parameter: {
    token: value: string
    // +usage=the model name
    model: *"gpt-3.5-turbo" | string
    // +usage=the prompt to use
    prompt: {
      type:    *"diagnose" | string
      lang:    *"English" | string
      content: {...}
    }
    timeout: *"30s" | string
  }
}

That's it! Apply this step type and we can use it in our Workflow like the above.

vela def apply chat-gpt.cue

Case 2: Audit the resource​

Now the ChatGPT is our Kubernetes expert and can diagnose the resource. Can it also give us some security advice for the resource? Definitely! It's just prompt. Let's modify the step type that we wrote in the previous case to add the audit feature. We can add a new prompt type audit and pass the resource to the prompt. You can check out the whole step type in GitHub.

In the Workflow, we can apply a Deployment with nginx image and pass it to the second step. The second step will use the audit prompt to audit the resource. The complete Workflow is shown below:

apiVersion: core.oam.dev/v1alpha1
kind: WorkflowRun
metadata:
  name: chat-gpt-audit
  namespace: default
spec:
  workflowSpec:
    steps:
    - name: apply
      type: apply-deployment
      # output the resource to the next step
      outputs:
        - name: resource
          valueFrom: output.value
      properties:
        image: nginx

    - name: chat-audit
      type: chat-gpt
      # use the resource as inputs and pass it to prompt.content
      inputs:
        - from: resource
          parameterKey: prompt.content
      properties:
        token: 
          value: <your token>
        prompt:
          type: audit

Use Diagnose & Audit in one Workflow​

Now that we have the capability to diagnose and audit the resource, we can use them in one Workflow, and use the if condition to control the execution of the steps. For example, if the apply step fails, then diagnose the resource, if it succeeds, audit the resource.

The complete Workflow is shown below:

apiVersion: core.oam.dev/v1alpha1
kind: WorkflowRun
metadata:
  name: chat-gpt
  namespace: default
spec:
  workflowSpec:
    steps:
    - name: apply
      type: apply-deployment
      outputs:
        - name: resource
          valueFrom: output.value
      properties:
        image: nginx

    # if the apply step fails, then diagnose the resource
    - name: chat-diagnose
      if: status.apply.failed
      type: chat-gpt
      inputs:
        - from: resource
          parameterKey: prompt.content
      properties:
        token: 
          value: <your token>
        prompt:
          type: diagnose
        
    # if the apply step succeeds, then audit the resource
    - name: chat-audit
      if: status.apply.succeeded
      type: chat-gpt
      inputs:
        - from: resource
          parameterKey: prompt.content
      properties:
        token:
          value: <your token>
        prompt:
          type: audit

Case 3: Use ChatGPT as a quality gate​

If we want to apply the resources to a production environment, can we let ChatGPT rate the quality of the resource first, only if the quality is high enough, then apply the resource to the production environment? Absolutely!

Note that to make the score evaluated by chat-gpt more convincing, it's better to pass metrics than the resource in this case.

Let's write our Workflow. KubeVela Workflow has the capability to apply resources to multi clusters. The first step is to apply the Deployment to the test environment. The second step is to use the ChatGPT to rate the quality of the resource. If the quality is high enough, then apply the resource to the production environment.

The complete Workflow is shown below:

apiVersion: core.oam.dev/v1alpha1
kind: WorkflowRun
metadata:
  name: chat-gpt-quality-gate
  namespace: default
spec:
  workflowSpec:
    steps:
    # apply the resource to the test environment
    - name: apply
      type: apply-deployment
      # output the resource to the next step
      outputs:
        - name: resource
          valueFrom: output.value
      properties:
        image: nginx
        cluster: test
    
    - name: chat-quality-check
      # this step will always be executed
      if: always
      type: chat-gpt
      # get the inputs from resource and pass it to the prompt.content
      inputs:
        - from: resource
          parameterKey: prompt.content
      # output the score of ChatGPT and use strconv.Atoi to convert the score string to int
      outputs:
        - name: chat-result
          valueFrom: |
            import "strconv"
            strconv.Atoi(result)
      properties:
        token: 
          value: <your token>
        prompt:
          type: quality-gate

    # if the score is higher than 60, then apply the resource to the production environment
    - name: apply-production
      type: apply-deployment
      # get the score from chat-result
      inputs:
        - from: chat-result
      # check if the score is higher than 60
      if: inputs["chat-result"] > 60
      properties:
        image: nginx
        cluster: prod

Apply this Workflow and we can see that if the score is higher than 60, then the resource will be applied to the production environment.

In the End​

ChatGPT brings imagination to the world of Kubernetes. Diagnose, audit, rate is just the beginning. In the new AI era, the most precious thing is idea. What do you want to do with ChatGPT? Share your insights with us in the KubeVela Community.

Taking Over Your Existing Workloads​

Taking over existing workloads has always been a highly demanded requirement within the community, with a clear scenario: existing workloads can be naturally migrated to the OAM standard system and be managed uniformly by KubeVela's application delivery control plane. The workload takeover feature also allows reuse of VelaUX's UI console functions, including a series of operations and maintenance characteristics, workflow steps, and a rich plugin ecosystem. In version 1.7, we officially released this feature. Before diving into the specific operation details, let's first have a basic understanding of its operation mode.

"read-only" and "take-over" policy​

To meet the needs of different usage scenarios, KubeVela provides two modes for unified management. One is the "read-only" mode, which is suitable for systems that already have a self-built platform internally and still have the main control capability for existing businesses. The new KubeVela-based platform system can only observe these applications in a read-only manner. The other mode is the "take-over" mode, which is suitable for users who want to directly migrate their workloads to the KubeVela system and achieve complete unified management.

• The "read-only" mode, as the name suggests, does not perform any "write" operations on resources. Workloads managed in read-only mode can be visualized through KubeVela's toolset (such as CLI and VelaUX), which satisfies the need for unified viewing and observability. At the same time, when the managed application generated under read-only mode is deleted, the underlying workload resources will not be reclaimed. If the underlying workload is artificially modified by other controllers, KubeVela can also observe these changes.

• The "take-over" mode means that the underlying workloads will be fully managed by KubeVela, just like other workloads created directly through the KubeVela system. The update, deletion, and other lifecycle of the workloads will be fully controlled by KubeVela's application system. By default, modifications to workloads by other systems will no longer take effect and will be changed back by KubeVela's end-state control loop, unless you add other management policies (such as apply-once).

The method of declaring the take-over mode uses KubeVela's policy system, as shown below:

apiVersion: core.oam.dev/v1beta1
kind: Application
metadata:
  name: read-only
spec:
  components:
    - name: nginx
      type: webservice
      properties:
        image: nginx
  policies:
    - type: read-only
      name: read-only
      properties:
        rules:
          - selector:
              resourceTypes: ["Deployment"]

In the "read-only" policy, we have defined multiple read-only rules. For example, if the read-only selector hits a "Deployment" resource, it means that only the resources related to Deployment are read-only. We can still create and modify resources such as "Ingress" and "Service" using operational features. However, modifying the number of instances of Deployment using the "scaler" operational feature will not take effect.

apiVersion: core.oam.dev/v1beta1
kind: Application
metadata:
  name: take-over
spec:
  components:
    - name: nginx-take-over
      type: k8s-objects
      properties:
        objects:
          - apiVersion: apps/v1
            kind: Deployment
            metadata:
              name: nginx
      traits:
        - type: scaler
          properties:
            replicas: 3
  policies:
    - type: take-over
      name: take-over
      properties:
        rules:
          - selector:
              resourceTypes: ["Deployment"]

In the "take-over" policy, we also include a series of selectors to ensure that the resources being taken over are controllable. In the example above, without the "take-over" policy, the operation would fail if there is already a Deployment resource named "nginx" in the system, because the resource already exists. On one hand, the take-over policy ensures that already existing resources can be included in management when creating applications; on the other hand, it also allows the reuse of previously existing workload configurations, and only modifications to the number of instances in operational features such as "scaler" will be "patched" to the original configuration as part of the new configuration.

Use command line to take over workloads​

After learning about the take-over mode, you might wonder if there is an easy way to take over workloads with a single command. Yes, KubeVela's command line provides such a convenient way to take over workloads such as common K8s resources and "Helm". It is very easy to use. Specifically, the vela CLI automatically recognizes the resources in the system and assembles them into an application for take-over. Our core principle in designing this feature is that "taking over resources cannot trigger a restart of the underlying workloads".

As shown below, by default, using vela adopt will manage the resources in "read-only" mode. Simply specify the type, namespace, and name of the native resources you want to take over, and an Application object for takeover will be automatically generated. The generated application spec is strictly consistent with the actual fields in the cluster.

$ vela adopt deployment/default/example configmap/default/example
apiVersion: core.oam.dev/v1beta1
kind: Application
metadata:
  labels:
    app.oam.dev/adopt: native
  name: example
  namespace: default
spec:
  components:
  - name: example.Deployment.example
    properties:
      objects:
      - apiVersion: apps/v1
        kind: Deployment
        metadata:
          name: example
          namespace: default
        spec:
          replicas: 1
          selector:
            matchLabels:
              app: example
          template:
            metadata:
              labels:
                app: example
            spec:
              containers:
              - image: nginx
                imagePullPolicy: Always
                name: nginx
              restartPolicy: Always
          ...
    type: k8s-objects
  - name: example.config
    properties:
      objects:
      - apiVersion: v1
        kind: ConfigMap
        metadata:
          name: example
          namespace: default
    type: k8s-objects
  policies:
  - name: read-only
    properties:
      rules:
      - selector:
          componentNames:
          - example.Deployment.example
          - example.config
    type: read-only

The currently supported default takeover types and their corresponding resource API names are as follows:

• crd: ["CustomResourceDefinition"]
• ns: ["Namespace"]
• workload: ["Deployment", "StatefulSet", "DaemonSet", "CloneSet"]
• service: ["Service", "Ingress", "HTTPRoute"]
• config: ["ConfigMap", "Secret"]
• sa: ["ServiceAccount", "Role", "RoleBinding", "ClusterRole", "ClusterRoleBinding"]
• operator: ["MutatingWebhookConfiguration", "ValidatingWebhookConfiguration", "APIService"]
• storage: ["PersistentVolume", "PersistentVolumeClaim"]

If you want to change the application to takeover mode and deploy it directly to the cluster, just add a few parameters:

vela adopt deployment/default/example --mode take-over --apply

In addition to native resources, the vela command line also supports taking over workloads created by Helm applications by default.

vela adopt mysql --type helm --mode take-over --apply --recycle -n default

The above command will manage the "mysql" Helm release in the "default" namespace through "take-over" mode. Specifying --recycle will clean up the original Helm release metadata after a successful deployment.

Once the workloads have been taken over, the corresponding KubeVela Applications have been generated, and the related operations have been integrated with the KubeVela system. You can see the taken-over applications on the VelaUX interface, and also view and operate the applications through other vela command line functions.

You can also use a command to take over all the workloads in your namespace in batches. Based on KubeVela's resource topology capabilities, the system will automatically recognize the associated resources and form a complete application. For custom resources such as CRDs, KubeVela also supports custom rules for association relationships.

vela adopt --all --apply

This command will automatically recognize the resources and their association relationships in the current namespace based on the built-in resource topology rules, and take over the applications accordingly. Taking a Deployment as an example, the automatically taken over application looks like the following, which not only takes over the main workload Deployment but also its corresponding resources, including ConfigMap, Service, and Ingress.

apiVersion: core.oam.dev/v1beta1
kind: Application
metadata:
  name: test2
  namespace: default
spec:
  components:
  - name: test2.Deployment.test2
    properties:
      objects:
      - apiVersion: apps/v1
        kind: Deployment
        metadata:
          name: test2
          namespace: default
        spec:
          ...
    type: k8s-objects
  - name: test2.Service.test2
    properties:
      objects:
      - apiVersion: v1
        kind: Service
        metadata:
          name: test2
          namespace: default
        spec:
          ...
    type: k8s-objects
  - name: test2.Ingress.test2
    properties:
      objects:
      - apiVersion: networking.k8s.io/v1
        kind: Ingress
        metadata:
          name: test2
          namespace: default
        spec:
          ...
    type: k8s-objects
  - name: test2.config
    properties:
      objects:
      - apiVersion: v1
        kind: ConfigMap
        metadata:
          name: record-event
          namespace: default
    type: k8s-objects
  policies:
  - name: read-only
    properties:
      rules:
      - selector:
          componentNames:
          - test2.Deployment.test2
          - test2.Service.test2
          - test2.Ingress.test2
          - test2.config
    type: read-only

The demonstration result is shown below:

If you want to use custom resource topology relationships to take over custom resources, you can use the following command:

vela adopt <your-crd> --all --resource-topology-rule=<your-rule-file.cue>

Flexible Definition of Takeover Rules​

Given KubeVela's highly extensible design principles, the workloads and takeover methods faced by resource takeover are also different. Therefore, we have also designed a fully extensible and programmable way to take over workloads. In fact, the one-click takeover capability of the command line is also based on KubeVela's extensible takeover rules as a special case. The core idea is to define a configuration transformation rule through CUE, and then specify the transformation rule when executing the vela adopt command, as shown below.

vela adopt deployment/my-workload --adopt-template=my-adopt-rule.cue

This mode is only suitable for advanced users, and we will not go into too much detail here. If you want to learn more details, you can refer to the official documentation on workload takeover.

Significant Performance Optimization​

Performance optimization is also a major highlight of this version. Based on the practical experience of different users in the community, we have improved the overall performance of the controller, the capacity of a single application, and the overall application processing throughput by 5 to 10 times without changing the default resource quotas. This also includes some changes to default configurations, with parameter trimming for some niche scenarios that affect performance.

In terms of single application capacity, because KubeVela applications may contain a large number of actual Kubernetes APIs, this often leads to the ResourceTracker behind the application that records the actual resource status and the ApplicationRevision object that records version information exceeding the 2MB limit of a single Kubernetes object. In version 1.7, we have added zstd compression functionality and enabled it by default, which directly compresses the size of resources by nearly 10 times. This also means that the resource capacity that a single KubeVela Application can support has increased by 10 times.

In addition, for some scenarios such as recording application and component versions, these version records themselves will increase proportionally with the number of applications, such as default recording of 10 application versions, which will increase by a factor of ten with the number of applications. Due to the list-watch mechanism of the controller itself, these additional objects will occupy controller memory, leading to a significant increase in memory usage. Many users (such as GitOps users) may have their own version management system. In order to avoid waste of memory, we have changed the default upper limit of application version records from 10 to 2. For component versions, which are relatively niche, we have disabled them by default. This reduces the controller's overall memory consumption to one-third of the original.

In addition, some parameter adjustments have been made, including reducing the number of historical versions recorded in the Definition from 20 to 2, and increasing the default Kubernetes API interaction limit QPS from 100 to 200, among others. In future versions, we will continue to optimize the performance of the controller.

Usability Improvement​

In addition to the core feature updates and performance improvements, this release also includes enhancements to the usability of many features.

Client-side multi-environment resource rendering​

"Dry run" is a popular concept in Kubernetes, which refers to the practice of running resources in a dry, non-destructive mode before they are actually applied to the cluster, to check if the configurations are valid. KubeVela also provides this feature, which not only checks if resources can be run, but also translates the abstract applications of OAM into the Kubernetes native API, allowing the client to convert from application abstraction to actual resources. The new feature added in version 1.7 is to specify different files for dry run, generating different actual resources.

For example, we can write different policy and workflow files for different environments, such as "test-policy.yaml" and "prod-policy.yaml". In this way, the same application can be specified with different policies and workflows in the client, generating different underlying resources, such as:

• Running in test environment

vela dry-run  -f app.yaml -f test-policy.yaml -f test-workflow.yaml

• Running in production environment

vela dry-run  -f app.yaml -f prod-policy.yaml -f prod-workflow.yaml

Here is the content of app.yaml, which specifies an external workflow to be referenced:

# app.yaml
apiVersion: core.oam.dev/v1beta1
kind: Application
metadata:
  name: first-vela-app
spec:
  components:
    - name: express-server
      type: webservice
      properties:
        image: oamdev/hello-world
        ports:
         - port: 8000
           expose: true
      traits:
        - type: scaler
          properties:
            replicas: 1
  workflow:
    ref: deploy-demo

And the content of prod-policy.yaml and prod-workflow.yaml are as follows:

apiVersion: core.oam.dev/v1alpha1
kind: Policy
metadata:
  name: env-prod
type: topology
properties:
  clusters: ["local"]
  namespace: "prod"
---
apiVersion: core.oam.dev/v1alpha1
kind: Policy
metadata:
  name: ha
type: override
properties:
  components:
  - type: webservice
    traits:
    - type: scaler
      properties:
        replicas: 2

apiVersion: core.oam.dev/v1alpha1
kind: Workflow
metadata:
  name: deploy-demo
  namespace: default
steps:
  - type: deploy
    name: deploy-prod
    properties:
      policies: ["ha", "env-prod"]

The corresponding YAML files for the test environment can be modified in the same way by changing the parameters. This feature is particularly useful for users who use KubeVela as a client abstraction tool and combine it with tools such as Argo to synchronize resources.

Enhanced Application Deletion Feature​

In many special scenarios, deleting applications has always been a painful experience. In version 1.7, we have added some convenient ways to support smooth application deletion for various special cases.

• Deleting certain workloads in case of cluster disconnection: We provide an interactive method to delete resources, which allows users to select the underlying workloads by viewing the cluster name, namespace, and resource type, to remove resources involved in special scenarios such as cluster disconnection.

• Retain underlying resources when deleting an application: If you only want to delete the metadata of the application while keeping the underlying workload and configuration, you can use the --orphan flag to retain the underlying resources when deleting the application.

• Deleting application when controller is uninstalled: When you have uninstalled the KubeVela controller but found some remaining applications that were not cleaned up, you can use --force flag to delete these applications.

Custom output after addon installation​

For the KubeVela plugin system, we have added a NOTES.cue file that allows plugin makers to dynamically output installation completion prompts. For example, for the Backstage plugin, the NOTES.cue file is as follows:

info: string
if !parameter.pluginOnly {
	info: """
		By default, the backstage app is strictly serving in the domain `127.0.0.1:7007`, check it by:
		            
		    vela port-forward addon-backstage -n vela-system
		
		You can build your own backstage app if you want to use it in other domains. 
		"""
}
if parameter.pluginOnly {
	info: "You can use the endpoint of 'backstage-plugin-vela' in your own backstage app by configuring the 'vela.host', refer to example https://github.com/wonderflow/vela-backstage-demo."
}
notes: (info)

This plugin's output will be displayed with different content based on the parameters used by the user during plugin installation.

Enhanced Workflow Capabilities​

In version 1.7, we have enhanced the workflow capabilities with more granular options:

• Support specifying a failed step for retry

vela workflow restart <app-name> --step=<step-name>

• The step name of a workflow can be left blank, and it will be generated automatically by the webhook.
• The parameter passing in workflows now supports overriding existing parameters.

In addition, we have added a series of new workflow steps in this version, including the typical built-push-image step that allows users to build an image and push it to a registry within the workflow. During the execution of the workflow steps, users can check the logs of a specific step using the vela workflow logs <name> --step <step-name> command. The full list of built-in workflow steps can be found in the official documentation.

Enhanced VelaUX Capabilities​

The VelaUX console has also been enhanced in version 1.7, including:

• Enhanced application workflow orchestration capabilities, supporting full workflow capabilities including sub-steps, input/output, timeouts, conditional statements, step dependencies, etc. Application workflow status viewing is also more comprehensive, with historical workflow records, step details, step logs, and input/output information available for query.
• Support for application version regression, allowing users to view the differences between multiple versions of an application and select a specific version to roll back to.
• Support for multi-tenancy, with stricter limits on multi-tenant permissions aligned with the Kubernetes RBAC model.

What's Next​

Recently, the official version of KubeVela 1.8 is also in full swing and is expected to meet you at the end of March. We will further enhance the following aspects:

Enhance the scalability and stability of the KubeVela core controller, provide a sharding solution for controller horizontal scaling, optimize the controller performance and mapping for 10,000-level application scale in multi-cluster scenarios, and provide a new performance evaluation for the community. VelaUX supports out-of-the-box gray release capabilities, and interfaces with observability plugins for interactive release processes. At the same time, VelaUX forms an extensible framework system, providing configuration capabilities for customizing UI and supporting business custom extensions and interfaces. Enhance the GitOps workflow capabilities to support the complete VelaUX experience for applications synchronized with Git repositories. If you want to learn more about our plans, become a contributor, or partner with us, you can contact us through community communication (https://github.com/kubevela/community), and we look forward to your participation!

Originally post in CNCF.

The CNCF Technical Oversight Committee (TOC) has voted to accept KubeVela as a CNCF incubating project.

KubeVela is an application delivery engine built with the Kubernetes control plane that makes deploying and operating applications across hybrid and multi-cloud environments easier, faster, and more reliable. KubeVela can orchestrate, deploy, and operate application components and cloud resources with a workflow-based application delivery model. The application delivery abstraction of KubeVela is powered by the Open Application Model (OAM).

The KubeVela project evolved from oam-kubernetes-runtime and developed with bootstrap contributions from more than eight different organizations, including Alibaba Cloud, Microsoft, Upbound and more. It was publicly announced as an open source project in November 2020, released as v1.0 in April 2021, and accepted as a CNCF sandbox project in June 2021. The project has more than 260 contributors and committers across multiple continents from organizations like Didi, JD.com, JiHu GitLab, SHEIN, and more.

“KubeVela pioneered a path for delivering applications across multi-cloud/multi-cluster environments with unified yet extensible abstractions,” said Lei Zhang, CNCF TOC sponsor. “This innovation unlocked the next-generation software delivery experience and filled the ‘last mile’ gap in existing practices which focus on the ‘deploy’ stage rather than ‘orchestrating’. We are excited to welcome more app-centric tools/platforms in the CNCF community and look forward to watching the adoption of KubeVela grow to a new level in the fast-growing application delivery ecosystem.”

KubeVela is used in production today by multiple companies across all major public clouds and on-premises deployments. Most users are adopting KubeVela as their internal “PaaS”, as part of their CI/CD pipeline, or as an extensible DevOps kernel for building their own IDP. Public adopters include Alibaba, which uses KubeVela as the core to deliver and manage applications across hybrid environments; Bytedance, which uses KubeVela and Crossplane to provide advanced Gaming PaaS abilities; China Merchants Bank, which uses KubeVela to build a hybrid cloud application platform to unify the whole process from build, ship and run; and many more.

“We’ve found capabilities such as gateway, security, and observability are being standardized with the emergence of corresponding open source tools and cloud services,” said Fang SITU, leader of aPaaS and Serverless team at Alibaba Cloud. “These capabilities can be integrated so that developers can self-serve, easily configure, automatically trigger, and get faster feedback, thereby greatly improving the application development efficiency. KubeVela is very suitable for the integration and customization of this kind of application delivery process, and it is the best practice of Platform Engineering.”

“KubeVela enables China Merchants Bank to quickly establish a large-scale unified OAM cloud native application management platform, reducing the complexity of cloud for FinTech and accelerating the standardized development and delivery of modern applications,” said Jiahang Xu, Senior Architect at China Merchants Bank and KubeVela maintainer. “KubeVela provides an application model and programmable CRD, workflow orchestration application delivery, observability, and configuration capabilities. KubeVela empowers cloud native applications along with the CNCF ecosystem.”

Main Components:

• Vela Core is the main component of KubeVela, also known as the KubeVela Control Plane. It provides operators and webhooks for rendering, orchestrating and delivering OAM applications.
• The Vela Workflow engine translates CUE-based steps and executes them. It’s a common library that can work as a standalone engine or run inside a KubeVela application.
• KubeVela CLI provides various commands that help you to operate applications, such as managing definitions, viewing resources, restarting workflow, and rolling versions.
• VelaUX is the Web UI for KubeVela. It incorporates business logic into fundamental APIs and provides out-of-box user experiences for non-k8s-expert users.
• The Terraform Controller in KubeVela allows users to use Terraform to manage cloud resources through Kubernetes Custom Resources.
• The Cluster Gateway provides a unified multi-cluster access interface.
• KubeVela also has a growing catalog with more than 50 community add-ons for integrations, including ArgoCD, FluxCD, Backstage, OpenKruise, Dapr, Crossplane, Terraform, OpenYurt and more.

Notable Milestones:

• >4.7k GitHub Stars
• >3.5k pull requests
• >1.6k issues
• >290 contributors
• >150 Releases

Since joining the CNCF Sandbox, KubeVela has released seven minor versions to v1.7 and added five new components, including standalone workflow, VelaUX, ClusterGateway, VelaD, and Vela Prism. The number of contributors has tripled from 90+ to 290+, GitHub stars have doubled from 1900+ to 4700+, and contributing organizations have tripled from 20+ to 70+.

“KubeVela improved the developer experience when it comes to complex multi-cloud environments with their modern open source software delivery control plane,” said Chris Aniszczyk, CNCF CTO. “We look forward to supporting the community in its growth and maturity towards a graduated project.”

Looking forward, the community plans to improve the user experience for cloud resource provisioning and consumption with delivery workflow, enhance the security for the whole CI/CD delivery process in hybrid/multi-cluster scenarios, support the KubeVela Dynamic API that allows users to make integration with third-party APIs easily, and more. Visit the roadmap to learn more.

“We’re humbled and grateful for the trust and support from users and contributors,” said Jianbo Sun, Staff Engineer at Alibaba Cloud and KubeVela maintainer. “KubeVela’s highly extensible design is very suitable for the diverse user scenarios of the community, bringing us a powerful engine to the ecosystem for application delivery. Thanks to the support and endorsement from CNCF, I believe that reaching incubation status is an important milestone for the project. The KubeVela maintainers look forward to collaborating with CNCF in our goals to make deploying and operating applications across today’s hybrid environments easier, faster and more reliable.”

“Thanks to Kubevela, the way we deploy and manage applications in Kubernetes has now become more accessible,” Daniel Higuero, CTO at Napptive and KubeVela maintainer. “The use of Application and Workflow as top-level entities greatly simplifies common processes on top of Kubernetes. The strength of the approach lies in its ability to simplify basic use cases while enabling complex ones with multi-tenant and multi-cluster support. This is combined with an inherently extensible system which supports community add-ons, allowing it to integrate with other tools and add custom definitions to tailor the experience to your use case.”

“The CNCF community has incubated a large number of cloud native operation and atomic management capabilities,” said QingGuo Zeng, Technical Expert at Alibaba Cloud and KubeVela maintainer. “We hope to integrate various capabilities through a unified, application-centric concept and help more and more platform developers easily implement standardized applications in enterprises. KubeVela is growing into a powerful helper for enterprises to practice Platform Engineering.”

“KubeVela aims to provide convenience and advancements of the rich cloud native infrastructures to various industries,” said Da Yin, Senior Engineer at Alibaba Cloud and KubeVela maintainer. “To meet the modern application delivery demands, KubeVela is always exploring extensible and flexible architecture and adding pioneering ideas, including multi-cluster delivery, programmable workflow, and automated observability. KubeVela also continuously cares for the security and stability of the control plane, which builds up production confidence for community adopters. We expect the openness of KubeVela could make it a frontier explorer in the cloud native era.”

As a CNCF-hosted project, KubeVela is part of a neutral foundation aligned with its technical interests and the larger Linux Foundation, which provides governance, marketing support, and community outreach. The project joins 35 other incubating technologies, including Backstage, Cilium, Istio, Knative, OpenTelemetry, and more. For more information on maturity requirements for each level, please visit the CNCF Graduation Criteria.

This article describes how to use KubeVela Workflow to upgrade the architecture of SAE and interprets multiple practice scenarios.

Challenges in the Serverless Era​

SAE is an application hosting platform for business application architecture and microservices. It is a Kubernetes-based cloud product that combines the Serverless architecture and the microservice model.

As shown in the preceding architecture diagram, SAE users can host multiple types of applications on SAE. At the underlying layer of SAE, the Java business layer processes the relevant business logic and interacts with Kubernetes resources. At the bottom, it relies on highly available, O&M-free, and pay-as-you-go elastic resource pools.

In this architecture, SAE mainly relies on its Java business layer to provide users with capabilities. This architecture helps users easily deploy applications. Still, at the same time, it brings some challenges as well.

With the continuous development of Serverless, SAE has encountered three major challenges:

1. Engineers in SAE have some complex and non-standardized operations. How can we automate these complex operations to reduce development consumption?
2. With the development of business, the application-delivering capability of SAE is favored by a large number of users. The growth of users also brings efficiency challenges.** How can we optimize the existing delivery capability and improve efficiency in high concurrency?**
3. As Serverless architecture continues to be implemented in enterprises, numerous of cloud vendors are making their product systems serverless. In such a wave, how can SAE quickly connect with internal Serverless capabilities and reduce development costs while launching new features?

The preceding three challenges show that SAE needs some kind of orchestration engine to upgrade the delivery process, integrate with internal capabilities, and automate operations.

So, what does this orchestration engine need to meet to solve these challenges?

1. High Scalability: The steps in the process need to be highly scalable for this orchestration engine. Only in this way can the originally non-standardized and complex operations be standardized with the steps. In the meanwhile, the steps can combine with the process control capability of the orchestration engine, thus reducing the consumption of development resources.
2. Lightweight and Efficient: This orchestration engine must be efficient and production-ready, so as to meet the high concurrency requirements of SAE in large-scale user scenarios.
3. Strong Integration and Process Control Capabilities: This orchestration engine needs to be able to quickly integrate with the atomic functions in the company. If the glue code that connects upstream and downstream capabilities can be transformed into the process in the orchestration engine, development costs can be reduced.

Based on these challenges and considerations, the SAE team and the KubeVela community have conducted in-depth cooperation and launched the KubeVela Workflow project as an orchestration engine.

Why Use Kubevela Workflow?​

Thanks to the booming ecological of cloud-native, there are already many mature workflow projects in the community (such as Tekton, Argo, etc.). There are also some orchestration engines within Alibaba Cloud. So, why reinvent the wheel instead of using existing technology?

This is because KubeVela Workflow has a fundamental difference in design. The steps in the workflow are designed for the cloud-native IaC system and support abstract encapsulation and reuse, which means that you can use atomic capabilities like a function call in every step, instead of just creating pods or containers.

In KubeVela Workflow, each step has a step type, and each step type corresponds to the resource callend WorkflowStepDefinition. You can use the CUE language (an IaC language, which is a superset of JSON) to write this step definition or directly use the step type defined in the community.

You can simply regard a WorkflowStepDefinition as a function declaration. Each time a new step type is defined, a new function is defined. The function requires some input parameters, and the step definition is the same. In the step definition, you can define the input parameters required for this step type in the parameter field. When the workflow is running, the workflow controller executes the CUE code in the corresponding step definition using the actual parameter values input by the user, just as it executes your custom function.

With such a layer of abstraction of the steps, it adds huge possibilities to the steps:

• If you want to customize the step type (just like writing a new function), you can import the official code packages in the step definition, thus introducing other atomic capabilities into the step, including HTTP requests, CRUD resources in multiple clusters, conditional waiting, etc. With such a programmable step type, you can easily integrate with any system. For example, in the SAE scenario, the step definition is integrate with other internal atomic capabilities (such as MSE, ACR, ALB, SLS, etc.). Then, the workflow orchestration capability is used to control the process.

• If you only want to use defined steps, just like calling a packaged third-party function, the only thing you need to care about is your input parameters and the step type you want. For example, in a typical image-building scenario, first, specify the step type as build-push-image and then specify your input parameters: the code source and branch of the built image, the name of the built image, and the secret key of the image repository to push.

apiVersion: core.oam.dev/v1alpha1
kind: WorkflowRun
metadata:
  name: build-push-image
  namespace: default
spec:
  workflowSpec:
   steps:
    - name: build-push
      type: build-push-image
      properties:
        context:
          git: github.com/FogDong/simple-web-demo
          branch: main
        image: fogdong/simple-web-demo:v1
        credentials:
          image:
            name: image-secret

In such an architecture, the abstraction of the step brings infinite possibilities to the workflow. When you need to add a step to the workflow, you no longer need to compile-build-package the business code and then use the pod to execute the code. You only need to modify the configuration code in the step definition (together with the workflow engine's orchestration and control capability) to integrate with new features.

This is also the main reason why SAE chooses Kubevela Workflow. Based on scalability, we can fully leverage the power of ecology and accelerate product upgrades.

Cases​

Next, let's go deeper into the user cases in SAE.

Case 1: Automated Operations​

The first scenario is an automated operations scenario for SREs in SAE.

In SAE, we write and update some base images for users. We need to preload these images to multiple clusters in different regions to provide a better experience for users who use these base images.

The original operation process is very complicated. It involves building images and pushing them across multiple regions using ACR, as well as creating image cache templates and managing those image caches. Regions here include Shanghai, US West, Shenzhen, and Singapore. These operations are non-standardized and time-consuming. This is because when an SRE needs to push these images from the local to foreign clusters, it is likely to fail due to network problems. Therefore, he needs to disperse his energy on these operations that could otherwise be automated.

This is also the scenario that KubeVela Workflow is suitable for: each step in these operations can be converted into a step in the workflow programmatically, so that these steps can be orchestrated and reused. In addition, KubeVela Workflow provides a visual dashboard. SREs only need to configure a pipeline template once, and can automate the process by triggering execution or by entering specific runtime parameters each time the execution is triggered.

The simplified steps is like below:

1. Use the HTTP request step type to build an image by requesting the service of ACR and pass the image ID to the next step through inputs/outputs. In this step definition, you need to wait until the ACR service is done before ending the execution of the current step.
2. If the first step fails, execute the error handle step.
3. If the first step is successfully built, use the HTTP request step to build image cache, at the same time, the service logs are used as the source of the current step. Here you can view the logs of the steps directly in the Dashboard to troubleshoot problems.
4. Use a step group with deploy step type to preload images for clusters in the China (Shanghai) region and the US (West) region. The multi-cluster management and control capabilities of KubeVela Workflow are used to distribute the ImagePullJob workload to multiple clusters to preload images.

In the preceding process, if you do not use KubeVela Workflow, you may need to write a bunch of business code to connect multiple services and clusters. Let’s take the last step of distributing the ImagePullJob workload to multiple clusters as an example: Not only do you need to manage the configuration of multiple clusters, but also need to watch the status of the workload (CRD) until the status of the workload becomes Ready before proceeding to the next step. This process actually corresponds to a simple Kubernetes Operator's reconcile logic: first create or update a resource, if the status of the resource is as expected, then end the reconcile, if not, continue to wait.

Do we need to implement an Operator for every new resource management in our operations? Is there any convenient way to free us from the complicated Operator development?

It is precisely because of the programmability of steps in KubeVela Workflow that it can completely cover these operations and resource management in SAE scenarios, which can help engineers reduce manpower consumption. Similar to the above logic, the step definition corresponding to KubeVela Workflow is very simple. No matter what kind of resources (or a HTTP interface request), it can be covered by a similar step template like:

template: {
  // First, read resources from the specified cluster
	read: op.#Read & {
  	value: {
    	apiVersion: parameter.apiVersion
    	kind: parameter.kind
    	metadata: {
      	name: parameter.name
      	namespace: parameter.namespace
    	}
  	}
  	cluster: parameter.cluster
	}
	// Second, wait until the resource is Ready. Otherwise, the step keeps waiting
	wait: op.#ConditionalWait & {
  	continue: read.value.status != _|_ && read.value.status.phase == "Ready"
	}
	// Third(optional), if the resource is Ready, then...
	// Custom Logic...
	
	// Users must input the defined parameter when using the step type
	parameter: {
  	apiVersion: string
  	kind: string
  	name: string
  	namespace: *context.namespace | string
  	cluster: *"" | string
 	}
}

Corresponding to the current case is:

• First: Read the status of ImagePullJob in a specified cluster, such as the cluster in the China (Shanghai) region.
• Second: If the ImagePullJob is ready and the image has been preloaded, continue to execute.
• **Third(Optional): **After the ImagePullJob is ready, clean up the ImagePullJob in the cluster.

Like this, no matter how many Region clusters or new type resources are added in the subsequent O&M scenarios, you can let KubeVela Workflow manage the cluster's KubeConfig, and use the defined step types with different cluster names or resource types to achieve a simplified Kubernetes Operator Reconcile process to reduce development costs.

Case 2: Optimize the Existing Delivery Process​

In addition to automating internal O&M operations, upgrading the original SAE product architecture to improve delivery efficiency for users is also an important reason of choosing KubeVela Workflow.

Original Architecture​

In the delivery scenario of SAE, a delivery process corresponds to a series of tasks, such as: initializing the environment, building images, releasing in batches, etc. This series of tasks corresponds to the SAE Tasks in the figure below.

These tasks are sequentially thrown to Java Executor for business logic in the original architecture of SAE, such as creating resources in Kubernetes, and synchronizing the status of current tasks with the MySQL database, etc.

After the current task is completed, JAVA Executor will get the next task from SAE's original orchestration engine, and at the same time, the orchestration engine will continuously put new tasks into the initial task list.

The biggest problem in this old architecture is the polling call. JAVA Executor will get it from the SAE task list every second to check whether there are new tasks; at the same time, after JAVA Executor creates Kubernetes resources, it will attempts to get the status of resources from the cluster every second.

The original architecture of SAE is not the controller model in the Kubernetes ecosystem, but the polling model. If the orchestration engine layer is upgraded to the controller model for event watching, it can better integrate with the entire Kubernetes ecosystem and improve efficiency.

However, the logic coupling of business is deep in the original architecture. If the traditional container-based cloud-native workflow is used, SAE needs to package the original business logic into images and maintain and update a large number of images, which is not a sustainable path. We hope the upgraded workflow engine can be easily integrated with the task orchestration, business execution layer, and Kubernetes cluster.

New Architecture​

With the high scalability of KubeVela Workflow, SAE engineers do not need to repackage the original capabilities into images or make large-scale modifications.

The new process is shown in the preceding figure. After a delivery paln is created on the SAE product side, the business side writes the model to the database, converts the model, and generates a KubeVele workflow, which corresponds to YAML on the right side.

At the same time, SAE's original Java Executor provides the original business capabilities as microservice APIs. When KubeVela Workflow is executed, each step is IaC-based, and the underlying implementation is in the CUE language. Some of these steps will call the business microservice API of SAE, while others will directly interact with the underlying Kubernetes resources. Data can be transferred between steps. If there is an error in the call, you can use the conditional judgment of the step to handle the error.

Such optimization is scalable and fully reuses the Kubernetes ecosystem. It extends workflow processes and atomic capabilities and is oriented to the final state. This combination of scalability and process control can cover the original business functions and reduce the amount of development. At the same time, the state update latency is reduced from the minute level to the millisecond level, which is agile and native. It has the YAML-level description capability but also improves the development efficiency from 1d to 1h.

Case 3: Launch New Features Quickly​

In addition to automated O&M and upgrading the original architecture, what else can KubeVela Workflow provide?

The reusability of steps and the ease of integration with other ecosystems bring more surprises to SAE in addition to upgrades: from writing business code to orchestrating different steps, so as to launch new product features quickly!

SAE has accumulated a large amount of JAVA foundation and supports a wealth of functions, such as: supporting single-batch release, multi-batch release, and canary release for JAVA microservices. However, with the increase of customers, some customers have put forward new requirements, and they hope to have the ability to publish multilingual north-south traffic in canary release.

There are also many mature open-source products for canary releases, such as Kruise Rollout. After investigation, SAE engineers found that Kruise Rollout can be used to complete the ability of canary release, and it can cooperate with the internal ingress controller(ALB) of Alibaba Cloud to split different traffic.

Such a solution is shown in the architecture diagram above. SAE distributes a KubeVela Workflow, and the steps in the Workflow will integrate with Alibaba Cloud ALB, open-source Kruise Rollout, and SAE's business components at the same time. Batch management is completed in the steps, thus completing the rapid launch of features.

In fact, after using KubeVela Workflow, it is no longer necessary to write new business codes for this feature. It is only necessary to write a step type for updating canary batches.

Due to the programmability of step types, we can easily use different patch policies in the definition to update the release batches of Rollout objects in the different clusters. Moreover, in the workflow, the step type of each step is reusable. This means when you develop a new step type, you are laying the foundation for the next time a new feature is launched. This reuse allows you to launch features quickly and reduce development costs.

Summary​

After SAE upgraded the KubeVela architecture, it improved delivery efficiency and reduced development costs. Based on the advantages of its underlying reliance on Serverless infrastructure, it can give full play to the advantages of the product in application hosting.

In addition, the architecture upgrade solution of KubeVela Workflow in SAE is an open-source solution that can be replicated. The community provides more than 50 built-in step types (including image building, image pushing, image scanning, and multi-cluster deployment, using Terraform to manage infrastructure, condition wait, message notification, etc) to help you open up CI/CD easily.

You can refer to the following documents for more usage scenarios:

• Build images, push images, and deploy resources
• Orchestrate multiple KubeVela Applications
• Initialize the Environment with One Click, use Terraform to pull up clusters, manage clusters, and distribute resources to new clusters
• Call the specified service and send a notification of the returned result through data passing
• Use different context parameters to control the deployment of resources

The End​

You can learn more about KubeVela and the OAM project through the following materials:

• Project Code Library: https://github.com/kubevela/kubevela Welcome to Star/Watch/Fork!
• Workflow Code Library: https://github.com/kubevela/workflow Welcome to Star/Watch/Fork!
• Official Project Homepage and Documents: kubevela.io
• Slack：CNCF #kubevela Channel

Background​

With the popularization of the Internet of Everything scenario, the computing power of edge devices is increasing. It is a new technological challenge to use the advantages of cloud computing to meet complex and diversified edge application scenarios and extend cloud-native technology to the end and edge. Cloud-Edge Collaboration is becoming a new technological focus. This article will focus on KubeVela and OpenYurt (two open-source projects of CNCF) and introduce the solution of cloud-edge collaboration in a practical Helm application delivery scenario.

OpenYurt focuses on extending Kubernetes to edge computing in a non-intrusive manner. Based on the container orchestration and scheduling capabilities of native Kubernetes, OpenYurt integrates edge computing power into the Kubernetes infrastructure for unified management. It provides capabilities (such as edge autonomy, efficient O&M channels, unitized edge management, edge traffic topology, secure containers, and edge Serverless/FaaS) and support for heterogeneous resources. In short, OpenYurt builds a unified infrastructure for cloud-edge collaboration in a Kubernetes-native manner.

Incubated in the OAM model, KubeVela focuses on helping enterprises build unified application delivery and management capabilities. It shields the complexity of underlying infrastructure for developers and provides flexible scaling capabilities. It also provides out-of-the-box microservice container management, cloud resource management, versioning and canary release, scaling, observability, resource dependency orchestration and data delivery, multi-cluster, CI docking, and GitOps. Maximize the R&D performance of developer self-service application management, which also meets the extensibility demands of the long-term evolution of the platform.

OpenYurt + KubeVela - What Problems Can be Solved?​

As mentioned before, OpenYurt supports the access of edge nodes, allowing users to manage edge nodes by operating native Kubernetes. "Edge nodes" are used to represent computing resources closer to users (such as virtual machines or physical servers in a nearby data center). After you add them through OpenYurt, these edge nodes are converted into nodes that can be used in Kubernetes. OpenYurt uses NodePool to describe a group of edge nodes in the same region. After basic resource management is met, we have the following core requirements for how to orchestrate and deploy applications to different NodePools in a cluster.

1. Unified Configuration: If you manually modify each resource to be distributed, there are many manual interventions, which are prone to errors. We need a unified way to configure parameters, which can not only facilitate batch management operations, but also connect security and audit to meet enterprise risk control and compliance requirements.
2. Differential Deployment: Most workloads deployed to different NodePools have the same attributes, but there are still personalized configuration differences. The key is how to set the parameters related to node selection. For example, NodeSelector can instruct the Kubernetes scheduler to schedule workloads to different NodePools.
3. Scalability: The cloud-native ecosystem is booming. Both workload types and different O&M functions are growing. We need the overall application architecture to be scalable so that we can fully benefit from the dividends of the cloud-native ecosystem and meet business needs flexibly.

KubeVela and OpenYurt can complement each other at the application layer to meet the preceding three core requirements. Next, I will show these functions with the operation process.

Deploy an Application to the Edge​

We will use the Ingress controller as an example to show how to use KubeVela to deploy applications to the edge. We want to deploy the Nginx Ingress controller to multiple NodePools to access the services provided by the specified NodePool through the edge Ingress. An Ingress can only be handled by the Ingress controller in the NodePool.

The cluster in the schematic diagram contains two NodePools: Beijing and Shanghai. The networks between them are not interconnected. We want to deploy an Nginx Ingress Controller, which can act as the network traffic ingress for each NodePool. A client close to Beijing can access the services provided in the Beijing NodePool by accessing the Ingress Controller of the Beijing NodePool and does not access the services provided in the Shanghai NodePool.

Basic Environment of Demo​

We will use Kubernetes clusters to simulate edge scenarios. The cluster has three nodes, and their roles are:

• Node 1: master node, cloud node
• Node 2: worker node, edge node, in NodePool Beijing
• Node 3: worker node, edge node, in NodePool Shanghai

Preparations​

1. Install YurtAppManager​

YurtAppManager is the core component of OpenYurt. It provides NodePool CRD and controller. There are other components in OpenYurt, but we only need YurtAppManager in this tutorial.

git clone https://github.com/openyurtio/yurt-app-managercd yurt-app-manager && helm install yurt-app-manager -n kube-system ./charts/yurt-app-manager/

2. Install KubeVela and Enable the FluxCD Addon​

Install the Vela command-line tool and install KubeVela in the cluster:

curl -fsSl https://kubevela.io/script/install.sh | bash
vela install

We want to reuse the mature Helm charts provided by the community, so we use Helm-type components to install the Nginx Ingress Controller. In KubeVela with microkernel design, Helm components are provided by the FluxCD addon. The following enables the FluxCD addon.

vela addon enable fluxcd

3. Prepare NodePool​

Create two NodePools: Beijing and Shanghai. Dividing NodePools by region is a common pattern in actual edge scenarios. Different groups of nodes often have obvious isolation attributes (such as network disconnection, no resource sharing, resource heterogeneity, and application independence). This is the origin of the NodePool concept. In OpenYurt, features (such as NodePools and service topologies) are used to help users deal with the preceding issues. In today's example, we will use NodePools to describe and manage nodes:

kubectl apply -f - <<EOF
apiVersion: apps.openyurt.io/v1beta1
kind: NodePool
metadata:
  name: beijing
spec:
  type: Edge
  annotations:
    apps.openyurt.io/example: test-beijing
  taints:
    - key: apps.openyurt.io/example
      value: beijing
      effect: NoSchedule
---
apiVersion: apps.openyurt.io/v1beta1
kind: NodePool
metadata:
  name: shanghai
spec:
  type: Edge
  annotations:
    apps.openyurt.io/example: test-shanghai
  taints:
    - key: apps.openyurt.io/example
      value: shanghai
      effect: NoSchedule
EOF

Add edge nodes to their respective NodePools. Please see how OpenYurt nodes are added for more information about how edge nodes are added.

kubectl label node <node1> apps.openyurt.io/desired-nodepool=Beijing
kubectl label node <node2> apps.openyurt.io/desired-nodepool=shanghai

kubectl get nodepool

Expected Output

NAME       TYPE   READYNODES   NOTREADYNODES   AGE
beijing    Edge   1            0               6m2s
shanghai   Edge   1            0               6m1s

Deploy Edge Applications in Batches​

Before we get into the details, let's look at how KubeVela describes and deploys applications to the edge. With the following application, we can deploy multiple Nginx Ingress Controller to their respective edge NodePools. Using the same application to uniformly configure the Nginx Ingress can eliminate duplication and reduce the management burden. It facilitates unified operations (such as the release and other O&M of components in the cluster).

apiVersion: core.oam.dev/v1beta1
kind: Application
metadata:
  name: edge-ingress
spec:
  components:
    - name: ingress-nginx
      type: helm
      properties:
        chart: ingress-nginx
        url: https://kubernetes.github.io/ingress-nginx
        repoType: helm
        version: 4.3.0
        values:
          controller:
            service:
              type: NodePort
            admissionWebhooks:
              enabled: false
      traits:
        - type: edge-nginx
  policies:
    - name: replication
      type: replication
      properties:
        selector: [ "ingress-nginx" ]
        keys: [ "beijing","shanghai" ]
  workflow:
    steps:
      - name: deploy
        type: deploy
        properties:
          policies: ["replication"]

A KubeVela application has three parts.

1. Helm Type Component: It describes the version of the Helm package that we want to install into the cluster. In addition, we have attached an trait to this component: edge-nginx. We'll show the details of this trait later. You can regard it as a patch that contains the properties of different NodePools.
2. The Strategy of Replication: It describes how to copy components to different NodePools. The field selector is used to select the components to be copied. Its keys field will convert one component to two components with different keys ("Beijing" and "Shanghai").
3. Deploy Workflow Steps: It describes how to deploy an application. It specifies the replication strategy to perform replication work.

Now, we can send the application to the cluster.

vela up -f app.yaml

Check the application status and resources created by KubeVela.

vela status edge-ingress --tree --detail

Expected Output

CLUSTER       NAMESPACE     RESOURCE                           STATUS    APPLY_TIME          DETAIL
local  ─── default─┬─ HelmRelease/ingress-nginx-beijing  updated   2022-11-02 12:00:24 Ready: True  Status: Release reconciliation succeeded  Age: 153m
                   ├─ HelmRelease/ingress-nginx-shanghai updated   2022-11-02 12:00:24 Ready: True  Status: Release reconciliation succeeded  Age: 153m
                   └─ HelmRepository/ingress-nginx       updated   2022-11-02 12:00:24 URL: https://kubernetes.github.io/ingress-nginx  Age: 153m
                                                                                         Ready: True
                                                                                         Status: stored artifact for revision '7bce426c58aee962d479ca84e5c
                                                                                         fc6931c19c8995e31638668cb958d4a3486c2'

Vela CLI can stand on a higher level to uniformly show the health status of applications. When it's needed, Vela CLI can help you penetrate applications and direct access to the underlying workloads. Also, it offers a rich selection of observability and Debug capability. For example, you can print the application's log through vela logs. You can forward the port that deploys applications locally through vela port-forward. You can use the vela exec command to go deep into the container at the edge and run Shell commands to troubleshoot the problem.

If you need an intuitive understanding of the application, KubeVela officials provide a Web console addon, VelaUX. You can view detailed resource topologies with the VelaUX addon .

vela addon enable velaux

Visit the Resource Topology page of VelaUX:

As you can see, KubeVela creates two HelmRelease to deliver the Nginx Ingress Controller Helm chart to two NodePools. HelmRelease resources are processed by the preceding FluxCD addon, and NGINX Ingress is installed in the two NodePools of the cluster. Run the following command to check whether pods of the Ingress controller are created in the Beijing NodePool. The same is true for the shanghai NodePool.

$ kubectl get node -l  apps.openyurt.io/nodepool=beijing                               
NAME                      STATUS   ROLES    AGE   VERSION
iz0xi0r2pe51he3z8pz1ekz   Ready    <none>   23h   v1.24.7+k3s1

$ kubectl get pod ingress-nginx-beijing-controller-c4c7cbf64-xthlp -oyaml|grep iz0xi0r2pe51he3z8pz1ekz
  nodeName: iz0xi0r2pe51he3z8pz1ekz

Differentiated Deployment​

How can a differentiated deployment of the same component be implemented during KubeVela application delivery? Let's continue to learn about the Trait and Policy that support the application. As mentioned above, we use KubeVela's built-in component replication policy in the workflow to attach a custom edge-nginx trait to ingress-nginx components.

• The component splitting policy splits the component into two components with a different context.replicaKey.
• edge-nginx Trait uses different context.replicaKey to deliver Helm charts with different configuration values to the cluster. Let the two Nginx Ingress Controller run in different NodePools and monitor Ingress resources with different ingressClasses. Patch the values of the Helm chart and modify the fields related to Node Selection, Affinity, and ingressClass.
• Different patch strategies is used when patching different fields. For example, a retainKeys strategy can overwrite the original value, while a jsonMergePatch strategy is merged with the original value.

"edge-nginx": {
  type: "trait"
  annotations: {}
  attributes: {
    podDisruptive: true
    appliesToWorkloads: ["helm"]
  }
}
template: {
  patch: {
    // +patchStrategy=retainKeys
    metadata: {
      name: "\(context.name)-\(context.replicaKey)"
    }
    // +patchStrategy=jsonMergePatch
    spec: values: {
      ingressClassByName: true
      controller: {
        ingressClassResource: {
          name:            "nginx-" + context.replicaKey
          controllerValue: "openyurt.io/" + context.replicaKey
        }
        _selector
      }
      defaultBackend: {
        _selector
      }
    }
  }
  _selector: {
    tolerations: [
      {
        key:      "apps.openyurt.io/example"
        operator: "Equal"
        value:    context.replicaKey
      },
    ]
    nodeSelector: {
      "apps.openyurt.io/nodepool": context.replicaKey
    }
  }
  parameter: null
}

Let More Types of Applications Go to the Edge​

As you can see, we only customize a trait with more than 40 rows and make full use of the built-in capabilities of KubeVela to deploy Nginx Ingress to different NodePools. More applications are likely to be deployed at the edge with the prosperous cloud-native ecosystem and cloud-edge collaboration trend. When a new application needs to be deployed in the edge NodePool in the new scenario, there is no need to worry. KubeVela makes it easy to expand a new edge application deployment trait following this pattern without writing code.

For example, we hope to deploy the implementation of the Kubernetes community's recent evolution hotspot Gateway API to the edge as well. We can use the Gateway API to enhance the expressive capability and extensibility of the edge NodePool to expose services and use role-based network APIs on edge nodes. For this scenario, we can easily complete the deployment task based on the preceding extension method. You only need to define a new trait (as shown below):

"gateway-nginx": {
  type: "trait"
  annotations: {}
  attributes: {
    podDisruptive: true
    appliesToWorkloads: ["helm"]
  }
}

template: {
  patch: {
    // +patchStrategy=retainKeys
    metadata: {
      name: "\(context.name)-\(context.replicaKey)"
    }
    // +patchStrategy=jsonMergePatch
    spec: values: {
      _selector
      fullnameOverride: "nginx-gateway-nginx-" + context.replicaKey
      gatewayClass: {
        name:           "nginx" + context.replicaKey
        controllerName: "k8s-gateway-nginx.nginx.org/nginx-gateway-nginx-controller-" + context.replicaKey
      }
    }
  }
  _selector: {
    tolerations: [
      {
        key:      "apps.openyurt.io/example"
        operator: "Equal"
        value:    context.replicaKey
      },
    ]
    nodeSelector: {
      "apps.openyurt.io/nodepool": context.replicaKey
    }
  }
  parameter: null
}

This Trait is similar to the one used to deploy the Nginx Ingress mentioned earlier. We also made some similar patch for Nginx Gateway chart values, including node selection, affinity, and resource name. The difference between the former trait is that this Trait specifies the gatewayClass instead of the IngressClass. Please see the GitHub repository for more information about the trait and application files. We extend the ability of the cluster to deploy a new application to the edge by customizing such a Trait. If we cannot predict application deployment requirements brought about by the development of edge computing in the future, at least we can adapt to new scenarios through this scalability way.

How KubeVela Solves Edge Deployment Puzzles​

Let's review how KubeVela addressed the key issues raised at the beginning of the article.

1. Unified Configuration::A component is used to describe the common properties of the ingress-nginx Helm chart to be deployed (such as Helm repository, chart name, version, and other unified configurations).
2. Attribute Differences: KubeVela uses a user-defined trait that describes the differences in Helm configurations sent to different NodePools. This trait can be repeatedly used to deploy the same Helm Chart.
3. Scalability: KubeVela can be programmable for common workloads (such as Deployment/StatefulSet) or other packaging methods (such as Helm or Kustomize). It requires only a few lines of code to push a new application to the edge.

It benefits from the powerful functions provided by KubeVela in the application delivery and management field. In addition to solving application definition, delivery, O&M, and observability issues within a single cluster, KubeVela natively supports application release and management in the multi-cluster mode. Currently, the Kubernetes deployment mode suitable for edge computing scenarios is not fixed. KubeVela can manage application tasks regardless of the architecture of single cluster + edge NodePool or multi-edge cluster architecture.

With OpenYurt and KubeVela, cloud-edge applications are deployed in a unified manner and share the same abstraction, O&M, and observability, which avoids the experience of being separated in different scenarios. In addition, cloud applications and edge applications can take advantage of KubeVela's excellent practices of a continuously integrated cloud-native ecosystem in the form of addons. In the future, the KubeVela community will continue to enrich out-of-the-box system addons and deliver better and easier-to-use application delivery and management capabilities.

If you want to know more about the capabilities of application deployment and management, you can read KubeVela’s official documents. If you want to know the latest development in the KubeVela community, you are welcome to participate in the discussion! If you are interested in OpenYurt, you are welcome to join the OpenYurt community!

You can learn more about KubeVela and the OAM project through the following materials:

• Project Code Base: https://github.com/oam-dev/kubevela Welcome Star/Watch/Fork!
• Project Official Homepage and Documents: kubevela.io

KubeVela is a modern software delivery platform that makes deploying and operating applications across multi environments easier, faster, and more reliable.

KubeVela is infrastructure agnostic and application-centric. It allows you to build robust software and deliver them anywhere! Kubevela provides an Open Application Model (OAM) based abstraction approach to ship applications and any resource across multiple environments.

Open Application Model (OAM) is a set of standard yet higher-level abstractions for modeling cloud-native applications on top of today’s hybrid and multi-cloud environments. You can find more conceptual details here.

apiVersion: core.oam.dev/v1beta1
kind: Application
metadata:
  name: first-vela-app
spec:
  components:
    - name: express-server
      type: webservice
      properties:
        image: oamdev/hello-world
        ports:
         - port: 8000
           expose: true
      traits:
        - type: scaler
          properties:
            replicas: 1

ArgoCD​

Argo CD is a declarative continuous delivery tool for Kubernetes applications. It uses the GitOps style to create and manage Kubernetes clusters. When any changes are made to the application configuration in Git, Argo CD will compare it with the configurations of the running application and notify users to bring the desired and live state into sync.

Flux​

By default, Kubevela works with Flux addon to complete the Gitops journey for applications. You need to enable the flux addon and perform git operations quickly. https://kubevela.io/docs/end-user/gitops/fluxcd.

vela addon enable fluxcd

Integration with ArgoCD​

If you want to use Kubevela with ArgoCD, there are two options to enable it.

• Kubevela dry-run option
• Kubevela + ArgoCD Gitops syncer mode

1. Vela dry-run approach

First approach is that without using Kubevela GitOps Controller, we can use ArgoCD as the GitOps Controller, which means ArgoCD applies a raw manifest to the target cluster. Kubevela provides us to export oam application components and traits to native Kubernetes resource therefore we can still use OAM based model and ArgoCD can manage resources with custom argo-repo-server plugin. Behind this approach that KubeVela and OAM can also works as a client side tool with the feature dry-run.

This part describe how ArgoCD apply dry-run mode on Kubevela resources. Firstly, ArgoCD and Kubevela same should be installed.

ArgoCD version v2.5.4

Kubevela version v1.6.4

Prerequisites​

Argo CD allows integrating more config management tools using config management plugins.

1. Write the plugin configuration file​

Plugins will be configured via a ConfigManagementPlugin manifest located inside the plugin container.

apiVersion: v1
kind: ConfigMap
metadata:
  name: vela-cmp
  namespace: argocd
data:
  plugin.yaml: |
    apiVersion: argoproj.io/v1alpha1
    kind: ConfigManagementPlugin
    metadata:
      name: vela
    spec:
      version: v1.0
      init:
        command: ["vela", "traits"]
      generate:
        command: ["sh"]
        args:
          - -c
          - |
            vela dry-run -f test-argo-oam.yml
      discover:
        # fileName: "-oam.yml*"
        find:
          # This does the same thing as fileName, but it supports double-start (nested directory) glob patterns.
          glob: "**/*oam*.yml"

kubectl apply -f vela-cmp.yml

This plugin is responsible for converting Kubevela OAM manifest definition to raw kubernetes object and ArgoCD should be responsible to deploy.

2. Register the plugin sidecar​

To install a plugin, patch argocd-repo-server to run the plugin container as a sidecar, with argocd-cmp-server as its entrypoint.

Vela plugin runs the vela command to export manifest when the plugin is discovered on git manifest. Before initializing the plugin, we need to install Kubevela CLI to run the order successfully. The below configuration adds an init container to download the necessary CLI.

initContainers:
 - name: kubevela
   image: nginx:1.21.6
   command:
    - bash
    - '-c'
    - |
     #!/usr/bin/env bash
     set -eo pipefail
     curl -fsSl https://kubevela.io/script/install.sh | bash -s 1.6.4
   env:
    - name: VELA_INSTALL_DIR
      value: /custom-tools
   resources:
    limits:
     cpu: 50m
     memory: 64Mi
    requests:
     cpu: 10m
     memory: 32Mi
   volumeMounts:
    - name: custom-tools
      mountPath: /custom-tools
   terminationMessagePath: /dev/termination-log
   terminationMessagePolicy: File
   imagePullPolicy: IfNotPresent

after adding init container, we need to add our custom sidecar binary to run plugins properly. To use configmap plugin configuration in our sidecar, we need to mount configmap plugin to our pods

containers:
        - name: vela
          image: nginx
          command:
            - /var/run/argocd/argocd-cmp-server
          env:
            - name: KUBECONFIG
              value: /kubeconfig
          resources: {}
          volumeMounts:
            - name: var-files
              mountPath: /var/run/argocd
            - name: vela-cli-dir
              mountPath: /.vela
            - name: kubeconfig
              mountPath: /kubeconfig
              subPath: kubeconfig
            - name: plugins
              mountPath: /home/argocd/cmp-server/plugins
            - name: vela-cmp
              mountPath: /home/argocd/cmp-server/config/plugin.yaml
              subPath: plugin.yaml
            - name: cmp-tmp
              mountPath: /tmp
            - name: custom-tools
              mountPath: /usr/local/bin/vela
              subPath: vela

3. Configure Argo CD to watch this repo for Git pushes.​

Create git repository and put oam manifest to target directory after you can create a simple ArgoCD application that watch git repository and apply manifests to local kubernetes cluster.

apiVersion: core.oam.dev/v1beta1
kind: Application
metadata:
  name: test-2
spec:
  components:
    - name: wf-poc-app-2
      type: webservice
      properties:
        image: oamdev/helloworld-python:v1
        env:
          - name: "TARGET"
            value: "ArgoCD"
        port: 8080

4. Final Output​

ArgoCD synced applications successfully and rendered kubernetes resources.

2. Kubevela Controller + ArgoCD Gitops syncer

Second approach is that we can use Kubevela gitops controller way as the server side and argocd can be our gitops syncer. This approach is flexible to use native kubevela feature set without using a custom plugin or dry run module. We just need to add below annotations to our manifest repository to ignore outofsync.

argocd.argoproj.io/compare-options: IgnoreExtraneous

In this example, ArgoCD tracks native Kubevela application resources and its revision.

Conclusion​

1. Vela Dry-run approach

• Kubevela can be installed different cluster from ArgoCD.
• Limited Kubevela features, we just use component and traits definition.
• Workflow and policy features don’t supported by this way.
• Depend on Kubevela dry run feature.
• No needed rbac, ui or any different experience from ARGO.

2. Kubevela Controller + ArgoCD Gitops syncer

• Kubevela must be installed to ArgoCD cluster.
• We can use all the features of Kubevela.
• No need depends on anything.

Reference: https://gokhan-karadas1992.medium.com/argocd-kubevela-integration-eb88dc0484e0

Big thanks to Erkan Zileli 🎉🎉🎉

Since Open Application Model invented in 2020, KubeVela has experienced tens of version changes and evolves advanced features towards modern application delivery. Recently, KubeVela has proposed to become a CNCF incubation project and delivered several public talks in the community. As a memorandum, this article will look back into the starting points and give a comprehensive introduction to the state of KubeVela in 2022.

KubeVela is a modern software platform that makes delivering and operating applications across today's hybrid, multi-cloud environments easier, faster and more reliable. It has three main features:

• Infrastructure agnotic: KubeVela is able to deploy your cloud-native application into various destinations, such as Kubernetes multi-clusters, cloud provider runtimes (like Alibaba Cloud, AWS or Azure) and edge devices.
• Programmable: KubeVela has abstraction layers for modeling applications and delivery process. The abstraction layers allow users to use programmable ways to build higher level reusable modules for application delivery and integrate arbitrary third-party projects (like FluxCD, Crossplane, Istio, Prometheus) in the KubeVela system.
• Application-centric: There are rich tools and eco-systems designed around the KubeVela applications, which add extra capabilities for deliverying and operating the applications, including CLI, UI, GitOps, Observability, etc.

KubeVela cares the whole lifecycle of the applications, including both the Day-1 Delivery and the Day-2 Operating stages. It is able to connect with a wide range of Continuous Integration tools, like Jenkins or GitLab CI, and help users deliver and operate applications across hybrid environments.

Why KubeVela?

Challenges and Difficulties​

Nowadays, the fast growing of the cloud native infrastructures has given more and more capabilities for users to deploying applications, such as High Availability and Security, but also exposes an increase number of complexities directly to application developers. For example, the Ingress resource on Kubernetes enables users to expose their applications easily, but developers need to handle the Ingress upgrades when the underlying Kubernetes version shifts, which requires knowledges for the Ingress resource. The hybrid deployment across various cloud provides can make this problem even harder. These difficulties are caused by the lack of operational input in the application definition and developers must face the infrastructure details directly if they want to enjoy the benefits brought by the rich cloud-native community.

Open Application Model​

To tackle the above challenges and bridge the gap between the use of applications and the understanding of infrasturcture details, Open Application Model (OAM) is jointly proposed by Alibaba Cloud and Microsoft Azure in 2020. The aim is to define a consistent application model for application delivery, irrelevant with the platforms and implementations. The defined application model describes an interface for developers on what an application consists of and how it should work. The former one is known as Component in OAM, which is usually used to model the workloads of the application. The latter one is defined as Trait in OAM, which attaches extra capabilities to Components.

KubeVela as OAM​

KubeVela is one of the implementations for the Open Application Model. In KubeVela, the abstraction layer is powered by CUE, a novel configuration programming language which can describe complex rendering logics and work as a superset of JSON. The abstraction layer simplifies the configuration of resources in Kubernetes, which hides the details of implementations and exposes limited parameters to the front developers. With KubeVela application, it is easy for developers to focus on the centric logic of applications, like what container image should be used and how the service should be made accessible. To achieve that, best practices of using Kubernetes native resources are summarized into KubeVela X-Definitions, which provide rendering templates of resources using CUE. These templates can be accessed from various sources, including official repositories, community addons or even self customized implementations by system operators. The templates are mostly infrastructure implemetation agnostic, in other words, not necessarily bond to specific infrastructures. The developers do not need to be aware of the underlying infra when using these templates.

Components & Traits​

The application model divides the abstraction of infra into two different aspects. The Component describes the main workload, which particularly in Kubernetes can be Deployments, StatefulSets, Helm Releases, etc. The Trait on the other hands, describes the added capability for the main workload, such as the scaler trait specifying the number of replicas and the gateway trait aggregates the endpoints for access. The separation of concerns in the design of Component and Trait give high extensibility and reusability to the abstraction. For example, the gateway trait could be backended by different infrastructures like Ingress or HTTPRoute. The application developer who uses the trait only needs to care about the exposed parameters, including the path, port and domain. The trait can be attached to various types of workloads, abstracted by different types of components, such as Deployment, StatefulSet, CloneSet, etc. In the cases where application developers and SRE are in the different teams, KubeVela makes clear division for their responsibilities.

• The platform team providing infrastructures, are responsible to build up X-Definitions where they enforce best practices and deployment confidence.
• The end users only need to choose the Component and Trait provided by the platform team and use them to assemble applications. They can simply enjoy PaaS-like experiences instead of directly interacting with the infra behind.

These are made possible thanks to the flexible, extensible and programmable system of KubeVela and can be applied under varying environments.

Unified Delivery​

Application delivery could happen everywhere. Therefore, another goal for KubeVela application is to build up unified delivery and provide consistent usage for users under various scenarios.

Hybrid-Cloud & Multi-Cluster​

In addition to the abstraction layer, KubeVela also supports hybrid-cloud or multi-cluster architecture natively as modern cloud native applications are not only about containers but involves lots of cloud resources as well. Besides, more and more users and teams start facing the difficulties of deliverying applications to various environments or multi-clusters for different purposes, such as testing or high availability. The KubeVela application allows user to define delivery targets and differentiated configurations through policies. The abstraction helps hide the details of how clusters are registered and connected and provide runtime-agnostic usages to app developers.

Addon Integration​

To enrich the delivery capability, users can leverage KubeVela addons to make extensions to their system. The addons are discoverable, reusable and easy-to-install capability bundles. They usually contain capability providers, including a wide range of third-party projects, like FluxCD, ClickHouse, Crossplane, etc. Addons not only install those projects into the system but create corresponding definitions for the integration concurrently, which extends the types of Component and Trait that application developers are able to use. The KubeVela community currently have 50+ addons already. Platform builders could enjoy these out-of-box integrations in systems depending on their customized demands.

With addons enabled in the system, it would be possible for end users to assemble applications in more customized ways, such as deploying cloud resources or using advanced workloads.

KubeVela Workflow​

While the Open Application Model defines the composition of an application, in real cases, the delivery process of the compositions could still vary a lot. For example, the different components in one application could have inter dependencies or data passing where delivery steps must be executed in specific order. Furthermore, the delivery process sometimes also involves more actions apart from the delivery of resources, such as rollouts or notifications. An extensible workflow is therefore designed to fulfill the needs of the process customization in KubeVela. Similar to Component and Trait, KubeVela workflow also leverages CUE to define workflow steps, providing flexibility, extensibility and programmability. It can be seen as another form of Infrastructure as Code (IaC). A bunch of build-in workflow steps has already provided rich out-of-box capabilities in KubeVela, such as making multi-cluster deployments and sending notifications through slack or email. The lightweight engine ensures the high performance and safety of step executions, compared to other types of engines involving running extra containers. Differ from the Component and Trait definitions in KubeVela, the WorkflowStep definition does not render templates into resources. Instead, it describes the actions to be executed in the step, which calls underlying atomic functions in various providers. With the use of workflow and addons, users are able to build arbitrary delivery process and make customized integrations. For example, it is possible to let the Continuous Integration tools to trigger the delivery of KubeVela applications and implement the GitOps solutions combining FluxCD and other addons.

Day-2 Management​

KubeVela cares more other than Day-1 Delivery. It also provides a unified Day-2 application management capability for all it's extensibility. The day-2 management is necessary for system operators and application developers to make continuous operation for the delivered applications and ensure the applications are always under control.

Resource Management​

The basic capabilities for application management are for its resources. KubeVela's core controller continuously watches the difference between the current state and the desired state of delivered resources. It makes sure that the live spec is accord with the declared spec recorded in the delivery process and therefore effectively prevents any configuration drifts. Besides, the automated garbage collection help recycle the resources that are not in-use during upgrades or deletion. There are also times resources need to be shared across multiple applications. These are all made possible in KubeVela application through the use of policies.

Version Control​

KubeVela application keeps history records for deliveries. These snapshots are useful when new version publish are out of expectations. The change inspectation could be used to diagnose the possible error changes and the rollback allows fast recovery to the previous successful states.

Observability​

KubeVela treats observability as first class citizen. It is the eyes to users for monitoring the state of applications and observing exceptions. There are multiple tools and methods in KubeVela to do the observation job. One of the most straightforward way is to use the CLI tool of KubeVela. The Vela CLI is able to provide in-time status info for the application in fine-grain or aggregated level. For users that prefer web interfaces, VelaUX provides an alternative way to view application status. In the cases applications are monitored through third-party projects, such as Grafana, Prometheus or Loki, KubeVela further provides addons for bootstrapping the observability infrastructures and empower users to customize the observing rules as codes in applications, through the abstraction layer. A series of out-of-box metrics and dashboards give users the basic capability of automated system observability. These can be used to diagnose system level exceptions and help improve the overall performance.

Eco-system​

In addition to the above mentioned tools, KubeVela also has several other tools in the eco-systems to facilitate application delivery.

• Vela CLI: KubeVela CLI provides various commands that helps you to operate applications, such as managing definitions, viewing resources, restarting workflow, rolling versions.
• VelaUX: VelaUX is the Web UI for KubeVela. Besides, it incorporates business logics into fundamental APIs and provides out-of-box user experiences for non-k8s-expert users.
• Terraform Controller: The terraform controller in KubeVela allows users to use Terraform to manage cloud resources through Kubernetes Custom Resources.
• Cluster Gateway: The gateway that provides unified multi-cluster access interface. Working as Kubernetes Aggregated API Server, the gateway leverages the native Authentication and Authorization modules and enforces secure and transparent access to managed clusters.
• VelaD: Building on top of k3s & k3d, VelaD integrates KubeVela with Kubernetes cores, which can be extremely helpful for building dev/test environment.
• Vela Prism: The extension API server for KubeVela built upon the Kubernetes Aggregated API Server. It projects native APIs like creating dashboards on Grafana into Kubernetes resource APIs, so that users can manage 3rd-party resources as Kubernetes native resources.
• Vela Workflow: The workflow engine translates CUE-based steps and executes them. It works as a pure delivery tool and can be used aside by the KubeVela application. Compared to Tekton, it mainly organize the process in CUE style, instead of using Pods and Jobs directly.

Stability​

To ensure KubeVela is able to handle certain amount of applications under limited resources, multiple load testings have been conducted under various circumstances. The experiments have demonstrated that the performance of KubeVela system is capable of dealing thousands of applications in an ordinary-sized cluster. The observability infrastructure further exposes the bottleneck of KubeVela and guides system operators to do customized tunning to improve the performance in specific use environments.

In a nutshell

Currently, KubeVela has already been applied in production by a number of adopters from various areas. Some mainly use KubeVela's abstraction capability to simplify the use and deploy of applications. Some build application-centric management system upon KubeVela. Some use the customized workflow to orchestrate the delivery process. It is especially welcomed in high-tech industries and shown to be helpful for delivering and managing enourmous applications.

The KubeVela community has attracted world-wide contributors and continuously evolves over the past two years. Nowadays, there are over 200 contributors from various contries have participated in the developing of KubeVela. Thousands of issues have been raised and 85% of them are already solved. There are also bi-weekly community meetings held in both English and Chinese community.

With more and more people coming into the community, KubeVela is consistently upgrading itself to fit into more complex, varying use cases and scenarios.

The cloud-native landscape is formed by a fast-growing ecosystem of tools with the aim of improving the development of modern applications in a cloud environment. Kubernetes has become the de facto standard to deploy enterprise workloads by improving development speed, and accommodating the needs of a dynamic environment.

Kubernetes offers a comprehensive set of entities that enables any potential application to be deployed into it, independent of its complexity. This however has a significant impact from the point of view of its adoption. Kubernetes is becoming as complex as it is powerful, and that translates into a steep learning curve for newcomers into the ecosystem. Thus, this has generated a new trend focused on providing developers with tools that improve their day-to-day activities without losing the underlying capabilities of the underlying system.

Napptive Application Platform​

The NAPPTIVE Playground offers a cloud native application platform focused on providing a simplified method to operate on Kubernetes clusters and deploy complex applications without needing to work with low-level Kubernetes entities. This is especially important to be able to accommodate non-developer users' personas as the computing resources of a company are being consolidated in multi-purpose, multi-tenant clusters. Data scientists, business analysts, modeling experts and many more can benefit from simple solutions that do not require any type of knowledge of cloud computing infrastructures, or orchestration systems such as Kubernetes, which enable them to run their applications with ease in the existing infrastructure.

Any tool that works in this space must start by analyzing the existing abstractions. In particular, at Napptive, we focus on the Applications. This quite understood abstraction is not present in Kubernetes, requiring users to manually tag, identify and reason about the different components involved in an application. The Open Application Model provides an excellent abstraction to represent any type of application independently of the cloud provider, containerization technology, or deployment framework. The model is highly customizable by means of adding Traits, Policies, or new Component Definitions.

KubeVela in Napptive​

The Napptive Playground leverages Kubevela as the OAM runtime for Kubernetes deployments. Our Playground provides an environment abstraction with multi-tenant guarantees that is equivalent to partitioning a shared cluster by means of differentiated namespaces. The benefit of our approach is the transparency of its configuration using a higher abstraction level that does not involve any Kubernetes knowledge.

The following diagram describes the overall architecture of Napptive and Kubevela deployed in a Kubernetes cluster.

The user has the ability to interact with the cluster by using the Napptive user interface (CLI or Web UI), or by means of using the standard Kubernetes API with kubectl. Isolated environments can be easily created to establish the logical separations such as type of environment (e.g., deployment, staging, production), or purpose (e.g., projectA, projectB), or any other approach to differentiate where to deploy an application. Once the OAM application is deployed on Kubernetes, Kubevela is in charge of managing the low-level application lifecycle creating the appropriate Kubernetes entities as a result. One of the main differences with other adopters of Kubevela, is the fact that we use Kubevela in a multi-tenant environment. The following figure shows the specifics of an application deployment in this type of scenario.

The Napptive Playground is integrated with Kubernetes RBAC and offers a native user management layer that works in both on-premise and cloud deployments. User identity is associated with each environment, and Kubevela is able to take that information to ensure that users can only access their allowed resources. Once a user deploys an OAM application, Kubevela will intercept the call and attach the user identity as an annotation. After that, the rendering process will ensure that the user has access to the entities (e.g., trait, component, policy, workflow step definitions) and that the target namespaces are also accessible by the user. Once the application render is ready, the workflow orchestrator will create the different entities in the Kubernetes namespace.

Napptive in the Community​

In terms of our involvement with the OAM/Kubevela community, it has evolved overtime from being passive members of the community simply exploring the possibilities of OAM in Kubernetes, to becoming active contributor members in different areas. In particular, we have closely worked with the core Kubevela development team to test and overcome the different challenges related to using a multi-tenant, RBAC compliance installation of Kubevela. Security in this type of installation is critical and it is one of our main focuses within the Kubevela community. We are committed to continue working with the community not only to ensure that multi-tenancy is maintained throughout the different releases, but also to add our own perspective representing our customer use cases.

The Kubevela community is growing at a fast pace, and we try to contribute our adaptations, features, feedback, and thoughts back to the community. This type of framework is deployed in a multitude of environments. The most common one is probably where one or more clusters are inside a company. In our particular application, we are interested in exploring methods to offer computing capabilities for a variety of user personas in a shared cluster, and we contribute our view and experience in the community.

Future Evolution​

In terms of future evolution, we believe the Napptive Playground is a great tool to experience working with the Open Application Model without the need of installing new clusters and frameworks. From the point of view of our contributions in the community, we are internally working in exploring QA mechanisms that ensure that customer applications remain working after upgrades, and identifying potential incompatibilities between releases. Once that work is ready, we plan to contribute our testing environment setup back to the community so that it can be adopted in the main branch. We are also excited with new functionalities that have been recently added to Kubevela such as multi-cluster support, and are exploring methods to adopt it inside the Playground. Moreover, we are actively working on an OAM-compatible application catalog that will simplify the way organizations store and make accessible application definitions so that they can be deployed into a cluster from a central repository. The catalog focuses on the OAM entities and relies on existing container registries for storing the image.

Background​

With the advent of the cloud-native era, developers have to use more and more complex APIs to build cloud-native-compliant application architectures from powerful infrastructure and clouds. But it's difficult to use and has a high learning curve for app developers. This also brings great stability risks due to the direct operation of the underlying infrastructure. Kubernetes provides a unified API integration interface for the infrastructure, but it’s a platform for platform developers. The application-centric interface is missing for upper-level application developers. Open Application Model (OAM) has come into being to meet these requirements. It was jointly released by Alibaba and Microsoft in 2019. The two companies brought a lot of practical experience in cloud-native application development and provided a theoretical basis for building an application platform in the cloud-native era.

After OAM was released, it was welcomed and adopted by many enterprises, including Oracle, Tencent, ByteDance, and 4Paradigm. However, for some enterprises, OAM is a theoretical model with no practical platforms that can be used directly. Thus, it is difficult to land. As a result, Alibaba Cloud engineers joined forces with enterprises that adopted OAM in the community and built an out-of-the-box OAM implementation engine based on common practice. Then, KubeVela was born.

Standard and Extensible Application Delivery Engine​

KubeVela was officially released in November 2020 and has always adhered to the design principles of flexibility, extensibility, and separation of concerns. The goal is to connect cloud-native technology components and enterprise-level applications to help developers in the enterprise quickly obtain an easy, secure, and reliable software delivery and management experience. Less than a year after becoming open-source, it has provided core features (such as multi-cluster unified orchestration, infrastructure independence, and application delivery workflow). It officially joined CNCF in June 2021 and was widely welcomed by community users.

Today, KubeVela has been adopted by more than 300 domestic and overseas enterprises, including China Merchants Bank, ByteDance, Li Auto, and Shein. The entire OAM and KubeVela community contains dozens of ecological projects, which have accumulated more than 8000 stars and received many contributions from more than 300 developers from dozens of countries worldwide.

Picture 1: Unified Application Delivery of KubeVela

KubeVela helps enterprises build unified application delivery capabilities. With KubeVela's unified architecture, enterprises can simultaneously deliver microservice containers, cloud database, frontend static pages, and their internal custom extensions in one configuration file, completing the orchestration and operational management of diverse workloads. Developers of business applications no longer need to worry about the differences in Kubernetes resource versions or different disk specifications during delivery. KubeVela provides comprehensive version management, grayscale release, CI/CD docking, multi-cluster management, and other functions for the description of a unified application, significantly reducing the threshold for enterprises to use cloud-native technology.

With the development of the community, more enterprises use KubeVela as the core engine of the internal PaaS platform. Based on the unified model and addon extension mechanism at the upper layer of KubeVela, a series of advanced features have been developed, including security, observability, and GitOps. A large number of core contributors have emerged, including China Merchants Bank, Napptive, and JD Cloud. They contribute many new features and addons to the KubeVela ecosystem, gradually extending the boundaries of KubeVela from application delivery to application day-2 management.

New Milestone: United App Delivery and Day-2 Management​

This release has four core directions.

1. Visualize Resource Status: From the first deployment of an application to the subsequent continuous delivery, KubeVela visualizes the delivery process and topology of the entire application resources and automatically generates an observability dashboard containing more than 100 core metrics to help developers locate problems by themselves. More importantly, the resource visualization capability can easily access user-defined resources to meet user-defined observability configuration requirements and realize unified observability at the multi-cluster level.

2. Make Platform Capabilities More Like Addons: KubeVela has built a flexible, extensible, and self-installed platform capability addon center and more than 50 out-of-the-box addons. These addons include cloud resources, GitOps, observability, FinOps, IoT, and other scenarios. They help business developers flexibly build solutions for different scenarios and reduce the mental burden of users through the practical experience of the community.

3. Automate the Delivery Process: The KubeVela declarative workflow system has the capability of operating system resources and adds a large number of advanced features (such as condition judgment, parameter transfer, and branch concurrency). Each process can be extended by configuration language programming. More than ten scenario-based workflows (including multi-cluster grayscale release, CI/CD docking, and secure and compliant access) help developers easily complete self-service application orchestration.

4. Unify Environment Management: You no longer need to prepare a Kubernetes cluster in advance to install KubeVela. KubeVela provides a self-service one-click offline installation tool for developers to install a complete set of control panels locally or on virtual machines. It combines multi-cluster delivery and management capabilities to help developers realize unified management in the whole process and multi-environment of development, testing, and production.

Picture 2: Application Platform with Integrated Delivery and Management

In-Depth Interpretation of New Version Functions​

Since KubeVela officially released version 1.0 in April 2021, OAM has been continuously verified by KubeVela and gradually stabilized. KubeVela's previous version iterations can always ensure the forward compatibility of the API, corresponding to OAM version 0.3.1. According to SemVer's international practice of naming versions, the version number released this time is 1.6. Although the version number has only increased by 0.1, this release is the overall presentation of KubeVela's application management capabilities since version 1.2 was released more than a year ago. Next, we will conduct an in-depth interpretation of the core functions one by one.

Prioritize Resource Visualization​

Based on the OAM concept, extensibility and abstraction are the two core features of KubeVela. However, abstraction often represents a black box. Once users encounter problems that are difficult to troubleshoot and can't see the progress and status of resource delivery, management will be more difficult. We have solved the problem of resource visualization based on extensibility in this release. KubeVela automatically discovers Kubernetes native resources and helps users build resource topology maps. You can automatically generate a topology map for user-defined extended resources by describing the configuration of a resource association relationship.

This also means resource visualization is prioritized in the KubeVela system. KubeVela can fully describe the topology, view underlying container events and logs, and obtain the overall delivery status for any workload contained in an application. The following picture shows the resource topology of KubeVela.

Picture 3: VelaUX Resource Visualization Process

VelaUX Documentation: https://kubevela.net/docs/reference/addons/velaux

In addition, KubeVela provides a top interactive interface for command-line users, which allows users to easily view the status of delivered resources and fully meet the usage habits of different developers.

Picture 4: Diagram of Vela Top Command Line Operation

Vela Top Command Line Operation Documentation: https://kubevela.net/docs/tutorials/vela-top

Support for Application-Level Observability​

Observability is the top priority in application management. This release of KubeVela has also comprehensively improved observability. Specifically, it consists of observability infrastructure construction, application-oriented observability, and observability as code.

KubeVela Observability Documentation: https://kubevela.net/docs/platform-engineers/operations/observability

Observability Infrastructure​

KubeVela's addon system can help users that do not have observability infrastructure create the entire observability software stack with one click. As shown in the following picture, it includes common observability services (such as Prometheus and Grafana) and metrics and log collection for business scenarios (such as mysql-exporter). After the addon is installed, you can automatically see the built-in dashboards of the platform and quickly build the observability application platform.

Picture 5: Out-of-the-Box Observability Addons

KubeVela allows users that already have observability services to integrate the deployed Prometheus and Grafana services through standard APIs, including the stable, reliable, and unlimited observability services that are self-created or provided by cloud vendors, such as Alibaba Cloud ARMS.

Application-Oriented Observability​

A major feature of KubeVela is that it drives complete application delivery through a top-level application description (YAML). Observability is no exception. The KubeVela controller automatically generates a monitoring dashboard for the application by selecting the O&M features corresponding to logs or metrics.

Picture 6: Application-Oriented Observability

The user portal is an application-oriented dashboard. You can view the component status, version information, and the generated resources in the application. It also contains the dashboard links corresponding to various sub-resources. You can dive into the sub-resources and see detailed information as needed. You can easily understand the full picture of resources from the perspective of applications, whether they are Kubernetes native resources or custom resources.

Observability as Code​

The ability to support the application observability underlying layer is completed by Infrastructure as Code (IaC). This also means KubeVela has made the full link IaC from metric collection (including log collection), parsing, enrichment, storage, data source registration and the dashboard visualization.

The following picture shows how to create a Grafana dashboard using IaC. The IaC module allows you to ”create-dashboard”. Similarly, these include creating data sources, importing dashboards, etc. These routine actions are fully built-in in the KubeVela community, and you can use them directly without learning details. If you want to do some customization, you can orchestrate your process through IaC to customize observability for your platform.

Picture 7: Observability as Code

Unified Management of Multi-Environment Pipeline​

Development, testing, and production of different environment management are the problems of application management. KubeVela's unified application model can deliver applications and initialize the environment. It naturally supports the management of multi-cluster and can install infrastructure components of different clusters with one click. In version 1.4, we released the VelaD project, which can pull up the complete environment of KubeVela on the local machine without relying on Kubernetes. Developers can quickly build and validate applications locally (or in virtual machines) for an environment consistent with production. The management of different environments has been improved in this release.

This release adds an independent pipeline capability. Compared with the application-level workflows of KubeVela, the independent pipeline has the following features:

1. It can manage multiple KubeVela applications and create them across multiple environments.

2. It is not bound to an application and can be used independently. For example, it scales in and out a group of resources, performs process-oriented grayscale release for an application, and performs a group of O&M operations in batches.

3. It is one-time and does not manage resources. Deleting the pipeline in time will not delete the created resources.

4. It is homologous to the execution engine of the application pipeline, which inherits the characteristics of KubeVela's lightweight workflow. Compared with the traditional container-based CI pipeline, KubeVela's pipeline does not rely on containers and does not require additional computing resources when performing various resource operations.

As shown in the following picture, when using it, we can obtain the built-in pipeline template, fill in the environment context parameters, and quickly execute. The following example shows multiple KubeVela platform addons opened through the pipeline. The KubeVela application is behind the addons.

Picture 8: KubeVela Pipeline UI Diagram

Integrate and Manage Configurations​

Configuration management is one of the cores of application management. KubeVela's configuration management mainly helps users share configurations between applications and integrate configurations with third-party external systems to realize unified configuration management. For example, you can connect to external systems (such as container image repositories and Helm repositories) and third-party cloud services (such as the ARMS observability suite mentioned above).

KubeVela configuration management uses the Secret API of Kubernetes as the carrier of configuration data. The application is used for multi-cluster distribution on demand. Applications can read configurations by mounting secrets commonly used in Kubernetes and connecting to the permission system. In addition, if a business application supports reading configurations from a third-party configuration management platform (such as Nacos), you can specify to export the configuration content to the services when defining configuration templates, thus reusing the configuration distribution capabilities of registries.

You can perform configuration management operations from UI/CLI and perform configuration reading and writing operations in application workflows and pipelines. It combines the data transfer capabilities and motion orchestration capabilities of a pipeline to implement diverse scenarios (such as configuration sharing and automatic configuration injection among applications).

Picture 9: Diagram of Configuration Management

The Ecology and Future of KubeVela​

This upgrade of KubeVela adds many application management capabilities to the original application delivery, which is a big step forward for the goal of helping application developers obtain an easy, reliable, and secure software development experience.

Today, KubeVela is a Kubernetes-based controller and a platform with many ecological projects. From the connectivity of the underlying infrastructure, and user-oriented interface at the upper layer, to the extensible ecosystem integration, KubeVela is growing rapidly, fulfilling the beautiful vision of OAM at the beginning of its release - to make the delivery and management of applications simpler, safer, and more reliable!

Picture 10: Tool Ecosystem of KubeVela

In the future, the KubeVela community will continue to enrich out-of-the-box system addons, enrich scenario-oriented application delivery and management solutions, and gradually settle community practices into OAM application standards, building a development, unified, and standardized cloud-native application ecosystem.

This blog introduces the main concepts of an addon and guides you to quickly start building one. Finally, we'll show you the experience of the end user, and how the capabilities provided are glued in a consistent user-friendly experience.

Why use addons​

We typically use addons with addon catalog, which is a registry contains addons that including all kinds of customizations that the KubeVela community carefully crafted. You can download and install these addons with just one click. For example, you can provide the ability to deploy Helm Chart in your KubeVela Application to your cluster by installing fluxcd addon.

Contrary to the convenience of one-key installation, without KubeVela addons, you have to manually install FluxCD in this way:

1. Install FluxCD using Helm Charts or downloaded yaml manifests
2. Glue FluxCD CRD with your system manually, it can be done by adding Component or Trait Definitions in KubeVela

Actually, this is how we install FluxCD before KubeVela v1.1. Although it seems like only 2 steps needed, we found it to be quite troublesome:

1. Complicated installation: users are required to refer to documentations to manually install FluxCD and tackle possible errors
2. Scattered resources: users need to obtain different resource files from different places
3. Hard distribution: users having to download manifests manually makes it hard to distribute all these resources in a uniformed way
4. Lack multi-cluster support: KubeVela emphasizes multi-cluster delivery a lot. However, manual installation makes multi-cluster support hard to carry out.
5. No version management: users need to manage the relationship between definitions, controllers, and corresponding versions by themselves.

KubeVela addons are born to solve these problems.

How KubeVela addons work​

KubeVela addons mainly contain two parts:

• One is the installation of the capability provider, which is usually a CRD Operator. The installation process are intrinsically leveraging KubeVela Application to work.
• Another one is the glue layer, which is OAM Definitions and other extensions. These OAM Definitions usually depend on the capability provider and provide user-friendly abstraction from best practices.

The whole working mechanism of addon is shown as above. KubeVela Application has the multi-clusters capability that help deliver CRD operators of addon into these clusters. The definition files are only used by KubeVela control plane, so they will only existing in the control plane clusters.

Let's build a Redis addon as example, which makes it possible to use Redis Components in Applications, to create a Redis cluster. Such addon will at least include a Redis Operator (to create Redis clusters) and a ComponentDefinition (to define what is a redis Component).

The installation process of an addon includes installing Applications (which includes a Redis Operator), Definitions and etc.

Create your own addon​

We will guide you through the whole process of making an redis addon. The full source code of this guide is located at catalog/redis-operator.

First, we need to consider what our addon can provide to end users. Let's say our Redis addon can provide a Component called redis-failover, which will create a whole Redis cluster when declared in application.

Then we will figure out how to achieve this. To define a redis-failover Component, we need to write a ComponentDefinition. To be able to create a Redis Cluster, we use a Redis Operator as the capability provider.

Now our goals are clear:

• Write an OAM Application, which includes the installation of a Redis Operator. (Refer to template.cue and resources/ in the full source code.)
• Write a ComponentDefinition, which defines Components called redis-failover. (Refer to definitions/ folder in the full source code.)

But before we start coding, we need to understand the structure of an addon directory. We will describe each file later, just have a basic understanding of what files are needed for now.

redis-operator/ 
├── definitions 
│   └── redis-failover.cue 
├── resources              
│   ├── crd.yaml       
│   ├── redis-operator.cue 
│   └── topology.cue       # (Optional)
├── metadata.yaml          
├── parameter.cue         
├── README.md            
└── template.cue       

Let's explain all these files and folders one by one:

1. redis-operator/ is the directory name, which is the same as the addon name.
2. definitions/ folder stores Definitions, including TraitDefinition, ComponentDefinition, and etc.
3. redis-failover.cue defines the ComponentDefinition that how the redis-failover component can be used, and how it integrates with the underlying resources.
4. resources/ folder contain resource files which will be composed as a whole application in template.cue.
5. crd.yaml inside the resources/ folder is the CRD that comes with Redis Operator. Yaml file inside this folder will be applied to Kubernetes directly.
6. redis-operator.cue defines a web-service Component, which installs the Redis Operator.
7. topology.cue is optional, it helps KubeVela build the relationships between Application and the underlying resources.
8. metadata.yaml defines metadata of this addon, including name, version, tags, maintainers and etc. This information can be an overview of addon exposed to any registry.
9. parameter.cue defines the parameters of this addon, end users can use them to customize their addon installation.
10. README.md guides the user to quickly start using this addon.
11. template.cue is the template of the addon which will form the whole Application when installed.

Now let's go through the deep of how to write them one by one.

parameter.cue​

parameter: {
	//+usage=Redis Operator image.
	image: *"quay.io/spotahome/redis-operator:v1.1.0" | string
	// ...omitted

The parameters defined in parameter.cue are customizable by users when installing addons, just like Helm Values. You can access the values of these parameters in CUE later by parameter.<parameter-name>. In our example, image is provided to let the user customize Redis Operator image and can be accessed in redis-operator.cue by parameter.image.

Apart from customizing some fields, you can also do something creative, such as parameterized installation of addons. For example, fluxcd addon have a parameter called onlyHelmComponents. Since fluxcd addon is a big one, which includes 5 different controllers, not all users want such a heavy installation. So if the onlyHelmComponents parameter is set to true by the user, only 2 controllers will be installed, making it a relatively light installation. If you are interested in how this is achieved, you can refer to fluxcd addon for details.

When you design what parameters are provided to the user, you can follow our best practices to make it user friendly.

template.cue and resources/ folder​

OAM Application is stored here, which contains the actual installation process of addons. In our case, we will create an Application that includes a Redis Operator to give our cluster the ability to create Redis clusters.

template.cue and resources/ directory both serve the same purpose -- to define an Application. Aside from historical reasons, the reason why we split it into two places is readability. When we have a ton of resource in template.cue, it will eventually be too long to read. So we typically put the Application scaffold in template.cue, and Application Components in resources directory.

template.cue​

template.cue defines the framework of one Application. Most of the contents here can be boilerplates, they're explained as annotations in the code block.

// package name should be the same as the CUE files in resources directory,
// so that we can refer to the files in resources/.
package main

output: {
	// This is just a plain OAM Application
	apiVersion: "core.oam.dev/v1beta1"
	kind:       "Application"
	// No metadata required
	spec: {
		components: [
			// Create a component that includes a Redis Operator
			redisOperator // defined in resources/redis-operator.cue
		]
		policies: [
		// What namespace to install, whether install to sub-clusters
		// Again, these are boilerplates. No need to remember them. Just refer to the full source code.
		// https://github.com/kubevela/catalog/blob/master/experimental/addons/redis-operator/template.cue
		// Documentation: https://kubevela.io/docs/end-user/policies/references
		]
	}
}
// Resource topology, which can help KubeVela glue resources together.
// We will discuss this in detail later.
// Documentation: https://kubevela.io/docs/reference/topology-rule
outputs: topology: resourceTopology // defined in resources/topology.cue

The output field is the keyword of this template, it contains the Application that will be deployed. In side the application, the spec.components field will reference the objects defined in the resources/ folder.

The outputs field is another keyword that can be used to define any auxiliary resources you want to be deployed along with this addon. These resources MUST follow Kubernetes API.

resources/ folder​

Here we define Application Components that will be referred to in template.cue. We will use a webservice Component to install Redis Operator. Of course, if you are comfortable with extra dependencies (FluxCD addon), you can use helm Components to install the Redis Operator Helm Chart directly. But one of the principles of writing an addon is to reduce external dependencies, so we use the webservice Component, which is a built-in Component of KubeVela, instead of helm.

// resources/redis-operator.cue

// package name is the same as the one in template.cue. So we can use `redisOperator` below in template.cue
package main

redisOperator: {
	// an OAM Application Component, which will create a Redis Operator
	// https://kubevela.io/docs/end-user/components/references
	name: "redis-operator"
	type: "webservice"
	properties: {
		// Redis Operator container image (parameter.image is defined in parameter.cue)
		image:           parameter.image
		imagePullPolicy: "IfNotPresent"
	}
	traits: [
		// ...omitted
	]
}

You can refer to the annotation inside the code block to learn the detail usage for every field.

Highlights of the glue capability provided by KubeVela​

One of the notable features is Topology Rules (or Resource Topologies, Resource Relationships). Although not required, it can help KubeVela build the topological relationships of the resources managed by a KubeVela Application. This is how KubeVela can glue all kinds of resources together into an Application. It is especially helpful when we use Kubernetes Custom Resource(CR) (which is exactly the case in this example).

// resources/topology.cue

package main

import "encoding/json"

resourceTopology: {
	apiVersion: "v1"
	kind:       "ConfigMap"
	metadata: {
		name:      "redis-operator-topology"
		namespace: "vela-system"
		labels: {
			"rules.oam.dev/resources":       "true"
			"rules.oam.dev/resource-format": "json"
		}
	}
	data: rules: json.Marshal([{
		parentResourceType: {
			group: "databases.spotahome.com"
			kind:  "RedisFailover"
		}
		// RedisFailover CR manages the three resource below
		childrenResourceType: [
			{
				apiVersion: "apps/v1"
				kind:  "StatefulSet"
			},
			// Topologies of Deployment and etc. are built-in.
			// So we don't need to go deeper to Pods.
			{
				apiVersion: "apps/v1"
				kind:  "Deployment"
			},
			{
				apiVersion: "v1"
				kind:  "Service"
			},
		]
	}])
}

In our case, the redis-failover Component will create a CR, named RedisFailover. If we don't have Topology Rules, although we know RedisFailover is managing several Deployments and Services, KubeVela doesn't magically know this. So we can tell KubeVela our understanding with Topology Rules. Once KubeVela know what's inside a RedisFailover, it will know the relationships of all the resources inside an Application.

You can refer to run our addon to see the real user experience.

definitions/ folder​

Definitions folder contains KubeVela Definitions, including ComponentDefinitions, TraitDefinitions, and etc. This is the most important part of an addon as it provides the real capability to end users. With the Component and Trait types defined here, users can use them in their applications.

Writing Definitions for an addon is the same as writing regular OAM Definitions, this is a huge topic so we won't go to deep in our addon introduction, you can refer to following docs to learn how to write each kinds of definitions.

• Component Definition
• Trait Definition
• Policy Definition
• Workflow Step Definition.

In our redis addon example, we should refer to Component Definition and Redis Operator to write our ComponentDefinition. We'll name this component type as redis-failover. It will create a CR called RedisFailover. With RedisFailover created, the Redis Operator in our addon Application will create a Redis cluster for us.

You can refer to the source code here.

metadata.yaml​

Just the name implies, these are all metadata of an addon, including addon name, version, system requirements, and etc. You can refer to this documentation for the detail list. The example listed below contains all the fields available.

# addon name, the same as our directory name
name: redis-operator
# addon description
description: Redis Operator creates/configures/manages high availability redis with sentinel automatic failover atop Kubernetes.
# tags to show in VelaUX
tags:
  - redis
# addon version
version: 0.0.1
# addon icon
icon: https://xxx.com
# the webpage of this addon
url: https://github.com/spotahome/redis-operator
# other addon dependencies, e.g. fluxcd
dependencies: []

# system version requirements
system:
  vela: ">=v1.5.0"
  kubernetes: ">=1.19"

Run our addon​

Now we have finished most of our work. It is time to run it! If there's any details that were skipped, you can download the full source code to complete them.

After we finish all the redis-operator addon, we can use command vela addon enable redis-operator/ to enable it locally. It can help us debug and write docs about the end user experience.

As to say our example addon, you can refer to the README to learn how should be introduced.

By using this addon, users only need to write 4 lines of yaml to get a Redis cluster with 3 nodes! Contrary to manually installing Redis Operator, even manually managing Redis instances, the use of an addon greatly improves user experience.

apiVersion: core.oam.dev/v1beta1
kind: Application
metadata:
  name: redis-operator-sample
spec:
  components:
    # This component is provided by redis-operator addon.
    # In this example, 2 redis instance and 2 sentinel instance
    # will be created.
    - type: redis-failover
      name: ha-redis
      properties:
        # You can increase/decrease this later to add/remove instances.
        replicas: 3

A whole tree of complicated resources are created with just a few lines of yaml, as shown in the figure below. Since we have written Topology Rules for our addon, users can easily see the topology of all the resources (Pods, Services) of a Redis Cluster. They are now not limited to the level of observability of KubeVela Applications, instead, they can have a glance on the statuses of low-level resource. For example, we can see certain Redis Pods are still not ready in our figure:

Users can also choose the low-level resources of our sample Application, i.e., 3 Redis Pods and 3 Sentinal Pods, when executing vela exec/log/port-forward.

vela status can get the status of an Application. With Topology Rules, we can take one step further -- find out endpoints in an Application directly. In our example, users can get endpoints of Redis Sentinel to connect to by:

Wrap up​

By the end of this guide, you probably already have a good grasp of what addons do and how to make addons. As a conclusion, you'll get the following benefits:

1. Extend platform capability into a bundle that can be easy to use and share with the whole community.
2. Orchestrate and template all the infrastructure resources into multi-clusters in a flexible way powered by KubeVela application and CUE.
3. Provide consistent experience for end users with all kinds of extended capabilities.

At last, if you have successfully built your own addon, you're extremely welcomed to contribute it to the addon catalog. As a result, everyone in the KubeVela community can discover and benefit from extending your powerful platform capabilities!

KubeVela has released five major versions over the past year. Each iteration is a leap forward. The release of version 1.1 brought the ability to connect multiple clusters. Version 1.2/1.3 brought an extended system and a more developer-friendly experience. Version 1.4 introduced a security mechanism in the complete process. The release of 1.5 today brings us closer to KubeVela's vision of making application delivery and management easier. Along the way, we have stayed true to the same design philosophy and built a platform that automates the complexity of the underlying differentiated infrastructure without losing scalability. It helps application developers upgrade from business development to cloud-native R&D at a low cost. Technically, it focuses on the complete process from code to cloud and from application delivery to management. It refines the framework capabilities of connecting infrastructure based on the Open Application Model (OAM). As shown in Figure 1, KubeVela has covered the complete capabilities of application definition, delivery, O&M, and management, all of which are based on OAM scalability (OAM Definition) to connect to ecological projects in an addon way. In essence, each definition converts the experience of a specific capability into a reusable best practice module, which can be shared by enterprises or communities through addon packaging.

Figure 1: Core Structure of KubeVela Extensibility

The blossoming of atomic capabilities in cloud-native is an asset that raises the threshold in the field. Platform builders need to learn a large number of open-source projects and aggregate experience in multiple fields. It often takes months or longer to build a cloud-native support platform for the enterprise. However, the platform often passes the complexity to the application developer, which means they have to learn a lot of additional knowledge. KubeVela's design concept and existing technical achievements may help you quickly enter the cloud-native world. KubeVela's major addon integration specifications and unified application delivery experience in version 1.5 and previous versions have solved the problem of discrete atomic capabilities in the cloud-native field.

Addon Specification Upgrade and a More Flexible Definition​

The KubeVela addon mechanism has been popular among community users since version 1.2. There are nearly 50 addons in the community addon repository. Nearly 50 developers participated in the contribution of the addon.

Please see this link for more information

Starting from version 1.5, addon developers could have a better experience. Addon definition, distribution, and visual management are all improved. In addition to using YAML to define addons, developers can fully use CUE to define addons if they want a more flexible combination of addon resources and higher-level parametric control. The current specifications for addon definitions include the following parts:

• Combined with the resources directory, template.cue or template.yaml is used to define the runtime of an addon. For example, you need to run an Operator behind to extend a workload. This part is optional. For example, some lightweight addons can have no runtime behind them or reuse other runtimes. The combination of YAML and CUE configuration can cover most scenarios.

• The definitions directory, which stores the definition, is the core part of the addon and defines how the capabilities extended by the addon are used by users.

• The schema directory, which is used to assist the definition, defines custom rendering rules for related definitions on the UI side.

• The views directory, which stores the syntax definition of VelaQL, is used to extend the query capabilities of VelaQL.

• Addon metadata definition files like metadata.yaml and readme.md describe the basic information and environment requirements of the addon.

Please refer to the document for details.

Here is an example of integrating the delivery capability of the Helm Chart package. Currently, projects in the community like FluxCD or ArgoCD provide the atomic capability of deploying Chart packages. Their implementations are different, and each has its advantages. For KubeVela users, these two projects can be introduced through addons. As shown in Figure 2, we need to define a standard API for the end user to expose the necessary parameters to the end user according to the specific situation of the enterprise.

Figure 2: Process of KubeVela Extending the Helm Chart Package

As shown in Figure 3, according to the standard API, the frontend UI can automatically generate the corresponding interactive page to help the end user easily and conveniently deploy the Helm Chart package. The platform side automatically generates the driver configuration of the underlying capability based on the user's input parameters and addon definitions and intelligently sends the relevant status feedback to the user. These are based on addon specifications, such as integrating FluxCD. This project includes multiple controllers providing different atomic capabilities. First, we use template.cue to define the deployment method of FluxCD and deploy different components based on different parameter inputs. Then, the user experience is defined by definitions and schema directories.

Please see this link for more information.

Figure 3: Interaction for KubeVela to Deliver Helm Chart Packages

Function Interpretation Based on Addon Extensions​

Telemetry Integrating Prometheus + Grafana + Exporters to Support System Observability​

The system of application observability is closely related to application release. A good application observability system can make the management of application reliability easy. The KubeVela community includes application observability into the core features. In version 1.5, the community selected the KubeVela system observability as a case for system capability development. The following key points have been realized:

1. The multi-cluster observable infrastructure is installed through addons easily. First, we formed an observable addon set around the solution of Prometheus + Grafana + Exporters. It conveniently installs basic capabilities for different basic environments.

2. It supports one-click open of multi-cluster Metrics data aggregation and uses the Thanos Query solution to implement multi-cluster metric aggregation query and visualization. A similar solution will gradually cover the Logger and Tracing dimensions.

3. Grafana IaC: The configuration of Grafana data sources and dashboards are described through application models. The innovative application of extensible API makes the Grafana API change into runtime that KubeVela can operate.

4. The Grafana dashboard is generated automatically. You can enable the Grafana addon to generate a KubeVela system observability dashboard automatically.

Figure 4 shows the dashboard of KubeVela system operating indicators. The dashboard is automatically generated through the IaC system. You only need to enable the corresponding addon.

Figure 4: KubeVela System Observable Dashboard

Figure 5 shows the monitoring dashboard of the Kubernetes API Server service connected to KubeVela. Use addons to issue Exporters to all sub-clusters, expose data to the Prometheus service of each cluster, and aggregate it to the control cluster for centralized visualization. You can complete the data monitoring and dashboard access of many clusters at the same time.

Figure 5: KubeVela Multi-Cluster API Observation Dashboard

In the next release, the community will gradually integrate the unified description and delivery of application observability into the application delivery process. It covers Metric, Logger, and Tracing data acquisition, intermediate processing and transmission, storage and analysis, alerts and visualizations, and applications to the complete process of the application release workflow.

Reference: https://kubevela.net/docs/platform-engineers/operations/observability

Integrate CloudShell for CLI and UI Collaborated Application Delivery​

Manipulating application delivery through a CLI black screen is convenient, easily replicated, and can be done in batches. Developers are fond of it. Delivering application interactions through UI is more elegant. Process operations help reduce learning costs and achieve stricter enterprise security controls. A high degree of visualization can help us master the application and perform relevant operations anytime and anywhere. KubeVela was very different in the CLI and UI dimensions in the past version, and data in the two dimensions do not communicate with each other. If the two terminal methods can be combined, application delivery and management can be smoother. In version 1.5, KubeVela introduced the CloudShell addon, which provides a Web Shell terminal for UI users. The unified portal solves the problem of CLI and UI separation and brings more capabilities. The main changes to this process are listed below:

1. Convenient Toolset: Unlike other platforms, which mainly provide Web Shell capabilities for entering the application runtime, CloudShell generates a terminal environment for each user, including CLI tools (such as Vela and Kubectl). You can manage multiple applications in the same environment.

2. Automatic Authorization: Users do not need to care about how to allocate KubeConfig. The system automatically completes the authorization of the black screen environment according to the permissions owned by UI users, thus realizing the consistency of basic permissions between the white screen and black screen.

3. Automatic Environment Recycling: The maximum survival time of each user's terminal environment is one hour. Automatic recycling after expiration prevents excessive resource consumption.

4. Enhanced Vela CLI Capability: It re-implemented log, status, exec, port-forward, and other operation commands for Debug applications, realizing seamless compatibility for differentiated workloads under applications, which allows users to complete relevant operations without awareness. Whether it is the basic Deployment resource, a Helm-packaged load resource set, or a custom Operator-driven workload, Vela can automatically discover the underlying operation objects related to commands.

5. Data Automatic Synchronization: CLI can create and update applications. Changes will be visualized on the UI until the user chooses to take over the application and subsequent releases through UI.

Figure 6: KubeVela CloudShell Operating Terminal

Integrate OpenKruise Rollout to Provide Canary Release Capabilities​

The KubeVela community incubated the Rollout project in the early days, similar to the implementation model of Argo Rollout. It works in the form of a new workload, mainly realizing the ability to release in batches. With the development of the community, KubeVela focuses more on the application global control layer and addon extension capabilities. Therefore, the rollout implementation at the workload level has been transferred to the OpenKruise community. With the joint efforts of both parties, canary release capabilities can be implemented for various workloads (such as native Deployment, StatefulSet, and OpenKruise extended workload CloneSet). At the same time, when it coexists with the Helm delivery mode in KubeVela, it can implement the canary release for Helm chart package applications without any changes. This is innovative in the industry and very convenient for users. Kruise Rollout is integrated into the KubeVela ecosystem as an addon. KubeVela users only need to enable addons to configure Rollout Trait in application components. At the same time, it can cooperate with gateway rules (such as Gateway and HTTPRoute). This implementation has the following advantages:

1. No Intrusion and Binding: Rollout capability is introduced through the bypass, and users do not need to make other changes to the existing application configuration. The introduction cost is low and can be removed at any time.

2. Easy to Use: With a simple switching rule of the traffic configuration, combined with KubeVela's UI visualization, you can effectively observe the change in the number of replicas in the Rollout process and the additional resource relationship introduced.

3. Good Compatibility: No matter what workload the user uses for packaging (Helm or custom Operator), Rollout can work in bypass form after discovering the current underlying load resources.

Reference: https://kubevela.net/docs/end-user/traits/rollout

VelaUX Adds Multi-Environment Differentiated Visualized Configuration​

VelaUX has had the multi-environment deployment capability since its launch. Until version 1.5, it supports the differentiation of multiple environments for user visual editing, truly matching the needs of user multi-environment application release. Override Policy configurations support the dimension differentiation of environments, clusters, or namespaces and the unified management of application baseline and configurations..

As shown in Figure 7, the Application Policy has a variety of available options built in, including differentiated configuration, application multi-cluster policies, application maintenance policies, and GC policies. Users can easily configure corresponding policies according to their needs through UI guidance.

Figure 7: KubeVela Policy Add/Edit Window

In version 1.5, the following convenient features are added before and after deployment for different environments:

• DryRun (Trial Run) Capability: You can perform a DryRun before deploying an environment. Then, you can evaluate whether the application configuration meets expectations based on the UI feedback results to prevent the incorrect configuration from affecting the stability of online services after deployment.

• Environment Difference Insight: When switching to different environment views, the comparison of local configuration and deployment configuration is automatically compared. If there is a difference, the user will be prompted, and the configuration items of the difference will be displayed. It can prevent configuration drift or forgetting to go online for configurations scheduled to go online.

• Version Detail Query and Difference Comparison: You can view the application configuration rendering results of each version on the Version Management page. You can also compare the version configuration with the current running configuration or the latest local configuration. It is convenient for users to track the configuration change process.

Reference: https://kubevela.net/docs/tutorials/multi-env

Improve Application Engine Capabilities​

In addition to the preceding addon capabilities, a large number of updates have been made to the application engine. The performance is significantly improved, and the CPU consumption during workflow execution is reduced by 75%. The number of parallel executions has increased significantly. The important changes are listed below:

1. Timeout control is added to Workflow. In the Workflow step, configure the timeout period. When the execution time is greater than the timeout period, Workflow will end and become terminated.

2. Workflow adds conditional judgment. Configure the If field in the Workflow step and support reading data from status or input to determine whether the current step needs to be executed. It also supports the If Always mechanism and supports scenarios where some steps need to be executed under any circumstances.

3. Workflow supports display switching mode, DAG, or the default StepByStep.

4. Add a shared resource policy. Different applications can describe the same resources, such as a namespace or a ConfigMap. If you set this parameter to a shared resource, it does not conflict.

5. The optimization of the application resource construction algorithm improves query efficiency in different scenarios and makes it easier to extend custom rules. It also adds some default rules.

Please refer to this link for more information about changes.

Summary​

Overall, KubeVela 1.5 has made significant progress in multiple dimensions (such as product capabilities, community ecology, and benchmark users). User cases include finance, intelligent manufacturing, the Internet, and other industries. We expect more users to share practical experiences and help the KubeVela community find a more accurate way forward. Version 1.6 plans to bring more complete application observability, independent application workflow capabilities, continuous release control of multiple applications, and collaboration with observable systems. Developers with relevant needs and ideas can participate in community discussions at any time.

Helm is an application packaging and deployment tool of client side widely used in the cloud-native field. Its simple design and easy-to-use features have been recognized by users and formed its ecosystem. Up to now, thousands applications have been packaged using Helm Chart. Helm's design concept is very concise and can be summarized in the following two aspects:

1. Packaging and templating complex Kubernetes APIs and then abstracting and simplifying them into small number of parameters

2. Giving Application Lifecycle Solutions: Production, upload (hosting), versioning, distribution (discovery), and deployment.

These two design principles ensure that Helm is flexible and simple enough to cover all Kubernetes APIs, which solves the problem of one-off cloud-native application delivery. However, for enterprises with a certain scale, using Helm for continuous software delivery poses quite a challenge.

Challenges of the Continuous Delivery of Helm​

Helm was initially designed to ensure simplicity and ease of use instead of complex component orchestration. Therefore, Helm delivers all resources to Kubernetes clusters during the application deployment. It is expected to solve application dependency and orchestration problems automatically using Kubernetes' final-state oriented self-healing capabilities. Such a design may not be a problem during its first deployment, but it is too idealistic for the enterprise with a certain scale of production environment.

On the one hand, updating all resources when the application is upgraded may easily cause overall service interruption due to the short-term unavailability of some services. On the other hand, if there is a bug in the software, it cannot be rolled back in time, which may cause more trouble and make it difficult to control. In some serious scenarios, if some configurations in the production environment have been manually modified in O&M, the original modifications would be overwritten due to the one-off deployment of Helm. What's more, the previous versions of Helm may be inconsistent with the production environment, so it cannot be recovered using rollback. All of these add up to a larger area of failure.

Thus, with a certain scale, the grayscale and rollback capabilities of the software in the production environment are extremely important, and Helm itself cannot guarantee sufficient stability.

How Do We Enable Canary-Rollout for Helm?​

Typically, a rigorous software upgrade process follows a similar process: It is roughly divided into three stages. The first stage upgrades small number of pods (such as 20% ) and switches a small amount of traffic to the new version. After completing this stage, the upgrade is paused first. After manual confirmation, continue the second phase, which is to upgrade a larger proportion of pods and traffic (such as 90% ), and pause again for manual confirmation. In the final stage, the full upgrade to the new version is completed, and the verification is completed, thus the entire rollout process is completed. If any exceptions, including business metrics, are found during the upgrade, such as an increase in the CPU or memory usage rate or excessive log requests error 500, you can roll back quickly.

The image above is a typical canary rollout scenario, so how do we complete the above process for the Helm chart application? There are two typical ways in the industry:

1. Modify the Helm chart to change workloads into two copies and expose different Helm parameters. During the rollout, the images, number of pods, and traffic ratio of the two workloads are continuously modified to implement the canary rollout.
2. Modify the Helm chart to change the original basic workload to a custom workload with the same features and with phased rollout capabilities and expose the Helm parameters. It's these canary rollout CRDs that are manipulated during the canary rollout.

The two solutions are complex with considerable modification costs, especially when your Helm chart is a third-party component that cannot be modified or cannot maintain a Helm chart. Even if the two are modified, there are still stability risks compared to the original simple workload model. The reason is that Helm is only a package management tool, and it is incompatible with the canary rollout or workloads management.

When we have in-depth communication with large number of users in the community, we find that most users' applications are not complicated, among which the most are classic types (such as Deployment and StatefulSet). Therefore, through the powerful addon mechanism of KubeVela and the OpenKruise community, we have made a canary rollout KubeVela Addon for these qualified types. This addon helps you easily complete the canary rollout of the Helm chart without any migration and modification. Also, if your Helm chart is more complicated, you can customize an addon for your scenario to get the same experience. Let's take you through a practical example (using Deployment Workload as an example) to get a glimpse of the complete process.

Use KubeVela for Canary Rollout​

Prepare the Environment​

• Install KubeVela

$ curl -fsSL https://kubevela.io/script/install-velad.sh | bash
velad install

See this document for more installation details.

• Enable related addon

vela addon enable fluxcd
vela addon enable ingress-nginx
vela addon enable kruise-rollout
vela addon enable velaux

In this step, the following addons are started:

1. The fluxcd addon helps us enable the capability of Helm delivery.
2. The ingress-nginx addon is used to provide traffic management capabilities of canary rollout.
3. The kruise-rollout provides canary rollout capability.
4. The velaux addon provides interface operation and visualization.

• Map the Nginx ingress-controller port to local

vela port-forward addon-ingress-nginx -n vela-system

First Deployment​

Run the following command to deploy the Helm application for the first time. In this step, the deployment is done through KubeVela's CLI tool. If you are familiar with Kubernetes, you can also deploy through kubectl apply. The two work the same.

cat <<EOF | vela up -f -
apiVersion: core.oam.dev/v1beta1
kind: Application
metadata:
  name: canary-demo
  annotations:
    app.oam.dev/publishVersion: v1
spec:
  components:
  - name: canary-demo
    type: helm
    properties:
      repoType: "helm"
      url: "https://wangyikewxgm.github.io/my-charts/"
      chart: "canary-demo"
      version: "1.0.0"
    traits:
    - type: kruise-rollout
      properties:
        canary:
          # The first batch of Canary releases 20% Pods, and 20% traffic imported to the new version, require manual confirmation before subsequent releases are completed
          steps:
          - weight: 20
          # The second batch of Canary releases 90% Pods, and 90% traffic imported to the new version.
          - weight: 90
          trafficRoutings:
          - type: nginx
EOF

In the example above, we declare an application named canary-demo, which contains a Helm-type component (KubeVela also supports other types of component deployment), and the parameter of the component contains information (such as chart address and version).

In addition, we declare the kruise-rollout OAM trait for this component, which are the capabilities added to the component after the kruise-rollout addon is installed. The upgrade policy of Helm can be specified. In the first stage, 20% of pods and traffic are upgraded. After manual approve, 90% is upgraded. Finally, the full version is upgraded to the latest version.

Note: We have prepared a chart to demonstrate the intuitive effect (reflecting the version changes). The body of the Helm chart contains a Deployment and Ingress object, which is the most common scenario when the Helm chart is made. If your Helm chart is also equipped with the resources above, you can also use this example to canary rollout your helm chart.

After the deployment is successful, we use the following command to access the gateway address in your cluster, and you will see the following result:

$ curl -H "Host: canary-demo.com" http://localhost:8080/version
Demo: V1

In addition, through the resource topology graph of VelaUX, we can see that the five V1 pods are all ready.

Upgrade an Application​

Apply the following yaml to upgrade your application:

cat <<EOF | vela up -f -
apiVersion: core.oam.dev/v1beta1
kind: Application
metadata:
  name: canary-demo
  annotations:
    app.oam.dev/publishVersion: v2
spec:
  components:
  - name: canary-demo
    type: helm
    properties:
      repoType: "helm"
      url: "https://wangyikewxgm.github.io/my-charts/"
      chart: "canary-demo"
      # Upgade to version 2.0.0
      version: "2.0.0"
    traits:
    - type: kruise-rollout
      properties:
        canary:
          # The first batch of Canary releases 20% Pods, and 20% traffic imported to the new version, require manual confirmation before subsequent releases are completed
          steps:
          - weight: 20
          # The second batch of Canary releases 90% Pods, and 90% traffic imported to the new version.
          - weight: 90
          trafficRoutings:
          - type: nginx
EOF

We noticed that the new application only has two changes compared to the first deployment:

1. We upgrade the annotation of the app.oam.dev/publishVersion from V1 to V2. This shows this revision is a new version.

2. We upgrade the version of the Helm chart to 2.0.0. The tag of the deployment image in this version of the chart is upgraded to V2.

After a while, we will find that the upgrade process stops at the first batch we defined above. Only 20% of pods and traffic are upgraded. At this time, if you execute the command above to access the gateway multiple times, you will find that Demo: V1 and Demo: V2 appear alternately, and there is a probability of almost 20% of getting the result of Demo: V2.