Objective:

This project aimed to maximize the efficiency of WEDOS’s data centers, with a focus on utilizing oil cooling technology and automating server management processes.

At the time of implementation, WEDOS operated two data centers. The second data center, fully built with oil cooling, was designed primarily for the Top-load HPE Moonshot 1500 Chassis, each housing 45 HPE ProLiant m710x Server Blades.

The entire data center was constructed using these servers.

Implementation:

A distinctive feature of the implemented system was its stateless automatic provisioning. Servers booted from a network-boot server and connected to the assigned Kubernetes cluster, which then scheduled user workloads on them. Some servers were dedicated to cloud platform, while others managed hosting workloads like web servers, databases, and mail servers. Due to stateless nature the servers can be simple updated by reboot operation, during next-booting process they will fetch new images from the network-boot server.

To manage the entire infrastructure, the Kubefarm project was developed. It facilitated the launch of Kubernetes control planes for user servers, provided DHCP configuration, and operated a network boot server for node loading.

Outcomes:

The result was a fully managed infrastructure-as-a-code. System images for servers were compiled using Dockerfile too, allowing seamless cluster updates and convenient management through a specially developed utility, moonshotctl. This approach significantly enhanced the data center’s operational efficiency and resource management.

The project was open sourced, and the expertise was transferred as much as possible to the infrastructure team, where the project is used and developed to this day.

Demonstration:

Argo CD – Applications / kubefarm-foo

Kubefarm cluster deployed using Argo CD

Additional Links:

Kubernetes-in-Kubernetes and the WEDOS PXE bootable server farm

When you own two data centers, thousands of physical servers, virtual machines and hosting for hundreds of thousands sites, Kubernetes can actually simplify the management of all these things.

kubefarm: Automated Kubernetes deployment and the PXE-bootable servers farm

Automated Kubernetes deployment and the PXE-bootable servers farm – GitHub – kubefarm/kubefarm: Automated Kubernetes deployment and the PXE-bootable servers farm

kubernetes-in-kubernetes: Deploy Kubernetes in Kubernetes using Helm

Deploy Kubernetes in Kubernetes using Helm. Contribute to kubefarm/kubernetes-in-kubernetes development by creating an account on GitHub.

Configuring routing for MetalLB in L2 mode

In this article I will show you how to configure source-based and policy-based routing for the external network on your cluster.

Presentations:

Objective:

The project aimed to develop a cloud virtualization system for creating VPS hosting and a public cloud platform.

Implementation:

Given the large number of servers, the management of OpenNebula was completely shifted into Kubernetes. This transition was facilitated by the development of the kube-opennebula project, which included fully automatic configuration of OpenNebula with all users, images, and data stores.

A highly-customizable script was used for node configuration, executed by Kubernetes on each node. The storage system initially utilized LINSTOR SDS and was later transitioned to HPE 3par. An extensive API reference was prepared for integration with billing systems and the client’s system, covering various scenarios. The project also involved the development of a system for automatic preparation and unification of images. Several storage drivers like linstor_un and 3par were developed, along with a monitoring system and a multitude of plugins.

Outcomes:

The project successfully enabled the maintenance of multiple environments, such as production, devel and staging. The similarity for enviroments allowed for seamless development and integration of new features, significantly enhancing the efficiency and scalability of the cloud infrastructure.

Screenshots:

Additional links:

kube-opennebula: Helm chart and OpenNebula images ready to deploy on Kubernetes

Helm chart and OpenNebula images ready to deploy on Kubernetes – GitHub – kvaps/kube-opennebula: Helm chart and OpenNebula images ready to deploy on Kubernetes

addon-linstor_un: Community driven full-feature Linstor storage driver for OpenNebula

Community driven full-feature Linstor storage driver for OpenNebula – GitHub – OpenNebula/addon-linstor_un: Community driven full-feature Linstor storage driver for OpenNebula

opennebula-images: OpenNebula Images Generator

OpenNebula Images Generator. Contribute to kvaps/opennebula-images development by creating an account on GitHub.

Virtual Desktop Infrastructure (VDI) is a technology where users operate on remote virtual machines instead of traditional physical computers.

Implementation:

Users utilize thin clients as their local computers. These are simple single-board computers that boot over the network and do not have a physical hard drive for data storage. The work process initiates only when a user connects to the virtualization system. For every specific environment, a Windows virtual machine with pre-installed software is available.

Key Features:

• The virtual machine acts as a template that can be independently updated. Upon connection, a snapshot of the selected virtual machine is created for the user, who then connects using the high-performant Spice protocol.
• Multiple screen support is available.
• Both local or remote USB devices can be passed through to the virtual machine using usbredir protocol.
• Users can be granted full administrative rights without risk, as the specific instance of the virtual machine is deleted right after disconnection.

Technologies Used:

• LTSP (Linux Terminal Server Project)
• Usbredir (Part of Spice specification)
• OpenNebula (Cloud Computing Platform)

Additional links:

one-connect

Simple and secure access to your OpenNebula VMs via SPICE protocol.

usbredirtools

Collection of scripts for passing through the usb-devices to VMs

Objective:

The goal was to create an efficient backup system capable of reading data in real-time from the disks of virtual machines across several hundred servers and transferring it to a backup servers.

Researched:

• Client Side: A comprehensive analysis of the efficiency and performance of various open-source backup systems was undertaken. Restic emerged as the most optimal solution based on this research.
• Server Side: Different methods and filesystems were evaluated for their performance capabilities to establish an effective backend for the backup system.

Used technologies:

• Restic as highly efficient and feature-reach backup solution.
• ZFS with dRAID and rest-server used for backend. This setup allowed for efficient data storage and swift recovery in the event of disk failures, even with large-capacity disks.
• Kubernetes used for orchestration.

Outcomes:

This project successfully established a robust and scalable backup solution, enhancing data security and recovery capabilities across a large server network. The use of open-source tools like Restic and the ZFS file system, combined with the orchestration power of Kubernetes, resulted in a highly reliable and efficient backup system suitable for virtual machines backed by block storage.

The same approach was later adapted for the efficient backup of databases, demonstrating its versatility and scalability.

Additional links:

Backup storage for thousands of virtual machines using free tools

We needed a backup solution to handle thousands of backups each week and do it as fast and efficiently as possible.

Restic: Effective Backup from Stdin

A ready-made solution for setting up effective backup from Stdin, with deduplication and automatic cleaning of the repository from old copie

Objective:

This research aimed to develop an innovative design for a Container Storage Interface (CSI) driver suitable for shared LUNs in Kubernetes, focusing on efficient snapshot creation and support for ReadWriteMany mode for live migration of virtual machines in KubeVirt.

Researched:

• Linux Kernel Subsystems: The study delved into advanced Linux kernel storage technologies, including ublk (Userspace Block Device Driver), VDPA (Virtual Data Path Acceleration), and vDUSE (Virtual Device User Space).
• QEMU Storage Daemon: Investigated the specialized daemon responsible for data storage, along with its API.
• Fencing Methods: Explored various techniques for handling faulty nodes.
• Performance Analysis: Assessed the performance of different snapshot technologies, such as LVM, LVM Thin, QCOW2, and various device export technologies, including ublk, nbd, vhost, and vdpa.

Outcomes:

• Design Documentation: Produced a comprehensive design of all components for the Kubernetes CSI driver, complete with detailed documentation that outlines the interaction of all components. This encompassed the design of the driver’s components in Kubernetes, their low-level API interactions, and a high-level user interface for configuring the storage subsystem.
• Community Engagement: The findings were presented to and discussed with the wider community, as well as with the core developers of QEMU.

Additional links:

Kubernetes CSI driver components

Graphical representation of dynamic PVC provisioning in Kubernetes

The Evolution of Network Virtualization Technologies in Linux

How and why device virtualization technologies developed, how they work, their strengths and weaknesses.

LVM+QCOW2: creating a perfect CSI driver for shared SAN in Kubernetes

Building a fast and universal driver with clustered LVM and the QCOW2 for using any SAN-like storage system in K8s.

Objective:

The goal was to implement cloud system functionality on the Deckhouse Kubernetes platform, ensuring maximum compatibility and management through Kubernetes objects.

Researched:

Several open-source cloud platforms, including OpenStack, OpenNebula, CloudStack, Ganeti, and KubeVirt, were evaluated. After thorough comparison, KubeVirt was chosen as the most suitable solution.

Implementation:

• Network Improvements: The initial network implementation in KubeVirt was not satisfactory, prompting significant enhancements to achieve maximum performance and stability. The realization of the new macvtap method led to a 20% reduction in network latency for standard CNI binding method.
• IPAM: A specialized IP Address Management service was developed to allocate static IP addresses for virtual machines, allowing them to operate alongside the ephemeral pods in Kubernetes.
• Distributed Router: A distributed router was implemented to monitor the location of virtual machines within the cluster and dynamically adjust routes accordingly.
• Cilium modifications: Cilium updated with custom patches to add functionality for preserving IP and MAC addresses during the live migration of virtual machines to different nodes.

Outcomes:

After the release of the alpha version, the development transferred to a dedicated virtualization team. This included the complete transfer of expertise and assistance in designing the API for the next version.

Additional Links:

Deckhouse v1.43 introduces the next-generation virtualization system

Over the past few years, we at Flant have been keeping a close eye on the technology leaders in the cloud-native space. But it’s more than…

Macvtap binding mode for pod network #7648

This PR adds macvtap networking mode for binding podNetwork.
This method does not requires any bridges, as a result we have less latency to communicate with other pods and VMs.

Allow live-migration for pod network in bridge mode #7768

This PR enables the live-migration feaure for pod networking in any mode.

Presentation:

Objective:

Prior to implementing a cloud virtualization system, it was essential to establish a reliable block storage solution for the data of virtual machines as well container workloads.

Researched:

In collaboration with experts, we developed specialized testing tool set. We conducted an extensive evaluation of various storage solutions, including Ceph, Mayastor, Vitastor, and LINSTOR.
The findings from our testing were publicly released and extensively discussed within the community.

Implementation:

The initial implementation of the module included:

• Docker images for LINSTOR and all it’s components built from source code;
• Configuration tool;
• Documentation;
• Integrating tests;
• Grafana dashboard;
• Prometheus alerts with detailed courses of actions.

Outcomes:

The LINSTOR module was successfully implemented and has been operational since the stable release of Deckhouse v1.33.
There were detailed training and knowledge transfer sessions, as well as several public presentations.

Additional links:

Comparing Ceph, LINSTOR, Mayastor, and Vitastor storage performance in Kubernetes

Choosing an Open Source storage solution for your K8s clusters? These benchmarks might help you find the most suitable option for your case.

Comparing LINSTOR, Ceph, Mayastor, & Vitastor Storage Performance in Kubernetes

This is a guest post by Andrei Kvapil, Solutions Architect at Flant.

LINSTOR / DRBD Dashboard | Grafana Labs

LINSTOR and DRBD dashboard ready to use with piraeus-operator.

The linstor module | Deckhouse

Deckhouse uses the linstor module to manage a replicated block storage in the Kubernetes cluster.

Presentation: