Every year, we estimate the proportion of the hundreds of companies that have contributed to Istio in the past year, and use that metric to proportionally allocate the nine Contribution Seats on our Steering Committee.

After that, four Community Seats are voted for by our project members, with candidates being from companies that did not receive Contribution Seats.

In February, we announced the Contribution Seat allocation, and invited candidates to stand for the Community Seat elections. As the election officer, I am pleased to announce the results of that election, as well as the individuals who will represent the top contributors.

Community Seats

Five candidates stood for our four open seats. Using Condorcet-method ranked voting, the following four candidates have been elected:

• Faseela K, Ericsson Software Technology
• Craig Box, Independent
• Tyler Schade, GEICO Tech
• Zhonghu Xu, Huawei

We’re excited to welcome this fantastic group of community leaders. Faseela is returning for her fourth term on Steering and also serves on the CNCF Technical Oversight Committee. Craig has been on the Steering Committee since 2020 and recently ended his term as the maintainer representative on the CNCF Governing Board. Tyler is a new face on Steering, bringing an end-user perspective from GEICO Tech, where he leads a team building a custom service mesh on Istio. Zhonghu has been an Istio contributor since the project’s early days, is a core maintainer in multiple working groups, and is transitioning this year from a Contribution Seat to a Community Seat.

Contribution Seats

Our supporting companies have made their choices for the nine Contribution Seats. They will be held by:

• Zack Butcher (Tetrate)
• Rob Cernich (Red Hat)
• John Howard (Solo.io)
• Steven Jin Xuan (Microsoft)
• Jack Ma (Microsoft)
• Keith Mattix (Microsoft)
• Louis Ryan (Solo.io)
• Lin Sun (Solo.io)
• Ram Vennam (Solo.io)

Seating the new committee

On behalf of the Steering Committee, I wish to congratulate our new and returning members. This group will serve for one year, starting this week.

We would also like to extend our heartfelt thanks to Idit Levine, Rob Scott, Pratima Nambiar, and Wilson Wu, whose terms have now ended.

The new team will continue to grow and improve Istio as a successful and sustainable open source project. We encourage everyone to get involved in the Istio community, and help us shape the future of the world’s most popular service mesh.

In fact, we are fully migrating all Istio artifacts out of Google Cloud, including Helm charts. Future communications will cover the migration of Helm charts and other artifacts. This post will focus on what you can do today in response to the 2027 container registry migration.

Am I affected?

By default, Istio installations use Docker Hub (docker.io/istio) as their container registry, but many users choose to use the gcr.io/istio-release mirror. You can check whether you are using the mirror using the following command.

$ kubectl get pods --all-namespaces -o json \
    | jq -r '.items[] | select(.spec.containers[].image | startswith("gcr.io/istio-release")) | "\(.metadata.namespace)/\(.metadata.name)"'

The above command will list all the pods that use images hosted on gcr.io/istio-release. If there are any such pods, you will likely need to migrate.

What to do today

Although we plan to keep images available on gcr.io/istio-release until late 2026, we have set up registry.istio.io as the new home for Istio images. Please migrate to using registry.istio.io as soon as possible.

Using istioctl

If you install Istio using istioctl, you can update your IstioOperator configuration as follows:

# istiooperator.yaml
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  # ...
  hub: registry.istio.io/release
  # Everything else can stay the same unless you reference `gcr.io/istio-release` images elsewhere

and install Istio using this configuration

$ istioctl install -f istiooperator.yaml

Alternatively, you can pass in the registry as a command line argument

$ istioctl install --set hub=registry.istio.io/release # the rest of your arguments

Using Helm

If you use Helm to install Istio, update your values file to have the following:

# ...
hub: registry.istio.io/release
global:
  hub: registry.istio.io/release
# Everything else can stay the same unless you reference `gcr.io/istio-release` images elsewhere

Then, update your Helm installation with your new values file.

Private mirrors

Your organization might pull images from gcr.io/istio-release, push them to a private registry, and reference the private registry in your Istio installation. This process will still work, but you will have to pull from registry.istio.io/release instead of gcr.io/istio-release.

This blog post highlights the risks of using Istio in multi-tenant clusters and explains how users can mitigate these risks and safely operate Istio in their deployments.

Please note that the issues even extend beyond the cluster scope in a “single mesh with multiple clusters” deployment.

The behavior described in this post applies to Istio version 1.29.0 and to all versions since the introduction of the mesh gateway option in the VirtualService resource.

Background

Namespace-based Multi-Tenancy

Namespaces in Kubernetes provide a mechanism for organizing groups of resources within a cluster. Namespaces provide a logical abstraction that allows teams, applications, or environments to share a single cluster while isolating their resources via controls such as Network Policies, RBAC, and so on.

In this blog post, we focus on running Istio in clusters where multiple tenants share the same cluster and service mesh, and can deploy Istio resources (networking.istio.io/v1) in their namespaces while relying on namespace boundaries for isolation.

Traffic Routing in Istio

Istio provides traffic management capabilities by separating application logic from network routing behavior. It introduces additional configuration resources through CRDs that allow operators to define how traffic should be routed between services in the mesh.

One of the central resources for this purpose is the VirtualService. A VirtualService defines a set of routing rules that determine how requests to hosts specified in spec.hosts.[] are handled. These rules can match requests based on properties such as HTTP headers, paths, or ports, and can then direct the traffic to one or more destination services.

Routing decisions defined in a VirtualService are not limited to a single workload or namespace. Depending on how the resource is configured, these rules can affect traffic routing across the entire mesh.

In contrast to the newer Kubernetes Gateway API, these CRDs were created and effectively stabilized before namespace-based RBAC even made its way to Kubernetes. Thus, namespace-based multi-tenancy that shares the same service mesh was not part of the threat model at the time. With the introduction of RBAC, such multi-tenant environments emerged. It is therefore important to highlight and address the security risks associated with those architectures.

In the following section, we demonstrate those risks and show that this mechanism can be abused to intercept traffic in a namespace-based multi-tenant cluster. Later, we introduce ways to mitigate those risks.

Man-in-the-Middle Attacks through VirtualService

In a namespace-based multi-tenant environment, it is often assumed that namespaces provide sufficient trust boundaries between resources across different namespaces. However, Istio’s traffic routing configuration operates at the mesh level, meaning that routing rules defined in one namespace will influence traffic originating from workloads in other namespaces.

An attacker who has permission to create or modify VirtualService resources can abuse this behavior by defining routing rules for arbitrary hosts. When the service mesh parameter mesh is set in the gateways section of the spec, the routing rules are applied to all sidecar proxies in the mesh (independent of their namespace).

This allows an attacker to create a malicious VirtualService that matches requests for specific hostnames and redirects them to an attacker-controlled service. As a result, traffic from other workloads in the mesh can be transparently routed through the attacker’s service before reaching its intended destination.

This behavior enables MITM attacks within the service mesh. The attacker-controlled service can intercept traffic from services in the mesh. This includes traffic to other services in the mesh as well as traffic to the external services. This allows the attacker to:

• act as the destination service.
• redirect traffic to alternative destinations.
• drop requests to disrupt the communication (denial-of-service).

The source service will send the request to the attacker-controlled service instead of the destination service as the VirtualService overrides the default behavior. Istio’s mutual TLS authentication does not help here, because the proxy identifies the attacker-controlled service as the legitimate destination of the overwritten hostname. However, forwarding this traffic to the destination service to read or modify communication between the two services is more challenging for the attacker, as they cannot bypass Istio’s Layer 4 and Layer 7 security features. As the attacker intercepts the communication, the end-to-end encryption and authentication between the source and the destination service are broken. Thus, the request forwarded from the attacker-controlled service to the destination service is authenticated as a request from the attacker-controlled service. As a result, Authorization Policies configured on the destination service may deny the request. In addition, destination service will see the attacker-controlled service identity in the X-Forwarded-Client-Cert header, and the authentication from the source service is lost.

Why does this behavior occur?

This behavior results from how Istio distributes and evaluates traffic routing configuration within the service mesh.

Istio service mesh is logically split into a data plane and a control plane. Istio’s control plane aggregates routing configuration from all VirtualService resources and distributes the resulting configuration to the Envoy sidecar proxies that make up the data plane. These proxies then enforce routing rules locally for the traffic they handle, see also Istio Architecture.

When a VirtualService is configured as a mesh gateway, its routing rules apply to all sidecars in the mesh, including internal service-to-service traffic. Since the effects of this configuration are not limited to the namespace in which the VirtualService resides, a configuration created in one namespace can match requests originating from workloads in other namespaces.

Mitigation and Best Practices

Operators running Istio in namespace-based multi-tenancy setups or operating a single mesh across multiple clusters should apply additional safeguards to maintain strong isolation. Without these controls, unintended cross-namespace traffic manipulation can occur at the data plane level.

Recommended Mitigation: Migrate to the Newer Gateway API

Ideally, permissions to create or modify Istio networking resources (networking.istio.io/v1 as well as security.istio.io/v1) should be limited to platform operators responsible for global routing.

As an alternative, operators can offer tenants access to the newer Gateway API, which was designed with safe cross-namespace support in mind. However, the platform operators still need to control access to shared resources such as gateways.

Configuration Scoping can be implemented as an additional control.

Mitigation in Legacy Setups

When such changes and restrictions aren’t feasible due to business or organizational requirements, routing configurations should be scoped to specific services or namespaces. Broad rules that affect the entire mesh should be avoided unless explicitly intended, and their implications are well understood.

One way to mitigate this kind of attack is to configure Scoping. For instance, to restrict the Egress listener in every namespace to trusted namespaces. However, this would only mitigate the issue in sidecar mode and ambient mode with waypoints, but not in L4-only ambient mode, and also not for hosts configured when an Istio Gateway is used.

Another way to mitigate this kind of attack is to implement an admission policy that limits which hosts can be used in the host section for each tenant. This will also mitigate the issue in ambient mode.

Conclusion

As shown in this post, Istio’s mesh gateway option allows rules defined in one namespace to affect the traffic of other namespaces. In namespace-based multi-tenancy setups or when running a single mesh across multiple clusters, this behavior may expose the service mesh to malicious actors, e.g., enabling MITM attacks, as explained in this blog post.

Istio does not claim (nor seek to claim) hard namespace-based multi-tenancy as the project chose the tradeoff that eases adoption. Thus, operators who rely on this kind of multi-tenancy should assess the risks involved in their architecture and address the weaknesses, e.g., by removing unnecessary RBAC permissions and enforcing strict admission controls.

References

• Istio Documentation — Security Model
• Security Bulletin ISTIO-SECURITY-2026-002
• Istio Documentation — Traffic Management
• Istio Documentation — VirtualService

Istio highlights at KubeCon EU 2026

• Join us at Cloud Native Theater on Tuesday, March 24, 2026 starting at 14:30 CET for 4 amazing Istio Day sessions:
  • Tales From the Mesh: Horrors and Successes of Running Istio in Production — A panel where real users of Istio share real world experiences, challenges, and wins with Istio service mesh.
  • Zero-Downtime Migration from ingress-nginx to Istio in a Multi-Cluster Kubernetes Platform at Bloomberg — Learn how Bloomberg migrated to Istio across multiple clusters without downtime.
  • The Good, The Ugly, and The Bad: Leaving Sidecars Behind with Istio Ambient Mesh — An honest look at migrating to ambient mesh architecture.
  • Running State of the Art Inference with Istio and LLM-D — Discover how Istio enables efficient AI inference serving.
• Join the Maintainer Track: Evolution or Revolution: Istio as the Network Platform for Cloud Native — two maintainers explore how Istio’s vision of a universal dataplane has guided the project’s evolution, from powering global multicluster connectivity to enabling AI inference. Learn how you can contribute as an “Istio power user contributor” and share feedback on future improvements.
• Participate in the Cloud Native Novice Track: From First Contribution to Leadership: Lessons on Becoming a CNCF Leader — a perfect session for newcomers interested in growing within the ecosystem. Learn about overcoming early challenges, maintaining momentum in large communities, and how authenticity, curiosity, and community support can accelerate your journey to leadership.

Recommended Sessions at KubeCon

Here are recommended sessions from the main conference with strong Istio relevance:

• From NLB Sprawl To Mesh Efficiency: How Skyscanner Handles 60M Requests Per Minute With Istio | Solutions Showcase
• Intelligent Routing for Optimized Inference | Solutions Showcase
• Bob and Alice Revisited: Understanding Encryption in Kubernetes
• Istio’s Ambient Mesh: The Real Cost of Sidecar-less Tracing

Meet Us in Person

Stop by the Istio kiosk in the Project Pavilion throughout the event to chat with maintainers, contributors, and users!

Don’t Miss Out

KubeCon + CloudNativeCon is always the perfect time to connect, learn, and celebrate the amazing work happening across the Istio community. Stay tuned for more updates as the event approaches.

Follow us on X, LinkedIn, or Bluesky to get live updates from the event. See you soon in Amsterdam!

Gaps in Telemetry

The benefits of a multicluster distributed system are not without their tradeoffs. Some complexity is inevitable with larger scale, making good telemetry even more important. The Istio team understands that point and we were aware of some gaps that needed to be covered. Thankfully, on release 1.29, telemetry is now more robust and complete when our ambient data plane operates over distributed clusters and networks.

If you’ve deployed alpha multicluster capabilities before in multi-network scenarios, you might have noticed some source or destination labels would show as “unknown”.

For context, in a local cluster (or clusters sharing the same network), waypoint and ztunnel are aware of all existing endpoints, and they acquire that information through xDS. Confusing metrics instead often occur in multi-network deployments where, given all the information that needs to be replicated across separate networks, the xDS peer discovery is unpractical. Unfortunately, that results in missing peer information when requests traverse network boundaries to reach a different Istio cluster.

Telemetry Enhancements

Overcoming that problem, Istio 1.29 now ships with augmented discovery mechanisms in its data plane for exchanging peer metadata between endpoints and gateways sitting across different networks. The HBONE protocol is now enriched with baggage headers, allowing for waypoint and ztunnel to exchange peer information transparently through east-west gateways.

In the diagram above, focusing on L7 metrics, we show how the peer metadata flows through baggage headers across different clusters sitting in different networks.

1. The client in Cluster A initiates a request, and ztunnel starts to establish an HBONE connection through the Waypoint. This means ztunnel sends a CONNECT request with a baggage header containing the peer metadata from downstream. That metadata is then stored in the waypoint.
2. The baggage header containing the metadata is removed, and the request is routed normally. In this case it goes to a different cluster.
3. On the receiving side, the Ztunnel in Cluster B receives the HBONE request and replies with a successful status, appending a baggage header, now containing the upstream peer metadata.
4. The upstream peer metadata is invisible to the east-west gateway. And as the response reaches the waypoint, it will now have all the information it needs to emit metrics about the two parties involved.

Note that this functionality is behind a feature flag at the moment. If you want to try these telemetry enhancements, they need to be explicitly activated with the AMBIENT_ENABLE_BAGGAGE feature option.

Other Improvements and Fixes

Some welcomed improvements were made regarding connectivity. Ingress gateways and waypoint proxies can now route requests directly to remote clusters. This sets the stage for easier resiliency and enables more flexible design patterns providing the benefits that Istio users expect in multicluster and multi-network deployments.

And of course, we’ve also added a couple of smaller fixes making multi-network multicluster more stable and robust. We’ve updated the multicluster documentation to reflect some of these changes, including the addition of a guide on how to set up Kiali for an ambient multi-network deployment.

Limitations and Next Steps

All that said, we still acknowledge some gaps weren’t fully covered. Most of the work here was targeting multi-network support. Note that multicluster in single network deployments is still considered alpha stage.

Also, the east-west gateway may give preference to a specific endpoint during a certain time span. This may have some impact on how load from requests coming from a different network is distributed between endpoints. And this is a behavior that impacts both ambient and sidecar data plane modes, and we have plans to address it for both cases.

We’re working with the fantastic Istio community to get these limitations addressed. For now, we’re excited to get this beta out there, and eager to get your feedback. The future is looking bright for Istio multi-network multicluster.

If you would like to try out ambient multi-network multicluster, please follow this guide. Remember, this feature is in beta status and not ready for production use. We welcome your bug reports, thoughts, comments, and use cases. You can reach us on GitHub or Slack.

Every year, the leaders in the Istio project estimate the proportion of the hundreds of companies that have contributed to Istio in the past year, and uses that metric to proportionally allocate nine Contribution Seats on our Steering Committee.

Then, four Community Seats are voted for by our project members, with candidates being from companies that did not receive Contribution Seats.

We are pleased to share the result of this year’s calculation, and to kick off our Community Seat election.

Contribution seats

The calculation for the 2026-2027 term reflects the deep investment of our vendors in the Istio open source project. We have four companies represented in our Contribution Seats:

The full allocation can be seen in our formula spreadsheet.

Community Seat election

As in previous years, the Community Seat elections immediately follow the allocation of the Contribution Seats. It is therefore now time to collect our nominations for candidates, and ensure our voter list is correct.

Candidates

Eligibility for candidacy is defined in the Steering Committee charter as a project member who does not work for a Company that will hold a Contribution Seat during the upcoming term.

We would now like to invite members from outside our Contribution Seat holders to stand for election. Nominations are due by March 6.

Voters

Eligibility to vote is defined in the charter as either:

• a project member who has had at least one Pull Request merged in the past 12 months, or
• someone who has submitted the voting exception form and has been accepted by the Steering Committee as having standing in the community through contribution of another kind.

The draft list of voters has been published. If you’re not on that list and you believe you have standing in the Istio community, please submit the exception form.

Exception requests are due by March 6. Voting will start on March 9 and last until March 20.

Announcement of the new committee

Upon the completion of the election, the entire 2026-2027 committee - election winners and company-selected Contribution Seat holders - will be announced.

The Steering Committee wishes to thank its members, old and new, and looks forward to continue to grow and improve Istio as a successful and sustainable open source project. We encourage everyone to get involved in the Istio community by contributing, standing for election, voting, and helping us shape the future of cloud native networking.

KubeCon + CloudNativeCon North America 2025 lit up Atlanta from November 10–13, bringing together one of the largest gatherings of open-source practitioners, platform engineers, and maintainers across the cloud native ecosystem. For the Istio community, the week was defined by packed rooms, long hallway conversations, and a genuine sense of shared progress across service mesh, Gateway API, security, and AI-driven platforms.

Before the main conference began, the community kicked things off with Istio Day on November 10, a colocated event filled with deep technical sessions, migration stories, and future-looking discussions that set the tone for the rest of the week.

Istio Day at KubeCon NA

Istio Day brought together practitioners, contributors, and adopters for an afternoon of learning, sharing, and open conversations about where service mesh—and Istio—are headed next.

Istio Day opened with Welcome + Opening Remarks from John Howard from Solo.io and Keith Mattix from Microsoft, setting the tone for an afternoon focused on real-world mesh evolution and the growing energy across the Istio community.

The day quickly moved into applied AI with Is Your Service Mesh AI Ready?, where John Howard explored how traffic management, security, and observability shape production-grade AI workloads.

Momentum continued with Istio Ambient Goes Multicluster as Jackie Maertens and Steven Jin Xuan from Microsoft demonstrated how Ambient Mesh behaves across distributed clusters—highlighting identity, connectivity, and operational simplifications in multi-cluster deployments.

A burst of energy came with the lightning talk Validating Your Istio Setups? The Tests Are Already Written, where Francisco Herrera Lira from Red Hat showed how built-in validation tooling can catch common configuration issues before they reach production.

In Optimizing Istio Autoscaling: From Resource-Centric to Connection-Aware, Punakshi Chaand and Pankaj Sikka shared how Intuit improved reliability by tuning autoscaling behaviors based on connection patterns rather than raw resource metrics.

Next, Running Databases in Istio’s Service Mesh with Tyler Schade and Michael Bolot from GEICO Tech challenged long-held assumptions, offering practical lessons on securing and operating stateful workloads inside a mesh.

Modernizing traffic entry points took the stage as Lin Sun from Solo.io and Ahmad Al-Masry from Harri walked through Is Zero-Downtime Migration Possible? Moving From Ingress & Sidecars to Gateway API, focusing on progressive migration strategies that avoid outages during architectural shifts.

The final session, Credit Karma’s Istio Migration: 50k+ Pods, Minimal Impact, Lessons Learned, saw Sumit Vij and Mark Gergely outline how they executed one of the largest Istio migrations to date with careful automation and rollout discipline.

The day closed with remarks from John Howard and Keith Mattix, celebrating the speakers, contributors, and a community that continues to push the boundaries of what Istio makes possible.

Istio at the Main KubeCon Conference

Outside of Istio Day, the project was highly visible across KubeCon, with maintainers, end users, and contributors sharing technical deep dives, production stories, and cutting-edge research.

This KubeCon was especially meaningful for the Istio community because Istio appeared not only across expo booths and breakout sessions, but also throughout several of the KubeCon keynotes, where companies showcased how Istio plays a critical role in powering their platforms at scale.

The week’s momentum fully met its stride when the Istio community reconvened with the Istio Project Update, where project leads shared latest releases, roadmap advances, and how Istio is meeting emerging demands from AI workloads, multicluster mesh, and operational scale.

In Istio: Set Sailing With Istio Without Sidecars, attendees explored how sidecar-less Ambient Mesh architecture is rapidly moving from experiment to adoption, opening new possibilities for simpler deployments and leaner data-planes.

The session Lessons Applied Building a Next-Generation AI Proxy took the crowd behind the scenes of how mesh technologies adapt to AI-driven traffic patterns—applying the mesh not just to services, but to model-serving, inference, and data flow.

Over at Automated Rightsizing for Istio DaemonSet Workloads (Poster Session), practitioners gathered to compare strategies for optimizing control-plane resources, tuning for high scale, and reducing cost without sacrificing performance.

The narrative of traffic-management evolution featured prominently in Gateway API: Table Stakes and its faster sibling Know Before You Go! Speedrun Intro to Gateway API. These sessions brought forward foundational and introductory paths to modern ingress and mesh control.

Meanwhile, Return of the Mesh: Gateway API’s Epic Quest for Unity scaled that conversation: how traffic, API, mesh, and routing converge into one architecture that simplifies complexity rather than multiplies it.

For long-term reflection, 5 Key Lessons From 8 Years of Building Kgateway delivered hard-earned wisdom from years of system design, refactoring, and iterative improvements.

In GAMMA in Action: How Careem Migrated To Istio Without Downtime, the real-world migration story—a major production rollout that stayed up during transition—provided a roadmap for teams seeking safe mesh adoption at scale.

Safety and rollout risks took center stage in Taming Rollout Risks in Distributed Web Apps: A Location-Aware Gradual Deployment Approach, where strategies for regional rollouts, steering traffic, and minimizing user impact were laid out.

Finally, operations and day-two reality were tackled in End-to-End Security With gRPC in Kubernetes and On-Call the Easy Way With Agents, reminding everyone that mesh isn’t just about architecture, but about how teams run software safely, reliably, and confidently.

Community Spaces: ContribFest, Maintainer Track & the Project Pavilion

At the Project Pavilion, the Istio kiosk was constantly buzzing, drawing users with questions about Ambient Mesh, AI workloads, and deployment best practices.

The Maintainer Track brought contributors together to collaborate on roadmap topics, triage issues, and discuss key areas of investment for the next year.

At ContribFest, new contributors joined maintainers to work through good-first issues, discuss contribution pathways, and get their first PRs lined up.

Istio Maintainers Recognized at the CNCF Community Awards

This year’s CNCF Community Awards were a proud moment for the project. Two Istio maintainers received well-deserved recognition:

• John Howard — Top Committer Award
• Daniel Hawton — “Chop Wood, Carry Water” Award

Beyond these awards, Istio was also represented prominently in conference leadership. Faseela K, one of the KubeCon NA co-chairs and an Istio maintainer, participated in a keynote panel on Cloud Native for Good.

During closing remarks, it was also announced that Lin Sun, another long-time Istio maintainer, will serve as an upcoming KubeCon co-chair, highlighting the project’s strong leadership presence within CNCF.

What We Heard in Atlanta

Across sessions, kiosks, and hallways, a few themes emerged:

• Ambient Mesh is shifting from exploration to real-world adoption.
• AI workloads are driving innovation in mesh traffic patterns and operational practices.
• Multicluster deployments are becoming commonplace, with attention to identity, control, and failover.
• Gateway API is solidifying as a core tool for modern traffic management.
• New contributors are joining in meaningful numbers, supported by ContribFest, hands-on guidance, and community engagement.

Looking Ahead

KubeCon NA 2025 showcased a community that is vibrant, growing, and tackling some of the hardest challenges in modern cloud infrastructure—from AI traffic management to zero-downtime migrations, from scaling planet-wide control planes to building the next generation of sidecar-less mesh.

As we look ahead to 2026, the energy from Atlanta gives us confidence: the future of service mesh is bright, and the Istio community is leading the way, together.

Istio Day and Key Sessions

• Join us at Istio Day on Monday, November 10 2025 — a full-day, community-focused event. Istio Day will feature sessions on AI readiness for service meshes, scaling Istio across multicluster environments with Ambient Mesh, validating and testing Istio setups, optimizing autoscaling, running stateful workloads like databases in Istio, and zero-downtime migrations to modern Gateway APIs. Attendees will also have hands-on workshops and opportunities to meet maintainers and contributors.
• Catch the TOC session: Istio Project Updates: AI Inference, Ambient Multicluster & Default Deny — hear about the latest features, what the community has been working on, and a preview of the 2026 roadmap.
• Participate in Istio ContribFest: From Farm (Fork) To Table (Feature): Growing Your First (Free-range Organic) Istio PR — a perfect session for newcomers and aspiring contributors to jump into the codebase and community.

Recommended Sessions at KubeCon

We gathered recommended sessions from the main conference with strong Istio relevance:

• Istio: Set Sailing With Istio Without Sidecars | Project Lightning Talk
• Lessons Applied Building a Next-generation AI Proxy
• Automated Rightsizing for Istio DaemonSet Workloads | Poster Session
• Gateway API: Table Stakes
• Know Before You Go! Speedrun Intro To Gateway API | Project Lightning Talk
• Return of the Mesh: Gateway API’s Epic Quest for Unity
• 5 Key Lessons From 8 Years of Building Kgateway | Project Lightning Talk
• GAMMA in Action: How Careem Migrated To Istio Without Downtime
• Taming Rollout Risks in Distributed Web Apps: A Location-Aware Gradual Deployment Approach
• End-to-End Security With gRPC in Kubernetes
• On-Call the Easy Way With Agents

Meet Us in Person

Stop by the Istio kiosk in the Project Pavilion throughout the event to chat with maintainers, contributors, and users!

Don’t Miss Out

KubeCon + CloudNativeCon is always the perfect time to connect, learn, and celebrate the amazing work happening across the Istio community. Stay tuned for more updates, demos, and announcements — including exciting conversations around AI inference, multicluster networking, and the evolution of Ambient Mesh.

Follow us on X, LinkedIn, or Bluesky to get live updates from the event. See you soon in Atlanta!

Each year, three of the six seats are elected. To bootstrap the process, we announced the 2025 election would cover the seats held by the three longest-serving members.

One of those three seats became vacant, prompting a by-election. Long-time maintainer Costin Manolache won that election. We thank Costin for his continuing contributions and completing the remainder of the term, after which he chose not to stand again.

Five candidates stood for the three available seats, and the Steering Committee has now concluded the election.

Lin Sun and Louis Ryan were re-elected to their two seats. Both have been involved with Istio since before its public launch, and continue to serve as active leaders across the project.

The third seat was won by Rama Chavali, a long-time contributor and maintainer of Istio through his work at Salesforce.

In Rama’s own words:

On behalf of the Istio community, I congratulate Rama on his election to the TOC and look forward to his continued leadership and impact.

The Power & Complexity of Multicluster

Multicluster architectures increase outage resilience, shrink your blast radius, and scale across data centers. That said, integrating multiple clusters poses connectivity, security, and operational challenges.

In a single Kubernetes cluster, every pod can directly connect to another pod via a unique pod IP or service VIP. These guarantees break down in multicluster architectures; IP address spaces of different clusters might overlap, and even without overlap, the underlying infrastructure would need configuration to route cross-cluster traffic.

Cross-cluster connectivity also presents security challenges. Pod-to-pod traffic will leave cluster boundaries and pods will accept connections from outside their cluster. Without identity verification at the edge of the cluster and strong encryption, an outside attacker could exploit a vulnerable pod or intercept unencrypted traffic.

A multicluster solution must securely connect clusters and do so through simple, declarative APIs that keep pace with dynamic environments where clusters are frequently added and removed.

Key Components

Ambient multicluster extends ambient with new components and minimal APIs to securely connect clusters using ambient’s lightweight, modular architecture. It builds on the namespace sameness model so services keep their existing DNS names across clusters, allowing you to control cross-cluster communication without changing application code.

East-West Gateways

Each cluster has an east-west gateway with a globally routable IP acting as an entry point for cross-cluster communication. A ztunnel connects to the remote cluster’s east-west gateway, identifying the destination service by its namespaced name. The east-west gateway then load balances the connection to a local pod. Using the east-west gateway’s routable IP removes the need for inter-cluster routing configuration, and addressing pods by namespaced name rather than IP eliminates issues with overlapping IP spaces. Together, these design choices enable cross-cluster connectivity without changing cluster networking or restarting workloads, even as clusters are added or removed.

Double HBONE

Ambient multicluster uses nested HBONE connections to efficiently secure traffic traversing cluster boundaries. An outer HBONE connection encrypts traffic to the east-west gateway and allows the source ztunnel and east-west gateway to verify each other’s identity. An inner HBONE connection encrypts traffic end-to-end, which allows the source ztunnel and destination ztunnel to verify each other’s identity. At the same time, the HBONE layers allow ztunnel to effectively reuse cross-cluster connections, minimizing TLS handshakes.

Service Discovery and Scope

Marking a service global enables cross-cluster communication. Istiod configures east-west gateways to accept and route global service traffic to local pods and programs ztunnels to load balance global service traffic to remote clusters.

Mesh administrators define the label-based criteria for global services via the ServiceScope API, and app developers label their services accordingly. The default ServiceScope is

serviceScopeConfigs:
  - servicesSelector:
      matchExpressions:
        - key: istio.io/global
          operator: In
          values: ["true"]
    scope: GLOBAL

meaning that any service with the istio.io/global=true label is global. Although the default value is straightforward, the ServiceScope API can express complex conditions using a mix of ANDs and ORs.

By default, ztunnel load balances traffic uniformly across all endpoints –even remote ones–, but this is configurable through the service’s trafficDistribution field to only cross cluster boundaries when there are no local endpoints. Thus, users have control over whether and when traffic crosses cluster boundaries with no changes to application code.

Limitations and Roadmap

Although the current implementation of ambient multicluster has the foundational features for a multicluster solution, there is still a lot of work to be done. We are looking to improve the following areas

• Service and waypoint configuration must be uniform across all clusters.
• No cross-cluster L7 failover (L7 policy is applied at the destination cluster).
• No support for direct pod addressing or headless services.
• Support only for multi-primary deployment model.
• Support only for one network per cluster deployment model.

We are also looking to improve our reference documentation, guides, testing, and performance.

If you would like to try out ambient multicluster, please follow this guide. Remember, this feature is in alpha status and not ready for production use. We welcome your bug reports, thoughts, comments, and use cases – you can reach us on GitHub or Slack.

That’s why we’re excited to announce Istio’s support for the Gateway API Inference Extension, bringing intelligent, model-aware and LoRA-aware routing to Istio.

Why AI Workloads Need Special Treatment

Traditional web services typically handle quick, stateless requests measured in milliseconds. AI inference workloads operate in a completely different paradigm that challenges conventional load balancing approaches in several fundamental ways.

The Scale and Duration Challenge

Unlike typical API responses that complete in milliseconds, AI inference requests often take significantly longer to process - sometimes several seconds or even minutes. This dramatic difference in processing time means that routing decisions have far more impact than in traditional web services. A single poorly-routed request can tie up expensive GPU resources for extended periods, creating cascading effects across the entire system.

The payload characteristics are equally challenging. AI inference requests frequently involve substantially larger payloads, especially when dealing with Retrieval-Augmented Generation (RAG) systems, multi-turn conversations with extensive context, or multi-modal inputs including images, audio, or video. These large payloads require different buffering, streaming, and timeout strategies than traditional HTTP APIs.

Resource Consumption Patterns

Perhaps most critically, a single inference request can consume an entire GPU’s resources during processing. This is fundamentally different from traditional request serving where multiple requests can be processed concurrently on the same compute resources. When a GPU is fully engaged with one request, additional requests must queue, making the scheduling and routing decision far more impactful than those for standard API workloads.

This resource exclusivity means that simple round-robin or least-connection algorithms can create severe imbalances. Sending requests to a server that’s already processing a complex inference task doesn’t just add latency, it can cause resource contention that impacts performance for all queued requests.

Stateful Considerations and Memory Management

AI models often maintain in-memory caches that significantly impact performance. KV caches store intermediate attention calculations for previously processed tokens, serving as the primary consumer of GPU memory during generation and often becoming the most common bottleneck. When KV cache utilization approaches limits, performance degrades dramatically, making cache-aware routing essential.

Additionally, many modern AI deployments use fine-tuned adapters like LoRA (Low-Rank Adaptation) to customize model behavior for specific users, organizations, or use cases. These adapters consume GPU memory and loading time when switched. A model server that already has the required LoRA adapter loaded can process requests immediately, while servers without the adapter face expensive loading overhead that can take seconds to complete.

Queue Dynamics and Criticality

AI inference workloads also introduce the concept of request criticality that’s less common in traditional services. Real-time interactive applications (like chatbots or live content generation) require low latency and should be prioritized, while batch processing jobs or experimental workloads can tolerate higher latency or even be dropped during system overload.

Traditional load balancers lack the context to make these criticality-based decisions. They can’t distinguish between a time-sensitive customer support query and a background batch job, leading to suboptimal resource allocation during peak demand periods.

This is where inference-aware routing becomes critical. Instead of treating all backends as equivalent black boxes, we need routing decisions that understand the current state and capabilities of each model server, including their queue depth, memory utilization, loaded adapters, and ability to handle requests of different criticality levels.

Gateway API Inference Extension: A Kubernetes-Native Solution

The Kubernetes Gateway API Inference Extension has introduced solutions to these challenges, building on the proven foundation of Kubernetes Gateway API while adding AI-specific intelligence. Rather than requiring organizations to patch together custom solutions or abandon their existing Kubernetes infrastructure, the extension provides a standardized, vendor-neutral approach to intelligent AI traffic management.

The extension introduces two key Custom Resource Definitions that work together to address the routing challenges we’ve outlined. The InferenceModel resource provides an abstraction for AI-Inference workload owners to define logical model endpoints, while the InferencePool resource gives platform operators the tools to manage backend infrastructure with AI workload awareness.

By extending the familiar Gateway API model rather than creating an entirely new paradigm, the inference extension enables organizations to leverage their existing Kubernetes expertise while gaining the specialized capabilities that AI workloads demand. This approach ensures that teams can adopt intelligent inference routing aligned with familiar networking knowledge and tooling.

Note: InferenceModel is likely to change in future Gateway API Inference Extension releases.

InferenceModel

The InferenceModel resource allows inference workload owners to define logical model endpoints that abstract the complexities of backend deployment.

apiVersion: inference.networking.x-k8s.io/v1alpha2
kind: InferenceModel
metadata:
  name: customer-support-bot
  namespace: ai-workloads
spec:
  modelName: customer-support
  criticality: Critical
  poolRef:
    name: llama-pool
  targetModels:
    - name: llama-3-8b-customer-v1
      weight: 80
    - name: llama-3-8b-customer-v2
      weight: 20

This configuration exposes a customer-support model that intelligently routes between two backend variants, enabling safe rollouts of new model versions while maintaining service availability.

InferencePool

The InferencePool acts as a specialized backend service that understands AI workload characteristics:

apiVersion: inference.networking.x-k8s.io/v1alpha2
kind: InferencePool
metadata:
  name: llama-pool
  namespace: ai-workloads
spec:
  targetPortNumber: 8000
  selector:
    app: llama-server
    version: v1
  extensionRef:
    name: llama-endpoint-picker

When integrated with Istio, this pool automatically discovers model servers through Istio’s service discovery.

How Inference Routing Works in Istio

Istio’s implementation builds on the service mesh’s proven traffic management foundation. When a request enters the mesh through a Kubernetes Gateway, it follows the standard Gateway API HTTPRoute matching rules. However, instead of using traditional load balancing algorithms, the backend is picked by an Endpoint Picker (EPP) service.

The EPP evaluates multiple factors to select the optimal backend:

• Request Criticality Assessment: Critical requests receive priority routing to available servers, while lower criticality requests (Standard or Sheddable) may be load-shed during high utilization periods.

• Resource Utilization Analysis: The extension monitors GPU memory usage, particularly KV cache utilization, to avoid overwhelming servers that are approaching capacity limits.

• Adapter Affinity: For models using LoRA adapters, requests are preferentially routed to servers that already have the required adapter loaded, eliminating expensive loading overhead.

• Prefix-Cache Aware Load Balancing: Routing decisions consider distributed KV cache states across model servers, and prioritize model servers that already have the prefix in their cache.

• Queue Depth Optimization: By tracking request queue lengths across backends, the system avoids creating hotspots that would increase overall latency.

This intelligent routing operates transparently within Istio’s existing architecture, maintaining compatibility with features like mutual TLS, access policies, and distributed tracing.

Inference Routing Request Flow

The Road Ahead

The future roadmap includes istio-related features such as:

• Support for Waypoints - As Istio continues to evolve toward ambient mesh architecture, inference-aware routing will be integrated into waypoint proxies to provide centralized, scalable policy enforcement for AI workloads.

Beyond Istio-specific innovations, the Gateway API Inference Extension community is also actively developing several advanced capabilities that will further enhance routing for AI inference workloads on Kubernetes:

• HPA Integration for AI Metrics: Horizontal Pod Autoscaling based on model-specific metrics rather than just CPU and memory.

• Multi-Modal Input Support: Optimized routing for large multi-modal inputs and outputs (images, audio, video) with intelligent buffering and streaming capabilities.

• Heterogeneous Accelerator Support: Intelligent routing across different accelerator types (GPUs, TPUs, specialized AI chips) with latency and cost-aware load balancing.

Getting Started with Istio Inference Extension

Ready to try inference-aware routing? The implementation is officially available starting with Istio 1.27!

For installation and guides, please follow the Istio-specific guidance on the Gateway API Inference Extension website.

Performance Impact and Benefits

Early evaluations show significant performance improvements with inference-aware routing, including substantially lower p90 latency at higher query rates and reduced end-to-end tail latencies compared to traditional load balancing.

For detailed benchmark results and methodology, see the Gateway API Inference Extension performance evaluation with testing data using H100 GPUs and vLLM deployments.

The integration with Istio’s existing infrastructure means these benefits come with minimal operational overhead, and your existing monitoring, security, and traffic management configurations continue to work unchanged.

Conclusion

The Gateway API Inference Extension represents a significant step forward in making Kubernetes truly AI-ready, and Istio’s implementation brings this intelligence to the service mesh layer where it can have maximum impact. By combining inference-aware routing with Istio’s proven security, observability, and traffic management capabilities, we’re enabling organizations to run AI workloads with the same operational excellence they expect from their traditional services.

Have a question or want to get involved? Join the Kubernetes Slack and then find us on the #gateway-api-inference-extension channel or discuss on the Istio Slack.

Looking Back

Since 2023, the Istio project has been focused on maturity and innovation, solidifying our position as the best service mesh regardless of sidecars or ambient. These efforts included our CNCF graduation in July 2023, the promotion of Telemetry API and Gateway API to Stable in Istio 1.22, and the promotion of ambient mode to Stable in Istio 1.24. As part of Istio ambient mode reaching GA, we have observed more and more users exploring and adopting it, some of the users are net new Istio users, while others are users of Istio sidecars. Some of them ran ambient in production and spoke about their experiences at KubeCon EU in April this year. These efforts have made Istio the service mesh of choice for cloud native developers around the world, and we have been excited to accept first code contributions from 154 people in the past 12 months.

2025 Themes

Sidecar to ambient migration

With the promotion of ambient mode to Stable, Istio can now lay claim to being the fastest and most efficient service mesh as well as the most widely used, while being easier to operate than ever. With graduation, we’ve seen a substantial increase in interest, and a corresponding number of requests for a comprehensive migration guide for existing sidecar users. While our previous efforts to stabilize ambient mode have been targeted at new Istio users, it is clear that the time has come to provide an onramp for our existing user base to migrate to ambient mesh. While the technical foundations for this migration have been in place for some time (and some brave users have migrated on their own), we will be making new investments in tooling to assess your readiness to migrate, rollback-safe interoperability, and documentation to guide users every step of the way.

In addition to tests, tooling, and documentation, users migrating between data planes should reasonably expect that the Istio features they know and love will continue to work in their new environment. For this reason, we are investing in closing the most significant functionality gaps between sidecar and ambient mode, specifically by adding support for multi-cluster traffic management and extensibility, which you can read about below.

As we have stated in previous years, we have no intention of ending support for sidecar mode as long as there are users for it. Migrating to ambient mesh is completely voluntary, and we expect many users will use sidecars for years to come.

Multi-cluster ambient mesh

Multi-cluster traffic management has long been one of the most valued enterprise features of Istio, and we are hard at work to bring this value to ambient mode users in 2025. With a multi-cluster mesh, service outages or anomalies in one cluster can dynamically cause requests to fail over to other clusters, potentially in other regions or clouds. This gives users the ability to run high-availability services in active-active configuration, optimizing compute utilization and traffic costs from a single control plane. Multi-cluster ambient mesh will be available as an Alpha in Istio 1.27, which we plan to release in August.

The future of extensibility

The Istio project has offered several APIs for extensibility since launch, and none of them has been able to mature to Stable. Of those in use today, Envoy Filters are a powerful tool for tweaking internal proxy configuration, and modifying traffic flow, but are very difficult to use, and pose significant risk during upgrades, which can change the filter integrations in ways that cannot always be predicted. WebAssembly (Wasm) emerged in 2019 as a powerful tool for Turing-complete modification of traffic, but community support for Wasm compilers and libraries outside the Istio ecosystem has waned substantially since that time, making it difficult for users to safely and securely use Wasm with Istio.

As we plan for 2025 and beyond, it is clear that we need a path to a mature extensibility model for users of sidecars and ambient mode alike. We plan to address the most common use cases for extensibility, such as local rate limiting, with first class APIs, reducing the frequency with which users require extensibility. However, we recognize that networks are complex, and there will always be cases our APIs don’t cover, when users need a “break glass” option. The architecture of ambient mode provides some options, such as leveraging the waypoint pattern to accomplish service insertion, adding arbitrary proxies to the network chain, which can then perform arbitrary modifications. Another similar development is Envoy’s ext-proc filter, which sends requests to an arbitrary service for modification before forwarding them to their destination.

With several options on the table for extensibility, who will decide which is best? As always, the final decision lies with you, our users. Please share your thoughts with us about the future of the project in the extensibility channel at slack.istio.io.

New and Improved Contributor Experience

The Istio community is full of many talented contributors whose daily efforts make this project possible, and the list of contributors is always growing! However, like all Open Source projects, we are always in need of new contributors, and we recognize that submitting your first PR to Istio is harder than it should be. In 2025, we aim to make authoring your first Istio contribution easier than ever with improved integration with GitHub Codespaces, and regular triage of good first issues! If you’re interested in contributing, we can always use help on Issues labeled User Experience and Documentation. If you’d like to get more involved, consider joining our release manager rotation, which will provide you with two releases as a shadow before taking on primary release management responsibilities. We will also aim to provide better recognition to our contributors through a revamped workgroup leads program, where top contributors can be recognized for their expertise! With these initiatives, we believe we are setting up the Istio community to grow for years to come.

Conclusion

This roadmap outlines an exciting near-term for Istio, focusing on a seamless migration path from sidecar to ambient mode, enhanced multi-cluster capabilities, and a refined approach to extensibility. We are also committed to fostering a more welcoming and rewarding environment for our invaluable contributors. These initiatives solidify Istio’s position as the leading service mesh, ready to empower cloud native developers with unmatched efficiency, control, and a thriving community.

We kick-started the activities in London with Istio Day - a KubeCon + CloudNativeCon co-located event on April 1st. The event was well-received, showcasing lessons learned from running Istio in production, hands-on experiences, and featuring maintainers from across the Istio ecosystem.

Istio Day kicked off with an opening keynote from the Program Committee chairs, Keith Mattix and Denis Jannot. The keynote was followed by the much-awaited talk from Microsoft about Istio Ambient Mesh support on Windows. We had a very interesting talk by Lior Lieberman from Google and Erik Parienty from Riskified on architecting Istio for large scale deployments, followed by a talk from Kiali maintainers Josune Cordoba and Hayk Hovsepyan, from RedHat, about troubleshooting Istio ambient mesh with Kiali 2.0.

Istio multi-cluster is always a hot topic, and Pamela Hernandez from BlackRock nailed it in the talk on navigating the maze of multi-cluster Istio, diving into the complexities of implementing a multi-cluster Istio service mesh at scale, covering a hub-and-spoke model. The audience was excited when Denis Jannot from Solo.io ran a live, representative benchmark at scale with Istio Ambient, debunking all myths about service mesh overhead and complexity. The event witnessed how Istio played a pivotal role in managing traffic and ensuring data security, ultimately enabling a secure and efficient AI platform that meets enterprise standards when SAP presented GenAI platform challenges in multi-tenant environments. Rounding out the talks was a lightning talk by Rob Salmond from SuperOrbit on How to get Istio help, which involved the best places to go, how to ask good questions, and avoid common missteps.

The slides for all the sessions can be found in the Istio Day EU 2025 schedule.

Our presence at the conference did not end with Istio Day. The first day keynote of KubeCon + CloudNativeCon started with an Istio project lightning talk from Mitch Connors.

There were several keynotes on the main stage where Istio was mentioned. At the opening day keynotes, Vasu Chandrasekhara, from SAP, announced the NeoNephos Foundation under the Linux Foundation Europe - a major step forward for Digital Sovereignty in Europe, and Istio was mentioned as a supported project.

Stephen Connolly shared HSBC’s journey with Kubernetes and also discussed plans to adopt Istio ambient mesh to save on costs. Ant Group, who won the CNCF End User Award, also highlighted their Istio usage. Idit Levine and Keith Babo, from Solo.io, announced a free cost-saving estimator and migration tool for Istio ambient mesh. Faseela K had a Telco end user panel keynote on Cloud Native Evolution in Telecom with Vodafone, Orange, and Swisscom, which again highlighted Istio usage for Telco Network Functions.

Istio’s maintainer track session was also well received, where Raymond Wong, from Forbes, joined maintainers Louis Ryan and Lin Sun to discuss about Forbe’s journey to Istio ambient in production. It was a packed room with a lot of questions afterwards.

A Contribfest session led by Mitch Connors (Microsoft), Daniel Hawton (Solo.io), and Jackie Maertens (Microsoft) walked through the structure of the Istio repositories, where each component’s code lives, finding issues to resolve, setting up and using integration tests, and making first contributions to the project as well as resources for getting development environments up and running and places to go to get assistance.

Istio maintainers Lin Sun and Faseela K had a book signing event post their Istio Phippy book reading session on “Izzy saves the Birthday”.

The following sessions at KubeCon were based on Istio and almost all of them had a huge crowd in attendance:

• Project Lightning Talk: What’s New in Istio?
• Sponsored Demo: Bringing Agentic AI to Cloud Native - Introducing kagent
• “Izzy Saves the Birthday” - A Story-Driven Live Demo Exploring the Magic of Service Mesh
• Trino and Data Governance on Kubernetes
• Journey at the New York Times: Is Sidecar-Less Service Mesh Disappearing Into Infrastructure?
• Lightning Talk: High Availability With ‘503: Unavailable’

Istio had a kiosk in the project pavilion, with the majority of questions asked being around extensibility and multi cluster enhancements. Many of our members and maintainers offered support at our kiosk, helping us answer all the questions from our users.

Many of our TOC members and maintainers also offered support at the booth, where a lot of interesting discussions happened around Istio ambient mesh as well.

We would like to express our heartfelt gratitude to our gold sponsor Microsoft Azure, for supporting Istio Day Europe! Last but not least, we would like to thank our Istio Day Program Committee members, for all their hard work and support!

See you in Atlanta in November 2025!

It is our intention that the ztunnel project be safe to install by default in every Kubernetes cluster, and to that end, it needs to be secure and performant.

We comprehensively demonstrated ztunnel’s performance, showing that it is the highest-bandwidth way to achieve a secure zero-trust network in Kubernetes — providing higher TCP throughput than even in-kernel data planes like IPsec and WireGuard — and that its performance has increased by 75% over the past 4 releases.

Today, we are excited to validate the security of ztunnel, publishing the results of an audit of the codebase performed by Trail of Bits.

We would like to thank the Cloud Native Computing Foundation for funding this work, and OSTIF for its coordination.

Scope and overall findings

Istio has been assessed in 2020 and 2023, with the Envoy proxy receiving independent assessment. The scope of this review was the new code in Istio’s ambient mode, the ztunnel component: specifically code relating to L4 authorization, inbound request proxying, transport-layer security (TLS), and certificate management.

The auditors stated that “the ztunnel codebase is well-written and structured”, and had no findings relating to vulnerabilities in the code. Their three findings — one of medium severity and two of informational — refer to recommendations regarding external factors, including software supply chain and testing.

Resolution and suggested improvements

Improving dependency management

At the time of the audit, the cargo audit report for ztunnel’s dependencies showed three versions with current security advisories. There was no suggestion that any vulnerable code paths in ztunnel dependencies could be reached, and the maintainers would regularly update the dependencies to the latest appropriate versions. To streamline this, we’ve adopted GitHub’s Dependabot for automated updates.

The auditors pointed out the risk of Rust crates in the dependency chain of ztunnel that either unmaintained or maintained by a single owner. This is a common situation in the Rust ecosystem (and indeed all of open source). We replaced the two crates that were explicitly identified.

Enhancing test coverage

The Trail of Bits team found that most ztunnel functionality is well-tested, but identified some error-handling code paths which were not covered by mutation testing.

We evaluated the suggestions and found that the gaps in coverage highlighted by these results apply to test code, and to code that does not affect correctness.

While mutation testing is useful to identify potential areas to improve, the goal is not to get to a point where a report returns no results. Mutations can trigger no test failures in a number of expected cases, such as behavior with no ‘correct’ result (e.g., log messages), behavior that impacts only performance but not correctness (measured outside of the scope the tooling is aware of), code paths that have multiple ways to achieve the same result, or code used only for testing. Testing and security is a core priority for the Istio team and we are constantly improving our test coverage — using tools like mutation testing and by developing novel solutions to test proxies.

Hardening HTTP header parsing

A third-party library was used for parsing the value of the HTTP Forwarded header, which may be present on connections made to ztunnel. The auditors pointed out that header parsing is a common area of attack, and expressed concern that the library we used was not fuzz tested. Given that we were only using this library for parsing one header, we wrote a custom parser for the Forwarded header, complete with a fuzzing harness to test it.

Get involved

With strong performance and now validated security, ambient mode continues to advance the state of the art in service mesh design. We encourage you to try it out today.

If you would like to get involved with Istio product security, or become a maintainer, we’d love to have you! Join our Slack workspace or our public meetings to raise issues or learn about what we are doing to keep Istio secure.

Simplified deployment & management

The Sail Operator is engineered to cut down the complexity of installing and running Istio. It automates manual tasks, ensuring a consistent, reliable, and uncomplicated experience from initial installation to ongoing maintenance and upgrades of Istio versions in your cluster. The Sail Operator APIs are built around Istio’s Helm chart APIs, which means that all the Istio configurations are available through the Sail Operator CRD’s values.

We encourage users to go through our documentation to learn more about this new way to manage your Istio environment.

The main resources that are part of the Sail Operator are:

• Istio: manages an Istio control plane.
• IstioRevision: represents a revision of the control plane.
• IstioRevisionTag: represents a stable revision tag, which functions as an alias for an Istio control plane revision.
• IstioCNI: manages Istio’s CNI node agent.
• ZTunnel: manage the ambient mode ztunnel DaemonSet (Alpha feature).

Main features and support

• Each component of the Istio control plane is managed independently by the Sail Operator through dedicated Kubernetes Custom Resources (CRs). The Sail Operator provides separate CRDs for components such as Istio, IstioCNI, and ZTunnel, allowing you to configure, manage, and upgrade them individually. Additionally, there are CRDs for IstioRevision and IstioRevisionTag to manage Istio control plane revisions.
• Support for multiple Istio versions. Currently the 1.0.0 version supports: 1.24.3, 1.24.2, 1.24.1, 1.23.5, 1.23.4, 1.23.3, 1.23.0.
• Two update strategies are supported: InPlace and RevisionBased. Check our documentation for more information about the update types supported.
• Support for multicluster Istio deployment models: multi-primary, primary-remote, external control plane. More information and examples in our documentation.
• Ambient mode support is Alpha: check our specific documentation.
• Addons are managed separately from the Sail Operator. They can be easily integrated with the Sail Operator, check this section for the documentation for examples and more information.

Why now?

As cloud native architectures continue to evolve, we feel a robust and user-friendly operator for Istio is more essential than ever. The Sail Operator offers developers and operations teams a consistent, secure, and efficient solution that feels familiar to those used to working with operators. Its GA release signals a mature solution, ready to support even the most demanding production environments.

Try it out

Would you like to try out Sail Operator? This example will show you how to safely do an update of your Istio control plane by using the revision-based upgrade strategy. This means you will have two Istio control planes running at the same time, allowing you to migrate workloads easily, minimizing the risk of traffic disruptions.

Prerequisites:

• Running cluster
• Helm
• Kubectl
• Istioctl

Install the Sail Operator using Helm

$ helm repo add sail-operator https://istio-ecosystem.github.io/sail-operator
$ helm repo update
$ kubectl create namespace sail-operator
$ helm install sail-operator sail-operator/sail-operator --version 1.0.0 -n sail-operator

The operator is now installed in your cluster:

NAME: sail-operator
LAST DEPLOYED: Tue Mar 18 12:00:46 2025
NAMESPACE: sail-operator
STATUS: deployed
REVISION: 1
TEST SUITE: None

Check the operator pod is running:

$ kubectl get pods -n sail-operator
NAME                             READY   STATUS    RESTARTS   AGE
sail-operator-56bf994f49-j67ft   1/1     Running   0          87s

Create Istio and IstioRevisionTag resources

Create an Istio resource with the version v1.24.2 and an IstioRevisionTag:

$ kubectl create ns istio-system
$ cat <<EOF | kubectl apply -f-
apiVersion: sailoperator.io/v1
kind: Istio
metadata:
  name: default
spec:
  namespace: istio-system
  updateStrategy:
    type: RevisionBased
    inactiveRevisionDeletionGracePeriodSeconds: 30
  version: v1.24.2
---
apiVersion: sailoperator.io/v1
kind: IstioRevisionTag
metadata:
  name: default
spec:
  targetRef:
    kind: Istio
    name: default
EOF

Note that the IstioRevisionTag has a target reference to the Istio resource with the name default

Check the state of the resources created:

• istiod pods are running

  $ kubectl get pods -n istio-system
  NAME                                    READY   STATUS    RESTARTS   AGE
  istiod-default-v1-24-2-bd8458c4-jl8zm   1/1     Running   0          3m45s

• Istio resource created

  $ kubectl get istio
  NAME      REVISIONS   READY   IN USE   ACTIVE REVISION   STATUS    VERSION   AGE
  default   1           1       1        default-v1-24-2   Healthy   v1.24.2   4m27s

• IstioRevisionTag resource created

  $ kubectl get istiorevisiontag
  NAME      STATUS                    IN USE   REVISION          AGE
  default   NotReferencedByAnything   False    default-v1-24-2   4m43s

Note that the IstioRevisionTag status is NotReferencedByAnything. This is because there are currently no resources using the revision default-v1-24-2.

Deploy sample application

Create a namespace and label it to enable Istio injection:

$ kubectl create namespace sample
$ kubectl label namespace sample istio-injection=enabled

After labeling the namespace you will see that the IstioRevisionTag resource status will change to ‘In Use: True’, because there is now a resource using the revision default-v1-24-2:

$ kubectl get istiorevisiontag
NAME      STATUS    IN USE   REVISION          AGE
default   Healthy   True     default-v1-24-2   6m24s

Deploy the sample application:

$ kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.29/samples/sleep/sleep.yaml -n sample

Confirm the proxy version of the sample app matches the control plane version:

$ istioctl proxy-status
NAME                              CLUSTER        CDS              LDS              EDS              RDS              ECDS        ISTIOD                                    VERSION
sleep-5fcd8fd6c8-q4c9x.sample     Kubernetes     SYNCED (78s)     SYNCED (78s)     SYNCED (78s)     SYNCED (78s)     IGNORED     istiod-default-v1-24-2-bd8458c4-jl8zm     1.24.2

Upgrade the Istio control plane to version 1.24.3

Update the Istio resource with the new version:

$ kubectl patch istio default -n istio-system --type='merge' -p '{"spec":{"version":"v1.24.3"}}'

Check the Istio resource. You will see that there are two revisions and they are both ‘ready’:

$ kubectl get istio
NAME      REVISIONS   READY   IN USE   ACTIVE REVISION   STATUS    VERSION   AGE
default   2           2       2        default-v1-24-3   Healthy   v1.24.3   10m

The IstioRevisiontag now references the new revision:

$ kubectl get istiorevisiontag
NAME      STATUS    IN USE   REVISION          AGE
default   Healthy   True     default-v1-24-3   11m

There are two IstioRevisions, one for each Istio version:

$ kubectl get istiorevision
NAME              TYPE   READY   STATUS    IN USE   VERSION   AGE
default-v1-24-2          True    Healthy   True     v1.24.2   11m
default-v1-24-3          True    Healthy   True     v1.24.3   92s

The Sail Operator automatically detects whether a given Istio control plane is being used and writes this information in the “In Use” status condition that you see above. Right now, all IstioRevisions and our IstioRevisionTag are considered “In Use”:

• The old revision default-v1-24-2 is considered in use because it is referenced by the sample application’s sidecar.
• The new revision default-v1-24-3 is considered in use because it is referenced by the tag.
• The tag is considered in use because it is referenced by the sample namespace.

Confirm there are two control plane pods running, one for each revision:

$ kubectl get pods -n istio-system
NAME                                      READY   STATUS    RESTARTS   AGE
istiod-default-v1-24-2-bd8458c4-jl8zm     1/1     Running   0          16m
istiod-default-v1-24-3-68df97dfbb-v7ndm   1/1     Running   0          6m32s

Confirm the proxy sidecar version remains the same:

$ istioctl proxy-status
NAME                              CLUSTER        CDS                LDS                EDS                RDS                ECDS        ISTIOD                                    VERSION
sleep-5fcd8fd6c8-q4c9x.sample     Kubernetes     SYNCED (6m40s)     SYNCED (6m40s)     SYNCED (6m40s)     SYNCED (6m40s)     IGNORED     istiod-default-v1-24-2-bd8458c4-jl8zm     1.24.2

Restart the sample pod:

$ kubectl rollout restart deployment -n sample

Confirm the proxy sidecar version is updated:

$ istioctl proxy-status
NAME                              CLUSTER        CDS              LDS              EDS              RDS              ECDS        ISTIOD                                      VERSION
sleep-6f87fcf556-k9nh9.sample     Kubernetes     SYNCED (29s)     SYNCED (29s)     SYNCED (29s)     SYNCED (29s)     IGNORED     istiod-default-v1-24-3-68df97dfbb-v7ndm     1.24.3

When an IstioRevision is no longer in use and is not the active revision of an Istio resource (for example, when it is not the version that is set in the spec.version field), the Sail Operator will delete it after a grace period, which defaults to 30 seconds. Confirm the deletion of the old control plane and IstioRevision:

• The old control plane pod is deleted

  $ kubectl get pods -n istio-system
  NAME                                      READY   STATUS    RESTARTS   AGE
  istiod-default-v1-24-3-68df97dfbb-v7ndm   1/1     Running   0          10m

• The old IstioRevision is deleted

  $ kubectl get istiorevision
  NAME              TYPE   READY   STATUS    IN USE   VERSION   AGE
  default-v1-24-3          True    Healthy   True     v1.24.3   13m

• The Istio resource now only has one revision

  $ kubectl get istio
  NAME      REVISIONS   READY   IN USE   ACTIVE REVISION   STATUS    VERSION   AGE
  default   1           1       1        default-v1-24-3   Healthy   v1.24.3   24m

Congratulations! You have successfully updated your Istio control plane using the revision-based upgrade strategy.

Conclusion

The Sail Operator automates manual tasks, ensuring a consistent, reliable, and uncomplicated experience from initial installation to ongoing maintenance and upgrades of Istio in your cluster. The Sail Operator is an istio-ecosystem project, and we encourage you to try it out and provide feedback to help us improve it, you can check our contribution guide for more information about how to contribute to the project.

An amazing lineup of Istio activities awaits you in London at KubeCon + CloudNativeCon Europe 2025!

• Join for the Istio Project Meeting hosted at the Maintainer Summit.

  

• Come to the Istio Day co-located event.

  

• Attend the Istio Maintainers’ Track session: Istio: The Past, Present and Future of the Project and Community

• Drop by the Istio Contribfest session: A Beginner’s Guide to Contributing to Istio - Hands-on Development and Contribution Workshop

• Add the following KubeCon sessions to your schedule, all of which have an Istio flavor:

  • Project Lightning Talk: What’s New in Istio?
  • Sponsored Demo: Bringing Agentic AI to Cloud Native - Introducing kagent
  • “Izzy Saves the Birthday” - A Story-Driven Live Demo Exploring the Magic of Service Mesh
  • Trino and Data Governance on Kubernetes
  • Journey at the New York Times: Is Sidecar-Less Service Mesh Disappearing Into Infrastructure?
  • Lightning Talk: High Availability With ‘503: Unavailable’

• Have a chat with maintainers and users at the Istio kiosk in the Project Pavilion throughout the event.

• We also have the Istio Phippy book signing event organized alongside the Izzy Saves the Birthday session. Do join the talk and grab a free, signed copy of the book from the authors!

Follow us on X, LinkedIn or Bluesky to get live updates from the event. See you soon!

However, the challenge with security is that it doesn’t come without a cost: it often involves a trade-off between complexity, user experience, and performance.

While most Cloud Native users will know of Istio as a service mesh, providing advanced HTTP functionality, it can also serve the role of providing a foundational network security layer. When we set out to build Istio’s ambient mode, these two layers were explicitly split. One of our primary objectives was to be able to offer security (and a long list of other features!) without compromise.

With ambient mode, Istio is now the highest-bandwidth way to achieve a secure zero-trust network in Kubernetes.

Lets look at some results before we dive into the how and why.

Putting it to the test

To test performance, we utilized a standard network benchmarking tool, iperf, to measure the bandwidth of TCP traffic flowing through various popular Kubernetes network security solutions.

The results speak for themselves: Istio decisively leads the pack as the highest-performing network security solution. Even more impressive is that this gap continues to grow with each Istio release:

Istio’s performance is driven by ztunnel, a purpose built data plane that is light, fast, and secure. Over the last 4 releases, the performance of Ztunnel has improved by 75%!

Outpacing the Kernel

A very common perception in networking performance is that doing everything in the kernel, either natively or by using eBPF extensions, is the optimal way to achieve high performance. However, these results show the opposite effect: the user-space implementations - Linkerd and Istio - substantially outperform the kernel implementations. What gives?

One major factor is the speed of innovation. Performance is not static, and there is a constant progression of micro-optimizations, innovations, and adaptations to hardware improvements. The kernel serves a large number of use cases, and must evolve deliberately. Even when improvements are made, they can take many years to filter through to real world environments.

In contrast, user-space implementations are able to rapidly change and adapt to their specific targeted use cases, and run on any kernel version. Ztunnel is a great example of this effect in action, with substantial performance improvements coming in each quarterly release. A few of the most impactful changes:

• Migrating to rustls, a high performance TLS library focusing on safety (#820).
• Reducing data copying on outbound traffic (#1012).
• Dynamically tuning buffer sizes of active connections (#1024).
• Optimizing memory copies (#1169).
• Moving the cryptography library to AWS-LC, a high-performance cryptography library optimized for modern hardware (#1466).

Some other factors include:

• WireGuard and Linkerd use the ChaCha20-Poly1305 encryption algorithm, while Istio uses AES-GCM. The latter is highly optimized on modern hardware.
• WireGuard and IPsec operate on individual packets (typically at most 1500 bytes, bound by the network MTU) while TLS operates on records of up to 16KB.

Try ambient mode today

If you’re looking to enhance your cluster’s security without compromising on complexity or performance, now is the perfect time to try Istio’s ambient mode!

Follow the getting started guide to learn how easy it is to install and enable.

You can engage with the developers in the #ambient channel on the Istio Slack, or use the discussion forum on GitHub for any questions you may have.

Every year, we estimate the proportion of the hundreds of companies that have contributed to Istio in the past year, and uses that metric to proportionally allocate the nine Contribution Seats on our Steering Committee.

After that, four Community Seats are voted for by our project members, with candidates being from companies that did not receive Contribution Seats.

In February, we announced the Contribution Seat allocation, and invited candidates to stand for the Community Seat elections. As the election officer, I am pleased to announce the results of that election, as well as the individuals who will represent the top contributors.

Community Seats

Four excellent candidates stood for our four open seats, and thus all are elected unopposed:

• Faseela K, Ericsson Software Technology
• Wilson Wu, DaoCloud
• Rob Cernich, Red Hat
• Pratima Nambiar, Salesforce

Wilson is a top 20 contributor to Istio, and a leader in the localization of Istio’s documentation into Chinese. The three other candidates have all previously served on the Steering Committee.

Contribution Seats

Our supporting companies have made their choices for the nine Contribution Seats. They will be held by:

• Craig Box (Solo.io)
• Zack Butcher (Tetrate)
• John Howard (Solo.io)
• Idit Levine (Solo.io)
• Keith Mattix (Microsoft)
• Justin Pettit (Google)
• Louis Ryan (Solo.io)
• Lin Sun (Solo.io)
• Zhonghu Xu (Huawei)

Seating the new committee

On behalf of the Steering Committee, I wish to congratulate our new and returning members. This group will serve for one year, starting this week.

We would also like to extend our heartfelt thanks to Iris Ding, Arunkumar Jayaraman, Abhi Joglekar, Kebe Liu and Jamie Longmuir, whose terms have now ended.

The new team will continue to grow and improve Istio as a successful and sustainable open source project. We encourage everyone to get involved in the Istio community, and help us shape the future of the world’s most popular service mesh.

Every year, the leaders in the Istio project estimate the proportion of the hundreds of companies that have contributed to Istio in the past year, and uses that metric to proportionally allocate nine Contribution Seats on our Steering Committee.

Then, four Community Seats are voted for by our project members, with candidates being from companies that did not receive Contribution Seats.

We are pleased to share the result of this year’s calculation, and to kick off our Community Seat election.

Contribution seats

The calculation for the 2025-2026 term reflects the deep investment of our vendors in the Istio open source project, especially in the area of ambient mode. As was the case last year, we have five companies represented in our Contribution Seats:

The full allocation can be seen in our formula spreadsheet.

Community Seat election

Last year, we changed the timing of the Community Seat elections to immediately follow the allocation of the Contribution Seats. It is therefore now time to collect our nominations for candidates, and ensure our voter list is correct.

Candidates

Eligibility for candidacy is defined in the Steering Committee charter as a project member who does not work for a Company that will hold a Contribution Seat during the upcoming term.

We would now like to invite members from outside our Contribution Seat holders to stand for election. Nominations are due by February 23.

Voters

Eligibility to vote is defined in the charter as either:

• a project member who has had at least one Pull Request merged in the past 12 months, or
• someone who has submitted the voting exception form and has been accepted by the Steering Committee as having standing in the community through contribution of another kind.

The draft list of voters has been published. If you’re not on that list and you believe you have standing in the Istio community, please submit the exception form.

Exception requests are due by February 23. Voting will start on February 24 and last until March 9.

Announcement of the new committee

Upon the completion of the election, the entire 2025-2026 committee - election winners and company-selected Contribution Seat holders - will be announced.

The Steering Committee wishes to thank its members, old and new, and looks forward to continue to grow and improve Istio as a successful and sustainable open source project. We encourage everyone to get involved in the Istio community by contributing, standing for election, voting, and helping us shape the future of cloud native networking.

We will show you how to get started with a simple example. You will come to see how this combination is a solid option to deliver policy quickly and transparently to application team everywhere in the business, while also providing the data the security teams need for audit and compliance.

Try it out

When integrated with Istio, the Kyverno Authz Server can be used to enforce fine-grained access control policies for microservices.

This guide shows how to enforce access control policies for a simple microservices application.

Prerequisites

• A Kubernetes cluster with Istio installed.
• The istioctl command-line tool installed.

Install Istio and configure your mesh options to enable Kyverno:

$ istioctl install -y -f - <<EOF
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  meshConfig:
    accessLogFile: /dev/stdout
    accessLogFormat: |
      [KYVERNO DEMO] my-new-dynamic-metadata: '%DYNAMIC_METADATA(envoy.filters.http.ext_authz)%'
    extensionProviders:
    - name: kyverno-authz-server
      envoyExtAuthzGrpc:
        service: kyverno-authz-server.kyverno.svc.cluster.local
        port: '9081'
EOF

Notice that in the configuration, we define an extensionProviders section that points to the Kyverno Authz Server installation:

[...]
    extensionProviders:
    - name: kyverno-authz-server
      envoyExtAuthzGrpc:
        service: kyverno-authz-server.kyverno.svc.cluster.local
        port: '9081'
[...]

Deploy the Kyverno Authz Server

The Kyverno Authz Server is a GRPC server capable of processing Envoy External Authorization requests.

It is configurable using Kyverno AuthorizationPolicy resources, either stored in-cluster or provided externally.

$ kubectl create ns kyverno
$ kubectl label namespace kyverno istio-injection=enabled
$ helm install kyverno-authz-server --namespace kyverno --wait --version 0.1.0 --repo https://kyverno.github.io/kyverno-envoy-plugin kyverno-authz-server

Deploy the sample application

httpbin is a well-known application that can be used to test HTTP requests and helps to show quickly how we can play with the request and response attributes.

$ kubectl create ns my-app
$ kubectl label namespace my-app istio-injection=enabled
$ kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.29/samples/httpbin/httpbin.yaml -n my-app

Deploy an Istio AuthorizationPolicy

An AuthorizationPolicy defines the services that will be protected by the Kyverno Authz Server.

$ kubectl apply -f - <<EOF
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: my-kyverno-authz
  namespace: istio-system # This enforce the policy on all the mesh, istio-system being the mesh root namespace
spec:
  selector:
    matchLabels:
      ext-authz: enabled
  action: CUSTOM
  provider:
    name: kyverno-authz-server
  rules: [{}] # Empty rules, it will apply to selectors with ext-authz: enabled label
EOF

Notice that in this resource, we define the Kyverno Authz Server extensionProvider you set in the Istio configuration:

[...]
  provider:
    name: kyverno-authz-server
[...]

Label the app to enforce the policy

Let’s label the app to enforce the policy. The label is needed for the Istio AuthorizationPolicy to apply to the sample application pods.

$ kubectl patch deploy httpbin -n my-app --type=merge -p='{
  "spec": {
    "template": {
      "metadata": {
        "labels": {
          "ext-authz": "enabled"
        }
      }
    }
  }
}'

Deploy a Kyverno AuthorizationPolicy

A Kyverno AuthorizationPolicy defines the rules used by the Kyverno Authz Server to make a decision based on a given Envoy CheckRequest.

It uses the CEL language to analyze an incoming CheckRequest and is expected to produce a CheckResponse in return.

The incoming request is available under the object field, and the policy can define variables that will be made available to all authorizations.

$ kubectl apply -f - <<EOF
apiVersion: envoy.kyverno.io/v1alpha1
kind: AuthorizationPolicy
metadata:
  name: demo-policy.example.com
spec:
  failurePolicy: Fail
  variables:
  - name: force_authorized
    expression: object.attributes.request.http.?headers["x-force-authorized"].orValue("")
  - name: allowed
    expression: variables.force_authorized in ["enabled", "true"]
  authorizations:
  - expression: >
      variables.allowed
        ? envoy.Allowed().Response()
        : envoy.Denied(403).Response()
EOF

Notice that you can build the CheckResponse by hand or use CEL helper functions like envoy.Allowed() and envoy.Denied(403) to simplify creating the response message:

[...]
  - expression: >
      variables.allowed
        ? envoy.Allowed().Response()
        : envoy.Denied(403).Response()
[...]

How it works

When applying the AuthorizationPolicy, the Istio control plane (istiod) sends the required configurations to the sidecar proxy (Envoy) of the selected services in the policy. Envoy will then send the request to the Kyverno Authz Server to check if the request is allowed or not.

The Envoy proxy works by configuring filters in a chain. One of those filters is ext_authz, which implements an external authorization service with a specific message. Any server implementing the correct protobuf can connect to the Envoy proxy and provide the authorization decision; The Kyverno Authz Server is one of those servers.

Reviewing Envoy’s Authorization service documentation, you can see that the message has these attributes:

• Ok response

  {
    "status": {...},
    "ok_response": {
      "headers": [],
      "headers_to_remove": [],
      "response_headers_to_add": [],
      "query_parameters_to_set": [],
      "query_parameters_to_remove": []
    },
    "dynamic_metadata": {...}
  }

• Denied response

  {
    "status": {...},
    "denied_response": {
      "status": {...},
      "headers": [],
      "body": "..."
    },
    "dynamic_metadata": {...}
  }

This means that based on the response from the authz server, Envoy can add or remove headers, query parameters, and even change the response body.

We can do this as well, as documented in the Kyverno Authz Server documentation.

Testing

Let’s test the simple usage (authorization) and then let’s create a more advanced policy to show how we can use the Kyverno Authz Server to modify the request and response.

Deploy an app to run curl commands to the httpbin sample application:

$ kubectl apply -n my-app -f https://raw.githubusercontent.com/istio/istio/release-1.29/samples/curl/curl.yaml

Apply the policy:

$ kubectl apply -f - <<EOF
apiVersion: envoy.kyverno.io/v1alpha1
kind: AuthorizationPolicy
metadata:
  name: demo-policy.example.com
spec:
  failurePolicy: Fail
  variables:
  - name: force_authorized
    expression: object.attributes.request.http.?headers["x-force-authorized"].orValue("")
  - name: allowed
    expression: variables.force_authorized in ["enabled", "true"]
  authorizations:
  - expression: >
      variables.allowed
        ? envoy.Allowed().Response()
        : envoy.Denied(403).Response()
EOF

The simple scenario is to allow requests if they contain the header x-force-authorized with the value enabled or true. If the header is not present or has a different value, the request will be denied.

In this case, we combined allow and denied response handling in a single expression. However it is possible to use multiple expressions, the first one returning a non null response will be used by the Kyverno Authz Server, this is useful when a rule doesn’t want to make a decision and delegate to the next rule:

[...]
  authorizations:
  # allow the request when the header value matches
  - expression: >
      variables.allowed
        ? envoy.Allowed().Response()
        : null
  # else deny the request
  - expression: >
      envoy.Denied(403).Response()
[...]

Simple rule

The following request will return 403:

$ kubectl exec -n my-app deploy/curl -- curl -s -w "\nhttp_code=%{http_code}" httpbin:8000/get

The following request will return 200:

$ kubectl exec -n my-app deploy/curl -- curl -s -w "\nhttp_code=%{http_code}" httpbin:8000/get -H "x-force-authorized: true"

Advanced manipulations

Now the more advanced use case, apply the second policy:

$ kubectl apply -f - <<EOF
apiVersion: envoy.kyverno.io/v1alpha1
kind: AuthorizationPolicy
metadata:
  name: demo-policy.example.com
spec:
  variables:
  - name: force_authorized
    expression: object.attributes.request.http.headers[?"x-force-authorized"].orValue("") in ["enabled", "true"]
  - name: force_unauthenticated
    expression: object.attributes.request.http.headers[?"x-force-unauthenticated"].orValue("") in ["enabled", "true"]
  - name: metadata
    expression: '{"my-new-metadata": "my-new-value"}'
  authorizations:
    # if force_unauthenticated -> 401
  - expression: >
      variables.force_unauthenticated
        ? envoy
            .Denied(401)
            .WithBody("Authentication Failed")
            .Response()
        : null
    # if force_authorized -> 200
  - expression: >
      variables.force_authorized
        ? envoy
            .Allowed()
            .WithHeader("x-validated-by", "my-security-checkpoint")
            .WithoutHeader("x-force-authorized")
            .WithResponseHeader("x-add-custom-response-header", "added")
            .Response()
            .WithMetadata(variables.metadata)
        : null
    # else -> 403
  - expression: >
      envoy
        .Denied(403)
        .WithBody("Unauthorized Request")
        .Response()
EOF

In that policy, you can see:

• If the request has the x-force-unauthenticated: true header (or x-force-unauthenticated: enabled), we will return 401 with the “Authentication Failed” body
• Else, if the request has the x-force-authorized: true header (or x-force-authorized: enabled), we will return 200 and manipulate request headers, response headers and inject dynamic metadata
• In all other cases, we will return 403 with the “Unauthorized Request” body

The corresponding CheckResponse will be returned to the Envoy proxy from the Kyverno Authz Server. Envoy will use those values to modify the request and response accordingly.

Change returned body

Let’s test the new capabilities:

$ kubectl exec -n my-app deploy/curl -- curl -s -w "\nhttp_code=%{http_code}" httpbin:8000/get

Now we can change the response body.

With 403 the body will be changed to “Unauthorized Request”, running the previous command, you should receive:

Unauthorized Request
http_code=403

Change returned body and status code

Running the request with the header x-force-unauthenticated: true:

$ kubectl exec -n my-app deploy/curl -- curl -s -w "\nhttp_code=%{http_code}" httpbin:8000/get -H "x-force-unauthenticated: true"

This time you should receive the body “Authentication Failed” and error 401:

Authentication Failed
http_code=401

Adding headers to request

Running a valid request:

$ kubectl exec -n my-app deploy/curl -- curl -s -w "\nhttp_code=%{http_code}" httpbin:8000/get -H "x-force-authorized: true"

You should receive the echo body with the new header x-validated-by: my-security-checkpoint and the header x-force-authorized removed:

[...]
    "X-Validated-By": [
      "my-security-checkpoint"
    ]
[...]
http_code=200

Adding headers to response

Running the same request but showing only the header:

$ kubectl exec -n my-app deploy/curl -- curl -s -I -w "\nhttp_code=%{http_code}" httpbin:8000/get -H "x-force-authorized: true"

You will find the response header added during the Authz check x-add-custom-response-header: added:

HTTP/1.1 200 OK
[...]
x-add-custom-response-header: added
[...]
http_code=200

Sharing data between filters

Finally, you can pass data to the following Envoy filters using dynamic_metadata.

This is useful when you want to pass data to another ext_authz filter in the chain or you want to print it in the application logs.

To do so, review the access log format you set earlier:

[...]
    accessLogFormat: |
      [KYVERNO DEMO] my-new-dynamic-metadata: "%DYNAMIC_METADATA(envoy.filters.http.ext_authz)%"
[...]

DYNAMIC_METADATA is a reserved keyword to access the metadata object. The rest is the name of the filter that you want to access.

In our case, the name envoy.filters.http.ext_authz is created automatically by Istio. You can verify this by dumping the Envoy configuration:

$ istioctl pc all deploy/httpbin -n my-app -oyaml | grep envoy.filters.http.ext_authz

You will see the configurations for the filter.

Let’s test the dynamic metadata. In the advance rule, we are creating a new metadata entry: {"my-new-metadata": "my-new-value"}.

Run the request and check the logs of the application:

$ kubectl exec -n my-app deploy/curl -- curl -s -I httpbin:8000/get -H "x-force-authorized: true"

$ kubectl logs -n my-app deploy/httpbin -c istio-proxy --tail 1

You will see in the output the new attributes configured by the Kyverno policy:

[...]
[KYVERNO DEMO] my-new-dynamic-metadata: '{"my-new-metadata":"my-new-value","ext_authz_duration":5}'
[...]

Conclusion

In this guide, we have shown how to integrate Istio and the Kyverno Authz Server to enforce policies for a simple microservices application. We also showed how to use policies to modify the request and response attributes.

This is the foundational example for building a platform-wide policy system that can be used by all application teams.

And here it is!

The Istio project is excited to unveil Izzy’s adventure sailing with the Phippy family at KubeCon North America 2024 this week, as together we celebrate the 10 year anniversary of Kubernetes. Copies are available at the CNCF Store, or on the online store shortly after the event.

Captain Kube hosts a grand birthday bash on a special cruise with Phippy and her friends, however the ship is in great danger! But there is nothing to worry about, when Izzy is in charge of the security! Join Izzy’s smart and adventurous chase of the pirates who want to spoil Captain Kube’s birthday bash.

Why the book?

The co-authors of the book, Faseela K. and Lin Sun, are both Istio maintainers and parents. They have often found themselves in a tough spot explaining what they do at work, particularly in a context that makes sense to younger people. Their children read and enjoyed the Illustrated Children’s Guide to Kubernetes but were curious to learn more about the other characters and their roles and responsibilities!

This book is for every one who has encountered curious little eyes that keep asking you what “Service Mesh” is. It’s also a great gift for anyone of any age who needs to understand what Istio is, or who thinks that service mesh is too complex.

Acknowledgements

The Istio Steering Committee would like to thank Faseela and Lin for writing this amazing book. Suri Patel and Alex Davy from CNCF did a wonderful job with the design and illustrations, bringing the story to life. Last, but not least, a huge thanks to Katie Greenley for her support throughout the process to make sure the book was released on time for Captain Kube’s birthday celebrations at our community’s largest international conference.

We are planning a book signing event at next year’s KubeCon EU in London.

Happy reading!

Ambient mesh — and its reference implementation with Istio’s ambient mode — was announced in September 2022. Since then, our community has put in 26 months of hard work and collaboration, with contributions from Solo.io, Google, Microsoft, Intel, Aviatrix, Huawei, IBM, Red Hat, and many others. Stable status in 1.24 indicates the features of ambient mode are now fully ready for broad production workloads. This is a huge milestone for Istio, bringing Istio to production readiness without sidecars, and offering users a choice.

Why ambient mesh?

From the launch of Istio in 2017, we have observed a clear and growing demand for mesh capabilities for applications — but heard that many users found the resource overhead and operational complexity of sidecars hard to overcome. Challenges that Istio users shared with us include how sidecars can break applications after they are added, the large CPU and memory requirement for a proxy with every workload, and the inconvenience of needing to restart application pods with every new Istio release.

As a community, we designed ambient mesh from the ground up to tackle these problems, alleviating the previous barriers of complexity faced by users looking to implement service mesh. The new concept was named ‘ambient mesh’ as it was designed to be transparent to your application, with no proxy infrastructure collocated with user workloads, no subtle changes to configuration required to onboard, and no application restarts required. In ambient mode it is trivial to add or remove applications from the mesh. All you need to do is label a namespace, and all applications in that namespace are instantly added to the mesh. This immediately secures all traffic within that namespace with industry-standard mutual TLS encryption — no other configuration or restarts required!. Refer to the Introducing Ambient Mesh blog for more information on why we built Istio’s ambient mode.

How does ambient mode make adoption easier?

The core innovation behind ambient mesh is that it slices Layer 4 (L4) and Layer 7 (L7) processing into two distinct layers. Istio’s ambient mode is powered by lightweight, shared L4 node proxies and optional L7 proxies, removing the need for traditional sidecar proxies from the data plane. This layered approach allows you to adopt Istio incrementally, enabling a smooth transition from no mesh, to a secure overlay (L4), to optional full L7 processing — on a per-namespace basis, as needed, across your fleet.

By utilizing ambient mesh, users bypass some of the previously restrictive elements of the sidecar model. Server-send-first protocols now work, most reserved ports are now available, and the ability for containers to bypass the sidecar — either maliciously or not — is eliminated.

The lightweight shared L4 node proxy is called the ztunnel (zero-trust tunnel). ztunnel drastically reduces the overhead of running a mesh by removing the need to potentially over-provision memory and CPU within a cluster to handle expected loads. In some use cases, the savings can exceed 90% or more, while still providing zero-trust security using mutual TLS with cryptographic identity, simple L4 authorization policies, and telemetry.

The L7 proxies are called waypoints. Waypoints process L7 functions such as traffic routing, rich authorization policy enforcement, and enterprise-grade resilience. Waypoints run outside of your application deployments and can scale independently based on your needs, which could be for the entire namespace or for multiple services within a namespace. Compared with sidecars, you don’t need one waypoint per application pod, and you can scale your waypoint effectively based on its scope, thus saving significant amounts of CPU and memory in most cases.

The separation between the L4 secure overlay layer and L7 processing layer allows incremental adoption of the ambient mode data plane, in contrast to the earlier binary “all-in” injection of sidecars. Users can start with the secure L4 overlay, which offers a majority of features that people deploy Istio for (mTLS, authorization policy, and telemetry). Complex L7 handling such as retries, traffic splitting, load balancing, and observability collection can then be enabled on a case-by-case basis.

Rapid exploration and adoption of ambient mode

The ztunnel image on Docker Hub has reached over 1 million downloads, with ~63,000 pulls in the last week alone.

We asked a few of our users for their thoughts on ambient mode’s GA:

What is in scope?

The general availability of ambient mode means the following things are now considered stable:

• Installing Istio with support for ambient mode, with Helm or istioctl.
• Adding your workloads to the mesh to gain mutual TLS with cryptographic identity, L4 authorization policies, and telemetry.
• Configuring waypoints to use L7 functions such as traffic shifting, request routing, and rich authorization policy enforcement.
• Connecting the Istio ingress gateway to workloads in ambient mode, supporting the Kubernetes Gateway APIs and all existing Istio APIs.
• Using waypoints for controlled mesh egress
• Using istioctl to operate waypoints, and troubleshoot ztunnel & waypoints.

Refer to the feature status page for more information.

Roadmap

We are not standing still! There are a number of features that we continue to work on for future releases, including some that are currently in Alpha/Beta.

In our upcoming releases, we expect to move quickly on the following extensions to ambient mode:

• Full support for sidecar and ambient mode interoperability
• Multi-cluster installations
• Multi-network support
• VM support

What about sidecars?

Sidecars are not going away, and remain first-class citizens in Istio. You can continue to use sidecars, and they will remain fully supported. While we believe most use cases will be best served with a mesh in ambient mode, the Istio project remains committed to ongoing sidecar mode support.

Try ambient mode today

With the 1.24 release of Istio and the GA release of ambient mode, it is now easier than ever to try out Istio on your own workloads.

• Follow the getting started guide to explore ambient mode.
• Read our user guides to learn how to incrementally adopt ambient for mutual TLS & L4 authorization policy, traffic management, rich L7 authorization policy, and more.
• Explore the new Kiali 2.0 dashboard to visualize your mesh.

You can engage with the developers in the #ambient channel on the Istio Slack, or use the discussion forum on GitHub for any questions you may have.

• Come to the Istio Day co-located event.

• Attend the Istio Maintainers’ Track session: Life of a Packet: Ambient Edition

• Drop by the Istio Contribfest session: Sidecarless Service Mesh: Let’s Work Together on Istio V2

• Add the following KubeCon sessions to your schedule, all of which have an Istio flavor:

  • Why Choose Istio in 2025 | Project Lightning Talk
  • Lightning Talk: Effortless, Sidecar-Less Mutual TLS and Rich Authorization Policies up and Running in 5 Minutes
  • Poster Session : Unleashing the Power of Prediction to Proactively Scale Control Plane Components
  • What Istio Got Wrong: Learnings from the Last Seven Years of Service Mesh
  • Tutorial: Live with Gateway API V1.2
  • Mish-Mesh: Abusing the Service Mesh to Compromise Kubernetes Environments
  • Engaging the KServe Community, The Impact of Integrating Solutions with Standardized CNCF Projects
  • How Google Built a New Cloud on Top of Kubernetes
  • Securing Outgoing Traffic: Building a Powerful Internet Egress Gateway for Reliable Connectivity
  • Testing Kubernetes Without Kubernetes: A Networking Deep Dive
  • How GoTo Financial Automates Upgrading 60+ Istio Service Mesh Seamlessly!

• Have a chat with maintainers and users at the Istio kiosk in the Project Pavilion throughout the event, where you can grab a cool Istio T-shirt with our brand new design.

• We also have an interesting surprise for all Istio lovers, to be released at the KubeCon North America CNCF store. Stay tuned!

Follow us on X, LinkedIn or Bluesky to get live updates from the event. See you soon!

Despite offering a similar feature set on the surface, the two projects have substantially different architectures, most notably Cilium’s use of eBPF and WireGuard for processing and encrypting L4 traffic in the kernel, contrasted with Istio’s ztunnel component for L4 in user space. These differences have resulted in substantial speculation about how Istio will perform at scale compared to Cilium.

While many comparisons have been made about tenancy models, security protocols and basic performance of the two projects, there has not yet been a full evaluation published at enterprise scale. Rather than emphasizing theoretical performance, we put Istio’s ambient mode and Cilium through their paces, focusing on key metrics like latency, throughput, and resource consumption. We cranked up the pressure with realistic load scenarios, simulating a bustling Kubernetes environment. Finally, we pushed the size of our AKS cluster up to 1,000 nodes on 11,000 cores, to understand how these projects perform at scale. Our results show areas where each can improve, but also indicate that Istio is the clear winner.

Test Scenario

In order to push Istio and Cilium to their limits, we created 500 different services, each backed by 100 pods. Each service is in a separate namespace, which also contains one Fortio load generator client. We restricted the clients to a node pool of 100 32-core machines, to eliminate noise from collocated clients, and allocated the remaining 900 8-core instances to our services.

For the Istio test, we used Istio’s ambient mode, with a waypoint proxy in every service namespace, and default install parameters. In order to make our test scenarios similar, we had to turn on a few non-default features in Cilium, including WireGuard encryption, L7 Proxies, and Node Init. We also created a Cilium Network Policy in each namespace, with HTTP path-based rules. In both scenarios, we generated churn by scaling one service to between 85 and 115 instances at random every second, and relabeling one namespace every minute. To see the precise settings we used, and to reproduce our results, see my notes.

Scalability Scorecard

Taking into account the resource used, Istio processed 2178 Queries Per Core, vs Cilium’s 1815, a 20% improvement.

• The Cilium Slowdown: Cilium, while boasting impressive low latency with default install parameters, slows down substantially when Istio’s baseline features such as L7 policy and encryption are turned on. Additionally, Cilium’s memory and CPU utilization remained high even when no traffic was flowing in the mesh. This can impact the overall stability and reliability of your cluster, especially as it grows.
• Istio, The Steady Performer: Istio’s ambient mode, on the other hand, showed its strength in stability and maintaining decent throughput, even with the added overhead of encryption. While Istio did consume more memory and CPU than Cilium under test, its CPU utilization settled to a fraction of Cilium’s when not under load.

Behind the Scenes: Why the Difference?

The key to understanding these performance differences lies in the architecture and design of each tool.

• Cilium’s Control Plane Conundrum: Cilium runs a control plane instance on each node, leading to API server strain and configuration overhead as your cluster expands. This frequently caused our API server to crash, followed by Cilium becoming unready, and the entire cluster becoming unresponsive.
• Istio’s Efficiency Edge: Istio, with its centralized control plane and identity-based approach, streamlines configuration and reduces the burden on your API server and nodes, directing critical resources to processing and securing your traffic, rather than processing configuration. Istio takes further advantage of the resources not used in the control plane by running as many Envoy instances as a workload needs, while Cilium is limited to one shared Envoy instance per node.

Digging Deeper

While the objective of this project is to compare Istio and Cilium scalability, several constraints make a direct comparison difficult.

Layer 4 Isn’t always Layer 4

While Istio and Cilium both offer L4 policy enforcement, their APIs and implementation differ substantially. Cilium implements Kubernetes NetworkPolicy, which uses labels and namespaces to block or allow access to and from IP Addresses. Istio offers an AuthorizationPolicy API, and makes allow and deny decisions based on the TLS identity used to sign each request. Most defense-in-depth strategies will need to make use of both NetworkPolicy and TLS-based policy for comprehensive security.

Not all Encryption is Created Equal

While Cilium offers IPsec for FIPS-compatible encryption, most other Cilium features such as L7 policy and load balancing are incompatible with IPsec. Cilium has much better feature compatibility when using WireGuard encryption, but WireGuard cannot be used in FIPS-compliant environments. Istio, on the other-hand, because it strictly complies with TLS protocol standards, always uses FIPS-compliant mTLS by default.

Hidden Costs

While Istio operates entirely in user space, Cilium’s L4 dataplane runs in the Linux kernel using eBPF. Prometheus metrics for resource consumption only measure user space resources, meaning that all kernel resources used by Cilium are not accounted for in this test.

Recommendations: Choosing the Right Tool for the Job

So, what’s the verdict? Well, it depends on your specific needs and priorities. For small clusters with pure L3/L4 use cases and no requirement for encryption, Cilium offers a cost-effective and performant solution. However, for larger clusters and a focus on stability, scalability, and advanced features, Istio’s ambient mode, along with an alternate NetworkPolicy implementation, is the way to go. Many customers choose to combine the L3/L4 features of Cilium with the L4/L7 and encryption features of Istio for a defense-in-depth strategy.

Remember, the world of cloud-native networking is constantly evolving. Keep an eye on developments in both Istio and Cilium, as they continue to improve and address these challenges.

Let’s Keep the Conversation Going

Have you worked with Istio’s ambient mode or Cilium? What are your experiences and insights? Share your thoughts in the comments below. Let’s learn from each other and navigate the exciting world of Kubernetes together!

The Steering Committee represents the companies and contributors that support the Istio project, while the TOC is the top of an individual contributor ladder made up of our members, maintainers and working group leads.

Each year, we build our Steering Committee with representatives from our top commercial contributors, and members elected by our maintainer community. This is the group with the responsibility of electing new TOC members, who have traditionally served indefinitely.

We want to ensure that all the members of our community have the opportunity to stand for, and serve in, our leadership positions. Today, we are pleased to announce our transition to a regularly-elected TOC, with members serving two-year terms, and call for candidates for our first election.

What does the Technical Oversight Committee do?

The charter for the TOC spells out the responsibilities of its members, including:

• Setting the overall technical direction and roadmap of the project.
• Resolving technical issues, disagreements, and escalations.
• Declaring maturity levels for Istio features.
• Approving the creation and dissolution of working groups and approving leadership changes of working groups.
• Ensuring the team adheres to our code of conduct and respects our values.
• Fostering an environment for a healthy and happy community of developers and contributors.

While the interest of our vendors is represented by our Steering Committee, TOC membership is associated with the individual, irrespective of their current employer. Members act independently, in their individual capacities, and must prioritize the best interests of the project and the community. This has always been achieved by method of consensus, and as such we seat an even number of members. The TOC has traditionally comprised 6 members, and this remains the case going forward.

What changes with the new charter?

The key changes in the new charter, recently ratified by the Steering Committee, are:

• Members will now serve 2 year terms.
• The Steering Committee will vote every year to (re-^†) seat 3 of the 6 members on the TOC.
• The mechanics for election are clearly defined, including the expectation for candidates to qualify for the election, and how they will be evaluated.
• The expectations of regular meetings between the Steering and TOC have been formalized.
• There is now a formal process for removing a TOC member, should they lose the confidence of the Steering Committee.

^† There is no limit on the number of terms a member may serve for, and incumbent TOC members are welcome to run again at the end of their term.

TOC member farewells

We recently announced the retirement of long-time contributor Eric Van Norman. We also now bid farewell to Neeraj Poddar from the Istio TOC. Neeraj has been involved with the project since 2017, co-founding Aspen Mesh within F5, and later leading Gloo Mesh as VP of Engineering at Solo.io. He was first elected to the TOC in 2020. Neeraj has taken a role as VP of Engineering at NimbleEdge, and we congratulate him and wish him well for the future.

Maintainers: stand in our first election

We have set our annual TOC elections to occur after the seating of the Steering Committee each year, which will put the first instance around March 2025.

However, as we currently have two vacancies, we are announcing our first election will be a by-election to fill these two seats for the remainder of their terms.

The bar for joining the TOC is deliberately set high. Candidates must be tenured maintainers, recognized within the Istio community as collaborative technical leaders, and meet qualification criteria which demonstrate their suitability for the position.

To stand for a TOC seat, please send an e-mail to elections@istio.io, including a link to a one-page Google Doc with your self-assessment against the qualification criteria. Nominations will close in two weeks, on 31 October.

Good luck!

Often work is given directly to application teams to implement, but there are some features that are best implemented once, and offered as a service to all teams. One feature within the reach of most platform teams is offering a standard, responsive system for Layer 7 application authorization policy. Policy as code enables teams to lift authorization decisions out of the application layer into a lightweight and performant decoupled system. It might sound like a challenge, but it doesn’t have to be, with the right tools for the job.

We’re going to dive into how Istio and Open Policy Agent (OPA) can be used to enforce Layer 7 policies in your platform. We’ll show you how to get started with a simple example. You will come to see how this combination is a solid option to deliver policy quickly and transparently to application team everywhere in the business, while also providing the data the security teams need for audit and compliance.

Try it out

When integrated with Istio, OPA can be used to enforce fine-grained access control policies for microservices. This guide shows how to enforce access control policies for a simple microservices application.

Prerequisites

• A Kubernetes cluster with Istio installed.
• The istioctl command-line tool installed.

Install Istio and configure your mesh options to enable OPA:

$ istioctl install -y -f - <<'EOF'
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  meshConfig:
    accessLogFile: /dev/stdout
    accessLogFormat: |
      [OPA DEMO] my-new-dynamic-metadata: "%DYNAMIC_METADATA(envoy.filters.http.ext_authz)%"
    extensionProviders:
    - name: "opa.local"
      envoyExtAuthzGrpc:
        service: "opa.opa.svc.cluster.local"
        port: "9191"
EOF

Notice that in the configuration, we define an extensionProviders section that points to the OPA standalone installation.

Deploy the sample application. Httpbin is a well-known application that can be used to test HTTP requests and helps to show quickly how we can play with the request and response attributes.

$ kubectl create ns my-app
$ kubectl label namespace my-app istio-injection=enabled

$ kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.29/samples/httpbin/httpbin.yaml -n my-app

Deploy OPA. It will fail because it expects a configMap containing the default Rego rule to use. This configMap will be deployed later in our example.

$ kubectl create ns opa
$ kubectl label namespace opa istio-injection=enabled

$ kubectl apply -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: opa
  name: opa
  namespace: opa
spec:
  replicas: 1
  selector:
    matchLabels:
      app: opa
  template:
    metadata:
      labels:
        app: opa
    spec:
      containers:
      - image: openpolicyagent/opa:0.61.0-envoy
        name: opa
        args:
          - "run"
          - "--server"
          - "--disable-telemetry"
          - "--config-file=/config/config.yaml"
          - "--log-level=debug" # Uncomment this line to enable debug logs
          - "--diagnostic-addr=0.0.0.0:8282"
          - "/policy/policy.rego" # Default policy
        volumeMounts:
          - mountPath: "/config"
            name: opa-config
          - mountPath: "/policy"
            name: opa-policy
      volumes:
        - name: opa-config
          configMap:
            name: opa-config
        - name: opa-policy
          configMap:
            name: opa-policy
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: opa-config
  namespace: opa
data:
  config.yaml: |
    # Here the OPA configuration you can find in the offcial documention
    decision_logs:
      console: true
    plugins:
      envoy_ext_authz_grpc:
        addr: ":9191"
        path: mypackage/mysubpackage/myrule # Default path for grpc plugin
    # Here you can add your own configuration with services and bundles
---
apiVersion: v1
kind: Service
metadata:
  name: opa
  namespace: opa
  labels:
    app: opa
spec:
  ports:
    - port: 9191
      protocol: TCP
      name: grpc
  selector:
    app: opa
---
EOF

Deploy the AuthorizationPolicy to define which services will be protected by OPA.

$ kubectl apply -f - <<EOF
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: my-opa-authz
  namespace: istio-system # This enforce the policy on all the mesh being istio-system the mesh config namespace
spec:
  selector:
    matchLabels:
      ext-authz: enabled
  action: CUSTOM
  provider:
    name: "opa.local"
  rules: [{}] # Empty rules, it will apply to selectors with ext-authz: enabled label
EOF

Let’s label the app to enforce the policy:

$ kubectl patch deploy httpbin -n my-app --type=merge -p='{
  "spec": {
    "template": {
      "metadata": {
        "labels": {
          "ext-authz": "enabled"
        }
      }
    }
  }
}'

Notice that in this resource, we define the OPA extensionProvider you set in the Istio configuration:

[...]
  provider:
    name: "opa.local"
[...]

How it works

When applying the AuthorizationPolicy, the Istio control plane (istiod) sends the required configurations to the sidecar proxy (Envoy) of the selected services in the policy. Envoy will then send the request to the OPA server to check if the request is allowed or not.

The Envoy proxy works by configuring filters in a chain. One of those filters is ext_authz, which implements an external authorization service with a specific message. Any server implementing the correct protobuf can connect to the Envoy proxy and provide the authorization decision; OPA is one of those servers.

Before, when you installed OPA server, you used the Envoy version of the server. This image allows the configuration of the gRPC plugin which implements the ext_authz protobuf service.

[...]
      containers:
      - image: openpolicyagent/opa:0.61.0-envoy # This is the OPA image version which brings the Envoy plugin
        name: opa
[...]

In the configuration, you have enabled the Envoy plugin and the port which will listened to:

[...]
    decision_logs:
      console: true
    plugins:
      envoy_ext_authz_grpc:
        addr: ":9191" # This is the port where the envoy plugin will listen
        path: mypackage/mysubpackage/myrule # Default path for grpc plugin
    # Here you can add your own configuration with services and bundles
[...]

Reviewing Envoy’s Authorization service documentation, you can see that the message has these attributes:

OkHttpResponse
{
  "status": {...},
  "denied_response": {...},
  "ok_response": {
      "headers": [],
      "headers_to_remove": [],
      "dynamic_metadata": {...},
      "response_headers_to_add": [],
      "query_parameters_to_set": [],
      "query_parameters_to_remove": []
    },
  "dynamic_metadata": {...}
}

This means that based on the response from the authz server, Envoy can add or remove headers, query parameters, and even change the response status. OPA can do this as well, as documented in the OPA documentation.

Testing

Let’s test the simple usage (authorization) and then let’s create a more advanced rule to show how we can use OPA to modify the request and response.

Deploy an app to run curl commands to the httpbin sample application:

$ kubectl -n my-app run --image=curlimages/curl curl -- /bin/sleep 100d

Apply the first Rego rule and restart the OPA deployment:

$ kubectl apply -f - <<EOF
apiVersion: v1
kind: ConfigMap
metadata:
  name: opa-policy
  namespace: opa
data:
  policy.rego: |
    package mypackage.mysubpackage

    import rego.v1

    default myrule := false

    myrule if {
      input.attributes.request.http.headers["x-force-authorized"] == "enabled"
    }

    myrule if {
      input.attributes.request.http.headers["x-force-authorized"] == "true"
    }
EOF

$ kubectl rollout restart deployment -n opa

The simple scenario is to allow requests if they contain the header x-force-authorized with the value enabled or true. If the header is not present or has a different value, the request will be denied.

There are multiple ways to create the Rego rule. In this case, we created two different rules. Executed in order, the first one which satisfies all the conditions will be the one that will be used.

Simple rule

The following request will return 403:

$ kubectl exec -n my-app curl -c curl  -- curl -s -w "\nhttp_code=%{http_code}" httpbin:8000/get

The following request will return 200 and the body:

$ kubectl exec -n my-app curl -c curl  -- curl -s -w "\nhttp_code=%{http_code}" httpbin:8000/get -H "x-force-authorized: enabled"

Advanced manipulations

Now the more advanced rule. Apply the second Rego rule and restart the OPA deployment:

$ kubectl apply -f - <<EOF
apiVersion: v1
kind: ConfigMap
metadata:
  name: opa-policy
  namespace: opa
data:
  policy.rego: |
    package mypackage.mysubpackage

    import rego.v1

    request_headers := input.attributes.request.http.headers

    force_unauthenticated if request_headers["x-force-unauthenticated"] == "enabled"

    default allow := false

    allow if {
      not force_unauthenticated
      request_headers["x-force-authorized"] == "true"
    }

    default status_code := 403

    status_code := 200 if allow

    status_code := 401 if force_unauthenticated

    default body := "Unauthorized Request"

    body := "Authentication Failed" if force_unauthenticated

    myrule := {
      "body": body,
      "http_status": status_code,
      "allowed": allow,
      "headers": {"x-validated-by": "my-security-checkpoint"},
      "response_headers_to_add": {"x-add-custom-response-header": "added"},
      "request_headers_to_remove": ["x-force-authorized"],
      "dynamic_metadata": {"my-new-metadata": "my-new-value"},
    }
EOF

$ kubectl rollout restart deployment -n opa

In that rule, you can see:

myrule["allowed"] := allow # Notice that `allowed` is mandatory when returning an object, like here `myrule`
myrule["headers"] := headers
myrule["response_headers_to_add"] := response_headers_to_add
myrule["request_headers_to_remove"] := request_headers_to_remove
myrule["body"] := body
myrule["http_status"] := status_code

Those are the values that will be returned to the Envoy proxy from the OPA server. Envoy will use those values to modify the request and response.

Notice that allowed is required when returning a JSON object instead of only true/false. This can be found in the OPA documentation.

Change returned body

Let’s test the new capabilities:

$ kubectl exec -n my-app curl -c curl  -- curl -s -w "\nhttp_code=%{http_code}" httpbin:8000/get

Now we can change the response body. With 403 the body in the Rego rule is changed to “Unauthorized Request”. With the previous command, you should receive:

Unauthorized Request
http_code=403

Change returned body and status code

Running the request with the header x-force-authorized: enabled you should receive the body “Authentication Failed” and error “401”:

$ kubectl exec -n my-app curl -c curl  -- curl -s -w "\nhttp_code=%{http_code}" httpbin:8000/get -H "x-force-unauthenticated: enabled"

Adding headers to request

Running a valid request, you should receive the echo body with the new header x-validated-by: my-security-checkpoint and the header x-force-authorized removed:

$ kubectl exec -n my-app curl -c curl  -- curl -s httpbin:8000/get -H "x-force-authorized: true"

Adding headers to response

Running the same request but showing only the header, you will find the response header added during the Authz check x-add-custom-response-header: added:

$ kubectl exec -n my-app curl -c curl  -- curl -s -I httpbin:8000/get -H "x-force-authorized: true"

Sharing data between filters

Finally, you can pass data to the following Envoy filters using dynamic_metadata. This is useful when you want to pass data to another ext_authz filter in the chain or you want to print it in the application logs.

To do so, review the access log format you set earlier:

[...]
    accessLogFormat: |
      [OPA DEMO] my-new-dynamic-metadata: "%DYNAMIC_METADATA(envoy.filters.http.ext_authz)%"
[...]

DYNAMIC_METADATA is a reserved keyword to access the metadata object. The rest is the name of the filter that you want to access. In your case, the name envoy.filters.http.ext_authz is created automatically by Istio. You can verify this by dumping the Envoy configuration:

$ istioctl pc all deploy/httpbin -n my-app -oyaml | grep envoy.filters.http.ext_authz

You will see the configurations for the filter.

Let’s test the dynamic metadata. In the advance rule, you are creating a new metadata entry: {"my-new-metadata": "my-new-value"}.

Run the request and check the logs of the application:

$ kubectl exec -n my-app curl -c curl  -- curl -s -I httpbin:8000/get -H "x-force-authorized: true"
$ kubectl logs -n my-app deploy/httpbin -c istio-proxy --tail 1

You will see in the output the new attributes configured by OPA Rego rules:

[...]
 my-new-dynamic-metadata: "{"my-new-metadata":"my-new-value","decision_id":"8a6d5359-142c-4431-96cd-d683801e889f","ext_authz_duration":7}"
[...]

Conclusion

In this guide, we have shown how to integrate Istio and OPA to enforce policies for a simple microservices application. We also showed how to use Rego to modify the request and response attributes. This is the foundational example for building a platform-wide policy system that can be used by all application teams.

Read the whole post at lucavall.in.

The Sail Operator manages the lifecycle of Istio control planes, making it easier and more efficient for cluster administrators to deploy, configure and upgrade Istio in large scale production environments. Instead of creating a new configuration schema and reinventing the wheel, the Sail Operator APIs are built around Istio’s Helm chart APIs. All installation and configuration options that are exposed by Istio’s Helm charts are available through the Sail Operator CRDs’ values fields. This means that you can easily manage and customize Istio using familiar configurations without adding additional items to learn.

The Sail Operator has 3 main resource concepts:

• Istio: used to manage the Istio control planes.
• Istio Revision: represents a revision of that control plane, which is an instance of Istio with a specific version and revision name.
• Istio CNI: used to manage the resource and lifecycle of Istio’s CNI plugin. To install the Istio CNI Plugin, you create an IstioCNI resource.

Currently, the main feature of the Sail Operator is the Update Strategy. The operator provides an interface that manages the upgrade of Istio control plane(s). It currently supports two update strategies:

• In Place: with the InPlace strategy, the existing Istio control plane is replaced with a new version, and the workload sidecars immediately connect to the new control plane. This way, workloads don’t need to be moved from one control plane instance to another.
• Revision Based: with the RevisionBased strategy, a new Istio control plane instance is created for every change to the Istio.spec.version field. The old control plane remains in place until all workloads have been moved to the new control plane instance. Optionally, the updateWorkloads flag can be set to automatically move workloads to the new control plane when it is ready.

We know that doing upgrades of the Istio control plane carries risk and can require a substantial manual effort for large deployments and this is why it is our current focus. For the future, we are looking at how the Sail Operator can better support use cases such as multi-tenancy and isolation, multi-cluster federation, and simplified integration with 3rd party projects.

The Sail Operator project is still alpha and under heavy development. Note that as an istio-ecosystem project, it is not supported as part of the Istio project. We are actively seeking feedback and contributions from the community. If you want to get involved with the project please refer to the repo documentation and contributing guidelines. If you are a user, you can also try the new operator by following the instructions in the user documentation.

For more information, contact us:

• Discussions
• Issues
• Slack

Does this affect you?

This deprecation only affects users of the In-Cluster Operator. Users who install Istio with the istioctl install command and an IstioOperator YAML file are not affected.

To determine if you are affected, run kubectl get deployment -n istio-system istio-operator and kubectl get IstioOperator. If both commands return non-empty values, your cluster will be affected. Based on recent polls, we expect that this will affect fewer than 10% of Istio users.

Operator-based Installations of Istio will continue to run indefinitely, but cannot be upgraded past 1.23.x.

When do I need to migrate?

In keeping with Istio’s deprecation policy for Beta features, the Istio In-Cluster Operator will be removed with the release of Istio 1.24, roughly three months from this announcement. Istio 1.23 will be supported through March 2025, at which time operator users will need to migrate to another install mechanism to retain support.

How do I migrate?

The Istio project will continue to support installation and upgrade via the istioctl command, as well as with Helm. Because of Helm’s popularity within the platform engineering ecosystem, we recommend most users migrate to Helm. istioctl install is based on Helm templates, and future versions may integrate deeper with Helm.

Helm installs can also be managed with GitOps tools like Flux or Argo CD.

Users who prefer the operator pattern for running Istio can migrate to either of two new Istio Ecosystem projects, the Classic Operator Controller, or the Sail Operator.

Migrating to Helm

Helm migration requires translating your IstioOperator YAML into Helm values. Istio 1.24 and above includes a manifest translate command to perform this operation. The output is a values.yaml file, and a shell script to install equivalent Helm charts.

$ istioctl manifest translate -f istio.yaml

Migrating to istioctl

Identify your IstioOperator custom resource: there should be only one result.

$ kubectl get IstioOperator

Using the name of your resource, download your operator configuration in YAML format:

$ kubectl get IstioOperator <name> -o yaml > istio.yaml

Disable the In-Cluster Operator. This will not disable your control plane or disrupt your current mesh traffic.

$ kubectl scale deployment -n istio-system istio-operator –replicas 0

When you are ready to upgrade Istio to version 1.24 or later, follow the upgrade instructions, using the istio.yaml file you downloaded above.

Once you have completed and verified your migration, run the following commands to clean up your operator resources:

$ kubectl delete deployment -n istio-system istio-operator
$ kubectl delete customresourcedefinition istiooperator

Migrating to the Classic Operator Controller

A new ecosystem project, the Classic Operator Controller, is a fork of the original controller built into Istio. This project maintains the same API and code base as the original operator, but is maintained outside of Istio core.

Because the API is the same, migration is straightforward: only the installation of the new operator will be required.

Classic Operator Controller is not supported by the Istio project.

Migrating to Sail Operator

A new ecosystem project, the Sail Operator, is able to install and manage the lifecycle of the Istio control plane in a Kubernetes or OpenShift cluster.

Sail Operator APIs are built around Istio’s Helm chart APIs. All installation and configuration options that are exposed by Istio’s Helm charts are available through the Sail Operator CRD’s values: fields.

Sail Operator is not supported by the Istio project.

What is an operator, and why did Istio have one?

The operator pattern was popularized by CoreOS in 2016 as a method for codifying human intelligence into code. The most common use case is a database operator, where a user might have multiple database instances in one cluster, with multiple ongoing operational tasks (backups, vacuums, sharding).

Istio introduced istioctl and the in-cluster operator in version 1.4, in response to problems with Helm v2. Around the same time, Helm v3 was introduced, which addressed the community’s concerns, and is a preferred method for installing software on Kubernetes today. Support for Helm v3 was added in Istio 1.8.

Istio’s in-cluster operator handled installation of the service mesh components - an operation you generally do one time, and for one instance, per cluster. You can think of it as a way to run istioctl inside your cluster. However, this meant you had a high-privilege controller running inside your cluster, which weakens your security posture. It doesn’t handle any ongoing administration tasks (backing up, taking snapshots etc, are not requirements for running Istio).

The Istio operator is something you have to install into the cluster, which means you already have to manage the installation of something. Using it to upgrade the cluster likewise first required you to download and run a new version of istioctl.

Using an operator means you have created a level of indirection, where you have to have options in your custom resource to configure everything you may wish to change about an installation. Istio worked around this by offering the IstioOperator API, which allows configuration of installation options. This resource is used by both the in-cluster operator and istioctl install, so there is a trivial migration path for operator users.

Three years ago — around the time of Istio 1.12 — we updated our documentation to say that use of the operator for new Istio installations is discouraged, and that users should use istioctl or Helm to install Istio.

Having three different installation methods has caused confusion, and in order to provide the best experience for people using Helm or istioctl - over 90% of our install base - we have decided to formally deprecate the in-cluster operator in Istio 1.23.

On this day in 2017, Google and IBM announced the launch of the Istio service mesh. Istio is an open technology that enables developers to seamlessly connect, manage, and secure networks of different services — regardless of platform, source, or vendor. We can hardly believe that Istio turns seven today! To celebrate the project’s 7th birthday, we wanted to highlight Istio’s momentum and its exciting future.

Rapid adoption among users

Istio, the most widely adopted service mesh project in the world, has been gathering significant momentum since its inception in 2017. Last year Istio joined Kubernetes, Prometheus, and other stalwarts of the cloud native ecosystem with its CNCF graduation. End users range from digital native startups to the world’s largest financial institutions and telcos, with case studies from companies including eBay, T-Mobile, Airbnb, Splunk, FICO, T-Mobile, Salesforce, and many others.

Istio’s control plane and sidecar are the #3 and #4 most downloaded images on Docker Hub, each with over 10 billion downloads.

We have over 35,000 GitHub stars on Istio’s main repository, with continuing growth. Thank you everyone who starred the istio/istio repo.

We asked a few of our users for their thoughts on the occasion of Istio’s 7th birthday:

Amazing diversity of contributors and vendors

Over the past year, our community has observed tremendous growth in terms of both the number of contributing companies and the number of contributors. Recall that Istio had 500 contributors when it turned three years old? We have had over 1,700 contributors in the past year!

With Microsoft’s Open Service Mesh team joining the Istio community, we added Azure to the list of clouds and enterprise Kubernetes vendors providing Istio-compatible solutions, including Google Cloud, Red Hat OpenShift, VMware Tanzu, Huawei Cloud, DaoCloud, Oracle Cloud, Tencent Cloud, Akamai Cloud and Alibaba Cloud. We are also delighted to see the Amazon Web Services team publish the EKS Blueprint for Istio due to high demand from users wanting to run Istio on AWS.

Specialist network software providers are also driving Istio forward, with Solo.io, Tetrate and F5 Networks all offering enterprise Istio solutions that will run in any environment.

Below are the top contributing companies for the past year, with Solo.io, Google, and DaoCloud taking the top three places. While most of these companies are Istio vendors, Salesforce and Ericsson are end users, running Istio in production!

Here are some thoughts from our community leaders:

Continuous technical innovation

We are firm believers that diversity drives innovation. What amazes us most is the continuous innovation from the Istio community, from making upgrades easier, to adopting Kubernetes Gateway API, to adding the new sidecar-less ambient data plane mode, to making Istio easy to use and as transparent as possible.

Istio’s ambient mode was introduced in September 2022, designed for simplified operations, broader application compatibility, and reduced infrastructure cost. Ambient mode introduces lightweight, shared Layer 4 (L4) node proxies and optional Layer 7 (L7) proxies, removing the need for traditional sidecar proxies from the data plane. The core innovation behind ambient mode is that it slices the L4 and L7 processing into two distinct layers. This layered approach allows you to adopt Istio incrementally, enabling a smooth transition from no mesh, to a secure overlay (L4), to optional full L7 processing — on a per-namespace basis, as needed, across your fleet.

As part of the Istio 1.22 release, ambient mode has reached beta and you can run Istio without sidecars in production with precautions.

Here are some thoughts and well-wishes from our contributors and users:

Learn more about Istio

If you are new to Istio, here are a few resources to help you learn more:

• Check out the project website and GitHub repository.
• Read the documentation.
• Join the community Slack.
• Follow the project on Twitter and LinkedIn.
• Attend the user community meetings.
• Join the working group meeting.
• Become an Istio contributor and developer by submitting a membership request, after you have a pull request merged.

If you are already part of the Istio community, please wish the Istio project a happy 7th birthday, and share your thoughts about the project on social media. Thank you for your help and support!

Ambient mode was announced in September 2022. Since then, our community has put in 20 months of hard work and collaboration, with contributions from Solo.io, Google, Microsoft, Intel, Aviatrix, Huawei, IBM, Red Hat, and many others. Beta status in 1.22 indicates the features of ambient mode are now ready for production workloads, with appropriate precautions. This is a huge milestone for Istio, bringing both Layer 4 and Layer 7 mesh features to production readiness without sidecars.

Why ambient mode?

In listening to feedback from Istio users, we observed a growing demand for mesh capabilities for applications — but heard that many of you found the resource overhead and operational complexity of sidecars hard to overcome. Challenges that sidecar users shared with us include how Istio can break applications after sidecars are added, the large consumption of CPU and memory by sidecars, and the inconvenience of the requirement to restart application pods with every new proxy release.

As a community, we designed ambient mode to tackle these problems, alleviating the previous barriers of complexity faced by users looking to implement service mesh. The new feature set was named ‘ambient mode’ as it was designed to be transparent to your application, ensuring no additional configuration was required to adopt it, and required no restarting of applications by users.

In ambient mode it is trivial to add or remove applications from the mesh. You can now simply label a namespace, and all applications in that namespace are added to the mesh. This immediately secures all traffic with mTLS, all without sidecars or the need to restart applications.

Refer to the Introducing Ambient Mesh blog for more information on why we built ambient mode.

How does ambient mode make adoption easier?

Istio’s ambient mode introduces lightweight, shared Layer 4 (L4) node proxies and optional Layer 7 (L7) proxies, removing the need for traditional sidecar proxies from the data plane. The core innovation behind ambient mode is that it slices the L4 and L7 processing into two distinct layers. This layered approach allows you to adopt Istio incrementally, enabling a smooth transition from no mesh, to a secure overlay (L4), to optional full L7 processing — on a per-namespace basis, as needed, across your fleet.

Ambient mode works without any modification required to your existing Kubernetes deployments. You can label a namespace to add all of its workloads to the mesh, or opt-in certain deployments as needed. By utilizing ambient mode, users bypass some of the previously restrictive elements of the sidecar model. Server-send-first protocols now work, most reserved ports are now available, and the ability for containers to bypass the sidecar — either maliciously or not — is eliminated.

The lightweight shared L4 node proxy is called the ztunnel (zero-trust tunnel). Ztunnel drastically reduces the overhead of running a mesh by removing the need to potentially over-provision memory and CPU within a cluster to handle expected loads. In some use cases, the savings can exceed 90% or more, while still providing zero-trust security using mutual TLS with cryptographic identity, simple L4 authorization policies, and telemetry.

The L7 proxies are called waypoints. Waypoints process L7 functions such as traffic routing, rich authorization policy enforcement, and enterprise-grade resilience. Waypoints run outside of your application deployments and can scale independently based on your needs, which could be for the entire namespace or for multiple services within a namespace. Compared with sidecars, you don’t need one waypoint per application pod, and you can scale your waypoint effectively based on its scope, thus saving significant amounts of CPU and memory in most cases.

The separation between the L4 secure overlay layer and L7 processing layer allows incremental adoption of the ambient mode data plane, in contrast to the earlier binary “all-in” injection of sidecars. Users can start with the secure L4 overlay, which offers a majority of features that people deploy Istio for (mTLS, authorization policy, and telemetry). Complex L7 handling such as retries, traffic splitting, load balancing, and observability collection can then be enabled on a case-by-case basis.

What is in the scope of the Beta?

We recommend you explore the following Beta functions of ambient mode in production with appropriate precautions, after validating them in test environments:

• Installing Istio with support for ambient mode.
• Adding your workloads to the mesh to gain mutual TLS with cryptographic identity, L4 authorization policies, and telemetry.
• Configuring waypoints to use L7 functions such as traffic shifting, request routing, and rich authorization policy enforcement.
• Connecting the Istio ingress gateway to workloads in ambient mode, supporting all existing Istio APIs.
• Using istioctl to operate waypoints, and troubleshoot ztunnel & waypoints.

Alpha features

Many other features we want to include in ambient mode have been implemented, but remain in Alpha status in this release. Please help test them, so they can be promoted to Beta in 1.23 or later:

• Multi-cluster installations
• DNS proxying
• Interoperability with sidecars
• IPv6/Dual stack
• SOCKS5 support (for outbound)
• Istio’s classic APIs (VirtualService and DestinationRule)

Roadmap

We have a number of features which are not yet implemented in ambient mode, but are planned for upcoming releases:

• Controlled egress traffic
• Multi-network support
• Improve status messages on resources to help troubleshoot and understand the mesh
• VM support

What about sidecars?

Sidecars are not going away, and remain first-class citizens in Istio. You can continue to use sidecars, and they will remain fully supported. For any feature outside of the Alpha or Beta scope for ambient mode, you should consider using the sidecar mode until the feature is added to ambient mode. Some use cases, such as traffic shifting based on source labels, will continue to be best implemented using the sidecar mode. While we believe most use cases will be best served with a mesh in ambient mode, the Istio project remains committed to ongoing sidecar mode support.

Try ambient mode today

With the 1.22 release of Istio and the Beta release of ambient mode, it is now easier than ever to try out Istio on your own workloads. Follow the getting started guide to explore ambient mode, or read our new user guides to learn how to incrementally adopt ambient for mutual TLS & L4 authorization policy, traffic management, rich L7 authorization policy, and more. You can engage with the developers in the #ambient channel on the Istio Slack, or use the discussion forum on GitHub for any questions you may have.

Most of the features powered by these APIs have been considered stable for some time, but the API version has remained at v1beta1. As a reflection of the stability, adoption, and value of these resources, the Istio community has decided to promote these APIs to v1 in Istio 1.22.

In Istio 1.22 we are happy to announce that a concerted effort has been made to graduate the below APIs to v1:

• Destination Rule
• Gateway
• Service Entry
• Sidecar
• Virtual Service
• Workload Entry
• Workload Group
• Telemetry API*
• Peer Authentication

Feature stability and API versions

Declarative APIs, such as those used by Kubernetes and Istio, decouple the description of a resource from the implementation that acts on it.

Istio’s feature phase definitions describe how a stable feature — one that is deemed ready for production use at any scale, and comes with a formal deprecation policy — should be matched with a v1 API. We are now making good on that promise, with our API versions matching our feature stability for both features that have been stable for some time, and those which are being newly designated as stable in this release.

Although there are currently no plans to discontinue support for the previous v1beta1 and v1alpha1 API versions, users are encouraged to manually transition to utilizing the v1 APIs by updating their existing YAML files.

Telemetry API

The v1 Telemetry API is the only API that was promoted that had changes from its previous API version. The following v1alpha1 features weren’t promoted to v1:

• metrics.reportingInterval

  • Reporting interval allows configuration of the time between calls out to for metrics reporting. This currently only supports TCP metrics but we may use this for long duration HTTP streams in the future.

    At this time, Istio lacks usage data to support the need for this feature.

• accessLogging.filter

  • If specified, this filter will be used to select specific requests/connections for logging.

    This feature is based on a relatively new feature in Envoy, and Istio needs to further develop the use case and implementation before graduating it to v1.

• tracing.useRequestIdForTraceSampling

  • This value is true by default. The format of this Request ID is specific to Envoy, and if the Request ID generated by the proxy that receives user traffic first is not specific to Envoy, Envoy will break the trace because it cannot interpret the Request ID. By setting this value to false, we can prevent Envoy from sampling based on the Request ID.

    There is not a strong use case for making this configurable through the Telemetry API.

Please share any feedback on these fields by creating issues on GitHub.

Overview of Istio CRDs

This is the full list of supported API versions:

Istio can also be configured using the Kubernetes Gateway API.

Using the v1 Istio APIs

There are some APIs in Istio that are still under active development and are subject to potential changes between releases. For instance, the Envoy Filter, Proxy Config and Wasm Plugin APIs.

Furthermore, Istio maintains a strictly identical schema across all versions of an API due to limitations in CRD versioning. Therefore, even though there is a v1 Telemetry API, the three v1alpha1 fields mentioned above can still be utilized when declaring a v1 Telemetry API resource.

For risk-averse environments, we have added a stable validation policy, a validating admission policy which can ensure that only v1 APIs and fields are used with Istio APIs.

In new environments, selecting the stable validation policy upon installing Istio will guarantee that all future Custom Resources created or updated are v1 and contain only v1 features.

If the policy is deployed into an existing Istio installation that has Custom Resources that do not comply with it, the only allowed action is to delete the resource or remove the usage of the offending fields.

To install Istio with the stable validation policy:

$ helm install istio-base -n istio-system --set experimental.stableValidationPolicy=true

To set a specific revision when installing Istio with the policy:

$ helm install istio-base -n istio-system --set experimental.stableValidationPolicy=true -set revision=x

This feature is compatible with Kubernetes 1.30 and higher. The validations are created using CEL expressions, and users can modify the validations for their specific needs.

Summary

The Istio project is committed to delivering stable APIs and features essential for the successful operation of your service mesh. We would love to receive your feedback to help guide us in making the right decisions as we continue to refine relevant use cases and stability blockers for our features. Please share your feedback by creating issues, posting in the relevant Istio Slack channel, or by joining us in our weekly working group meeting.

What is the Gateway API?

The Gateway API is a collection of APIs that are part of Kubernetes, focusing on traffic routing and management. The APIs are inspired by, and serve many of the same roles as, Kubernetes’ Ingress and Istio’s VirtualService and Gateway APIs.

These APIs have been under development both in Istio, as well as with broad collaboration, since 2020, and have come a long way since then. While the API initially targeted only serving ingress use cases (which went GA last year), we had always envisioned allowing the same APIs to be used for traffic within a cluster as well.

With this release, that vision is made a reality: Istio users can use the same routing API for all of their traffic!

Getting started

Throughout the Istio documentation, all of our examples have been updated to show how to use the Gateway API, so explore some of the tasks to gain a deeper understanding.

Using Gateway API for service mesh should feel familiar both to users already using Gateway API for ingress, and users using VirtualService for service mesh today.

• Compared to Gateway API for ingress, routes target a Service instead of a Gateway.
• Compared to VirtualService, where routes associate with a set of hosts, routes target a Service.

Here is a simple example, which demonstrates routing requests to two different versions of a Service based on the request header:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: reviews
spec:
  parentRefs:
  - group: ""
    kind: Service
    name: reviews
    port: 9080
  rules:
  - matches:
    - headers:
      - name: my-favorite-service-mesh
        value: istio
    filters:
    - type: RequestHeaderModifier
      requestHeaderModifier:
      add:
        - name: hello
          value: world
    backendRefs:
    - name: reviews-v2
      port: 9080
  - backendRefs:
    - name: reviews-v1
      port: 9080

Breaking this down, we have a few parts:

• First, we identify what routes we should match. By attaching our route to the reviews Service, we will apply this routing configuration to all requests that were originally targeting reviews.
• Next, matches configures criteria for selecting which traffic this route should handle.
• Optionally, we can modify the request. Here, we add a header.
• Finally, we select a destination for the request. In this example, we are picking between two versions of our application.

For more details, see Istio’s traffic routing internals and Gateway API’s Service documentation.

Which API should I use?

With overlapping responsibilities (and names!), picking which APIs to use can be a bit confusing.

Here is the breakdown:

You may wonder, given the above, why the Istio APIs were promoted to v1 concurrently? This was part of an effort to accurate categorize the stability of the APIs. While we view Gateway API as the future (and present!) of traffic routing APIs, our existing APIs are here to stay for the long run, with full compatibility. This mirrors Kubernetes’ approach with Ingress, which was promoted to v1 while directing future work towards the Gateway API.

Community

This stability graduation represents the culmination of countless hours of work and collaboration across the project. It is incredible to look at the list of organizations involved in the API and consider back at how far we have come.

A special thanks goes out to my co-leads on the effort: Flynn, Keith Mattix, and Mike Morris, as well as the countless others involved.

Interested in getting involved, or even just providing feedback? Check out Istio’s community page or the Gateway API contributing guide!

The Istio Steering Committee is excited to unveil Izzy Dolphin, the Istio Indo-Pacific Bottlenose, who today dives into the family of “Phippy and Friends”.

Istio stands on the shoulders of several other CNCF projects, including Kubernetes, Envoy, Prometheus, and Helm. Izzy is proud to join Phippy, Hazel, and Captain Kube’s gang, taking cloud native to the masses.

Izzy not only represents the hard work and imagination of Istio’s maintainers from diverse companies, but will help us illustrate the concepts of service mesh and Istio’s new ambient mode in an easy manner. Keep tuned, as we build out our illustrated guides where Izzy will demystify Istio and service mesh in terms a child could understand! Next time you’re breaking down these concepts for people who don’t share your background knowledge, how about using Izzy?

Istio was initially developed by Google and IBM and built on the Envoy project from Lyft. The project now has maintainers from more than 16 companies, including many of the largest networking vendors and cloud organizations worldwide. Istio provides zero-trust networking, policy enforcement, traffic management, load balancing, and monitoring without requiring applications to be rewritten.

Over the years Istio has made substantial strides in simplifying the complex problem of cloud native networking. We understand that these concepts remain complicated for many, and that is why we are proud to join Phippy’s mission to talk about tech in an accessible, straight-forward manner. We would like to open the doors of service mesh technology to more folks than ever before through Izzy and enable you to join #teamcloudnative!

Every year, the leaders in the Istio project estimate the proportion of the hundreds of companies that have contributed to Istio in the past year, and uses that metric to proportionally allocate nine Contribution Seats on our Steering Committee.

Then, four Community Seats are voted for by our project members, with candidates being from companies that did not receive Contribution Seats.

We are pleased to share the result of this year’s calculation, and changes to our Community Seat holders as a result.

Contribution seats

The calculation for the 2024-2025 term brings the most diverse set of company representation ever in the Contribution Seats, with five companies¹ represented:

The full allocation can be seen in our formula spreadsheet.

Community seats

As a result of this year’s calculation, two of our Community Seat holders move to Contribution Seats. This creates two extra vacancies, which are allocated to the runners-up of our last election².

We are pleased to welcome our two newest Community Seat holders, Mitch Connors from Aviatrix and Keith Mattix from Microsoft. Both are highly active maintainers and leaders in the project, and we are delighted to have them join the Steering Committee.

Proposed changes to election timing

Our charter currently allocates Contribution Seats in February and holds the Community Seat election in July. We previously anticipated a situation where people would change seat types mid-term, and this has now come to pass.

We will therefore be voting on a change to our Charter which will move our Community Seat elections to February, to be held immediately after the allocation of Contribution Seats. It is our intention that the next annual election be held in February 2025.

The full group

Following these changes, we now have representation from nine companies:

Our sincerest thanks to Ameer Abbas, April Kyle Nassi, Cale Rath and Chaomeng Zhang, whose terms have come to an end.

The Steering Committee wishes to thank its members, old and new, and looks forward to continue to grow and improve Istio as a successful and sustainable open source project. We encourage everyone to get involved in the Istio community by contributing, voting, and helping us shape the future of cloud native networking.

Our alpha was focused on proving out the value of the ambient data plane mode under limited configurations and environments. However, the conditions were quite limited. Ambient mode relies on transparently redirecting traffic between workload pods and ztunnel, and the initial mechanism we used to do that conflicted with several categories of 3rd-party Container Networking Interface (CNI) implementations. Through GitHub issues and Slack discussions, we heard our users wanted to be able to use ambient mode in minikube and Docker Desktop, with CNI implementations like Cilium and Calico, and on services that ship in-house CNI implementations like OpenShift and Amazon EKS. Getting broad support for Kubernetes anywhere has become the No. 1 requirement for ambient mesh moving to beta — people have come to expect Istio to work on any Kubernetes platform and with any CNI implementation. After all, ambient wouldn’t be ambient without being all around you!

At Solo, we’ve been integrating ambient mode into our Gloo Mesh product, and came up with an innovative solution to this problem. We decided to upstream our changes in late 2023 to help ambient reach beta faster, so more users can operate ambient in Istio 1.21 or newer, and enjoy the benefits of ambient sidecar-less mesh in their platforms regardless of their existing or preferred CNI implementation.

How did we get here?

Service meshes and CNIs: it’s complicated

Istio is a service mesh, and all service meshes by strict definition are not CNI implementations - service meshes require a spec-compliant, primary CNI implementation to be present in every Kubernetes cluster, and rest on top of that.

This primary CNI implementation may be provided by your cloud provider (AKS, GKE, and EKS all ship their own), or by third-party CNI implementations like Calico and Cilium. Some service meshes may also ship bundled with their own primary CNI implementation, which they explicitly require to function.

Basically, before you can do things like secure pod traffic with mTLS and apply high-level authentication and authorization policy at the service mesh layer, you must have a functional Kubernetes cluster with a functional CNI implementation, to make sure the basic networking pathways are set up so that packets can get from one pod to another (and from one node to another) in your cluster.

Though some service meshes may also ship and require their own in-house primary CNI implementation, and it is sometimes possible to run two primary CNI implementations in parallel within the same cluster (for instance, one shipped by the cloud provider, and a 3rd-party implementation), in practice this introduces a whole host of compatibility issues, strange behaviors, reduced feature sets, and some incompatibilities due to the wildly varying mechanisms each CNI implementation might employ internally.

To avoid this, the Istio project has chosen not to ship or require our own primary CNI implementation, or even require a “preferred” CNI implementation - instead choosing to support CNI chaining with the widest possible ecosystem of CNI implementations, and ensuring maximum compatibility with managed offerings, cross-vendor support, and composability with the broader CNCF ecosystem.

Traffic redirection in ambient alpha

The istio-cni component is an optional component in the sidecar data plane mode, commonly used to remove the requirement for the NET_ADMIN and NET_RAW capabilities for users deploying pods into the mesh. istio-cni is a required component in the ambient data plane mode. The istio-cni component is not a primary CNI implementation, it is a node agent that extends whatever primary CNI implementation is already present in the cluster.

Whenever pods are added to an ambient mesh, the istio-cni component configures traffic redirection for all incoming and outgoing traffic between the pods and the ztunnel running on the pod’s node, via the node-level network namespace. The key difference between the sidecar mechanism and the ambient alpha mechanism is that in the latter, pod traffic was redirected out of the pod network namespace, and into the co-located ztunnel pod network namespace - necessarily passing through the host network namespace on the way, which is where the bulk of the traffic redirection rules to achieve this were implemented.

As we tested more broadly in multiple real-world Kubernetes environments, which have their own default CNI, it became clear that capturing and redirecting pod traffic in the host network namespace, as we were during alpha development, was not going to meet our requirements. Achieving our goals in a generic manner across these diverse environments was simply not feasible with this approach.

The fundamental problem with redirecting traffic in the host network namespace is that this is precisely the same spot where the cluster’s primary CNI implementation must configure traffic routing/networking rules. This created inevitable conflicts, most critically:

• The primary CNI implementation’s basic host-level networking configuration could interfere with the host-level ambient networking configuration from Istio’s CNI extension, causing traffic disruption and other conflicts.
• If users deployed a network policy to be enforced by the primary CNI implementation, the network policy might not be enforced when the Istio CNI extension is deployed (depending on how the primary CNI implementation enforces NetworkPolicy)

While we could design around this on a case-by-case basis for some primary CNI implementations, we could not sustainably approach universal CNI support. We considered eBPF, but realized any eBPF implementation would have the same basic problem, as there is no standardized way to safely chain/extend arbitrary eBPF programs at this time, and we would still potentially have a hard time supporting non-eBPF CNIs with this approach.

Addressing the challenges

A new solution was necessary - doing redirection of any sort in the node’s network namespace would create unavoidable conflicts, unless we compromised our compatibility requirements.

In sidecar mode, it is trivial to configure traffic redirection between the sidecar and application pod, as both operate within the pod’s network namespace. This led to a light-bulb moment: why not mimic sidecars, and configure the redirection in the application pod’s network namespace?

While this sounds like a “simple” thought, how would this even be possible? A critical requirement of ambient is that ztunnel must run outside application pods, in the Istio system namespace. After some research, we discovered a Linux process running in one network namespace could create and own listening sockets within another network namespace. This is a basic capability of the Linux socket API. However, to make this work operationally and cover all pod lifecycle scenarios, we had to make architectural changes to the ztunnel as well as to the istio-cni node agent.

After prototyping and sufficiently validating that this novel approach does work for all the Kubernetes platforms we have access to, we built confidence in the work and decided to contribute to upstream this new traffic redirection model, an in-Pod traffic redirection mechanism between workload pods and the ztunnel node proxy component that has been built from the ground up to be highly compatible with all major cloud providers and CNIs.

The key innovation is to deliver the pod’s network namespace to the co-located ztunnel so that ztunnel can start its redirection sockets inside the pod’s network namespace, while still running outside the pod. With this approach, the traffic redirection between ztunnel and application pods happens in a way that’s very similar to sidecars and application pods today and is strictly invisible to any Kubernetes primary CNI operating in the node network namespace. Network policy can continue to be enforced and managed by any Kubernetes primary CNI, regardless of whether the CNI uses eBPF or iptables, without any conflict.

Technical deep dive of in-Pod traffic redirection

First, let’s go over the basics of how a packet travels between pods in Kubernetes.

Linux, Kubernetes, and CNI - what’s a network namespace, and why does it matter?

In Linux, a container is one or more Linux processes running within isolated Linux namespaces. A Linux namespace is simply a kernel flag that controls what processes running within that namespace are able to see. For instance, if you create a new Linux network namespace via the ip netns add my-linux-netns command and run a process inside it, that process can only see the networking rules created within that network namespace. It can not see any network rules created outside of it - even though everything running on that machine is still sharing one Linux networking stack.

Linux namespaces are conceptually a lot like Kubernetes namespaces - logical labels that organize and isolate different active processes, and allow you to create rules about what things within a given namespace can see and what rules are applied to them - they simply operate at a much lower level.

When a process running within a network namespace creates a TCP packet outward bound for something else, the packet must be processed by any local rules within the local network namespace first, then leave the local network namespace, passing into another one.

For example, in plain Kubernetes without any mesh installed, a pod might create a packet and send it to another pod, and the packet might (depending on how networking was set up):

• Be processed by any rules within the source pod’s network namespace.
• Leave the source pod network namespace, and bubble up into the node’s network namespace where it is processed by any rules in that namespace.
• From there, finally be redirected into the target pod’s network namespace (and processed by any rules there).

In Kubernetes, the Container Runtime Interface (CRI) is responsible for talking to the Linux kernel, creating network namespaces for new pods, and starting processes within them. The CRI then invokes the Container Networking Interface (CNI), which is responsible for wiring up the networking rules in the various Linux network namespaces, so that packets leaving and entering the new pod can get where they’re supposed to go. It doesn’t matter much to Kubernetes or the container runtime what topology or mechanism the CNI uses to accomplish this - as long as packets get where they’re supposed to be, Kubernetes works and everyone is happy.

Why did we drop the previous model?

In Istio ambient mesh, every node has a minimum of two containers running as Kubernetes DaemonSets:

• An efficient ztunnel which handles mesh traffic proxying duties, and L4 policy enforcement.
• A istio-cni node agent that handles adding new and existing pods into the ambient mesh.

In the previous ambient mesh implementation, this is how application pod is added to the ambient mesh:

• The istio-cni node agent detects an existing or newly-started Kubernetes pod with its namespace labeled with istio.io/dataplane-mode=ambient, indicating that it should be included in the ambient mesh.
• The istio-cni node agent then establishes network redirection rules in the host network namespace, such that packets entering or leaving the application pod would be intercepted and redirected to that node’s ztunnel on the relevant proxy ports (15008, 15006, or 15001).

This means that for a packet created by a pod in the ambient mesh, that packet would leave that source pod, enter the node’s host network namespace, and then ideally would be intercepted and redirected to that node’s ztunnel (running in its own network namespace) for proxying to the destination pod, with the return trip being similar.

This model worked well enough as a placeholder for the initial ambient mesh alpha implementation, but as mentioned, it has a fundamental problem - there are many CNI implementations, and in Linux there are many fundamentally different and incompatible ways in which you can configure how packets get from one network namespace to another. You can use tunnels, overlay networks, go through the host network namespace, or bypass it. You can go through the Linux user space networking stack, or you can skip it and shuttle packets back and forth in the kernel space stack, etc. For every possible approach, there’s probably a CNI implementation out there that makes use of it.

Which meant that with the previous redirection approach, there were a lot of CNI implementations ambient simply wouldn’t work with. Given its reliance on host network namespace packet redirection - any CNI that didn’t route packets thru the host network namespace would need a different redirection implementation. And even for CNIs that did do this, we would have unavoidable and potentially unresolvable problems with conflicting host-level rules. Do we intercept before the CNI, or after? Will some CNIs break if we do one, or the other, and they aren’t expecting that? Where and when is NetworkPolicy enforced, since NetworkPolicy must be enforced in the host network namespace? Do we need lots of code to special-case every popular CNI?

Istio ambient traffic redirection: the new model

In the new ambient model, this is how application pod is added to the ambient mesh:

• The istio-cni node agent detects a Kubernetes pod (existing or newly-started) with its namespace labeled with istio.io/dataplane-mode=ambient, indicating that it should be included in the ambient mesh.
  • If a new pod is started that should be added to the ambient mesh, a CNI plugin (as installed and managed by the istio-cni agent) is triggered by the CRI. This plugin is used to push a new pod event to the node’s istio-cni agent, and block pod startup until the agent successfully configures redirection. Since CNI plugins are invoked by the CRI as early as possible in the Kubernetes pod creation process, this ensures that we can establish traffic redirection early enough to prevent traffic escaping during startup, without relying on things like init containers.
  • If an already-running pod becomes added to the ambient mesh, a new pod event is triggered. The istio-cni node agent’s Kubernetes API watcher detects this, and redirection is configured in the same manner.
• The istio-cni node agent enters the pod’s network namespace and establishes network redirection rules inside the pod network namespace, such that packets entering and leaving the pod are intercepted and transparently redirected to the node-local ztunnel proxy instance listening on well-known ports (15008, 15006, 15001).
• The istio-cni node agent then informs the node ztunnel over a Unix domain socket that it should establish local proxy listening ports inside the pod’s network namespace, (on 15008, 15006, and 15001), and provides ztunnel with a low-level Linux file descriptor representing the pod’s network namespace.
  • While typically sockets are created within a Linux network namespace by the process actually running inside that network namespace, it is perfectly possible to leverage Linux’s low-level socket API to allow a process running in one network namespace to create listening sockets in another network namespace, assuming the target network namespace is known at creation time.
• The node-local ztunnel internally spins up a new proxy instance and listen port set, dedicated to the newly-added pod.
• Once the in-Pod redirect rules are in place and the ztunnel has established the listen ports, the pod is added in the mesh and traffic begins flowing thru the node-local ztunnel, as before.

Here’s a basic diagram showing the flow of application pod being added to the ambient mesh:

Once the pod is successfully added to the ambient mesh, traffic to and from pods in the mesh will be fully encrypted with mTLS by default, as always with Istio.

Traffic will now enter and leave the pod network namespace as encrypted traffic - it will look like every pod in the ambient mesh has the ability to enforce mesh policy and securely encrypt traffic, even though the user application running in the pod has no awareness of either.

Here’s a diagram to illustrate how encrypted traffic flows between pods in the ambient mesh in the new model:

And, as before, unencrypted plaintext traffic from outside the mesh can still be handled and policy enforced, for use cases where that is necessary:

The new ambient traffic redirection: what this gets us

The end result of the new ambient capture model is that all traffic capture and redirection happens inside the pod’s network namespace. To the node, the CNI, and everything else, it looks like there is a sidecar proxy inside the pod, even though there is no sidecar proxy running in the pod at all. Remember that the job of CNI implementations is to get packets to and from the pod. By design and by the CNI spec, they do not care what happens to packets after that point.

This approach automatically eliminates conflicts with a wide range of CNI and NetworkPolicy implementations, and drastically improves Istio ambient mesh compatibility with all major managed Kubernetes offerings across all major CNIs.

Wrapping up

Thanks to significant amounts of effort from our lovely community in testing the change with a large variety of Kubernetes platforms and CNIs, and many rounds of reviews from Istio maintainers, we are glad to announce that the ztunnel and istio-cni PRs implementing this feature merged to Istio 1.21 and are enabled by default for ambient, so Istio users can start running ambient mesh on any Kubernetes platforms with any CNIs in Istio 1.21 or newer. We’ve tested this with GKE, AKS, and EKS and all the CNI implementations they offer, as well as with 3rd-party CNIs like Calico and Cilium, as well as platforms like OpenShift, with solid results.

We are extremely excited that we are able to move Istio ambient mesh forward to run everywhere with this innovative in-Pod traffic redirection approach between ztunnel and users’ application pods. With this top technical hurdle to ambient beta resolved, we can’t wait to work with the rest of the Istio community to get ambient mesh to beta soon! To learn more about ambient mesh’s beta progress, join us in the #ambient and #ambient-dev channel in Istio’s slack, or attend the weekly ambient contributor meeting on Wednesdays, or check out the ambient mesh beta project board and help us fix something!

• Come to the Istio Day co-located event.

• The following KubeCon sessions will be based on Istio, add them to your schedule:

  • Keynote: Platform Building Blocks: How to Build ML Infrastructure with CNCF Projects
  • What Not Do When You’re Updating Istio in a Critical Environment?
  • Comparing Sidecar-Less Service Mesh from Cilium and Istio
  • Next-Level Security: Implementing mTLS in Istio Multi-Cluster Environments Using SPIRE
  • Scaling Service Mesh: Self Service Beyond 300 Clusters
  • Lightning Talk: Help! My Envoy Sidecar Is Consuming 8GBs of Memory!
  • Poster Session: Kubernetes in the Confidential Computing Marvels: Unlocking SMPC Across Multi-Cloud Clusters
  • Poster Session: Serve CAKES for Your Developers: Introducing the Cloud Native CAKES Stack for Zero Trust!
  • Tutorial: Configuring Your Service Mesh with Gateway API
  • Product Market Misfit: Adventures in User Empathy

• Attend the co-located Observability Day session related to Istio: A Practical Guide on How to Monitor and Compare Service Mesh Infrastructure Costs

• Attend the co-located Multi-TenancyCon session related to Istio: Lightning Talk: Establishing Trust Between Microservices in a Multi Tenant Cloud

• Attend the co-located Platform Engineering Day session related to Istio: Building an AI-Powered, Paved Road Platform with Cloud-Native OSS

• Attend the co-located AppDeveloperCon session related to Istio: Navigating the Complexities of Service to Service Invocations: Deep and Brief Dive Into Causality

• Attend the co-located Cloud Native AI Day session related to Istio: Platform Building Blocks: How to Build ML Infrastructure with CNCF Projects

• Attend the Istio Maintainers’ Track session

• Come and have a chat at the Istio kiosk in the Project Pavilion throughout the event.

See you soon in Paris!

The use-case of configuring an egress gateway to handle arbitrary wildcard domains had been included in the official Istio docs up until version 1.13, but was subsequently removed because the documented solution was not officially supported or recommended and was subject to breakage in future versions of Istio. Nevertheless, the old solution was still usable with Istio versions before 1.20. Istio 1.20, however, dropped some Envoy functionality that was required for the approach to work.

This post attempts to describe how we resolved the issue and filled the gap with a similar approach using Istio version-independent components and Envoy features, but without the need for a separate Nginx SNI proxy. Our approach allows users of the old solution to seamlessly migrate configurations before their systems face the breaking changes in Istio 1.20.

Problem to solve

The currently documented egress gateway use-cases rely on the fact that the target of the traffic (the hostname) is statically configured in a VirtualService, telling Envoy in the egress gateway pod where to TCP proxy the matching outbound connections. You can use multiple, and even wildcard, DNS names to match the routing criteria, but you are not able to route the traffic to the exact location specified in the application request. For example you can match traffic for targets *.wikipedia.org, but you then need to forward the traffic to a single final target, e.g., en.wikipedia.org. If there is another service, e.g., anyservice.wikipedia.org, that is not hosted by the same server(s) as en.wikipedia.org, the traffic to that host will fail. This is because, even though the target hostname in the TLS handshake of the HTTP payload contains anyservice.wikipedia.org, the en.wikipedia.org servers will not be able to serve the request.

The solution to this problem at a high level is to inspect the original server name (SNI extension) in the application TLS handshake (which is sent in plain-text, so no TLS termination or other man-in-the-middle operation is needed) in every new gateway connection and use it as the target to dynamically TCP proxy the traffic leaving the gateway.

When restricting egress traffic via egress gateways, we need to lock down the egress gateways so that they can only be used by clients within the mesh. This is achieved by enforcing ISTIO_MUTUAL (mTLS peer authentication) between the application sidecar and the gateway. That means that there will be two layers of TLS on the application L7 payload. One that is the application originated end-to-end TLS session terminated by the final remote target, and another one that is the Istio mTLS session.

Another thing to keep in mind is that in order to mitigate any potential application pod corruption, the application sidecar and the gateway should both perform hostname list checks. This way, any compromised application pod will still only be able to access the allowed targets and nothing more.

Low-level Envoy programming to the rescue

Recent Envoy releases include a dynamic TCP forward proxy solution that uses the SNI header on a per- connection basis to determine the target of an application request. While an Istio VirtualService cannot configure a target like this, we are able to use EnvoyFilters to alter the Istio generated routing instructions so that the SNI header is used to determine the target.

To make it all work, we start by configuring a custom egress gateway to listen for the outbound traffic. Using a DestinationRule and a VirtualService we instruct the application sidecars to route the traffic (for a selected list of hostnames) to that gateway, using Istio mTLS. On the gateway pod side we build the SNI forwarder with the EnvoyFilters, mentioned above, introducing internal Envoy listeners and clusters to make it all work. Finally, we patch the internal destination of the gateway-implemented TCP proxy to the internal SNI forwarder.

The end-to-end request flow is shown in the following diagram:

This diagram shows an egress HTTPS request to en.wikipedia.org using SNI as a routing key.

• Application container

  Application originates HTTP/TLS connection towards the final destination. Puts destination’s hostname into the SNI header. This TLS session is not decrypted inside the mesh. Only SNI header is inspected (as it is in cleartext).

• Sidecar proxy

  Sidecar intercepts traffic to matching hostnames in the SNI header from the application originated TLS sessions. Based on the VirtualService, the traffic is routed to the egress gateway while wrapping original traffic into Istio mTLS as well. Outer TLS session has the gateway Service address in the SNI header.

• Mesh listener

  A dedicated listener is created in the Gateway that mutually authenticates the Istio mTLS traffic. After the outer Istio mTLS termination, it unconditionally sends the inner TLS traffic with a TCP proxy to the other (internal) listener in the same Gateway.

• SNI forwarder

  Another listener with SNI forwarder performs a new TLS header inspection for the original TLS session. If the inner SNI hostname matches the allowed domain names (including wildcards), it TCP proxies the traffic to the destination, read from the header per connection. This listener is internal to Envoy (allowing it to restart traffic processing to see the inner SNI value), so that no pods (inside or outside the mesh) can connect to it directly. This listener is 100% manually configured through EnvoyFilter.

Deploy the sample

In order to deploy the sample configuration, start by creating the istio-egress namespace and then use the following YAML to deploy an egress gateway, along with some RBAC and its Service. We use the gateway injection method to create the gateway in this example. Depending on your install method, you may want to deploy it differently (for example, using an IstioOperator CR or using Helm).

# New k8s cluster service to put egressgateway into the Service Registry,
# so application sidecars can route traffic towards it within the mesh.
apiVersion: v1
kind: Service
metadata:
  name: egressgateway
  namespace: istio-egress
spec:
  type: ClusterIP
  selector:
    istio: egressgateway
  ports:
  - port: 443
    name: tls-egress
    targetPort: 8443

---
# Gateway deployment with injection method
apiVersion: apps/v1
kind: Deployment
metadata:
  name: istio-egressgateway
  namespace: istio-egress
spec:
  selector:
    matchLabels:
      istio: egressgateway
  template:
    metadata:
      annotations:
        inject.istio.io/templates: gateway
      labels:
        istio: egressgateway
        sidecar.istio.io/inject: "true"
    spec:
      containers:
      - name: istio-proxy
        image: auto # The image will automatically update each time the pod starts.
        securityContext:
          capabilities:
            drop:
            - ALL
          runAsUser: 1337
          runAsGroup: 1337

---
# Set up roles to allow reading credentials for TLS
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: istio-egressgateway-sds
  namespace: istio-egress
rules:
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get", "watch", "list"]
- apiGroups:
  - security.openshift.io
  resourceNames:
  - anyuid
  resources:
  - securitycontextconstraints
  verbs:
  - use

---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: istio-egressgateway-sds
  namespace: istio-egress
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: istio-egressgateway-sds
subjects:
- kind: ServiceAccount
  name: default

Verify the gateway pod is up and running in the istio-egress namespace and then apply the following YAML to configure the gateway routing:

# Define a new listener that enforces Istio mTLS on inbound connections.
# This is where sidecar will route the application traffic, wrapped into
# Istio mTLS.
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: egressgateway
  namespace: istio-system
spec:
  selector:
    istio: egressgateway
  servers:
  - port:
      number: 8443
      name: tls-egress
      protocol: TLS
    hosts:
      - "*"
    tls:
      mode: ISTIO_MUTUAL

---
# VirtualService that will instruct sidecars in the mesh to route the outgoing
# traffic to the egress gateway Service if the SNI target hostname matches
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: direct-wildcard-through-egress-gateway
  namespace: istio-system
spec:
  hosts:
    - "*.wikipedia.org"
  gateways:
  - mesh
  - egressgateway
  tls:
  - match:
    - gateways:
      - mesh
      port: 443
      sniHosts:
        - "*.wikipedia.org"
    route:
    - destination:
        host: egressgateway.istio-egress.svc.cluster.local
        subset: wildcard
# Dummy routing instruction. If omitted, no reference will point to the Gateway
# definition, and istiod will optimise the whole new listener out.
  tcp:
  - match:
    - gateways:
      - egressgateway
      port: 8443
    route:
    - destination:
        host: "dummy.local"
      weight: 100

---
# Instruct sidecars to use Istio mTLS when sending traffic to the egress gateway
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: egressgateway
  namespace: istio-system
spec:
  host: egressgateway.istio-egress.svc.cluster.local
  subsets:
  - name: wildcard
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL

---
# Put the remote targets into the Service Registry
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: wildcard
  namespace: istio-system
spec:
  hosts:
    - "*.wikipedia.org"
  ports:
  - number: 443
    name: tls
    protocol: TLS

---
# Access logging for the gateway
apiVersion: telemetry.istio.io/v1alpha1
kind: Telemetry
metadata:
  name: mesh-default
  namespace: istio-system
spec:
  accessLogging:
    - providers:
      - name: envoy

---
# And finally, the configuration of the SNI forwarder,
# it's internal listener, and the patch to the original Gateway
# listener to route everything into the SNI forwarder.
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: sni-magic
  namespace: istio-system
spec:
  configPatches:
  - applyTo: CLUSTER
    match:
      context: GATEWAY
    patch:
      operation: ADD
      value:
        name: sni_cluster
        load_assignment:
          cluster_name: sni_cluster
          endpoints:
          - lb_endpoints:
            - endpoint:
                address:
                  envoy_internal_address:
                    server_listener_name: sni_listener
  - applyTo: CLUSTER
    match:
      context: GATEWAY
    patch:
      operation: ADD
      value:
        name: dynamic_forward_proxy_cluster
        lb_policy: CLUSTER_PROVIDED
        cluster_type:
          name: envoy.clusters.dynamic_forward_proxy
          typed_config:
            "@type": type.googleapis.com/envoy.extensions.clusters.dynamic_forward_proxy.v3.ClusterConfig
            dns_cache_config:
              name: dynamic_forward_proxy_cache_config
              dns_lookup_family: V4_ONLY

  - applyTo: LISTENER
    match:
      context: GATEWAY
    patch:
      operation: ADD
      value:
        name: sni_listener
        internal_listener: {}
        listener_filters:
        - name: envoy.filters.listener.tls_inspector
          typed_config:
            "@type": type.googleapis.com/envoy.extensions.filters.listener.tls_inspector.v3.TlsInspector

        filter_chains:
        - filter_chain_match:
            server_names:
            - "*.wikipedia.org"
          filters:
            - name: envoy.filters.network.sni_dynamic_forward_proxy
              typed_config:
                "@type": type.googleapis.com/envoy.extensions.filters.network.sni_dynamic_forward_proxy.v3.FilterConfig
                port_value: 443
                dns_cache_config:
                  name: dynamic_forward_proxy_cache_config
                  dns_lookup_family: V4_ONLY
            - name: envoy.tcp_proxy
              typed_config:
                "@type": type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy
                stat_prefix: tcp
                cluster: dynamic_forward_proxy_cluster
                access_log:
                - name: envoy.access_loggers.file
                  typed_config:
                    "@type": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
                    path: "/dev/stdout"
                    log_format:
                      text_format_source:
                        inline_string: '[%START_TIME%] "%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%
                          %PROTOCOL%" %RESPONSE_CODE% %RESPONSE_FLAGS% %RESPONSE_CODE_DETAILS% %CONNECTION_TERMINATION_DETAILS%
                          "%UPSTREAM_TRANSPORT_FAILURE_REASON%" %BYTES_RECEIVED% %BYTES_SENT% %DURATION%
                          %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% "%REQ(X-FORWARDED-FOR)%" "%REQ(USER-AGENT)%"
                          "%REQ(X-REQUEST-ID)%" "%REQ(:AUTHORITY)%" "%UPSTREAM_HOST%" %UPSTREAM_CLUSTER%
                          %UPSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_REMOTE_ADDRESS%
                          %REQUESTED_SERVER_NAME% %ROUTE_NAME%

                          '
  - applyTo: NETWORK_FILTER
    match:
      context: GATEWAY
      listener:
        filterChain:
          filter:
            name: "envoy.filters.network.tcp_proxy"
    patch:
      operation: MERGE
      value:
        name: envoy.tcp_proxy
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy
          stat_prefix: tcp
          cluster: sni_cluster

Check the istiod and gateway logs for any errors or warnings. If all went well, your mesh sidecars are now routing *.wikipedia.org requests to your gateway pod while the gateway pod is then forwarding them to the exact remote host specified in the application request.

Try it out

Following other Istio egress examples, we will use the sleep pod as a test source for sending requests. Assuming automatic sidecar injection is enabled in your default namespace, deploy the test app using the following command:

$ kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.29/samples/sleep/sleep.yaml

Get your sleep and gateway pods:

$ export SOURCE_POD=$(kubectl get pod -l app=sleep -o jsonpath={.items..metadata.name})
$ export GATEWAY_POD=$(kubectl get pod -n istio-egress -l istio=egressgateway -o jsonpath={.items..metadata.name})

Run the following command to confirm that you are able to connect to the wikipedia.org site:

$ kubectl exec "$SOURCE_POD" -c sleep -- sh -c 'curl -s https://en.wikipedia.org/wiki/Main_Page | grep -o "<title>.*</title>"; curl -s https://de.wikipedia.org/wiki/Wikipedia:Hauptseite | grep -o "<title>.*</title>"'
<title>Wikipedia, the free encyclopedia</title>
<title>Wikipedia – Die freie Enzyklopädie</title>

We could reach both English and German wikipedia.org subdomains, great!

Normally, in a production environment, we would block external requests that are not configured to redirect through the egress gateway, but since we didn’t do that in our test environment, let’s access another external site for comparison:

$ kubectl exec "$SOURCE_POD" -c sleep -- sh -c 'curl -s https://cloud.ibm.com/login | grep -o "<title>.*</title>"'
<title>IBM Cloud</title>

Since we have access logging turned on globally (with the Telemetry CR in the manifest), we can now inspect the logs to see how the above requests were handled by the proxies.

First, check the gateway logs:

$ kubectl logs -n istio-egress $GATEWAY_POD
[...]
[2023-11-24T13:21:52.798Z] "- - -" 0 - - - "-" 813 111152 55 - "-" "-" "-" "-" "185.15.59.224:443" dynamic_forward_proxy_cluster 172.17.5.170:48262 envoy://sni_listener/ envoy://internal_client_address/ en.wikipedia.org -
[2023-11-24T13:21:52.798Z] "- - -" 0 - - - "-" 1531 111950 55 - "-" "-" "-" "-" "envoy://sni_listener/" sni_cluster envoy://internal_client_address/ 172.17.5.170:8443 172.17.34.35:55102 outbound_.443_.wildcard_.egressgateway.istio-egress.svc.cluster.local -
[2023-11-24T13:21:53.000Z] "- - -" 0 - - - "-" 821 92848 49 - "-" "-" "-" "-" "185.15.59.224:443" dynamic_forward_proxy_cluster 172.17.5.170:48278 envoy://sni_listener/ envoy://internal_client_address/ de.wikipedia.org -
[2023-11-24T13:21:53.000Z] "- - -" 0 - - - "-" 1539 93646 50 - "-" "-" "-" "-" "envoy://sni_listener/" sni_cluster envoy://internal_client_address/ 172.17.5.170:8443 172.17.34.35:55108 outbound_.443_.wildcard_.egressgateway.istio-egress.svc.cluster.local -

There are four log entries, representing two of our three curl requests. Each pair shows how a single request flows through the envoy traffic processing pipeline. They are printed in reverse order, but we can see the 2nd and the 4th line show that the requests arrived at the gateway service and were passed through the internal sni_cluster target. The 1st and 3rd line show that the final target is determined from the inner SNI header, i.e., the target host set by the application. The request is forwarded to dynamic_forward_proxy_cluster which finally sends on the request from Envoy to the remote target.

Great, but where is the third request to IBM Cloud? Let’s check the sidecar logs:

$ kubectl logs $SOURCE_POD -c istio-proxy
[...]
[2023-11-24T13:21:52.793Z] "- - -" 0 - - - "-" 813 111152 61 - "-" "-" "-" "-" "172.17.5.170:8443" outbound|443|wildcard|egressgateway.istio-egress.svc.cluster.local 172.17.34.35:55102 208.80.153.224:443 172.17.34.35:37020 en.wikipedia.org -
[2023-11-24T13:21:52.994Z] "- - -" 0 - - - "-" 821 92848 55 - "-" "-" "-" "-" "172.17.5.170:8443" outbound|443|wildcard|egressgateway.istio-egress.svc.cluster.local 172.17.34.35:55108 208.80.153.224:443 172.17.34.35:37030 de.wikipedia.org -
[2023-11-24T13:21:55.197Z] "- - -" 0 - - - "-" 805 15199 158 - "-" "-" "-" "-" "104.102.54.251:443" PassthroughCluster 172.17.34.35:45584 104.102.54.251:443 172.17.34.35:45582 cloud.ibm.com -

As you can see, Wikipedia requests were sent through the gateway while the request to IBM Cloud went straight out from the application pod to the internet, as indicated by the PassthroughCluster log.

Conclusion

We implemented controlled routing for egress HTTPS/TLS traffic using egress gateways, supporting arbitrary and wildcard domain names. In a production environment, the example shown in this post would be extended to support HA requirements (e.g., adding zone aware gateway Deployments, etc.) and to restrict the direct external network access of your application so that the application can only access the public network through the gateway, which is limited to a predefined set of remote hostnames.

The solution scales easily. You can include multiple domain names in the configuration, and they will be allow-listed as soon as you roll it out! No need to configure per domain VirtualServices or other routing details. Be careful, however, as the domain names are listed in multiple places in the config. If you use tooling for CI/CD (e.g., Kustomize), it’s best to extract the domain name list into a single place from which you can render into the required configuration resources.

That’s all! I hope this was helpful. If you’re an existing user of the previous Nginx-based solution, you can now migrate to this approach before upgrading to Istio 1.20, which will otherwise disrupt your current setup.

Happy SNI routing!

References

• Envoy docs for the SNI forwarder
• Previous solution with Nginx as an SNI proxy container in the gateway

With Istio Day NA over, that’s a wrap for our major community events for 2023. In case you missed them, Istio Day Europe was held in April, and alongside our Virtual IstioCon 2023 event, IstioCon China 2023 was held on September 26 in Shanghai, China.

Istio Day kicked off with an opening keynote from the Program Committee chairs, Faseela K and Zack Butcher. The keynote made sure to recognize the day-to-day efforts of our contributors, maintainers, release managers, and users, with some awards for our top contributors and community helpers. Rob Salmond and Andrea Ma were recognized for their selfless efforts in the Istio community, and the top 20 contributors in the last 6 months were also called out.

The opening keynote also announced the availability of the Istio Certified Associate (ICA) exam for enrollment starting November 6th.

We were also proud to showcase a small video of many of our contributors, vendors and end-users congratulating us for the CNCF graduation!

The keynote was followed by an end user talk by Kush Trivedi and Khushboo Mittal from DevRev about their usage of Istio. We had a much-awaited session on architecting ambient for scale from John Howard, which stirred some interesting discussions in the community. We also had an interesting talk showcasing the collaboration between Lilt and Intel about Scaling AI powered translation services using Istio.

After this we stepped into another end user talk from Intuit where Karim Lakhani explained about Intuit’s modern SaaS platform deploying multiple cloud native projects including Istio. The audience was excited when Mitch Connors and Christian Hernandez did a live demo of upgrading Istio ambient mesh with Argo on a live public site, with a publicly accessible availability monitor.

The event witnessed more focus on security in subsequent talks with Jackie Elliot from Microsoft taking a dig into Istio Identity, followed by a lightning talk from Kush Mansing from Speedscale showing the impacts of running services with arbitrary code on Istio. We also had a lightning talk from Xiangfeng Zhu, a PhD student at the University of Washington, where he showcased a tool developed to analyze and predict the performance overhead of Istio.

The talk from the Kiali maintainers Jay Shaughnessy and Nick Fox, was very interesting, as it demonstrated many advanced ways of using Kiali for better debugging of Istio use cases. Ekansh Gupta from Zeta, and Nirupama Singh from Reskill pitched in another end user talk explaining the best practices while upgrading Istio in their production deployments.

Istio multi-cluster is always a hot topic, and Lukonde Mwila and Ovidiu from AWS nailed it in the talk on bridging trust between multi-cluster meshes.

We also had an interactive panel discussion with the Istio TOC Members, where a lot of questions came in from the audience, and the good attendance for the discussion was a testament to the continued popularity of Istio. Istio Day concluded with a brilliant workshop on getting started with ambient mesh from Christian Posta and Jim Barton from Solo.io, which is the hot topic all of the audience were looking forward to.

The slides for all the sessions can be found in the Istio Day NA 2023 schedule.

Our presence at the conference did not end with Istio Day. The first day keynote of KubeCon + CloudNativeCon started with a project update video from Mitch Connors. It was also a proud moment for us, when two of our contributors, Lin Sun and Faseela K, took home the prestigious CNCF community “Chop Wood Carry Water” award, presented by Chris Aniszczyk, CTO CNCF, at the second day keynote.

Some of our maintainers and contributors made it to the CNCF Fall 2023 Ambassadors list as well, Lin Sun, Mitch Connors, and Faseela K, to name a few.

The KubeCon maintainer track session for Istio, presented by TOC members John Howard and Louis Ryan, grabbed great attention as they talked about the current ongoing efforts and future roadmap of Istio. The technologies described in the talk, and the resulting size of the audience, underlined why Istio continues to be the most popular service mesh in the industry.

The Contribfest Hands-on Development and Contribution Workshop by Lin Sun, Eric Van Norman, Steven Landow, and Faseela K was also well received. It was great to see so many people interested in contributing to Istio and pushing their first pull request at the end of the workshop.

A much-awaited panel discussion on Service Mesh Battle Scars: Technology, Timing and Tradeoffs, led by the maintainers from three CNCF Service Mesh projects, had a huge crowd in attendance, and a lot of interesting discussions.

Istio came up as a hot topic of discussion in several other KubeCon talks as well. Here are a few we noticed:

• Take It to the Edge: Creating a Globally Distributed Ingress with Istio & K8gb
• Under the Hood: Exploring Istio’s Lock Contention and Its Impact on Expedia’s Compute Platform
• Untangling Your Service Mesh with Feature Gates
• Flavors of Certificates in Service Mesh: The Whys and Hows!

Istio had a kiosk in the project pavilion, with the majority of questions asked being around the schedule for ambient mesh being production ready.

We are glad that the major question which we had at the Istio kiosk in Europe — the schedule for CNCF graduation — has been answered, and we assured everyone that we are working on ambient mesh with the same level of seriousness.

Many of our members and maintainers offered support at our kiosk, helping us answer all the questions from our users.

Another highlight of our kiosk was that we had new Istio T-shirts sponsored by Microsoft, Solo.io, Stackgenie and Tetrate for everyone to grab!

We would like to express our heartfelt gratitude to our platinum sponsors Google Cloud, for supporting Istio Day North America! Last but not least, we would like to thank our Istio Day Program Committee members, for all their hard work and support!

See you in Paris in March 2024!

What do you need to secure the communications among your applications?

Modern cloud native applications are frequently distributed across multiple Kubernetes clusters or virtual machines. New versions are being staged frequently and they can rapidly scale up and down based on user requests. As modern applications gain resource utilization efficiency by not being dependent on co-location, it is paramount to be able to apply access policy to and secure the communications among these distributed applications due to increased multiple entry points resulting in a larger attack surface. To ignore this is to invite massive business risk from data loss, data theft, forged data, or simple mishandling.

The following are the common key requirements for secure communications between applications:

Identities

Identity is a fundamental component of any security architecture. Before your applications can send their data securely, identities must be established for the applications. This establishing an identity process is called identity validation - it involves some well-known, trusted authority performing one or more checks on the application workload to establish that it is what it claims to be. Once the authority is satisfied, it grants the workload an identity.

Consider the act of being issued a passport - you will request one from some authority, that authority will probably ask you for several different identity validations that prove you are who you say you are - a birth certificate, current address, medical records, etc. Once you have satisfied all the identity validations, you will (hopefully) be granted the identity document. You can give that identity document to someone else as proof that you have satisfied all the identity validation requirements of the issuing authority, and if they trust the issuing authority (and the identity document itself), they can trust what it says about you (or they can contact the trusted authority and verify the document).

An identity could take any form, but, as with any form of identity document, the weaker the identity validations are, the easier it is to forge, and the less useful that identity document is to anyone using it to make a decision. That’s why, in computing, cryptographically verifiable identities are so important - they are signed by a verifiable authority, similar to your passport and driver’s license. Identities based around anything less are a security weakness that is relatively easy to exploit.

Your system may have identities derived from network properties such as IP addresses with distributed identity caches that track the mapping between identities and these network properties. These identities don’t have strong guarantees as cryptographically verifiable identities because IP addresses could be re-allocated to different workloads and identity caches may not always be updated to the latest.

Using cryptographically verifiable identities for your applications is desired, because exchanging cryptographically verifiable identities for applications during connection establishment is inherently more reliable and secure than systems dependent on mapping IP addresses to identities. These systems depend on distributed identity caches with eventual consistency and staleness issues which could create a structural weakness in Kubernetes, where high rates of automated pod churn are the norm.

Confidentiality

Encrypting the data transmitted among applications is critical - because in a world where breaches are common, costly, and effectively trivial, relying entirely on secure internal environments or other security perimeters has long since ceased to be adequate. To prevent a man-in-the-middle attack, you require a unique encryption channel for a source-destination pair because you want a strong identity uniqueness guarantee to avoid confused deputy problems. In other words, it is not enough to simply encrypt the channel - it must be encrypted using unique keys directly derived from the unique source and destination identities so that only the source and destination can decrypt the data. Further, you may need to customize the encryption, e.g. by choosing specific ciphers, in accordance with what your security team requires.

Integrity

The encrypted data sent over the network from source to destination can’t be modified by any identities other than the source and destination once it is sent. In other words, data received is the same as data sent. If you don’t have data integrity, someone in the middle could modify some bits or the entire content of the data during the communication between the source and destination.

Access Policy Enforcement

Application owners need to apply access policies to their applications and have them enforced properly, consistently, and unambiguously. In order to apply policy for both ends of a communication channel, we need an application identity for each end. Once we have a cryptographically verifiable identity with an unambiguous provenance chain for both ends of a potential communication channel, we can begin to apply policies about who can communicate with what. Standard TLS, the widely used cryptographic protocol that secures communication between clients (e.g., web browsers) and servers (e.g., web servers), only really verifies and mandates an identity for one side - the server. But for comprehensive end-to-end policy enforcement, it is critical to have a reliable, verifiable, unambiguous identity for both sides - client and server. This is a common requirement for internal applications - imagine for example a scenario where only a frontend application should call the GET method for a backend checkout application, but should not be allowed to call the POST or DELETE method. Or a scenario where only applications that have a JWT token issued by a particular JWT issuer can call the GET method for a checkout application. By leveraging cryptographic identities on both ends, we can ensure powerful access policies are enforced correctly, securely, and reliably, with a validatable audit trail.

FIPS compliance

Federal Information Processing Standards (FIPS) are standards and guidelines for federal computer systems that are developed by National Institute of Standards and Technology (NIST). Not everyone requires FIPS compliance, but FIPS compliance means meeting all the necessary security requirements established by the U.S. government for protecting sensitive information. It is required when working with the federal government. To follow the guidelines developed by the U.S. government relating to cybersecurity, many in the private sector voluntarily use these FIPS standards.

To illustrate the above secure application requirements (identity, confidentiality and integrity), let’s use the example that the frontend application calls the checkout application. Remember, you can think of ID in the diagram as any kind of identity document such as a government issued passport, photo identifier:

How does mTLS satisfy the above requirements?

TLS 1.3 (the most recent TLS version at the time of writing) specification’s primary goal is to provide a secure channel between two communicating peers. The TLS secure channel has the following properties:

1. Authentication: the server side of the channel is always authenticated, the client side is optionally authenticated. When the client is also authenticated, the secure channel becomes a mutual TLS channel.
2. Confidentiality: Data is encrypted and only visible to the client and server. Data must be encrypted using keys that are unambiguously cryptographically bound to the source and destination identity documents in order to reliably protect the application-layer traffic.
3. Integrity: data sent over the channel can’t be modified without detection. This is guaranteed by the fact that only source and destination have the key to encrypt and decrypt the data for a given session.

mTLS internals

We’ve established that cryptographically verifiable identities are key for securing channels and supporting access policy enforcement, and we’ve established that mTLS is a battle-tested protocol that mandates some extremely important guarantees for using cryptographically verifiable identities on both ends of a channel - let’s get into some detail on how the mTLS protocol actually works under the hood.

Handshake protocol

The handshake protocol authenticates the communicating peers, negotiates cryptographic modes and parameters, and establishes shared keying material. In other words, the role of the handshake is to verify the communicating peers’ identities and negotiate a session key, so that the rest of the connection can be encrypted based on the session key. When your applications make a mTLS connection, server and client negotiate a cipher suite, which dictates what encryption algorithm your applications will use for the rest of the connection and your applications also negotiate the cryptographic session key to use. The whole handshake is designed to resist tampering - interference by any entities that do not possess the same unique, cryptographically verifiable identity document as the source and/or destination will be rejected. For this reason, it is important to check the whole handshake and verify its integrity before any communicating peer continues with the application data.

The handshake can be thought of as having three phases per the handshake protocol overview in the TLS 1.3 specification - again, let’s use the example of a frontend application calling a backend checkout application:

1. Phase 1: frontend and checkout negotiates the cryptographic parameters and encryption keys that can be used to protect the rest of the handshake and traffic data.
2. Phase 2: everything in this phase and after are encrypted. In this phase, frontend and checkout establish other handshake parameters, and whether or not the client is also authenticated - that is, mTLS.
3. Phase 3: frontend authenticates checkout via its cryptographically verifiable identity (and, in mTLS, checkout authenticates frontend in the same way).

There are a few major differences since TLS 1.2 related to handshake, refer to the TLS 1.3 specification for more details:

1. All handshake messages (phase 2 and 3) are encrypted using the encryption keys negotiated in phase 1.
2. Legacy symmetric encryption algorithms have been pruned.
3. A zero round-trip time (0-RTT) mode was added, saving a round trip at connection setup.

Record protocol

Having negotiated the TLS protocol version, session-key & HMAC during the handshake phase, the peers can now securely exchange encrypted data that is chunked by the record protocol. It is critical (and required as part of the spec) to use the exact same negotiated parameters from the handshake to encrypt the traffic to ensure the traffic confidentiality and integrity.

Putting the two protocols from the TLS 1.3 specification together and using the frontend and checkout applications to illustrate the flow as below:

Who issues the identity certificates for frontend and checkout? They are commonly issued by a certificate authority (CA) which either has its own root certificate or uses an intermediate certificate from its root CA. A root certificate is basically a public key certificate that identifies a root CA, which you likely already have in your organization. The root certificate is distributed to frontend (or checkout) in addition to its own root-signed identity certificate. This is how everyday, basic Public Key Infrastructure (PKI) works - a CA has responsibility for validating an entity’s identity document, and then grants it an unforgeable identity document in the form of a certificate.

You can rely on your CA and intermediate CAs as source of identity truth in a structural fashion that maintains high availability and stable, persistently-verifiable identity guarantees in a way that a massive distributed cache of IP and identity maps simply cannot. When the frontend and checkout identity certificates are issued by the same root certificate, frontend and checkout can verify their peer identities consistently and reliably regardless of which cluster or nodes or scale they run.

You learned about how mTLS provides cryptographic identity, confidentiality and integrity, what about scalability as you grow to thousands or more applications among multiple clusters? If you establish a single root certificate across multiple clusters, the system doesn’t need to care when your application gets a connection request from another cluster as long as it is trusted by the root certificate - the system knows the identity on the connection is cryptographically verified. As your application pod changes IP or is redeployed to a different cluster or network, your application (or component acting on behalf of it) simply originates the traffic with its trusted certificate minted by the CA to the destination. It can be 500+ network hops or can be direct; your access policies for your application are enforced in the same fashion regardless of the topology, without needing to keep track of the identity cache and calculate which IP address maps to which application pod.

What about FIPS compliance? Per TLS 1.3 specification, TLS-compliant applications must implement the TLS_AES_128_GCM_SHA256 cipher suite, and are recommended to implement TLS_AES_256_GCM_SHA384, both of which are also in the guidelines for TLS by NIST. RSA or ECDSA server certificates are also recommended by both TLS 1.3 specification and NIST’s guideline for TLS. As long as you use mTLS and FIPS 140-2 or 140-3 compliant cryptographic modules for your mTLS connections, you will be on the right path for FIPS 140-2 or 140-3 validation.

What could go wrong

It is critical to implement mTLS exactly as the TLS 1.3 specification dictates. Without using proper mTLS following the TLS specification, here are a few things that can go wrong without detection:

What if someone in the middle of the connection silently captures the encrypted data?

If the connection doesn’t follow exactly the handshake and record protocols as outlined in the TLS specification, for example, the connection follows the handshake protocol but not using the negotiated session key and parameters from the handshake in the record protocol, you may have your connection’s handshake unrelated to the record protocol where identities could be different between the handshake and record protocols. TLS requires that the handshake and record protocols share the same connection because separating them increases the attack surface for man-in-the-middle attacks.

A mTLS connection has a consistent end-to-end security from start of the handshake to finish. The encrypted data is encrypted with the session key negotiated using the public key in the certificate. Only the source and destination can decrypt the data with the private key. In other words, only the owner of the certificate who has the private key can decrypt the data. Unless a hacker has control of the private key of the certificate, he or she doesn’t have a way to mess around with the mTLS connection to successfully execute a man-in-the-middle attack.

What if either source or destination identity is not cryptographically secure?

If the identity is based on network properties such as IP address, which could be re-allocated to other pods, the identity can’t be validated using cryptographic techniques. Since this type of identity isn’t based on cryptographic identity, your system likely has an identity cache to track the mapping between the identity, the pod’s network labels, the corresponding IP address and the Kubernetes node info where the pod is deployed. With an identity cache, you could run into pod IP addresses being reused and identity mistaken where policy isn’t enforced properly when the identity cache gets out of sync for a short period of time. For example, if you don’t have cryptographic identity on the connection between the peers, your system would have to get the identity from the identity cache which could be outdated or incomplete.

These identity caches that map identity to workload IPs are not ACID (Atomicity, Consistency, Isolation, and Durability) and you want your security system to be applied to something with strong guarantees. Consider the following properties and questions you may want to ask yourself:

• Staleness: How can a peer verify that an entry in the cache is current?
• Incompleteness: If there’s a cache miss and the system fails to close the connection, does the network become unstable when it’s only the cache synchronizer that is failing?
• What if something simply doesn’t have an IP? For example, an AWS Lambda service doesn’t by default have a public IP.
• Non-transactional: If you read the identity twice will you see the same value? If you are not careful in your access policy or auditing implementation this can cause real issues.
• Who will guard the guards themselves? Are there established practices to protect the cache like a CA has? What proof do you have that the cache has not been tampered with? Are you forced to reason about (and audit) the security of some complex infrastructure that is not your CA?

Some of the above are worse than others. You can apply the failing closed principle but that does not solve all of the above.

Identities are also used in enforcing access policies such as authorization policy, and these access policies are in the request path where your system has to make decisions fast to allow or deny the access. Whenever identities become mistaken, access policies could be bypassed without being detected or audited. For example, your identity cache may have your checkout pod’s prior allocated IP address associated as one of the checkout identities. If the checkout pod gets recycled and the same IP address is just allocated to one of the frontend pods, that frontend pod could have the checkout’s identity before the cache is updated, which could cause wrong access policies to be enforced.

Let us illustrate the identity cache staleness problem assuming the following large scale multi-cluster deployment:

1. 100 clusters where each cluster has 100 nodes with 20 pods per node. The number of total pods is 200,000.
2. 0.25% of pods are being churned at all times (rollout, restarts, recovery, node churn, …), each churn is a 10 second window.
3. 500 pods which are being churned are distributed to 10,000 nodes (caches) every 10 secs
4. If the cache synchronizer stalls what % stale is the system after 5 minutes - potentially as high as 7.5%!

Above assumes the cache synchronizer is in a steady state. If cache synchronizer has a brown-out it would affect its health-checking which increases churn rate, leading to cascading instability.

CA could also be compromised by an attacker who claims to present someone else and trick the CA to issue a certificate. The attacker can then use that certificate to communicate with other peers. This is where certificate revocation can remediate the situation by revoking the certificate so it is no longer valid. Otherwise the attacker can exploit the compromised certificate till expiry. It is critical to keep the private key for the root certificates in an HSM that is kept offline and use intermediate certificates for signing workload certificates. In the event when CA is brown-out or stalled for 5 minutes, you won’t be able to obtain new or renewed workload certificates but the previously issued and valid certificates continue to provide strong identity guarantees for your workloads. For increased reliability for issuance, you can deploy Intermediate CAs to different zones and regions.

mTLS in Istio

Enable mTLS

Enabling mTLS in Istio for intra-mesh applications is very simple. All you need is to add your applications to the mesh, which can be done by labeling your namespace for either sidecar injection or ambient. In the case of sidecar, a rollout restart would be required for sidecar to be injected to your application pods.

Cryptographic identity

In Kubernetes environment, Istio creates an application’s identity based on its service account. Identity certificate is provided to each application pod in the mesh after you add your application to the mesh.

By default, your pod’s identity certificate expires in 24 hours and Istio rotates the pod identity certificate every 12 hours so that in the event of a compromise (for example, compromised CA or stolen private key for the pod), the compromised certificate only works for a very limited period of time until the certificate expires and therefore limit the damage it can cause.

Enforce strict mTLS

The default mTLS behavior is mTLS whenever possible but not strictly enforced. To strictly enforce your application to accept only mTLS traffic, you can use Istio’s PeerAuthentication policy, mesh-wide or per namespace or workload. In addition, you can also apply Istio’s AuthorizationPolicy to control access for your workloads.

TLS version

TLS version 1.3 is the default in Istio for intra-mesh application communication with the Envoy’s default cipher suites (for example TLS_AES_256_GCM_SHA384 for Istio 1.19.0). If you need an older TLS version, you can configure a different mesh-wide minimum TLS protocol version for your workloads.

Wrapping up

The TLS protocol, as established by the Internet Engineering Task Force (IETF), is one of the most widely-reviewed, expert-approved, battle-tested data security protocols in existence. TLS is also widely used globally - whenever you visit any secured website, you shop with confidence partly because of the padlock icon to indicate that you are securely connected to a trusted site by using TLS. The TLS 1.3 protocol was designed with end-to-end authentication, confidentiality, and integrity to ensure your application’s identity and communications are not compromised, and to prevent man-in-the-middle attacks. In order to achieve that (and to be considered standards-compliant TLS), it is not only important to properly authenticate the communicating peers but also critical to encrypt the traffic using the keys established from the handshake. Now that you know mTLS excels at satisfying your secure application communication requirements (cryptographic identities, confidentiality, integrity and access policy enforcement), you can simply use Istio to upgrade your intra-mesh application communication with mTLS out of the box - with very little configuration!

Huge thanks to Louis Ryan, Ben Leggett, John Howard, Christian Posta, Justin Pettit who contributed significant time in reviewing and proposing updates to the blog!

IstioCon is committed to the industry-leading service mesh that provides a platform to explore insights gained from real-world Istio deployments, engage in interactive hands-on activities, and connect with maintainers across the entire Istio ecosystem.

Alongside our virtual IstioCon 2023 event, IstioCon China 2023 was held on September 26 in Shanghai, China. Part of the KubeCon + CloudNativeCon + Open Source Summit China, the event was arranged and hosted by the Istio maintainers and the CNCF. We were very proud to have a strong program for IstioCon in Shanghai and pleased to bring together members of the Chinese Istio community. The event was a testament to Istio’s immense popularity in the Asia-Pacific ecosystem.

IstioCon China kicked off with an opening keynote from Program Committee members Jimmy Song and Zhonghu Xu. The event was packed with great content, ranging from new features to end user talks, with major focus on the new Istio ambient mesh.

The welcome speech was followed by a sponsored keynote from Justin Pettit from Google, on “Istio Ambient Mesh as a Managed Infrastructure” which highlighted the importance and priority of the ambient model in the Istio community, especially for our top supporters like Google Cloud.

Perfectly placed after the keynote, Huailong Zhang from Intel and Yuxing Zeng from Alibaba discussed configurations for the co-existence of Ambient and Sidecar: a very relevant topic for existing users who want to experiment with the new ambient model.

Huawei’s new Istio data plane based on eBPF intends to implement the capabilities of L4 and L7 in the kernel,to avoid kernel-state and user-mode switching and reduce the latency of the data plane. This was explained by an interesting talk from Xie SongYang and Zhonghu Xu. Chun Li and Iris Ding from Intel also integrated eBPF with Istio, with their talk “Harnessing eBPF for Traffic Redirection in Istio ambient mode”, leading to more interesting discussions. DaoCloud also had a presence at the event, with Kebe Liu sharing Merbridge’s innovation in eBPF and Xiaopeng Han presenting about MirageDebug for localized Istio development.

The talk from Tetrate’s Jimmy Song, about the perfect union of different GitOps and Observability tools, was also very well received. Chaomeng Zhang from Huawei presented on how cert-manager helps enhance the security and flexibility of Istio’s certificate management system, and Xi Ning Wang and Zehuan Shi from Alibaba Cloud shared the idea of using VK (Virtual Kubelet) to implement serverless mesh.

While Shivanshu Raj Shrivastava gave a perfect introduction to WebAssembly through his talk “Extending and Customizing Istio with Wasm”, Zufar Dhiyaulhaq from GoTo Financial, Indonesia shared the practice of using Coraza Proxy Wasm to extend Envoy and quickly implement custom Web Application Firewalls. Huabing Zhao from Tetrate shared Aeraki Mesh’s Dubbo service governance practices with Qin Shilin from Boss Direct. While multi-tenancy is always a hot topic with Istio, John Zheng from HP described in detail about multi-tenant management in HP OneCloud Platform.

The slides for all the sessions can be found in the IstioCon China 2023 schedule and all the presentations will be available in the CNCF YouTube Channel soon for the audience in other parts of the world.

On the show floor

Istio had a full time kiosk in the project pavilion at KubeCon + CloudNativeCon + Open Source Summit China 2023 , with the majority of questions asked around ambient mesh. Many of our members and maintainers offered support at the booth, where a lot of interesting discussions happened.

Another highlight was the Istio Steering Committee members and authors of the Istio books “Cloud Native Service Mesh Istio” and “Istio: the Definitive Guide”, Zhonghu Xu and Chaomeng Zhang, spent time at the Istio booth interacting with our users and contributors.

We would like to express our heartfelt gratitude to our diamond sponsors Google Cloud, for supporting IstioCon 2023!

Last but not least, we would like to thank our IstioCon China Program Committee members for all their hard work and support!

See you all in Chicago in November!

There are 2 deployment modes for Istio: ambient mode and sidecar mode. The former is still on the way, the latter is the classic one. Therefore, the coexistence of ambient mode and sidecar mode should be a normal deployment form and the reason why this blog may be helpful for Istio users.

Background

In the architecture of modern microservices, communication and management among services is critical. To address the challenge, Istio emerged as a service mesh technology. It provides traffic control, security, and superior observation capabilities by utilizing the sidecar. In order to further improve the adaptability and flexibility of Istio, the Istio community began to explore a new mode - ambient mode. In this mode, Istio no longer relies on explicit sidecar injection, but achieves communication and mesh management among services through ztunnel and waypoint proxies. Ambient also brings a series of improvements, such as lower resource consumption, simpler deployment, and more flexible configuration options. When enabling ambient mode, we don’t have to restart pods anymore which enables Istio to play a better role in various scenarios.

There are many blogs, which can be found in the Reference Resources section of this blog, that introduce and analyze ambient, and this blog will analyze the network traffic path in Istio ambient and sidecar modes.

To clarify the network traffic paths and make it easier to understand, this blog post explores the following two scenarios with corresponding diagrams:

• The network path of services in ambient mode to services in sidecar mode
• The network path of services in sidecar mode to services in ambient mode

Information about the analysis

The analysis is based on Istio 1.18.2, where ambient mode uses iptables for redirection.

Ambient mode sleep to sidecar mode httpbin

Deployment and configuration for the first scenario

• sleep is deployed in namespace foo
  • sleep pod is scheduled to Node A
• httpbin is deployed in namespace bar
  • httpbin is scheduled to Node B
• foo namespace enables ambient mode (foo namespace contains label: istio.io/dataplane-mode=ambient)
• bar namespace enables sidecar injection (bar namespace contains label: istio-injection: enabled)

With the above description, the deployment and network traffic paths are:

ztunnel will be deployed as a DaemonSet in istio-system namespace if ambient mode is enabled, while istio-cni and ztunnel would generate iptables rules and routes for both the ztunnel pod and pods on each node.

All network traffic coming in/out of the pod with ambient mode enabled will go through ztunnel based on the network redirection logic. The ztunnel will then forward the traffic to the correct endpoints.

Network traffic path analysis of ambient mode sleep to sidecar mode httpbin

According to above diagram, the details of network traffic path is demonstrated as below:

(1) (2) (3) Request traffic of the sleep service is sent out from the veth of the sleep pod where it will be marked and forwarded to the istioout device in the node by following the iptables rules and route rules. The istioout device on node A is a Geneve tunnel, and the other end of the tunnel is pistioout, which is inside the ztunnel pod on the same node.

(4) (5) When the traffic arrives through the pistioout device, the iptables rules inside the pod intercept and redirect it through the eth0 interface in the pod to port 15001.

(6) According to the original request information, ztunnel can obtain the endpoint list of the target service. It will then handle sending the request to the endpoint, such as one of the httpbin pods. Finally, the request traffic would get into the httpbin pod via the container network.

(7) The request traffic arriving in httpbin pod will be intercepted and redirected through port 15006 of the sidecar by its iptables rules.

(8) Sidecar handles the inbound request traffic coming in via port 15006, and forwards the traffic to the httpbin container in the same pod.

Sidecar mode sleep to ambient mode httpbin and helloworld

Deployment and configuration for the second scenario

• sleep is deployed in namespace foo
  • sleep pod is scheduled to Node A
• httpbin deployed in namespace bar-1
  • httpbin pod is scheduled to Node B
  • the waypoint proxy of httpbin is disabled
• helloworld is deployed in namespace bar-2
  • helloworld pod is scheduled to Node D
  • the waypoint proxy of helloworld is enabled
  • the waypoint proxy is scheduled to Node C
• foo namespace enables sidecar injection (foo namespace contains label: istio-injection: enabled)
• bar-1 namespace enables ambient mode (bar-1 namespace contains label: istio.io/dataplane-mode=ambient)

With the above description, the deployment and network traffic paths are:

Network traffic path analysis of sidecar mode sleep to ambient mode httpbin

Network traffic path of a request from the sleep pod (sidecar mode) to the httpbin pod (ambient mode) is depicted in the top half of the diagram above.

(1) (2) (3) (4) the sleep container sends a request to httpbin. The request is intercepted by iptables rules and directed to port 15001 on the sidecar in the sleep pod. Then, the sidecar handles the request and routes the traffic based on the configuration received from istiod (control plane) forwarding the traffic to an IP address corresponding to the httpbin pod on node B.

(5) (6) After the request is sent to the device pair (veth httpbin <-> eth0 inside httpbin pod), the request is intercepted and forwarded using the iptables and route rules to the istioin device on node B where httpbin pod is running by following its iptables and route rules. The istioin device on node B and the pistion device inside the ztunnel pod on the same node are connected by a Geneve tunnel.

(7) (8) After the request enters the pistioin device of the ztunnel pod, the iptables rules in the ztunnel pod intercept and redirect the traffic through port 15008 on the ztunnel proxy running inside the pod.

(9) The traffic getting into the port 15008 would be considered as a inbound request, and the ztunnel will then forward the request to the httpbin pod in the same node B.

Network traffic path analysis of sidecar mode sleep to ambient mode httpbin via waypoint proxy

Comparing with the top part of the diagram, the bottom part inserts a waypoint proxy in the path between the sleep, ztunnel and httpbin pods. The Istio control plane has all the service information and configuration of the service mesh. When helloworld pod is deployed with a waypoint proxy, the EDS configuration of helloworld service received by the sleep pod sidecar will be changed to the type of envoy_internal_address. This causes that the request traffic going through the sidecar to be forwarded to port 15008 of the waypoint proxy on node C via the HTTP Based Overlay Network (HBONE) protocol.

Waypoint proxy is an instance of Envoy proxy and forwards the request to the helloworld pod based on the routing configuration received from the control plane. Once traffic reaches the veth on node D, it follows the same path as the previous scenario.

Wrapping up

The sidecar mode is what made Istio a great service mesh. However, the sidecar mode can also cause problems as it requires the app and sidecar containers to run in the same pod. Istio ambient mode implements communication among services through centralized proxies (ztunnel and waypoint). The ambient mode provides greater flexibility and scalability, reduces resource consumption as it doesn’t require a sidecar for each pod in the mesh, and allows more precise configuration. Therefore, there’s no doubt ambient mode is the next evolution of Istio. It’s obvious that the coexistence of sidecar and ambient modes may be last a very long time, although the ambient mode is still in alpha stage and the sidecar mode is still the recommended mode of Istio, it will give users a more light-weight option of running and adopting the Istio service mesh as the ambient mode moves towards beta and future releases.

Reference Resources

• Traffic in ambient mesh: Istio CNI and node configuration
• Traffic in ambient mesh: Redirection using iptables and Geneve tunnels
• Traffic in ambient mesh: ztunnel, eBPF configuration, and waypoint proxies

• Craig Box, ARMO
• Iris Ding, Intel
• Lin Sun, Solo.io
• Faseela K, Ericsson Software Technology

The winners will serve on the Steering Committee for one year, starting on September 1, 2023. They will be responsible for helping to guide the development and governance of Istio, the world’s most popular service mesh.

The election was held in August 2023, and was open to any member of the Istio community who submitted a pull request or made other significant project contributions. Over 120 eligible voters evaluated the candidates on their contributions to Istio, their experience in open source governance, and their commitment to the project’s mission.

In addition to the four Community Seats, the Istio Steering Committee also consists of nine Contribution Seats, which are awarded proportionally to organizations which made significant contributions to the project. The Contribution Seats for 2023 are held by:

• Google
• IBM / Red Hat
• Huawei

The Steering Committee congratulates the winners of the election, and looks forward to working with them to continue to grow and improve Istio as a successful and sustainable open source project. We encourage everyone to get involved in the Istio community, contribute, vote, and help us shape the future of service mesh.

This has caused a number of problems: what if you have a job that terminates by design, but a sidecar container that doesn’t? This exact use case is the most popular ever on the Kubernetes issue tracker.

A formal proposal for adding sidecar support in Kubernetes was raised in 2019. With many stops and starts along the way, and after a reboot of the project last year, formal support for sidecars is being released to Alpha in Kubernetes 1.28. Istio has implemented support for this feature, and in this post you can learn how to take advantage of it.

Sidecar woes

Sidecar containers give a lot of power, but come with some issues. While containers within a pod can share some things, their lifecycle’s are entirely decoupled. To Kubernetes, both of these containers are functionally the same.

However, in Istio they are not the same - the Istio container is required for the primary application container to run, and has no value without the primary application container.

This mismatch in expectation leads to a variety of issues:

• If the application container starts faster than Istio’s container, it cannot access the network. This wins the most +1’s on Istio’s GitHub by a landslide.
• If Istio’s container shuts down before the application container, the application container cannot access the network.
• If an application container intentionally exits (typically from usage in a Job), Istio’s container will still run and keep the pod running indefinitely. This is also a top GitHub issue.
• InitContainers, which run before Istio’s container starts, cannot access the network.

Countless hours have been spent in the Istio community and beyond to work around these issues - to limited success.

Fixing the root cause

While increasingly-complex workarounds in Istio can help alleviate the pain for Istio users, ideally all of this would just work - and not just for Istio. Fortunately, the Kubernetes community has been hard at work to address these directly in Kubernetes.

In Kubernetes 1.28, a new feature to add native support for sidecars was merged, closing out over 5 years of ongoing work. With this merged, all of our issues can be addressed without workarounds!

While we are on the “GitHub issue hall of fame”, these two issues account for #1 and #6 all time issues in Kubernetes - and have finally been closed!

A special thanks goes to the huge group of individuals involved in getting this past the finish line.

Trying it out

While Kubernetes 1.28 was just released, the new SidecarContainers feature is Alpha (and therefore, off by default), and the support for the feature in Istio is not yet shipped, we can still try it out today - just don’t try this in production!

First, we need to spin up a Kubernetes 1.28 cluster, with the SidecarContainers feature enabled:

$ cat <<EOF | kind create cluster --name sidecars --image gcr.io/istio-testing/kind-node:v1.28.0 --config=-
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
featureGates:
  SidecarContainers: true
EOF

Then we can download the latest Istio 1.19 pre-release (as 1.19 is not yet out). I used Linux here. This is a pre-release of Istio, so again - do not try this in production! When we install Istio, we will enable the feature flag for native sidecar support and turn on access logs to help demo things later.

$ TAG=1.19.0-beta.0
$ curl -L https://github.com/istio/istio/releases/download/$TAG/istio-$TAG-linux-amd64.tar.gz | tar xz
$ ./istioctl install --set values.pilot.env.ENABLE_NATIVE_SIDECARS=true -y --set meshConfig.accessLogFile=/dev/stdout

And finally we can deploy a workload:

$ kubectl label namespace default istio-injection=enabled
$ kubectl apply -f samples/sleep/sleep.yaml

Let’s look at the pod:

$ kubectl get pods
NAME                     READY   STATUS    RESTARTS   AGE
sleep-7656cf8794-8fhdk   2/2     Running   0          51s

Everything looks normal at first glance… If we look under the hood, we can see the magic, though.

$ kubectl get pod -o "custom-columns="\
"NAME:.metadata.name,"\
"INIT:.spec.initContainers[*].name,"\
"CONTAINERS:.spec.containers[*].name"

NAME                     INIT                     CONTAINERS
sleep-7656cf8794-8fhdk   istio-init,istio-proxy   sleep

Here we can see all the containers and initContainers in the pod.

Surprise! istio-proxy is now an initContainer.

More specifically, it is an initContainer with restartPolicy: Always set (a new field, enabled by the SidecarContainers feature). This tells Kubernetes to treat it as a sidecar.

This means that later containers in the list of initContainers, and all normal containers will not start until the proxy container is ready. Additionally, the pod will terminate even if the proxy container is still running.

Init container traffic

To put this to the test, let’s make our pod actually do something. Here we deploy a simple pod that sends a request in an initContainer. Normally, this would fail.

apiVersion: v1
kind: Pod
metadata:
  name: sleep
spec:
  initContainers:
  - name: check-traffic
    image: istio/base
    command:
    - curl
    - httpbin.org/get
  containers:
  - name: sleep
    image: istio/base
    command: ["/bin/sleep", "infinity"]

Checking the proxy container, we can see the request both succeeded and went through the Istio sidecar:

$ kubectl logs sleep -c istio-proxy | tail -n1
[2023-07-25T22:00:45.703Z] "GET /get HTTP/1.1" 200 - via_upstream - "-" 0 1193 334 334 "-" "curl/7.81.0" "1854226d-41ec-445c-b542-9e43861b5331" "httpbin.org" ...

If we inspect the pod, we can see our sidecar now runs before the check-traffic initContainer:

$ kubectl get pod -o "custom-columns="\
"NAME:.metadata.name,"\
"INIT:.spec.initContainers[*].name,"\
"CONTAINERS:.spec.containers[*].name"

NAME    INIT                                  CONTAINERS
sleep   istio-init,istio-proxy,check-traffic   sleep

Exiting pods

Earlier, we mentioned that when applications exit (common in Jobs), the pod would live forever. Fortunately, this is addressed as well!

First we deploy a pod that will exit after one second and doesn’t restart:

apiVersion: v1
kind: Pod
metadata:
  name: sleep
spec:
  restartPolicy: Never
  containers:
- name: sleep
  image: istio/base
  command: ["/bin/sleep", "1"]

And we can watch its progress:

$ kubectl get pods -w
NAME    READY   STATUS     RESTARTS   AGE
sleep   0/2     Init:1/2   0          2s
sleep   0/2     PodInitializing   0          2s
sleep   1/2     PodInitializing   0          3s
sleep   2/2     Running           0          4s
sleep   1/2     Completed         0          5s
sleep   0/2     Completed         0          12s

Here we can see the application container exited, and shortly after Istio’s sidecar container exits as well. Previously, the pod would be stuck in Running, while now it can transition to Completed. No more zombie pods!

What about ambient mode?

Last year, Istio announced ambient mode - a new data plane mode for Istio that doesn’t rely on sidecar containers. So with ambient mode coming, does any of this even matter?

I would say a resounding “Yes”!

While the impacts of sidecar are lessened when ambient mode is used for a workload, I expect that almost all large scale Kubernetes users have some sort of sidecar in their deployments. This could be Istio workloads they don’t want to migrate to ambient, that they haven’t yet migrated, or things unrelated to Istio. So while there may be fewer scenarios where this matters, it still is a huge improvement for the cases where sidecars are used.

You may wonder the opposite - if all our sidecar woes are addressed, why do we need ambient mode at all? There are still a variety of benefits ambient brings with these sidecar limitations addressed. For example, this blog post goes into details about why decoupling proxies from workloads is advantageous.

Try it out yourself

We encourage the adventurous readers to try this out themselves in testing environments! Feedback for these experimental and alpha features is critical to ensure they are stable and meeting expectations before promoting them. If you try it out, let us know what you think in the Istio Slack!

In particular, the Kubernetes team is interested in hearing more about:

• Handling of shutdown sequence, especially when there are multiple sidecars involved.
• Backoff restart handling when sidecar containers are crashing.
• Edge cases they have not yet considered.

Load balancing is a core networking solution used to distribute traffic across multiple servers in a server farm. Load balancers improve application availability and responsiveness and prevent server overload. Each load balancer sits between client devices and backend servers, receiving and then distributing incoming requests to any available server capable of fulfilling them.

For a common web server, it usually has multiple workers (processors or threads). If many clients connect to a single worker, this worker becomes busy and brings long tail latency while other workers run in the free state, affecting the performance of the web server. Connection load balancing is the solution for this situation, which is also known as connection balancing.

What does Istio do for connection load balancing?

Istio uses Envoy as the data plane.

Envoy provides a connection load balancing implementation called Exact connection balancer. As its name says, a lock is held during balancing so that connection counts are nearly exactly balanced between workers. It is “nearly” exact in the sense that a connection might close in parallel thus making the counts incorrect, but this should be rectified on the next accept. This balancer sacrifices accept throughput for accuracy and should be used when there are a small number of connections that rarely cycle, e.g., service mesh gRPC egress.

Obviously, it is not suitable for an ingress gateway since an ingress gateway accepts thousands of connections within a short time, and the resource cost from the lock brings a big drop in throughput.

Now, Envoy has integrated Intel® Dynamic Load Balancing (Intel®DLB) connection load balancing to accelerate in high connection count cases like ingress gateway.

How Intel® Dynamic Load Balancing accelerates connection load balancing in Envoy

Intel DLB is a hardware managed system of queues and arbiters connecting producers and consumers. It is a PCI device envisaged to live in the server CPU uncore and can interact with software running on cores, and potentially with other devices.

Intel DLB implements the following load balancing features:

• Offloads queue management from software — useful where there are significant queuing-based costs.
  • Especially with multi-producer / multi-consumer scenarios and enqueue batching to multiple destinations.
  • The overhead locks are required to access shared queues in the software. Intel DLB implements lock-free access to shared queues.
• Dynamic, flow aware load balancing and reordering.
  • Ensures equal distribution of tasks and better CPU core utilization. Can provide flow-based atomicity if required.
  • Distributes high bandwidth flows across many cores without loss of packet order.
  • Better determinism and avoids excessive queuing latencies.
  • Uses less IO memory footprint and saves DDR Bandwidth.
• Priority queuing (up to 8 levels) — allows for QOS.
  • Lower latency for traffic that is latency sensitive.
  • Optional delay measurements in the packets.
• Scalability
  • Allows dynamic sizing of applications, seamless scale up/down.
  • Power aware; application can drop workers to lower power state in cases of lighter load.

There are three types of load balancing queues:

• Unordered: For multiple producers and consumers. The order of tasks is not important, and each task is assigned to the processor core with the lowest current load.
• Ordered: For multiple producers and consumers where the order of tasks is important. When multiple tasks are processed by multiple processor cores, they must be rearranged in the original order.
• Atomic: For multiple producers and consumers, where tasks are grouped according to certain rules. These tasks are processed using the same set of resources and the order of tasks within the same group is important.

An ingress gateway is expected to process as much data as possible as quickly as possible, so Intel DLB connection load balancing uses an unordered queue.

How to use Intel DLB connection load balancing in Istio

With the 1.17 release, Istio officially supports Intel DLB connection load balancing.

The following steps show how to use Intel DLB connection load balancing in an Istio Ingress Gateway in an SPR (Sapphire Rapids) machine, assuming the Kubernetes cluster is running.

Step 1: Prepare DLB environment

Install the Intel DLB driver by following the instructions on the Intel DLB driver official site.

Install the Intel DLB device plugin with the following command:

$ kubectl apply -k https://github.com/intel/intel-device-plugins-for-kubernetes/deployments/dlb_plugin?ref=v0.26.0

For more details about the Intel DLB device plugin, please refer to Intel DLB device plugin homepage.

You can check the Intel DLB device resource:

$ kubectl describe nodes | grep dlb.intel.com/pf
  dlb.intel.com/pf:   2
  dlb.intel.com/pf:   2
...

Step 2: Download Istio

In this blog we use 1.17.2. Let’s download the installation:

$ curl -L https://istio.io/downloadIstio | ISTIO_VERSION=1.17.2 TARGET_ARCH=x86_64 sh -
$ cd istio-1.17.2
$ export PATH=$PWD/bin:$PATH

You can check the version is 1.17.2:

$ istioctl version
no running Istio pods in "istio-system"
1.17.2

Step 3: Install Istio

Create an install configuration for Istio, notice that we assign 4 CPUs and 1 DLB device to ingress gateway and set concurrency as 4, which is equal to the CPU number.

$ cat > config.yaml << EOF
apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  profile: default
  components:
    ingressGateways:
    - enabled: true
      name: istio-ingressgateway
      k8s:
        overlays:
          - kind: Deployment
            name: istio-ingressgateway
        podAnnotations:
          proxy.istio.io/config: |
            concurrency: 4
        resources:
          requests:
            cpu: 4000m
            memory: 4096Mi
            dlb.intel.com/pf: '1'
          limits:
            cpu: 4000m
            memory: 4096Mi
            dlb.intel.com/pf: '1'
        hpaSpec:
          maxReplicas: 1
          minReplicas: 1
  values:
    telemetry:
      enabled: false
EOF

Use istioctl to install:

$ istioctl install -f config.yaml --set values.gateways.istio-ingressgateway.runAsRoot=true -y
✔ Istio core installed
✔ Istiod installed
✔ Ingress gateways installed
✔ Installation complete                                                                                                                                                                                                                                                                       Making this installation the default for injection and validation.

Thank you for installing Istio 1.17.  Please take a few minutes to tell us about your install/upgrade experience!  https://forms.gle/hMHGiwZHPU7UQRWe9

Step 4: Setup Backend Service

Since we want to use DLB connection load balancing in Istio ingress gateway, we need to create a backend service first.

We’ll use an Istio-provided sample to test, httpbin.

$ kubectl apply -f samples/httpbin/httpbin.yaml
$ kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: httpbin-gateway
spec:
  # The selector matches the ingress gateway pod labels.
  # If you installed Istio using Helm following the standard documentation, this would be "istio=ingress"
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "httpbin.example.com"
EOF
$ kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: httpbin
spec:
  hosts:
  - "httpbin.example.com"
  gateways:
  - httpbin-gateway
  http:
  - match:
    - uri:
        prefix: /status
    - uri:
        prefix: /delay
    route:
    - destination:
        port:
          number: 8000
        host: httpbin
EOF

You have now created a virtual service configuration for the httpbin service containing two route rules that allow traffic for paths /status and /delay.

The gateways list specifies that only requests through your httpbin-gateway are allowed. All other external requests will be rejected with a 404 response.

Step 5: Enable DLB Connection Load Balancing

$ kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: dlb
  namespace: istio-system
spec:
  workloadSelector:
    labels:
      istio: ingressgateway
  configPatches:
  - applyTo: LISTENER
    match:
      context: GATEWAY
    patch:
      operation: MERGE
      value:
        connection_balance_config:
            extend_balance:
              name: envoy.network.connection_balance.dlb
              typed_config:
                "@type": type.googleapis.com/envoy.extensions.network.connection_balance.dlb.v3alpha.Dlb
EOF

It is expected that if you check the log of ingress gateway pod istio-ingressgateway-xxxx you will see log entries similar to:

$ export POD="$(kubectl get pods -n istio-system | grep gateway | awk '{print $1}')"
$ kubectl logs -n istio-system ${POD} | grep dlb
2023-05-05T06:16:36.921299Z     warning envoy config external/envoy/contrib/network/connection_balance/dlb/source/connection_balancer_impl.cc:46        dlb device 0 is not found, use dlb device 3 instead     thread=35

Envoy will auto detect and choose the DLB device.

Step 6: Test

$ export HOST="<YOUR-HOST-IP>"
$ export PORT="$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.ports[?(@.name=="http2")].nodePort}')"
$ curl -s -I -HHost:httpbin.example.com "http://${HOST}:${PORT}/status/200"
HTTP/1.1 200 OK
server: istio-envoy
...

Note that you use the -H flag to set the Host HTTP header to httpbin.example.com since now you have no DNS binding for that host and are simply sending your request to the ingress IP.

You can also add the DNS binding in /etc/hosts and remove -H flag:

$ echo "$HOST httpbin.example.com" >> /etc/hosts
$ curl -s -I "http://httpbin.example.com:${PORT}/status/200"
HTTP/1.1 200 OK
server: istio-envoy
...

Access any other URL that has not been explicitly exposed. You should see an HTTP 404 error:

$ curl -s -I -HHost:httpbin.example.com "http://${HOST}:${PORT}/headers"
HTTP/1.1 404 Not Found
...

You can turn on debug log level to see more DLB related logs:

$ istioctl pc log ${POD}.istio-system --level debug
istio-ingressgateway-665fdfbf95-2j8px.istio-system:
active loggers:
  admin: debug
  alternate_protocols_cache: debug
  aws: debug
  assert: debug
  backtrace: debug
...

Run curl to send one request and you will see something like below:

$ kubectl logs -n istio-system ${POD} | grep dlb
2023-05-05T06:16:36.921299Z     warning envoy config external/envoy/contrib/network/connection_balance/dlb/source/connection_balancer_impl.cc:46        dlb device 0 is not found, use dlb device 3 instead     thread=35
2023-05-05T06:37:45.974241Z     debug   envoy connection external/envoy/contrib/network/connection_balance/dlb/source/connection_balancer_impl.cc:269   worker_3 dlb send fd 45 thread=47
2023-05-05T06:37:45.974427Z     debug   envoy connection external/envoy/contrib/network/connection_balance/dlb/source/connection_balancer_impl.cc:286   worker_0 get dlb event 1        thread=46
2023-05-05T06:37:45.974453Z     debug   envoy connection external/envoy/contrib/network/connection_balance/dlb/source/connection_balancer_impl.cc:303   worker_0 dlb recv 45    thread=46
2023-05-05T06:37:45.975215Z     debug   envoy connection external/envoy/contrib/network/connection_balance/dlb/source/connection_balancer_impl.cc:283   worker_0 dlb receive none, skip thread=46

For more details about Istio Ingress Gateway, please refer to Istio Ingress Gateway Official Doc.

We would like to thank our TOC sponsors Emily Fox and Nikhita Raghunath, and everyone who has collaborated over the past six years on Istio’s design, development, and deployment.

As before, project work continues uninterrupted. We were excited to bring ambient mesh to Alpha in Istio 1.18 and are continuing to drive it to production readiness. Sidecar deployments remain the recommended method of using Istio, and our 1.19 release will support a new sidecar container feature in Alpha in Kubernetes 1.28.

We have been delighted to welcome Microsoft to our community after their decision to archive the Open Service Mesh project and collaborate together on Istio. As the third most active CNCF project in terms of PRs, and with support from over 20 vendors and dozens of contributing companies, there is simply no better choice for a service mesh.

We would like to invite the Istio community to submit a talk to the upcoming virtual IstioCon 2023, the companion full day, in-person event co-located with KubeCon China in Shanghai, or Istio Day co-located with KubeCon NA in Chicago.

Watch a video

In this video for Techstrong TV, I talk about the history of the project, and what graduation means to us.

Words of support from our alumni

When we announced our incubation, we mentioned that the journey began with Istio’s inception in 2016. One of the great things about collaborative open source projects is that people come and go from employers, but their affiliation with a project can remain. Some of our original contributors founded companies based on Istio; some moved to other companies that support it; and some are still working on it at Google or IBM, six years later.

The announcement from the CNCF and blog posts from Intel, Red Hat, Solo.io, Tetrate, VMware and DaoCloud summarize the thoughts and feelings of those working on the project today.

We also reached out to some contributors who have moved on from the project, to share their thoughts.

We all had a blast at Istio Day Europe in April. The event was incredibly well received, but organizers and attendees alike felt that a half-day was not enough to showcase all that Istio has to offer. Due to the overwhelming response, we are glad to share with all of you that Istio Day North America is going to be a full-day event, co-located with KubeCon North America in Chicago.

Submit a talk

We now encourage Istio users, developers, partners, and advocates to submit a session proposal through the CNCF event portal, which is open until August 6th.

We want to see real world examples, case studies, and success stories that can inspire newcomers to use Istio in production. The content will cover introductory to advanced levels, split into four main topic tracks:

• New Features: What have you been working on that the community should know about?
• Case Studies: How have you built a platform or service on top of Istio?
• Istio Recipes: How you can solve a specific business problem using Istio.
• Project Updates: The evolution of Istio, and the latest updates from the project maintainers.

You can pick one of these formats to submit a session proposal:

• Presentation: 25 minutes, 1 or 2 speaker(s) presenting a topic
• Panel Discussion: 35 minutes of discussion among 3 to 5 speakers
• Lightning Talk: A brief 5-minute presentation, maximum of 1 speaker

Accepted speakers will receive a complimentary All-Access In-Person ticket for all four days of KubeCon + CloudNativeCon.

Timeline

• Thursday, June 15: CFP + Sponsor Prospectus Launch
• Sunday, August 6: CFP Closes
• Tuesday, August 8 - Monday, August 21: CFP Review Window
• Thursday, September 7: Speaker Notifications
• Week of September 11: Schedule Launch & Announcement
• Wednesday, September 20: Sponsor Sales Close
• Monday, November 6: Event Day

Sponsor the event

Do you want to put your product or service in front of the most discerning Cloud Native users: those who demand 25% more conference than the crowd? Check out page 19 of the CNCF events prospectus to learn more. Contact sponsor@cncf.io to secure your sponsorship today! Signed contracts must be received by September 20.

Register to attend

Istio Day is a KubeCon + CloudNativeCon North America CNCF-hosted Co-located Event. In-person KubeCon + CloudNativeCon attendees have the option to buy an All-Access ticket which includes entry to all the CNCF-hosted “day 0” events, as well as the main three days of the conference. You must be attending KubeCon to attend Istio Day, but virtual registration options are available, and the recordings will be posted to YouTube soon after the event.

For those of you who can’t make it, keep your eyes peeled for announcements of IstioCon 2023 (Virtual) and Istio Day China.

Stay tuned to hear more about the event, and we hope you can join us in Chicago for Istio Day!

Istio Day kicked off with an opening keynote from the Program Committee chairs, Mitch Connors and Faseela K. The event was packed with great content, ranging from new features to end user talks, and the hall was always jam-packed. The opening keynote was an ice-breaker with some Istio fun in the form of a pop quiz, and recognition for the day-to-day efforts of our contributors, maintainers, release managers, and users.

This was followed by a 2023 roadmap update session from TOC members Lin Sun and Louis Ryan. We had our much awaited session on the security posture of Ambient Mesh, from Christian Posta and John Howard, which stirred some interesting discussions in the community. After this we stepped into our first end user talk from John Keates from Wehkamp, a local Dutch company, followed by speakers from Bloomberg, Alexa Griffith and Zhenni Fu, on how they secure their highly privileged financial information using Istio. Istio Day witnessed more focus on security, which became even more prominent when Zack Butcher talked about using Istio for Controls Compliance. We also had lightning talks covering faster Istio development environments, guide for Istio resource isolation and securing hybrid cloud deployments from Mitch Connors, Zhonghu Xu and Matt Turner respectively.

A number of our ecosystem members had Istio-related announcements at the event. Microsoft announced Istio as a managed add-on for Azure Kubernetes Service, and support for Istio is now generally available in D2iQ Kubernetes Platform.

Tetrate announced Tetrate Service Express, an Istio-based service connectivity, security and resilience automation solution for Amazon EKS, and Solo.io announced Gloo Fabric, with Istio-based application networking capabilities expanded to VM-based, container, and serverless applications across cloud environments.

Istio’s presence at the conference did not end with Istio Day. The second day keynote started with a project update video from Lin Sun. It was also a proud moment for us, when our steering committee member Craig Box was recognized as a CNCF mentor in the keynote. The maintainer track for Istio presented by TOC member Neeraj Poddar grabbed great attention as he talked about the current ongoing efforts and future roadmap of Istio. The talk, and the size of the audience, underlined why Istio continues to be the most popular service mesh in the industry.

The following sessions at KubeCon were based on Istio and almost all of them had a huge crowd in attendance:

• Future of Istio - Sidecar, Sidecarless or Both?
• Operate Multi Tenancy Istio with ArgoCD in production
• Create Istio Filters with Any Programming Language
• Automated Cloud-Native Incident Response with Kubernetes and Service Mesh
• Autoscaling Elastic Kubernetes Infrastructure for Stateful Applications Using Proxyless gRPC and Istio
• Developing a Mental Model of Istio: From Kubernetes to Sidecars to Ambient
• Future of ServiceMesh - Sidecar, Sidecarless or Proxyless? - Panel Discussion
• The Top 10 List of Istio Security Risks and Mitigation Strategies

Istio had a full time kiosk in the KubeCon project pavilion, with the majority of questions asked being on the status of our CNCF graduation. We are so excited to know that our users are eagerly waiting for news of our graduation, and we promise we are actively working towards it!

Many of our TOC members and maintainers also offered support at the booth, where a lot of interesting discussions happened around Istio Ambient Mesh as well.

Another highlight was Istio TOC and steering members and authors Lin Sun and Christian Posta signing copies of the “Istio Ambient Explained” book.

Last, but not least, we would like to express our heartfelt gratitude to our platinum sponsors Tetrate, for supporting Istio Day!

2023 is going to be really big for Istio, with more events planned for the coming months! Stay tuned for updates on IstioCon 2023 and Istio’s presence at KubeCon in China and North America.

This post will explore the tools and practices leveraged by Splunk to secure their Kubernetes network infrastructure, starting with VPC design and connectivity and going all the way up the stack to HTTP Request based security. Along the way, we’ll see what it takes to provide comprehensive network security for your cloud native stack, how these tools interoperate, and where some of them can improve. Splunk uses a variety of tools to secure their network, including:

• AWS Functionality
• Kubernetes
• Istio
• Envoy
• Aviatrix

About Splunk’s Use Case

Splunk is a technology company that provides a platform for collecting, analyzing and visualizing data generated by various sources. It is primarily used for searching, monitoring, and analyzing machine-generated big data through a web-style interface. Splunk Cloud is an initiative to move Splunk’s internal infrastructure to a cloud native architecture. Today Splunk Cloud consists of over 35 fully replicated clusters in AWS and GCP in regions around the world.

Securing Layer 3/4: AWS, Aviatrix and Kubernetes

At Splunk Cloud, we use a pattern called “cookie cutter VPCs” where each cluster is provisioned with it’s own VPC, with identical private subnets for Pod and Node IPs, a public subnet for ingress and egress to and from the public internet, and an internal subnet for traffic between clusters. This keeps Pods and Nodes from separate clusters completely isolated, while allowing traffic outside the cluster to have particular rules enforced in the public and internal subnets. Additionally, this pattern avoids the possibility of RFC 1918 private IP exhaustion when leveraging many clusters.

Within each VPC, Network ACLs and Security Groups are set up to restrict connectivity to what is absolutely required. As an example, we restrict public connectivity to our Ingress nodes (that will deploy Envoy ingress gateways). In addition to ordinary east/west and north/south traffic, there are also shared services at Splunk that every cluster needs to access. Aviatrix is used to provide overlapping VPC access, while also enforcing some high level security rules (segmentation per domain).

The next security layer in Splunk’s stack is Kubernetes itself. Validating Webhooks are used to prevent the deployment of K8S objects that would allow insecure traffic in the cluster (typically around NLBs and services). Splunk also relies on NetworkPolicies for securing and restricting Pod to Pod connectivity.

Securing Layer 7: Istio

Splunk uses Istio to enforce policy on the application layer based on the details of each request. Istio also emits Telemetry data (metrics, logs, traces) that is useful for validating request-level security.

One of the key benefits of Istio’s injection of Envoy sidecars is that Istio can provide in-transit encryption for the entire mesh without requiring any modifications to the applications. The applications send plain text HTTP requests, but the Envoy sidecar intercepts the traffic and implements Mutual TLS encryption to protect against interception or modification.

Istio manages Splunk’s ingress gateways, which receive traffic from public and internal NLBs. The gateways are managed by the platform team and run in the Istio Gateway namespace, allowing users to plug into them, but not modify them. The Gateway service is also provisioned with certificates to enforce TLS by default, and Validating Webhooks ensure that services can only connect to gateways for their own hostnames. Additionally, gateways enforce request authentication at ingress, before traffic is able to impact application pods.

Because Istio and related K8S objects are relatively complex to configure, Splunk created an abstraction layer, which is a controller that configures everything for the service, including virtual services, destination rules, gateways, certificates, and more. It sets up DNS that goes directly to the right NLB. It’s a one-click solution for end-to-end network deployment. For more complex use cases, the services teams can still bypass the abstraction and configure these settings directly.

Pain Points

While Splunk’s architecture meets many of our needs, there are a few pain points worth discussing. Istio operates by creating as many Envoy Sidecars as application pods, which is an inefficient use of resources. In addition, when a particular application has unique needs from its sidecar, such as additional CPU or Memory, it can be difficult to adjust these settings without adjusting them for all sidecars in the mesh. Istio Sidecar injection involves a lot of magic, using a mutating webhook to add a sidecar container to every pod as it is created, which means those pods no longer match their corresponding deployments. Additionally, injection can only happen at pod creation time, which means that any time a sidecar version or parameter is updated, all pods must be restarted before they will get the new settings. Overall, this magic complicates running a service mesh in production, and adds a great deal of operational uncertainty to your application.

The Istio project is aware of these limitations, and believes they will be substantially improved by the new Ambient mode for Istio. In this mode, Layer 4 constructs like identity and encryption will be applied by a Daemon running on the node, but not in the same pod as the application. Layer 7 features will still be handled by Envoy, but Envoy will be run in an adjacent pod as part of its own deployment, rather than relying on the magic of sidecar injection. Application pods will not be modified in any way in ambient mode, which should add a good deal of predictability to service mesh operations. Ambient mode is expected to reach Alpha quality in Istio 1.18.

Conclusion

With all these layers to network security at Splunk Cloud, it is helpful to take a step back and examine the life of a request as it traverses these layers. When a client sends a request, they first connect to the NLB, which will be allowed or blocked by the VPC ACLs. The NLB then proxies the request to one of the ingress nodes, which terminates TLS and inspects the request at Layer 7, choosing to allow or block the request. The Envoy Gateway then validates the request using ExtAuthZ to ensure it is properly authenticated, and meets quota restrictions before being allowed into the cluster. Next, the Envoy Gateway proxies the request upstream, and the network policies from Kubernetes take effect again to make sure this proxying is allowed. The upstream sidecar on the workload inspects the Layer 7 requests and if allowed, it will decrypt the request and send it to the workload in clear text.

Securing Splunk’s Cloud Native Network Stack while meeting the scalability needs of this large enterprise company requires careful security planning at each layer.

While applying identity, observability, and policy principles at every layer in the stack may appear redundant at first glance, each layer is able to make up for the shortcomings of the others, so that together these layers form a tight and effective barrier to unwanted access.

If you are interested in diving deeper into Splunk’s Network Security Stack, you can watch our Cloud Native SecurityCon presentation.

Architecture of waypoint proxies

Similar to sidecar, the waypoint proxy is also Envoy-based and is dynamically configured by Istio to serve your applications configuration. What is unique about the waypoint proxy is that it runs either per-namespace (default) or per-service account. By running outside of the application pod, a waypoint proxy can install, upgrade, and scale independently from the application, as well as reduce operational costs.

Waypoint proxies are deployed declaratively using Kubernetes Gateway resources or the helpful istioctl command:

$ istioctl experimental waypoint generate
apiVersion: gateway.networking.k8s.io/v1beta1
kind: Gateway
metadata:
  name: namespace
spec:
  gatewayClassName: istio-waypoint
  listeners:
  - name: mesh
    port: 15008
    protocol: HBONE

Istiod will monitor these resources and deploy and manage the corresponding waypoint deployment for users automatically.

Shift source proxy configuration to destination proxy

In the existing sidecar architecture, most traffic-shaping (for example request routing or traffic shifting or fault injection) policies are implemented by the source (client) proxy while most security policies are implemented by the destination (server) proxy. This leads to a number of concerns:

• Scaling - each source sidecar needs to know information about every other destination in the mesh. This is a polynomial scaling problem. Worse, if any destination configuration changes, we need to notify all sidecars at once.
• Debugging - because policy enforcement is split between the client and server sidecars, it can be hard to understand the behavior of the system when troubleshooting.
• Mixed environments - if we have systems where not all clients are part of the mesh, we get inconsistent behavior. For example, a non-mesh client wouldn’t respect a canary rollout policy, leading to unexpected traffic distribution.
• Ownership and attribution - ideally a policy written in one namespace should only affect work done by proxies running in the same namespace. However, in this model, it is distributed and enforced by each sidecar. While Istio has designed around this constraint to make this secure, it is still not optimal.

In ambient, all policies are enforced by the destination waypoint. In many ways, the waypoint acts as a gateway into the namespace (default scope) or service account. Istio enforces that all traffic coming into the namespace goes through the waypoint, which then enforces all policies for that namespace. Because of this, each waypoint only needs to know about configuration for its own namespace.

The scalability problem, in particular, is a nuisance for users running in large clusters. If we visualize it, we can see just how big an improvement the new architecture is.

Consider a simple deployment, where we have 2 namespaces, each with 2 (color coded) deployments. The Envoy (XDS) configuration required to program the sidecars is shown as circles:

In the sidecar model, we have 4 workloads, each with 4 sets of configuration. If any of those configurations changed, all of them would need to be updated. In total there are 16 configurations distributed.

In the waypoint architecture, however, the configuration is dramatically simplified:

Here, we see a very different story. We have only 2 waypoint proxies, as each one is able to serve the entire namespace, and each one only needs configuration for its own namespace. In total we have 25% of the amount of configuration sent, even for a simple example.

If we scale each namespace up to 25 deployments with 10 pods each and each waypoint deployment with 2 pods for high availability, the numbers are even more impressive - the waypoint config distribution requires just 0.8% of the configuration distribution of the sidecar, as the table below illustrates!

While we use namespace scoped waypoint proxies to illustrate the simplification above, the simplification is similar when you apply it to service account waypoint proxies.

This reduced configuration means lower resource usage (CPU, RAM, and network bandwidth) for both the control plane and data plane. While users today can see similar improvements with careful usage of exportTo in their Istio networking resources or of the Sidecar API, in ambient mode this is no longer required, making scaling a breeze.

What if my destination doesn’t have a waypoint proxy?

The design of ambient mode centers around the assumption that most configuration is best implemented by the service producer, rather than the service consumer. However, this isn’t always the case - sometimes we need to configure traffic management for destinations we don’t control. A common example of this would be connecting to an external service with improved resilience to handle occasional connection issues (e.g., to add a timeout for calls to example.com).

This is an area under active development in the community, where we design how traffic can be routed to your egress gateway and how you can configure the egress gateway with your desired policies. Look out for future blog posts in this area!

A deep-dive of waypoint configuration

Assuming you have followed the ambient get started guide up to and including the control traffic section, you have deployed a waypoint proxy for the bookinfo-reviews service account to direct 90% traffic to reviews v1 and 10% traffic to reviews v2.

Use istioctl to retrieve the listeners for the reviews waypoint proxy:

$ istioctl proxy-config listener deploy/bookinfo-reviews-istio-waypoint --waypoint
LISTENER              CHAIN                                                 MATCH                                         DESTINATION
envoy://connect_originate                                                       ALL                                           Cluster: connect_originate
envoy://main_internal inbound-vip|9080||reviews.default.svc.cluster.local-http  ip=10.96.104.108 -> port=9080                 Inline Route: /*
envoy://main_internal direct-tcp                                            ip=10.244.2.14 -> ANY                         Cluster: encap
envoy://main_internal direct-tcp                                            ip=10.244.1.6 -> ANY                          Cluster: encap
envoy://main_internal direct-tcp                                            ip=10.244.2.11 -> ANY                         Cluster: encap
envoy://main_internal direct-http                                           ip=10.244.2.11 -> application-protocol='h2c'  Cluster: encap
envoy://main_internal direct-http                                           ip=10.244.2.11 -> application-protocol='http/1.1' Cluster: encap
envoy://main_internal direct-http                                           ip=10.244.2.14 -> application-protocol='http/1.1' Cluster: encap
envoy://main_internal direct-http                                           ip=10.244.2.14 -> application-protocol='h2c'  Cluster: encap
envoy://main_internal direct-http                                           ip=10.244.1.6 -> application-protocol='h2c'   Cluster: encap
envoy://main_internal direct-http                                           ip=10.244.1.6 -> application-protocol='http/1.1'  Cluster: encap
envoy://connect_terminate default                                               ALL                                           Inline Route:

For requests arriving on port 15008, which by default is Istio’s inbound HBONE port, the waypoint proxy terminates the HBONE connection and forwards the request to the main_internal listener to enforce any workload policies such as AuthorizationPolicy. If you are not familiar with internal listeners, they are Envoy listeners that accepts user space connections without using the system network API. The --waypoint flag added to the istioctl proxy-config command, above, instructs it to show the details of the main_internal listener, its filter chains, chain matches, and destinations.

Note 10.96.104.108 is the reviews’ service VIP and 10.244.x.x are the reviews’ v1/v2/v3 pod IPs, which you can view for your cluster using the kubectl get svc,pod -o wide command. For plain text or HBONE terminated inbound traffic, it will be matched on the service VIP and port 9080 for reviews or by pod IP address and application protocol (either ANY, h2c, or http/1.1).

Checking out the clusters for the reviews waypoint proxy, you get the main_internal cluster along with a few inbound clusters. Other than the clusters for infrastructure, the only Envoy clusters created are for services and pods running in the same service account. No clusters are created for services or pods running elsewhere.

$ istioctl proxy-config clusters deploy/bookinfo-reviews-istio-waypoint
SERVICE FQDN                         PORT SUBSET  DIRECTION   TYPE         DESTINATION RULE
agent                                -    -       -           STATIC
connect_originate                    -    -       -           ORIGINAL_DST
encap                                -    -       -           STATIC
kubernetes.default.svc.cluster.local 443  tcp     inbound-vip EDS
main_internal                        -    -       -           STATIC
prometheus_stats                     -    -       -           STATIC
reviews.default.svc.cluster.local    9080 http    inbound-vip EDS
reviews.default.svc.cluster.local    9080 http/v1 inbound-vip EDS
reviews.default.svc.cluster.local    9080 http/v2 inbound-vip EDS
reviews.default.svc.cluster.local    9080 http/v3 inbound-vip EDS
sds-grpc                             -    -       -           STATIC
xds-grpc                             -    -       -           STATIC
zipkin                               -    -       -           STRICT_DNS

Note that there are no outbound clusters in the list, which you can confirm using istioctl proxy-config cluster deploy/bookinfo-reviews-istio-waypoint --direction outbound! What’s nice is that you didn’t need to configure exportTo on any other bookinfo services (for example, the productpage or ratings services). In other words, the reviews waypoint is not made aware of any unnecessary clusters, without any extra manual configuration from you.

Display the list of routes for the reviews waypoint proxy:

$ istioctl proxy-config routes deploy/bookinfo-reviews-istio-waypoint
NAME                                                    DOMAINS MATCH              VIRTUAL SERVICE
encap                                                   *       /*
inbound-vip|9080|http|reviews.default.svc.cluster.local *       /*                 reviews.default
default

Recall that you didn’t configure any Sidecar resources or exportTo configuration on your Istio networking resources. You did, however, deploy the bookinfo-productpage route to configure an ingress gateway to route to productpage but the reviews waypoint has not been made aware of any such irrelevant routes.

Displaying the detailed information for the inbound-vip|9080|http|reviews.default.svc.cluster.local route, you’ll see the weight-based routing configuration directing 90% of the traffic to reviews v1 and 10% of the traffic to reviews v2, along with some of Istio’s default retry and timeout configurations. This confirms the traffic and resiliency policies are shifted from the source to destination oriented waypoint as discussed earlier.

$ istioctl proxy-config routes deploy/bookinfo-reviews-istio-waypoint --name "inbound-vip|9080|http|reviews.default.svc.cluster.local" -o yaml
- name: inbound-vip|9080|http|reviews.default.svc.cluster.local
 validateClusters: false
 virtualHosts:
 - domains:
   - '*'
   name: inbound|http|9080
   routes:
   - decorator:
       operation: reviews:9080/*
     match:
       prefix: /
     metadata:
       filterMetadata:
         istio:
           config: /apis/networking.istio.io/v1alpha3/namespaces/default/virtual-service/reviews
     route:
       maxGrpcTimeout: 0s
       retryPolicy:
         hostSelectionRetryMaxAttempts: "5"
         numRetries: 2
         retriableStatusCodes:
         - 503
         retryHostPredicate:
         - name: envoy.retry_host_predicates.previous_hosts
           typedConfig:
             '@type': type.googleapis.com/envoy.extensions.retry.host.previous_hosts.v3.PreviousHostsPredicate
         retryOn: connect-failure,refused-stream,unavailable,cancelled,retriable-status-codes
       timeout: 0s
       weightedClusters:
         clusters:
         - name: inbound-vip|9080|http/v1|reviews.default.svc.cluster.local
           weight: 90
         - name: inbound-vip|9080|http/v2|reviews.default.svc.cluster.local
           weight: 10

Check out the endpoints for reviews waypoint proxy:

$ istioctl proxy-config endpoints deploy/bookinfo-reviews-istio-waypoint
ENDPOINT                                            STATUS  OUTLIER CHECK CLUSTER
127.0.0.1:15000                                     HEALTHY OK            prometheus_stats
127.0.0.1:15020                                     HEALTHY OK            agent
envoy://connect_originate/                          HEALTHY OK            encap
envoy://connect_originate/10.244.1.6:9080           HEALTHY OK            inbound-vip|9080|http/v2|reviews.default.svc.cluster.local
envoy://connect_originate/10.244.1.6:9080           HEALTHY OK            inbound-vip|9080|http|reviews.default.svc.cluster.local
envoy://connect_originate/10.244.2.11:9080          HEALTHY OK            inbound-vip|9080|http/v1|reviews.default.svc.cluster.local
envoy://connect_originate/10.244.2.11:9080          HEALTHY OK            inbound-vip|9080|http|reviews.default.svc.cluster.local
envoy://connect_originate/10.244.2.14:9080          HEALTHY OK            inbound-vip|9080|http/v3|reviews.default.svc.cluster.local
envoy://connect_originate/10.244.2.14:9080          HEALTHY OK            inbound-vip|9080|http|reviews.default.svc.cluster.local
envoy://main_internal/                              HEALTHY OK            main_internal
unix://./etc/istio/proxy/XDS                        HEALTHY OK            xds-grpc
unix://./var/run/secrets/workload-spiffe-uds/socket HEALTHY OK            sds-grpc

Note that you don’t get any endpoints related to any services other than reviews, even though you have a few other services in the default and istio-system namespace.

Wrapping up

We are very excited about the waypoint simplification focusing on destination oriented waypoint proxies. This is another significant step towards simplifying Istio’s usability, scalability and debuggability which are top priorities on Istio’s roadmap. Follow our getting started guide to try the ambient alpha build today and experience the simplified waypoint proxy!

In Istio’s new ambient mode, the istio-cni component running on each Kubernetes worker node is responsible for redirecting application traffic to the zero-trust tunnel (ztunnel) on that node. By default it relies on iptables and Generic Network Virtualization Encapsulation (Geneve) overlay tunnels to achieve this redirection. We have now added support for an eBPF-based method of traffic redirection.

Why eBPF

Although performance considerations are essential in the implementation of Istio ambient mode redirection, it’s also important to consider ease of programmability, to enable the implementation of versatile and customized requirements. With eBPF, you can leverage additional context in the kernel to bypass complex routing and simply send packets to their final destination.

Furthermore, eBPF enables deeper visibility and additional context for packets in the kernel, allowing for more efficient and flexible management of data flow compared with iptables.

How it works

An eBPF program, attached to the traffic control ingress and egress hook, has been compiled into the Istio CNI component. istio-cni will watch pod events and attach/detach the eBPF program to other related network interfaces when the pod is moved into or out of ambient mode.

Using an eBPF program (instead of iptables) eliminates the need to encapsulate tasks (for Geneve), allowing the routing tasks to be customized in kernel space instead. This yields both gains in performance, and additional flexibility, in routing.

All traffic to/from the application pod will be intercepted by eBPF and redirected to the corresponding ztunnel pod. On the ztunnel side, proper redirection will be performed based on connection lookup results within the eBPF program. This provides more efficient control of the network traffic between the application and ztunnel.

How to enable eBPF redirection in Istio ambient mode

Follow the instructions in Getting Started with Ambient Mesh to set up your cluster, with a small change: when you install Istio, set the values.cni.ambient.redirectMode configuration parameter to ebpf.

$ istioctl install --set profile=ambient --set values.cni.ambient.redirectMode="ebpf"

Check the istio-cni logs to confirm eBPF redirection is on:

ambient Writing ambient config: {"ztunnelReady":true,"redirectMode":"eBPF"}

Performance gains

The latency and throughput (QPS) for eBPF redirection are somewhat better than using iptables. The following tests were run in a kind cluster with a Fortio client sending requests to a Fortio server, both running in ambient mode (with eBPF debug logging disabled) and on the same Kubernetes worker node.

$ fortio load -uniform -t 60s -qps 0 -c <num_connections> http://<fortio-svc-name>:8080

$ fortio load -uniform -t 60s -qps 8000 -c <num_connections> http://<fortio-svc-name>:8080

Wrapping up

Both eBPF and iptables have their own advantages and disadvantages when it comes to traffic redirection. eBPF is a modern, flexible, and powerful alternative that allows for more customization in rule creation and offers better performance. However, it does require a modern kernel version (4.20 or later for redirection case) which may not be available on some systems. On the other hand, iptables is widely used and compatible with most Linux distributions, even those with older kernels. However, it lacks the flexibility and extensibility of eBPF and may have lower performance.

Ultimately, the choice between eBPF and iptables for traffic redirection will depend on the specific needs and requirements of the system, as well as the user’s level of expertise in using each tool. Some users may prefer the simplicity and compatibility of iptables, while others may require the flexibility and performance of eBPF.

There is still plenty of work to be done, including integration with various CNI plugins, and contributions to improve the ease of use would be greatly welcomed. Please join us in #ambient on the Istio slack.

Background

The journey has taken us longer than anticipated and we continue to have work to do. The team initially started with a design based on a reference implementation from F5. The design led to an RFC that caused us to re-examine our approach. Notably, there were concerns about memory and performance issues that the community wanted to be addressed before implementation. The original design had to duplicate Envoy configuration for listeners, clusters, routes and endpoints. Given that many people already experience Envoy memory and CPU consumption issues, early feedback wanted us to completely re-evaluate this approach. Many proxies transparently handle outbound dual-stack traffic regardless of how the traffic was originated. Much of the earliest feedback was to implement the same behavior in Istio and Envoy.

Redefining Dual Stack Support

Much of the feedback provided by the community for the original RFC was to update Envoy to better support dual-stack use cases internally instead of supporting this within Istio. This has led us to a new design where we have taken lessons learned as well as feedback and have applied them to fit a simplified design.

Support for Dual Stack in Istio 1.17

We have worked with the Envoy community to address numerous concerns which is a reason why dual-stack enablement has taken us a while to implement. We have implemented matched IP Family for outbound listener and supported multiple addresses per listener. Alex Xu has also been working fervently to get long outstanding issues resolved, with the ability for Envoy to have a smarter way to pick endpoints for dual-stack. Some of these improvements to Envoy, such as the ability to enable socket options on multiple addresses, have landed in the Istio 1.17 release (e.g. extra source addresses on inbound clusters).

The Envoy API changes made by the team can be found at their site at Listener addresses and bind config. Making sure we can have proper support at both the downstream and upstream connection for Envoy is important for realizing dual-stack support.

In total the team has submitted over a dozen PRs to Envoy and are working on at least a half dozen more to make Envoy adoption of dual stack easier for Istio.

Meanwhile, on the Istio side you can track the progress in Issue #40394. Progress has slowed down a bit lately as we continue working with Envoy on various issues, however, we are happy to announce experimental support for dual stack in Istio 1.17!

A Quick Experiment using Dual Stack

1. Enable dual stack experimental support on Istio 1.17.0+ with the following:

   $ istioctl install -y -f - <<EOF
   apiVersion: install.istio.io/v1alpha1
   kind: IstioOperator
   spec:
     meshConfig:
       defaultConfig:
         proxyMetadata:
           ISTIO_DUAL_STACK: "true"
     values:
       pilot:
         env:
           ISTIO_DUAL_STACK: "true"
   EOF

2. Create three namespaces:

   • dual-stack: tcp-echo will listen on both an IPv4 and IPv6 address.
   • ipv4: tcp-echo will listen on only an IPv4 address.
   • ipv6: tcp-echo will listen on only an IPv6 address.

   $ kubectl create namespace dual-stack
   $ kubectl create namespace ipv4
   $ kubectl create namespace ipv6

3. Enable sidecar injection on all of those namespaces as well as the default namespace:

   $ kubectl label --overwrite namespace default istio-injection=enabled
   $ kubectl label --overwrite namespace dual-stack istio-injection=enabled
   $ kubectl label --overwrite namespace ipv4 istio-injection=enabled
   $ kubectl label --overwrite namespace ipv6 istio-injection=enabled

4. Create tcp-echo deployments in the namespaces:

   $ kubectl apply --namespace dual-stack -f https://raw.githubusercontent.com/istio/istio/release-1.29/samples/tcp-echo/tcp-echo-dual-stack.yaml
   $ kubectl apply --namespace ipv4 -f https://raw.githubusercontent.com/istio/istio/release-1.29/samples/tcp-echo/tcp-echo-ipv4.yaml
   $ kubectl apply --namespace ipv6 -f https://raw.githubusercontent.com/istio/istio/release-1.29/samples/tcp-echo/tcp-echo-ipv6.yaml

5. Create sleep deployment in the default namespace:

   $ kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.29/samples/sleep/sleep.yaml

6. Verify the traffic:

   $ kubectl exec -it "$(kubectl get pod -l app=sleep -o jsonpath='{.items[0].metadata.name}')" -- sh -c "echo dualstack | nc tcp-echo.dual-stack 9000"
   hello dualstack
   $ kubectl exec -it "$(kubectl get pod -l app=sleep -o jsonpath='{.items[0].metadata.name}')" -- sh -c "echo ipv4 | nc tcp-echo.ipv4 9000"
   hello ipv4
   $ kubectl exec -it "$(kubectl get pod -l app=sleep -o jsonpath='{.items[0].metadata.name}')" -- sh -c "echo ipv6 | nc tcp-echo.ipv6 9000"
   hello ipv6

Now you can experiment with dual-stack services in your environment!

Important Changes to Listeners and Endpoints

For the above experiment, you’ll notice changes are made to listeners and routes:

$ istioctl proxy-config listeners "$(kubectl get pod -n dual-stack -l app=tcp-echo -o jsonpath='{.items[0].metadata.name}')" -n dual-stack --port 9000

You will see listeners are now bound to multiple addresses, but only for dual stack services. Other services will only be listening on a single IP address.

"name": "fd00:10:96::f9fc_9000",
"address": {
    "socketAddress": {
        "address": "fd00:10:96::f9fc",
        "portValue": 9000
    }
},
"additionalAddresses": [
    {
        "address": {
            "socketAddress": {
                "address": "10.96.106.11",
                "portValue": 9000
            }
        }
    }
],

Virtual inbound addresses are now also configured to listen on both 0.0.0.0 and [::].

"name": "virtualInbound",
"address": {
    "socketAddress": {
        "address": "0.0.0.0",
        "portValue": 15006
    }
},
"additionalAddresses": [
    {
        "address": {
            "socketAddress": {
                "address": "::",
                "portValue": 15006
            }
        }
    }
],

Envoy’s endpoints now are configured to route to both IPv4 and IPv6:

$ istioctl proxy-config endpoints "$(kubectl get pod -l app=sleep -o jsonpath='{.items[0].metadata.name}')" --port 9000
ENDPOINT                 STATUS      OUTLIER CHECK     CLUSTER
10.244.0.19:9000         HEALTHY     OK                outbound|9000||tcp-echo.ipv4.svc.cluster.local
10.244.0.26:9000         HEALTHY     OK                outbound|9000||tcp-echo.dual-stack.svc.cluster.local
fd00:10:244::1a:9000     HEALTHY     OK                outbound|9000||tcp-echo.dual-stack.svc.cluster.local
fd00:10:244::18:9000     HEALTHY     OK                outbound|9000||tcp-echo.ipv6.svc.cluster.local

Get Involved

Plenty of work remains, and you are welcome to help us with the remaining tasks needed for dual stack support to get to Alpha here.

For instance, Iris Ding (Intel) and Li Chun (Intel) are already working with the community for getting redirection of network traffic for ambient, and we are hoping to have ambient support dual stack for its upcoming alpha release in Istio 1.18.

We would love your feedback and if you are eager to work with us please stop by our slack channel, #dual-stack within the Istio Slack.

Thank you to the team that has worked on Istio dual-stack!

• Intel: Steve Zhang, Alex Xu, Iris Ding
• F5: Jacob Delgado
• Yingchun Cai (formerly of F5)

Major Changes from the Initial Launch

Ambient mesh is designed for simplified operations, broader application compatibility, and reduced infrastructure cost. The ultimate goal of ambient is to be transparent to your applications and we have made a few changes to make the ztunnel and waypoint components simpler and lightweight.

• The ztunnel component has been rewritten from the ground up to be fast, secure, and lightweight. Refer to Introducing Rust-Based Ztunnel for Istio Ambient Service Mesh for more information.
• We made significant changes to simplify waypoint proxy’s configuration to improve its debuggability and performance. Refer to Istio Ambient Waypoint Proxy Made Simple for more information.
• Added the istioctl x waypoint command to help you conveniently deploy waypoint proxies, along with istioctl pc workload to help you view workload information.
• We gave users the ability to explicitly bind Istio policies such as AuthorizationPolicy to waypoint proxies vs selecting the destination workload.

Get involved

Follow our getting started guide to try the ambient pre-alpha build today. We’d love to hear from you! To learn more about ambient:

• Join us in the #ambient and #ambient-dev channel in Istio’s slack.
• Attend the weekly ambient contributor meeting on Wednesdays.
• Check out the Istio and ztunnel repositories, submit issues or PRs!

Because ztunnel is designed to run on all of your Kubernetes worker nodes, it is critical to keep its resource footprint small. Ztunnel is designed to be an invisible (or “ambient”) part of your service mesh with minimal impact on your workloads.

Ztunnel architecture

Similar to sidecars, ztunnel also serves as an xDS client and CA client:

1. During startup, it securely connects to the Istiod control plane using its service account token. Once the connection from ztunnel to Istiod is established securely using TLS, it starts to fetch xDS configuration as an xDS client. This works similarly to sidecars or gateways or waypoint proxies, except that Istiod recognizes the request from ztunnel and sends the purpose-built xDS configuration for ztunnel, which you will learn more about soon.
2. It also serves as a CA client to manage and provision mTLS certificates on behalf of all co-located workloads it manages.
3. As traffic comes in or goes out, it serves as a core proxy that handles the inbound and outbound traffic (either out-of-mesh plain text or in-mesh HBONE) for all co-located workloads it manages.
4. It provides L4 telemetry (metrics and logs) along with an admin server with debugging information to help you debug ztunnel if needed.

Why not reuse Envoy?

When Istio ambient service mesh was announced on Sept 7, 2022, the ztunnel was implemented using an Envoy proxy. Given that we use Envoy for the rest of Istio - sidecars, gateways, and waypoint proxies - it was natural for us to start implementing ztunnel using Envoy.

However, we found that while Envoy was a great fit for other use cases, it was challenging to implement ztunnel in Envoy, as many of the tradeoffs, requirements, and use cases are dramatically different than that of a sidecar proxy or ingress gateway. In addition, most of the things that make Envoy such a great fit for those other use cases, such as its rich L7 feature set and extensibility, went to waste in ztunnel which didn’t need those features.

A purpose-built ztunnel

After having trouble bending Envoy to our needs, we started investigating making a purpose-built implementation of the ztunnel. Our hypothesis was that by designing with a single focused use case in mind from the beginning, we could develop a solution that was simpler and more performant than molding a general purpose project to our bespoke use cases. The explicit decision to make ztunnel simple was key to this hypothesis; similar logic wouldn’t hold up to rewriting the gateway, for example, which has a huge list of supported features and integrations.

This purpose-built ztunnel involved two key areas:

• The configuration protocol between ztunnel and its Istiod
• The runtime implementation of ztunnel

Configuration protocol

Envoy proxies use the xDS Protocol for configuration. This is a key part of what makes Istio work well, offering rich and dynamic configuration updates. However, as we tread off the beaten path, the config becomes more and more bespoke, which means it’s much larger and more expensive to generate. In a sidecar, a single Service with 1 pod, generates roughly ~350 lines of xDS (in YAML), which already has been challenging to scale. The Envoy-based ztunnel was far worse, and in some areas had N^2 scaling attributes.

To keep the ztunnel configuration as small as possible, we investigated using a purpose built configuration protocol, that contains precisely the information we need (and nothing more), in an efficient format. For example, a single pod could be represented concisely:

name: helloworld-v1-55446d46d8-ntdbk
namespace: default
serviceAccount: helloworld
node: ambient-worker2
protocol: TCP
status: Healthy
waypointAddresses: []
workloadIp: 10.244.2.8
canonicalName: helloworld
canonicalRevision: v1
workloadName: helloworld-v1
workloadType: deployment

This information is transported over the xDS transport API, but uses a custom ambient-specific type. Refer to the workload xDS configuration section to learn more about the configuration details.

By having a purpose built API, we can push logic into the proxy instead of in Envoy configuration. For example, to configure mTLS in Envoy, we need to add an identical large set of configuration tuning the precise TLS settings for each service; with ztunnel, we need only a single enum to declare whether mTLS should be used or not. The rest of the complex logic is embedded directly into ztunnel code.

With this efficient API between Istiod and ztunnel, we found we could configure ztunnels with information about large meshes (such as those with 100,000 pods) with orders of magnitude less configuration, which means less CPU, memory, and network costs.

Runtime implementation

As the name suggests, ztunnel uses an HTTPS tunnel to carry users requests. While Envoy supports this tunneling, we found the configuration model limiting for our needs. Roughly speaking, Envoy operates by sending requests through a series of “filters”, starting with accepting a request and ending with sending a request. With our requirements, which have multiple layers of requests (the tunnel itself and the users’ requests), as well as a need to apply per-pod policy after load balancing, we found we would need to loop through these filters 4 times per connection when implementing our prior Envoy-based ztunnel. While Envoy has some optimizations for essentially “sending a request to itself” in memory, this was still very complex and expensive.

By building out our own implementation, we could design around these constraints from the ground up. In addition, we have more flexibility in all aspects of the design. For example, we could choose to share connections across threads or implement more bespoke requirements around isolation between service accounts. After establishing that a purpose built proxy was viable, we set out to choose the implementation details.

A Rust-based ztunnel

With the goal to make ztunnel fast, secure, and lightweight, Rust was an obvious choice. However, it wasn’t our first. Given Istio’s current extensive usage of Go, we had hoped we could make a Go-based implementation meet these goals. In initial prototypes, we built out some simple versions of both a Go-based implementation as well as a Rust-based one. From our tests, we found that the Go-based version didn’t meet our performance and footprint requirements. While it’s likely we could have optimized it further, we felt that a Rust-based proxy would give us the long-term optimal implementation.

A C++ implementation – likely reusing parts of Envoy – was also considered. However, this option was not pursued due to lack of memory safety, developer experience concerns, and a general industry trend towards Rust.

This process of elimination left us with Rust, which was a perfect fit. Rust has a strong history of success in high performance, low resource utilization applications, especially in network applications (including service mesh). We chose to build on top of the Tokio and Hyper libraries, two of the de-facto standards in the ecosystem that are extensively battle-tested and easy to write highly performant asynchronous code with.

A quick tour of the Rust-based ztunnel

Workload xDS configuration

The workload xDS configurations are very easy to understand and debug. You can view them by sending a request to localhost:15000/config_dump from one of your ztunnel pods, or use the convenient istioctl pc workload command. There are two key workload xDS configurations: workloads and policies.

Before your workloads are included in your ambient mesh, you will still be able to see them in ztunnel’s config dump, as ztunnel is aware of all of the workloads regardless of whether they are ambient enabled or not. For example, below contains a sample workload configuration for a newly deployed helloworld v1 pod which is out-of-mesh indicated by protocol: TCP:

{
  "workloads": {
    "10.244.2.8": {
      "workloadIp": "10.244.2.8",
      "protocol": "TCP",
      "name": "helloworld-v1-cross-node-55446d46d8-ntdbk",
      "namespace": "default",
      "serviceAccount": "helloworld",
      "workloadName": "helloworld-v1-cross-node",
      "workloadType": "deployment",
      "canonicalName": "helloworld",
      "canonicalRevision": "v1",
      "node": "ambient-worker2",
      "authorizationPolicies": [],
      "status": "Healthy"
    }
  }
}

After the pod is included in ambient (by labeling the namespace default with istio.io/dataplane-mode=ambient), the protocol value is replaced with HBONE, instructing ztunnel to upgrade all incoming and outgoing communications from the helloworld-v1 pod to be HBONE.

{
  "workloads": {
    "10.244.2.8": {
      "workloadIp": "10.244.2.8",
      "protocol": "HBONE",
      ...
}

After you deploy any workload level authorization policy, the policy configuration will be pushed as xDS configuration from Istiod to ztunnel and shown under policies:

{
  "policies": {
    "default/hw-viewer": {
      "name": "hw-viewer",
      "namespace": "default",
      "scope": "WorkloadSelector",
      "action": "Allow",
      "groups": [[[{
        "principals": [{"Exact": "cluster.local/ns/default/sa/sleep"}]
      }]]]
    }
  }
  ...
}

You’ll also notice the workload’s configuration is updated with reference to the authorization policy.

{
  "workloads": {
    "10.244.2.8": {
    "workloadIp": "10.244.2.8",
    ...
    "authorizationPolicies": [
        "default/hw-viewer"
    ],
  }
  ...
}

L4 telemetry provided by ztunnel

You may be pleasantly surprised that the ztunnel logs are easy to understand. For example, you’ll see the HTTP Connect request on the destination ztunnel that indicates the source pod IP (peer_ip) and destination pod IP.

2023-02-15T20:40:48.628251Z  INFO inbound{id=4399fa68cf25b8ebccd472d320ba733f peer_ip=10.244.2.5 peer_id=spiffe://cluster.local/ns/default/sa/sleep}: ztunnel::proxy::inbound: got CONNECT request to 10.244.2.8:5000

You can view L4 metrics of your workloads by accessing the localhost:15020/metrics API which provides the full set of TCP standard metrics, with same labels that sidecars expose. For example:

istio_tcp_connections_opened_total{
  reporter="source",
  source_workload="sleep",
  source_workload_namespace="default",
  source_principal="spiffe://cluster.local/ns/default/sa/sleep",
  destination_workload="helloworld-v1",
  destination_workload_namespace="default",
  destination_principal="spiffe://cluster.local/ns/default/sa/helloworld",
  request_protocol="tcp",
  connection_security_policy="mutual_tls"
  ...
} 1

If you install Prometheus and Kiali, you can view these metrics easily from Kiali’s UI.

Wrapping up

We are super excited that the new Rust-based ztunnel is drastically simplified, more lightweight and performant than the prior Envoy-based ztunnel. With the purposefully designed workload xDS for the Rust-based ztunnel, you’ll not only be able to understand the xDS configuration much more easily, but also have drastically reduced network traffic and cost between the Istiod control plane and ztunnels. With Istio ambient now merged to upstream master, you can try the new Rust-based ztunnel by following our getting started guide.

Last year, we elected four members to the community seats. It’s now time to announce the companies who fuel our growth by selecting the Contribution Seat members. As per the Steering charter, every February we look at which companies have made the most contributions to Istio based on an annually agreed metric.

According to our seat allocation process, this year Google will be allocated 5 seats and IBM/Red Hat will be allocated 2. As the third largest contributor to Istio in the last 12 months, we are pleased to announce that Huawei has earned two Contribution Seats.

Based on this, here is the complete list of Istio Steering Committee members, including both the Contribution and Community Seats:

• Ameer Abbas (Google)
• Craig Box (ARMO)
• Rob Cernich (Red Hat)
• Iris Ding (Intel)
• Cameron Etezadi (Google)
• Jianpeng He (Huawei)
• John Howard (Google)
• Faseela K (Ericsson Software Technology)
• April Kyle Nassi(Google)
• Justin Pettit (Google)
• Christian Posta (Solo.io)
• Cale Rath (IBM)
• Zhonghu Xu (Huawei)

Our sincerest thanks to Louis Ryan, Srihari Angaluri, Kebe Liu and Jason McGee, all long-time contributors to the Istio project, whose terms have come to an end.

The auditors’ assessment was that “Istio is a well-maintained project that has a strong and sustainable approach to security”. No critical issues were found; the highlight of the report was the discovery of a vulnerability in the Go programming language.

We would like to thank the Cloud Native Computing Foundation for funding this work, as a benefit offered to us after we joined the CNCF in August. It was arranged by OSTIF, and performed by ADA Logics.

Scope and overall findings

Istio received its first security assessment in 2020, with its data plane, the Envoy proxy, having been independently assessed in 2018 and 2021. The Istio Product Security Working Group and ADA Logics therefore decided on the following scope:

• Produce a formal threat model, to guide this and future security audits
• Carry out a manual code audit for security issues
• Review the fixes for the issues found in the 2020 audit
• Review and improve Istio’s fuzzing suite
• Perform a SLSA review of Istio

Once again, no Critical issues were found in the review. The assessment found 11 security issues; two High, four Medium, four Low and one informational. All the reported issues have been fixed.

Aside from their observations above, the auditors note that Istio follows a high level of industry standards in dealing with security. In particular, they highlight that:

• The Istio Product Security Working Group responds swiftly to security disclosures
• The documentation on the project’s security is comprehensive, well-written and up to date
• Security vulnerability disclosures follow industry standards and security advisories are clear and detailed
• Security fixes include regression tests

Resolution and learnings

Request smuggling vulnerability in Go

The auditors uncovered a situation where Istio could accept traffic using HTTP/2 Over Cleartext (h2c), a method of making an unencrypted connection with HTTP/1.1 and then upgrading to HTTP/2. The Go library for h2c connections reads the entire request into memory, and notes that if you wish to avoid this, the request should be wrapped in a MaxBytesHandler.

In fixing this bug, Istio TOC member John Howard noticed that the recommended fix introduces a request smuggling vulnerability. The Go team thus published CVE-2022-41721 — the only vulnerability discovered by this audit!

Istio has since been changed to disable h2c upgrade support throughout.

Improvements to file fetching

The most common class of issue found were related to Istio fetching files over a network (for example, the Istio Operator installing Helm charts, or the WebAssembly module downloader):

• A crafted Helm chart could exhaust disk space (#1) or overwrite other files in the Operator’s pod (#2)
• File handles were not closed in the case of an error, and could be exhausted (#3)
• Crafted files could exhaust memory (#4 and #5)

To execute these code paths, an attacker would need enough privilege to either specify a URL for a Helm chart or a WebAssembly module. With such access, they would not need an exploit: they could already cause an arbitrary chart to be installed to the cluster or an arbitrary WebAssembly module to be loaded into memory on the proxy servers.

The auditors and maintainers both note that the Operator is not recommended as a method of installation, as this requires a high-privilege controller to run in the cluster.

Other issues

The remaining issues found were:

• In some testing code, or where a control plane component connects to another component over localhost, minimum TLS settings were not enforced (#6)
• Operations that failed may not return error codes (#7)
• A deprecated library was being used (#8)
• TOC/TOU race conditions in a library used for copying files (#9)
• A user could exhaust the memory of the Security Token Service if running in Debug mode (#11)

Please refer to the full report for details.

Reviewing the 2020 report

All 18 issues reported in Istio’s first security assessment were found to have been fixed.

Fuzzing

The OSS-Fuzz project helps open source projects perform free fuzz testing. Istio is integrated into OSS-Fuzz with 63 fuzzers running continuously: this support was built by ADA Logics and the Istio team in late 2021.

The assessment notes that “Istio benefits largely from having a substantial fuzz test suite that runs continuously on OSS-Fuzz”, and identified a few APIs in security-critical code that would benefit from further fuzzing, Six new fuzzers were contributed as a result of this work; by the end of the audit, the new tests had run over 3 billion times.

SLSA

Supply chain Levels for Software Artifacts (SLSA) is a check-list of standards and controls to prevent tampering, improve integrity, and secure software packages and infrastructure. It is organized into a series of levels that provide increasing integrity guarantees.

Istio does not currently generate provenance artifacts, so it does not meet the requirements for any SLSA levels. Work on reaching SLSA Level 1 is currently underway. If you would like to get involved, please join the Istio Slack and reach out to our Test and Release working group.

Get involved

If you want to get involved with Istio product security, or become a maintainer, we’d love to have you! Join our public meetings to raise issues or learn about what we are doing to keep Istio secure.

Istio Day is the perfect opportunity to meet the Istio maintainers and contributors in person, and hear from users why Istio is constantly ranked the #1 service mesh in production.

Submit a talk

We now encourage Istio users, developers, partners, and advocates to submit a session proposal through the CNCF event portal, which is open until February 12.

We want to see real world examples, case studies, and success stories that can inspire newcomers to use Istio in production. The content will cover introductory to advanced levels, split into four main topic tracks:

• New Features: What have you been working on that the community should know about?
• Case Studies: How have you built a platform or service on top of Istio?
• Istio Recipes: How you can solve a specific business problem using Istio.
• Project Updates: The evolution of Istio, and the latest updates from the project maintainers.

You can pick one of these formats to submit a session proposal:

• Presentation: 25 minutes, 1 or 2 speaker(s) presenting a topic
• Panel Discussion: 35 minutes of discussion among 3 to 5 speakers
• Lightning Talk: A brief 5-minute presentation, maximum of 1 speaker

Accepted speakers will receive a complimentary All-Access In-Person ticket for all four days of KubeCon + CloudNativeCon.

Sponsor the event

Do you want to put your product or service in front of the most discerning Cloud Native users: those who demand 25% more conference than the crowd? Check out page 11 of the CNCF events prospectus to learn more.

Register to attend

Istio Day is a CNCF-hosted co-located event on 18 April 2023. KubeCon + CloudNativeCon Europe in-person attendees now have the option to buy an All-Access ticket, which includes entry to all the Day 0 events, as well as the main three days of the conference. You must be attending KubeCon to attend Istio Day, but virtual registration options are available, and the recordings will be posted to YouTube soon after the event.

For those of you who can’t make it, keep your eyes peeled for announcements of IstioCon 2023 and Istio Day North America later this year.

Stay tuned to hear more about the event, and we hope you can join us at Istio Day Europe!

The Kubernetes Ingress resource has for some time been known to have significant shortcomings, especially when using it to configure ingress traffic for large applications and when working with protocols other than HTTP. One problem is that it configures both the client-side L4-L6 properties (e.g., ports, TLS, etc.) and service-side L7 routing in a single resource, configurations that for large applications should be managed by different teams and in different namespaces. Also, by trying to draw a common denominator across different HTTP proxies, Ingress is only able to support the most basic HTTP routing and ends up pushing every other feature of modern proxies into non-portable annotations.

To overcome Ingress’ shortcomings, Istio introduced its own configuration API for ingress traffic management. With Istio’s API, the client-side representation is defined using an Istio Gateway resource, with L7 traffic moved to a VirtualService, not coincidentally the same configuration resource used for routing traffic between services inside the mesh. Although the Istio API provides a good solution for ingress traffic management for large-scale applications, it is unfortunately an Istio-only API. If you are using a different service mesh implementation, or no service mesh at all, you’re out of luck.

Enter Gateway API

There’s a lot of excitement surrounding a new Kubernetes traffic management API, dubbed Gateway API, which has recently been promoted to Beta. Gateway API provides a set of Kubernetes configuration resources for ingress traffic control that, like Istio’s API, overcomes the shortcoming of Ingress, but unlike Istio’s, is a standard Kubernetes API with broad industry agreement. There are several implementations of the API in the works, including a Beta implementation in Istio, so now may be a good time to start thinking about how you can start moving your ingress traffic configuration from Kubernetes Ingress or Istio Gateway/VirtualService to the new Gateway API.

Whether or not you use, or plan to use, Istio to manage your service mesh, the Istio implementation of the Gateway API can easily be used to get started with your cluster ingress control. Even though it’s still a Beta feature in Istio, mostly driven by the fact that the Gateway API is itself still a Beta level API, Istio’s implementation is quite robust because under the covers it uses Istio’s same tried-and-proven internal resources to implement the configuration.

Gateway API quick-start

To get started using the Gateway API, you need to first download the CRDs, which don’t come installed by default on most Kubernetes clusters, at least not yet:

$ kubectl get crd gateways.gateway.networking.k8s.io &> /dev/null || \
  { kubectl kustomize "github.com/kubernetes-sigs/gateway-api/config/crd?ref=v1.4.0" | kubectl apply -f -; }

Once the CRDs are installed, you can use them to create Gateway API resources to configure ingress traffic, but in order for the resources to work, the cluster needs to have a gateway controller running. You can enable Istio’s gateway controller implementation by simply installing Istio with the minimal profile:

$ curl -L https://istio.io/downloadIstio | sh -
$ cd istio-1.29.1
$ ./bin/istioctl install --set profile=minimal -y

Your cluster will now have a fully-functional implementation of the Gateway API, via Istio’s gateway controller named istio.io/gateway-controller, ready to use.

Deploy a Kubernetes target service

To try out the Gateway API, we’ll use the Istio helloworld sample as an ingress target, but only running as a simple Kubernetes service without sidecar injection enabled. Because we’re only going to use the Gateway API to control ingress traffic into the “Kubernetes cluster”, it makes no difference if the target service is running inside or outside of a mesh.

We’ll use the following command to deploy the helloworld service:

$ kubectl create ns sample
$ kubectl apply -f @samples/helloworld/helloworld.yaml@ -n sample

The helloworld service includes two backing deployments, corresponding to different versions (v1 and v2). We can confirm they are both running using the following command:

$ kubectl get pod -n sample
NAME                             READY   STATUS    RESTARTS   AGE
helloworld-v1-776f57d5f6-s7zfc   1/1     Running   0          10s
helloworld-v2-54df5f84b-9hxgww   1/1     Running   0          10s

Configure the helloworld ingress traffic

With the helloworld service up and running, we can now use the Gateway API to configure ingress traffic for it.

The ingress entry point is defined using a Gateway resource:

$ kubectl create namespace sample-ingress
$ kubectl apply -f - <<EOF
apiVersion: gateway.networking.k8s.io/v1beta1
kind: Gateway
metadata:
  name: sample-gateway
  namespace: sample-ingress
spec:
  gatewayClassName: istio
  listeners:
  - name: http
    hostname: "*.sample.com"
    port: 80
    protocol: HTTP
    allowedRoutes:
      namespaces:
        from: All
EOF

The controller that will implement a Gateway is selected by referencing a GatewayClass. There must be at least one GatewayClass defined in the cluster to have functional Gateways. In our case, we’re selecting Istio’s gateway controller, istio.io/gateway-controller, by referencing its associated GatewayClass (named istio) with the gatewayClassName: istio setting in the Gateway.

Notice that unlike Ingress, a Kubernetes Gateway doesn’t include any references to the target service, helloworld. With the Gateway API, routes to services are defined in separate configuration resources that get attached to the Gateway to direct subsets of traffic to specific services, like helloworld in our example. This separation allows us to define the Gateway and routes in different namespaces, presumably managed by different teams. Here, while acting in the role of cluster operator, we’re applying the Gateway in the sample-ingress namespace. We’ll add the route, below, in the sample namespace, next to the helloworld service itself, on behalf of the application developer.

Because the Gateway resource is owned by a cluster operator, it can very well be used to provide ingress for more than one team’s services, in our case more than just the helloworld service. To emphasize this point, we’ve set hostname to *.sample.com in the Gateway, allowing routes for multiple subdomains to be attached.

After applying the Gateway resource, we need to wait for it to be ready before retrieving its external address:

$ kubectl wait -n sample-ingress --for=condition=programmed gateway sample-gateway
$ export INGRESS_HOST=$(kubectl get -n sample-ingress gateway sample-gateway -o jsonpath='{.status.addresses[0].value}')

Next, we attach an HTTPRoute to the sample-gateway (i.e., using the parentRefs field) to expose and route traffic to the helloworld service:

$ kubectl apply -n sample -f - <<EOF
apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
  name: helloworld
spec:
  parentRefs:
  - name: sample-gateway
    namespace: sample-ingress
  hostnames: ["helloworld.sample.com"]
  rules:
  - matches:
    - path:
        type: Exact
        value: /hello
    backendRefs:
    - name: helloworld
      port: 5000
EOF

Here we’ve exposed the /hello path of the helloworld service to clients outside of the cluster, specifically via host helloworld.sample.com. You can confirm the helloworld sample is accessible using curl:

$ for run in {1..10}; do curl -HHost:helloworld.sample.com http://$INGRESS_HOST/hello; done
Hello version: v1, instance: helloworld-v1-78b9f5c87f-2sskj
Hello version: v2, instance: helloworld-v2-54dddc5567-2lm7b
Hello version: v1, instance: helloworld-v1-78b9f5c87f-2sskj
Hello version: v2, instance: helloworld-v2-54dddc5567-2lm7b
Hello version: v2, instance: helloworld-v2-54dddc5567-2lm7b
Hello version: v1, instance: helloworld-v1-78b9f5c87f-2sskj
Hello version: v1, instance: helloworld-v1-78b9f5c87f-2sskj
Hello version: v2, instance: helloworld-v2-54dddc5567-2lm7b
Hello version: v1, instance: helloworld-v1-78b9f5c87f-2sskj
Hello version: v2, instance: helloworld-v2-54dddc5567-2lm7b

Since no version routing has been configured in the route rule, you should see an equal split of traffic, about half handled by helloworld-v1 and the other half handled by helloworld-v2.

Configure weight-based version routing

Among other “traffic shaping” features, you can use Gateway API to send all of the traffic to one of the versions or split the traffic based on request percentages. For example, you can use the following rule to distribute the helloworld traffic 90% to v1, 10% to v2:

$ kubectl apply -n sample -f - <<EOF
apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
  name: helloworld
spec:
  parentRefs:
  - name: sample-gateway
    namespace: sample-ingress
  hostnames: ["helloworld.sample.com"]
  rules:
  - matches:
    - path:
        type: Exact
        value: /hello
    backendRefs:
    - name: helloworld-v1
      port: 5000
      weight: 90
    - name: helloworld-v2
      port: 5000
      weight: 10
EOF

Gateway API relies on version-specific backend service definitions for the route targets, helloworld-v1 and helloworld-v2 in this example. The helloworld sample already includes service definitions for the helloworld versions v1 and v2, we just need to run the following command to define them:

$ kubectl apply -n sample -f @samples/helloworld/gateway-api/helloworld-versions.yaml@

Now, we can run the previous curl commands again:

$ for run in {1..10}; do curl -HHost:helloworld.sample.com http://$INGRESS_HOST/hello; done
Hello version: v1, instance: helloworld-v1-78b9f5c87f-2sskj
Hello version: v1, instance: helloworld-v1-78b9f5c87f-2sskj
Hello version: v1, instance: helloworld-v1-78b9f5c87f-2sskj
Hello version: v1, instance: helloworld-v1-78b9f5c87f-2sskj
Hello version: v1, instance: helloworld-v1-78b9f5c87f-2sskj
Hello version: v1, instance: helloworld-v1-78b9f5c87f-2sskj
Hello version: v1, instance: helloworld-v1-78b9f5c87f-2sskj
Hello version: v1, instance: helloworld-v1-78b9f5c87f-2sskj
Hello version: v2, instance: helloworld-v2-54dddc5567-2lm7b
Hello version: v1, instance: helloworld-v1-78b9f5c87f-2sskj

This time we see that about 9 out of 10 requests are now handled by helloworld-v1 and only about 1 in 10 are handled by helloworld-v2.

Gateway API for internal mesh traffic

You may have noticed that we’ve been talking about the Gateway API only as an ingress configuration API, often referred to as north-south traffic management, and not an API for service-to-service (aka, east-west) traffic management within a cluster.

If you are using a service mesh, it would be highly desirable to use the same API resources to configure both ingress traffic routing and internal traffic, similar to the way Istio uses VirtualService to configure route rules for both. Fortunately, the Kubernetes Gateway API is working to add this support. Although not as mature as the Gateway API for ingress traffic, an effort known as the Gateway API for Mesh Management and Administration (GAMMA) initiative is underway to make this a reality and Istio intends to make Gateway API the default API for all of its traffic management in the future.

The first significant Gateway Enhancement Proposal (GEP) has recently been accepted and is, in-fact, already available to use in Istio. To try it out, you’ll need to use the experimental version of the Gateway API CRDs, instead of the standard Beta version we installed above, but otherwise, you’re ready to go. Check out the Istio request routing task to get started.

Summary

In this article, we’ve seen how a light-weight minimal install of Istio can be used to provide a Beta-quality implementation of the new Kubernetes Gateway API for cluster ingress traffic control. For Istio users, the Istio implementation also lets you start trying out the experimental Gateway API support for east-west traffic management within the mesh.

Much of Istio’s documentation, including all of the ingress tasks and several mesh-internal traffic management tasks, already includes parallel instructions for configuring traffic using either the Gateway API or the Istio configuration API. Check out the Gateway API task for more information about the Gateway API implementation in Istio.

• Craig Box (ARMO)
• Iris Ding (Intel)
• Faseela K (Ericsson Software Technology)
• Christian Posta (Solo.io)

We would like to extend our heartfelt thanks to Zack Butcher, Lin Sun and Zhonghu Xu, whose terms have now ended. With Contribution Seat holders from Google, IBM, Red Hat and DaoCloud, we have representation from 8 organizations on the Steering Committee, reflecting the breadth of our worldwide contributor ecosystem.

Thank you to everyone who participated in the election process, with special thanks to our election officers Josh Berkus, Cameron Etezadi and Ram Vennam.

In April, Istio applied to become a CNCF project. Today, the TOC announced they have voted to accept our application.

This journey began with Istio’s inception in 2016. We are grateful for all who have collaborated over the last six years on Istio’s design, development, and deployment.

We especially appreciate the efforts of TOC sponsor Dave Zolotusky, TAG Network, and the engineering teams at Airbnb, Intuit, Splunk, and WP Engine for sharing their feedback as end users.

While project work continues uninterrupted, with the acceptance of Istio, we now will begin the processes of transferring trademarks and build infrastructure to CNCF ownership. We are hard at work on our upcoming 1.16 release, while continuing to collect feedback on ambient mesh and driving it to production readiness. Our project members are currently electing our community representatives to the Steering Committee for the next year.

As a CNCF project, we will now be much more visible at KubeCon NA in October. Come to the maintainer session, find us in the project pavilion, or grab an Istio t-shirt at the CNCF Store. Watch our Twitter throughout the conference for more exciting updates!

To recap, Istio’s ambient mode introduces a layered mesh data plane with a secure overlay responsible for transport security and routing, that has the option to add L7 capabilities for namespaces that need them. To understand more, please see the announcement blog and the getting started blog. The secure overlay consists of a node-shared component, the ztunnel, that is responsible for L4 telemetry and mTLS which is deployed as a DaemonSet. The L7 layer of the mesh is provided by waypoint proxies, full L7 Envoy proxies that are deployed per identity/workload type. Some of the core implications of this design include:

• Separation of application from data plane
• Components of the secure overlay layer resemble that of a CNI
• Simplicity of operations is better for security
• Avoiding multi-tenant L7 proxies
• Sidecars are still a first-class supported deployment

Separation of application and data plane

Although the primary goal of ambient mesh is simplifying operations of the service mesh, it does serve to improve security as well. Complexity breeds vulnerabilities and enterprise applications (and their transitive dependencies, libraries, and frameworks) are exceedingly complex and prone to vulnerabilities. From handling complex business logic to leveraging OSS libraries or buggy internal shared libraries, a user’s application code is a prime target for attackers (internal or external). If an application is compromised, credentials, secrets, and keys are exposed to an attacker including those mounted or stored in memory. When looking at the sidecar model, an application compromise includes takeover of the sidecar and any associated identity/key material. In Istio’s ambient mode, no data plane components run in the same pod as the application and therefore an application compromise does not lead to the access of secrets.

What about Envoy Proxy as a potential target for vulnerabilities? Envoy is an extremely hardened piece of infrastructure under intense scrutiny and run at scale in critical environments (e.g., used in production to front Google’s network). However, since Envoy is software, it is not immune to vulnerabilities. When those vulnerabilities do arise, Envoy has a robust CVE process for identifying them, fixing them quickly, and rolling them out to customers before they have the chance for wide impact.

Circling back to the earlier comment that “complexity breeds vulnerabilities”, the most complex parts of Envoy Proxy is in its L7 processing, and indeed historically the majority of Envoy’s vulnerabilities have been in its L7 processing stack. But what if you just use Istio for mTLS? Why take the risk of deploying a full-blown L7 proxy which has a higher chance of CVE when you don’t use that functionality? Separating L4 and L7 mesh capabilities comes into play here. While in sidecar deployments you adopt all of the proxy, even if you use only a fraction of the functionality, in ambient mode we can limit the exposure by providing a secure overlay and only layering in L7 as needed. Additionally, the L7 components run completely separate from the applications and do not give an attack avenue.

Pushing L4 down into the CNI

The L4 components of the ambient mode data plane run as a DaemonSet, or one per node. This means it is shared infrastructure for any of the pods running on a particular node. This component is particularly sensitive and should be treated at the same level as any other shared component on the node such as any CNI agents, kube-proxy, kubelet, or even the Linux kernel. Traffic from workloads is redirected to the ztunnel which then identifies the workload and selects the right certificates to represent that workload in a mTLS connection.

The ztunnel uses a distinct credential for every pod which is only issued if the pod is currently running on the node. This ensures that the blast radius for a compromised ztunnel is that only credentials for pods currently scheduled on that node could be stolen. This is a similar property to other well implemented shared node infrastructure including other secure CNI implementations. The ztunnel does not use cluster-wide or per-node credentials which, if stolen, could immediately compromise all application traffic in the cluster unless a complex secondary authorization mechanism is also implemented.

If we compare this to the sidecar model, we notice that the ztunnel is shared and compromise could result in exfiltration of the identities of the applications running on the node. However, the likelihood of a CVE in this component is lower than that of an Istio sidecar since the attack surface is greatly reduced (only L4 handling); the ztunnel does not do any L7 processing. In addition, a CVE in a sidecar (with a larger attack surface with L7) is not truly contained to only that particular workload which is compromised. Any serious CVE in a sidecar is likely repeatable to any of the workloads in the mesh as well.

Simplicity of operations is better for security

Ultimately, Istio is a critical piece of infrastructure that must be maintained. Istio is trusted to implement some of the tenets of zero-trust network security on behalf of applications and rolling out patches on a schedule or on demand is paramount. Platform teams often have predictable patching or maintenance cycles which is quite different from that of applications. Applications likely get updated when new capabilities and functionality are required and usually part of a project. This approach to application changes, upgrades, framework and library patches, is highly unpredictable, allows a lot of time to pass, and does not lend itself to safe security practices. Therefore, keeping these security features part of the platform and separate from the applications is likely to lead to a better security posture.

As we’ve identified in the announcement blog, operating sidecars can be more complex because of the invasive nature of them (injecting the sidecar/changing the deployment descriptors, restarting the applications, race conditions between containers, etc). Upgrades to workloads with sidecars require a bit more planning and rolling restarts that may need to be coordinated to not bring down the application. With ambient mode, upgrades to the ztunnel can coincide with any normal node patching or upgrades, while the waypoint proxies are part of the network and can be upgraded completely transparently to the applications as needed.

Avoiding multi-tenant L7 proxies

Supporting L7 protocols such as HTTP 1/2/3, gRPC, parsing headers, implementing retries, customizations with Wasm and/or Lua in the data plane is significantly more complex than supporting L4. There is a lot more code to implement these behaviors (including user-custom code for things like Lua and Wasm) and this complexity can lead to the potential for vulnerabilities. Because of this, CVEs have a higher chance of being discovered in these areas of L7 functionality.

In ambient mode, we do not share L7 processing in a proxy across multiple identities. Each identity (service account in Kubernetes) has its own dedicated L7 proxy (waypoint proxy) which is very similar to the model we use with sidecars. Trying to co-locate multiple identities and their distinct complex policies and customizations adds a lot of variability to a shared resource which leads to unfair cost attribution at best and total proxy compromise at worst.

Sidecars are still a first-class supported deployment

We understand that some folks are comfortable with the sidecar model and their known security boundaries and wish to stay on that model. With Istio, sidecars are a first-class citizen to the mesh and platform owners have the choice to continue using them. If a platform owner wants to support both sidecar and ambient modes, they can. A workload with the ambient data plane can natively communicate with workloads that have a sidecar deployed. As folks better understand the security posture of ambient mode, we are confident that it will be the preferred data plane mode of Istio, with sidecars used for specific optimizations.

Ambient mesh is a new data plane mode for Istio introduced today. Following this getting started guide, you can experience how ambient mesh can simplify your application onboarding, help with ongoing operations, and reduce service mesh infrastructure resource usage.

Install Istio with Ambient Mode

1. Download the preview version of Istio with support for ambient mesh.
2. Check out supported environments. We recommend using a Kubernetes cluster that is version 1.21 or newer that has two nodes or more. If you don’t have a Kubernetes cluster, you can set up using locally (e.g. using kind as below) or deploy one in Google or AWS Cloud:

$ kind create cluster --config=- <<EOF
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: ambient
nodes:
- role: control-plane
- role: worker
- role: worker
EOF

The ambient profile is designed to help you get started with ambient mesh. Install Istio with the ambient profile on your Kubernetes cluster, using the istioctl downloaded above:

$ istioctl install --set profile=ambient

After running the above command, you’ll get the following output that indicates these four components are installed successfully!

✔ Istio core installed
✔ Istiod installed
✔ Ingress gateways installed
✔ CNI installed
✔ Installation complete

By default, the ambient profile has the Istio core, Istiod, ingress gateway, zero-trust tunnel agent (ztunnel) and CNI plugin enabled. The Istio CNI plugin is responsible for detecting which application pods are part of the ambient mesh and configuring the traffic redirection between the ztunnels. You’ll notice the following pods are installed in the istio-system namespace with the default ambient profile:

$ kubectl get pod -n istio-system
NAME                                    READY   STATUS    RESTARTS   AGE
istio-cni-node-97p9l                    1/1     Running   0          29s
istio-cni-node-rtnvr                    1/1     Running   0          29s
istio-cni-node-vkqzv                    1/1     Running   0          29s
istio-ingressgateway-5dc9759c74-xlp2j   1/1     Running   0          29s
istiod-64f6d7db7c-dq8lt                 1/1     Running   0          47s
ztunnel-bq6w2                           1/1     Running   0          47s
ztunnel-tcn4m                           1/1     Running   0          47s
ztunnel-tm9zl                           1/1     Running   0          47s

The istio-cni and ztunnel components are deployed as Kubernetes DaemonSets which run on every node. Each Istio CNI pod checks all pods co-located on the same node to see if these pods are part of the ambient mesh. For those pods, the CNI plugin configures traffic redirection so that all incoming and outgoing traffic to the pods are redirected to the co-located ztunnel first. As new pods are deployed or removed on the node, CNI plugin continues to monitor and update the redirection logic accordingly.

Deploy Your Applications

You’ll use the sample bookinfo application, which is part of your Istio download from previous steps. In ambient mode, you deploy applications to your Kubernetes cluster exactly the same way you would without Istio. This means you can have your applications running in your Kubernetes before you enable ambient mesh, and have them join the mesh without needing to restart or reconfigure your applications.

$ kubectl apply -f samples/bookinfo/platform/kube/bookinfo.yaml
$ kubectl apply -f https://raw.githubusercontent.com/linsun/sample-apps/main/sleep/sleep.yaml
$ kubectl apply -f https://raw.githubusercontent.com/linsun/sample-apps/main/sleep/notsleep.yaml

Note: sleep and notsleep are two simple applications that can serve as curl clients.

Connect productpage to the Istio ingress gateway so you can access the bookinfo app from outside of the cluster:

$ kubectl apply -f samples/bookinfo/networking/bookinfo-gateway.yaml

Test your bookinfo application, it should work with or without the gateway. Note: you can replace istio-ingressgateway.istio-system below with its load balancer IP (or hostname) if it has one:

$ kubectl exec deploy/sleep -- curl -s http://istio-ingressgateway.istio-system/productpage | head -n1
$ kubectl exec deploy/sleep -- curl -s http://productpage:9080/ | head -n1
$ kubectl exec deploy/notsleep -- curl -s http://productpage:9080/ | head -n1

Adding your application to the ambient mesh

You can enable all pods in a given namespace to be part of the ambient mesh by simply labeling the namespace:

$ kubectl label namespace default istio.io/dataplane-mode=ambient

Congratulations! You have successfully added all pods in the default namespace to the ambient mesh. The best part is that there is no need to restart or redeploy anything!

Send some test traffic:

$ kubectl exec deploy/sleep -- curl -s http://istio-ingressgateway.istio-system/productpage | head -n1
$ kubectl exec deploy/sleep -- curl -s http://productpage:9080/ | head -n1
$ kubectl exec deploy/notsleep -- curl -s http://productpage:9080/ | head -n1

You’ll immediately gain mTLS communication among the applications in the Ambient mesh.

If you are curious about the X.509 certificate for each identity, you can learn more about it by stepping through a certificate:

$ istioctl pc secret ds/ztunnel -n istio-system -o json | jq -r '.dynamicActiveSecrets[0].secret.tlsCertificate.certificateChain.inlineBytes' | base64 --decode | openssl x509 -noout -text -in /dev/stdin

For example, the output shows the certificate for the sleep principle that is valid for 24 hours, issued by the local Kubernetes cluster.

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 307564724378612391645160879542592778778 (0xe762cfae32a3b8e3e50cb9abad32b21a)
    Signature Algorithm: SHA256-RSA
        Issuer: O=cluster.local
        Validity
            Not Before: Aug 29 21:00:14 2022 UTC
            Not After : Aug 30 21:02:14 2022 UTC
        Subject:
        Subject Public Key Info:
            Public Key Algorithm: RSA
                Public-Key: (2048 bit)
                Modulus:
                    ac:db:1a:77:72:8a:99:28:4a:0c:7e:43:fa:ff:35:
                    75:aa:88:4b:80:4f:86:ca:69:59:1c:b5:16:7b:71:
                    dd:74:57:e2:bc:cf:ed:29:7d:7b:fa:a2:c9:06:e6:
                    d6:41:43:2a:3c:2c:18:8e:e8:17:f6:82:7a:64:5f:
                    c4:8a:a4:cd:f1:4a:9c:3f:e0:cc:c5:d5:79:49:37:
                    30:10:1b:97:94:2c:b7:1b:ed:a2:62:d9:3b:cd:3b:
                    12:c9:b2:6c:3c:2c:ac:54:5b:a7:79:97:fb:55:89:
                    ca:08:0e:2e:2a:b8:d2:e0:3b:df:b2:21:99:06:1b:
                    60:0d:e8:9d:91:dc:93:2f:7c:27:af:3e:fc:42:99:
                    69:03:9c:05:0b:c2:11:25:1f:71:f0:8a:b1:da:4a:
                    da:11:7c:b4:14:df:6e:75:38:55:29:53:63:f5:56:
                    15:d9:6f:e6:eb:be:61:e4:ce:4b:2a:f9:cb:a6:7f:
                    84:b7:4c:e4:39:c1:4b:1b:d4:4c:70:ac:98:95:fe:
                    3e:ea:5a:2c:6c:12:7d:4e:24:ab:dc:0e:8f:bc:88:
                    02:f2:66:c9:12:f0:f7:9e:23:c9:e2:4d:87:75:b8:
                    17:97:3c:96:83:84:3f:d1:02:6d:1c:17:1a:43:ce:
                    68:e2:f3:d7:dd:9e:a6:7d:d3:12:aa:f5:62:91:d9:
                    8d
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage:
                Server Authentication, Client Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Authority Key Identifier:
                keyid:93:49:C1:B8:AB:BF:0F:7D:44:69:5A:C3:2A:7A:3C:79:19:BE:6A:B7
            X509v3 Subject Alternative Name: critical
                URI:spiffe://cluster.local/ns/default/sa/sleep

Note: If you don’t get any output, it may mean ds/ztunnel has selected a node that doesn’t manage any certificates. You can specify a specific ztunnel pod (e.g. istioctl pc secret ztunnel-tcn4m -n istio-system) that manages either one of the sample application pods instead.

Secure application access

After you have added your application to ambient mesh, you can secure application access using L4 authorization policies. This lets you control access to and from a service based on client workload identities, but not at the L7 level, such as HTTP methods like GET and POST.

L4 Authorization Policies

Explicitly allow the sleep service account and istio-ingressgateway service accounts to call the productpage service:

$ kubectl apply -f - <<EOF
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
 name: productpage-viewer
 namespace: default
spec:
 selector:
   matchLabels:
     app: productpage
 action: ALLOW
 rules:
 - from:
   - source:
       principals: ["cluster.local/ns/default/sa/sleep", "cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account"]
EOF

Confirm the above authorization policy is working:

$ # this should succeed
$ kubectl exec deploy/sleep -- curl -s http://istio-ingressgateway.istio-system/productpage | head -n1
$ # this should succeed
$ kubectl exec deploy/sleep -- curl -s http://productpage:9080/ | head -n1
$ # this should fail with an empty reply
$ kubectl exec deploy/notsleep -- curl -s http://productpage:9080/ | head -n1

Layer 7 Authorization Policies

Using the Kubernetes Gateway API, you can deploy a waypoint proxy for the productpage service that uses the bookinfo-productpage service account. Any traffic going to the productpage service will be mediated, enforced and observed by the Layer 7 (L7) proxy.

$ kubectl apply -f - <<EOF
apiVersion: gateway.networking.k8s.io/v1beta1
kind: Gateway
metadata:
 name: productpage
 annotations:
   istio.io/service-account: bookinfo-productpage
spec:
 gatewayClassName: istio-mesh
EOF

Note the gatewayClassName has to be istio-mesh for the waypoint proxy.

View the productpage waypoint proxy status; you should see the details of the gateway resource with Ready status:

$ kubectl get gateway productpage -o yaml
...
status:
  conditions:
  - lastTransitionTime: "2022-09-06T20:24:41Z"
    message: Deployed waypoint proxy to "default" namespace for "bookinfo-productpage"
      service account
    observedGeneration: 1
    reason: Ready
    status: "True"
    type: Ready

Update our AuthorizationPolicy to explicitly allow the sleep service account and istio-ingressgateway service accounts to GET the productpage service, but perform no other operations:

$ kubectl apply -f - <<EOF
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
 name: productpage-viewer
 namespace: default
spec:
 selector:
   matchLabels:
     app: productpage
 action: ALLOW
 rules:
 - from:
   - source:
       principals: ["cluster.local/ns/default/sa/sleep", "cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account"]
   to:
   - operation:
       methods: ["GET"]
EOF

Confirm the above authorization policy is working:

$ # this should fail with an RBAC error because it is not a GET operation
$ kubectl exec deploy/sleep -- curl -s http://productpage:9080/ -X DELETE | head -n1
$ # this should fail with an RBAC error because the identity is not allowed
$ kubectl exec deploy/notsleep -- curl -s http://productpage:9080/  | head -n1
$ # this should continue to work
$ kubectl exec deploy/sleep -- curl -s http://productpage:9080/ | head -n1

With the productpage waypoint proxy deployed, you’ll also automatically get L7 metrics for all requests to the productpage service:

$ kubectl exec deploy/bookinfo-productpage-waypoint-proxy -- curl -s http://localhost:15020/stats/prometheus | grep istio_requests_total

You’ll notice the metric with response_code=403 and some metrics response_code=200, like below:

istio_requests_total{
  response_code="403",
  source_workload="notsleep",
  source_workload_namespace="default",
  source_principal="spiffe://cluster.local/ns/default/sa/notsleep",
  destination_workload="productpage-v1",
  destination_principal="spiffe://cluster.local/ns/default/sa/bookinfo-productpage",
  connection_security_policy="mutual_tls",
  ...
}

The metric shows two 403 responses when the source workload (notsleep) calls the destination workload(productpage-v1) along with source and destination principals via mutual TLS connection.

Control Traffic

Deploy a waypoint proxy for the review service, using the bookinfo-review service account, so that any traffic going to the review service will be mediated by the waypoint proxy.

$ kubectl apply -f - <<EOF
apiVersion: gateway.networking.k8s.io/v1beta1
kind: Gateway
metadata:
 name: reviews
 annotations:
   istio.io/service-account: bookinfo-reviews
spec:
 gatewayClassName: istio-mesh
EOF

Apply the reviews virtual service to control 90% traffic to reviews v1 and 10% traffic to reviews v2.

$ kubectl apply -f samples/bookinfo/networking/virtual-service-reviews-90-10.yaml
$ kubectl apply -f samples/bookinfo/networking/destination-rule-reviews.yaml

Confirm that roughly 10% traffic from the 100 requests go to reviews-v2:

$ kubectl exec -it deploy/sleep -- sh -c 'for i in $(seq 1 100); do curl -s http://istio-ingressgateway.istio-system/productpage | grep reviews-v.-; done'

Wrapping up

The existing Istio resources continue to work, regardless if you choose to use the sidecar or ambient data plane mode.

Take a look at the short video to watch Lin run through the Istio ambient mesh demo in 5 minutes:

What’s next

We are super excited about the new Istio ambient data plane with its simple “ambient” architecture. Onboarding your applications onto a service mesh with ambient mode is now as easy as labeling a namespace. Your applications will gain instant benefits such as mTLS with cryptographic identity for mesh traffic and L4 observability. If you need to control access or routes or increase resiliency or gain L7 metrics among your applications in ambient mesh, you can apply waypoint proxies to your applications as needed. We’re big fans of paying for only what we need, as it not only saves resources but also saves operation cost from constantly updating many proxies! We invite you to try the new Istio ambient data plane architecture to experience how simple it is. We look forward to your feedback in the Istio community!

Today, we are excited to introduce “ambient mesh”, and its reference implementation: a new Istio data plane mode that’s designed for simplified operations, broader application compatibility, and reduced infrastructure cost. Ambient mesh gives users the option to forgo sidecar proxies in favor of a data plane that’s integrated into their infrastructure, all while maintaining Istio’s core features of zero-trust security, telemetry, and traffic management. We are sharing a preview of ambient mesh with the Istio community that we are working to bring to production readiness in the coming months.

Istio and sidecars

Since its inception, a defining feature of Istio’s architecture has been the use of sidecars – programmable proxies deployed alongside application containers. Sidecars allow operators to reap Istio’s benefits, without requiring applications to undergo major surgery and its associated costs.

Although sidecars have significant advantages over refactoring applications, they do not provide a perfect separation between applications and the Istio data plane. This results in a few limitations:

• Invasiveness - Sidecars must be “injected” into applications by modifying their Kubernetes pod spec and redirecting traffic within the pod. As a result, installing or upgrading sidecars requires restarting the application pod, which can be disruptive for workloads.
• Underutilization of resources - Since the sidecar proxy is dedicated to its associated workload, the CPU and memory resources must be provisioned for worst case usage of each individual pod. This adds up to large reservations that can lead to underutilization of resources across the cluster.
• Traffic breaking - Traffic capture and HTTP processing, as typically done by Istio’s sidecars, is computationally expensive and can break some applications with non-conformant HTTP implementations.

While sidecars have their place — more on that later — we think there is a need for a less invasive and easier option that will be a better fit for many service mesh users.

Slicing the layers

Traditionally, Istio implements all data plane functionality, from basic encryption through advanced L7 policy, in a single architectural component: the sidecar. In practice, this makes sidecars an all-or-nothing proposition. Even if a workload just needs simple transport security, administrators still need to pay the operational cost of deploying and maintaining a sidecar. Sidecars have a fixed operational cost per workload that does not scale to fit the complexity of the use case.

The ambient data plane takes a different approach. It splits Istio’s functionality into two distinct layers. At the base, there’s a secure overlay that handles routing and zero trust security for traffic. Above that, when needed, users can enable L7 processing to get access to the full range of Istio features. The L7 processing mode, while heavier than the secure overlay, still runs as an ambient component of the infrastructure, requiring no modifications to application pods.

This layered approach allows users to adopt Istio in a more incremental fashion, smoothly transitioning from no mesh, to the secure overlay, to full L7 processing — on a per-namespace basis, as needed. Furthermore, workloads running in different ambient layers, or with sidecars, interoperate seamlessly, allowing users to mix and match capabilities based on the particular needs as they change over time.

Building an ambient mesh

Istio’s ambient data plane mode uses a shared agent, running on each node in the Kubernetes cluster. This agent is a zero-trust tunnel (or ztunnel), and its primary responsibility is to securely connect and authenticate elements within the mesh. The networking stack on the node redirects all traffic of participating workloads through the local ztunnel agent. This fully separates the concerns of Istio’s data plane from those of the application, ultimately allowing operators to enable, disable, scale, and upgrade the data plane without disturbing applications. The ztunnel performs no L7 processing on workload traffic, making it significantly leaner than sidecars. This large reduction in complexity and associated resource costs make it amenable to delivery as shared infrastructure.

Ztunnels enable the core functionality of a service mesh: zero trust. A secure overlay is created when ambient mode is enabled for a namespace. It provides workloads with mTLS, telemetry, authentication, and L4 authorization, without terminating or parsing HTTP.

After ambient mode is enabled and a secure overlay is created, a namespace can be configured to utilize L7 features. This allows a namespace to implement the full set of Istio capabilities, including the Virtual Service API, L7 telemetry, and L7 authorization policies. Namespaces operating in this mode use one or more Envoy-based waypoint proxies to handle L7 processing for workloads in that namespace. Istio’s control plane configures the ztunnels in the cluster to pass all traffic that requires L7 processing through the waypoint proxy. Importantly, from a Kubernetes perspective, waypoint proxies are just regular pods that can be auto-scaled like any other Kubernetes deployment. We expect this to yield significant resource savings for users, as the waypoint proxies can be auto-scaled to fit the real time traffic demand of the namespaces they serve, not the maximum worst-case load operators expect.

Ambient mesh uses HTTP CONNECT over mTLS to implement its secure tunnels and insert waypoint proxies in the path, a pattern we call HBONE (HTTP-Based Overlay Network Environment). HBONE provides for a cleaner encapsulation of traffic than TLS on its own while enabling interoperability with common load-balancer infrastructure. FIPS builds are used by default to meet compliance needs. More details on HBONE, its standards-based approach, and plans for UDP and other non-TCP protocols will be provided in a future blog.

Mixing sidecar and ambient modes in a single mesh does not introduce limitations on the capabilities or security properties of the system. The Istio control plane ensures that policies are properly enforced regardless of the deployment model chosen. Ambient mode simply introduces an option that has better ergonomics and more flexibility.

Why no L7 processing on the local node?

Ambient mode uses a shared ztunnel agent on the node, which handles the zero trust aspects of the mesh, while L7 processing happens in the waypoint proxy in separately scheduled pods. Why bother with the indirection, and not just use a shared full L7 proxy on the node? There are several reasons for this:

• Envoy is not inherently multi-tenant. As a result, we have security concerns with commingling complex processing rules for L7 traffic from multiple unconstrained tenants in a shared instance. By strictly limiting to L4 processing, we reduce the vulnerability surface area significantly.
• The mTLS and L4 features provided by the ztunnel need a much smaller CPU and memory footprint when compared to the L7 processing required in the waypoint proxy. By running waypoint proxies as a shared namespace resource, we can scale them independently based on the needs of that namespace, and its costs are not unfairly distributed across unrelated tenants.
• By reducing ztunnel’s scope we allow for it to be replaced by other secure tunnel implementations that can meet a well-defined interoperability contract.

But what about those extra hops?

With ambient mode, a waypoint isn’t necessarily guaranteed to be on the same node as the workloads it serves. While at first glance this may appear to be a performance concern, we’re confident that latency will ultimately be in-line with Istio’s current sidecar implementation. We’ll discuss more in a dedicated performance blog post, but for now we’ll summarize with two points:

• The majority of Istio’s network latency does not, in fact, come from the network (modern cloud providers have extremely fast networks). Instead the biggest culprit is the intensive L7 processing Istio needs to implement its sophisticated feature set. Unlike sidecars, which implement two L7 processing steps for each connection (one for each sidecar), ambient mode collapses these two steps into one. In most cases, we expect this reduced processing cost to compensate for an additional network hop.
• Users often deploy a mesh to enable a zero-trust security posture as a first-step and then selectively enable L7 capabilities as needed. Ambient mode allows those users to bypass the cost of L7 processing entirely when it’s not needed.

Resource overhead

Overall we expect Istio’s ambient mode to have fewer and more predictable resource requirements for most users. The ztunnel’s limited responsibilities allows it to be deployed as a shared resource on the node. This will substantially reduce the per-workload reservations required for most users. Furthermore, since the waypoint proxies are normal Kubernetes pods, they can be dynamically deployed and scaled based on the real-time traffic demands of the workloads they serve.

Sidecars, on the other hand, need to reserve memory and CPU for the worst case for each workload. Making these calculations are complicated, so in practice administrators tend to over-provision. This leads to underutilized nodes due to high reservations that prevent other workloads from being scheduled. Ambient mode’s lower fixed per-node overhead and dynamically scaled waypoint proxies will require far fewer resource reservations in aggregate, leading to more efficient use of a cluster.

What about security?

With a radically new architecture naturally comes questions around security. The ambient mode security blog does a deep dive, but we’ll summarize here.

Sidecars co-locate with the workloads they serve and as a result, a vulnerability in one compromises the other. In the ambient mesh model, even if an application is compromised, the ztunnels and waypoint proxies can still enforce strict security policy on the compromised application’s traffic. Furthermore, given that Envoy is a mature battle-tested piece of software used by the world’s largest network operators, it is likely less vulnerable than the applications it runs alongside.

While the ztunnel is a shared resource, it only has access to the keys of the workloads currently on the node it’s running. Thus, its blast radius is no worse than any other encrypted CNI that relies on per-node keys for encryption. Also, given the ztunnel’s limited L4 only attack surface area and Envoy’s aforementioned security properties, we feel this risk is limited and acceptable.

Finally, while the waypoint proxies are a shared resource, they can be limited to serving just one service account. This makes them no worse than sidecars are today; if one waypoint proxy is compromised, the credential associated with that waypoint is lost, and nothing else.

Is this the end of the road for the sidecar?

Definitely not. While we believe ambient mesh will be the best option for many mesh users going forward, sidecars continue to be a good choice for those that need dedicated data plane resources, such as for compliance or performance tuning. Istio will continue to support sidecars, and importantly, allow them to interoperate seamlessly with ambient mode. In fact, the ambient mode code we’re releasing today already supports interoperation with sidecar-based Istio.

Learn more

Take a look at a short video to watch Christian run through the Istio ambient mode components and demo some capabilities:

Get involved

What we have released today is an early version of ambient mode in Istio, and it is very much still under active development. We are excited to share it with the broader community and look forward to getting more people involved in shaping it as we move to production readiness in 2023.

We would love your feedback to help shape the solution. A build of Istio which supports ambient mode is available to download and try in the Istio Experimental repo. A list of missing features and work items is available in the README. Please try it out and let us know what you think!

Thank you to the team that contributed to the launch of ambient mesh!

• Google: Craig Box, John Howard, Ethan J. Jackson, Abhi Joglekar, Steven Landow, Oliver Liu, Justin Pettit, Doug Reid, Louis Ryan, Kuat Yessenov, Francis Zhou
• Solo.io: Aaron Birkland, Kevin Dorosh, Greg Hanson, Daniel Hawton, Denis Jannot, Yuval Kohavi, Idit Levine, Yossi Mesika, Neeraj Poddar, Nina Polshakova, Christian Posta, Lin Sun, Eitan Yarmush

The history of Istio’s traffic management APIs

API design is more of an art than a science, and Istio is often used as an API to configure the serving of other APIs! In the case of traffic routing alone, we must consider producer vs consumer, routing vs. post-routing, and how to express a complex feature set with the correct number of objects — factoring in that these must be owned by different teams.

When we launched Istio in 2017, we brought many years of experience from Google’s production API serving infrastructure and IBM’s Amalgam8 project, and mapped it onto Kubernetes. We soon came up against the limitations of Kubernetes’ Ingress API. A desire to support all proxy implementations meant that Ingress only supported the most basic of HTTP routing features, with other features often implemented as vendor-specific annotations. The Ingress API was shared between infrastructure admins (“create and configure a load balancer”), cluster operators (“manage a TLS certificate for my entire domain”) and application users (“use it to route /foo to the foo service”).

We rewrote our traffic APIs in early 2018 to address user feedback, and to more adequately address these concerns.

A primary feature of Istio’s new model was having separate APIs that describe infrastructure (the load balancer, represented by the Gateway), and application (routing and post-routing, represented by the VirtualService and DestinationRule).

Ingress worked well as a lowest common denominator between different implementations, but its shortcomings led SIG Network to investigate the design of a “version 2”. A user survey in 2018 was followed by a proposal for new APIs in 2019, based in large part on Istio’s traffic APIs. That effort came to be known as the “Gateway API”.

The Gateway API was built to be able to model many more use cases, with extension points to enable functionality that differs between implementations. Furthermore, adopting the Gateway API opens a service mesh up to compatibility with the whole ecosystem of software that is written to support it. You don’t have to ask your vendor to support Istio routing directly: all they need to do is create Gateway API objects, and Istio will do what it needs to do, out of the box.

Support for the Gateway API in Istio

Istio added support for the Gateway API in November 2020, with support marked Alpha along with the API implementation. With the Beta release of the API spec we are pleased to announce support for ingress use in Istio is being promoted to Beta. We also encourage early adopters to start experimenting with the Gateway API for mesh (service-to-service) use, and we will move that support to Beta when SIG Network has standardized the required semantics.

Around the time of the v1 release of the API, we intend to make the Gateway API the default method for configuring all traffic routing in Istio - for ingress (north-south) and service-to-service (east-west). At that time, we will change our documentation and examples to reflect the recommendation.

Just like Kubernetes intends to support the Ingress API for many years after the Gateway API goes stable, the Istio APIs (Gateway, VirtualService and DestinationRule) will remain supported for the foreseeable future.

Not only that, but you can continue to use the existing Istio traffic APIs alongside the Gateway API, for example, using an HTTPRoute with an Istio VirtualService.

The similarity between the APIs means that we will be able to offer a tool to easily convert Istio API objects to Gateway API objects, and we will release this alongside the v1 version of the API.

Other parts of Istio functionality, including policy and telemetry, will continue to be configured using Istio-specific APIs while we work with SIG Network on standardization of these use cases.

Welcoming the SMI community to the Gateway API project

Throughout its design and implementation, members of the Istio team have been working with members of SIG Network on the implementation of the Gateway API, making sure the API was suitable for use in mesh use cases.

We are delighted to be formally joined in this effort by members of the Service Mesh Interface (SMI) community, including leaders from Linkerd, Consul and Open Service Mesh, who have collectively decided to standardize their API efforts on the Gateway API. To that end, we have set up a Gateway API Mesh Management and Administration (GAMMA) workstream within the Gateway API project. John Howard, a member of the Istio Technical Oversight Committee and a lead of our Networking WG, will be a lead of this group.

Our combined next steps are to provide enhancement proposals to the Gateway API project to support mesh use cases. We have started looking at API semantics for mesh traffic management, and will work with vendors and communities implementing Gateway API in their projects to build on a standard implementation. After that, we intend to build a representation for authorization and authentication policy.

With SIG Network as a vendor neutral forum for ensuring the service mesh community implements the Gateway API using the same semantics, we look forward to having a standard API which works with all projects, regardless of their technology stack or proxy.

Depending upon use cases, when an ingress gateway must handle a large number of incoming TLS and secured service-to-service connections through sidecar proxies, the load on Envoy increases. The potential performance depends on many factors, such as size of the cpuset on which Envoy is running, incoming traffic patterns, and key size. These factors can impact Envoy serving many new incoming TLS requests. To achieve performance improvements and accelerated handshakes, a new feature was introduced in Envoy 1.20 and Istio 1.14. It can be achieved with 3rd Gen Intel® Xeon® Scalable processors, the Intel® Integrated Performance Primitives (Intel® IPP) crypto library, CryptoMB Private Key Provider Method support in Envoy, and Private Key Provider configuration in Istio using ProxyConfig.

CryptoMB

The Intel IPP crypto library supports multi-buffer crypto operations. Briefly, multi-buffer cryptography is implemented with Intel® Advanced Vector Extensions 512 (Intel® AVX-512) instructions using a SIMD (single instruction, multiple data) mechanism. Up to eight RSA or ECDSA operations are gathered into a buffer and processed at the same time, providing potentially improved performance. Intel AVX-512 instructions are available on recently launched 3rd generation Intel Xeon Scalable processor server processors (Ice Lake server).

The idea of Envoy’s CryptoMB private key provider is that incoming TLS handshakes’ RSA operations are accelerated using Intel AVX-512 multi-buffer instructions.

Accelerate Envoy with Intel AVX-512 instructions

Envoy uses BoringSSL as the default TLS library. BoringSSL supports setting private key methods for offloading asynchronous private key operations, and Envoy implements a private key provider framework to allow creation of Envoy extensions which handle TLS handshakes private key operations (signing and decryption) using the BoringSSL hooks.

CryptoMB private key provider is an Envoy extension which handles BoringSSL TLS RSA operations using Intel AVX-512 multi-buffer acceleration. When a new handshake happens, BoringSSL invokes the private key provider to request the cryptographic operation, and then the control returns to Envoy. The RSA requests are gathered in a buffer. When the buffer is full or the timer expires, the private key provider invokes Intel AVX-512 processing of the buffer. When processing is done, Envoy is notified that the cryptographic operation is done and that it may continue with the handshakes.

The Envoy worker thread has a buffer size for eight RSA requests. When the first RSA request is stored in the buffer, a timer will be initiated (timer duration is set by the poll_delay field in the CryptoMB configuration).

When the buffer is full or when the timer expires, the crypto operations are performed for all RSA requests simultaneously. The SIMD (single instruction, multiple data) processing gives the potential performance benefit compared to the non-accelerated case.

Envoy CryptoMB Private Key Provider configuration

A regular TLS configuration only uses a private key. When a private key provider is used, the private key field is replaced with a private key provider field. It contains two fields, provider name and typed config. Typed config is CryptoMbPrivateKeyMethodConfig, and it specifies the private key and the poll delay.

TLS configuration with just a private key.

tls_certificates:
  certificate_chain: { "filename": "/path/cert.pem" }
  private_key: { "filename": "/path/key.pem" }

TLS configuration with CryptoMB private key provider.

tls_certificates:
  certificate_chain: { "filename": "/path/cert.pem" }
  private_key_provider:
    provider_name: cryptomb
    typed_config:
      "@type": type.googleapis.com/envoy.extensions.private_key_providers.cryptomb.v3alpha.CryptoMbPrivateKeyMethodConfig
      private_key: { "filename": "/path/key.pem" }
      poll_delay: 10ms

Istio CryptoMB Private Key Provider configuration

In Istio, CryptoMB private key provider configuration can be applied mesh wide, gateways specific or pod specific configurations using pod annotations. The User will provide the PrivateKeyProvider in the ProxyConfig with the pollDelay value. This configuration will be applied to mesh wide (gateways and all sidecars).

Istio Mesh wide Configuration

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  namespace: istio-system
  name: example-istiocontrolplane
spec:
  profile: demo
  components:
    egressGateways:
    - name: istio-egressgateway
      enabled: true
    ingressGateways:
    - name: istio-ingressgateway
      enabled: true
  meshConfig:
    defaultConfig:
      privateKeyProvider:
        cryptomb:
          pollDelay: 10ms

Istio Gateways Configuration

If a user wants to apply a private key provider configuration for ingress gateway only, follow the below sample configuration.

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  namespace: istio-system
  name: example-istiocontrolplane
spec:
  profile: demo
  components:
    egressGateways:
    - name: istio-egressgateway
      enabled: true
    ingressGateways:
    - name: istio-ingressgateway
      enabled: true
      k8s:
        podAnnotations:
          proxy.istio.io/config: |
            privateKeyProvider:
              cryptomb:
                pollDelay: 10ms

Istio Sidecar Configuration using pod annotations

If a user wants to apply private key provider configuration to application specific pods, configure them using pod annotations like the below sample.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: httpbin
---
apiVersion: v1
kind: Service
metadata:
  name: httpbin
  labels:
    app: httpbin
    service: httpbin
spec:
  ports:
  - name: http
    port: 8000
    targetPort: 80
  selector:
    app: httpbin
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: httpbin
spec:
  replicas: 1
  selector:
    matchLabels:
      app: httpbin
      version: v1
  template:
    metadata:
      labels:
        app: httpbin
        version: v1
      annotations:
        proxy.istio.io/config: |
          privateKeyProvider:
            cryptomb:
              pollDelay: 10ms
    spec:
      serviceAccountName: httpbin
      containers:
      - image: docker.io/kennethreitz/httpbin
        imagePullPolicy: IfNotPresent
        name: httpbin
        ports:
        - containerPort: 80

Performance

The potential performance benefit depends on many factors. For example, the size of the cpuset Envoy is running on, incoming traffic pattern, encryption type (RSA or ECDSA), and key size.

Below, we show performance based on the total latency between k6, gateway and Fortio server. These show relative performance improvement using the CryptoMB provider, and are in no way representative of Istio’s general performance or benchmark results. Our measurements use different client tools (k6 and fortio), different setup (client, gateway and server running on separate nodes) and we create a new TLS handshake with every HTTP request.

We have published a white paper with general cryptographic performance numbers.

Configuration used in above comparison.

• Azure AKS Kubernetes cluster
  • v1.21
  • Three-node cluster
  • Each node Standard_D4ds_v5: 3rd Generation Intel® Xeon® Platinum 8370C (Ice Lake), 4 vCPU, 16 GB memory
• Istio
  • 1.14-dev
  • Istio ingress gateway pod
    • resources.request.cpu: 2
    • resources.request.memory: 4 GB
    • resources.limits.cpu: 2
    • resources.limits.memory: 4 GB
• K6
  • loadimpact/k6:latest
• Fortio
  • fortio/fortio:1.27.0
• K6 client, envoy and fortio pods are forced to run on separate nodes via Kubernetes AntiAffinity and node selectors
• In above picture
  • Istio is installed with above configuration
  • Istio with CryptoMB (AVX-512) with above configuration + below settings

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  components:
    ingressGateways:
    - enabled: true
      name: istio-ingressgateway
      k8s:
        # this controls the SDS service which configures ingress gateway
        podAnnotations:
          proxy.istio.io/config: |
            privateKeyProvider:
              cryptomb:
                pollDelay: 1ms
  values:
    # Annotate pods with
    #     inject.istio.io/templates: sidecar, cryptomb
    sidecarInjectorWebhook:
      templates:
        cryptomb: |
          spec:
            containers:
            - name: istio-proxy

It is almost 5 years since Google, IBM and Lyft launched Istio 0.1 in May 2017. That first version set the standard for what a service mesh should be: traffic management, policy enforcement, and observability, powered by sidecars next to workloads. We’re proud to be the most popular service mesh according to a recent CNCF survey, and look forward to working closer with the CNCF communities around networking and service mesh.

As we deepen our integration with Kubernetes through the Gateway API and gRPC with proxyless mesh — not to mention Envoy, which has grown up beside Istio — we think it’s time to unite the premier Cloud Native stack under a single umbrella.

What’s next?

Today is just the start of a journey. The CNCF Technical Oversight Committee will carefully consider our application, and perform due diligence. After that, they’ll open up for a vote, and if successful, the project will be transferred.

The work we did in establishing guidelines for the Istio trademark through the Open Usage Commons (OUC) will ensure the whole ecosystem can continue to use the Istio trademarks in a free and fair fashion. The trademarks will move to the Linux Foundation but continue to be managed under OUC’s trademark guidelines.

Google currently funds and manages Istio’s build/test infrastructure. The company has committed to continue sponsoring this infrastructure as it moves to management by the CNCF, and it will be supported with credits from Google and other contributors after the transition is complete.

Nothing about our current open governance model has to change as a result of this transfer. We will continue to reward corporate contribution, community influence and long-term maintainership through our Steering Committee and Technical Oversight Committee model. Istio is key to the future of Google Cloud and Google intends to continue investing heavily in the project.

We want to thank the ecosystem of Istio users, integrated projects, and professional services vendors. Please send us a PR if you want to be listed on our site!

Istio is the building block for products by over 20 different vendors. No other service mesh has a comparable footprint. We want to thank all the clouds, technology enterprises, startups and everyone else who has built a product based on Istio, or who makes Istio available with their hosted Kubernetes service. We look forward to our continued collaboration.

Finally, we want to thank Google for their stewardship of the Istio community to date, their immeasurable contributions to Istio, and for their continued support during this transition.

See also

For more perspectives on today’s news, please read blog posts from Google, IBM, Tetrate, VMware, Solo.io, Aspen Mesh and Red Hat.

$ istioctl proxy-status
Error: unable to find any Istiod instances

Notice that the error message doesn’t just say that it’s unable to access the istiod service, it specifically mentions its inability to find istiod instances. This is because the istioctl proxy-status implementation needs to retrieve the sync status of not just any single istiod instance, but rather all of them. When there is more than one istiod instance (replica) running, each instance is only connected to a subset of the service proxies running in the mesh. The istioctl command needs to return the status for the entire mesh, not just the subset managed by one of the instances.

In an ordinary Istio installation where the istiod service is running locally on the cluster (i.e., a primary cluster), the command is implemented by simply finding all of the running istiod pods, calling each one in turn, and then aggregating the result before returning it to the user.

When using a remote cluster, on the other hand, this is not possible since the istiod instances are running outside of the mesh cluster and not accessible to the mesh user. The instances may not even be deployed using pods on a Kubernetes cluster.

Fortunately, istioctl provides a configuration option to address this issue. You can configure istioctl with the address of an external proxy service that will have access to the istiod instances. Unlike an ordinary load-balancer service, which would delegate incoming requests to one of the instances, this proxy service must instead delegate to all of the istiod instances, aggregate the responses, and then return the combined result.

If the external proxy service is, in fact, running on another Kubernetes cluster, the proxy implementation code can be very similar to the implementation code that istioctl runs in the primary cluster case, i.e., find all of the running istiod pods, call each one in turn, and then aggregate the result.

An Istio Ecosystem project that includes an implementation of such an istioctl proxy server can be found here. To try it out, you’ll need two clusters, one of which is configured as a remote cluster using a control plane installed in the other cluster.

Install Istio with a remote cluster topology

To demonstrate istioctl working on a remote cluster, we’ll start by using the external control plane install instructions to set up a single remote cluster mesh with an external control plane running in a separate external cluster.

After completing the installation, we should have two environment variables, CTX_REMOTE_CLUSTER and CTX_EXTERNAL_CLUSTER, containing the context names of the remote (mesh) and external (control plane) clusters, respectively.

We should also have the helloworld and sleep samples running in the mesh, i.e., on the remote cluster:

$ kubectl get pod -n sample --context="${CTX_REMOTE_CLUSTER}"
NAME                             READY   STATUS    RESTARTS   AGE
helloworld-v1-776f57d5f6-tmpkd   2/2     Running   0          10s
sleep-557747455f-v627d           2/2     Running   0          9s

Notice that if you try to run istioctl proxy-status in the remote cluster, you will see the error message described earlier:

$ istioctl proxy-status --context="${CTX_REMOTE_CLUSTER}"
Error: unable to find any Istiod instances

Configure istioctl to use the sample proxy service

To configure istioctl, we first need to deploy the proxy service next to the running istiod pods. In our installation, we’ve deployed the control plane in the external-istiod namespace, so we start the proxy service on the external cluster using the following command:

$ kubectl apply -n external-istiod --context="${CTX_EXTERNAL_CLUSTER}" \
    -f https://raw.githubusercontent.com/istio-ecosystem/istioctl-proxy-sample/main/istioctl-proxy.yaml
service/istioctl-proxy created
serviceaccount/istioctl-proxy created
secret/jwt-cert-key-secret created
deployment.apps/istioctl-proxy created
role.rbac.authorization.k8s.io/istioctl-proxy-role created
rolebinding.rbac.authorization.k8s.io/istioctl-proxy-role created

You can run the following command to confirm that the istioctl-proxy service is running next to istiod:

$ kubectl get po -n external-istiod --context="${CTX_EXTERNAL_CLUSTER}"
NAME                              READY   STATUS    RESTARTS   AGE
istioctl-proxy-664bcc596f-9q8px   1/1     Running   0          15s
istiod-666fb6694d-jklkt           1/1     Running   0          5m31s

The proxy service is a gRPC server that is serving on port 9090:

$ kubectl get svc istioctl-proxy -n external-istiod --context="${CTX_EXTERNAL_CLUSTER}"
NAME             TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
istioctl-proxy   ClusterIP   172.21.127.192   <none>        9090/TCP   11m

Before we can use it, however, we need to expose it outside of the external cluster. There are many ways to do that, depending on the deployment environment. In our setup, we have an ingress gateway running on the external cluster, so we could update it to also expose port 9090, update the associated virtual service to direct port 9090 requests to the proxy service, and then configure istioctl to use the gateway address for the proxy service. This would be a “proper” approach.

However, since this is just a simple demonstration where we have access to both clusters, we will simply port-forward the proxy service to localhost:

$ kubectl port-forward -n external-istiod service/istioctl-proxy 9090:9090 --context="${CTX_EXTERNAL_CLUSTER}"

We now configure istioctl to use localhost:9090 to access the proxy by setting the ISTIOCTL_XDS_ADDRESS environment variable:

$ export ISTIOCTL_XDS_ADDRESS=localhost:9090
$ export ISTIOCTL_ISTIONAMESPACE=external-istiod
$ export ISTIOCTL_PREFER_EXPERIMENTAL=true

Because our control plane is running in the external-istiod namespace, instead of the default istio-system, we also need to set the ISTIOCTL_ISTIONAMESPACE environment variable.

Setting ISTIOCTL_PREFER_EXPERIMENTAL is optional. It instructs istioctl to redirect istioctl command calls to an experimental equivalent, istioctl x command, for any command that has both a stable and experimental implementation. In our case we need to use istioctl x proxy-status, the version that implements the proxy delegation feature.

Run the istioctl proxy-status command

Now that we’re finished configuring istioctl we can try it out by running the proxy-status command again:

$ istioctl proxy-status --context="${CTX_REMOTE_CLUSTER}"
NAME                                                      CDS        LDS        EDS        RDS        ISTIOD         VERSION
helloworld-v1-776f57d5f6-tmpkd.sample                     SYNCED     SYNCED     SYNCED     SYNCED     <external>     1.12.1
istio-ingressgateway-75bfd5668f-lggn4.external-istiod     SYNCED     SYNCED     SYNCED     SYNCED     <external>     1.12.1
sleep-557747455f-v627d.sample                             SYNCED     SYNCED     SYNCED     SYNCED     <external>     1.12.1

As you can see, this time it correctly displays the sync status of all the services running in the mesh. Notice that the ISTIOD column returns the generic value <external>, instead of the instance name (e.g., istiod-666fb6694d-jklkt) that would be displayed if the pod was running locally. In this case, this detail is not available, or needed, by the mesh user. It’s only available on the external cluster for the mesh operator to see.

Summary

In this article, we used a sample proxy server to configure istioctl to work with an external control plane installation. We’ve seen how some of the istioctl CLI commands don’t work out of the box on a remote cluster managed by an external control plane. Commands such as istioctl proxy-status, among others, need access to the istiod service instances managing the mesh, which are unavailable when the control plane is running outside of the mesh cluster. To address this issue, istioctl was configured to delegate to a proxy server, running along side the external control plane, which accesses the istiod instances on its behalf.

In 2021, more than 4,000 people from across 84 countries joined the event online, to hear from 27 end-user companies how they are using Istio in production. Participants were able to learn how Airbnb navigated scalability issues to finally find a solution in Istio, how HP set up a secure and wise platform with Istio, and how eBay used Istio to create federated access points, among many more examples of using Istio in production.

IstioCon 2022 will be an industry-focused event, a platform to connect contributors and users to discuss uses of Istio in different architectural setups, what are some limitations, and where to take the project next. The main focus will be in end-user companies, as we look forward to sharing a diversity of case studies showing how to use Istio in production. The content will be categorized according to expertise level.

This community-led event also has an interactive social hour to take the load off and mesh with the Istio community, vendors, and maintainers. Participation in the event is free of charge, register today for a chance to get the conference t-shirt!

There are shortcomings in using iptables to perform this interception. Since netfilter is a highly versatile tool for filtering packets, several routing rules and data filtering processes are applied before reaching the destination socket. For example, from the network layer to the transport layer, netfilter will be used for processing for several times with the rules predefined, like pre_routing, post_routing and etc. When the packet becomes a TCP packet or UDP packet, and is forwarded to user space, some additional steps like packet validation, protocol policy processing and destination socket searching will be performed. When a sidecar is configured to intercept traffic, the original data path can become very long, since duplicated steps are performed several times.

Over the past two years, eBPF has become a trending technology, and many projects based on eBPF have been released to the community. Tools like Cilium and Pixie show great use cases for eBPF in observability and network packet processing. With eBPF’s sockops and redir capabilities, data packets can be processed efficiently by directly being transported from an inbound socket to an outbound socket. In an Istio mesh, it is possible to use eBPF to replace iptables rules, and accelerate the data plane by shortening the data path.

We have created an open source project called Merbridge, and by applying the following command to your Istio-managed cluster, you can use eBPF to achieve such network acceleration.

$ kubectl apply -f https://raw.githubusercontent.com/merbridge/merbridge/main/deploy/all-in-one.yaml

With Merbridge, the packet datapath can be shortened directly from one socket to another destination socket, and here’s how it works.

Using eBPF sockops for performance optimization

Network connection is essentially socket communication. eBPF provides a function bpf_msg_redirect_hash, to directly forward the packets sent by the application in the inbound socket to the outbound socket. By entering the function mentioned before, developers can perform any logic to decide the packet destination. According to this characteristic, the datapath of packets can noticeably be optimized in the kernel.

The sock_map is the crucial piece in recording information for packet forwarding. When a packet arrives, an existing socket is selected from the sock_map to forward the packet to. As a result, we need to save all the socket information for packets to make the transportation process function properly. When there are new socket operations — like a new socket being created — the sock_ops function is executed. The socket metadata is obtained and stored in the sock_map to be used when processing packets. The common key type in the sock_map is a “quadruple” of source and destination addresses and ports. With the key and the rules stored in the map, the destination socket will be found when a new packet arrives.

The Merbridge approach

Let’s introduce the detailed design and implementation principles of Merbridge step by step, with a real scenario.

Istio sidecar traffic interception based on iptables

When external traffic hits your application’s ports, it will be intercepted by a PREROUTING rule in iptables, forwarded to port 15006 of the sidecar container, and handed over to Envoy for processing. This is shown as steps 1-4 in the red path in the above diagram.

Envoy processes the traffic using the policies issued by the Istio control plane. If allowed, the traffic will be sent to the actual container port of the application container.

When the application tries to access other services, it will be intercepted by an OUTPUT rule in iptables, and then be forwarded to port 15001 of the sidecar container, where Envoy is listening. This is steps 9-12 on the red path, similar to inbound traffic processing.

Traffic to the application port needs to be forwarded to the sidecar, then sent to the container port from the sidecar port, which is overhead. Moreover, iptables’ versatility determines that its performance is not always ideal because it inevitably adds delays to the whole datapath with different filtering rules applied. Although iptables is the common way to do packet filtering, in the Envoy proxy case, the longer datapath amplifies the bottleneck of packet filtering process in the kernel.

If we use sockops to directly connect the sidecar’s socket to the application’s socket, the traffic will not need to go through iptables rules, and thus performance can be improved.

Processing outbound traffic

As mentioned above, we would like to use eBPF’s sockops to bypass iptables to accelerate network requests. At the same time, we also do not want to modify any parts of Istio, to make Merbridge fully adaptive to the community version. As a result, we need to simulate what iptables does in eBPF.

Traffic redirection in iptables utilizes its DNAT function. When trying to simulate the capabilities of iptables using eBPF, there are two main things we need to do:

1. Modify the destination address, when the connection is initiated, so that traffic can be sent to the new interface.
2. Enable Envoy to identify the original destination address, to be able to identify the traffic.

For the first part, we can use eBPF’s connect program to process it, by modifying user_ip and user_port.

For the second part, we need to understand the concept of ORIGINAL_DST which belongs to the netfilter module in the kernel.

When an application (including Envoy) receives a connection, it will call the get_sockopt function to obtain ORIGINAL_DST. If going through the iptables DNAT process, iptables will set this parameter, with the “original IP + port” value, to the current socket. Thus, the application can get the original destination address according to the connection.

We have to modify this call process through eBPF’s get_sockopts function. (bpf_setsockopt is not used here because this parameter does not currently support the optname of SO_ORIGINAL_DST).

Referring to the figure below, when an application initiates a request, it will go through the following steps:

1. When the application initiates a connection, the connect program will modify the destination address to 127.x.y.z:15001, and use cookie_original_dst to save the original destination address.
2. In the sockops program, the current socket information and the quadruple are saved in sock_pair_map. At the same time, the same quadruple and its corresponding original destination address will be written to pair_original_dest. (Cookie is not used here because it cannot be obtained in the get_sockopt program).
3. After Envoy receives the connection, it will call the get_sockopt function to read the destination address of the current connection. get_sockopt will extract and return the original destination address from pair_original_dst, according to the quadruple information. Thus, the connection is completely established.
4. In the data transport step, the redir program will read the sock information from sock_pair_map according to the quadruple information, and then forward it directly through bpf_msg_redirect_hash to speed up the request.

Why do we set the destination address to 127.x.y.z instead of 127.0.0.1? When different pods exist, there might be conflicting quadruples, and this gracefully avoids conflict. (Pods’ IPs are different, and they will not be in the conflicting condition at any time.)

Inbound traffic processing

The processing of inbound traffic is basically similar to outbound traffic, with the only difference: revising the port of the destination to 15006.

It should be noted that since eBPF cannot take effect in a specified namespace like iptables, the change will be global, which means that if we use a Pod that is not originally managed by Istio, or an external IP address, serious problems will be encountered — like the connection not being established at all.

As a result, we designed a tiny control plane (deployed as a DaemonSet), which watches all pods — similar to the kubelet watching pods on the node — to write the pod IP addresses that have been injected into the sidecar to the local_pod_ips map.

When processing inbound traffic, if the destination address is not in the map, we will not do anything to the traffic.

Otherwise, the steps are the same as for outbound traffic.

Same-node acceleration

Theoretically, acceleration between Envoy sidecars on the same node can be achieved directly through inbound traffic processing. However, Envoy will raise an error when accessing the application of the current pod in this scenario.

In Istio, Envoy accesses the application by using the current pod IP and port number. With the above scenario, we realized that the pod IP exists in the local_pod_ips map as well, and the traffic will be redirected to the pod IP on port 15006 again because it is the same address that the inbound traffic comes from. Redirecting to the same inbound address causes an infinite loop.

Here comes the question: are there any ways to get the IP address in the current namespace with eBPF? The answer is yes!

We have designed a feedback mechanism: When Envoy tries to establish the connection, we redirect it to port 15006. However, in the sockops step, we will determine if the source IP and the destination IP are the same. If yes, it means the wrong request is sent, and we will discard this connection in the sockops process. In the meantime, the current ProcessID and IP information will be written into the process_ip map, to allow eBPF to support correspondence between processes and IPs.

When the next request is sent, the same process need not be performed again. We will check directly from the process_ip map if the destination address is the same as the current IP address.

Connection relationship

Before applying eBPF using Merbridge, the data path between pods is like:

After applying Merbridge, the outbound traffic will skip many filter steps to improve the performance:

If two pods are on the same machine, the connection can even be faster:

Performance results

Let’s see the effect on overall latency using eBPF instead of iptables (lower is better):

We can also see overall QPS after using eBPF (higher is better). Test results are generated with wrk.

Summary

We have introduced the core ideas of Merbridge in this post. By replacing iptables with eBPF, the data transportation process can be accelerated in a mesh scenario. At the same time, Istio will not be changed at all. This means if you do not want to use eBPF any more, just delete the DaemonSet, and the datapath will be reverted to the traditional iptables-based routing without any problems.

Merbridge is a completely independent open source project. It is still at an early stage, and we are looking forward to having more users and developers to get engaged. It would be greatly appreciated if you would try this new technology to accelerate your mesh, and provide us with some feedback!

See also

• Merbridge on GitHub
• Using eBPF instead of iptables to optimize the performance of service grid data plane by Liu Xu, Tencent
• Sidecar injection and transparent traffic hijacking process in Istio explained in detail by Jimmy Song, Tetrate
• Accelerate the Istio data plane with eBPF by Yizhou Xu, Intel
• Envoy’s Original Destination filter

IstioCon provides an opportunity to showcase the lessons learned from running Istio in production, hands-on experiences from the Istio community, and will feature maintainers from across the Istio ecosystem. This year’s IstioCon features sessions focused on sharing real world examples, case studies, and success stories that can inspire newcomers to use Istio in production. The content will range from introductory to advanced levels, split into four main topic tracks:

• Getting started & getting involved
• Tools, features & functionality: Observability, traceability, and other things built on top of Istio.
• Infrastructure & networking: How Istio works, with deep-dives into performance, cost, and multi-cloud environments.
• Tech evolution & what’s next: The evolution of Istio, new standards, new extensions, and how to address problems that are interesting to tackle.

At this time, we encourage Istio users, developers, partners, and advocates to submit a session proposal through the conference’s CFP portal. The conference offers a mix of keynotes, technical talks, lightning talks, workshops, and roadmap sessions. Choose from the following formats to submit a session proposal for IstioCon:

• Presentation: 40 minute presentation, maximum of 2 speakers
• Panel: 40 minutes of discussion among 3 to 5 speakers
• Workshop: 160 minute (2h 40m), in-depth, hands-on presentation with 1–4 speakers
• Lighting Talk: 10 minute presentation, limited to 1 speaker

This community-led event also has an interactive social hour to take the load off and mesh with the Istio community, vendors, and maintainers. Participation in the event is free of charge, and will only require participants to register in order to attend.

Stay tuned to hear more about this conference, and we hope you can join us at IstioCon 2022!

Some of the complexities involved with joining a virtual machine are due to the sheer number of features that a service mesh provides. But, what if you only need a subset of those features? For example, secure communication from your virtual machine to services running inside your service mesh. With only a few tradeoffs, you can give your virtual machine service mesh features without all of the overhead.

What about local development? As more and more micro-services are deployed to Kubernetes and as your dependency graph resembles a spider web, it has become increasingly difficult to do local development. What if a local machine could simply join a service mesh and make calls to mesh applications. This solution could potentially save time and money by not requiring developers to wait for their code to be deployed.

Reduce Complexity

Today, adding a virtual machine to your Istio service mesh involves a lot of moving parts. You must create a Kubernetes service account, Istio workload entry and then generate configuration all before on-boarding a single virtual machine. There are also complexities to automating this, especially for auto scaling VMs. Finally you are required to expose Istiod externally to your cluster.

The complexity of adding a virtual machine comes from the expectation that the VM should participate 100% within the service mesh. For many this is not a necessity, by looking at the actual requirements of your system you may be able to simplify your virtual machine on-boarding and still get the features you need.

So what are some use cases that could be met but yet still make virtual machines easier to work with service mesh?

Single Direction Traffic Flow

Sometimes a virtual machine just needs to talk securely to applications within the service mesh. This is often the case when migrating VM based applications to Kubernetes in which other VMs may have depended on those applications. With the described approach below you can still achieve this without all of the operational overhead as shown above.

Developer Access to Service Mesh

Engineers often do not have the resources to run all of the required micro-services for their environment. The approach below explains how you can achieve this in the same way that virtual machines securely communicate with mesh applications.

Decouple Envoy and Istio

The largest amount of complexity as it relates to virtual machines is the connecting of envoy to istiod to get its configuration. A simpler approach is to just not connect them anymore. Even though Istio will no longer know about the virtual machines that are communicating in the mesh, that communication can still be secure and authenticated. The trick is issuing virtual machines their own workload certificates that are rooted in the same trust chain as the mesh workloads. This also means that the end user will be responsible for configuring envoy manually on the virtual machine. For most this shouldn’t be an issue because it is not expected that it will change very often.

A Simpler On-boarding Experience

We can achieve a simpler setup by utilizing some built-in Istio features. First we need to expose a secure tunnel for applications outside the mesh to communicate with applications within.

To do this we simply need to create an Istio east-west gateway and enable AUTO_PASSTHROUGH. This automatically configures the east-west gateway to pass traffic through to the correct service over mTLS. This gives your virtual machine end to end authenticated encryption with the application its trying to reach.

Due to the complexity involved in configuring envoy to talk to istiod, it is more practical to directly configure the virtual machine envoy. At first this sounds quite daunting, but due to the reduced complexity we only need to enable a few features to make this work. Envoy will need to be configured to know about each service mesh application that the virtual machine will need to communicate with. We then will configure these as clusters within envoy and set them up to use mTLS communication passing through the east-west gateway in the service mesh. Secondly a listener will need to be exposed to handle incoming traffic from the virtual machine application. Finally certificates will need to be issued for each virtual machine that share the same root of trust as the service mesh applications. This allows end to end encryption as well as the ability to authorize which applications the virtual machine can communicate with.

Easier to Automate

Given that no initialization has to occur on the service mesh cluster when on-boarding a virtual machine, it is much easier to automate. The configuration needed for the virtual machine envoy can be added to your pipeline; the envoy container can either be pulled via docker, or added to your image building infrastructure; the mTLS certificates can also be provisioned and maintained by a third party such as Hashicorp’s Vault.

More Runtime Support

Due to the fact that this installation method does not require access to the underlying OS networking. You can run this approach in more types of environments including Windows and Docker. The only requirement is that your Envoy include the Istio extensions found here. Using Docker, you can now run the Envoy proxy on your local machine and communicate with the service mesh directly.

Advanced Use Cases

gRPC to JSON

This technique can also be leveraged to enable virtual machine applications to communicate with gRPC applications without having to implement the gRPC endpoints. Using envoys gRPC / JSON transformation, the virtual machine application can communicate with its local envoy over REST and envoy will translate that to gRPC.

Multi Direction

Even though your service mesh may not know about the virtual machines that are communicating with it, you can still add them as external endpoints using Service Entries. That service entry could be an HTTPS Load Balancer endpoint that manages traffic to multiple virtual machines. This setup is still often more feasible than fully on-boarding virtual machines into the virtual mesh.

Forwarding Proxy

Maybe installing envoy on every virtual machine is still too complex. An alternative is to run envoy (or an autoscaling group) to run on its own virtual machine and act as a forwarding proxy into the mesh. This is a much simpler solution to accessing mesh services as the virtual machines that run the applications are left untouched.

Part 2…

In part 2, I will explain how to configure Istio as well as a virtual machine to communicate within the mesh. If you would like a preview, feel free to reach out to nick.nellis@solo.io

Special Thanks

A special thanks to Dave Ortiz for this virtual machine idea and congrats to Constant Contact a new registered Istio user!

New WasmPlugin API

With the new WasmPlugin CRD in the extensions.istio.io namespace, we’re introducing a new high-level API for extending the functionality of the Istio proxy with custom Wasm modules. This effort builds on the excellent work that has gone into the Proxy-Wasm specification and implementation over the last two years. From now on, you no longer need to use EnvoyFilter resources to add custom Wasm modules to your proxies. Instead, you can now use a WasmPlugin resource:

apiVersion: extensions.istio.io/v1alpha1
kind: WasmPlugin
metadata:
  name: your-filter
spec:
  selector:
    matchLabels:
      app: server
  phase: AUTHN
  priority: 10
  pluginConfig:
    someSetting: true
    someOtherSetting: false
    youNameIt:
    - first
    - second
  url: docker.io/your-org/your-filter:1.0.0

There are a lot of similarities and a few differences between WasmPlugin and EnvoyFilter, so let’s go through the fields one by one.

The above example deploys a Wasm module to all workloads (including gateway pods) that match the selector field - this very much works the same as in an EnvoyFilter.

The next field below that is the phase. This determines where in the proxy’s filter chain the Wasm module will be injected. We have defined four distinct phases for injection:

• AUTHN: prior to any Istio authentication and authorization filters.
• AUTHZ: after the Istio authentication filters and before any first-class authorization filters, i.e., before AuthorizationPolicy resources have been applied.
• STATS: after all authorization filters and prior to the Istio stats filter.
• UNSPECIFIED_PHASE: let the control plane decide where to insert. This will generally be at the end of the filter chain, right before the router. This is the default value for this phase field.

The pluginConfig field is used for configuring your Wasm plugin. Whatever you put into this field will be encoded in JSON and passed on to your filter, where you can access it in the configuration callback of the Proxy-Wasm SDKs. For example, you can retrieve the config with onConfigure in the C++ SDK, on_configure in the Rust SDK or the OnPluginStart call back in the Go SDK.

The url field specifies where to pull the Wasm module. You’ll notice that the url in this example is a docker URI. Apart from loading Wasm modules via HTTP, HTTPS and the local file system (using file://), we are introducing the OCI image format as the preferred mechanism for distributing Wasm modules.

One last thing to note is currently the Wasm Plugin API only applies to inbound HTTP filter chains. Support for network filters and outbound traffic will be added in the future.

Wasm image specification

We believe that containers are the ideal way to store, publish and manage proxy extensions, so we worked with Solo.io to extend their existing Proxy-Wasm container format with a variant that aims to be compatible with all registries and the CLI toolchain. Depending on your processes, you can now build your proxy extension containers using your existing container CLI tooling such as Docker CLI or buildah.

To learn how to build OCI images, please refer to these instructions.

Image fetcher in Istio agent

Since Istio 1.9, Istio-agent has provided a reliable solution for loading Wasm binaries, fetched from remote HTTP sources configured in the EnvoyFilters, by leveraging the xDS proxy inside istio-agent and Envoy’s Extension Configuration Discovery Service (ECDS). The same mechanism applies for the new Wasm API implementation in Istio 1.12. You can use HTTP remote resources reliably without concern that Envoy might get stuck with a bad configuration when a remote fetch fails.

In addition, Istio 1.12 expands this capability to Wasm OCI images. This means the Istio-agent is now able to fetch Wasm images from any OCI registry including Docker Hub, Google Container Registry(GCR), Amazon Elastic Container Registry (Amazon ECR), etc. After fetching images, Istio-agent extracts and caches Wasm binaries from them, and then inserts them into the Envoy filter chains.

Improvements in Envoy Wasm runtime

The Wasm runtime powered by V8 in Envoy has been shipped since Istio 1.5 and there have been a lot of improvements since then.

WASI supports

First, some of the WASI (WebAssembly System Interface) system calls are now supported. For example, the clock_time_get system call can be made from Wasm programs so you can use std::time::SystemTime::now() in Rust or time.Now().UnixNano() in Go in your Envoy Wasm extensions, just like any other native platform. Another example is random_get is now supported by Envoy, so the “crypto/rand” package is available in Go as a cryptographically secure random number generator. We are also currently looking into file system support as we have seen requests for reading and writing local files from Wasm programs running in Envoy.

Debuggability

Next is the improvement in debuggability. The Envoy runtime now emits the stack trace of your program when it causes runtime errors, for example, when null pointer exceptions occur in C++ or the panic function is called in Go or Rust. While Envoy error messages did not previously include anything about the cause, they now show the trace which you can use to debug your program:

Function: proxy_on_request_headers failed: Uncaught RuntimeError: unreachable
Proxy-Wasm plugin in-VM backtrace:
  0:  0xdbd - runtime._panic
  1:  0x103ab - main.anotherCalculation
  2:  0x10399 - main.someCalculation
  3:  0xea57 - main.myHeaderHandler
  4:  0xea15 - proxy_on_request_headers

The above is an example stack trace from a Go SDK based Wasm extension. You might notice that the output does not include file names and line numbers in the trace. This is an important future work item and open issue related to the DWARF format for WebAssembly and the Exception Handling proposal for the WebAssembly specification.

Strace support for Wasm programs

You can see strace equivalent logs emitted by Envoy. With Istio proxy’s component log level wasm:trace, you can observe all the system calls and Proxy-Wasm ABI calls that go across the boundary between Wasm virtual machines and Envoy. The following is an example of such an strace log stream:

[host->vm] proxy_on_context_create(2, 1)
[host<-vm] proxy_on_context_create return: void
[host->vm] proxy_on_request_headers(2, 8, 1)
[vm->host] wasi_snapshot_preview1.random_get(86928, 32)
[vm<-host] wasi_snapshot_preview1.random_get return: 0
[vm->host] env.proxy_log(2, 87776, 18)

This is especially useful to debug a Wasm program’s execution at runtime, for example, to verify it is not making any malicious system calls.

Arbitrary Prometheus namespace for in-Wasm metrics

The last update is about metrics. Wasm extensions have been able to define their own custom metrics and expose them in Envoy, just like any other metric, but prior to Istio 1.12, all of these custom metrics were prefixed with the envoy_ Prometheus namespace and users were not able to use their own namespaces. Now, you can choose whatever namespace you want, and your metrics will be exposed in Envoy as-is, without being prefixed by envoy_.

Note that in order to actually expose these custom metrics, you have to configure ProxyConfig.proxyStatsMatcher in meshConfig for global configuration or in proxy.istio.io/config for per proxy configuration. For detail, please refer to Envoy Statistics.

Future work and looking for feedback

Although we have announced the alpha availability of Wasm plugins, there is still a lot of work left to be done. One important work item is “Image pull secrets” support in the Wasm API which will allow you to easily consume OCI images in a private repository. Others include first-class support for L4 filters, signature verification of Wasm binaries, runtime improvements in Envoy, Proxy-Wasm SDK improvements, documentation, etc.

This is just the beginning of our plan to provide 1st-class Wasm support in Istio. We would love to hear your feedback so that we can improve the developer experience using Wasm plugins, in future releases of Istio!

Istio 1.11 adds experimental support for adding gRPC services directly to the mesh. We support basic service discovery, some VirtualService based traffic policy, and mutual TLS.

Supported Features

The current implementation of the xDS APIs within gRPC is limited in some areas compared to Envoy. The following features should work, although this is not an exhaustive list and other features may have partial functionality:

• Basic service discovery. Your gRPC service can reach other pods and virtual machines registered in the mesh.
• DestinationRule:
  • Subsets: Your gRPC service can split traffic based on label selectors to different groups of instances.
  • The only Istio loadBalancer currently supported is ROUND_ROBIN, consistentHash will be added in future versions of Istio (it is supported by gRPC).
  • tls settings are restricted to DISABLE or ISTIO_MUTUAL. Other modes will be treated as DISABLE.
• VirtualService:
  • Header match and URI match in the format /ServiceName/RPCName.
  • Override destination host and subset.
  • Weighted traffic shifting.
• PeerAuthentication:
  • Only DISABLE and STRICT are supported. Other modes will be treated as DISABLE.
  • Support for auto-mTLS may exist in a future release.

Other features including faults, retries, timeouts, mirroring and rewrite rules may be supported in a future release. Some of these features are awaiting implementation in gRPC, and others require work in Istio to support. The status of xDS features in gRPC can be found here. The status of Istio’s support will exist in future official docs.

Architecture Overview

Although this doesn’t use a proxy for data plane communication, it still requires an agent for initialization and communication with the control-plane. First, the agent generates a bootstrap file at startup the same way it would generate bootstrap for Envoy. This tells the gRPC library how to connect to istiod, where it can find certificates for data plane communication, and what metadata to send to the control plane. Next, the agent acts as an xDS proxy, connecting and authenticating with istiod on the application’s behalf. Finally, the agent fetches and rotates certificates used in data plane traffic.

Changes to application code

To enable the xDS features in gRPC, there are a handful of required changes your application must make. Your gRPC version should be at least 1.39.0.

In the client

The following side-effect import will register the xDS resolvers and balancers within gRPC. It should be added in your main package or in the same package calling grpc.Dial.

import _ "google.golang.org/grpc/xds"

When creating a gRPC connection the URL must use the xds:/// scheme.

conn, err := grpc.DialContext(ctx, "xds:///foo.ns.svc.cluster.local:7070")

Additionally, for (m)TLS support, a special TransportCredentials option has to be passed to DialContext. The FallbackCreds allow us to succeed when istiod doesn’t send security config.

import "google.golang.org/grpc/credentials/xds"

...

creds, err := xds.NewClientCredentials(xds.ClientOptions{
FallbackCreds: insecure.NewCredentials()
})
// handle err
conn, err := grpc.DialContext(
ctx,
"xds:///foo.ns.svc.cluster.local:7070",
grpc.WithTransportCredentials(creds),
)

On the server

To support server-side configurations, such as mTLS, there are a couple of modifications that must be made.

First, we use a special constructor to create the GRPCServer:

import "google.golang.org/grpc/xds"

...

server = xds.NewGRPCServer()
RegisterFooServer(server, &fooServerImpl)

If your protoc generated Go code is out of date, you may need to regenerate it to be compatible with the xDS server. Your generated RegisterFooServer function should look like the following:

func RegisterFooServer(s grpc.ServiceRegistrar, srv FooServer) {
s.RegisterService(&FooServer_ServiceDesc, srv)
}

Finally, as with the client-side changes, we must enable security support:

creds, err := xds.NewServerCredentials(xdscreds.ServerOptions{FallbackCreds: insecure.NewCredentials()})
// handle err
server = xds.NewGRPCServer(grpc.Creds(creds))

In your Kubernetes Deployment

Assuming your application code is compatible, the Pod simply needs the annotation inject.istio.io/templates: grpc-agent. This adds a sidecar container running the agent described above, and some environment variables that gRPC uses to find the bootstrap file and enable certain features.

For gRPC servers, your Pod should also be annotated with proxy.istio.io/config: '{"holdApplicationUntilProxyStarts": true}' to make sure the in-agent xDS proxy and bootstrap file are ready before your gRPC server is initialized.

Example

In this guide you will deploy echo, an application that already supports both server-side and client-side proxyless gRPC. With this app you can try out some supported traffic policies enabling mTLS.

Prerequisites

This guide requires the Istio (1.11+) control plane to be installed before proceeding.

Deploy the application

Create an injection-enabled namespace echo-grpc. Next deploy two instances of the echo app as well as the Service.

$ kubectl create namespace echo-grpc
$ kubectl label namespace echo-grpc istio-injection=enabled
$ kubectl -n echo-grpc apply -f samples/grpc-echo/grpc-echo.yaml

Make sure the two pods are running:

$ kubectl -n echo-grpc get pods
NAME                       READY   STATUS    RESTARTS   AGE
echo-v1-69d6d96cb7-gpcpd   2/2     Running   0          58s
echo-v2-5c6cbf6dc7-dfhcb   2/2     Running   0          58s

Test the gRPC resolver

First, port-forward 17171 to one of the Pods. This port is a non-xDS backed gRPC server that allows making requests from the port-forwarded Pod.

$ kubectl -n echo-grpc port-forward $(kubectl -n echo-grpc get pods -l version=v1 -ojsonpath='{.items[0].metadata.name}') 17171 &

Next, we can fire off a batch of 5 requests:

$ grpcurl -plaintext -d '{"url": "xds:///echo.echo-grpc.svc.cluster.local:7070", "count": 5}' :17171 proto.EchoTestService/ForwardEcho | jq -r '.output | join("")'  | grep Hostname
Handling connection for 17171
[0 body] Hostname=echo-v1-7cf5b76586-bgn6t
[1 body] Hostname=echo-v2-cf97bd94d-qf628
[2 body] Hostname=echo-v1-7cf5b76586-bgn6t
[3 body] Hostname=echo-v2-cf97bd94d-qf628
[4 body] Hostname=echo-v1-7cf5b76586-bgn6t

You can also use Kubernetes-like name resolution for short names:

$ grpcurl -plaintext -d '{"url": "xds:///echo:7070"}' :17171 proto.EchoTestService/ForwardEcho | jq -r '.output | join
("")'  | grep Hostname
[0 body] Hostname=echo-v1-7cf5b76586-ltr8q
$ grpcurl -plaintext -d '{"url": "xds:///echo.echo-grpc:7070"}' :17171 proto.EchoTestService/ForwardEcho | jq -r
'.output | join("")'  | grep Hostname
[0 body] Hostname=echo-v1-7cf5b76586-ltr8q
$ grpcurl -plaintext -d '{"url": "xds:///echo.echo-grpc.svc:7070"}' :17171 proto.EchoTestService/ForwardEcho | jq -r
'.output | join("")'  | grep Hostname
[0 body] Hostname=echo-v2-cf97bd94d-jt5mf

Creating subsets with destination rule

First, create a subset for each version of the workload.

$ cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: echo-versions
  namespace: echo-grpc
spec:
  host: echo.echo-grpc.svc.cluster.local
  subsets:
  - name: v1
    labels:
      version: v1
  - name: v2
    labels:
      version: v2
EOF

Traffic shifting

Using the subsets defined above, you can send 80 percent of the traffic to a specific version:

$ cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: echo-weights
  namespace: echo-grpc
spec:
  hosts:
  - echo.echo-grpc.svc.cluster.local
  http:
  - route:
    - destination:
        host: echo.echo-grpc.svc.cluster.local
        subset: v1
      weight: 20
    - destination:
        host: echo.echo-grpc.svc.cluster.local
        subset: v2
      weight: 80
EOF

Now, send a set of 10 requests:

$ grpcurl -plaintext -d '{"url": "xds:///echo.echo-grpc.svc.cluster.local:7070", "count": 10}' :17171 proto.EchoTestService/ForwardEcho | jq -r '.output | join("")'  | grep ServiceVersion

The response should contain mostly v2 responses:

[0 body] ServiceVersion=v2
[1 body] ServiceVersion=v2
[2 body] ServiceVersion=v1
[3 body] ServiceVersion=v2
[4 body] ServiceVersion=v1
[5 body] ServiceVersion=v2
[6 body] ServiceVersion=v2
[7 body] ServiceVersion=v2
[8 body] ServiceVersion=v2
[9 body] ServiceVersion=v2

Enabling mTLS

Due to the changes to the application itself required to enable security in gRPC, Istio’s traditional method of automatically detecting mTLS support is unreliable. For this reason, the initial release requires explicitly enabling mTLS on both the client and server.

To enable client-side mTLS, apply a DestinationRule with tls settings:

$ cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: echo-mtls
  namespace: echo-grpc
spec:
  host: echo.echo-grpc.svc.cluster.local
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL
EOF

Now an attempt to call the server that is not yet configured for mTLS will fail.

$ grpcurl -plaintext -d '{"url": "xds:///echo.echo-grpc.svc.cluster.local:7070"}' :17171 proto.EchoTestService/ForwardEcho | jq -r '.output | join("")'
Handling connection for 17171
ERROR:
Code: Unknown
Message: 1/1 requests had errors; first error: rpc error: code = Unavailable desc = all SubConns are in TransientFailure

To enable server-side mTLS, apply a PeerAuthentication.

$ cat <<EOF | kubectl apply -f -
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: echo-mtls
  namespace: echo-grpc
spec:
  mtls:
    mode: STRICT
EOF

Requests will start to succeed after applying the policy.

$ grpcurl -plaintext -d '{"url": "xds:///echo.echo-grpc.svc.cluster.local:7070"}' :17171 proto.EchoTestService/ForwardEcho | jq -r '.output | join("")'
Handling connection for 17171
[0] grpcecho.Echo(&{xds:///echo.echo-grpc.svc.cluster.local:7070 map[] 0  5s false })
[0 body] x-request-id=0
[0 body] Host=echo.echo-grpc.svc.cluster.local:7070
[0 body] content-type=application/grpc
[0 body] user-agent=grpc-go/1.39.1
[0 body] StatusCode=200
[0 body] ServiceVersion=v1
[0 body] ServicePort=17070
[0 body] Cluster=
[0 body] IP=10.68.1.18
[0 body] IstioVersion=
[0 body] Echo=
[0 body] Hostname=echo-v1-7cf5b76586-z5p8l

Limitations

The initial release comes with several limitations that may be fixed in a future version:

• Auto-mTLS isn’t supported, and permissive mode isn’t supported. Instead we require explicit mTLS configuration with STRICT on the server and ISTIO_MUTUAL on the client. Envoy can be used during the migration to STRICT.
• grpc.Serve(listener) or grpc.Dial("xds:///...") called before the bootstrap is written or xDS proxy is ready can cause a failure. holdApplicationUntilProxyStarts can be used to work around this, or the application can be more robust to these failures.
• If the xDS-enabled gRPC server uses mTLS then you will need to make sure your health checks can work around this. Either a separate port should be used, or your health-checking client needs a way to get the proper client certificates.
• The implementation of xDS in gRPC does not match Envoys. Certain behaviors may be different, and some features may be missing. The feature status for gRPC provides more detail. Make sure to test that any Istio configuration actually applies on your proxyless gRPC apps.

Performance

Experiment Setup

• Using Fortio, a Go-based load testing app
  • Slightly modified, to support gRPC’s XDS features (PR)
• Resources:
  • GKE 1.20 cluster with 3 e2-standard-16 nodes (16 CPUs + 64 GB memory each)
  • Fortio client and server apps: 1.5 vCPU, 1000 MiB memory
  • Sidecar (istio-agent and possibly Envoy proxy): 1 vCPU, 512 MiB memory
• Workload types tested:
  • Baseline: regular gRPC with no Envoy proxy or Proxyless xDS in use
  • Envoy: standard istio-agent + Envoy proxy sidecar
  • Proxyless: gRPC using the xDS gRPC server implementation and xds:/// resolver on the client
  • mTLS enabled/disabled via PeerAuthentication and DestinationRule

Latency

There is a marginal increase in latency when using the proxyless gRPC resolvers. Compared to Envoy this is a massive improvement that still allows for advanced traffic management features and mTLS.

istio-proxy container resource usage

Even though we still require an agent, the agent uses less than 0.1% of a full vCPU, and only 25 MiB of memory, which is less than half of what running Envoy requires.

These metrics don’t include additional resource usage by gRPC in the application container, but serve to demonstrate the resource usage impact of the istio-agent when running in this mode.

Lack of Protocols Support in Service Mesh

We are now facing some challenges with service meshes:

• Istio and other popular service mesh implementations have very limited support for layer 7 protocols other than HTTP and gRPC.
• Envoy RDS(Route Discovery Service) is solely designed for HTTP. Other protocols such as Dubbo and Thrift can only use listener in-line routes for traffic management, which breaks existing connections when routes change.
• It takes a lot of effort to introduce a proprietary protocol into a service mesh. You’ll need to write an Envoy filter to handle the traffic in the data plane, and a control plane to manage those Envoys.

Those obstacles make it very hard, if not impossible, for users to manage the traffic of other widely-used layer-7 protocols in microservices. For example, in a microservices application, we may have the below protocols:

• RPC: HTTP, gRPC, Thrift, Dubbo, Proprietary RPC Protocol …
• Messaging: Kafka, RabbitMQ …
• Cache: Redis, Memcached …
• Database: MySQL, PostgreSQL, MongoDB …

If you have already invested a lot of effort in migrating to a service mesh, of course, you want to get the most out of it — managing the traffic of all the protocols in your microservices.

Aeraki’s approach

To address these problems, we create an open-source project, Aeraki Mesh, to provide a non-intrusive, extendable way to manage any layer-7 traffic in an Istio service mesh.

As this diagram shows, Aeraki Framework consists of the following components:

• Aeraki: Aeraki provides high-level, user-friendly traffic management rules to operations, translates the rules to envoy filter configurations, and leverages Istio’s EnvoyFilter API to push the configurations to the sidecar proxies. Aeraki also serves as the RDS server for MetaProtocol proxies in the data plane. Contrary to Envoy RDS, which focuses on HTTP, Aeraki RDS is aimed to provide a general dynamic route capability for all layer-7 protocols.
• MetaProtocol Proxy: MetaProtocol Proxy provides common capabilities for Layer-7 protocols, such as load balancing, circuit breaker, load balancing, routing, rate limiting, fault injection, and auth. Layer-7 protocols can be built on top of MetaProtocol. To add a new protocol into the service mesh, the only thing you need to do is implementing the codec interface and a couple of lines of configuration. If you have special requirements which can’t be accommodated by the built-in capabilities, MetaProtocol Proxy also has an application-level filter chain mechanism, allowing users to write their own layer-7 filters to add custom logic into MetaProtocol Proxy.

Dubbo and Thrift have already been implemented based on MetaProtocol. More protocols are on the way. If you’re using a close-source, proprietary protocol, you can also manage it in your service mesh simply by writing a MetaProtocol codec for it.

Most request/response style, stateless protocols can be built on top of the MetaProtocol Proxy. However, some protocols’ routing policies are too “special” to be normalized in MetaProtocol. For example, Redis proxy uses a slot number to map a client query to a specific Redis server node, and the slot number is computed by the key in the request. Aeraki can still manage those protocols as long as there’s an available Envoy Filter in the Envoy proxy side. Currently, for protocols in this category, Redis and Kafka are supported in Aeraki.

Deep Dive Into MetaProtocol

Let’s look into how MetaProtocol works. Before MetaProtocol is introduced, if we want to proxy traffic for a specific protocol, we need to write an Envoy filter that understands that protocol and add the code to manipulate the traffic, including routing, header modification, fault injection, traffic mirroring, etc.

For most request/response style protocols, the code for traffic manipulation is very similar. Therefore, to avoid duplicating these functionalities in different Envoy filters, Aeraki Framework implements most of the common functions of a layer-7 protocol proxy in a single place — the MetaProtocol Proxy filter.

This approach significantly lowers the barrier to write a new Envoy filter: instead of writing a fully functional filter, now you only need to implement the codec interface. In addition to that, the control plane is already in place — Aeraki works at the control plane to provides MetaProtocol configuration and dynamic routes for all protocols built on top of MetaProtocol.

There are two important data structures in MetaProtocol Proxy: Metadata and Mutation. Metadata is used for routing, and Mutation is used for header manipulation.

At the request path, the decoder(the decode method of the codec implementation) populates the Metadata data structure with key-value pairs parsed from the request, then the Metadata will be passed to the MetaProtocol Router. The Router selects an appropriate upstream cluster after matching the route configuration it receives from Aeraki via RDS and the Metadata.

A custom filter can populate the Mutation data structure with arbitrary key-value pairs if the request needs to be modified: adding a header or changing the value of a header. Then the Mutation data structure will be passed to the encoder(the encode method of the codec implementation). The encoder is responsible for writing the key-value pairs into the wire protocol.

The response path is similar to the request path, only in a different direction.

An Example

If you need to implement an application protocol based on MetaProtocol, you can follow the below steps(use Thrift as an example):

Data Plane

• Implement the codec interface to encode and decode the protocol package. You can refer to Dubbo codec and Thrift codec as writing your own implementation.

• Define the protocol with Aeraki ApplicationProtocol CRD, as this YAML snippet shows:

apiVersion: metaprotocol.aeraki.io/v1alpha1
kind: ApplicationProtocol
metadata:
  name: thrift
  namespace: istio-system
spec:
  protocol: thrift
  codec: aeraki.meta_protocol.codec.thrift

Control Plane

You don’t need to implement the control plane. Aeraki watches services and traffic rules, generates the configurations for the sidecar proxies, and sends the configurations to the data plane via EnvoyFilter and MetaProtocol RDS.

Protocol selection

Similar to Istio, protocols are identified by service port prefix. Please name service ports with this pattern: tcp-metaprotocol-{application protocol}-xxx. For example, a Thrift service port should be named tcp-metaprotocol-thrift.

Traffic management

You can change the route via MetaRouter CRD. For example: send 20% of the requests to v1 and 80% to v2:

apiVersion: metaprotocol.aeraki.io/v1alpha1
kind: MetaRouter
metadata:
  name: test-metaprotocol-route
spec:
  hosts:
    - thrift-sample-server.thrift.svc.cluster.local
  routes:
    - name: traffic-spilt
      route:
        - destination:
            host: thrift-sample-server.thrift.svc.cluster.local
            subset: v1
          weight: 20
        - destination:
            host: thrift-sample-server.thrift.svc.cluster.local
            subset: v2
          weight: 80

Hope this helps if you need to manage protocols other than HTTP in a service mesh. Reach out to zhaohuabing if you have any questions.

Reference

• Aeraki Mesh website
• Aeraki Mesh on GitHub
• Live Demo: Kiali Dashboard
• Live Demo: Service Metrics: Grafana
• Live Demo: Service Metrics: Prometheus
• Istio meetup China(Chinese): Full Stack Service Mesh - Manage Any Layer-7 Traffic in an Istio Service Mesh with Aeraki
• IstioCon 2021: How to Manage Any Layer-7 Traffic in an Istio Service Mesh?

During this extended period of support, Istio 1.9 will receive CVE and critical bug fixes only, as our goal is simply to provide users with time to migrate off the release and on to 1.10 or 1.11. And speaking of users, we would love to hear how we’re doing at improving your Day 2 experience of Istio. Is two upgrades per year not the right number? Is a six week upgrade window too short? Please share your thoughts with us on slack (in the user-experience channel), or on twitter. Thanks!

To achieve this goal, the Istio community contracted the NCC Group last year to conduct a third-party security assessment of the project. The goal of the review was “to identify security issues related to the Istio code base, highlight high-risk configurations commonly used by administrators, and provide perspective on whether security features sufficiently address the concerns they are designed to provide”.

NCC Group carried out the review over a period of five weeks with collaboration from subject matter experts across the Istio community. In this blog, we will examine the key findings of the report, actions taken to implement various fixes and recommendations, and our plan of action for continuous security evaluation and improvement of the Istio project. You can download and read the unabridged version of the security assessment report.

Scope and Key Findings

The assessment evaluated Istio’s architecture as a whole for security related issues with focus on key components like istiod (Pilot), Ingress/Egress gateways, and Istio’s overall Envoy usage as its data plane proxy. Additionally, Istio documentation, including security guides, were audited for correctness and clarity. The report was compiled against Istio version 1.6.5, and since then the Product Security Working Group has issued several security releases as new vulnerabilities were disclosed, along with fixes to address concerns raised in the new report.

An important conclusion from the report is that the auditors found no “Critical” issues within the Istio project. This finding validates the continuous and proactive security review and vulnerability management process implemented by Istio’s Product Security Working Group (PSWG). For the remaining issues surfaced by the report, the PSWG went to work on addressing them, and we are glad to report that all issues marked “High”, and several marked “Medium/Low”, have been resolved in the releases following the report.

The report also makes strategic recommendations around creating a hardening guide which is now available in our Security Best Practices guide. This is a comprehensive document which pulls together recommendations from security experts within the Istio community, and industry leaders running Istio in production. Work is underway to create an opinionated and hardened security profile for installing Istio in secure environments, but in the interim we recommend users follow the Security Best Practices guide and configure Istio to meet their security requirements. With that, let’s look at the analysis and resolution for various issues raised in the report.

Resolution and learnings

Inability to secure control plane network communications

The report flags configuration options that were available in older versions of Istio to control how communication is secured to the control plane. Since 1.7, Istio by default secures all control plane communication and many configuration options mentioned in the report to manage control plane encryption are no longer required.

The debug endpoint mentioned in the report is enabled by default (as of Istio 1.10) to allow users to debug their Istio service mesh using the istioctl tool. It can be disabled by setting the environment variable ENABLE_DEBUG_ON_HTTP to false as mentioned in the Security Best Practices guide. Additionally, in an upcoming version (1.11), this debug endpoint will be secured by default and a valid Kubernetes service account token will be required to gain access.

Lack of security related documentation

The report points out gaps in the security related documentation published with Istio 1.6. Since then, we have created a detailed Security Best Practices guide with recommendations to ensure users can deploy Istio securely to meet their requirements. Moving forward, we will continue to augment this documentation with more hardening recommendations. We advise users to monitor the guide for updates.

Lack of VirtualService Gateway field validation enables request hijacking

For this issue, the report uses a valid but permissive Gateway configuration that can cause requests to be routed incorrectly. Similar to the Kubernetes RBAC, Istio APIs, including Gateways, can be tuned to be permissive or restrictive depending upon your requirements. However, the report surfaced missing links in our documentation related to best practices and guiding our users to secure their environments. To address them, we have added a section to our Security Best Practices guide with steps for running Gateways securely. In particular, the section describing using namespace prefixes in hosts specification on Gateway resources is strongly recommended to harden your configuration and prevent this type of request hijacking.

Ingress Gateway configuration generation enables request hijacking

The report raises possible request hijacking when using the default mechanism of selecting gateway workloads by labels across namespaces in a Gateway resource. This behavior was chosen by default as it allows delegation of managing Gateway and VirtualService resources to the applications team while allowing operations teams to centrally manage the ingress gateway workloads for meeting their unique security requirements like running on dedicated nodes for instance. As highlighted in the report, if this deployment topology is not a requirement in your environment it is strongly recommended to co-locate Gateway resources with your gateway workloads and set the environment variable PILOT_SCOPE_GATEWAY_TO_NAMESPACE to true.

Please refer to the gateway deployment topologies guide to understand the various recommended deployment models by the Istio community. Additionally, as mentioned in the Security Best Practices guide, Gateway resource creation should be access controlled using Kubernetes RBAC or other policy enforcement mechanisms to ensure only authorized entities can create them.

Other Medium and Low Severity Issues

There are two medium severity issues reported related to debug information exposed at various levels within the project which can be used to gain access to sensitive information or orchestrate Denial of Service (DOS) attacks. While Istio by default enables these debug interfaces for profiling or enabling tools like “istioctl”, they can be disabled by setting the environment variable ENABLE_DEBUG_ON_HTTP to false as discussed above.

The report correctly points out that various utilities like sudo, tcpdump, etc. installed in the default images shipped by Istio can lead to privilege escalation attacks. These utilities are provided to aid runtime debugging of packets flowing through the mesh, and users are recommended to use hardened versions of these images in production.

The report also surfaces a known architectural limitation with any sidecar proxy-based service mesh implementation which uses iptables for intercepting traffic. This mechanism is susceptible to sidecar proxy bypass, which is a valid concern for secure environments. It can be addressed by following the defense-in-depth recommendation of the Security Best Practices guide. We are also investigating more secure options in collaboration with the Kubernetes community.

The tradeoff between useful and secure

You may have noticed a trend in the findings of the assessment and the recommendations made to address them. Istio provides various configuration options to create a more secure installation based on your requirement, and we have introduced a comprehensive Security Best Practices guide for our users to follow. As Istio is widely adopted in production, it is a tradeoff for us between switching to secure defaults and possible migration issues for our existing users on upgrades. The Istio Product Security Working Group evaluates each of these issues and creates a plan of action to enable secure default on a case-by-case basis after giving our users a number of releases to opt-in the secure configuration and migrate their workloads.

Lastly, there were several lessons for us during and after undergoing a neutral security assessment. The primary one was to ensure our security practices are robust to quickly respond to the findings, and more importantly making security enhancements while maintaining our standards for upgrades without disruption.

To continue this endeavor, we are always looking for feedback and participation in the Istio Product Security Working Group, so join our public meetings to raise issues or learn about what we are doing to keep Istio secure!

We welcomed thousands of Istio users and developers to the first IstioCon in February 2021, and the attendees expressed an interest in participating in more meetups and helping to grow the community at the local level.

To this end, the Istio community united six partners — the China Academy of Information and Communications Technology, Alibaba Cloud, Huawei Cloud, Intel, Tencent Cloud, and Tetrate — to co-host the first official Istio Community Meetup China. We have invited a number of industry experts to share comprehensive Istio technical practices with everyone at an in-person meetup. We will serve some refreshments, and seats are limited, so we will operate on a first-come first-served basis. Please register to attend.

Time and day: 13:30-17:30 (CST), July 10, 2021

Venue: Industrial Internet Exhibition Hall, 2nd Floor, Research Building, China Academy of Information and Communications Technology, No. 52 Huayuan North Road, Haidian District, Beijing

Agenda

We want to thank our community in China who have worked on this event, especially Iris Ding, Wei W Hu, Jimmy Song, Zhonghu Xu, Xining Wang, and Huabing Zhao. We hope you can join!

It’s now time to kick off our 2021 election for Community Seats. Members have two weeks to submit nominations, and voting will run from 12 to 25 July. You can learn all about the election, including how to stand and how to vote, in the istio/community repository on GitHub.

Just like last year, any project member can stand for election. All Istio members who have been active in the last 12 months are eligible to vote.

Technical Oversight Committee updates

We wish to offer our thanks to Dan Berg and Joshua Blatt, both long-time contributors to the Istio project, who have recently taken new jobs outside the service mesh space. That left two vacancies on the Istio Technical Oversight Committee (TOC), responsible for cross-cutting product and design decisions.

TOC members are elected by the Steering Committee from the working group leads, and last week we voted for two new members:

• John Howard, from Google, has become one of the most active contributors to Istio since joining the project in January 2019. He is currently a lead in the Networking working group, and has also served as an Environments working group lead and release manager for version 1.4.
• Brian Avery, from Red Hat, has been active in the Istio community for over 3 years. He served as Release Manager for Istio 1.3 and 1.6, and has remained actively involved in the Istio release process, including introducing tooling for release notes, streamlining the feature maturity process, and working on documentation testing. Most recently, Brian was a lead in the Test and Release and Product Security working groups.

Congratulations to John and Brian!

As our new TOC members step into their roles, they will be vacating their current positions as working group leads. We are always on the lookout for community members who are interested in joining, or leading, Istio working groups. If you’re interested, please reach out in the working group channels on Slack, or during the public working group meetings of your interest.

Similar to services running inside the service mesh, you can configure Istio to detect outliers and failover to a healthy endpoint, while still being completely transparent to your application. In this example, we’ll use Amazon DynamoDB endpoints and pick a primary region that is the same or close to workloads running in a Google Kubernetes Engine (GKE) cluster. We’ll also configure a failover region.

Define external endpoints using a ServiceEntry

Locality load balancing works based on region or zone, which are usually inferred from labels set on the Kubernetes nodes. First, determine the location of your workloads:

$ kubectl describe node | grep failure-domain.beta.kubernetes.io/region
                    failure-domain.beta.kubernetes.io/region=us-east1
                    failure-domain.beta.kubernetes.io/region=us-east1

In this example, the GKE cluster nodes are running in us-east1.

Next, create a ServiceEntry which aggregates the endpoints you want to use. In this example, we have selected mydb.com as the host. This is the address your application should be configured to connect to. Set the locality of the primary endpoint to the same region as your workload:

apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
  name: external-svc-dns
spec:
  hosts:
  - mydb.com
  location: MESH_EXTERNAL
  ports:
  - number: 80
    name: http
    protocol: HTTP
  resolution: DNS
  endpoints:
  - address: dynamodb.us-east-1.amazonaws.com
    locality: us-east1
    ports:
      http: 80
  - address: dynamodb.us-west-1.amazonaws.com
    locality: us-west
    ports:
      http: 80

Let’s deploy a sleep container to use as a test source for sending requests.

$ kubectl apply -f @samples/sleep/sleep.yaml@

From the sleep container try going to http://mydb.com 5 times:

$ for i in {1..5}; do kubectl exec deploy/sleep -c sleep -- curl -sS http://mydb.com; echo; sleep 2; done
healthy: dynamodb.us-east-1.amazonaws.com
healthy: dynamodb.us-west-1.amazonaws.com
healthy: dynamodb.us-west-1.amazonaws.com
healthy: dynamodb.us-east-1.amazonaws.com
healthy: dynamodb.us-east-1.amazonaws.com

You will see that Istio is sending requests to both endpoints. We only want it to send to the endpoint marked with the same region as our nodes.

For that, we need to configure a DestinationRule.

Set failover conditions using a DestinationRule

Istio’s DestinationRule lets you configure load balancing, connection pool, and outlier detection settings. We can specify the conditions used to identify an endpoint as unhealthy and remove it from the load balancing pool.

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: mydynamodb
spec:
  host: mydb.com
  trafficPolicy:
    outlierDetection:
      consecutive5xxErrors: 1
      interval: 15s
      baseEjectionTime: 1m

The above DestinationRule configures the endpoints to be scanned every 15 seconds, and if any endpoint fails with a 5xx error code, even once, it will be marked unhealthy for one minute. If this circuit breaker is not triggered, the traffic will route to the same region as the pod.

If we run our curl again, we should see that traffic is always going to the us-east1 endpoint.

$ for i in {1..5}; do kubectl exec deploy/sleep -c sleep -- curl -sS http://mydb.com; echo; sleep 2; done

healthy: dynamodb.us-east-1.amazonaws.com
healthy: dynamodb.us-east-1.amazonaws.com
healthy: dynamodb.us-east-1.amazonaws.com
healthy: dynamodb.us-east-1.amazonaws.com
healthy: dynamodb.us-east-1.amazonaws.com

Simulate a failure

Next, let’s see what happens if the us-east endpoint goes down. To simulate this, let’s modify the ServiceEntry and set the us-east endpoint to an invalid port:

apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
  name: external-svc-dns
spec:
  hosts:
  - mydb.com
  location: MESH_EXTERNAL
  ports:
  - number: 80
    name: http
    protocol: HTTP
  resolution: DNS
  endpoints:
  - address: dynamodb.us-east-1.amazonaws.com
    locality: us-east1
    ports:
      http: 81 # INVALID - This is purposefully wrong to trigger failover
  - address: dynamodb.us-west-1.amazonaws.com
    locality: us-west
    ports:
      http: 80

Running our curl again shows that traffic is automatically failed over to our us-west region after failing to connect to the us-east endpoint:

$ for i in {1..5}; do kubectl exec deploy/sleep -c sleep -- curl -sS http://mydb.com; echo; sleep 2; done
upstream connect error or disconnect/reset before headers. reset reason: connection failure
healthy: dynamodb.us-west-1.amazonaws.com
healthy: dynamodb.us-west-1.amazonaws.com
healthy: dynamodb.us-west-1.amazonaws.com
healthy: dynamodb.us-west-1.amazonaws.com

You can check the outlier status of the us-east endpoint by running:

$ istioctl pc endpoints <sleep-pod> | grep mydb
ENDPOINT                         STATUS      OUTLIER CHECK     CLUSTER
52.119.226.80:81                 HEALTHY     FAILED            outbound|80||mydb.com
52.94.12.144:80                  HEALTHY     OK                outbound|80||mydb.com

Failover for HTTPS

Configuring failover for external HTTPS services is just as easy. Your application can still continue to use plain HTTP, and you can let the Istio proxy perform the TLS origination to the HTTPS endpoint.

apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
  name: external-svc-dns
spec:
  hosts:
  - mydb.com
  ports:
  - number: 80
    name: http-port
    protocol: HTTP
    targetPort: 443
  resolution: DNS
  endpoints:
  - address: dynamodb.us-east-1.amazonaws.com
    locality: us-east1
  - address: dynamodb.us-west-1.amazonaws.com
    locality: us-west

The above ServiceEntry defines the mydb.com service on port 80 and redirects traffic to the real DynamoDB endpoints on port 443.

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: mydynamodb
spec:
  host: mydb.com
  trafficPolicy:
    tls:
      mode: SIMPLE
    loadBalancer:
      simple: ROUND_ROBIN
      localityLbSetting:
        enabled: true
        failover:
          - from: us-east1
            to: us-west
    outlierDetection:
      consecutive5xxErrors: 1
      interval: 15s
      baseEjectionTime: 1m

The DestinationRule now performs TLS origination and configures the outlier detection. The rule also has a failover field configured where you can specify exactly what regions are failover targets. This is useful when you have several regions defined.

Wrapping Up

Istio’s VirtualService and DestinationRule API’s provide traffic routing, failure recovery and fault injection features so that you can create resilient applications. The ServiceEntry API extends many of these features to external services that are not part of your service mesh.

Background

In Istio 1.6, we added basic support for upgrading the service mesh following a canary pattern using revisions. Using this approach, you can run multiple control planes side-by-side without impacting an existing deployment and slowly migrate workloads from the old control plane to the new.

To support this revision-based upgrade, Istio introduced a istio.io/rev label for namespaces. This indicates which control plane revision should inject sidecar proxies for the workloads in the respective namespace. For example, a label of istio.io/rev=1-9-5 indicates the control plane revision 1-9-5 should inject the data plane using proxies for 1-9-5 for workloads in that namespace.

If you wanted to upgrade the data-plane proxies for a particular namespace, you would update the istio.io/rev label to point to a new version, such as istio.io/rev=1-10-0. Manually changing (or even trying to orchestrate) changes of labels across a large number of namespaces can be error-prone and lead to unintended downtime.

Introducing Revision Tags

In Istio 1.10, we’ve improved revision-based upgrades with a new feature called revision tags. A revision tag reduces the number of changes an operator has to make to use revisions, and safely upgrade an Istio control plane. You use the tag as the label for your namespaces, and assign a revision to that tag. This means you don’t have to change the labels on a namespace while upgrading, and minimizes the number of manual steps and configuration changes.

For example, you can define a tag named prod-stable and point it to the 1-9-5 revision of a control plane. You can also define another tag named prod-canary which points to the 1-10-0 revision. You may have a lot of important namespaces in your cluster, and you can label those namespaces with istio.io/rev=prod-stable. In other namespaces you may be willing to test the new version of Istio, and you can label that namespace istio.io/rev=prod-canary. The tag will indirectly associate those namespaces with the 1-9-5 revision for prod-stable and 1-10-0 for prod-canary respectively.

Once you’ve determined the new control plane is suitable for the rest of the prod-stable namespaces, you can change the tag to point to the new revision. This enables you to update all the namespaces labeled prod-stable to the new 1-10-0 revision without making any changes to the labels on the namespaces. You will need to restart the workloads in a namespace once you’ve changed the tag to point to a different revision.

Once you’re satisfied with the upgrade to the new control-plane revision, you can remove the old control plane.

Stable revision tags in action

To create a new prod-stable tag for a revision 1-9-5, run the following command:

$ istioctl x revision tag set prod-stable --revision 1-9-5

You can then label your namespaces with the istio.io/rev=prod-stable label. Note, if you installed a default revision (i.e., no revision) of Istio, you will first have to remove the standard injection label:

$ kubectl label ns istioinaction istio-injection-
$ kubectl label ns istioinaction istio.io/rev=prod-stable

You can list the tags in your mesh with the following:

$ istioctl x revision tag list

TAG         REVISION NAMESPACES
prod-stable 1-9-5    istioinaction

A tag is implemented with a MutatingWebhookConfiguration. You can verify a corresponding MutatingWebhookConfiguration has been created:

$ kubectl get MutatingWebhookConfiguration

NAME                             WEBHOOKS   AGE
istio-revision-tag-prod-stable   2          75s
istio-sidecar-injector           1          5m32s

Let’s say you are trying to canary a new revision of the control plane based on 1.10.0. First you would install the new version using a revision:

$ istioctl install -y --set profile=minimal --revision 1-10-0

You can create a new tag called prod-canary and point that to your 1-10-0 revision:

$ istioctl x revision tag set prod-canary --revision 1-10-0

Then label your namespaces accordingly:

$ kubectl label ns istioinaction-canary istio.io/rev=prod-canary

If you list out the tags in your mesh, you will see two stable tags pointing to two different revisions:

$ istioctl x revision tag list

TAG         REVISION NAMESPACES
prod-stable 1-9-5    istioinaction
prod-canary 1-10-0   istioinaction-canary

Any of the namespaces that you have labeled with istio.io/rev=prod-canary will be injected by the control plane that corresponds to the prod-canary stable tag name (which in this example points to the 1-10-0 revision). When you’re ready, you can switch the prod-stable tag to the new control plane with:

$ istioctl x revision tag set prod-stable --revision 1-10-0 --overwrite

Any time you switch a tag to point to a new revision, you will need to restart the workloads in any respective namespace to pick up the new revision’s proxy.

When both the prod-stable and prod-canary no longer point to the old revision, it may be safe to remove the old revision as follows:

$ istioctl x uninstall --revision 1-9-5

Wrapping up

Using revisions makes it safer to canary changes to an Istio control plane. In large environments with lots of namespaces, you may prefer to use stable tags, as we’ve introduced in this blog, to remove the number of moving pieces and simplify any automation you may build around updating an Istio control plane. Please check out the 1.10 release and the new tag feature and give us your feedback!

Four years ago today, the Istio project was born to the open source world. To celebrate this anniversary, we are hosting a week-long birthday celebration that focuses on contributions to the Istio project that stem from using Istio in production. Read on to learn how to participate in this celebration and enter a chance to win some Istio swag.

A year of important developments for Istio

Over the last 12 months, the Istio project has been very focused on the day-0 & day-1 experience for users by actively listening to our users through UX surveys and GitHub issues.

• We simplified the control plane architecture and made Istio easier to install, configure and upgrade.
• We provided clarity and process to our feature status and promotion of features and APIs.
• We simplified the debugging experience with various istioctl commands.
• We expanded the mesh to services running in VMs and multiple clusters.
• We made StatefulSet easier to use in Istio 1.10 with zero-configuration.
• We made various performance improvements to the Istio control plane and data plane via discovery selectors, sidecar resources etc.
• We introduced WebAssembly as our extensibility platform which has helped users tailor Istio to their needs.
• We beefed up our CVE management and release processes to meet enterprise needs.

Read more about improvements to Istio in 2020 that made this technology easier to use.

In February 2021, we celebrated the first IstioCon! This community-led event was an opportunity for users and developers to share many examples of how they use Istio in production and lessons learned from it. IstioCon was a great opportunity to feature more than 25 end-user companies, like Salesforce, T-Mobile, and Airbnb, among others, to feature maintainers from across the Istio ecosystem, and to share the Istio project roadmap.

This inaugural community conference was a major success, with more than 4,000 registrants from 80 countries participating. The program was conducted during US and Asia time zones, and in English and Chinese languages to accommodate big user communities in various continents. Learn more about the impact of IstioCon and find the presentations on the conference website.

How to participate in Istio’s 4th Birthday celebration

Contributions are key to the long life of an open source project. This is why, on its 4th birthday, we want to hear about your contributions to the Istio project. To participate in this campaign, share on Twitter a contribution you made to the project and why it matters, using the hashtag #IstioTurns4 and #IstioBirthday. You can submit posts from Monday, May 24th at 9 am Pacific, until Friday, May 28th, at 12 pm Pacific, to enter a chance to win some Istio swag.

The other way of participating in this campaign is by joining the Istio community meetup, which will take place on Thursday, May 27th at 10 am Pacific. At this event, we will have Pratima Nambiar discuss contributions that have stemmed from using Istio in production at Salesforce. Join the event, and ask a question or make a comment on the demo, and enter a chance to win some Istio swag.

With the release of Istio 1.10, we are announcing Alpha level support for upgrading directly from Istio 1.8.x to 1.10.x, without upgrading to 1.9.x. We hope this will reduce the operational burden of running Istio, in keeping with our 2021 theme of improving Day 2 Operations.

Upgrade From 1.8 to 1.10

For direct upgrades we recommend using the canary upgrade method so that control plane functionality can be verified before cutting workloads over to the new version. We’ll also be using revision tags in this guide, an improvement to canary upgrades that was introduced in 1.10, so users don’t have to change the labels on a namespace while upgrading.

First, using a version 1.10 or newer istioctl, create a revision tag stable pointed to your existing 1.8 revision. From now on let’s assume this revision is called 1-8-5:

$ istioctl x revision tag set stable --revision 1-8-5

If your 1.8 installation did not have an associated revision, we can create this revision tag with:

$ istioctl x revision tag set stable --revision default

Now, relabel your namespaces that were previously labeled with istio-injection=enabled or istio.io/rev=<REVISION> with istio.io/rev=stable. Download the Istio 1.10.0 release and install the new control plane with a revision:

$ istioctl install --revision 1-10-0 -y

Now evaluate that the 1.10 revision has come up correctly and is healthy. Once satisfied with the stability of new revision you can set the revision tag to the new revision:

$ istioctl x revision tag set stable --revision 1-10-0 --overwrite

Verify that the revision tag stable is pointing to the new revision:

$ istioctl x revision tag list
TAG    REVISION NAMESPACES
stable 1-10-0        ...

Once prepared to move existing workloads over to the new 1.10 revision, the workloads must be restarted so that the sidecar proxies will use the new control plane. We can go through namespaces one by one and roll the workloads over to the new version:

$ kubectl rollout restart deployments -n …

Notice an issue after rolling out workloads to the new Istio version? No problem! Since you’re using canary upgrades, the old control plane is still running and we can just switch back over.

$ istioctl x revision tag set prod --revision 1-8-5

Then after triggering another rollout, your workloads will be back on the old version.

We look forward to hearing about your experience with direct upgrades, and look forward to improving and expanding this functionality in the future.

The Istio community has been making gradual progress towards zero-configuration support for StatefulSets; from automatic mTLS, to eliminating the need to create DestinationRule or ServiceEntry resources, to the most recent pod networking changes in Istio 1.10.

What is unique about using a StatefulSet with a service mesh? The StatefulSet pods are created from the same spec, but are not interchangeable: each has a persistent identifier that it maintains across any rescheduling. The kind of apps that run in a StatefulSet are often those that need to communicate among their pods, and, as they come from a world of hard-coded IP addresses, may listen on the pod IP only, instead of 0.0.0.0.

ZooKeeper, for example, is configured by default to not listen on all IPs for quorum communication:

quorumListenOnAllIPs=false

Over the last few releases, the Istio community has reported many issues around support for applications running in StatefulSets.

StatefulSets in action, prior to Istio 1.10

In a GKE cluster running Kubernetes 1.19, we have Istio 1.9.5 installed. We enabled automatic sidecar injection in the default namespace, then we installed ZooKeeper using the Helm charts provided by Bitnami, along with the Istio sleep pod for interactive debugging:

$ helm repo add bitnami https://charts.bitnami.com/bitnami
$ helm install my-release bitnami/zookeeper --set replicaCount=3
$ kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.29/samples/sleep/sleep.yaml

After a few minutes, all pods come up nicely with sidecar proxies:

$ kubectl get pods,svc
NAME                             READY   STATUS    RESTARTS   AGE
my-release-zookeeper-0           2/2     Running   0          3h4m
my-release-zookeeper-1           2/2     Running   0          3h4m
my-release-zookeeper-2           2/2     Running   0          3h5m
pod/sleep-8f795f47d-qkgh4        2/2     Running   0          3h8m

NAME                            TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)                            AGE
my-release-zookeeper            ClusterIP   10.100.1.113   <none>        2181/TCP,2888/TCP,3888/TCP         3h
my-release-zookeeper-headless   ClusterIP   None           <none>        2181/TCP,2888/TCP,3888/TCP         3h
service/sleep                   ClusterIP   10.100.9.26    <none>        80/TCP                             3h

Are our ZooKeeper services working and is the status Running? Let’s find out! ZooKeeper listens on 3 ports:

• Port 2181 is the TCP port for clients to connect to the ZooKeeper service
• Port 2888 is the TCP port for peers to connect to other peers
• Port 3888 is the dedicated TCP port for leader election

By default, the ZooKeeper installation configures port 2181 to listen on 0.0.0.0 but ports 2888 and 3888 only listen on the pod IP. Let’s check out the network status on each of these ports from one of the ZooKeeper pods:

$ kubectl exec my-release-zookeeper-1 -c istio-proxy -- netstat -na | grep -E '(2181|2888|3888)'
tcp        0      0 0.0.0.0:2181            0.0.0.0:*               LISTEN
tcp        0      0 10.96.7.7:3888          0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:2181          127.0.0.1:37412         TIME_WAIT
tcp        0      0 127.0.0.1:2181          127.0.0.1:37486         TIME_WAIT
tcp        0      0 127.0.0.1:2181          127.0.0.1:37456         TIME_WAIT
tcp        0      0 127.0.0.1:2181          127.0.0.1:37498         TIME_WAIT
tcp        0      0 127.0.0.1:2181          127.0.0.1:37384         TIME_WAIT
tcp        0      0 127.0.0.1:2181          127.0.0.1:37514         TIME_WAIT
tcp        0      0 127.0.0.1:2181          127.0.0.1:37402         TIME_WAIT
tcp        0      0 127.0.0.1:2181          127.0.0.1:37434         TIME_WAIT
tcp        0      0 127.0.0.1:2181          127.0.0.1:37526         TIME_WAIT
tcp        0      0 127.0.0.1:2181          127.0.0.1:37374         TIME_WAIT
tcp        0      0 127.0.0.1:2181          127.0.0.1:37442         TIME_WAIT
tcp        0      0 127.0.0.1:2181          127.0.0.1:37464         TIME_WAIT

There is nothing ESTABLISHED on port 2888 or 3888. Next, let us get the ZooKeeper server status:

$ kubectl exec my-release-zookeeper-1 -c zookeeper -- /opt/bitnami/zookeeper/bin/zkServer.sh status
/opt/bitnami/java/bin/java
ZooKeeper JMX enabled by default
Using config: /opt/bitnami/zookeeper/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost. Client SSL: false.
Error contacting service. It is probably not running.

From the above output, you can see the ZooKeeper service is not functioning properly. Let us check the cluster configuration for one of the ZooKeeper pods:

$ istioctl proxy-config cluster my-release-zookeeper-1 --port 3888 --direction inbound -o json
[
    {
        "name": "inbound|3888||",
        "type": "STATIC",
        "connectTimeout": "10s",
        "loadAssignment": {
            "clusterName": "inbound|3888||",
            "endpoints": [
                {
                    "lbEndpoints": [
                        {
                            "endpoint": {
                                "address": {
                                    "socketAddress": {
                                        "address": "127.0.0.1",
                                        "portValue": 3888
                                    }
                                }
                            }
                        }
                    ]
                }
            ]
        },
...

What is interesting here is that the inbound on port 3888 has 127.0.0.1 as its endpoint. This is because the Envoy proxy, in versions of Istio prior to 1.10, redirects the inbound traffic to the loopback interface, as described in our blog post about the change.

StatefulSets in action with Istio 1.10

Now, we have upgraded our cluster to Istio 1.10 and configured the default namespace to enable 1.10 sidecar injection. Let’s rolling restart the ZooKeeper StatefulSet to update the pods to use the new version of the sidecar proxy:

$ kubectl rollout restart statefulset my-release-zookeeper

Once the ZooKeeper pods reach the running status, let’s check out the network connections for these 3 ports from any of the ZooKeeper pods:

$ kubectl exec my-release-zookeeper-1 -c istio-proxy -- netstat -na | grep -E '(2181|2888|3888)'
tcp        0      0 0.0.0.0:2181            0.0.0.0:*               LISTEN
tcp        0      0 10.96.8.10:2888         0.0.0.0:*               LISTEN
tcp        0      0 10.96.8.10:3888         0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.6:42571         10.96.8.10:2888         ESTABLISHED
tcp        0      0 10.96.8.10:2888         127.0.0.6:42571         ESTABLISHED
tcp        0      0 127.0.0.6:42655         10.96.8.10:2888         ESTABLISHED
tcp        0      0 10.96.8.10:2888         127.0.0.6:42655         ESTABLISHED
tcp        0      0 10.96.8.10:37876        10.96.6.11:3888         ESTABLISHED
tcp        0      0 10.96.8.10:44872        10.96.7.10:3888         ESTABLISHED
tcp        0      0 10.96.8.10:37878        10.96.6.11:3888         ESTABLISHED
tcp        0      0 10.96.8.10:44870        10.96.7.10:3888         ESTABLISHED
tcp        0      0 127.0.0.1:2181          127.0.0.1:54508         TIME_WAIT
tcp        0      0 127.0.0.1:2181          127.0.0.1:54616         TIME_WAIT
tcp        0      0 127.0.0.1:2181          127.0.0.1:54664         TIME_WAIT
tcp        0      0 127.0.0.1:2181          127.0.0.1:54526         TIME_WAIT
tcp        0      0 127.0.0.1:2181          127.0.0.1:54532         TIME_WAIT
tcp        0      0 127.0.0.1:2181          127.0.0.1:54578         TIME_WAIT
tcp        0      0 127.0.0.1:2181          127.0.0.1:54634         TIME_WAIT
tcp        0      0 127.0.0.1:2181          127.0.0.1:54588         TIME_WAIT
tcp        0      0 127.0.0.1:2181          127.0.0.1:54610         TIME_WAIT
tcp        0      0 127.0.0.1:2181          127.0.0.1:54550         TIME_WAIT
tcp        0      0 127.0.0.1:2181          127.0.0.1:54560         TIME_WAIT
tcp        0      0 127.0.0.1:2181          127.0.0.1:54644         TIME_WAIT

There are ESTABLISHED connections on both port 2888 and 3888! Next, let us check out the ZooKeeper server status:

$ kubectl exec my-release-zookeeper-1 -c zookeeper -- /opt/bitnami/zookeeper/bin/zkServer.sh status
/opt/bitnami/java/bin/java
ZooKeeper JMX enabled by default
Using config: /opt/bitnami/zookeeper/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost. Client SSL: false.
Mode: follower

The ZooKeeper service is now running!

We can connect to each of the ZooKeeper pods from the sleep pod and run the below command to discover the server status of each pod within the StatefulSet. Note that there is no need to create ServiceEntry resources for any of the ZooKeeper pods and we can call these pods directly using their DNS names (e.g. my-release-zookeeper-0.my-release-zookeeper-headless) from the sleep pod.

$ kubectl exec -it deploy/sleep -c sleep -- sh  -c 'for x in my-release-zookeeper-0.my-release-zookeeper-headless my-release-zookeeper-1.my-release-zookeeper-headless my-release-zookeeper-2.my-release-zookeeper-headless; do echo $x; echo srvr|nc $x 2181; echo; done'
my-release-zookeeper-0.my-release-zookeeper-headless
Zookeeper version: 3.7.0-e3704b390a6697bfdf4b0bef79e3da7a4f6bac4b, built on 2021-03-17 09:46 UTC
Latency min/avg/max: 1/7.5/20
Received: 3845
Sent: 3844
Connections: 1
Outstanding: 0
Zxid: 0x200000002
Mode: follower
Node count: 6

my-release-zookeeper-1.my-release-zookeeper-headless
Zookeeper version: 3.7.0-e3704b390a6697bfdf4b0bef79e3da7a4f6bac4b, built on 2021-03-17 09:46 UTC
Latency min/avg/max: 0/0.0/0
Received: 3856
Sent: 3855
Connections: 1
Outstanding: 0
Zxid: 0x200000002
Mode: follower
Node count: 6

my-release-zookeeper-2.my-release-zookeeper-headless
Zookeeper version: 3.7.0-e3704b390a6697bfdf4b0bef79e3da7a4f6bac4b, built on 2021-03-17 09:46 UTC
Latency min/avg/max: 0/0.0/0
Received: 3855
Sent: 3854
Connections: 1
Outstanding: 0
Zxid: 0x200000002
Mode: leader
Node count: 6
Proposal sizes last/min/max: 48/48/48

Now our ZooKeeper service is running, let’s use Istio to secure all communication to our regular and headless services. Apply mutual TLS to the default namespace:

$ kubectl apply -n default -f - <<EOF
apiVersion: "security.istio.io/v1beta1"
kind: "PeerAuthentication"
metadata:
  name: "default"
spec:
  mtls:
    mode: STRICT
EOF

Continue sending some traffic from the sleep pod and bring up the Kiali dashboard to visualize the services in the default namespace:

The padlock icons on the traffic flows indicate that the connections are secure.

Wrapping up

With the new networking changes in Istio 1.10, a Kubernetes pod with a sidecar has the same networking behavior as a pod without a sidecar. This change enables stateful applications to function properly in Istio as we have shown you in this post. We believe this is a huge step towards Istio’s goal of providing transparent service mesh and zero-configuration Istio.

We currently disclose vulnerabilities and security releases via numerous channels:

• istio.io via our Release Announcements and Security Bulletins
• Discuss
• announcements channel on Slack
• Twitter
• RSS

When operating any software, it is preferable to plan for possible downtime when upgrading. Given the work that the Istio community is doing around Day 2 operations in 2021, the Environments working group has done a good job to streamline many upgrade issues users have seen. The Product Security Working Group intends to help Day 2 operations by having routine security release days so that upgrade operations can be planned in advance for our users.

Patch Tuesdays

The Product Security working group is intending to ship a security release the 2nd Tuesday of each month. These security releases may contain fixes for multiple CVEs. It is the intent of the Product Security working group to have these security releases not contain any other fixes, although that may not always be possible.

When the Product Security working group intends to ship an upcoming security patch, an announcement will be made on the Istio discussion board 2 weeks prior to release. If you’re running Istio in production, we suggest you watch the Announcements category to be notified of such a release. If no such announcement is made there will not be a security release for that month, barring some exceptions listed below.

First Patch Tuesday

We are pleased to announce that Istio 1.9.5, and the final release of Istio 1.8, 1.8.6, are the first security releases to fit this pattern. As Istio 1.10 will be shipping soon we are intending to continue this new tradition in June.

These releases fix 3 CVEs. Please see the release pages for information regarding the specific CVEs fixed.

Unscheduled security releases

0-day vulnerabilities

Unfortunately, 0-day vulnerabilities cannot be planned. Upon disclosure, the Product Security Working Group will need to issue an out-of-band security release. The above methods will be used to disclose such issues, so please use at least one of them to be notified of such disclosures.

Third party embargoes

Similar to 0-day vulnerabilities, security releases can be dictated by third party embargoes, namely Envoy. When this occurs, Istio will release a same-day patch once the embargo is lifted.

Security Best Practices

The Istio Security Best Practices has seen many improvements over the past few months. We recommend you check it regularly, as many of our recent security bulletins can be mitigated by utilizing methods discussed in the Security Best Practices page.

Early Disclosure List

If you meet the criteria to be a part of the Istio Early Disclosure list, please apply for membership. Patches for upcoming security releases will be made available to the early disclosure list ~2 weeks prior to Istio’s Patch Tuesday.

There will be times when an upcoming Istio security release will also need patches from Envoy. We cannot redistribute Envoy patches due to their embargo. Please refer to Envoy’s guidance on how to join their early disclosure list.

Security Feedback

The Product Security Working Group holds bi-weekly meetings on Tuesdays from 9:00-9:30 Pacific. For more information see the Istio Working Group Calendar.

Our next public meeting will be held on May 25, 2021. Please join us!

Both in the community as well as for our large-scale customers at Solo.io, we need a way to dynamically restrict the set of namespaces that are part of the mesh so that the Istio control plane only processes resources in those namespaces. The ability to restrict the namespaces enables Istiod to watch and push fewer resources and associated changes to the sidecars, thus improving the overall performance on the control plane and data plane.

Background

By default, Istio watches all Namespaces, Services, Endpoints and Pods in a cluster. For example, in my Kubernetes cluster, I deployed the sleep service in the default namespace, and the httpbin service in the ns-x namespace. I’ve added the sleep service to the mesh, but I have no plan to add the httpbin service to the mesh, or have any service in the mesh interact with the httpbin service.

Use istioctl proxy-config endpoint command to display all the endpoints for the sleep deployment:

Note that the httpbin service endpoint in the ns-x namespace is in the list of discovered endpoints. This may not be an issue when you only have a few services. However, when you have hundreds of services that don’t interact with any of the services running in the Istio service mesh, you probably don’t want your Istio control plane to watch these services and send their information to the sidecars of your services in the mesh.

Introducing Discovery Selectors

Starting with Istio 1.10, we are introducing the new discoverySelectors option to MeshConfig, which is an array of Kubernetes selectors. The exact type is []LabelSelector, as defined here, allowing both simple selectors and set-based selectors. These selectors apply to labels on namespaces.

You can configure each label selector for expressing a variety of use cases, including but not limited to:

• Arbitrary label names/values, for example, all namespaces with label istio-discovery=enabled
• A list of namespace labels using set-based selectors which carries OR semantics, for example, all namespaces with label istio-discovery=enabled OR region=us-east1
• Inclusion and/or exclusion of namespaces, for example, all namespaces with label istio-discovery=enabled AND label key app equal to helloworld

Note: discoverySelectors is not a security boundary. Istiod will continue to have access to all namespaces even when you have configured your discoverySelectors.

Discovery Selectors in Action

Assuming you know which namespaces to include as part of the service mesh, as a mesh administrator, you can configure discoverySelectors at installation time or post-installation by adding your desired discovery selectors to Istio’s MeshConfig resource. For example, you can configure Istio to discover only the namespaces that have the label istio-discovery=enabled.

1. Using our examples earlier, let’s label the default namespace with label istio-discovery=enabled.

   $ kubectl label namespace default istio-discovery=enabled

2. Use istioctl to apply the yaml with discoverySelectors to update your Istio installation. Note, to avoid any impact to your stable environment, we recommend that you use a different revision for your Istio installation:

   $ istioctl install --skip-confirmation -f - <<EOF
   apiVersion: install.istio.io/v1alpha1
   kind: IstioOperator
   metadata:
   namespace: istio-system
   spec:
   # You may override parts of meshconfig by uncommenting the following lines.
     meshConfig:
       discoverySelectors:
         - matchLabels:
             istio-discovery: enabled
   EOF

3. Display the endpoint configuration for the sleep deployment:

   

   Note this time the httpbin service in the ns-x namespace is NOT in the list of discovered endpoints, along with many other services that are not in the default namespace. If you display routes (or cluster or listeners) information for the sleep deployment, you will also notice much less configuration is returned:

   

You can use matchLabels to configure multiple labels with AND semantics or use matchLabels sets to configure OR semantics among multiple labels. Whether you deploy services or pods to namespaces with different sets of labels or multiple application teams in your organization use different labeling conventions, discoverySelectors provides the flexibility you need. Furthermore, you could use matchLabels and matchExpressions together per our documentation. Refer to the Kubernetes selector docs for additional detail on selector semantics.

Discovery Selectors vs Sidecar Resource

The discoverySelectors configuration enables users to dynamically restrict the set of namespaces that are part of the mesh. A Sidecar resource also controls the visibility of sidecar configurations and what gets pushed to the sidecar proxy. What are the differences between them?

• The discoverySelectors configuration declares what Istio control plane watches and processes. Without discoverySelectors configuration, the Istio control plane watches and processes all namespaces/services/endpoints/pods in the cluster regardless of the sidecar resources you have.
• discoverySelectors is configured globally for the mesh by the mesh administrators. While Sidecar resources can also be configured for the mesh globally by the mesh administrators in the MeshConfig root namespace, they are commonly configured by service owners for their namespaces.

You can use discoverySelectors with Sidecar resources. You can use discoverySelectors to configure at the mesh-wide level what namespaces the Istio control plane should watch and process. For these namespaces in the Istio service mesh, you can create Sidecar resources globally or per namespace to further control what gets pushed to the sidecar proxies. Let us add Bookinfo services to the ns-y namespace in the mesh as shown in the diagram below. discoverySelectors enables us to define the default and ns-y namespaces are part of the mesh. How can we configure the sleep service not to see anything other than the default namespace? Adding a Sidecar resource for the default namespace, we can effectively configure the sleep sidecar to only have visibility to the clusters/routes/listeners/endpoints associated with its current namespace plus any other required namespaces.

Wrapping up

Discovery selectors are powerful configurations to tune the Istio control plane to only watch and process specific namespaces. If you don’t want all namespaces in your Kubernetes cluster to be part of the service mesh or you have multiple Istio service meshes within your Kubernetes cluster, we highly recommend that you explore this configuration and reach out to us for feedback on our Istio slack or GitHub.

While Kubernetes networking is customizable, a typical pod’s network will look like this:

An application may choose to bind to either the loopback interface lo (typically binding to 127.0.0.1), or the pods network interface eth0 (typically to the pod’s IP), or both (typically binding to 0.0.0.0).

Binding to lo allows calls such as curl localhost to work from within the pod. Binding to eth0 allows calls to the pod from other pods.

Typically, an application will bind to both. However, applications which have internal logic, such as an admin interface may choose to bind to only lo to avoid access from other pods. Additionally, some applications, typically stateful applications, choose to bind only to eth0.

Current behavior

In Istio prior to release 1.10, the Envoy proxy, running in the same pod as the application, binds to the eth0 interface and redirects all inbound traffic to the lo interface.

This has two important side effects that cause the behavior to differ from standard Kubernetes:

• Applications binding only to lo will receive traffic from other pods, when otherwise this is not allowed.
• Applications binding only to eth0 will not receive traffic.

Applications that bind to both interfaces (which is typical) will not be impacted.

Future behavior

Starting with Istio 1.10, the networking behavior is changed to align with the standard behavior present in Kubernetes.

Here we can see that the proxy no longer redirects the traffic to the lo interface, but instead forwards it to the application on eth0. As a result, the standard behavior of Kubernetes is retained, but we still get all the benefits of Istio. This change allows Istio to get closer to its goal of being a drop-in transparent proxy that works with existing workloads with zero configuration. Additionally, it avoids unintended exposure of applications binding only to lo.

Am I impacted?

For new users, this change should only be an improvement. However, if you are an existing user, you may have come to depend on the old behavior, intentionally or accidentally.

To help detect these situations, we have added a check to find pods that will be impacted. You can run the istioctl experimental precheck command to get a report of any pods binding to lo on a port exposed in a Service. This command is available in Istio 1.10+. Without action, these ports will no longer be accessible upon upgrade.

$ istioctl experimental precheck
Error [IST0143] (Pod echo-local-849647c5bd-g9wxf.default) Port 443 is exposed in a Service but listens on localhost. It will not be exposed to other pods.
Error [IST0143] (Pod echo-local-849647c5bd-g9wxf.default) Port 7070 is exposed in a Service but listens on localhost. It will not be exposed to other pods.
Error: Issues found when checking the cluster. Istio may not be safe to install or upgrade.
See https://istio.io/latest/docs/reference/config/analysis for more information about causes and resolutions.

Migration

If you are currently binding to lo, you have a few options:

• Switch your application to bind to all interfaces (0.0.0.0 or ::).

• Explicitly configure the port using the Sidecar ingress configuration to send to lo, preserving the old behavior.

  For example, to configure request to be sent to localhost for the ratings application:

  apiVersion: networking.istio.io/v1beta1
  kind: Sidecar
  metadata:
    name: ratings
  spec:
    workloadSelector:
      labels:
        app: ratings
    ingress:
    - port:
        number: 8080
        protocol: HTTP
        name: http
      defaultEndpoint: 127.0.0.1:8080

• Disable the change entirely with the PILOT_ENABLE_INBOUND_PASSTHROUGH=false environment variable in Istiod, to enable the same behavior as prior to Istio 1.10. This option will be removed in the future.

WebAssembly support merged in upstream Envoy

After adding experimental support for Wasm and the WebAssembly for Proxies (Proxy-Wasm) ABI to Istio’s fork of Envoy, we collected some great feedback from our community of early adopters. This, combined with the experience gained from developing core Istio Wasm extensions, helped us mature and stabilize the runtime. These improvements unblocked merging Wasm support directly into Envoy upstream in October 2020, allowing it to become part of all official Envoy releases. This was a significant milestone, since it indicates that:

• The runtime is ready for wider adoption.
• The programming ABI/API, extension configuration API, and runtime behavior, are becoming stable.
• You can expect a larger community of adoption and support moving forward.

wasm-extensions Ecosystem Repository

As an early adopter of the Envoy Wasm runtime, the Istio Extensions and Telemetry working group gained a lot of experience in developing extensions. We built several first-class extensions, including metadata exchange, Prometheus stats, and attribute generation. In order to share our learning more broadly, we created a wasm-extensions repository in the istio-ecosystem organization. This repository serves two purposes:

• It provides canonical example extensions, covering several highly demanded features (such as basic authentication).
• It provides a guide for Wasm extension development, testing, and release. The guide is based on the same build tool chains and test frameworks that are used, maintained and tested by the Istio extensibility team.

The guide currently covers WebAssembly extension development and unit testing with C++, as well as integration testing with a Go test framework, which simulates a real runtime by running a Wasm module with the Istio proxy binary. In the future, we will also add several more canonical extensions, such as an integration with Open Policy Agent, and header manipulation based on JWT tokens.

Wasm module distribution via the Istio Agent

Prior to Istio 1.9, Envoy remote data sources were needed to distribute remote Wasm modules to the proxy. In this example, you can see two EnvoyFilter resources are defined: one to add a remote fetch Envoy cluster, and the other one to inject a Wasm filter into the HTTP filter chain. This method has a drawback: if remote fetch fails, either due to bad configuration or transient error, Envoy will be stuck with the bad configuration. If a Wasm extension is configured as fail closed, a bad remote fetch will stop Envoy from serving. To fix this issue, a fundamental change is needed to the Envoy xDS protocol to make it allow asynchronous xDS responses.

Istio 1.9 provides a reliable distribution mechanism out of the box by leveraging the xDS proxy inside istio-agent and Envoy’s Extension Configuration Discovery Service (ECDS).

istio-agent intercepts the extension config resource update from istiod, reads the remote fetch hint from it, downloads the Wasm module, and rewrites the ECDS configuration with the path of the downloaded Wasm module. If the download fails, istio-agent will reject the ECDS update and prevent a bad configuration reaching Envoy. For more detail, please see our docs on Wasm module distribution.

Istio Wasm SIG and Future Work

Although we have made a lot of progress on Wasm extensibility, there are still many aspects of the project that remain to be completed. In order to consolidate the efforts from various parties and better tackle the challenges ahead, we have formed an Istio WebAssembly SIG, with aim of providing a standard and reliable way for Istio to consume Wasm extensions. Here are some of the things we are working on:

• A first-class extension API: Currently Wasm extensions needs to be injected via Istio’s EnvoyFilter API. A first-class extension API will make using Wasm with Istio easier, and we expect this to be introduced in Istio 1.10.
• Distribution artifacts interoperability: Built on top of Solo.io’s WebAssembly OCI image spec effort, a standard Wasm artifacts format will make it easy to build, pull, publish, and execute.
• Container Storage Interface (CSI) based artifacts distribution: Using istio-agent to distribute modules is easy for adoption, but may not be efficient as each proxy will keep a copy of the Wasm module. As a more efficient solution, with Ephemeral CSI, a DaemonSet will be provided which could configure storage for pods. Working similarly to a CNI plugin, a CSI driver would fetch the Wasm module out-of-band from the xDS flow and mount it inside the rootfs when the pod starts up.

If you would like to join us, the group will meet every other week Tuesdays at 2PM PT. You can find the meeting on the Istio working group calendar.

We look forward to seeing how you will use Wasm to extend Istio!

The old APIs were deprecated in Istio 1.4. Two releases after the v1beta1 APIs were introduced, Istio 1.6 removed support for the v1alpha1 APIs.

If you are using a version of Istio prior to 1.6 and you want to upgrade, you will have to migrate your alpha security policy objects to the beta API. This tutorial will help you make that move.

Overview

Your control plane must first be upgraded to a version that supports the v1beta1 security policy.

It is recommended to first upgrade to Istio 1.5 as a transitive version, because it is the only version that supports both v1alpha1 and v1beta1 security policies. You will complete the security policy migration in Istio 1.5, remove the v1alpha1 security policy, and then continue to upgrade to later Istio versions. For a given workload, the v1beta1 version will take precedence over the v1alpha1 version.

Alternatively, if you want to do a skip-level upgrade directly from Istio 1.4 to 1.6 or later, you should use the canary upgrade method to install a new Istio version as a separate control plane, and gradually migrate your workloads to the new control plane completing the security policy migration at the same time.

In either case, it is recommended to migrate using namespace granularity: for each namespace, find all the v1alpha1 policies that have an effect on workloads in the namespace and migrate all the policies to v1beta1 at the same time. This allows a safer migration as you can make sure everything is working as expected, and then move forward to the next namespace.

Major differences

Before starting the migration, read through the v1beta1 authentication and authorization documentation to understand the v1beta1 policy.

You should examine all of your existing v1alpha1 security policies, find out what fields are used and which policies need migration, compare the findings with the major differences listed below and confirm there are no blocking issues (e.g., using an alpha feature that is no longer supported in beta):

Although RequestAuthentication in v1beta1 security policy is similar to the v1alpha1 JWT policy, there is a notable semantics change. The v1alpha1 JWT policy needs to be migrated to two v1beta1 resources: RequestAuthentication and AuthorizationPolicy. This will change the JWT deny message due to the use of AuthorizationPolicy. In the alpha version, the HTTP code 401 is returned with the body Origin authentication failed. In the beta version, the HTTP code 403 is returned with the body RBAC: access denied.

The v1alpha1 JWT policy triggerRule field is replaced by the AuthorizationPolicy with the exception that the regex field is no longer supported.

Migration flow

This section describes in detail how to migrate a v1alpha1 security policy.

Step 1: Find related policies

For each namespace, find all v1alpha1 security policies that have an effect on workloads in the namespace. The result could include:

• a single MeshPolicy that applies to all services in the mesh;
• a single namespace-level Policy that applies to all workloads in the namespace;
• multiple service-level Policy objects that apply to the selected services in the namespace;
• a single ClusterRbacConfig that enables the RBAC on the whole namespace or some services in the namespace;
• multiple namespace-level ServiceRole and ServiceRoleBinding objects that apply to all services in the namespace;
• multiple service-level ServiceRole and ServiceRoleBinding objects that apply to the selected services in the namespace;

Step 2: Convert service name to workload selector

The v1alpha1 policy selects targets using their service name. You should refer to the corresponding service definition to decide the workload selector that should be used in the v1beta1 policy.

A single v1alpha1 policy may include multiple services. It will need to be migrated to multiple v1beta1 policies because the v1beta1 policy currently only supports at most one workload selector per policy.

Also note the v1alpha1 policy uses service port but the v1beta1 policy uses the workload port. This means the port number might be different in the migrated v1beta1 policy.

Step 3: Migrate authentication policy

For each v1alpha1 authentication policy, migrate with the following rules:

1. If the whole namespace is enabled with mTLS or JWT, create the PeerAuthentication, RequestAuthentication and AuthorizationPolicy without a workload selector for the whole namespace. Fill out the policy based on the semantics of the corresponding MeshPolicy or Policy for the namespace.

2. If a workload is enabled with mTLS or JWT, create the PeerAuthentication, RequestAuthentication and AuthorizationPolicy with a corresponding workload selector for the workload. Fill out the policy based on the semantics of the corresponding MeshPolicy or Policy for the workload.

3. For mTLS related configuration, use STRICT mode if the alpha policy is using STRICT, or use PERMISSIVE in all other cases.

4. For JWT related configuration, refer to the end-user authentication documentation to learn how to migrate to RequestAuthentication and AuthorizationPolicy.

A security policy migration tool is provided to automatically migrate authentication policy automatically. Please refer to the tool’s README for its usage.

Step 4: Migrate RBAC policy

For each v1alpha1 RBAC policy, migrate with the following rules:

1. If the whole namespace is enabled with RBAC, create an AuthorizationPolicy without a workload selector for the whole namespace. Add an empty rule so that it will deny all requests to the namespace by default.

2. If a workload is enabled with RBAC, create an AuthorizationPolicy with a corresponding workload selector for the workload. Add rules based on the semantics of the corresponding ServiceRole and ServiceRoleBinding for the workload.

Step 5: Verify migrated policy

1. Double check the migrated v1beta1 policies: make sure there are no policies with duplicate names, the namespace is specified correctly and all v1alpha1 policies for the given namespace are migrated.

2. Dry-run the v1beta1 policy with the command kubectl apply --dry-run=server -f beta-policy.yaml to make sure it is valid.

3. Apply the v1beta1 policy to the given namespace and closely monitor the effect. Make sure to test both allow and deny scenarios if JWT or authorization are used.

4. Migrate the next namespace. Only remove the v1alpha1 policy after completing migration for all namespaces successfully.

Example

v1alpha1 policy

This section gives a full example showing the migration for namespace foo. Assume the namespace foo has the following v1alpha1 policies that affect the workloads in it:

# A MeshPolicy that enables mTLS globally, including the whole foo namespace
apiVersion: "authentication.istio.io/v1alpha1"
kind: "MeshPolicy"
metadata:
  name: "default"
spec:
  peers:
  - mtls: {}
---
# A Policy that enables mTLS permissive mode and enables JWT for the httpbin service on port 8000
apiVersion: authentication.istio.io/v1alpha1
kind: Policy
metadata:
  name: httpbin
  namespace: foo
spec:
  targets:
  - name: httpbin
    ports:
    - number: 8000
  peers:
  - mtls:
      mode: PERMISSIVE
  origins:
  - jwt:
      issuer: testing@example.com
      jwksUri: https://www.example.com/jwks.json
      triggerRules:
      - includedPaths:
        - prefix: /admin/
        excludedPaths:
        - exact: /admin/status
  principalBinding: USE_ORIGIN
---
# A ClusterRbacConfig that enables RBAC globally, including the foo namespace
apiVersion: "rbac.istio.io/v1alpha1"
kind: ClusterRbacConfig
metadata:
  name: default
spec:
  mode: 'ON'
---
# A ServiceRole that enables RBAC for the httpbin service
apiVersion: "rbac.istio.io/v1alpha1"
kind: ServiceRole
metadata:
  name: httpbin
  namespace: foo
spec:
  rules:
  - services: ["httpbin.foo.svc.cluster.local"]
    methods: ["GET"]
---
# A ServiceRoleBinding for the above ServiceRole
apiVersion: "rbac.istio.io/v1alpha1"
kind: ServiceRoleBinding
metadata:
  name: httpbin
  namespace: foo
spec:
  subjects:
  - user: cluster.local/ns/foo/sa/sleep
    roleRef:
      kind: ServiceRole
      name: httpbin

httpbin service

The httpbin service has the following definition:

apiVersion: v1
kind: Service
metadata:
  name: httpbin
  namespace: foo
spec:
  ports:
  - name: http
    port: 8000
    targetPort: 80
  selector:
    app: httpbin

This means the service name httpbin should be replaced by the workload selector app: httpbin, and the service port 8000 should be replaced by the workload port 80.

v1beta1 authentication policy

The migrated v1beta1 policies for the v1alpha1 authentication policies in foo namespace are listed below:

# A PeerAuthentication that enables mTLS for the foo namespace, migrated from the MeshPolicy
# Alternatively the MeshPolicy could also be migrated to a PeerAuthentication at mesh level
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: foo
spec:
  mtls:
    mode: STRICT
---
# A PeerAuthentication that enables mTLS for the httpbin workload, migrated from the Policy
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: httpbin
  namespace: foo
spec:
  selector:
    matchLabels:
      app: httpbin
  # port level mtls set for the workload port 80 corresponding to the service port 8000
  portLevelMtls:
    80:
      mode: PERMISSIVE
--
# A RequestAuthentication that enables JWT for the httpbin workload, migrated from the Policy
apiVersion: security.istio.io/v1beta1
kind: RequestAuthentication
metadata:
  name: httpbin
  namespace: foo
spec:
  selector:
    matchLabels:
      app: httpbin
  jwtRules:
  - issuer: testing@example.com
    jwksUri: https://www.example.com/jwks.json
---
# An AuthorizationPolicy that enforces to require JWT validation for the httpbin workload, migrated from the Policy
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: httpbin-jwt
  namespace: foo
spec:
  # Use DENY action to explicitly deny requests without JWT token
  action: DENY
  selector:
    matchLabels:
      app: httpbin
  rules:
  - from:
    - source:
        # This makes sure requests without JWT token will be denied
        notRequestPrincipals: ["*"]
    to:
    - operation:
        # This should be the workload port 80, not the service port 8000
        ports: ["80"]
        # The path and notPath is converted from the trigger rule in the Policy
        paths: ["/admin/*"]
        notPaths: ["/admin/status"]

v1beta1 authorization policy

The migrated v1beta1 policies for the v1alpha1 RBAC policies in foo namespace are listed below:

# An AuthorizationPolicy that denies by default, migrated from the ClusterRbacConfig
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: default
  namespace: foo
spec:
  # An empty rule that allows nothing
  {}
---
# An AuthorizationPolicy that enforces to authorization for the httpbin workload, migrated from the ServiceRole and ServiceRoleBinding
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: httpbin
  namespace: foo
spec:
  selector:
    matchLabels:
      app: httpbin
      version: v1
  action: ALLOW
  rules:
  - from:
    - source:
        principals: ["cluster.local/ns/foo/sa/sleep"]
    to:
    - operation:
        methods: ["GET"]

Finish the upgrade

Congratulations; having reached this point, you should only have v1beta1 policy objects, and you will be able to continue upgrading Istio to 1.6 and beyond.

One great part about Istio, however, is that it aims to bring as much value to users out of the box without any configuration at all. This enables users to get most of the benefits of Istio with minimal efforts. For some users with simple requirements, custom configurations may never be required at all. Others will be able to incrementally add Istio configurations once they are more comfortable and as they need them, such as to add ingress routing, fine-tune networking settings, or lock down security policies.

Getting started

To get started, check out our getting started documentation, where you will learn how to install Istio. If you are already familiar, you can simply run istioctl install.

Next, we will explore all the benefits Istio provides us, without any configuration or changes to application code.

Security

Istio automatically enables mutual TLS for traffic between pods in the mesh. This enables applications to forgo complex TLS configuration and certificate management, and offload all transport layer security to the sidecar.

Once comfortable with automatic TLS, you may choose to allow only mTLS traffic, or configure custom authorization policies for your needs.

Observability

Istio automatically generates detailed telemetry for all service communications within a mesh. This telemetry provides observability of service behavior, empowering operators to troubleshoot, maintain, and optimize their applications – without imposing any additional burdens on service developers. Through Istio, operators gain a thorough understanding of how monitored services are interacting, both with other services and with the Istio components themselves.

All of this functionality is added by Istio without any configuration. Integrations with tools such as Prometheus, Grafana, Jaeger, Zipkin, and Kiali are also available.

For more information about the observability Istio provides, check out the observability overview.

Traffic Management

While Kubernetes provides a lot of networking functionality, such as service discovery and DNS, this is done at Layer 4, which can have unintended inefficiencies. For example, in a simple HTTP application sending traffic to a service with 3 replicas, we can see unbalanced load:

$ curl http://echo/{0..5} -s | grep Hostname
Hostname=echo-cb96f8d94-2ssll
Hostname=echo-cb96f8d94-2ssll
Hostname=echo-cb96f8d94-2ssll
Hostname=echo-cb96f8d94-2ssll
Hostname=echo-cb96f8d94-2ssll
Hostname=echo-cb96f8d94-2ssll
$ curl http://echo/{0..5} -s | grep Hostname
Hostname=echo-cb96f8d94-879sn
Hostname=echo-cb96f8d94-879sn
Hostname=echo-cb96f8d94-879sn
Hostname=echo-cb96f8d94-879sn
Hostname=echo-cb96f8d94-879sn
Hostname=echo-cb96f8d94-879sn

The problem here is Kubernetes will determine the backend to send to when the connection is established, and all future requests on the same connection will be sent to the same backend. In our example here, our first 5 requests are all sent to echo-cb96f8d94-2ssll, while our next set (using a new connection) are all sent to echo-cb96f8d94-879sn. Our third instance never receives any requests.

With Istio, HTTP traffic (including HTTP/2 and gRPC) is automatically detected, and our services will automatically be load balanced per request, rather than per connection:

$ curl http://echo/{0..5} -s | grep Hostname
Hostname=echo-cb96f8d94-wf4xk
Hostname=echo-cb96f8d94-rpfqz
Hostname=echo-cb96f8d94-cgmxr
Hostname=echo-cb96f8d94-wf4xk
Hostname=echo-cb96f8d94-rpfqz
Hostname=echo-cb96f8d94-cgmxr

Here we can see our requests are round-robin load balanced between all backends.

In addition to these better defaults, Istio offers customization of a variety of traffic management settings, including timeouts, retries, and much more.

You can now find the full schedule of events which includes a series of English sessions and Chinese sessions.

By attending the conference, you’ll connect with community members from across the globe. Each day you will find keynotes, technical talks, lightning talks, panel discussions, workshops and roadmap sessions led by diverse speakers representing the Istio community. You can also connect with other Istio and Open Source ecosystem community members through social hour events that include activities on the social platform Gather.town, a live cartoonist, virtual swag bags, raffles, live music and games.

Don’t miss it! Registration is free. We look forward to seeing you at the first IstioCon!

Istio’s authorization policy provides access control for services in the mesh. It is fast, powerful and a widely used feature. We have made continuous improvements to make policy more flexible since its first release in Istio 1.4, including the DENY action, exclusion semantics, X-Forwarded-For header support, nested JWT claim support and more. These features improve the flexibility of the authorization policy, but there are still many use cases that cannot be supported with this model, for example:

• You have your own in-house authorization system that cannot be easily migrated to, or cannot be easily replaced by, the authorization policy.

• You want to integrate with a 3rd-party solution (e.g. Open Policy Agent or oauth2 proxy) which may require use of the low-level Envoy configuration APIs in Istio, or may not be possible at all.

• Authorization policy lacks necessary semantics for your use case.

Solution

In Istio 1.9, we have implemented extensibility into authorization policy by introducing a CUSTOM action, which allows you to delegate the access control decision to an external authorization service.

The CUSTOM action allows you to integrate Istio with an external authorization system that implements its own custom authorization logic. The following diagram shows the high level architecture of this integration:

At configuration time, the mesh admin configures an authorization policy with a CUSTOM action to enable the external authorization on a proxy (either gateway or sidecar). The admin should verify the external auth service is up and running.

At runtime,

1. A request is intercepted by the proxy, and the proxy will send check requests to the external auth service, as configured by the user in the authorization policy.

2. The external auth service will make the decision whether to allow it or not.

3. If allowed, the request will continue and will be enforced by any local authorization defined by ALLOW/DENY action.

4. If denied, the request will be rejected immediately.

Let’s look at an example authorization policy with the CUSTOM action:

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: ext-authz
  namespace: istio-system
spec:
  # The selector applies to the ingress gateway in the istio-system namespace.
  selector:
    matchLabels:
      app: istio-ingressgateway
  # The action "CUSTOM" delegates the access control to an external authorizer, this is different from
  # the ALLOW/DENY action that enforces the access control right inside the proxy.
  action: CUSTOM
  # The provider specifies the name of the external authorizer defined in the meshconfig, which tells where and how to
  # talk to the external auth service. We will cover this more later.
  provider:
    name: "my-ext-authz-service"
  # The rule specifies that the access control is triggered only if the request path has the prefix "/admin/".
  # This allows you to easily enable or disable the external authorization based on the requests, avoiding the external
  # check request if it is not needed.
  rules:
  - to:
    - operation:
        paths: ["/admin/*"]

It refers to a provider called my-ext-authz-service which is defined in the mesh config:

extensionProviders:
# The name "my-ext-authz-service" is referred to by the authorization policy in its provider field.
- name: "my-ext-authz-service"
  # The "envoyExtAuthzGrpc" field specifies the type of the external authorization service is implemented by the Envoy
  # ext-authz filter gRPC API. The other supported type is the Envoy ext-authz filter HTTP API.
  # See more in https://www.envoyproxy.io/docs/envoy/v1.16.2/intro/arch_overview/security/ext_authz_filter.
  envoyExtAuthzGrpc:
    # The service and port specifies the address of the external auth service, "ext-authz.istio-system.svc.cluster.local"
    # means the service is deployed in the mesh. It can also be defined out of the mesh or even inside the pod as a separate
    # container.
    service: "ext-authz.istio-system.svc.cluster.local"
    port: 9000

The authorization policy of CUSTOM action enables the external authorization in runtime, it could be configured to trigger the external authorization conditionally based on the request using the same rule that you have already been using with other actions.

The external authorization service is currently defined in the meshconfig API and referred to by its name. It could be deployed in the mesh with or without proxy. If with the proxy, you could further use PeerAuthentication to enable mTLS between the proxy and your external authorization service.

The CUSTOM action is currently in the experimental stage; the API might change in a non-backward compatible way based on user feedback. The authorization policy rules currently don’t support authentication fields (e.g. source principal or JWT claim) when used with the CUSTOM action. Only one provider is allowed for a given workload, but you can still use different providers on different workloads.

For more information, please see the Better External Authorization design doc.

Example with OPA

In this section, we will demonstrate using the CUSTOM action with the Open Policy Agent as the external authorizer on the ingress gateway. We will conditionally enable the external authorization on all paths except /ip.

You can also refer to the external authorization task for a more basic introduction that uses a sample ext-authz server.

Create the example OPA policy

Run the following command create an OPA policy that allows the request if the prefix of the path is matched with the claim “path” (base64 encoded) in the JWT token:

$ cat > policy.rego <<EOF
package envoy.authz

import input.attributes.request.http as http_request

default allow = false

token = {"valid": valid, "payload": payload} {
    [_, encoded] := split(http_request.headers.authorization, " ")
    [valid, _, payload] := io.jwt.decode_verify(encoded, {"secret": "secret"})
}

allow {
    is_token_valid
    action_allowed
}

is_token_valid {
  token.valid
  now := time.now_ns() / 1000000000
  token.payload.nbf <= now
  now < token.payload.exp
}

action_allowed {
  startswith(http_request.path, base64url.decode(token.payload.path))
}
EOF
$ kubectl create secret generic opa-policy --from-file policy.rego

Deploy httpbin and OPA

Enable the sidecar injection:

$ kubectl label ns default istio-injection=enabled

Run the following command to deploy the example application httpbin and OPA. The OPA could be deployed either as a separate container in the httpbin pod or completely in a separate pod:

$ kubectl apply -f - <<EOF
apiVersion: v1
kind: Service
metadata:
  name: httpbin-with-opa
  labels:
    app: httpbin-with-opa
    service: httpbin-with-opa
spec:
  ports:
  - name: http
    port: 8000
    targetPort: 80
  selector:
    app: httpbin-with-opa
---
# Define the service entry for the local OPA service on port 9191.
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: local-opa-grpc
spec:
  hosts:
  - "local-opa-grpc.local"
  endpoints:
  - address: "127.0.0.1"
  ports:
  - name: grpc
    number: 9191
    protocol: GRPC
  resolution: STATIC
---
kind: Deployment
apiVersion: apps/v1
metadata:
  name: httpbin-with-opa
  labels:
    app: httpbin-with-opa
spec:
  replicas: 1
  selector:
    matchLabels:
      app: httpbin-with-opa
  template:
    metadata:
      labels:
        app: httpbin-with-opa
    spec:
      containers:
        - image: docker.io/kennethreitz/httpbin
          imagePullPolicy: IfNotPresent
          name: httpbin
          ports:
          - containerPort: 80
        - name: opa
          image: openpolicyagent/opa:latest-envoy
          securityContext:
            runAsUser: 1111
          volumeMounts:
          - readOnly: true
            mountPath: /policy
            name: opa-policy
          args:
          - "run"
          - "--server"
          - "--addr=localhost:8181"
          - "--diagnostic-addr=0.0.0.0:8282"
          - "--set=plugins.envoy_ext_authz_grpc.addr=:9191"
          - "--set=plugins.envoy_ext_authz_grpc.query=data.envoy.authz.allow"
          - "--set=decision_logs.console=true"
          - "--ignore=.*"
          - "/policy/policy.rego"
          livenessProbe:
            httpGet:
              path: /health?plugins
              scheme: HTTP
              port: 8282
            initialDelaySeconds: 5
            periodSeconds: 5
          readinessProbe:
            httpGet:
              path: /health?plugins
              scheme: HTTP
              port: 8282
            initialDelaySeconds: 5
            periodSeconds: 5
      volumes:
        - name: proxy-config
          configMap:
            name: proxy-config
        - name: opa-policy
          secret:
            secretName: opa-policy
EOF

$ kubectl apply -f - <<EOF
apiVersion: v1
kind: Service
metadata:
  name: opa
  labels:
    app: opa
spec:
  ports:
  - name: grpc
    port: 9191
    targetPort: 9191
  selector:
    app: opa
---
kind: Deployment
apiVersion: apps/v1
metadata:
  name: opa
  labels:
    app: opa
spec:
  replicas: 1
  selector:
    matchLabels:
      app: opa
  template:
    metadata:
      labels:
        app: opa
    spec:
      containers:
        - name: opa
          image: openpolicyagent/opa:latest-envoy
          securityContext:
            runAsUser: 1111
          volumeMounts:
          - readOnly: true
            mountPath: /policy
            name: opa-policy
          args:
          - "run"
          - "--server"
          - "--addr=localhost:8181"
          - "--diagnostic-addr=0.0.0.0:8282"
          - "--set=plugins.envoy_ext_authz_grpc.addr=:9191"
          - "--set=plugins.envoy_ext_authz_grpc.query=data.envoy.authz.allow"
          - "--set=decision_logs.console=true"
          - "--ignore=.*"
          - "/policy/policy.rego"
          ports:
          - containerPort: 9191
          livenessProbe:
            httpGet:
              path: /health?plugins
              scheme: HTTP
              port: 8282
            initialDelaySeconds: 5
            periodSeconds: 5
          readinessProbe:
            httpGet:
              path: /health?plugins
              scheme: HTTP
              port: 8282
            initialDelaySeconds: 5
            periodSeconds: 5
      volumes:
        - name: proxy-config
          configMap:
            name: proxy-config
        - name: opa-policy
          secret:
            secretName: opa-policy
EOF

$ kubectl apply -f @samples/httpbin/httpbin.yaml@

Define external authorizer

Run the following command to edit the meshconfig:

$ kubectl edit configmap istio -n istio-system

Add the following extensionProviders to the meshconfig:

apiVersion: v1
data:
  mesh: |-
    # Add the following contents:
    extensionProviders:
    - name: "opa.local"
      envoyExtAuthzGrpc:
        service: "local-opa-grpc.local"
        port: "9191"

apiVersion: v1
data:
  mesh: |-
    # Add the following contents:
    extensionProviders:
    - name: "opa.default"
      envoyExtAuthzGrpc:
        service: "opa.default.svc.cluster.local"
        port: "9191"

Create an AuthorizationPolicy with a CUSTOM action

Run the following command to create the authorization policy that enables the external authorization on all paths except /ip:

$ kubectl apply -f - <<EOF
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: httpbin-opa
spec:
  selector:
    matchLabels:
      app: httpbin-with-opa
  action: CUSTOM
  provider:
    name: "opa.local"
  rules:
  - to:
    - operation:
        notPaths: ["/ip"]
EOF

$ kubectl apply -f - <<EOF
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: httpbin-opa
spec:
  selector:
    matchLabels:
      app: httpbin
  action: CUSTOM
  provider:
    name: "opa.default"
  rules:
  - to:
    - operation:
        notPaths: ["/ip"]
EOF

Test the OPA policy

1. Create a client pod to send the request:

   Zip

   $ kubectl apply -f @samples/sleep/sleep.yaml@
   $ export SLEEP_POD=$(kubectl get pod -l app=sleep -o jsonpath={.items..metadata.name})

2. Use a test JWT token signed by the OPA:

   $ export TOKEN_PATH_HEADERS="eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJwYXRoIjoiTDJobFlXUmxjbk09IiwibmJmIjoxNTAwMDAwMDAwLCJleHAiOjE5MDAwMDAwMDB9.9yl8LcZdq-5UpNLm0Hn0nnoBHXXAnK4e8RSl9vn6l98"

   The test JWT token has the following claims:

   {
     "path": "L2hlYWRlcnM=",
     "nbf": 1500000000,
     "exp": 1900000000
   }

   The path claim has value L2hlYWRlcnM= which is the base64 encode of /headers.

3. Send a request to path /headers without a token. This should be rejected with 403 because there is no JWT token:

   Deploy OPA in the same pod Deploy OPA in a separate pod

   $ kubectl exec ${SLEEP_POD} -c sleep  -- curl http://httpbin-with-opa:8000/headers -s -o /dev/null -w "%{http_code}\n"
   403

4. Send a request to path /get with a valid token. This should be rejected with 403 because the path /get is not matched with the token /headers:

   Deploy OPA in the same pod Deploy OPA in a separate pod

   $ kubectl exec ${SLEEP_POD} -c sleep  -- curl http://httpbin-with-opa:8000/get -H "Authorization: Bearer $TOKEN_PATH_HEADERS" -s -o /dev/null -w "%{http_code}\n"
   403

5. Send a request to path /headers with valid token. This should be allowed with 200 because the path is matched with the token:

   Deploy OPA in the same pod Deploy OPA in a separate pod

   $ kubectl exec ${SLEEP_POD} -c sleep  -- curl http://httpbin-with-opa:8000/headers -H "Authorization: Bearer $TOKEN_PATH_HEADERS" -s -o /dev/null -w "%{http_code}\n"
   200

6. Send request to path /ip without token. This should be allowed with 200 because the path /ip is excluded from authorization:

   Deploy OPA in the same pod Deploy OPA in a separate pod

   $ kubectl exec ${SLEEP_POD} -c sleep  -- curl http://httpbin-with-opa:8000/ip -s -o /dev/null -w "%{http_code}\n"
   200

7. Check the proxy and OPA logs to confirm the result.

Summary

In Istio 1.9, the CUSTOM action in the authorization policy allows you to easily integrate Istio with any external authorization system with the following benefits:

• First-class support in the authorization policy API

• Ease of usage: define the external authorizer simply with a URL and enable with the authorization policy, no more hassle with the EnvoyFilter API

• Conditional triggering, allowing improved performance

• Support for various deployment type of the external authorizer:

  • A normal service and pod with or without proxy

  • Inside the workload pod as a separate container

  • Outside the mesh

We’re working to promote this feature to a more stable stage in following versions and welcome your feedback at discuss.istio.io.

Acknowledgements

Thanks to Craig Box, Christian Posta and Limin Wang for reviewing drafts of this blog.

We can set Istio up as a proxy service for these upstream services. This allows us to benefit from capabilities like authorization/authentication, traceability and observability, even while legacy services stand as they are.

At the end of this article there is a hands-on exercise where you can simulate the scenario. In the exercise, an upstream service hosted at https://httpbin.org will be proxied by an Istio egress gateway.

If you are familiar with Istio, one of the methods offered to connect to upstream services is through an egress gateway.

You can deploy one to control all the upstream traffic or you can deploy multiple in order to have fine-grained control and satisfy the single-responsibility principle as this picture shows:

With this model, one egress gateway is in charge of exactly one upstream service.

Although the Operator spec allows you to deploy multiple egress gateways, the manifest can become unmanageable:

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
[...]
spec:
    egressGateways:
    - name: egressgateway-1
      enabled: true
    - name: egressgateway-2
      enabled: true
    [egressgateway-3, egressgateway-4, ...]
    - name: egressgateway-N
      enabled: true
[...]

As a benefit of decoupling egress getaways from the Operator manifest, you have enabled the possibility of setting up custom readiness probes to have both services (Gateway and upstream Service) aligned.

You can also inject OPA as a sidecar into the pod to perform authorization with complex rules (OPA envoy plugin).

As you can see, your possibilities increase and Istio becomes very extensible.

Let’s look at how you can implement this pattern.

Solution

There are several ways to perform this task, but here you will find how to define multiple Operators and deploy the generated resources.

In the following section you will deploy an egress gateway to connect to an upstream service: httpbin (https://httpbin.org/)

At the end, you will have:

Hands on

Prerequisites

• kind (Kubernetes-in-Docker - perfect for local development)
• istioctl

Kind

Save this as config.yaml.

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
kubeadmConfigPatches:
  - |
    apiVersion: kubeadm.k8s.io/v1beta2
    kind: ClusterConfiguration
    metadata:
      name: config
    apiServer:
      extraArgs:
        "service-account-issuer": "kubernetes.default.svc"
        "service-account-signing-key-file": "/etc/kubernetes/pki/sa.key"

$ kind create cluster --name <my-cluster-name> --config config.yaml

Where <my-cluster-name> is the name for the cluster.

Istio Operator with Istioctl

Install the Operator

$ istioctl operator init --watchedNamespaces=istio-operator

$ kubectl create ns istio-system

Save this as operator.yaml:

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  name: istio-operator
  namespace: istio-operator
spec:
  profile: default
  tag: 1.8.0
  meshConfig:
    accessLogFile: /dev/stdout
    outboundTrafficPolicy:
      mode: REGISTRY_ONLY

$ kubectl apply -f operator.yaml

Deploy Egress Gateway

The steps for this task assume:

• The service is installed under the namespace: httpbin.
• The service name is: http-egress.

Istio 1.8 introduced the possibility to apply overlay configuration, to give fine-grain control over the created resources.

Save this as egress.yaml:

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  profile: empty
  tag: 1.8.0
  namespace: httpbin
  components:
    egressGateways:
    - name: httpbin-egress
      enabled: true
      label:
        app: istio-egressgateway
        istio: egressgateway
        custom-egress: httpbin-egress
      k8s:
        overlays:
        - kind: Deployment
          name: httpbin-egress
          patches:
          - path: spec.template.spec.containers[0].readinessProbe
            value:
              failureThreshold: 30
              exec:
                command:
                  - /bin/sh
                  - -c
                  - curl http://localhost:15021/healthz/ready && curl https://httpbin.org/status/200
              initialDelaySeconds: 1
              periodSeconds: 2
              successThreshold: 1
              timeoutSeconds: 1
  values:
    gateways:
      istio-egressgateway:
        runAsRoot: true

Create the namespace where you will install the egress gateway:

$ kubectl create ns httpbin

As it is described in the documentation, you can deploy several Operator resources. However, they have to be pre-parsed and then applied to the cluster.

$ istioctl manifest generate -f egress.yaml | kubectl apply -f -

Istio configuration

Now you will configure Istio to allow connections to the upstream service at https://httpbin.org.

Certificate for TLS

You need a certificate to make a secure connection from outside the cluster to your egress service.

How to generate a certificate is explained in the Istio ingress documentation.

Create and apply one to be used at the end of this article to access the service from outside the cluster (<my-proxied-service-hostname>):

$ kubectl create -n istio-system secret tls <my-secret-name> --key=<key> --cert=<cert>

Where <my-secret-name> is the name used later for the Gateway resource. <key> and <cert> are the files for the certificate. <cert>.

Ingress Gateway

Create a Gateway resource to operate ingress gateway to accept requests.

An example:

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: my-ingressgateway
  namespace: istio-system
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - "<my-proxied-service-hostname>"
    port:
      name: http
      number: 80
      protocol: HTTP
    tls:
     httpsRedirect: true
  - port:
      number: 443
      name: https
      protocol: https
    hosts:
    - "<my-proxied-service-hostname>"
    tls:
      mode: SIMPLE
      credentialName: <my-secret-name>

Where <my-proxied-service-hostname> is the hostname to access the service through the my-ingressgateway and <my-secret-name> is the secret which contains the certificate.

Egress Gateway

Create another Gateway object, but this time to operate the egress gateway you have already installed:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: "httpbin-egress"
  namespace: "httpbin"
spec:
  selector:
    istio: egressgateway
    service.istio.io/canonical-name: "httpbin-egress"
  servers:
  - hosts:
    - "<my-proxied-service-hostname>"
    port:
      number: 80
      name: http
      protocol: HTTP

Where <my-proxied-service-hostname> is the hostname to access through the my-ingressgateway.

Virtual Service

Create a VirtualService for three use cases: