Forem: Jeongwoo Kim

[Hands-on] Kubernetes Pod Certificate Request introduced in v1.35

Jeongwoo Kim — Tue, 07 Apr 2026 00:04:40 +0000

Goal

[!NOTE]
In hurry? Jump to the result!

The goal of this document is to generate auto signed certificate for any pod with the following projected volumes:

volumes:
- name: creds
  projected:
    sources:
    - podCertificate:
        signerName: row-major.net/spiffe
        keyType: ED25519
        credentialBundlePath: service.crt
        keyPath: service.key
    - clusterTrustBundle:
        name: row-major.net:spiffe:primary-bundle
        path: ca.crt

cover_image: ./thumbnail.png
Goal
Table of Contents
Walkthrough
- Setup: Working Directory
- Setup: Kind Cluster with Cert Provisioning Enabled
- Setup: Mash Controller Deployed
- Verify: Auto Distributed Certificate Feature on Sample Deployment
What's next?
Closing

Walkthrough

Here is the step-by-step record of how I achieved the goal.

Setup: Working Directory

Let's quickly create a test directory build:

test_name=pod_certificate_request
tmp_dir=$(date +%y%m%d_%H%M%S_$test_name)
mkdir -p ~/test_dive/$tmp_dir
cd ~/test_dive/$tmp_dir

Setup: Kind Cluster with Cert Provisioning Enabled

[!NOTE]
Please note that the point of this blog's release already contains v1.35+ that includes the feature, so we do not need to set specific version.

Create a kind cluster locally with the following command:

_cluster_name="cert-provisioning"
_k8s_version="v1.35.0"

cat <<EOF | kind create cluster --name "$_cluster_name" --image kindest/node:$_k8s_version --config -
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
featureGates:
  PodCertificateRequest: true
  ClusterTrustBundle: true
  ClusterTrustBundleProjection: true
runtimeConfig:
  "certificates.k8s.io/v1beta1/podcertificaterequests": "true"
  "certificates.k8s.io/v1beta1/clustertrustbundles": "true"
nodes:
- role: control-plane
- role: worker
EOF

Check new cluster created with version v1.35.0:

kubectl get nodes

# NAME                              STATUS   ROLES           AGE   VERSION
# cert-provisioning-control-plane   Ready    control-plane   29s   v1.35.0
# cert-provisioning-worker          Ready    <none>          19s   v1.35.0

Setup: Mash Controller Deployed

Let's deploy the mesh-controller developed by @ahmedtd as a sample. First, clone the helper project:

git clone https://github.com/ahmedtd/mesh-example.git mesh_example

Before we deploy the mesh-controller, we need to create a CA pool secret that the controller can use to sign the certificate requests. The mesh-controller expects two different CA pool secrets:

service-dns-ca-pool: for service DNS certificates between kubeapi server and mesh controller
spiffe-ca-pool: for ca certificate to be used to sign the auto distributed certificates to pods

The helper project provides the following command to create the CA pool & save as k8 secrets:

(cd mesh_example && go run ./cmd/meshtool make-ca-pool-secret --namespace mesh-controller --name service-dns-ca-pool --ca-id 1)
(cd mesh_example && go run ./cmd/meshtool make-ca-pool-secret --namespace mesh-controller --name spiffe-ca-pool --ca-id 1)

Check secrets created:

kubectl get secrets -n mesh-controller

# NAME                  TYPE     DATA   AGE
# service-dns-ca-pool   Opaque   1      9s
# spiffe-ca-pool        Opaque   1      9s

With secrets available as prerequisite, we can deploy the mesh controller:

kubectl apply -f ./mesh_example/controller-manifests

# namespace/mesh-controller configured
# clusterrole.rbac.authorization.k8s.io/meshtool-signer created
# clusterrolebinding.rbac.authorization.k8s.io/meshtool-is-a-meshtool-signer created
# role.rbac.authorization.k8s.io/meshtool-controller created
# rolebinding.rbac.authorization.k8s.io/meshtool-is-a-meshtool-controller created
# deployment.apps/mesh-controller created

The command above will create a namespace mesh-controller and deploy the mesh-controller in it. mesh-controller is a custom controller and its current job is to:

Watch PodCertificateRequest resources, created by Kubernetes's kubelet with user's deployment's request
Sign the certificate request with desiginated CA pool injected as a secret
Return the signed certificate and key back to the kubelet

Check if the mesh controller is running:

kubectl get pods -n mesh-controller

# NAME                               READY   STATUS    RESTARTS   AGE
# mesh-controller-547cd7996b-blqsf   1/1     Running   0          20s

Verify: Auto Distributed Certificate Feature on Sample Deployment

With all the prerequisites and the mesh controller in place, it's time to put it to the test. Let's deploy a sample application and verify if the certificates are automatically distributed and correctly mounted inside the pod.

First of all, let's create a namespace for our sample application:

kubectl create namespace podcert-demo

# namespace/podcert-demo created

Next, deploy a k8s deployment:

cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: spiffe-demo-deploy
  namespace: podcert-demo
spec:
  replicas: 2
  selector:
    matchLabels:
      app: spiffe-demo
  template:
    metadata:
      labels:
        app: spiffe-demo
    spec:
      serviceAccountName: default
      automountServiceAccountToken: false
      containers:
      - name: app
        image: alpine/openssl
        command: ["sh", "-lc", "sleep infinity"]
        volumeMounts:
        - name: creds
          mountPath: /var/run/tls
          readOnly: true
      volumes:
      - name: creds
        projected:
          sources:
          - podCertificate:
              signerName: row-major.net/spiffe
              keyType: ED25519
              credentialBundlePath: service.crt
              keyPath: service.key
          - clusterTrustBundle:
              name: row-major.net:spiffe:primary-bundle
              path: ca.crt
EOF

# deployment.apps/spiffe-demo-deploy created

Note that the kubelet can notice that this deployment (or its pods) require the certificate by:

- name: creds
  projected:
    sources:
    - podCertificate:
        signerName: row-major.net/spiffe
        keyType: ED25519
        credentialBundlePath: service.crt
        keyPath: service.key
    - clusterTrustBundle:
        name: row-major.net:spiffe:primary-bundle
        path: ca.crt

The mesh-controller by default is watching the signerName row-major.net/spiffe and row-major.net/service-dns. If we ever build our own signer, we can simply change the singerName in the deployment manifest.

Finally, let's check the certificate and key are mounted on /var/run/tls inside the pod:

kubectl exec -it deploy/spiffe-demo-deploy -n podcert-demo -- ls -al /var/run/tls

# total 4
# drwxrwxrwt    3 root     root           140 Apr  6 02:46 .
# drwxr-xr-x    1 root     root          4096 Apr  6 02:46 ..
# drwxr-xr-x    2 root     root           100 Apr  6 02:46 ..2026_04_06_02_46_52.3629103651
# lrwxrwxrwx    1 root     root            32 Apr  6 02:46 ..data -> ..2026_04_06_02_46_52.3629103651
# lrwxrwxrwx    1 root     root            13 Apr  6 02:46 ca.crt -> ..data/ca.crt
# lrwxrwxrwx    1 root     root            18 Apr  6 02:46 service.crt -> ..data/service.crt
# lrwxrwxrwx    1 root     root            18 Apr  6 02:46 service.key -> ..data/service.key

Check the certificate auto distributed:

kubectl exec -it deploy/spiffe-demo-deploy -n podcert-demo \
  -- openssl x509 -in /var/run/tls/service.crt -noout -text

# Certificate:
#     Data:
#         Version: 3 (0x2)
#         Serial Number:
#             02:20:72:93:62:05:37:22:fe:41:4f:ad:3f:ce:c5:bd:54:31:9e:0c
#         Signature Algorithm: ED25519
#         Issuer:
#         Validity
#             Not Before: Apr  6 02:44:52 2026 GMT
#             Not After : Apr  7 02:44:52 2026 GMT
#         Subject:
#         Subject Public Key Info:
#             Public Key Algorithm: ED25519
#                 ED25519 Public-Key:
#                 pub:
#                     0e:bb:f2:f7:fc:ee:5f:21:83:c3:87:da:3c:eb:79:
#                     dd:fe:5c:41:7e:90:cb:aa:e0:96:a8:64:e0:c5:1c:
#                     11:90
#         X509v3 extensions:
#             X509v3 Key Usage: critical
#                 Digital Signature
#             X509v3 Extended Key Usage:
#                 TLS Web Client Authentication, TLS Web Server Authentication
#             X509v3 Basic Constraints: critical
#                 CA:FALSE
#             X509v3 Subject Alternative Name: critical
#                 URI:spiffe://cluster.local/ns/podcert-demo/sa/default
#     Signature Algorithm: ED25519

Let's see the root CA with one year validity:

kubectl exec -it deploy/spiffe-demo-deploy -n podcert-demo \
  -- openssl x509 -in /var/run/tls/ca.crt -noout -text

# Certificate:
#     Data:
#         Version: 3 (0x2)
#         Serial Number:
#             70:f5:36:4e:2b:6f:9e:fc:fb:2d:cf:5d:32:77:65:a5:fe:5e:08:64
#         Signature Algorithm: ED25519
#         Issuer:
#         Validit
#             Not Before: Apr  6 02:39:27 2026 GMT
#             Not After : Apr  6 02:39:27 2027 GMT
#         Subject:
#         Subject Public Key Info:
#             Public Key Algorithm: ED25519
#                 ED25519 Public-Key:
#                 pub:
#                     87:6d:54:67:8f:0c:d7:ed:5a:e4:a5:fd:a0:fb:81:
#                     7d:e2:7e:84:44:dc:5f:2f:99:12:63:b4:08:2c:b0:
#                     0b:5b
#         X509v3 extensions:
#             X509v3 Key Usage: critical
#                 Digital Signature, Certificate Sign
#             X509v3 Basic Constraints: critical
#                 CA:TRUE
#             X509v3 Subject Key Identifier:
#                 A0:96:CE:C1:73:8E:63:5A:26:C2:0B:BE:45:46:4D:25:50:C4:7E:EB
#     Signature Algorithm: ED25519

This is it! The kubelet automatically created a PodCertificateRequest and the mesh-controller signed it and returned the signed certificate and key back to the kubelet. The kubelet then mounted the certificate and key as a projected volume in the pod.

What's next?

The mesh-controller sample is a really good starting point to understand how the PodCertificateRequest works. However, if we want to generate a certificate signed by external CA, how can we do this? The next step is to create a controller that does detect the PodCertificateRequest, and instead of signing by itself with secret injected CA, it forwards the request to the external authentication/authorization service (like Athenz), and returns the signed certificate and key back to the kubelet.

Closing

If you enjoyed this deep dive, please leave a like & subscribe for more!

Also, leave comments if you have any questions or suggestions. Thank you in advance!

[Hands-on] Testing Separate Ports in Athenz ZTS

Jeongwoo Kim — Wed, 01 Apr 2026 00:13:15 +0000

[Hands-on] Testing Separate Ports in Athenz ZTS

The goal of this test is to verify the new port-URI filtering feature introduced in PR: Adding support to filter requests based on port-uri combination #3190.

Result

I successfully reproduced the multiple ports configuration on the ZTS server. Initially, the server was only listening on the default port 4443:

# Defaulted container "athenz-zts-server" out of: athenz-zts-server, zms-cli (init), athenz-conf (init), athenz-plugins (init)
# sh: 1: ss: not found
# Active Internet connections (only servers)
# Proto Recv-Q Send-Q Local Address           Foreign Address         State
# tcp6       0      0 :::4443                 :::*                    LISTEN

After applying the new configuration, it now listens on both the original port 4443 AND the newly added port 8443:

# Active Internet connections (only servers)
# Proto Recv-Q Send-Q Local Address           Foreign Address         State
# tcp6       0      0 :::8443                 :::*                    LISTEN
# tcp6       0      0 :::4443                 :::*                    LISTEN

[Hands-on] Testing Separate Ports in Athenz ZTS
Result
Table of Contents
Steps to Reproduce
- Setup: Working Directory & Athenz Server
- Setup: port-uri configuration
- Setup: Volume Injection
- Setup: port open in ZTS
- Setup: Readiness/Liveness watching the port 8443
- Setup: athenz.properties
- Setup: Rollback to 4443
What I learned
What's next?
Closing

Steps to Reproduce

Walk through the following steps to achieve the same result.

Setup: Working Directory & Athenz Server

Let's quickly create a test directory build:

test_name=separate_port_in_athenz
tmp_dir=$(date +%y%m%d_%H%M%S_$test_name)
mkdir -p ~/test_dive/$tmp_dir
cd ~/test_dive/$tmp_dir

This tutorial has a one-command setup script that will create a local kubernetes cluster with kind, and deploy athenz server into it:

git clone https://github.com/mlajkim/dive-manifest.git manifest
make -C manifest setup

# ...
# ✅ Athenz Server deployment finished

Check that every pod is running:

kubectl get pods -n athenz

# NAME                                 READY   STATUS    RESTARTS   AGE
# athenz-cli-574d747dff-qk8lr          1/1     Running   0          5m5s
# athenz-db-0                          1/1     Running   0          5m5s
# athenz-ui-59f7f77667-wgr5l           2/2     Running   0          5m4s
# athenz-zms-server-568d4cfd89-whcjn   1/1     Running   0          5m4s
# athenz-zts-server-6966ff7f66-897lg   1/1     Running   0          5m4

Setup: port-uri configuration

Please take a look what is inside the https://raw.githubusercontent.com/AthenZ/athenz/refs/heads/master/containers/jetty/conf/port-uri.json.example.

This sample configuration file is basically saying:

This file is a security rulebook that decides which API paths can be accessed and whether strict security (mTLS) is required for four different server ports.
Port 4443 is the main port that allows all API requests, but it absolutely requires mTLS to keep things safe.
The other three ports don't need mTLS, but they are strictly locked down to only handle their own specific jobs—like registration (9443), health checks (8443), or OpenID (443).

Use the following commands to create a K8s cm from the sample file:

_ns=athenz
_cm_name=port-uri-config
_cm_file_name=port-uri.json

cat <<EOF > $_cm_file_name
{
  "ports": [
    {
      "port": 4443,
      "mtls_required": false,
      "description": "Main API port - all endpoints with default 4443, but no mTLS required for test sake",
      "allowed_endpoints": []
    },
    {
      "port": 8443,
      "mtls_required": false,
      "description": "Health/status port - /zts/v1/status (ZTS API) and /status (file-based health check returning OK)",
      "allowed_endpoints": [
        {
          "path": "/zts/v1/status",
          "methods": ["GET"],
          "description": "ZTS API status - returns JSON { \"code\": 200, \"message\": \"OK\" }"
        },
        {
          "path": "/status",
          "methods": ["GET"],
          "description": "Legacy file-based health check - returns OK when athenz.health_check_uri_list includes /status"
        }
      ]
    }
  ]
}
EOF

kubectl create ns $_ns 2>/dev/null || true && \
kubectl create configmap $_cm_name -n $_ns --from-file=$_cm_file_name --dry-run=client -o yaml | kubectl apply -f - && \
rm $_cm_file_name

# configmap/port-uri-config created

Check:

kubectl get cm $_cm_name -n $_ns

# NAME               DATA   AGE
# port-uri-config    1      18s

Setup: Volume Injection

Let's mount the cm created above into the zts server pod.

_ns=athenz
_cm_name=port-uri-config
_cm_file_name=port-uri.json
_deploy_name=athenz-zts-server
_mnt_path=/opt/athenz/zts/conf/$_cm_file_name
_patch_json=$(cat <<EOF
[
  {
    "op": "add",
    "path": "/spec/template/spec/volumes/-",
    "value": {
      "name": "port-uri-vol",
      "configMap": {
        "name": "${_cm_name}"
      }
    }
  },
  {
    "op": "add",
    "path": "/spec/template/spec/containers/0/volumeMounts/-",
    "value": {
      "name": "port-uri-vol",
      "mountPath": "${_mnt_path}",
      "subPath": "${_cm_file_name}"
    }
  }
]
EOF
)

kubectl patch deployment $_deploy_name -n $_ns --type='json' -p="$_patch_json"

# deployment.apps/athenz-zts-server patched

Check:

_deploy_name=athenz-zts-server
_ns=athenz

kubectl exec -it deployment/$_deploy_name -n $_ns -- sh -c 'ls -l /opt/athenz/zts/conf/port-uri.json && cat /opt/athenz/zts/conf/port-uri.json'

# ...
#         {
#           "path_starts_with": "/zts/v1/.well-known",
#           "path_ends_with": "openid-configuration",
#           "methods": ["GET"],
#           "description": "OpenID discovery (alternative using path_starts_with and path_ends_with)"
#         }
#       ]
#     }
#   ]
# }

Setup: port open in ZTS

[!TIPS]
You can learn what kind of port is required with the port-uri.json

[!NOTE]
We only need the status, but for reference sake, let's set up multiple ports open (not security best practice, but this is tutorial!)

The default port https:4443 is used for all API requests by manifest default. Let's open other ports as well in Kubernetes:

_deploy_name=athenz-zts-server
_ns=athenz

kubectl patch deployment $_deploy_name -n $_ns --type='json' -p='[
  { "op": "add", "path": "/spec/template/spec/containers/0/ports/-", "value": { "containerPort": 8443, "name": "status", "protocol": "TCP" } },
  { "op": "add", "path": "/spec/template/spec/containers/0/ports/-", "value": { "containerPort": 9443, "name": "instance", "protocol": "TCP" } },
  { "op": "add", "path": "/spec/template/spec/containers/0/ports/-", "value": { "containerPort": 443, "name": "openid", "protocol": "TCP" } }
]'

# deployment.apps/athenz-zts-server patched

Check:

_deploy_name=athenz-zts-server
_ns=athenz

kubectl get deployment $_deploy_name -n $_ns -o custom-columns="PORTS:.spec.template.spec.containers[0].ports[*].containerPort"

# PORTS
# 4443,8443,9443,443

Setup: Readiness/Liveness watching the port 8443

[!NOTE]
This phase deliberately fails readiness state and will be fixed in the next phase.

We have done the following so far:

k8s cm created and mounted as a file in ZTS pod
ZTS deployment patched to open ports required to communicate between ZTS and requsters

Now we want to change the readiness probe to watch the port 8443 instead of 4443:

_ns=athenz
_deploy=athenz-zts-server
_new_status_port=8443
_patch_json=$(cat <<EOF
[
  {
    "op": "replace",
    "path": "/spec/template/spec/containers/0/readinessProbe/exec/command",
    "value": [
      "curl",
      "-s",
      "--fail",
      "--resolve",
      "athenz-zts-server.athenz:${_new_status_port}:127.0.0.1",
      "https://athenz-zts-server.athenz:${_new_status_port}/zts/v1/status"
    ]
  },
  {
    "op": "replace",
    "path": "/spec/template/spec/containers/0/livenessProbe/exec/command",
    "value": [
      "curl",
      "-s",
      "--fail",
      "--resolve",
      "athenz-zts-server.athenz:${_new_status_port}:127.0.0.1",
      "https://athenz-zts-server.athenz:${_new_status_port}/zts/v1/status"
    ]
  }
]
EOF
)

kubectl patch deployment $_deploy -n $_ns --type='json' -p="$_patch_json"

# deployment.apps/athenz-zts-server patched

Let's check the status of the deployment:

_ns=athenz

kubectl get pods -n $_ns


# NAME                                 READY   STATUS    RESTARTS      AGE
# athenz-cli-574d747dff-7466j          1/1     Running   0             14m
# athenz-db-0                          1/1     Running   0             14m
# athenz-ui-59f7f77667-99ws6           2/2     Running   0             14m
# athenz-zms-server-568d4cfd89-tk8td   1/1     Running   0             14m
# athenz-zts-server-68686cbb54-rhxzs   0/1     Running   1 (48s ago)   109s

As expected, the pod fails the readiness check and goes into a restart loop. This happens because the ZTS server doesn't know about our mounted port-uri.json file yet, so it's still only listening on the default port 4443. We can verify this by checking the active ports inside the container:

_deploy_name=athenz-zts-server
_ns=athenz

kubectl exec -it deployment/$_deploy_name -n $_ns -- sh -c 'ss -ltn || netstat -tln'

# Defaulted container "athenz-zts-server" out of: athenz-zts-server, zms-cli (init), athenz-conf (init), athenz-plugins (init)
# sh: 1: ss: not found
# Active Internet connections (only servers)
# Proto Recv-Q Send-Q Local Address           Foreign Address         State
# tcp6       0      0 :::4443                 :::*                    LISTEN

Setup: athenz.properties

To fix the readiness problem, let's finally add a property athenz.port_uri_config to athenz.properties:

_cm_name=athenz-zts-conf
_ns=athenz
_prop_file="athenz.properties"
_new_line="athenz.port_uri_config=/opt/athenz/zts/conf/port-uri.json"

kubectl get cm $_cm_name -n $_ns -o json | \
jq --arg file "$_prop_file" --arg line "$_new_line" '.data[$file] += "\n" + $line + "\n"' | \
kubectl apply -f -

# configmap/athenz-zts-conf replaced

In k8s, you need to restart the deployment to read the new cm:

_deploy_name=athenz-zts-server
_ns=athenz

kubectl rollout restart deployment/$_deploy_name -n $_ns

# deployment.apps/athenz-zts-server restarted

Let's see if listening ports have been changed:

_deploy_name=athenz-zts-server
_ns=athenz

kubectl exec -it deployment/$_deploy_name -n $_ns -- sh -c 'ss -ltn || netstat -tln'

# Active Internet connections (only servers)
# Proto Recv-Q Send-Q Local Address           Foreign Address         State
# tcp6       0      0 :::8443                 :::*                    LISTEN
# tcp6       0      0 :::4443                 :::*                    LISTEN

Let's see the pod status as well:

_ns=athenz

kubectl get pods -n $_ns

# NAME                                 READY   STATUS    RESTARTS   AGE
# athenz-cli-574d747dff-7466j          1/1     Running   0          19m
# athenz-db-0                          1/1     Running   0          19m
# athenz-ui-59f7f77667-99ws6           2/2     Running   0          19m
# athenz-zms-server-568d4cfd89-tk8td   1/1     Running   0          19m
# athenz-zts-server-797757d5cb-dqzs8   1/1     Running   0          26s

Setup: Rollback to 4443

What happens when we roll back to the 4443 from 8443 for readiness and liveness?

_ns=athenz
_deploy=athenz-zts-server
_new_status_port=4443
_patch_json=$(cat <<EOF
[
  {
    "op": "replace",
    "path": "/spec/template/spec/containers/0/readinessProbe/exec/command",
    "value": [
      "curl",
      "-s",
      "--fail",
      "--resolve",
      "athenz-zts-server.athenz:${_new_status_port}:127.0.0.1",
      "https://athenz-zts-server.athenz:${_new_status_port}/zts/v1/status"
    ]
  },
  {
    "op": "replace",
    "path": "/spec/template/spec/containers/0/livenessProbe/exec/command",
    "value": [
      "curl",
      "-s",
      "--fail",
      "--resolve",
      "athenz-zts-server.athenz:${_new_status_port}:127.0.0.1",
      "https://athenz-zts-server.athenz:${_new_status_port}/zts/v1/status"
    ]
  }
]
EOF
)

kubectl patch deployment $_deploy -n $_ns --type='json' -p="$_patch_json"

# deployment.apps/athenz-zts-server patched

Let's see the pod status will be changed from Running:

sleep 5
_ns=athenz

kubectl get pods -n $_ns

# NAME                                 READY   STATUS    RESTARTS   AGE
# athenz-cli-574d747dff-7466j          1/1     Running   0          21m
# athenz-db-0                          1/1     Running   0          21m
# athenz-ui-59f7f77667-99ws6           2/2     Running   0          21m
# athenz-zms-server-568d4cfd89-tk8td   1/1     Running   0          21m
# athenz-zts-server-78b94c8948-884qq   1/1     Running   0          35s

Why is it still Running after rolling back to 4443? If you recall our port-uri.json configuration, the 4443 port has an empty allowed_endpoints array [], meaning it accepts all paths, including /status. This overlapping configuration enables a seamless, zero-downtime migration to the new port in production environments.

What I learned

I learned that I really enjoyed the following command as it was a bit hassle to do it manually for cluster setup & Athenz setup:

git clone https://github.com/mlajkim/dive-manifest.git manifest
make -C manifest setup

Also, sometimes I crash ZMS/ZTS servers, which is a bit hassle to delete again + redeploy again.

What's next?

I will keep writing hands-on tutorials for other PRs as well. Stay tuned!

Closing

If you enjoyed this deep dive, please leave a like & subscribe for more!

Also, leave comments if you have any questions or suggestions. Thank you in advance!

Read Athenz Patch Note v1.12.27

Jeongwoo Kim — Fri, 06 Mar 2026 20:11:35 +0000

Goal

[!TIP]
In hurry? Jump directly to Result section to see the outcome of this dive.

The goal of this dive is to read and understand the patch notes for Athenz v1.12.27.

Goal
ToC
- PR: expose on-call URL value in client-side config#3055
- Glossary
- PR summary in my own words
- PR summary from AI
- Can I test it out?
  - Setup: extend-config.js
  - Setup: ON_CALL_URL env for the athenz-ui deployment
  - Verify: On Call URL
- PR: ui - switch from zms to msd for policy creation by @ArtjomsPorss in #3034
- Glossary
- PR summary in my own words
- PR summary from AI
- Can I test it out?
- PR: feat: Add functionality to search My Domains in UI by @chandrasekhar1996 in #3058
- PR Summary in my own words
- PR Summary from AI
- Can I test it out?
- PR: fix: preserve domain contacts when updating an individual contact without page refresh by @chandrasekhar1996 in #3083
- PR Summary in my own words
- PR Summary from AI
- Can I test it out?
  - Setup: ZMS properties domain_contact_types
  - Setup: UI users_data.json
  - Setup: Patch UI deployment to read the users_data.json
  - Verify: UI Behavior
  - Verify: UI Network
  - Verify: DB Table
- PR: Use correct URL path and query param for athenz role. #3089
- PR Summary from AI
- PR Summary in my own words
- Can I test it out?
- What I learned
- PR: use metadata to specify use of default identity #3084
- Prerequisites
- PR Summary in my own words
- PR Summary from AI
- Can I test it out?: Yes, but skipped
- Future Potentials
- PR: Make ZpeUpdPolLoader ScheduledExecutorService thread daemon #3086
- Prerequisites
  - 1
  - 2
  - 3
  - 4
  - 5
- PR Summary from AI
- PR Summary in my own words
- Can I test it out?: Yes
  - Setup: Namespace athenz
  - Setup: Java code as a CM
  - Verify: v1.12.26 vs v1.12.27
- Future Potentials
- PR: make otel metric options more configurable #3090
- Prerequisites
  - 1
  - 2
  - 3
- PR Summary in my own words
- PR Summary from AI
- Breaking Changes?: Yes
- Can I test it out?: Yes, but skipped
- Future Potentials
- PR: expose openid_issuer field for access tokens in zts java client #3091
- Prerequisites
- PR Summary in my own words
- PR Summary from AI
- Breaking Changes?: No
- Can I test it out?: Yes, but skipped
- Future Potentials
- PR: Add FreeBSD support to libs/go/sia/util #3093
- Glossary
- Prerequisites
- PR Summary in my own words
- PR Summary from AI
- Breaking Changes?: No
- Can I test it out?: Yes, but skipped
- Future Potentials
- PR: expose x509/ssh key id for instance register/refresh operations #3092
- Prerequisites
- PR Summary in my own words
- PR Summary from AI
- Breaking Changes?: No
- Can I test it out?: Yes, but skipped
- Future Potentials
- PR: fix util test os filenames + new GetGroupGID impl #3094
- Prerequisites
- PR Summary in my own words
- PR Summary from AI
- Breaking Changes?: No
- Can I test it out?: Yes, but skipped
- Future Potentials
- PR: update go and java dependencies to their latest releases #3095
- Prerequisites
- PR Summary in my own words
- PR Summary from AI
- Breaking Changes?: No
- Can I test it out?: Yes, but skipped
- Future Potential
- PR: allow wildcard in first domain component of StaticWorkloadName #3096
- Glossary
- Prerequisites
- PR Summary in my own words
- PR Summary from AI
- Breaking Changes?: No
- Can I test it out?: No
- Future Potential
- What's Next?
Closing

PR: expose on-call URL value in client-side config#3055

https://github.com/AthenZ/athenz/pull/3055

Glossary

On call: Emergency calls
client-side: Web browser

PR summary in my own words

Simply speaking:

Before: We had to set the NEXT_PUBLIC_ONCALL_URL environment variable at build time.
After: We can now set onCallUrl="" in the client-side configuration or as an ENV variable. There is no need to rebuild => Simply modify the config and restart the pod.

Just like we can set the ZMS_URL for k8s-athenz-sia without needing to rebuild the image.

PR summary from AI

This pull request refactors how the on-call URL is exposed to the client-side in the UI application. It moves away from
relying on NEXT_PUBLIC environment variables, which are baked in at build time, to a more robust client-side configuration mechanism using Next.js's publicRuntimeConfig. This ensures the onCallUrl is consistently available for client-side usage without being tied to the build process.

Can I test it out?

Yes, I've done it.

Setup: `extend-config.js`

The following change allows the athenz-ui to read the ENV value ON_CALL_URL:

apiVersion: v1
data:
  extended-config.js: |
    'use strict';

    const config = {
        authProxy: {
            onCallUrl: process.env.ON_CALL_URL, << HERE
            timeZone: 'Asia/Tokyo',

Setup: `ON_CALL_URL` env for the `athenz-ui` deployment

And pass the env ON_CALL_URL inside the k8s deployment of the athenz-ui pod:

        - name: NODE_EXTRA_CA_CERTS
          value: /etc/ssl/certs/ca-certificates.crt
        - name: ON_CALL_URL
          value: https://www.google.com < HERE
        image: ghcr.io/mlajkim/athenz-ui:latest

Verify: On Call URL

Choose any athenz domain in UI (Sample URL)
Click More Details
Click add on the On Call section (The name of the team will be displayed if you already have set up)
Set whatever team member you want to test as
Click the link; You will be redirected to {ON_CALL_URL}/{whatever team member}

PR: ui - switch from zms to msd for policy creation by @ArtjomsPorss in #3034

https://github.com/AthenZ/athenz/pull/3034

Glossary

MSD: Or Micro Segmentation Daemon, is basically an API server that sets up IP policies in the ZMS Server to achieve micro-segmentation. Note that IPs change all the time in a cloud environment, so something has to handle these changes continuously.

PR summary in my own words

The UI used to handle Micro-Segmentation business logic by communicating directly with ZMS. Now, the MSD handles this for the UI instead. The UI simply calls the API, and it's done.

PR summary from AI

This pull request refactors the microsegmentation policy management workflow within the UI. It transitions the backend interaction from orchestrating multiple calls to the ZMS to directly communicating with the Microsegmentation Daemon (MSD). By utilizing the createOrUpdateTransportPolicy API, this change significantly simplifies client-side logic by removing redundant ZMS calls and delegates policy validation and IP mapping directly to the MSD service.

Can I test it out?

No. MSD is not yet open-sourced.
So only Yahoo Inc. can test this. However, I learned that I should do the following:

  featureFlag: true,
  pageFeatureFlag: {
      microsegmentation: {
          policyValidation: true,
      },
  },

This enables the MS feature on the UI. But since the PR is just a refactor, the behavior is expected to remain the same.

PR: feat: Add functionality to search My Domains in UI by @chandrasekhar1996 in #3058

https://github.com/AthenZ/athenz/pull/3058

PR Summary in my own words

A Athenz-UI-side filter (search) feature for the domain list.

PR Summary from AI

This pull request introduces a new search capability for the 'My Domains' section in the UI. It allows users to efficiently filter their list of domains using a search bar, improving navigation and usability for large domain sets. The implementation includes client-side filtering with intelligent result prioritization and robust input handling.

Can I test it out?

Yes, you can directly search on UI:

PR: fix: preserve domain contacts when updating an individual contact without page refresh by @chandrasekhar1996 in #3083

https://github.com/AthenZ/athenz/pull/3083

PR Summary in my own words

It fixes a bug that required users to refresh the page to see updated contact info.

PR Summary from AI

This pull request implements a crucial fix to prevent the loss of domain contact information when an individual contact (like a product owner or security owner) is updated without a full page refresh. By introducing a new helper method, the system now intelligently merges existing contacts with the newly updated ones, ensuring that all contact types are preserved during the update process. This enhances data integrity and provides a smoother user experience.

Can I test it out?

I was not able to test it out right away and got the following status:

✅ Was able to add the contact
✅ Was able to see the contact in the DB
❌ Failed to show the added contact in the UI
❌ Failed to reproduce the issue

Setup: ZMS properties `domain_contact_types`

The UI defaults to these two and sends these types based on that, so we need to let ZMS know about them too:

zms.properties: |
    athenz.zms.domain_contact_types=product-owner,security-owner

This will fix the following error:

Setup: UI users_data.json

Add hard-coded users to the UI for testing:

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
  name: athenz-ui-users-cm
  namespace: athenz
data:
  users_data.json: |
    [
      {
        "is_human": 1,
        "login": "__admin__",
        "gecos": "__fullname__",
        "enabled_status": 1
      },
      {
        "is_human": 1,
        "login": "jekim",
        "gecos": "Jeongwoo Kim",
        "enabled_status": 1
      },
      {
        "is_human": 1,
        "login": "athenz_admin",
        "gecos": "Athenz Admin",
        "enabled_status": 1
      },
      {
        "is_human": 1,
        "login": "test",
        "gecos": "Test User",
        "enabled_status": 1
      }
    ]
EOF

Setup: Patch UI deployment to read the users_data.json

kubectl patch deployment athenz-ui -n athenz --type='json' -p='[
  {
    "op": "add",
    "path": "/spec/template/spec/volumes/-",
    "value": {
      "name": "users-config",
      "configMap": {
        "name": "athenz-ui-users-cm"
      }
    }
  },
  {
    "op": "add",
    "path": "/spec/template/spec/containers/0/volumeMounts/-",
    "value": {
      "name": "users-config",
      "mountPath": "/home/athenz/src/config/users_data.json",
      "subPath": "users_data.json"
    }
  }
]'

Verify: UI Behavior

[!WARNING]
I was not able to reproduce the issue yet, but I was able to add the contact.

Verify: UI Network

Confirm that an API call was made to /api/v1/domain;domain=user?returnMeta=true containing the contacts field:

{
  "data": {
    "enabled": true,
    "auditEnabled": false,
    "ypmId": 0,
    "contacts": {
      "product-owner": "user.athenz_admin",
      "security-owner": "user.jekim"
    },
    "autoDeleteTenantAssumeRoleAssertions": false,
    "name": "user",
    "modified": "2026-02-09T23:15:19.456Z",
    "id": "75911440-052f-11f1-b5de-8cd6cbf5b517"
  },
  "meta": {}
}

Verify: DB Table

mariadb

SHOW DATABASES;
USE zms_server;

SHOW TABLES;
SELECT * FROM domain_contacts
WHERE domain_id = (
    SELECT domain_id
    FROM domain
    WHERE name = 'user'
);

# +-----------+----------------+-------------------+
# | domain_id | type           | name              |
# +-----------+----------------+-------------------+
# |         1 | product-owner  | user.athenz_admin |
# |         1 | security-owner | user.jekim        |
# +-----------+----------------+-------------------+

PR: Use correct URL path and query param for athenz role. #3089

https://github.com/AthenZ/athenz/pull/3089

PR Summary from AI

Fixes the documentation incorrectness

PR Summary in my own words

Uses Athenz user role members as a SSOT to control who has access to specific Azure resources,
without relying on Azure Console's user management. It simply fixes an inaccuracy in the documentation:

// principal authorized to request a given scope in the credentials).
resource ExternalCredentialsResponse POST "/external/{provider}/domain/{domainName}/creds" {
    SimpleName provider; //provider name to request credentials from

https://github.com/AthenZ/athenz/blob/master/core/zts/src/main/rdl/ExternalCredentials.rdli#L22

And NOT athenzRole but:

public static final String ZTS_EXTERNAL_ATTR_ROLE_NAME     = "athenzRoleName";

https://github.com/AthenZ/athenz/blob/4af98e4a2a338c638c44a30f0322e6ee328c953e/servers/zts/src/main/java/com/yahoo/athenz/zts/ZTSConsts.java#L248

Can I test it out?

Not efficient as I do not have Azure subscription.

What I learned

I have heard that Yahoo Inc uses the Athenz as a SSOT for login for other PFs but learned that it includes for Azure too, implemented in Jonmv/assume azure services #2634

sequenceDiagram
    autonumber
    actor User as Athenz Client (User/Service)
    participant ZTS as Athenz ZTS Server
    participant AzureARM as Azure Resource Manager
    participant AzureAD as Azure AD (Entra ID)

    Note over User, ZTS: 1. Request Token
    User->>ZTS: POST /external/azure/.../creds<br/>{Role: "azure-log-reader",<br/>IdentityName: "log-reader",<br/>ResourceGroup: "system"}

    Note over ZTS, AzureARM: 2. Lookup Identity Info (Name -> Client ID)
    ZTS->>AzureARM: Query Client ID for "log-reader" in Resource Group<br/>(Using ZTS's own Azure Identity permissions)
    AzureARM-->>ZTS: Returns Client ID (UUID)

    Note over ZTS, AzureAD: 3. Token Exchange (Federation)
    ZTS->>ZTS: Generate ID Token<br/>(iss: ZTS, sub: "coretech:role.azure-log-reader")
    ZTS->>AzureAD: Submit ID Token & Request Access Token<br/>(aud: api://AzureADTokenExchange)
    AzureAD->>AzureAD: Validate Federated Credential<br/>(Check Issuer & Subject match)
    AzureAD-->>ZTS: Issues Azure Access Token

    Note over ZTS, User: 4. Final Response
    ZTS-->>User: Returns Azure Access Token

classDiagram
    direction LR

    class AthenzDomain {
        Name: coretech
        AzureSubscription: (ID)
        AzureTenant: (ID)
    }

    class AthenzRole {
        Name: azure-log-reader
        FullARN: coretech:role.azure-log-reader
        Members: (Users/Services)
    }

    class AzureManagedIdentity {
        Name: log-reader
        ResourceGroup: system
        ClientID: (UUID)
    }

    class FederatedCredential {
        Issuer: <ZTS API URL>
        Subject: coretech:role.azure-log-reader
        Audience: api://AzureADTokenExchange
    }

    %% Relationship Definitions
    AthenzDomain "1" *-- "many" AthenzRole : contains
    AzureManagedIdentity "1" *-- "1" FederatedCredential : contains config

    AthenzRole .. FederatedCredential : 1. Mapping (Subject Match)
    FederatedCredential .. AzureManagedIdentity : 2. Grant Access (Allows usage of this ID)

    note for FederatedCredential "Key Connector:\nAzure grants access based on this credential\nwhen a specific Athenz Role requests it."

PR: use metadata to specify use of default identity #3084

https://github.com/AthenZ/athenz/pull/3084

Prerequisites

GCP Service Account Name must be at least 6 characters long
Yahoo Inc has many services that are shorter than 6 characters, like zts, zms, msd already
GCP instances allow you to specify metadata of any key-value pairs

PR Summary in my own words

Allows GCP users to set a new metadata key defaultServiceIdentity in their GCP instances. This maps to an Athenz service name that differs from the native GCP service account name, allowing them to use their short Athenz service names (like zts or zms, shorter than 6 characters).

PR Summary from AI

This pull request introduces a mechanism to use the instance's default identity for a service by specifying a defaultServiceIdentity metadata attribute. This is useful when the desired service name doesn't match the GCP service account name. The changes in attestation.go implement this logic, and new tests are added in attestation_test.go to cover the new functionality. My review focuses on improving the robustness of the new logic and correcting issues in the new tests. I've suggested handling a potential edge case with empty service names and improving error visibility in the main logic. For the tests, I've pointed out a logic bug that prevents a test from failing correctly and recommended using test-specific logging for cleaner output. Overall, the changes are good and the tests are comprehensive, but these adjustments will improve the quality and maintainability.

Can I test it out?: Yes, but skipped

Probably yes, but I would need to:

Create a GCP instance
Setup an Athenz service that is shorter than 6 characters
Setup GCP instance metadata to use the short service name with the new key defaultServiceIdentity

Which I don't think is easy to do by myself. Since I understand the concept, I think it's fine to skip it.

Future Potentials

If we ever face a situation where we need to use a short service name for any other 3rd party systems, we can consider to have a alias stored inside the instance metadata.

PR: Make ZpeUpdPolLoader ScheduledExecutorService thread daemon #3086

https://github.com/AthenZ/athenz/pull/3086

Prerequisites

1

In Java, there are two types of threads:

Non-Daemon Thread: A thread that runs in the foreground and is essential to the application's execution. It is like a head chef in a restaurant. (Which also implies you usually do not shutdown your restraunt when your head chef is still cooking (running))
Daemon Thread: A thread that runs in the background and is not essential to the application's execution. It is like a background music in a restaurant. (Which also implies you can shutdown your restraunt even when the background music is still playing)

Platform threads are designated daemon or non-daemon threads.
— Class Thread - Oracle Docs

2

Starting of Java 21 (Sep 2023~), there are two types of threads:

Platform threads: The threads that we are familiar with, which are mapped to OS threads.
Virtual threads: Lightweight threads introduced in Java 21 that are managed by the JVM and are not mapped to OS threads. Can run millions of them.

3

[!NOTE]
Please note that Daemon Thread does not initiate shutdown sequence by itself.

The JVM initiates the shutdown sequence in response to one of several events:

when the number of live non-daemon threads drops to zero for the first time

when the Runtime.exit or System.exit method is called for the first time

when some external event occurs, such as an interrupt or a signal is received from the operating system

— Shutdown Sequence - Class Runtime - Oracle Docs

Then it does for any daemon or non-daemon threads:

the registered shutdown hooks are started in some unspecified order.

Finally:

The shutdown sequence finishes when all shutdown hooks have terminated.

And note that:

one or more shutdown hooks do not terminate, for example, because of an infinite loop. In this case, the shutdown sequence will never finish.

4

Since Java's shutdown sequence can be initiated with external requests as well, it is important to make your threads to have a safe shutdown method like close() or shutdown() that will be running in the registered shutdown hooks stage, to prevent:

dead lock for your relying DB services
resource leak
etc

5

Relying users uses AuthZpeClient.allowAccess() to determine if certain requests are allowed or not. AuthZpeClient uses ZpeUpdPolLoader to load policies from ZMS. ZpeUpdPolLoader load policies by creating ScheduledExecutorService thread behind the scene (without letting the relying users know it) to load policies from ZMS periodically. ZpeUpdPolLoader loads policy over asking every time is for ZMS's over load protection & distributed access check despite of ZMS's availability.

PR Summary from AI

This pull request addresses a resource management issue where the ScheduledExecutorService in ZpeUpdPolLoader could prevent graceful application shutdown by keeping non-daemon threads alive. The change modifies the executor's thread creation to mark them as daemon threads and provides them with explicit names, ensuring they do not block the JVM exit and improving troubleshooting capabilities.

PR Summary in my own words

Make ScheduledExecutorService non-daemon thread => daemon thread so that it does not require to close() thread that is initiated by the library - which developers of the library do not know it exists.

Can I test it out?: Yes

[!TIP]
Does not require Athenz server to be up & running.

Setup: Namespace `athenz`

kubectl create ns athenz

Setup: Java code as a CM

Create a configmap with athenz zpe java client & its code that uses ZpeUpdPolLoader:

cat << 'EOF' | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
  name: athenz-thread-test-code
  namespace: athenz
data:
  pom.xml: |
    <project>
      <modelVersion>4.0.0</modelVersion>
      <groupId>demo</groupId>
      <artifactId>repro</artifactId>
      <version>1.0</version>
      <properties>
        <maven.compiler.source>17</maven.compiler.source>
        <maven.compiler.target>17</maven.compiler.target>
      </properties>
      <dependencies>
        <dependency>
          <groupId>com.yahoo.athenz</groupId>
          <artifactId>athenz-zpe-java-client</artifactId>
          <version>${athenz.version}</version>
        </dependency>
      </dependencies>
      <build>
        <plugins>
          <plugin>
            <groupId>org.codehaus.mojo</groupId>
            <artifactId>exec-maven-plugin</artifactId>
            <version>3.1.0</version>
            <configuration>
              <mainClass>demo.Repro</mainClass>
            </configuration>
          </plugin>
        </plugins>
      </build>
    </project>

  Repro.java: |
    package demo;
    import com.yahoo.athenz.zpe.ZpeUpdater;
    import java.nio.file.Files;
    import java.nio.file.Path;

    public class Repro {
      public static void main(String[] args) throws Exception {
        // Dummy environment settings to prevent errors during ZPE initialization
        System.setProperty("athenz.zpe.skip_policy_dir_check", "true");
        Path tmp = Files.createTempDirectory("athenz-zpe-pol");
        System.setProperty("athenz.zpe.policy_dir", tmp.toString());

        System.out.println("[INFO] Initializing ZpeUpdater...");
        // This line creates a scheduler thread in the background.
        ZpeUpdater zpe = new ZpeUpdater(); 

        Thread.sleep(1000); // Wait for thread creation

        Thread.getAllStackTraces().keySet().stream()
            .filter(t -> t.isAlive() && !t.isDaemon())
            .forEach(t -> System.out.println("  - " + t.getName()));

        System.out.println("[INFO] main() finishes here. No explicit close() called.");
      }
    }
EOF

Verify: v1.12.26 vs v1.12.27

You can see that the pod with Athenz v1.12.27+ shutdowns and completes successfully, while the pod with Athenz v1.12.26+ hangs there keep in Running state:

To verify it yourself, you can create a pod that runs the java code with athenz-zpe-java-client version 1.12.26:

kubectl run test-bug-1-12-26 --image=maven:3.9-eclipse-temurin-17 \
  --namespace=athenz \
  --restart=Never \
  --overrides='{
    "spec": {
      "containers": [{
        "name": "test",
        "image": "maven:3.9-eclipse-temurin-17",
        "command": ["/bin/sh", "-c"],
        "args": ["mkdir -p /app/src/main/java/demo && cp /config/pom.xml /app/ && cp /config/Repro.java /app/src/main/java/demo/ && cd /app && mvn -q compile exec:java -Dathenz.version=1.12.26"],
        "volumeMounts": [{"name": "code-vol", "mountPath": "/config"}]
      }],
      "volumes": [{"name": "code-vol", "configMap": {"name": "athenz-thread-test-code"}}]
    }
  }'

Then compare with the pod that runs the java code with athenz-zpe-java-client version 1.12.27:

kubectl run test-fix-1-12-27 --image=maven:3.9-eclipse-temurin-17 \
  --namespace=athenz \
  --restart=Never \
  --overrides='{
    "spec": {
      "containers": [{
        "name": "test",
        "image": "maven:3.9-eclipse-temurin-17",
        "command": ["/bin/sh", "-c"],
        "args": ["mkdir -p /app/src/main/java/demo && cp /config/pom.xml /app/ && cp /config/Repro.java /app/src/main/java/demo/ && cd /app && mvn -q compile exec:java -Dathenz.version=1.12.27"],
        "volumeMounts": [{"name": "code-vol", "mountPath": "/config"}]
      }],
      "volumes": [{"name": "code-vol", "configMap": {"name": "athenz-thread-test-code"}}]
    }
  }'

Future Potentials

Not for this specific improvement, but I might be able to spot similar thread-related problems in the future.

PR: make otel metric options more configurable #3090

https://github.com/AthenZ/athenz/pull/3090

Prerequisites

1

Otel offers three metrics:

counter: total amount of requests + total number of errors etc
gauge: memory usage, current live users
histogram: latency (heavy with its nature)

Unlike logs, Otel stores its metrics in memory with unique label combination, for example:

Label (Time Series)	Value
athenz_api_request_duration_seconds{api="getDomain", domain="sys.auth", method="GET", status="200"}	0.045
athenz_api_request_duration_seconds{api="getDomain", domain="my.custom.domain", method="GET", status="200"}	0.051

The problem of this label above is that production level can have many domains (some internal case has 200,000 domains), and the otel will suffer from OOM due this massive number of combination. We call it as "Cardinality Explosion".

2

What is Cardinality?

Cardinality refers to the number of unique combinations of labels (tags) attached to a metric. In time-series databases (like Prometheus or Datadog), every unique combination creates a brand new, separate data stream called a Time Series.

3

What is then "Cardinality Explosion"?

The number of time series grows exponentially by multiplying the number of possible values for each label.

Safe: Method (GET, POST = 2) × Status (200, 400, 500 = 3) 👉 Total 6 time series. (Easy to manage)
💥 Explosion: Add Domain (10,000 user domains) 👉 2 × 3 × 10,000 = Total 60,000 time series!

Why is it dangerous?

The monitoring server has to keep a separate "memory bucket" active for every single time series. A sudden explosion in cardinality will cause:

OOM (Out of Memory) Crashes: Your monitoring server (e.g., Prometheus) runs out of RAM and dies.
Slow Queries: Grafana dashboards will freeze or time out while trying to load the massive number of series.
Massive Bills: Cloud monitoring tools like Datadog charge by the number of custom metrics/series. It can lead to a huge unexpected bill.

The Golden Rule: Never use labels with unbounded or highly variable values (like User IDs, IP addresses, or highly diverse Domain names) in your metrics.

PR Summary in my own words

[!TIP]
Histogram data (latency) is heavy and expensive. If you enable the separate option, skipping the domain-specific data is highly recommended to save resources. That is why the default for skip_domain_histogram is true.

For Histogram (athenz.otel_separate_domain_histogram_metrics):

false (default): Creates a single "fat" metric containing all labels, including domain names. (High risk of cardinality explosion)
true: Separates the metric into 3 distinct metrics (Core API metric, _requestDomain, and _principalDomain).
- athenz.otel_skip_domain_histogram_metrics:
- true (default): Skips recording the two domain-specific metrics. Only the core metric (with api, method, status) is recorded. (Maximum cost efficiency, but no domain-level latency tracking)
- false: Records all 3 separated metrics. (Moderate cost efficiency, retains domain-level latency tracking)

For Counter (athenz.otel_separate_domain_counter_metrics):

false (default): Creates a single "fat" metric containing all labels, including domain names.
true: Separates the metric into 3 distinct metrics (Core API metric, _requestDomain, and _principalDomain).
- athenz.otel_skip_domain_counter_metrics:
- false (default): Records all 3 separated metrics. (Reduces core cardinality while still tracking domain-level request counts)
- true: Skips recording the two domain-specific metrics. Only the core metric is recorded. (Maximum cost efficiency, but no domain-level count tracking)

PR Summary from AI

This pull request enhances the configurability of OpenTelemetry metrics by separating the settings for histogram and counter metrics, and adding options to skip domain-specific metrics to control cardinality. The changes are logical and well-implemented, with corresponding updates to tests.

A key point to consider is that renaming the configuration property athenz.otel_separate_domain_metrics to athenz.otel_separate_domain_histogram_metrics constitutes a breaking change for users who might have the old property configured. It would be beneficial to update the pull request description to acknowledge this.

Breaking Changes?: Yes

Users using athenz.otel_separate_domain_metrics will need to update their configuration to athenz.otel_separate_domain_histogram_metrics.

Can I test it out?: Yes, but skipped

It seems like Athenz by defaut has the library, so all I have to do is to apply the config change to the cluster, and run something like:

curl localhost:8080/metrics | grep athenz_api_request_duration_seconds

But skipped for now.

Future Potentials

Once we deploy the otel support, this knowledge will be very benefitial.

PR: expose openid_issuer field for access tokens in zts java client #3091

https://github.com/AthenZ/athenz/pull/3091

Prerequisites

AWS EKS can be configured to use Athenz as OIDC Authentication Provider to authorize access to configured EKS clusters

https://athenz.github.io/athenz/oidc_aws_eks/

ZTS has openid_issuer field for AccessToken Request if you want the AccessToken with issuer field for openid

PR Summary in my own words

ZTS Java Client has not supported any parameters to request for openid_issuer=true agasint ZTS. This PR allows you to do so so that your Java application may connect to Athenz protected AWS EKS.

Also they introduced AccessTokenRequestBuilder so that can set default behavior and never pass it as a parameter.

PR Summary from AI

This pull request successfully exposes the openid_issuer field for access tokens in the ZTS Java client. The introduction of the AccessTokenRequestBuilder is a great design choice to avoid further overloading the getAccessToken method, improving the client's usability and maintainability. The changes are well-implemented, and the tests have been updated accordingly to cover the new functionality, including a new comprehensive test for PrefetchTokenScheduledItem. My review includes one suggestion to improve the maintainability of a newly added test method by refactoring it into smaller, more focused tests.

Breaking Changes?: No

No, just a feature that had been missing; not a bug but more of missing core feature.

Can I test it out?: Yes, but skipped

I think I could test getting the AccessToken with openid_issuer=true withmy own Java Code to test it, but I skipped it for now.

Future Potentials

If I ever work with Athenz <=> AWS EKS, I might be able to use it.

PR: Add FreeBSD support to libs/go/sia/util #3093

https://github.com/AthenZ/athenz/pull/3093

Glossary

What is FreeBSD?

An OS great for networking, a direct descendant of Unix

└── unix
    ├── BSD (by UC Berkly in 1970s)
    │   └── MacOS
    │   └── FreeBSD
    │           └── Netflix
    │           └── Uber
    ├── Linux (Only Influenced Though)

What is Vendor Build?

Normal Build: Every build fetches dependencies from the internet (versions may differ).
Vendor Build: Allows you to build using local ingredients. It's always the same because the local ingredients never change, making it much safer and guaranteed to work.

Prerequisites

OSS SIA allows you to vendor-build for:

darwin
linux
windows

but has not yet supported freebsd.

PR Summary in my own words

It now supports vendor builds for freebsd as well, using libs/go/sia/util/os_util_freebsd.go

PR Summary from AI

This pull request adds support for FreeBSD by introducing os_util_freebsd.go, which is a good step towards broader platform compatibility.

Breaking Changes?: No

Enhancement; you can now vendor-build on FreeBSD OS too.

Can I test it out?: Yes, but skipped

I need to prepare FreeBSD & Try to vendor-build. Skipped for now.

Future Potentials

If we want to vendor-build somewhere else, I think I can work on it with it.

PR: expose x509/ssh key id for instance register/refresh operations #3092

https://github.com/AthenZ/athenz/pull/3092

Prerequisites

[!NOTE]
The same as Description in https://github.com/AthenZ/athenz/pull/3092

You can fetch X.509 cert with custom CA for specific:

Athenz domain
Athenz service

This however requires system admin to set it.

Also note that Vespa used Yahoo Inc.'s Athenz domain and wanted to use their custom CA, so they allowed Vespa-owned domains to have custom CAs. However, adding custom CAs for a specific domain/service can only be done by system admins.

PR Summary in my own words

This PR allows a service admin (not a system admin) to test a custom or default CA. They no longer have to ask system admins to modify CA settings just for testing purposes, allowing them to try different CAs without changing the custom CA setting for the domain/service permanently.

Non-system admins can test it by inclduing new field:

x509CertSignerKeyId
sshCertSignerKeyId

It also allows a domain using a custom CA to connect to services that only accept the default CA. It does this by letting them request a cert signed by the default CA, potentially allowing a single service to operate with both a default and custom CA.

PR Summary from AI

This pull request successfully exposes x509/ssh key IDs for instance register/refresh and role certificate operations, enhancing testing capabilities for service admins. The changes across the Go and Java codebases are mostly well-implemented and consistent. I've identified a few opportunities for improvement, primarily around reducing code duplication in both Go validation logic and the new Java tests. Additionally, I've found a minor bug in one of the new tests that should be addressed.

Breaking Changes?: No

More like enhancement: allow you to submit extra field for InstanceRefreshInformation or InstanceRegisterInformation

Can I test it out?: Yes, but skipped

I think I can register a CA for a specific athenz service, and try to generate X.509 cert signed by the specific CA, but skipped for now.

Future Potentials

If one of our services spinned out, maybe yes.

PR: fix util test os filenames + new GetGroupGID impl #3094

https://github.com/AthenZ/athenz/pull/3094

Prerequisites

SIA wants to ensure that the outputted cert files have the correct GroupID. SIA builders can specify a group name (not a GroupID; defaults to athenz if unset) to allow members of that group to read the cert file. SIA reads the /etc/group file to find the GID matching the provided group name, ensuring the outputted cert file is assigned the correct GroupID.

PR Summary in my own words

Uses the standard Go package os.Read() to parse /etc/group data instead of relying on the grep command. This makes it more robust and cross-platform compatible across different OSes.

PR Summary from AI

This pull request significantly refactors the group ID retrieval mechanism within the utility package by introducing a new GetGroupGID function that directly reads and parses the /etc/group file. This change moves away from executing external grep commands, enhancing the robustness and cross-platform compatibility of the group lookup logic. Additionally, the PR includes important fixes to OS-specific test file names and expands the test suite for the updated group ID retrieval functionality, ensuring comprehensive validation of the new implementation.

Breaking Changes?: No

It simply makes it better reading the /etc/group.

Can I test it out?: Yes, but skipped

I think I can create an instance with grep removed and try to build the sia package, which is expected to fail.

And then this enhanceement will allow me to build it because it no longer uses the grep.

Future Potentials

A good reminder that using standard Go libraries is much better than depending on external OS packages.

External packages might be:

deleted
modified
or differ between OSes

It's dangerous to rely on them for important data outputs, so internal handling is always safer.

PR: update go and java dependencies to their latest releases #3095

https://github.com/AthenZ/athenz/pull/3095

Prerequisites

Athenz used to handle validation internally because its dependency package didn't enforce email validation in libs/go/athenzutils/principal.go:

// athenz always verifies that we include a valid
// email in the certificate

idx := strings.Index(emails[0], "@")
if idx == -1 {
  return "", fmt.Errorf("certificate email is invalid: %s", emails[0])
}

PR Summary in my own words

A periodic package upgrade. One of the upgrades includes net/mail's stricter parsing of email addresses, meaning we no longer need to maintain internal logic to test email validation.

While reading this PR, I realized that the internal email check might no longer be needed at all. I wasn't completely sure, so I submitted a PR to get feedback: https://github.com/AthenZ/athenz/pull/3226

PR Summary from AI

This pull request updates various Go and Java dependencies to their latest versions. The changes primarily consist of version bumps in go.mod, go.sum, and pom.xml. A key consequence of these updates is the removal of Go test cases that relied on a certificate with an invalid email format. As noted in the description, stricter parsing in newer libraries now causes these tests to fail during certificate loading, making this a reasonable and well-documented adjustment. The dependency updates are beneficial for project maintenance and security. The changes appear correct and well-justified.

Breaking Changes?: No

No breaking change, the same email invalid checking logic, simply done by different entity.

Can I test it out?: Yes, but skipped

I think I can prepare one certificate with invalid email address in it, and send it to Athenz server, but skipped for now.

Future Potential

If we ever upgrade the package to 1.25.2+ from other package, the stricker checking mechanism may bring breaking changes

PR: allow wildcard in first domain component of StaticWorkloadName #3096

https://github.com/AthenZ/athenz/pull/3096

Glossary

StaticWorkload: The opposite of a ServiceIdentity (e.g., IP Address, FQDN, VIP, LB, NAT). These are resources you don't name using an Athenz Service.

Prerequisites

Before this PR, if you want to include StaticWorkloadFQDN for the following domains, you had to add them all individually:

keep.google.com
docs.google.com
drive.google.com
mail.google.com
and more

PR Summary in my own words

This PR allows you to use * (wildcard) for the static workload's name like *.google.com to include subdomains.

Note that for stricter checking, it only allows you to use the * only once and it must be at the front of the domain name.

You can see as a sample from test to see what evaluates to true (allowed) or false (not allowed).

It also silently added a CLAUDE.md for AIs

PR Summary from AI

This pull request introduces the ability to use a wildcard character (*) in the first domain component of a StaticWorkloadName. This change is implemented across the RDL definition, Java schema, and Go client schema for the MSD (Microservice Daemon) component. Additionally, a new CLAUDE.md file has been added to provide guidance for AI assistants working with the repository.

Breaking Changes?: No

Nope.

Can I test it out?: No

MSD is not yet open sourced

Future Potential

If we ever use the MSD, I think I can think of the * usage for the StaticWorkloadFQDN

What's Next?

Reading Athenz Note definitely helped me out understanding the recent changes. I will continue doing this for later releases as well.

Closing

If you enjoyed this deep dive, please leave a like & subscribe for more!

[Post-Mortem] Deploying Athenz with Okta Sign-In: A Partial Success

Jeongwoo Kim — Sat, 21 Feb 2026 22:54:20 +0000

Goal

[!TIP]
In hurry? Jump directly to Result section to see the outcome of this dive.

[!NOTE]
Although I am organizing and writing this on 2/21 after my LF AI & Data Japan RUG presentation has successfully concluded, I believe it is crucial to leave a record of failures and roadblocks. Therefore, I am documenting this past dive retrospectively.

The goal of this dive is to integrate Okta Sign-In with the Athenz ecosystem, by achieving the following:

Login into Athenz UI with Okta SSO
Successfully run zms-cli with Okta SSO

Goal
ToC
Result
Setup
- Setup: Sign up Okta
- Setup: Okta with Work Email
- Setup: Okta Verify for Mac OS locally
- Test: Login Okta
- Setup: Okta App Integration
- Setup: Okta Authorization Server API
- Setup: Okta Policy & Rules
- Test: Token Preview
- Setup: sub as athenz user service name
- Test: sub from Token Preview
- Setup: Access Token
- Test: Access Token
- Setup: Adding ZMS properties so that ZMS can trust the okta verify
- Setup: Make sure to restart the ZMS server to get the changes!
- Setup: Set k8s secret for proxy credentials
- Setup: Create proxy container
- Setup: Remove STATIC_USER_NAME
- Setup: Modify the UI config
- Verify: Does it work?
What I learned
What's next?
Closing

Result

I was able to login into Athenz UI successfully using Okta SSO.

However, due to the complexity of Identity mapping, I couldn't achieve the following goals:

Acting normally as the mapped user user.ajkim post-login. (The system authenticates me as the full email ajkim@ajktown.com, failing authorization checks).
Logging into zms-cli directly using the Okta integration.

Setup

Setup: Sign up Okta

https://developer.okta.com/signup/

Setup: Okta with Work Email

Setup: Okta Verify for Mac OS locally

Install Okta Verify from App Store.

Test: Login Okta

Setup: Okta App Integration

[!TIP]
Official Athenz Doc: https://github.com/AthenZ/athenz/blob/master/docker/docs/IdP/Auth0.md

https://integrator-8302118-admin.okta.com/admin/apps/active

Click next with OIDC & Web Application:

Proof of possession: Makes required signed token, for now we skip

Setup: Okta Authorization Server API

https://integrator-8302118-admin.okta.com/admin/oauth2/as

Setup: Okta Policy & Rules

Create a scope athenz:

Test: Token Preview

Setup: `sub` as athenz user service name

[!NOTE]

Source code for authorized client ids not mandatory

Source code for using sub as service principal

Athenz only sees the sub field to define who you are. You can set up conversion field too but for now we can simply do this:

Test: `sub` from Token Preview

Setup: Access Token

CLIENT_ID="0oaz36xyehsYf8Cwz697"
CLIENT_SECRET="PUT_YOUR_SECRET_HERE"

curl -X POST \
  -H "Content-Type: application/x-www-form-urlencoded" \
  -u "${CLIENT_ID}:${CLIENT_SECRET}" \
  "https://integrator-8302118.okta.com/oauth2/default/v1/token" \
  -d "grant_type=client_credentials" \
  -d "scope=athenz" | jq

# {
#   "token_type": "Bearer",
#   "expires_in": 3600,
#   "access_token": "CENSORED",
#   "scope": "athenz"
# }

Test: Access Token

TOKEN="PUT_YOUR_ACCESS_TOKEN"
curl -k -H "Authorization: Bearer $TOKEN" \
  "https://localhost:4443/zms/v1/domain"

Setup: Adding ZMS properties so that ZMS can trust the okta verify

We can omit athenz.auth.oauth.jwt.authorized_client_ids_path because we setup the email address as the SSOT.

### Okta Configuration for ZMS ###

athenz.zms.authority_classes=com.yahoo.athenz.auth.impl.CertificateAuthority,com.yahoo.athenz.auth.impl.AuthorizedServiceAuthHeaderAuthority,com.yahoo.athenz.auth.oauth.OAuthCertBoundJwtAccessTokenAuthority

# Issuer:
athenz.auth.oauth.jwt.claim.aud=api://default
athenz.auth.oauth.jwt.claim.iss=https://integrator-8302118.okta.com/oauth2/default
athenz.auth.oauth.jwt.claim.scope=athenz

athenz.auth.oauth.jwt.parser.jwks_url=https://integrator-8302118.okta.com/oauth2/default/v1/keys
athenz.auth.oauth.jwt.auth0.claim_client_id=cid

athenz.auth.oauth.jwt.verify_cert_thumbprint=false
athenz.auth.oauth.jwt.cert.exclude_role_certificates=false

Setup: Make sure to restart the ZMS server to get the changes!

kubectl rollout restart deployment/athenz-zms-server -n athenz

Setup: Set k8s secret for proxy credentials

Athenz UI does not have any mechanism to handle the Oauth (completely separated) so we want to build our own proxy so that all Athenz cares is about the token stored as Cookie,

But before we do anything, we need the id and secret to represent the proxy so that proxy is trusted by Okta server. To do this correctly, we want to create a k8s secret:

printf "Enter Okta Client ID: "
read _CLIENT_ID
printf "Enter Okta Client Secret: "
read _CLIENT_SECRET
echo ""

_COOKIE_SECRET=$(openssl rand -base64 32 | tr -- '+/' '-_')
_ns="athenz"

echo "🔐 Creating Kubernetes Secret [oauth2-proxy-creds] in ns [$_ns]..."
kubectl delete secret oauth2-proxy-creds -n $_ns --ignore-not-found > /dev/null
kubectl create secret generic oauth2-proxy-creds \
  -n $_ns \
  --from-literal=client-id=${_CLIENT_ID} \
  --from-literal=client-secret=${_CLIENT_SECRET} \
  --from-literal=cookie-secret=${_COOKIE_SECRET}

Setup: Create proxy container

Copy the following container spec right under the spec.template.spec.containers section:


      - name: oauth2-proxy
        image: quay.io/oauth2-proxy/oauth2-proxy:latest
        ports:
        - containerPort: 3100
          name: auth-proxy
        env:
        - name: OAUTH2_PROXY_CLIENT_ID
          valueFrom:
            secretKeyRef:
              name: oauth2-proxy-creds
              key: client-id
        - name: OAUTH2_PROXY_CLIENT_SECRET
          valueFrom:
            secretKeyRef:
              name: oauth2-proxy-creds
              key: client-secret
        - name: OAUTH2_PROXY_COOKIE_SECRET
          valueFrom:
            secretKeyRef:
              name: oauth2-proxy-creds
              key: cookie-secret
        args:
        # destination of this container:
        - --upstream=http://127.0.0.1:3000/
        # this container's port, the same as the .ports[0].containerPort:
        - --http-address=0.0.0.0:3100
        # Okta setting:
        - --provider=oidc
        - --oidc-issuer-url=https://integrator-8302118.okta.com/oauth2/default
        - --email-domain=*
        # "Athenz-Principal-Auth" is the default (modifiable) cookie name
        - --cookie-name=Athenz-Principal-Auth
        - --cookie-secure=false
        - --pass-access-token=true
        # Let the UI read?
        - --set-xauthrequest=true
        - --pass-user-headers=true

Setup: Remove `STATIC_USER_NAME`

Remove the following line from the spec.template.spec.containers section, this is only for test and if you want to use the admin back, please do add them later once again:


        - name: STATIC_USER_NAME
          value: athenz_admin

Setup: Modify the UI config

[!NOTE]
docker/ui/conf/extended-config.js

Modify

authUserEmailHeader to X-Forwarded-Email
authUserNameHeader to X-Forwarded-Preferred-Username:

Verify: Does it work?

Please refer to the Result section above to see the verification steps and outcome.

What I learned

Not every tech dive ends with a perfectly working architecture, and that is completely fine.

One of my core principles for maintaining a Weekly Dive habit is knowing how to manage failure gracefully. If I stubbornly pushed to resolve this Okta identity mapping issue, I would have compromised the preparation time for my upcoming LF AI & Data Japan RUG presentation.

By defining this as a [Post-Mortem], I accomplished two things:

I left a clear, reproducible architecture footprint (oauth2-proxy + Okta OIDC) that I can easily pick back up later.
I maintained my documentation consistency without feeling guilty about an "incomplete" project.

What's next?

For the next 4 weeks, my Weekly Dives will be entirely dedicated to an Epic: AI Data Japan RUG Prep. Sometimes the best engineering decision you can make is knowing when to pause a task, log the exact error state, and pivot to the higher-priority deadline.

Closing

If you enjoyed this deep dive, please leave a like & subscribe for more!

I Deployed Athenz/k8s-athenz-syncer locally! (Feat. Fixing Legacy Code)

Jeongwoo Kim — Sun, 11 Jan 2026 19:38:38 +0000

Goal

[!TIP]
In hurry? Jump directly to Result section to see the outcome of this dive.

Athenz provides the following API endpoints for getting Athenz domain and its role/policy information:

/v1/domain/{domainName}/group/admin
/v1/domain/{domainName}/group/viewer

However, if your applications are running in Kubernetes, constantly querying these endpoints can be inefficient. Ideally, you want a synchronization mechanism to cache this data within the cluster. Furthermore, you might want to decouple your applications from a direct dependency on Athenz to improve resilience.

Let's be realistic—building a custom client to fetch, cache, and manage these resources within Kubernetes is a hassle. Why reinvent the wheel when you just want to consume the data?

That's why I looked into Athenz/k8s-athenz-syncer. It is an existing tool designed to sync Athenz data into Kubernetes Custom Resources (CRDs) called AthenzDomain, effectively handling the heavy lifting for us. In this post, I’ll walk through how to deploy this syncer, fix a few build issues I encountered, and explore how it can save us from writing unnecessary boilerplate code.

Goal
Result
Setup
- Setup: Working directory
- Setup: Athenz and Local Kubernetes Cluster
- Test
- Setup: Set UI for web page
- Test
- Setup: Clone k8s-athenz-syncer
- Setup: Build image
- Setup: Load image to kind cluster
- Setup: Deploy manifests
- Test
- Setup: Create a secret to represent k8s-athenz-syncer
- Setup: Create our custom deployment
- Verify: Does it work?
What I learned
What's next?
Dive Hours: 24.5 Hours
Closing

Result

I successfully deployed k8s-athenz-syncer, and the AthenzDomain CRD was registered as follows:

Next, I created a domain named home.syncer and a role can-i-see-this-role-in-crd to see if syncer can really sync the AthenzDomain CRD:

Finally, I created a Kubernetes namespace home-syncer (Note: Athenz domains replace dots . with dashes - in K8s namespaces). I verified that the syncer successfully generated the AthenzDomain resource, with its roles and policies correctly synced:

Setup

Setup: Working directory

Let's set up the working directory. Feel free to use your own, but here is an idempotent script for a quick start:

test_name=deploy_k8s_athenz_syncer
tmp_dir=$(date +%y%m%d_%H%M%S_$test_name)
mkdir -p ~/test_dive/$tmp_dir
cd ~/test_dive/$tmp_dir

Setup: Athenz and Local Kubernetes Cluster

[!WARNING]
The following script only works on macOS. Let me know in comments if you want to use other platforms.

The following script will set up a local Kubernetes cluster and install the Athenz server:

git clone https://github.com/mlajkim/dive-manifest.git manifest
make -C manifest setup

Test

Let's verify that the Athenz server is running:

kubectl get pods -n athenz

## Setup: Set UI for web page

> [!TIP]
> - To forward: `kubectl -n athenz athenz-ui deployment/athenz-ui 3000:3000 > /dev/null 2>&1 &`
> - To kill later: `pkill -f "athenz-ui"`


### Test

Open the UI in browser:

sh
open http://localhost:3000


## Setup: Clone k8s-athenz-syncer

> [!NOTE]
> Once the PR https://github.com/AthenZ/k8s-athenz-syncer/pull/45 is released, we will use `mlajkim`'s fork.

sh
git clone -b fix/deprecated-Dockerfile-images-and-CRD-definition-API https://github.com/mlajkim/k8s-athenz-syncer.git syncer


## Setup: Build image

Let's build the image locally:

sh
(cd syncer && docker build -t local/k8s-athenz-syncer:latest .)


## Setup: Load image to kind cluster

Since we are using Kind, we need to load the locally built `k8s-athenz-syncer` image into the cluster:

sh
kind load docker-image local/k8s-athenz-syncer:latest

Image: "local/k8s-athenz-syncer:latest" with ID "sha256:bb5bcf2d9c362a46444f9476791f6c9e3f81ce6abf6ebc07d3228b9b7da53fa8" not yet present on node "kind-control-plane", loading...



## Setup: Deploy manifests

> [!TIP]
> For the detailed explanation of each command, please refer to the [following](https://github.com/mlajkim/k8s-athenz-syncer/tree/fix/deprecated-Dockerfile-images-and-CRD-definition-API?tab=readme-ov-file#install)

> [!WARNING]
> We will create a custom `deployment.yaml` later, as the sample provided in the OSS repository requires configurations that are a bit complex for a quick demo.

sh
kubectl create ns kube-yahoo
kubectl apply -f ./syncer/k8s/athenzdomain.yaml
kubectl apply -f ./syncer/k8s/serviceaccount.yaml
kubectl apply -f ./syncer/k8s/clusterrole.yaml
kubectl apply -f ./syncer/k8s/clusterrolebinding.yaml


### Test

> [!TIP]
> It is okay to see `No resources found` at this stage, as the syncer managing the `AthenzDomain` is not yet deployed.

The commands above registered the `AthenzDomain CRD` (shortened to `domain`). Let's quickly check:

sh
kubectl get domain

No resources found


## Setup: Create a secret to represent `k8s-athenz-syncer`

Unlike a standard production setup which uses [CopperArgos](https://github.com/AthenZ/athenz/blob/master/docs/copper_argos.md) to auto-distribute X.509 certificates, we will simply use the root certificate for this quick demo to represent `k8s-athenz-syncer` as an Athenz service.

sh
kubectl create secret generic k8s-athenz-syncer-cert \
-n kube-yahoo \
--from-file=cert.pem=./athenz/certs/athenz_admin.cert.pem \
--from-file=key.pem=./athenz/keys/athenz_admin.private.pem \
--from-file=ca.pem=./athenz/certs/ca.cert.pem

secret/k8s-athenz-syncer-cert created


## Setup: Create our custom deployment

> [!NOTE]
> Note the following configurations:
> - We set `zms-url=https://athenz-zms-server.athenz:4443/zms/v1` because we are sharing the same Kubernetes cluster with the Athenz server
> - We set `update-cron=5s` to see the synchronization results quickly

As mentioned earlier, we are using a custom manifest that mounts the Secret we just created:

sh
cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
name: k8s-athenz-syncer
namespace: kube-yahoo
labels:
app: k8s-athenz-syncer
spec:
replicas: 1
selector:
matchLabels:
app: k8s-athenz-syncer
strategy:
rollingUpdate:
maxSurge: 50%
maxUnavailable: 0%
type: RollingUpdate
template:
metadata:
labels:
app: k8s-athenz-syncer
spec:
containers:
- name: syncer
image: local/k8s-athenz-syncer
imagePullPolicy: IfNotPresent
resources:
limits:
cpu: 1
memory: 1Gi
requests:
cpu: 1
memory: 1Gi
args:
- --zms-url=https://athenz-zms-server.athenz:4443/zms/v1
- --update-cron=5s
- --cert=/var/run/athenz/cert.pem
- --key=/var/run/athenz/key.pem
- --cacert=/var/run/athenz/ca.pem
- --exclude-namespaces=kube-system,kube-public,kube-k8s-athenz-syncer,default,local-path-storage,kube-node-lease,athenz,ajktown-api,kube-yahoo
volumeMounts:
- name: athenz-certs
mountPath: /var/run/athenz
readOnly: true
serviceAccountName: k8s-athenz-syncer
volumes:
- name: athenz-certs
secret:
secretName: k8s-athenz-syncer-cert
EOF

deployment.apps/k8s-athenz-syncer created




## Verify: Does it work?

Please refer to the [Result](#result) section above to see the verification steps and outcome.

# What I learned

Here's what I've learned:

- I discovered that when members change in a Trusted Domain (delegated role), the Modified Date and ETag of the Provider Domain do NOT change.
- I learned that `CustomResourceDefinition`'s `v1beta1` API has been deprecated
- I learned that you can enforce type in CRD

# What's next?

I will dive into jag token and how Athenz integrates with it, and what Athenz lacks so far.

# Dive Hours: 24.5 Hours

> [!NOTE]
> `aegis` that utilizes syncer's CRD and kubernetes RBAC enforcer has been stopped as they do not sync super well yet.

- `1/1 Thu`: 6.75 Hours
- `1/2 Fri`: 4.75 Hours
- `1/3 Sat`: 6.5 Hours
- `1/11 Sun`: 6.5 Hours


With the separate PRs of the following:

- https://github.com/mlajkim/aegis/pull/1
- https://github.com/mlajkim/dive-manifest/pull/1
- https://github.com/mlajkim/dive-manifest/pull/2
- https://github.com/mlajkim/dive-manifest/pull/3
- https://github.com/AthenZ/k8s-athenz-syncer/pull/45
- https://github.com/AthenZ/athenz/pull/3166

# Closing

If you enjoyed this deep dive, please leave a like & subscribe for more!

![cats_thumbs_up](https://raw.githubusercontent.com/mlajkim/dive-deep/main/weekly_dives/athenz/week_01_2026/assets/cats_thumbs_up.png)

Integrate Athenz Notification Feature with AWS SES

Jeongwoo Kim — Wed, 07 Jan 2026 08:41:13 +0000

Goal

[!TIP]
In hurry? Jump directly to Result section to see the outcome of this dive.

I’ve been diving into Athenz, an open-source RBAC/ABAC platform, running it on a local Kubernetes (Kind) cluster. Everything was working great until I needed to test the approval workflow.

I looked through official documentations and found out this "Email Notifications - Athenz", and they tell me that you can simply utilize already built AWS SES integration if you only run your Athenz server on AWS infrastructure, but I was running it locally. So, I had to figure out how to make it work on my own.

The official doc also mentioned as the following that you can build your own notification plugin, so I decided to give it a try:

This requires Athenz to be deployed on AWS. Users may use other Email Providers by following the steps to Enable Notifications using other Providers

General Architecture

This is the general architecture of how the Athenz Custom Plugin works:

Goal
General Architecture
Table of Contents
Result
Setup
- Setup: Working directory
- Setup: Athenz and Local Kubernetes Cluster
- Setup: Clone Athenz Plugin
- Setup: AWS SES Recipient Setup
- Setup: Open AWS SES Console
- Setup: Get AWS SES SMTP Credentials
- Setup: Get Smtp credentials
- Setup: Create secret for AWS SES
- Setup: Build jar and deploy plugin as configmap in Kubernetes
- Setup: Modify Athenz ZMS Server Deployment to use the Plugin and Secret
- Verify: Does it work?
What I learned
- What's Next?
Dive Hours: 9.25h
Closing

Result

I have successfully built and deployed a custom Athenz notification plugin that integrates with AWS SES. The plugin monitors specific Athenz events—such as role membership changes or domain modifications—and triggers email notifications via AWS SES.

The following GIF demonstrates the end-to-end workflow:

Filter Logs: Search for AWS to isolate logs related to the AWS SES Plugin.
Configure Recipient: Add a recipient by defining a user.<identifier> rule (e.g., adding user.jkim67cloud maps to jkim67cloud@gmail.com). (Only supports gmail.com for now)
Create Role: Create a new role that requires an approval review.
Request Membership: Attempt to add user.test as a member to the role.
Trigger Notification: The ZMS Server detects the pending request and triggers a notification to the role administrators.
Verify Email: Check the inbox to confirm the receipt of the notification email sent via AWS SES.

Setup

Setup: Working directory

test_name=email_notification_plugin
tmp_dir=$(date +%y%m%d_%H%M%S_$test_name)
mkdir -p ~/test_dive/$tmp_dir
cd ~/test_dive/$tmp_dir

Setup: Athenz and Local Kubernetes Cluster

[!TIP]
Soon we are coming with the local cluster + Athenz server setup guide! Meanwhile, please refer to the following guides

Local k8s server setup guide: https://dev.to/mlajkim/stop-using-magic-building-a-kubernetes-operator-from-scratch-mo2#a-local-kubernetes-cluster-kind
Local athenz server setup guide: https://dev.to/mlajkim/stop-using-magic-building-a-kubernetes-operator-from-scratch-mo2#b-deploy-athenz-server

Setup: Clone Athenz Plugin

git clone https://github.com/mlajkim/athenz-amazon-ses-notification-plugin.git plugin

Setup: AWS SES Recipient Setup

First, we need to set up trusted email addresses in AWS SES. AWS restricts sending emails to unverified addresses to prevent them from being flagged as spam. Therefore, we must first verify the email addresses we intend to use. For this example, we will simply use our personal email address:

Setup: Open AWS SES Console

Open Amazon SES Console's identity management page, and hit the Create identity button:

Select Email address as identity type, and input your personal email address that you want to use it as the recipient of Athenz notification emails:

You will shortly receive a verification email from AWS SES. Open the email and click the Verify email address button to complete the verification process:

Setup: Get AWS SES SMTP Credentials

For Athenz Server to connect to the public AWS SES service, we need to create SMTP credentials that Athenz server will use to authenticate itself when sending emails via AWS SES:

Setup: Get Smtp credentials

Click Create SMTP Credentials button on the SMTP Settings page:

Store the generated username and password somewhere safe, as we will need them later when creating Kubernetes secret:

Setup: Create secret for AWS SES

With the STMP Username and SMTP Password we just created, we can now create a Kubernetes secret that will store these credentials securely. The plugin repo contains a Makefile target that automates this process for us. Simply run the following command:

make -C plugin create-aws-ses-secret

Setup: Build jar and deploy plugin as configmap in Kubernetes

We will build the plugin (as RedBox) jar file and deploy it as a configmap in our local Kubernetes cluster. The plugin repo contains a Makefile target that automates this process for us. Simply run the following command:

make -C plugin deploy

Setup: Modify Athenz ZMS Server Deployment to use the Plugin and Secret

We have the following ready so far:

AWS SES SMTP Credentials stored as Kubernetes Secret
Athenz Notification Plugin stored as Kubernetes ConfigMap

Now we need to let ZMS Server know about these resources by modifying its deployment manifest. The plugin repo contains a Makefile target that automates this process for us. Simply run the following command:

make -C plugin patch

Verify: Does it work?

Please refer to the Result section above to see the verification steps and outcome.

What I learned

Through this project, I gained hands-on experience in extending a complex open-source platform and integrating it with cloud infrastructure.

Plugin Injection & Classpath: I discovered that Athenz supports custom plugin injection via the USER_CLASSPATH environment defined path variable. This allows for seamless integration of custom logic into a running ZMS server without modifying the core codebase
AWS SES Integration: Successfully implemented a real-world email notification workflow using AWS SES, gaining a deeper understanding of SMTP authentication and identity verification
Enhanced Developer UX: I focused on improving the CLI experience by adding color-coded outputs and streamlined Makefile targets, making the deployment process more intuitive
Scripting & Database Exploration: Refined my automation skills by improving hack scripts and performed direct hands-on database queries within the Athenz DB to verify internal data states.
Visual Documentation: Utilized Excalidraw to create technical diagrams, recognizing that visual aids significantly improve the readability and accessibility of complex architectures.

What's Next?

As I reflect on this dive, I plan to create a robust, "one-click" Makefile that orchestrates the entire Kubernetes cluster and Athenz server setup.

Dive Hours: 9.25h

This post took me approximately 9.25 hours of focused work and development, broken down as follows:

1/5/26: 5.25h
1/6/26: 4h

With the separate PRs of the following:

Closing

If you enjoyed this deep dive, please leave a like & subscribe for more!

Stop Using Magic: Building a Kubernetes Operator from Scratch

Jeongwoo Kim — Tue, 30 Dec 2025 09:55:38 +0000

Goal

[!TIP]
In hurry? Jump directly to Result section to see the outcome of this dive.

Hello everyone! This post's primary goal is to demystify how Kubernetes Controllers work by building a custom Athenz/k8s-athenz-syncer from scratch! Instead of relying on "magic" libraries or copying existing production code, we are diving deep into the core concepts by implementing the operator ourselves.

By doing this, we can:

Understand the core concepts of Kubernetes Operators and the Reconciliation Loop.
Learn how to interact with the Athenz ZMS server via its API programmatically.
Implement a logic that automatically maps external identity roles (Athenz) to Kubernetes RBAC.
Identify the subtle details often overlooked when simply deploying pre-made operators.
Know exactly what to look for to catch the subtle details that other reviewers would miss

Goal
Table of Contents
Result
What I Learned
Walkthrough
- Prerequisites & Setup
- a. Local Kubernetes Cluster (Kind)
- b. Deploy Athenz Server
- c. Configure Athenz Domains
- Implementation: The Hard & Clean Way
- 1. Initialize the Project
- 2. Make an operator that works from scratch
- 3. Refactor the operator to make it cleaner
- 4. Write demo/local setup guide in README.md
- 5. Make make run writes a config
- Verification: Does it actually work?
What's next?
- I: The "Weekly Dive": Performance Optimization
- II: Dissecting the Production Code (athenz/k8s-athenz-syncer)
- III: Contributing Back
Dive Hours: 28 hours
Closing

Result

I successfully built a working Kubernetes operator and you can find it here:

How to setup guide: mlajkim/k8s-athenz-syncer-the-hard-clean-way/README.md
PR: PR: k8s-athenz-syncer-the-hard-clean-way.

The operator, k8s-athenz-syncer-the-hard-clean-way, performs the following actions automatically when a Namespace <ns> is created in the cluster:

Creates a corresponding Athenz Domain (e.g., eks.users.<ns>).
Creates Athenz Roles within that domain based on a configuration.
Creates Kubernetes RBAC Roles that map to those Athenz roles.
Periodically polls Athenz to sync member changes into Kubernetes RoleBindings.

Demo 1: Auto-creation of Resources in Athenz with k8s namespace only

https://github.com/mlajkim/dive-deep/blob/main/weekly_dives/athenz/week_51_2025/assets/01_create_ns.gif

Demo 2: Giving permission with Syncing Membership (Polling)

Operator k8s-athenz-syncer-the-hard-clean-way periodically polls Athenz roles under certain parent domain (e.g., eks.users), and syncs the members of the Athenz roles into corresponding Kubernetes RBAC Roles, which results in automatic access control based on Athenz role membership.

https://github.com/mlajkim/dive-deep/blob/main/weekly_dives/athenz/week_51_2025/assets/02_polling_athenz_roles.gif

Operator k8s-athenz-syncer-the-hard-clean-way makes sure that if you delete members from Athenz roles, the members are also removed from corresponding Kubernetes RBAC Roles.

Demo 3: Restricting permission with Syncing Membership (Polling)

https://github.com/mlajkim/dive-deep/blob/main/weekly_dives/athenz/week_51_2025/assets/03_remove_athenz_role_members.gif

What I Learned

Through this "hard way" implementation, I gained several key technical insights:

The Power of Kubebuilder Scaffolding
Starting from scratch doesn't mean writing boilerplate. kubebuilder abstracts away the complexity of leader election, metrics server, and signal handling, allowing us to focus purely on the Reconcile logic.
Level-Triggered vs. Edge-Triggered
Kubernetes controllers are primarily level-triggered. While I implemented a polling mechanism for the external Athenz API, the internal Kubernetes state relies on the Reconcile loop constantly attempting to move the current state to the desired state.
I learned that for external resources (like Athenz), we explicitly need to manage the polling interval or set up webhooks, as Kubernetes cannot "watch" an external API by default.
Security Integration Details
Connecting Kubernetes RBAC with an external system isn't just about mapping strings. It involves handling X.509 certificates for authentication (Athenz) and correctly signing Kubernetes CSRs for user testing (user.mlajkim).

Walkthrough

Here is the step-by-step record of how I achieved the result.

Prerequisites & Setup

a. Local Kubernetes Cluster (Kind)

I used kind to run a cluster locally.

kind create cluster
kubectl cluster-info --context kind-kind

b. Deploy Athenz Server

I utilized @ctyano's athenz-distribution to deploy a local Athenz instance.

# Clone and deploy Athenz
git clone https://github.com/ctyano/athenz-distribution.git athenz_distribution
make -C ./athenz_distribution clean-kubernetes-athenz deploy-kubernetes-athenz

# Port forward the UI
kubectl -n athenz port-forward deployment/athenz-ui 3000:3000 &
open http://localhost:3000

c. Configure Athenz Domains

Set up the ZMS server access and create the Top Level Domain (TLD) for testing.

# Port forward ZMS
kubectl -n athenz port-forward deployment/athenz-zms-server 4443:4443 &

# Create TLD "eks"
curl -k -X POST "https://localhost:4443/zms/v1/domain" \
  --cert ./athenz_distribution/certs/athenz_admin.cert.pem \
  --key ./athenz_distribution/keys/athenz_admin.private.pem \
  -H "Content-Type: application/json" \
  -d '{"name": "eks", "adminUsers": ["user.athenz_admin"]}'

# Create Subdomain "eks.users"
curl -k -X POST "https://localhost:4443/zms/v1/subdomain/eks" \
  --cert ./athenz_distribution/certs/athenz_admin.cert.pem \
  --key ./athenz_distribution/keys/athenz_admin.private.pem \
    -H "Content-Type: application/json" \
  -d '{"parent": "eks", "name": "users", "adminUsers": ["user.athenz_admin"]}'

Implementation: The Hard & Clean Way

Here is how I built the operator from scratch in a clean way.

1. Initialize the Project

I initialized the project using kubebuilder.

repo="github.com/mlajkim"
mkdir -p k8s-athenz-syncer-the-hard-clean-way && cd k8s-athenz-syncer-the-hard-clean-way
kubebuilder init --domain "ajktown.com" --repo "$repo/k8s-athenz-syncer-the-hard-clean-way"

2. Make an operator that works from scratch

I first created an operator that works in bare minimum and deployed it public to the repository: k8s-athenz-syncer-the-hard-way

3. Refactor the operator to make it cleaner

But I found myself to improve the code structure and make it cleaner. So I created a new repository: k8s-athenz-syncer-the-hard-clean-way, with clean meaning that I re-organized the code structure to make it more modular and readable.

This is amount of changes I made in a PR to make it clean.

4. Write demo/local setup guide in README.md

I realized that visuals speak louder than words when demonstrating infrastructure tools. Instead of greeting users with a wall of text, I structured the README.md to lead with GIFs that showcase the operator's features immediately. Once I've captured the reader's interest, I provide a "copy-paste friendly" local setup guide to make the onboarding process as seamless as possible.

5. Make `make run` writes a config

To make the README.md easier to follow, I made make run write a config file config.yaml automatically with hack/ensure-config.sh, so that users don't have to manually create the config file, that they have no idea what to write in there. Also I've utilized read command in bash to make it interactive, so that users can just copy and paste the values when they run make run.

Verification: Does it actually work?

You can see the verifications from the step above named "Result"

What's next?

Now that I have a working prototype built from scratch, I want to bridge the gap between this "clean" version and a robust, production-grade operator. My roadmap for the coming weeks is as follows:

I: The "Weekly Dive": Performance Optimization

I plan to dedicate a full week (my "Weekly Dive") to deeply thinking about performance:

Optimization Strategy: instead of blindly fetching full role memberships every time, I want to investigate using HTTP headers (like ETag or Last-Modified) to implement a delta-sync mechanism.
Scaling Complexity: I am also curious about how Assumed Roles affect performance with expand=true in API and logic compared to standard domain roles.

II: Dissecting the Production Code (athenz/k8s-athenz-syncer)

After my independent study, I will deploy the official upstream Athenz/k8s-athenz-syncer locally:

Documentation: I aim to document exactly what configurations are required to run it and create easy-to-use manifests so anyone can deploy it effortlessly.
Code Audit: I will conduct a line-by-line analysis of the production code. My goal is to reverse-engineer the "why" behind their design decisions—what edge cases did they handle that I missed? Why did they choose specific architectural patterns?

III: Contributing Back

Finally, I don't just want to be an observer:

Feedback Loop: If I find performance bottlenecks or logic gaps during my audit, I plan to raise Issues or submit PRs to the upstream repository.
Community: I hope to start a conversation with the maintainers (Yahoo Inc.) to validate my assumptions and share my findings.
Guide others: If some teams want to use Athenz role as SSoT for Kubernetes RBAC, I want to help them by sharing my learnings and possibly providing a more production-ready version of my "hard & clean way" operator.

Dive Hours: 28 hours

This post took me approximately 28 hours of focused work and development, broken down as follows:

12/26 Fri: 4.5h
12/27 Sat: 5.5h
12/28 Sun: 8.5h
12/30 Tue: 9.5h

Closing

If you enjoyed this deep dive, please leave a like & subscribe for more!

Why this K8s TODO survived 3 years: An Autopsy of fieldnamedocscheck

Jeongwoo Kim — Sun, 21 Dec 2025 10:51:08 +0000

Week 50, 2025 Weekly Dive

Week 50, 2025 Weekly Dive
- Goal of this week
- Conclusion
- Where do we dive this week?
- What can I do as a first time contributor?
- What is fieldnamedocscheck?
- Where is fieldnamedocscheck used?
- Zero-brain Run verify-fieldname-docs.sh
  - Dissects each code of the 60 lines of sh
  - Dissection: Safety check
  - Dissection: KUBE_ROOT setup
    - Di-Dissection: Can you run somewhere else with this logic?
    - We can't fix the path issue
  - Dissection: go language check
  - Dissection: store every type.go
- Zero-brain Run fieldnamedocscheck
- What is that -s flag?
- Can we get a help command for that -s flag, without looking at the source code?
- Successfully run with -s!
- Dissects the core logic
- Dissection: Store all the field names in lower case
- Dissection: Understand what's inside the variables
- Check if the go file actually works
- Understand TODOs
- What's next?: Create a sample enforcing mechanism
- Returned 1402 lines of errors just for one file
- What's next?: Create a auto lint fixer
- Create a linter
- Apply the code
- Find the smallest code
- Realize that the changes are too big and go beyond expected
- Let's see if others have worked on this
- Create PR, even if it is WIP
- Fixes CLA problem
- Closes the PR
References

Goal of this week

The goal of this week is to make my first contribution to kubernetes project.

Conclusion

I have successfully made a PR here: feat(tools): support incremental enforcement of backticks in check-missing-backticks in field_name_docs_check.go

Where do we dive this week?

I want to be a part of Contributor in kubernetes

And eventually have the @kubernetes handle in GitHub:

What can I do as a first time contributor?

So I've git cloned the kubernetes repository without thinking too much, but still have no idea what to fix.

So I asked AI inside the project, what kind of things I can fix as a first time contributor (basically easy stuff)

And the AI has found this todo inside fieldnamedocscheck:

// TODO: a manual pass adding back-ticks to the doc strings, then update the linter to
// TODO: check the existence of back-ticks

First of all I have no idea what this field_name_docs_check.go does so ... let's just dive in without knowing anything :)

What is fieldnamedocscheck?

It seems like it checks if the fields in the types are properly documented or not.

Where is fieldnamedocscheck used?

So what is fieldnamedocscheck? Let see where this cmd fieldnamedocscheck is used:

Zero-brain Run `verify-fieldname-docs.sh`

[!NOTE]
You may be asked to run brew install bash if your bash version is insufficient

This script has checked 64 lines of output:

./hack/verify-fieldname-docs.sh
# Checking ./staging/src/k8s.io/kube-aggregator/pkg/apis/apiregistration/v1
# Checking ./staging/src/k8s.io/kube-aggregator/pkg/apis/apiregistration/v1beta1
# Checking ./staging/src/k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1
# Checking ./staging/src/k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1beta1
# Checking ./staging/src/k8s.io/api/rbac/v1
# Checking ./staging/src/k8s.io/api/rbac/v1beta1
# Checking ./staging/src/k8s.io/api/rbac/v1alpha1
# Checking ./staging/src/k8s.io/api/apiserverinternal/v1alpha1
# ...

Dissects each code of the 60 lines of sh

Let's open the verify-fieldname-docs.sh file and dissect each code.

Dissection: Safety check

set -o errexit
set -o nounset
set -o pipefail

errexit : Exit immediately if a command exits with a non-zero status
nounset : Treat unset variables as an error when substituting
pipefail: Prevent errors in a pipeline from being masked

Dissection: KUBE_ROOT setup

[!TIP]
Once you echo $KUBE_ROOT, you will get ./hack/.. as a sample.
That .. at the end

KUBE_ROOT=$(dirname "${BASH_SOURCE[0]}")/.. # i.e) ./hack/.. or ../../..
source "${KUBE_ROOT}/hack/lib/init.sh"
source "${KUBE_ROOT}/hack/lib/util.sh"

Di-Dissection: Can you run somewhere else with this logic?

In conclusion, we need to run the verify-fieldname-docs.sh from the base directory of kubernetes repository.

./oss_workspace/oss_kubernetes/hack/verify-fieldname-docs.sh
# KUBE_ROOT=./oss_workspace/oss_kubernetes/hack/..
# go: go.mod file not found in current directory or any parent directory; see 'go help modules'

../../verify-fieldname-docs.sh
# KUBE_ROOT=../../..
# stat ~/oss_workspace/oss_kubernetes/hack/hello/can/cmd/fieldnamedocscheck: directory not found

./verify-fieldname-docs.sh
# KUBE_ROOT=./..
# stat ~/oss_workspace/oss_kubernetes/hack/cmd/fieldnamedocscheck: directory not found

We can't fix the path issue

Since this chunk of code is used, I do not think we can modify this:

Dissection: go language check

Checks if golang sufficient environment is set

kube::golang::setup_env

Dissection: store every type.go

Store every type.go files inside versioned_api_files variable:

versioned_api_files=$(find_files) || true

Zero-brain Run fieldnamedocscheck

[!NOTE]
It is important to give a shot even if you don't know what you are doing :)

cd ./cmd/fieldnamedocscheck
go run field_name_docs_check.go
# F1211 11:49:45.848446   80482 field_name_docs_check.go:33] Please define -s flag as it is the api type file
# exit status 255

What is that `-s` flag?

Not officially, probably, it is a type source file, it seems:

typeSrc = flag.StringP("type-src", "s", "", "From where we are going to read the types")

if *typeSrc == "" {
  klog.Fatalf("Please define -s flag as it is the api type file")
}

Can we get a help command for that `-s` flag, without looking at the source code?

go run field_name_docs_check.go --help
Usage of ~/Library/Caches/go-build/81/..hash../-d/field_name_docs_check:
  -s, --type-src string   From where we are going to read the types

Successfully run with `-s`!

[!NOTE]
If nothing happens, it means all the fields are properly documented :)

go run field_name_docs_check.go -s ../../staging/src/k8s.io/api/core/v1/types.go

Dissects the core logic

Dissection: Store all the field names in lower case

if p.Name != "" {
  typesMap[strings.ToLower(p.Name)] = p
}

// Sample keys:
// key: ipfamilypolicy  key: trafficdistribution  key: externalips ...

To find all the fields, including right and wrong ones, we store all the field names in lower case inside typesMap.

Dissection: Understand what's inside the variables

TCPSocket *TCPSocketAction `json:"tcpSocket,omitempty" protobuf:"bytes,3,opt,name=tcpSocket"`

Struct: TCPSocketAction
p.Name:  tcpSocket
typesMap: tcpsocket

Check if the go file actually works

It seems like it checks only when it has the back-tick quoted field names in the doc string.

So I've set up TCPSocket => TcPSocket, and created a new sentence // TcPSocket is deprecated and...

// LifecycleHandler defines a specific action that should be taken in a lifecycle
// hook. One and only one of the fields, except `TcPSocket` must be specified.
// `TcPSocket` is deprecated and not supported as a LifecycleHandler.
type LifecycleHandler struct {
  // Deprecated. `TcPSocket` is NOT supported as a LifecycleHandler and kept
  // for backward compatibility. There is no validation of this field and
  // lifecycle hooks will fail at runtime when it is specified.
  // +optional
  TCPSocket *TCPSocketAction `json:"tcpSocket,omitempty" protobuf:"bytes,3,opt,name=tcpSocket"`
}

And got two errors (Note that even if there are three places of TcPSocket, it only shows two errors because the second rule skips the already found mismatched names):

go run field_name_docs_check.go -s ../../staging/src/k8s.io/api/core/v1/types.go
# Error: doc for LifecycleHandler contains: TcPSocket, which should be: tcpSocket
# Error: doc for LifecycleHandler.tcpSocket contains: TcPSocket, which should be: tcpSocket
# exit status 1

Understand TODOs

// The rule is:
// 1. Get all back-tick quoted names in the doc
// 2. Skip the name which is already found mismatched.
// 3. Skip the name whose lowercase is different from the lowercase of tag names,
//    because some docs use back-tick to quote field value or nil
// 4. Check if the name is different from its tag name

// TODO: a manual pass adding back-ticks to the doc strings, then update the linter to
// TODO: check the existence of back-ticks

Right now, setting the back-ticks is not mandatory, but the TODO suggests that we should add back-ticks to the doc strings, and then update the linter to check the existence of back-ticks.

What's next?: Create a sample enforcing mechanism

I created the following enforcing mechanism to make sure all the field names are properly documented with back-ticks:

package main

import (
    "fmt"
    "os"
    "regexp"
    "strings"

    flag "github.com/spf13/pflag"
    kruntime "k8s.io/apimachinery/pkg/runtime"
    "k8s.io/apimachinery/pkg/util/sets"
    "k8s.io/klog/v2"
)

var (
    typeSrc    = flag.StringP("type-src", "s", "", "From where we are going to read the types")
    re         = regexp.MustCompile("`(\\b\\w+\\b)`")
    reAllWords = regexp.MustCompile("\\b\\w+\\b")
)

// kubeTypesMap is a map from field name to its tag name and doc.
type kubeTypesMap map[string]kruntime.Pair

func main() {
    flag.Parse()

    if *typeSrc == "" {
        klog.Fatalf("Please define -s flag as it is the api type file")
    }

    docsForTypes := kruntime.ParseDocumentationFrom(*typeSrc)
    rc := false

    for _, ks := range docsForTypes {
        typesMap := make(kubeTypesMap)

        for _, p := range ks[1:] {
            // skip the field with no tag name
            if p.Name != "" {
                typesMap[strings.ToLower(p.Name)] = p
            }
        }

        structName := ks[0].Name

        rc = checkFieldNameAndDoc(structName, "", ks[0].Doc, typesMap) || rc
        for _, p := range ks[1:] {
            rc = checkFieldNameAndDoc(structName, p.Name, p.Doc, typesMap) || rc
        }
    }

    if rc {
        os.Exit(1)
    }
}

func checkFieldNameAndDoc(structName, fieldName, doc string, typesMap kubeTypesMap) bool {
    rc := false
    visited := sets.Set[string]{}

    // The rule is:
    // 1. Get all back-tick quoted names in the doc
    // 2. Skip the name which is already found mismatched.
    // 3. Skip the name whose lowercase is different from the lowercase of tag names,
    //    because some docs use back-tick to quote field value or nil
    // 4. Check if the name is different from its tag name
    // 5. Check if there are any unquoted field names in the doc

    nameGroups := re.FindAllStringSubmatch(doc, -1)
    for _, nameGroup := range nameGroups {
        name := nameGroup[1]
        if visited.Has(name) {
            continue
        }
        if p, ok := typesMap[strings.ToLower(name)]; ok && p.Name != name {
            rc = true
            visited.Insert(name)

            fmt.Fprintf(os.Stderr, "Error (Case Mismatch): doc for %s", structName)
            if fieldName != "" {
                fmt.Fprintf(os.Stderr, ".%s", fieldName)
            }

            fmt.Fprintf(os.Stderr, " contains: %s, which should be: %s\n", name, p.Name)
        }
    }

    wordIndices := reAllWords.FindAllStringIndex(doc, -1)

    for _, loc := range wordIndices {
        start, end := loc[0], loc[1]
        word := doc[start:end]

        p, isField := typesMap[strings.ToLower(word)]
        if !isField {
            continue
        }

        if visited.Has(word) {
            continue
        }

        hasBacktick := false
        if start > 0 && end < len(doc) {
            if doc[start-1] == '`' && doc[end] == '`' {
                hasBacktick = true
            }
        }

        if !hasBacktick {
            rc = true
            visited.Insert(word)

            fmt.Fprintf(os.Stderr, "Error (Missing Backticks): doc for %s", structName)
            if fieldName != "" {
                fmt.Fprintf(os.Stderr, ".%s", fieldName)
            }
            fmt.Fprintf(os.Stderr, " mentions field %s without backticks. Should be: `%s`\n", word, p.Name)
        }
    }

    return rc
}

Returned 1402 lines of errors just for one file

I got 1,402 lines of errors:

go run field_name_docs_check.go -s ../../staging/src/k8s.io/api/core/v1/types.go
# Error (Missing Backticks): doc for AWSElasticBlockStoreVolumeSource.volumeID mentions field volumeID without backticks. Should be: `volumeID`
# Error (Missing Backticks): doc for AWSElasticBlockStoreVolumeSource.fsType mentions field fsType without backticks. Should be: `fsType`
# Error (Missing Backticks): doc for AWSElasticBlockStoreVolumeSource.partition mentions field partition without backticks. Should be: `partition`

What's next?: Create a auto lint fixer

There is no way that I manually fix 1,402 lines of errors, and this applies for only 1 file.

Also, even after I fix this somehow, if someone else adds a new field without back-ticks, BEFORE I enforce the back-ticks check, the problem will happen again.

Therefore, I think it is better, if I make a mechanism that enforces one page first, and then eventually scope out to the entire repository.

Also there are some words like, name, type, kind, etc, that are used very often in the doc strings, but they are ALSO field names. That case, we need to have some kind of AI mechanism to figure out whether it is a field name or just a normal word.

Create a linter

I simply used AI for the linter. I won't put source code because it was not hard to generate.

Apply the code

go run field_name_docs_check_lint.go -s ../../staging/src/k8s.io/api/core/v1/types.go
# ✅ Successfully fixed backticks in: ../../staging/src/k8s.io/api/core/v1/types.go

Find the smallest code

I do not want to scare them but also want them to merge the PR & somewhat has dependency in my changes later in the future.

So maybe enforcing one file first with the smallest lines of code is a good idea.

find staging/src/k8s.io/api -name "types.go" | xargs wc -l | sort -n | head -n 5
# 48 staging/src/k8s.io/api/authentication/v1alpha1/types.go
# 75 staging/src/k8s.io/api/scheduling/v1/types.go
# 82 staging/src/k8s.io/api/scheduling/v1beta1/types.go
# 83 staging/src/k8s.io/api/imagepolicy/v1alpha1/types.go

Realize that the changes are too big and go beyond expected

// `azureDisk` represents an Azure Data Disk mount on the host and bind mount to the pod.
// Deprecated: `azureDisk` is deprecated. All operations for the in-tree `azureDisk` type
// are redirected to the disk.`csi`.azure.com `csi` driver.

The problem of the changes above:

The link is corrupted because of the back-ticks: disk.csi.azure.com => should be disk.csi.azure.com
Not sure if we can start with azureDisk rather than AzureDisk for the first word
- It seems like the preview mode works fine, but not sure if the actual kubernetes doc generator works fine with this.

Let's see if others have worked on this

I was not able to find one

Create PR, even if it is WIP

It won't build but who cares. Do the zero-brain PR: https://github.com/kubernetes/kubernetes/pull/135727

I see you Kubernetes Contributor badge :)

Fixes CLA problem

I got this warning in my PR:

Did the sign here:

Now it says:

Closes the PR

My goal was to create a PR out there and has definitely experienced the process of contribution.

But I realize that why such a TODO has been out there for 3 years without being fixed; The fix required too many changes that goes beyond the expected range like:

url corruption
handle general terms like port, name, type etc that are used very often in the doc strings
passes the k8s native rules for doc generation

So I close the PR:

References

NONE

How 'cd' actually works in Linux Kernel: A Source Code Analysis

Jeongwoo Kim — Sun, 30 Nov 2025 03:07:18 +0000

Goal

[!NOTE]
This dive took 6.5 hours

The goal of this document is to dive deep into the following commands:

cd /
cd -
cd
cd ..

Target Audience

This document is intended for users who have a basic understanding of Linux command line operations and wish to deepen their knowledge of directory navigation commands, with simple copy-paste examples.

Goal
Target Audience
TOC
Backgrounds
Dive in
- How to check source code of cd command?
- How do we know which shell we are using?
- Let's quickly check the version
- Dive into the source code of cd's bin_cd command in zsh
- Restricted shell check
- cd with chasing symbolic links
  - setopt
  - How to check the options set in your shell?
  - How to check what options are not yet enabled in your shell?
- Signal handling for control the signals during cd execution
- zpushnode(dirstack, ztrdup(pwd));
- Dive into cd_get_dest()
  - if (!argv[0])
- Dive into cd_new_pwd()
Others
- First time seeing 25 years old commit

Backgrounds

If you have had a chance to work with Linux, the commands above are pretty common for navigating through directories. You may already know what they do, but let's dive deep into each of them to understand their behaviors.

cd /: Navigates to the Root directory
cd ..: Navigates to the Parent directory
cd: Navigates to the Home directory (equivalent to cd ~)
cd -: Navigates to the Previous directory

While the first three are straightforward, cd - behaves like a "Back" button. But how does the shell remember where I was?

Dive in

How to check source code of `cd` command?

type cd
# cd is a shell builtin

shell builtin means that the command cd is implemented directly within the shell (like bash, zsh, etc.)

How do we know which shell we are using?

echo $SHELL
# /bin/zsh

OR you may:

echo $0
# -zsh

Let's quickly check the version

/bin/zsh --version
# or: `zsh --version`
# zsh 5.9 (arm64-apple-darwin24.0)

Dive into the source code of `cd`'s `bin_cd` command in `zsh`

[!INFO]
The following is a mirror link of the official repository of zsh

https://github.com/zsh-users/zsh

The core functions are the following:

cd_get_dest(): does the initial argument processing
cd_do_chdir(): actually changes directory, if possible
cd_new_pwd(): does the ancillary processing associated with actually changing directory

Restricted shell check

if (isset(RESTRICTED)) {
    zwarnnam(nam, "restricted");
    return 1;
}

[!INFO]
zwarnnam is a compound word for: z(ZSH) + warn (WARNING) + nam (NAME)
A function for error output with consistent formatting.

This code checks if the shell is running in restricted mode by using the isset(RESTRICTED) function. If it is, it issues a warning message "restricted" using the zwarnnam function and returns 1, indicating an error. In restricted mode, certain commands, including changing directories - cd -, may be limited to enhance security.

Will return the following output when you try to use cd in restricted mode:

Prepare restricted shell:

zsh -r
# shell_session_history_enable:1: /usr/bin/touch: restricted
# shell_session_history_enable:2: HISTFILE: restricted

# cd ..
zsh: cd: restricted

cd with chasing symbolic links

chasinglinks = OPT_ISSET(ops,'P') || (isset(CHASELINKS) && !OPT_ISSET(ops,'L'));

chasinglinks is a boolean value, meaning is_chasing_symbolic_links or not.
It is true if :

If user specifies -P option
If user has a preference setting CHASELINKS enabled
- You can temporarily override this setting by specifying -L option

Setup:

tmp_dir=$(date +%y%m%d_%H%M%S_test)
mkdir $tmp_dir && cd $tmp_dir
mkdir physical_path
ln -s physical_path symbolic_link

Check Setup:

ls -al
# total 0
# drwxr-xr-x   4 ajk  staff   128 Nov 29 13:58 .
# drwxr-x---+ 73 ajk  staff  2336 Nov 29 13:54 ..
# drwxr-xr-x   3 ajk  staff    96 Nov 29 13:58 physical_path
# lrwxr-xr-x   1 ajk  staff    13 Nov 29 13:58 symbolic_link -> physical_path

Test yourself:

cd symbolic_link
pwd
# /path/to/your/tmp_dir/symbolic_link

Test with -P option:

cd ..
cd -P symbolic_link
pwd
# /path/to/your/tmp_dir/physical_path

Test with CHASE_LINKS enabled:

cd ..
setopt CHASE_LINKS
cd symbolic_link
pwd
# /path/to/your/tmp_dir/physical_path

Override the CHASE_LINKS with -L option:

cd ..
cd -L symbolic_link
pwd
# /path/to/your/tmp_dir/symbolic_link

setopt

the command setopt is in Src/options.c

How to check the options set in your shell?

/* With no arguments or options, display options. */
if (!*args) {
    scanhashtable(optiontab, 1, 0, OPT_ALIAS, optiontab->printnode, !isun);
    return 0;
}

setopt
# chaselinks
# combiningchars
# interactive
# ...

How to check what options are not yet enabled in your shell?

unsetopt
# noaliases
# aliasfuncdef
# allexport
# ...

Signal handling for control the signals during `cd` execution

queue_signals();

Make sure that other signals (like interrupts) do not interfere with the execution of the cd command. This is important because changing directories is a critical operation, and we want to ensure it completes without interruption.

zpushnode(dirstack, ztrdup(pwd));

pwd contains the current working directory before changing it.

You can check dirstack by:

dirs -v
0   ~/Workspaces
1   ~/Workspaces/oss

Dive into `cd_get_dest()`

cd_get_dest returns the destination directory based on the arguments provided to the cd command. It handles various cases, such as:

No arguments: returns the home directory
-: returns the previous directory from the directory stack
Specific path: returns the specified path

It returns NULL if there is an error, such as when the previous directory is not set.

Also it is shared by the following commands (Please note that they had to do this for performance/size optimization back in the days when memory and storage were more limited):

cd
pushd
popd

if (!argv[0])

If argument is empty:

[!INFO]
popd stands for "pop directory" and is used to remove the top entry from the directory stack and change the current working directory to that entry.

if (func == BIN_POPD && !nextnode(firstnode(dirstack))) {
  zwarnnam(nam, "directory stack empty");
  return NULL;
}

You can manipulate this error log when your directory stack is empty:

# dirs -v
# 0 ~/Workspaces/directory
popd
# popd: directory stack empty

strcmp(argv[0], "-")

This part checks if the first argument to the cd command is a hyphen (-). If it is, the function retrieves the previous directory from the directory stack. If the stack is empty, it issues a warning and returns NULL.

Dive into cd_new_pwd()

Basically write the new working directory to the PWD environment variable and update the directory stack accordingly.

Forem: Jeongwoo Kim

[Hands-on] Kubernetes Pod Certificate Request introduced in v1.35

Goal

Table of Contents

Walkthrough

Setup: Working Directory

Setup: Kind Cluster with Cert Provisioning Enabled

Setup: Mash Controller Deployed

Verify: Auto Distributed Certificate Feature on Sample Deployment

What's next?

Closing

[Hands-on] Testing Separate Ports in Athenz ZTS

[Hands-on] Testing Separate Ports in Athenz ZTS

Result

Table of Contents

Steps to Reproduce

Setup: Working Directory & Athenz Server

Setup: port-uri configuration

Setup: Volume Injection

Setup: port open in ZTS

Setup: Readiness/Liveness watching the port 8443

Setup: athenz.properties

Setup: Rollback to 4443

What I learned

What's next?

Closing

Read Athenz Patch Note v1.12.27

Goal

ToC

PR: expose on-call URL value in client-side config#3055

Glossary

PR summary in my own words

PR summary from AI

Can I test it out?

Setup: extend-config.js

Setup: ON_CALL_URL env for the athenz-ui deployment

Verify: On Call URL

PR: ui - switch from zms to msd for policy creation by @ArtjomsPorss in #3034

Glossary

PR summary in my own words

PR summary from AI

Can I test it out?

PR: feat: Add functionality to search My Domains in UI by @chandrasekhar1996 in #3058

PR Summary in my own words

PR Summary from AI

Can I test it out?

PR: fix: preserve domain contacts when updating an individual contact without page refresh by @chandrasekhar1996 in #3083

PR Summary in my own words

PR Summary from AI

Can I test it out?

Setup: ZMS properties domain_contact_types

Setup: UI users_data.json

Setup: Patch UI deployment to read the users_data.json

Verify: UI Behavior

Verify: UI Network

Verify: DB Table

PR: Use correct URL path and query param for athenz role. #3089

PR Summary from AI

PR Summary in my own words

Can I test it out?

What I learned

PR: use metadata to specify use of default identity #3084

Prerequisites

PR Summary in my own words

PR Summary from AI

Can I test it out?: Yes, but skipped

Future Potentials

PR: Make ZpeUpdPolLoader ScheduledExecutorService thread daemon #3086

Prerequisites

1

2

3

4

5

PR Summary from AI

PR Summary in my own words

Can I test it out?: Yes

Setup: Namespace athenz

Setup: Java code as a CM

Verify: v1.12.26 vs v1.12.27

Setup: `extend-config.js`

Setup: `ON_CALL_URL` env for the `athenz-ui` deployment

Setup: ZMS properties `domain_contact_types`

Setup: Namespace `athenz`