DEV Community: John Ajera

Argo CD and AWS CodeConnections: The Upside, the Redeploy Pain, and How I Fixed It

John Ajera — Sat, 28 Mar 2026 10:46:16 +0000

Argo CD and AWS CodeConnections: The Upside, the Redeploy Pain, and How I Fixed It

I run Argo CD on Amazon EKS using the managed Argo CD capability and AWS CodeConnections for Git. CodeConnections has been a clear win for day-to-day operations. Then I had to recreate the connection (new resource, new identity in the URL). Every Application went to Sync: Unknown until I updated URLs in two places—Git and the live cluster—and fixed ApplicationSets so they stopped writing the old URL back. This article leads with why I still choose CodeConnections, then what breaks on redeploy, then what I did when it inevitably happened, in that order.

1. Why CodeConnections is worth it for Argo CD on EKS

No SSH keys or personal tokens in the cluster. Argo pulls Git using IAM: the capability role is allowed to use the connection (UseConnection, GetConnection). You are not copying PATs into Secrets or rotating leaked keys because someone printed kubectl get secret.

One connection, many repos. The HTTPS URL includes your account, region, a connection UUID, and then owner/repo. Same connection, different path segment for each repository. Setup details and Terraform patterns are in Argo CD on EKS: Git repo access with AWS CodeConnections and Terraform.

Fits how enterprises already govern access. Connections are AWS resources; approval and auditing live next to the rest of your cloud controls.

That does not mean zero tradeoffs. Managed Argo CD often applies one Git credential broadly (for example every github.com fetch through Kustomize can inherit it). If that bites you, vendoring or URL strategy fixes it—see Why Your Kustomize Remote Bases Break on Managed Argo CD (and How to Fix It). The tradeoff this article focuses on is different: replacing the connection.

2. What actually hurts when you update or redeploy the connection

The Git URL Argo uses is not abstract. It embeds a connection UUID in the path. A new connection is a new UUID. Anything that still points at the old path keeps asking AWS to authorize the wrong resource, so repo fetch fails and sync never reconciles.

Pain point one — Git only is not enough. Your GitOps repo is the source of truth, but Kubernetes already has Application and ApplicationSet objects applied yesterday. Their spec.source.repoURL (and generator URLs on sets) stay on the old string until something updates them. Pushing Git does not retroactively patch those CRs.

Pain point two — ApplicationSets fight you. Many sets declare repoURL twice: on the git generator and again on the template. If you patch child Applications but leave the set on the old URL, the controller reconciles and puts the old URL back on the apps.

Pain point three — Terraform layout. If CodeConnections and the EKS cluster share one tangled module and state, recreating the connection can feel like you are planning half the platform when you only wanted a new Git pipe. I now prefer the connection (and its IAM attachment) in something standalone, with outputs the cluster stack consumes—so the next rotation is a smaller blast radius.

What this is not. If Argo shows forbidden listing some API group (for example heavy CRD surfaces from controllers like ACK), that is usually cluster RBAC / EKS access policies for the Argo identity, not the CodeConnections URL. Fix that on its own; do not confuse it with a UUID swap.

Do not mass-delete Applications to “fix” a bad URL. Workloads may still be fine; you risk prune tearing down real resources. Fix the URLs.

3. Prerequisites

kubectl against the cluster, with permission to read and patch applications and applicationsets in the Argo CD namespace (below I use argocd, the usual default)
Rights to commit and push every Git repo that hardcodes the CodeConnections URL
The new clone URL or at least the new UUID from the AWS Console or Terraform output

4. If it happens to you: what worked for me

These steps assume you already know the old and new connection UUID (search your shell history, Terraform state, or an old Application YAML). The host pattern is documented in the CodeConnections and EKS guides linked at the end.

Step A — Fix Git first, everywhere

Search across all repos that participate in GitOps—not only the “main” infra repo:

rg 'codeconnections\.' --glob '*.yaml' --glob '*.yml' --glob '*.tf' --glob '*.md'

Update bootstrap Application manifests, every ApplicationSet generator and template repoURL, any child Application checked in with a literal URL, and docs or scripts that build repository secrets. Commit and push.

Step B — Patch Applications in the cluster

Still in a broken state, the cluster cannot always sync from Git, so you patch live objects once. Set shell variables to your real values (namespace if not argocd, old UUID, new UUID):

NS=argocd
OLD_UUID=aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee
NEW_UUID=ffffffff-eeee-dddd-cccc-bbbbbbbbbbbb

List apps whose single spec.source.repoURL still contains the old UUID:

kubectl get applications -n "$NS" -o json | jq -r --arg u "$OLD_UUID" \
  '.items[] | select(.spec.source.repoURL // "" | contains($u)) | .metadata.name'

For each name, the safest approach is to take the existing URL from the object and swap the UUID in the shell, then merge-patch:

app=my-app
oldurl=$(kubectl get application "$app" -n "$NS" -o jsonpath='{.spec.source.repoURL}')
newurl="${oldurl//$OLD_UUID/$NEW_UUID}"
kubectl patch application "$app" -n "$NS" --type merge -p "{\"spec\":{\"source\":{\"repoURL\":\"$newurl\"}}}"

If you use spec.sources (multi-source), repeat the idea per entry that points at CodeConnections.

Step C — Patch ApplicationSets (do not skip this)

Confirm the old UUID still appears on spec (ignore status for a moment):

kubectl get applicationsets -n "$NS" -o yaml | rg "$OLD_UUID"

To replace the UUID everywhere under each set’s JSON (review before you run this in production—I exported one set first and eyeballed it):

for name in $(kubectl get applicationsets -n "$NS" -o json | jq -r --arg u "$OLD_UUID" \
  '.items[] | select(.. | strings? | contains($u)) | .metadata.name' | sort -u); do
  kubectl get applicationset "$name" -n "$NS" -o json | \
    jq --arg o "$OLD_UUID" --arg n "$NEW_UUID" \
    'del(.status) | walk(if type == "string" then gsub($o; $n) else . end)' | \
    kubectl replace -f -
done

The walk replaces every occurrence of the old UUID string in that JSON. Pick a UUID substring unique to the connection so you do not hit unrelated fields.

Step D — Refresh and converge

Hard refresh apps in the UI or CLI, then let GitOps catch up. If Git still cannot be fetched until one app is fixed, patch your bootstrap Application (the one that applies the rest of the tree) to the new URL first, then repeat the rest—classic chicken-and-egg.

5. Troubleshooting

Application stuck deleting

kubectl patch application my-app -n argocd -p '{"metadata":{"finalizers":null}}' --type=merge

I deleted an app and it came back

An ApplicationSet owns it (ownerReferences). Fix the set and Git; deleting the app alone will not stick.

6. References

Host a Static Site on EC2 with Terraform (VPC, Optional ALB)

John Ajera — Sat, 28 Mar 2026 08:54:20 +0000

Host a Static Site on EC2 with Terraform (VPC, Optional ALB, Session Manager)

For static sites, S3 + CloudFront is usually the better default. This post points at a small Terraform demo and pulls a few excerpts from main.tf, variables.tf, iam.tf, and user_data.tftpl. Full layout: tf-aws-ec2-static-demo (local path ~/workspace/jdevto/tf-aws-ec2-static-demo if you keep it beside this blog repo). S3 + CloudFront with Terraform: art0018.

Overview

The demo provisions a VPC, nginx on Amazon Linux 2023, index.html (AZ + private IP from IMDSv2), and robots.txt. use_alb=false (default): one instance in one public subnet; clients hit :80 on the instance public IP (CIDR from allowed_http_cidr). use_alb=true: internet ALB across az_count ≥ 2 public subnets; az_count instances in private subnets (one per AZ), NAT for egress, no instance public IP; instances register to one target group with HTTP / health check expecting 200; instance SG allows :80 from the ALB SG and from the VPC CIDR. enable_ssm=true (default): Session Manager with AmazonSSMManagedInstanceCore—no SSH in the template.

locals {
  subnet_count   = var.use_alb ? var.az_count : 1
  instance_count = var.use_alb ? var.az_count : 1
}

variable "az_count" {
  default = 3
  # ...
  validation {
    condition     = var.az_count >= 1 && var.az_count <= 6 && (!var.use_alb || var.az_count >= 2)
    error_message = "az_count must be between 1 and 6, and at least 2 when use_alb is true (ALB requirement)."
  }
}

Why EC2?

Learning stack (Terraform + VPC + optional ELB), policy that prefers VPC-hosted sites, temporary lift-and-shift, or a box that might grow non-static behavior later. None of that makes EC2 the default for a pure static site.

VPC

Every instance is in a VPC and a subnet (EC2-Classic is gone for new accounts). The demo creates its own VPC—public subnets always; private subnets + NAT when use_alb=true.

Architecture

use_alb = false
  +----------+   :80 (public IP)   +----------------+
  | Clients  | ------------------> | EC2 (nginx)    |
  +----------+                     +----------------+

use_alb = true
  +----------+   :80   +-----+   :80   +----------------+
  | Clients  | ------> | ALB | ------> | EC2 × az_count |
  +----------+         +-----+         | (private)      |
                                      +----------------+

NAT is billed when the ALB path is on. Repeated curl to the ALB can show different AZ / private IP in index.html as backends rotate. user_data fills those fields via IMDSv2 after nginx is up:

IMDS_TOKEN=$(curl -sS -X PUT "http://169.254.169.254/latest/api/token" \
  -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")
PRIVATE_IP=$(curl -sS -H "X-aws-ec2-metadata-token: $IMDS_TOKEN" \
  "http://169.254.169.254/latest/meta-data/local-ipv4")
AZ=$(curl -sS -H "X-aws-ec2-metadata-token: $IMDS_TOKEN" \
  "http://169.254.169.254/latest/meta-data/placement/availability-zone")

main.tf — placement and bootstrap:

resource "aws_instance" "web" {
  count = local.instance_count

  subnet_id              = var.use_alb ? aws_subnet.private[count.index].id : aws_subnet.public[0].id
  vpc_security_group_ids = [aws_security_group.instance.id]
  user_data              = templatefile("${path.module}/user_data.tftpl", { enable_ssm = var.enable_ssm })
  user_data_replace_on_change = true
  iam_instance_profile   = var.enable_ssm ? aws_iam_instance_profile.ssm[0].name : null
  # ...
}

main.tf — instance ingress (ALB on) and target group health check:

dynamic "ingress" {
  for_each = var.use_alb ? [1] : []
  content {
    description     = "HTTP from ALB security group (forwarded client traffic)"
    from_port       = 80
    to_port         = 80
    protocol        = "tcp"
    security_groups = [aws_security_group.alb[0].id]
  }
}

dynamic "ingress" {
  for_each = var.use_alb ? [1] : []
  content {
    description = "HTTP from VPC for ALB health checks and internal probes"
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = [aws_vpc.main.cidr_block]
  }
}

resource "aws_lb_target_group" "web" {
  count = var.use_alb ? 1 : 0

  port             = 80
  protocol         = "HTTP"
  protocol_version = "HTTP1"
  # ...

  health_check {
    enabled             = true
    protocol            = "HTTP"
    port                = "traffic-port"
    path                = "/"
    matcher             = "200"
    interval            = 15
    timeout             = 10
    healthy_threshold   = 2
    unhealthy_threshold = 3
  }
}

Session Manager

Agent + AmazonSSMManagedInstanceCore; user_data.tftpl via templatefile so #!/bin/bash is at column 0 (indented heredoc in Terraform often breaks the shebang). With enable_ssm, the template pulls the SSM Agent RPM from S3 and starts the service. Use the AMI’s curl—do not dnf install curl on AL2023 (conflicts with curl-minimal, set -e aborts before nginx). Egress: NAT or SSM VPC endpoints. Docs: agent install, status. Your principal needs ssm:StartSession; Session Manager plugin for CLI. After apply, wait for Online in Fleet Manager; with use_alb=true, use instance_ids or ssm_start_session_command (first instance). %{ if enable_ssm ~} … %{ endif ~} wraps the SSM block in the template.

iam.tf:

resource "aws_iam_role_policy_attachment" "ssm_core" {
  count      = var.enable_ssm ? 1 : 0
  role       = aws_iam_role.ssm[0].name
  policy_arn = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
}

user_data.tftpl (SSM path):

#!/bin/bash
# Column-0 shebang required: indented Terraform heredocs break #!/bin/bash and cloud-init may skip the script.
set -eux
# Do not `dnf install curl` here: AL2023 ships curl-minimal; installing full curl conflicts and aborts the whole script under set -e.
SSM_RPM=""
case "$(uname -m)" in
  x86_64) SSM_RPM="https://s3.amazonaws.com/ec2-downloads-windows/SSMAgent/latest/linux_amd64/amazon-ssm-agent.rpm" ;;
  aarch64) SSM_RPM="https://s3.amazonaws.com/ec2-downloads-windows/SSMAgent/latest/linux_arm64/amazon-ssm-agent.rpm" ;;
  *) echo "Unsupported arch for SSM agent RPM"; exit 1 ;;
esac
curl -sS -o /tmp/amazon-ssm-agent.rpm "$SSM_RPM"
dnf install -y /tmp/amazon-ssm-agent.rpm
rm -f /tmp/amazon-ssm-agent.rpm
systemctl enable amazon-ssm-agent
systemctl restart amazon-ssm-agent

robots.txt

Crawler hint file—not security. The demo writes:

User-agent: *
Allow: /

user_data.tftpl:

cat >/usr/share/nginx/html/robots.txt <<'ROBOTS'
User-agent: *
Allow: /
ROBOTS

Google: robots.txt.

Run it

git clone https://github.com/jdevto/tf-aws-ec2-static-demo.git
cd tf-aws-ec2-static-demo
terraform init && terraform apply

ALB (needs ≥2 AZs; default az_count is 3):

terraform apply -var="use_alb=true"
# terraform apply -var="use_alb=true" -var="az_count=2"

Wait 1–2 minutes after first boot for user_data. user_data_replace_on_change = true replaces instances when the template changes. Variables: repo README.

variable "use_alb" {
  type    = bool
  default = false
}

variable "allowed_http_cidr" {
  type    = string
  default = "0.0.0.0/0"
}

variable "enable_ssm" {
  type    = bool
  default = true
}

terraform output verify_commands
# use_alb=false:
curl -sS "http://$(terraform output -raw instance_public_ip)/robots.txt"
# use_alb=true (no instance public IP):
# curl -sS "$(terraform output -raw website_url_alb)/robots.txt"
terraform output -raw ssm_start_session_command

Troubleshooting

Issue	Check
Timeout / connection refused	`use_alb=true`: use ALB URL, not instance IP. `false`: SG, `allowed_http_cidr`, `user_data`, `instance_public_ip`.
ALB unhealthy / 502	SG :80 from ALB + VPC; `/` returns 200. `cloud-init-output.log`: failed `user_data` (e.g. `dnf install curl` vs `curl-minimal`).
Apply limits	Other region/account or limit increase.
SSM offline / denied	`enable_ssm`, agent time, `ssm:StartSession`, outbound to SSM (NAT or endpoints).

References

Using GoAccess to View nginx Logs

John Ajera — Sat, 28 Mar 2026 08:54:19 +0000

Using GoAccess to View nginx Logs

GoAccess is a fast terminal and HTML log analyzer for nginx access logs (requests/sec, status codes, top URLs, referrers, user agents). This guide covers installation, optional S3 download, and typical run modes.

1. Overview

Install GoAccess (package manager, source, or jdevto/cli-tools script)
Optional: sync logs from S3 when they are not on the machine you analyze from
Run TUI, HTML report, live tail, or compressed input; use COMBINED or COMMON to match nginx log_format

Skip §4 if logs are already on disk.

2. Prerequisites

Readable nginx access logs (usually combined or common format)
sudo for distro package installs
S3 (optional): AWS CLI v2 and s3:GetObject / s3:ListBucket for the prefix you use

3. Install GoAccess

Fedora / RHEL / CentOS Stream

sudo dnf install -y goaccess

Debian / Ubuntu

sudo apt update && sudo apt install -y goaccess

goaccess --version

Newer or custom build: official build instructions.

jdevto/cli-tools installer (optional)

Builds from the official tarball; details and env vars in install_goaccess.md.

bash <(curl -s https://raw.githubusercontent.com/jdevto/cli-tools/main/scripts/install_goaccess.sh) install

bash <(curl -s https://raw.githubusercontent.com/jdevto/cli-tools/main/scripts/install_goaccess.sh) uninstall

4. Optional: Download nginx Logs from S3

Single file

mkdir -p ~/nginx-logs
aws s3 cp s3://my-bucket/path/to/access.log ~/nginx-logs/access.log

Key is .gz only: download, decompress, or stream without extracting:

aws s3 cp s3://my-bucket/path/to/access.log.gz ~/nginx-logs/access.log.gz
gzip -d ~/nginx-logs/access.log.gz
# or: zcat ~/nginx-logs/access.log.gz | goaccess - --log-format=COMBINED

Prefix (many files)

mkdir -p ~/nginx-logs
aws s3 sync s3://my-bucket/nginx-logs/ ~/nginx-logs/

Replace bucket and prefix. If you only have *.gz, after sync use zcat ~/nginx-logs/*.gz > ~/nginx-logs/merged-access.log (order across files may not be chronological) or decompress then merge plain *.log.

Merge plain logs (optional)

cat ~/nginx-logs/*.log > ~/nginx-logs/merged-access.log

5. Run GoAccess

Match --log-format to nginx (COMBINED is the usual default).

Terminal UI

goaccess /var/log/nginx/access.log --log-format=COMBINED

HTML report

goaccess /var/log/nginx/access.log --log-format=COMBINED -o report.html --no-parsing-spinner
xdg-open report.html

Live log

tail -f /var/log/nginx/access.log | goaccess - --log-format=COMBINED

Common log format

goaccess /var/log/nginx/access.log --log-format=COMMON

Custom log_format: map fields with GoAccess custom format tokens.

Compressed on disk

zcat /var/log/nginx/access.log.*.gz | goaccess - --log-format=COMBINED

6. Summary: Copy-Paste

sudo dnf install -y goaccess
mkdir -p ~/nginx-logs && aws s3 sync s3://YOUR_BUCKET/YOUR_PREFIX/ ~/nginx-logs/   # optional
goaccess /var/log/nginx/access.log --log-format=COMBINED
goaccess /var/log/nginx/access.log --log-format=COMBINED -o report.html --no-parsing-spinner && xdg-open report.html
tail -f /var/log/nginx/access.log | goaccess - --log-format=COMBINED

7. Troubleshooting

Parsed 0 lines / wrong numbers: Log line shape ≠ COMBINED/COMMON. Compare head -1 to nginx log_format and adjust --log-format.

Permission denied on log: sudo goaccess ..., fix group on log files, or copy the log to your home directory.

S3 Access Denied: IAM on bucket/prefix; check AWS_PROFILE / role.

Live tail idle: Confirm nginx access_log path and read permissions on the active file.

8. References

Using GoAccess to View nginx Logs

John Ajera — Sun, 22 Mar 2026 23:22:42 +0000

Using GoAccess to View nginx Logs

1. Overview

Install GoAccess (package manager, source, or jdevto/cli-tools script)
Optional: sync logs from S3 when they are not on the machine you analyze from
Run TUI, HTML report, live tail, or compressed input; use COMBINED or COMMON to match nginx log_format

Skip §4 if logs are already on disk.

2. Prerequisites

Readable nginx access logs (usually combined or common format)
sudo for distro package installs
S3 (optional): AWS CLI v2 and s3:GetObject / s3:ListBucket for the prefix you use

3. Install GoAccess

Fedora / RHEL / CentOS Stream

sudo dnf install -y goaccess

Debian / Ubuntu

sudo apt update && sudo apt install -y goaccess

goaccess --version

Newer or custom build: official build instructions.

jdevto/cli-tools installer (optional)

Builds from the official tarball; details and env vars in install_goaccess.md.

bash <(curl -s https://raw.githubusercontent.com/jdevto/cli-tools/main/scripts/install_goaccess.sh) install

bash <(curl -s https://raw.githubusercontent.com/jdevto/cli-tools/main/scripts/install_goaccess.sh) uninstall

4. Optional: Download nginx Logs from S3

Single file

mkdir -p ~/nginx-logs
aws s3 cp s3://my-bucket/path/to/access.log ~/nginx-logs/access.log

Key is .gz only: download, decompress, or stream without extracting:

aws s3 cp s3://my-bucket/path/to/access.log.gz ~/nginx-logs/access.log.gz
gzip -d ~/nginx-logs/access.log.gz
# or: zcat ~/nginx-logs/access.log.gz | goaccess - --log-format=COMBINED

Prefix (many files)

mkdir -p ~/nginx-logs
aws s3 sync s3://my-bucket/nginx-logs/ ~/nginx-logs/

Merge plain logs (optional)

cat ~/nginx-logs/*.log > ~/nginx-logs/merged-access.log

5. Run GoAccess

Match --log-format to nginx (COMBINED is the usual default).

Terminal UI

goaccess /var/log/nginx/access.log --log-format=COMBINED

HTML report

goaccess /var/log/nginx/access.log --log-format=COMBINED -o report.html --no-parsing-spinner
xdg-open report.html

Live log

tail -f /var/log/nginx/access.log | goaccess - --log-format=COMBINED

Common log format

goaccess /var/log/nginx/access.log --log-format=COMMON

Custom log_format: map fields with GoAccess custom format tokens.

Compressed on disk

zcat /var/log/nginx/access.log.*.gz | goaccess - --log-format=COMBINED

6. Summary: Copy-Paste

sudo dnf install -y goaccess
mkdir -p ~/nginx-logs && aws s3 sync s3://YOUR_BUCKET/YOUR_PREFIX/ ~/nginx-logs/   # optional
goaccess /var/log/nginx/access.log --log-format=COMBINED
goaccess /var/log/nginx/access.log --log-format=COMBINED -o report.html --no-parsing-spinner && xdg-open report.html
tail -f /var/log/nginx/access.log | goaccess - --log-format=COMBINED

7. Troubleshooting

Parsed 0 lines / wrong numbers: Log line shape ≠ COMBINED/COMMON. Compare head -1 to nginx log_format and adjust --log-format.

Permission denied on log: sudo goaccess ..., fix group on log files, or copy the log to your home directory.

S3 Access Denied: IAM on bucket/prefix; check AWS_PROFILE / role.

Live tail idle: Confirm nginx access_log path and read permissions on the active file.

8. References

Why Your Kustomize Remote Bases Break on Managed Argo CD (and How to Fix It)

John Ajera — Mon, 16 Mar 2026 10:50:40 +0000

Why Your Kustomize Remote Bases Break on Managed Argo CD (and How to Fix It)

You switched to managed Argo CD (e.g. EKS Capabilities) and use AWS CodeConnections for Git. Your Applications that pull from public GitHub remotes in Kustomize suddenly fail with errors like "Password authentication is not supported" or "Authentication failed". The same kustomization worked with self-managed Argo CD. This article explains why CodeConnections causes that and how vendoring those remote bases into your repo fixes it.

1. Overview

What this guide does:

Explains why Kustomize remote resources (e.g. github.com/open-policy-agent/gatekeeper-library/...) break when Argo CD uses CodeConnections for Git
Shows that the credential helper is applied to every github.com URL, including public repos, and GitHub rejects those credentials
Describes when to vendor and a minimal how-to: download remote files at a pinned ref, add them to your repo, and point your kustomization at local files instead of remote URLs
Suggests keeping a README in the vendored directory so future upgrades are a repeatable re-download and commit

Why it breaks:

With managed Argo CD + CodeConnections, one credential is used for all GitHub access. When Kustomize runs git fetch for a remote base, Argo CD passes that same credential. Public repos don't need auth—but they receive it anyway, and GitHub rejects the format (e.g. "Password authentication is not supported").

2. Prerequisites

Managed Argo CD (e.g. EKS Argo CD capability) with AWS CodeConnections used for Git (see Argo CD on EKS: Git repo access with AWS CodeConnections and Terraform for setup)
At least one Argo CD Application whose source uses Kustomize with remote resources or bases pointing at GitHub (e.g. github.com/org/repo/path?ref=master)

3. Why remote bases break with CodeConnections

When Argo CD syncs an Application, the repo-server runs kustomize build over your repo. If your kustomization.yaml lists remote resources like:

resources:
  - github.com/open-policy-agent/gatekeeper-library/library/general/httpsonly?ref=master

Kustomize invokes git fetch for that URL. Argo CD configures a Git credential helper so that any git operation gets credentials. With CodeConnections, that helper returns the same credential (the one for your connected repos) for every github.com request—including requests to public repos such as open-policy-agent/gatekeeper-library.

Git sends those credentials to GitHub. GitHub then responds with something like:

fatal: Authentication failed for 'https://github.com/open-policy-agent/gatekeeper-library/'
remote: Invalid username or token. Password authentication is not supported for Git operations.

So the build fails even though the remote repo is public and would work with no credentials. With self-managed Argo CD you might have used SSH for your app repo; Kustomize’s HTTPS fetches to public GitHub then didn’t use that credential and succeeded. With CodeConnections, one HTTPS credential is used everywhere, and it gets sent to public URLs where it’s invalid.

4. When to vendor

Vendor remote bases when:

You use managed Argo CD with CodeConnections and your Kustomize app references public GitHub remotes (e.g. gatekeeper-library, shared bases from other orgs).
You see authentication or "Password authentication is not supported" errors during manifest generation for those remotes.

You can also vendor remotes you don’t control so that upgrades are explicit and reproducible (pin to a commit, re-download when you want to upgrade).

5. How to vendor

Vendoring means copying the remote manifest files into your repo and pointing Kustomize at local files instead of remote URLs.

Steps:

Choose a ref — Use a commit SHA or tag from the upstream repo (e.g. master or a specific SHA) so you can reproduce the same content later.
Download the remote files — Each remote resource is usually a directory containing one or more YAML files (e.g. template.yaml). Download those files using the raw GitHub URL pattern: https://raw.githubusercontent.com/<org>/<repo>/<ref>/<path>/<file>.yaml Save them under a directory in your repo (e.g. infrastructure/my-app/vendored/) with clear names (e.g. httpsonly.yaml, requiredlabels.yaml).
Update your kustomization — Replace remote resources entries with the local file names:

# Before (remote – fails with CodeConnections)
resources:
  - github.com/open-policy-agent/gatekeeper-library/library/general/httpsonly?ref=master

# After (vendored)
resources:
  - httpsonly.yaml

Document the source and upgrade process — Add a README in the vendored directory that lists:
- The upstream repo and the ref (commit SHA) you used
- The mapping from each local file to its upstream path
- How to upgrade: re-download from a new ref, overwrite the files, run kustomize build . to verify, commit, and update the README with the new ref.

Optionally, add a small script or one-liner (e.g. a shell loop with curl) that downloads all vendored files given a ref, so upgrades are a single command plus commit.

6. Summary

Problem: Managed Argo CD + CodeConnections sends one GitHub credential to every git URL. Kustomize remote bases to public GitHub repos get that credential; GitHub rejects it and the build fails.
Fix: Vendor those remote bases: download the manifest files at a pinned ref, put them in your repo, and point kustomization.yaml at the local files. No remote fetch at build time, so no credential is used for those resources.
Upgrades: Re-download from a new ref, overwrite the vendored files, verify with kustomize build ., commit, and update the README with the new ref.

Checklist when vendoring:

Pick a ref (e.g. latest master or a commit SHA) from the upstream repo.
Download each remote file via https://raw.githubusercontent.com/<org>/<repo>/<ref>/<path>/<file>.yaml into a directory in your repo.
Replace remote resources in kustomization.yaml with the local file names.
Add a README in that directory with source ref, file mapping, and upgrade steps (and optionally a download script).

7. Troubleshooting

Manifest generation still fails after vendoring

Confirm kustomization.yaml lists only local file names (e.g. httpsonly.yaml), not github.com/... URLs.
Run kustomize build . from the directory that contains the kustomization and vendored files; fix any path or resource errors before relying on Argo CD.

Upstream added or removed a file

Re-download from the ref you want (e.g. new commit on master). Add or remove the corresponding local file and update the resources list and README.

8. References

Argo CD – Kustomize (including private remote bases)
Argo CD on EKS: Git repo access with AWS CodeConnections and Terraform (CodeConnections setup)
gatekeeper-library (example upstream that’s often used as a Kustomize remote)

Argo CD on EKS: Git repo access with AWS CodeConnections and Terraform

John Ajera — Sat, 14 Mar 2026 07:31:29 +0000

Argo CD on EKS: Git repo access with AWS CodeConnections and Terraform

Argo CD on EKS Capabilities needs to pull from your Git repos. Instead of storing personal access tokens or SSH keys in the cluster, use AWS CodeConnections: one connection authorizes Argo CD to access GitHub via IAM. This guide gives the minimal Terraform (connection + IAM policy on the Argo CD role) and the one-time Console steps to move the connection from Pending to Available. One connection can serve many repos; you only change the owner/repo in the URL.

1. Overview

What this guide does:

Creates a CodeStar connection (GitHub) and grants the Argo CD capability role codeconnections:UseConnection and codeconnections:GetConnection so Argo CD can pull from Git without credentials in the cluster
Walks through the one-time Console step to complete the connection (Pending → Available), including using the GitHub App (e.g. AWS Connector for GitHub) for org repos
Shows the CodeConnections repo URL format for Argo CD Applications (connection ID is the UUID only, not the ARN)

Why CodeConnections?

No PAT or SSH key in Kubernetes; the Argo CD role gets IAM permission to use the connection
One connection = multiple GitHub repos (same connection ID, different owner/repo in the URL)

2. Prerequisites

EKS cluster with the Argo CD capability enabled (Identity Center configured)
The name of the Argo CD capability IAM role (e.g. from your EKS module: cluster_name-argocd-capability-role, or from AWS Console → IAM → Roles)
AWS Console access to complete the connection (GitHub OAuth)

3. Terraform: connection and IAM

Add the following to your Terraform. Replace argocd_capability_role_name with the actual role name (the role EKS created for the Argo CD capability).

variable "argocd_capability_role_name" {
  description = "Name of the IAM role used by the Argo CD EKS Capability"
  type        = string
}

resource "aws_codestarconnections_connection" "github" {
  name          = "github"
  provider_type = "GitHub"
}

data "aws_iam_policy_document" "argocd_codeconnections" {
  statement {
    effect = "Allow"
    actions = [
      "codeconnections:UseConnection",
      "codeconnections:GetConnection"
    ]
    resources = [aws_codestarconnections_connection.github.arn]
  }
}

resource "aws_iam_role_policy" "argocd_codeconnections" {
  name   = "argocd-codeconnections"
  role   = var.argocd_capability_role_name
  policy = data.aws_iam_policy_document.argocd_codeconnections.json
}

After terraform apply, the connection exists in Pending state. Terraform cannot complete it; you do that once in the Console.

4. After apply: complete the connection (Pending → Available)

Open the AWS Console in the same region as your Terraform.
Go to Developer Tools → Connections (or search Connections).
Find the connection (e.g. github). Status will be Pending.
Click the connection name → Update pending connection (or Connect to GitHub). You’ll land on the Connect to GitHub page.

GitHub connection settings:

Connection name — Confirm or set the name (e.g. github to match your Terraform name). This is how the connection appears in the Console.
App Installation (optional) — Two choices:
- Leave blank to connect as a GitHub user. That user must have (at least read) access to the repos you use in Argo CD. Fine for personal or single-account repos.
- Use a GitHub App (recommended for org repos): e.g. choose AWS Connector for GitHub via “Install a new app”, then select the org where your repos live (e.g. my-org) as the installation target.
Click the orange Connect button to start the GitHub authorization flow (or complete it after installing the app).

If you chose the GitHub App, you’ll land on GitHub’s Install & Authorize AWS Connector for GitHub page for that org. Choose Only select repositories, click Select repositories, and pick the repo(s) Argo CD will use (e.g. my-org/my-repo). Review the permissions (read/write access to code, pull requests, etc.), then click Install & Authorize. You’ll be redirected to https://redirect.codestar.aws/return. Back on the AWS Connect to GitHub page, the App Installation field will show your installation (e.g. an installation ID such as 123456789—yours will differ). Confirm Connection name (e.g. github), then click the orange Connect button. The connection status should become Available.
If you connected as a GitHub user, complete the OAuth prompt, then return to the Console and confirm status is Available.

5. Test the connection

Once the connection is Available:

Register the repo in Argo CD (one-time). Create a repository Secret so the repo appears under Settings → Repositories. Use your real region, account ID, connection UUID, and org/repo—do not leave literal OWNER/REPO in the URL or you’ll get “repository not found”. Example (replace the values if yours differ):

   export REGION=ap-southeast-2
   export ACCOUNT=$(aws sts get-caller-identity --query Account --output text)
   export CONN_ID=a1b2c3d4-e5f6-7890-abcd-ef1234567890   # your connection UUID from Console or Terraform
   export OWNER_REPO=my-org/my-repo                     # your org and repo
   kubectl create secret generic argocd-repo-github -n argocd --from-literal=url="https://codeconnections.${REGION}.amazonaws.com/git-http/${ACCOUNT}/${REGION}/${CONN_ID}/${OWNER_REPO}.git"
   kubectl label secret argocd-repo-github -n argocd argocd.argoproj.io/secret-type=repository

If the secret already exists with the wrong URL, delete it first: kubectl delete secret argocd-repo-github -n argocd, then run the create and label commands above.

In Argo CD → Settings → Repositories, click REFRESH LIST. The repo should show with a green check (Successful). If it shows Failed, see the Troubleshooting section.
Create an Application that uses this repo: set source.repoURL to the same CodeConnections URL and a path/targetRevision. Sync the application and confirm it syncs without errors.

6. Using the repo URL in Argo CD

CodeConnections URL format (replace placeholders):

https://codeconnections.<region>.amazonaws.com/git-http/<account-id>/<region>/<connection-id>/OWNER/REPO.git

connection-id: Must be the connection ID (UUID only), e.g. 9b6c3274-31da-4d1d-8955-e7d5cd9d15e2. Do not put the full ARN in the URL path—that causes HTTP 400. In Terraform the AWS provider’s aws_codestarconnections_connection.github.id is the ARN; use the UUID (the segment after the last / in the ARN) or an output from your module that exposes the UUID.
OWNER/REPO: Your Git org and repo (e.g. my-org/my-repo).

In your Argo CD Application, set source.repoURL to that URL. One connection works for many repos—same connection ID, change only owner/repo.

7. Summary: copy-paste

Terraform: Connection + IAM policy (role name from your EKS setup):

resource "aws_codestarconnections_connection" "github" {
  name          = "github"
  provider_type = "GitHub"
}

resource "aws_iam_role_policy" "argocd_codeconnections" {
  name   = "argocd-codeconnections"
  role   = var.argocd_capability_role_name  # your Argo CD capability role name
  policy = jsonencode({
    Version   = "2012-10-17"
    Statement = [{
      Effect   = "Allow"
      Action   = ["codeconnections:UseConnection", "codeconnections:GetConnection"]
      Resource = aws_codestarconnections_connection.github.arn
    }]
  })
}

Console: Developer Tools → Connections → select connection → Update pending connection → on the Connect to GitHub page set/confirm Connection name, then either leave App Installation blank (user connection) or install AWS Connector for GitHub and select the org that owns your repos (e.g. my-org) → click Connect → complete GitHub auth → status Available.

Argo CD: Use the CodeConnections URL with the connection ID and your OWNER/REPO as source.repoURL.

8. Troubleshooting

Issue	What to check
Connection stays Pending	Complete the connection in the AWS Console (authorize in GitHub or install the app). Check region.
HTTP 400 when Argo CD tests repo	The repo URL path must use the connection ID (UUID only), not the full ARN. If your URL contains `arn:aws:codestar-connections:` in the path, fix the URL to use only the UUID (e.g. from Terraform use a value that outputs the UUID, not `connection.id` which is the ARN).
repository not found / ProviderResourceNotFoundException	The URL still has literal `OWNER/REPO` instead of your real org and repo (e.g. `my-org/my-repo`). Delete the repo Secret and recreate it with the correct URL (see §5).
authorization failed: Write access to repository not granted	The GitHub identity (user or app) used by the connection doesn’t have access to that repo. Install the GitHub App (e.g. AWS Connector for GitHub) on the org that owns the repo (e.g. my-org), or ensure the GitHub user that authorized has read access to the repo. Then re-authorize the connection if needed.
Argo CD cannot fetch repo	Argo CD role has `codeconnections:UseConnection` and `codeconnections:GetConnection` on the connection ARN; connection status Available; `source.repoURL` uses the UUID as connection-id and correct owner/repo.
Wrong region	Connection is regional. Console and Terraform region must match.

9. References

Configure repository access (Argo CD, CodeConnections) – AWS EKS User Guide
EKS Capabilities – Argo CD, ACK, KRO

Bitwarden Secrets Manager on EKS – Per-App Integration with Atlantis

John Ajera — Mon, 02 Mar 2026 10:01:15 +0000

Bitwarden Secrets Manager on EKS – Per-App Integration with Atlantis

Sync secrets from Bitwarden Secrets Manager into Kubernetes on EKS using the sm-operator, AWS Secrets Manager for the machine token, and the Secrets Store CSI Driver. This guide expands on the base integration with a per-app, per-namespace pattern and uses Atlantis as a concrete example. It covers Terraform, Kustomize overlays, Argo CD, sync waves, and troubleshooting.

Note: Use placeholder values for org IDs and secret IDs. Never commit real tokens. For production, follow least-privilege IAM and rotation practices.

1. Overview

What this guide does:

Integrates Bitwarden Secrets Manager with EKS via sm-operator, AWS Secrets Manager, and Secrets Store CSI Driver
Uses a per-app namespace pattern: each app (e.g. Atlantis) gets its own SecretProviderClass, bw-auth-token-sync, and BitwardenSecret in its own namespace
Walks through Terraform (EKS + Pod Identity per app), manifests, Argo CD Applications, validation, and force-sync
Uses Atlantis (Terraform PR automation) as a worked example with GitHub App credentials

Why per-app namespaces?

Isolation: each app has its own Bitwarden machine token (scoped in AWS)
Simplicity: the sm-operator creates the K8s Secret in the app’s namespace; no cross-namespace wiring
Consistency: same pattern for every new app

2. Prerequisites

EKS cluster with Secrets Store CSI Driver installed
Argo CD installed
Terraform-managed EKS (e.g. terraform-aws-eks-basic with Secrets Manager support)
Bitwarden Secrets Manager organization with Machine Account
Per-app AWS secret path: bitwarden/sm-operator/<app>/machine-token

3. Architecture Overview

Bitwarden SM (Machine Account + App Secrets)
        │
        │ machine token (per app: bitwarden/sm-operator/<app>/machine-token)
        ▼
AWS Secrets Manager
        │
        │ Pod Identity + CSI (in app namespace, e.g. atlantis-1)
        ▼
SecretProviderClass + bw-auth-token-sync → creates bw-auth-token
        │
        ▼
BitwardenSecret (authToken: bw-auth-token)
        │
        │ sm-operator fetches via Bitwarden API
        ▼
Output K8s Secret (e.g. atlantis-1-vcs) in app namespace

Flow (per app, e.g. atlantis-1):

Machine token stored in AWS Secrets Manager as bitwarden/sm-operator/<app>/machine-token (JSON: {"token":"<value>"}).
SecretProviderClass + bw-auth-token-sync Deployment in the app namespace: CSI Driver mounts the token and creates K8s Secret bw-auth-token.
BitwardenSecret in the same namespace: authToken.secretName: bw-auth-token. The sm-operator reads this token and calls the Bitwarden API.
sm-operator creates the output K8s Secret (e.g. atlantis-1-vcs) in the app’s namespace. Atlantis (or any app) uses it directly.

4. Bitwarden Setup

Machine Account and token: Bitwarden Admin → Machine Accounts → Create Access Token. Copy the token.
Per-app machine token (optional): For isolation, create a separate token per app and store in bitwarden/sm-operator/<app>/machine-token in AWS.
App secrets: In Bitwarden Secrets Manager, create the secrets your apps need. For Atlantis (GitHub App): webhook secret, private key (key.pem).
Copy IDs: From Bitwarden Admin → Settings → Organization, copy organizationId. For each secret, copy its bwSecretId (UUID).

5. Terraform

Requirement: EKS module must enable Secrets Manager with Pod Identity. Add an association per app namespace (e.g. { namespace = "atlantis-1", service_account = "awssm-sync" }) and prefix bitwarden/sm-operator.

Create the AWS Secrets Manager secret per app. Pass the token via -var or TF_VAR_. Never commit it.

variable "bitwarden_sm_machine_token_atlantis" {
  description = "Bitwarden SM machine token for atlantis-1"
  type        = string
  default     = "REPLACE_WITH_REAL_TOKEN"
  sensitive   = true
}

resource "aws_secretsmanager_secret" "bitwarden_atlantis" {
  name        = "bitwarden/sm-operator/atlantis-1/machine-token"
  description = "Bitwarden SM machine token for atlantis-1"
  tags        = var.tags
}

resource "aws_secretsmanager_secret_version" "bitwarden_atlantis" {
  secret_id     = aws_secretsmanager_secret.bitwarden_atlantis.id
  secret_string = jsonencode({ token = var.bitwarden_sm_machine_token_atlantis })
}

6. Per-App Manifests (Atlantis Example)

Each app gets its own overlay with:

SecretProviderClass
ServiceAccount + bw-auth-token-sync Deployment
BitwardenSecret
The app itself (e.g. Atlantis Helm chart)

6.1 SecretProviderClass

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: bitwarden-sm-token
  namespace: atlantis-1
  annotations:
    argocd.argoproj.io/sync-wave: "-1"
spec:
  provider: aws
  parameters:
    region: ap-southeast-2
    usePodIdentity: "true"
    objects: |
      - objectName: "bitwarden/sm-operator/atlantis-1/machine-token"
        objectType: "secretsmanager"
        jmesPath:
          - path: token
            objectAlias: token
  secretObjects:
    - secretName: bw-auth-token
      type: Opaque
      data:
        - objectName: token
          key: token

6.2 ServiceAccount + bw-auth-token-sync

apiVersion: v1
kind: ServiceAccount
metadata:
  name: awssm-sync
  namespace: atlantis-1
  annotations:
    argocd.argoproj.io/sync-wave: "-2"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: bw-auth-token-sync
  namespace: atlantis-1
  annotations:
    argocd.argoproj.io/sync-wave: "-1"
  labels:
    app: bw-auth-token-sync
spec:
  replicas: 1
  selector:
    matchLabels:
      app: bw-auth-token-sync
  template:
    metadata:
      labels:
        app: bw-auth-token-sync
    spec:
      serviceAccountName: awssm-sync
      containers:
        - name: pause
          image: registry.k8s.io/pause:3.9
          resources:
            requests:
              cpu: 1m
              memory: 4Mi
          volumeMounts:
            - name: secrets-store
              mountPath: "/mnt/secrets-store"
              readOnly: true
      volumes:
        - name: secrets-store
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: bitwarden-sm-token

6.3 BitwardenSecret

apiVersion: k8s.bitwarden.com/v1
kind: BitwardenSecret
metadata:
  name: atlantis-1-vcs
  namespace: atlantis-1
  annotations:
    argocd.argoproj.io/sync-wave: "0"
spec:
  organizationId: "REPLACE_ORG_ID"
  secretName: atlantis-1-vcs
  onlyMappedSecrets: true
  map:
    - secretKeyName: github_secret
      bwSecretId: "REPLACE_WEBHOOK_SECRET_ID"
    - secretKeyName: key.pem
      bwSecretId: "REPLACE_PRIVATE_KEY_ID"
  authToken:
    secretName: bw-auth-token
    secretKey: token

6.4 Atlantis Helm values

# application-patch.yaml
vcsSecretName: atlantis-1-vcs
githubApp:
  id: "YOUR_GH_APP_ID"
  installationId: "YOUR_GH_INSTALLATION_ID"
ingress:
  enabled: false   # Enable when you have ingress + atlantisUrl

Use argocd.argoproj.io/sync-wave: "5" on the Atlantis Application so it is created after the BitwardenSecret and bw-auth-token-sync.

7. Sync Waves (Ordering)

To avoid the app starting before secrets exist:

-2: ServiceAccount awssm-sync
-1: SecretProviderClass, bw-auth-token-sync Deployment
0: BitwardenSecret, ConfigMaps
5: Atlantis Application (Helm)

8. Validation

# Verify secrets in app namespace
kubectl get secret atlantis-1-vcs bw-auth-token -n atlantis-1

# BitwardenSecret status
kubectl get bitwardensecret atlantis-1-vcs -n atlantis-1
kubectl get bitwardensecret atlantis-1-vcs -n atlantis-1 \
  -o jsonpath='{.status.conditions[?(@.type=="SuccessfulSync")].status}'

# Output secret keys (should show github_secret, key.pem)
kubectl get secret atlantis-1-vcs -n atlantis-1 -o jsonpath='{.data}' | jq -r 'keys[]'

# Atlantis pod
kubectl get pods -n atlantis-1
kubectl logs -n atlantis-1 -l app=atlantis --tail=20

9. Local Access (Port-Forward)

The Atlantis Service exposes port 80 (targetPort 4141):

kubectl port-forward svc/atlantis-1 -n atlantis-1 4141:80

Then open http://localhost:4141.

10. Force Sync (Token Refresh)

When the machine token in AWS Secrets Manager changes:

# Per-app namespace
kubectl delete secret bw-auth-token -n atlantis-1
kubectl delete pod -n atlantis-1 -l app=bw-auth-token-sync --force --grace-period=0
kubectl wait --for=condition=ready pod -l app=bw-auth-token-sync -n atlantis-1 --timeout=60s
kubectl rollout restart deployment/sm-operator-controller-manager -n sm-operator-system
kubectl rollout restart statefulset atlantis-1 -n atlantis-1

11. Adding Another App

For each new app:

Create app directory: apps/<app>/overlays/<cluster>/
Add SecretProviderClass (AWS path: bitwarden/sm-operator/<app>/machine-token)
Add bw-auth-token-sync (SA + Deployment)
Add BitwardenSecret with the app’s secret mapping
Add Terraform: { namespace = "<app>", service_account = "awssm-sync" } + AWS secret
Deploy the app (Helm, etc.) with the created secret

12. Summary: Copy-Paste

Validation:

kubectl get secret atlantis-1-vcs bw-auth-token -n atlantis-1
kubectl get bitwardensecret atlantis-1-vcs -n atlantis-1 -o jsonpath='{.status.conditions[?(@.type=="SuccessfulSync")].status}'
kubectl get secret atlantis-1-vcs -n atlantis-1 -o jsonpath='{.data}' | jq -r 'keys[]'

Port-forward:

kubectl port-forward svc/atlantis-1 -n atlantis-1 4141:80

Force sync (token changed):

kubectl delete secret bw-auth-token -n atlantis-1
kubectl delete pod -n atlantis-1 -l app=bw-auth-token-sync --force --grace-period=0
kubectl wait --for=condition=ready pod -l app=bw-auth-token-sync -n atlantis-1 --timeout=60s
kubectl rollout restart deployment/sm-operator-controller-manager -n sm-operator-system
kubectl rollout restart statefulset atlantis-1 -n atlantis-1

13. Troubleshooting

Issue: CreateContainerConfigError – Secret atlantis-1-vcs missing

Solution: Wait for sm-operator to sync BitwardenSecret; or restart pod after secret exists.

Issue: Pod Identity / token association error

Solution: Add { namespace = "atlantis-1", service_account = "awssm-sync" } to secrets_manager_associations in Terraform.

Issue: ResourceNotFoundException – AWS secret missing

Solution: Create bitwarden/sm-operator/atlantis-1/machine-token in Secrets Manager (JSON: {"token":"<value>"}).

Issue: Error: --gh-app-id/--gh-app-key-file... – VCS secret not reaching container

Solution: Ensure vcsSecretName and githubApp.id/installationId in Helm values; verify atlantis-1-vcs exists.

Issue: Argo CD app stuck "Progressing"

Solution: Ingress without controller. Set ingress.enabled: false until ingress is configured.

14. References

Bitwarden Secrets Manager on EKS – End-to-End Integration

John Ajera — Sun, 01 Mar 2026 18:41:10 +0000

Bitwarden Secrets Manager on EKS – End-to-End Integration

Note: Use placeholder values for org IDs and secret IDs. Never commit real tokens. For production, follow least-privilege IAM and rotation practices.

1. Overview

What this guide does:

Integrates Bitwarden Secrets Manager with EKS via sm-operator, AWS Secrets Manager, and Secrets Store CSI Driver
Stores machine token in AWS Secrets Manager; CSI Driver mounts it; sm-operator syncs Bitwarden secrets into K8s Secrets
Walks through Terraform (EKS + secret), flat manifests, Argo CD Application, validation, and force-sync

2. Prerequisites

EKS cluster with Secrets Store CSI Driver installed
Argo CD installed
Terraform-managed EKS (e.g. terraform-aws-eks-basic or equivalent)
Bitwarden Secrets Manager organization with Machine Account

3. Architecture Overview

Bitwarden SM (Machine Account + App Secrets)
        │
        │ machine token
        ▼
AWS Secrets Manager (bitwarden/sm-operator/machine-token)
        │
        │ Pod Identity + CSI
        ▼
Secrets Store CSI Driver → creates bw-auth-token
        │
        ▼
sm-operator (reads BitwardenSecret CR, uses bw-auth-token)
        │
        │ Bitwarden API fetches secrets
        ▼
github-app-secrets (K8s Secret)

Flow:

Machine token stored in AWS Secrets Manager as {"token":"<value>"}
CSI Driver mounts it into a pod; creates K8s Secret bw-auth-token
sm-operator uses bw-auth-token to auth to Bitwarden API and sync secrets into github-app-secrets (or your chosen output secret)

4. Bitwarden Setup

Machine Account and token: Bitwarden Admin → Machine Accounts → Create Access Token. Copy the token.
App secrets: In Bitwarden Secrets Manager, create the secrets your apps need (e.g. github-app-id, github-app-key, github-app-webhook-secret, github-app-installation-id).
Copy IDs: From Bitwarden Admin → Settings → Organization, copy organizationId. For each secret, copy its bwSecretId (UUID).

5. Terraform

Requirement: EKS module must enable Secrets Manager with Pod Identity for sm-operator-system / awssm-sync and secret prefix bitwarden/sm-operator.

Create the AWS Secrets Manager secret and pass the token via -var or TF_VAR_bitwarden_sm_machine_token. Never commit it.

variable "bitwarden_sm_machine_token" {
  description = "Bitwarden SM machine token from Machine Account → Create Access Token"
  type        = string
  default     = "REPLACE_WITH_REAL_TOKEN"
  sensitive   = true
}

resource "aws_secretsmanager_secret" "bitwarden_sm_token" {
  name        = "bitwarden/sm-operator/machine-token"
  description = "Bitwarden Secrets Manager machine token for sm-operator"
  tags        = var.tags
}

resource "aws_secretsmanager_secret_version" "bitwarden_sm_token" {
  secret_id     = aws_secretsmanager_secret.bitwarden_sm_token.id
  secret_string = jsonencode({ token = var.bitwarden_sm_machine_token })
}

6. Deploy sm-operator

Save as sm-operator.yaml and apply. Replace placeholders in BitwardenSecret; adjust region if needed.

# Namespace for sm-operator and CSI resources
apiVersion: v1
kind: Namespace
metadata:
  name: sm-operator-system
  labels:
    name: sm-operator-system
    app.kubernetes.io/name: sm-operator
---
# ServiceAccount with Pod Identity for AWS Secrets Manager access
apiVersion: v1
kind: ServiceAccount
metadata:
  name: awssm-sync
  namespace: sm-operator-system
  annotations:
    argocd.argoproj.io/sync-wave: "-2"
---
# Fetches machine token from AWS SM and creates K8s Secret bw-auth-token
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: bitwarden-sm-token
  namespace: sm-operator-system
  annotations:
    argocd.argoproj.io/sync-wave: "-1"
spec:
  provider: aws
  parameters:
    region: ap-southeast-2
    usePodIdentity: "true"
    objects: |
      - objectName: "bitwarden/sm-operator/machine-token"
        objectType: "secretsmanager"
        jmesPath:
          - path: token
            objectAlias: token
  secretObjects:
    - secretName: bw-auth-token
      type: Opaque
      data:
        - objectName: token
          key: token
---
# Pod that triggers CSI mount; creates bw-auth-token used by sm-operator
apiVersion: apps/v1
kind: Deployment
metadata:
  name: bw-auth-token-sync
  namespace: sm-operator-system
  annotations:
    argocd.argoproj.io/sync-wave: "-1"
  labels:
    app: bw-auth-token-sync
spec:
  replicas: 1
  selector:
    matchLabels:
      app: bw-auth-token-sync
  template:
    metadata:
      labels:
        app: bw-auth-token-sync
    spec:
      serviceAccountName: awssm-sync
      containers:
        - name: pause
          image: registry.k8s.io/pause:3.9
          resources:
            requests:
              cpu: 1m
              memory: 4Mi
          volumeMounts:
            - name: secrets-store
              mountPath: "/mnt/secrets-store"
              readOnly: true
      volumes:
        - name: secrets-store
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: bitwarden-sm-token
---
# Maps Bitwarden secrets to github-app-secrets K8s Secret
apiVersion: k8s.bitwarden.com/v1
kind: BitwardenSecret
metadata:
  name: bitwarden-secret
  namespace: sm-operator-system
spec:
  organizationId: "REPLACE_ORG_ID"
  secretName: github-app-secrets
  onlyMappedSecrets: true
  map:
    - secretKeyName: github-app-id
      bwSecretId: "REPLACE_SECRET_ID_1"
    - secretKeyName: github-app-key
      bwSecretId: "REPLACE_SECRET_ID_2"
    - secretKeyName: github-app-webhook-secret
      bwSecretId: "REPLACE_SECRET_ID_3"
    - secretKeyName: github-app-installation-id
      bwSecretId: "REPLACE_SECRET_ID_4"
  authToken:
    secretName: bw-auth-token
    secretKey: token
---
# Argo CD Application; deploys sm-operator Helm chart
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: sm-operator
  namespace: argocd
spec:
  project: platform
  source:
    repoURL: https://charts.bitwarden.com/
    chart: sm-operator
    targetRevision: 2.0.0
    helm:
      values: |
        settings:
          bwSecretsManagerRefreshInterval: 300
          cloudRegion: US
          replicas: 1
  destination:
    server: https://kubernetes.default.svc
    namespace: sm-operator-system
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
  syncOptions:
    - CreateNamespace=true

7. Validation

After Argo CD syncs the app:

# BitwardenSecret status
kubectl get bitwardensecret -n sm-operator-system
kubectl get bitwardensecret bitwarden-secret -n sm-operator-system \
  -o jsonpath='{.status.conditions[?(@.type=="SuccessfulSync")].status}'

# Output secret keys
kubectl get secret github-app-secrets -n sm-operator-system \
  -o jsonpath='{.data}' | jq -r 'keys[]'

Expected: SuccessfulSync status True and the mapped keys listed.

8. Force Sync (Token Refresh)

When the machine token in AWS Secrets Manager changes:

kubectl delete secret bw-auth-token -n sm-operator-system
kubectl delete pod -n sm-operator-system -l app=bw-auth-token-sync --force --grace-period=0
kubectl wait --for=condition=ready pod -l app=bw-auth-token-sync -n sm-operator-system --timeout=60s
kubectl rollout restart deployment/sm-operator-controller-manager -n sm-operator-system

When only BitwardenSecret mapping changes (e.g. new secrets):

kubectl delete bitwardensecret bitwarden-secret -n sm-operator-system
# Argo CD recreates it; wait for sync
kubectl annotate bitwardensecret bitwarden-secret -n sm-operator-system \
  force-reconcile="$(date +%s)" --overwrite

9. Summary: Copy-Paste

Validation (after sync):

kubectl get bitwardensecret -n sm-operator-system
kubectl get bitwardensecret bitwarden-secret -n sm-operator-system \
  -o jsonpath='{.status.conditions[?(@.type=="SuccessfulSync")].status}'
kubectl get secret github-app-secrets -n sm-operator-system -o jsonpath='{.data}' | jq -r 'keys[]'

Force sync (machine token changed):

kubectl delete secret bw-auth-token -n sm-operator-system
kubectl delete pod -n sm-operator-system -l app=bw-auth-token-sync --force --grace-period=0
kubectl wait --for=condition=ready pod -l app=bw-auth-token-sync -n sm-operator-system --timeout=60s
kubectl rollout restart deployment/sm-operator-controller-manager -n sm-operator-system

Force sync (BitwardenSecret mapping changed):

kubectl delete bitwardensecret bitwarden-secret -n sm-operator-system
kubectl annotate bitwardensecret bitwarden-secret -n sm-operator-system force-reconcile="$(date +%s)" --overwrite

10. Troubleshooting

Issue: Has an invalid identifier

Solution: One or more bwSecretId UUIDs are wrong. Verify each ID in Bitwarden Secrets Manager.

Issue: CSI secret not created

Solution: Check Pod Identity for SA awssm-sync in sm-operator-system; ensure SecretProviderClass and bw-auth-token-sync pod exist. Check CSI driver logs: kubectl logs -n kube-system -l app=csi-secrets-store.

Issue: No changes / Skipping sync

Solution: Delete the BitwardenSecret; Argo recreates it and the operator performs a fresh reconcile.

11. References

FastSchema on EKS – Install and Test

John Ajera — Sun, 01 Mar 2026 04:09:39 +0000

FastSchema on EKS – Install and Test

Deploy FastSchema via Argo CD on EKS and access it locally with port-forward. Uses SQLite with EBS-backed PVC. First-run setup token from pod logs.

FastSchema has no official Helm chart. This guide uses a community chart at k8sforge/fastschema-chart.

Note: This setup is for evaluation, not production-ready. For production, use external database (MySQL/PostgreSQL), configure auth providers, and follow security best practices.

1. Overview

What this guide does:

Deploys FastSchema using the community Helm chart via an Argo CD Application
Persists data with SQLite on EBS-backed PVC
Uses port-forward for local access (no ingress required)
Walks through first-run setup: token from pod logs, create admin account, verify storage

Prerequisites:

kubectl configured for your EKS cluster
Argo CD installed

2. Prerequisites

Before starting, ensure you have:

kubectl configured for your EKS cluster
Argo CD installed

3. Install FastSchema

Save the manifest below as fastschema-application.yaml and apply:

kubectl apply -f fastschema-application.yaml

Wait until the Application shows Synced in the Argo CD UI or CLI.

Manifest:

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: devops
  namespace: argocd
spec:
  clusterResourceWhitelist:
    - group: "*"
      kind: "*"
  description: DevOps-managed applications
  destinations:
    - namespace: "*"
      server: "*"
  namespaceResourceWhitelist:
    - group: "*"
      kind: "*"
  sourceRepos:
    - "*"
---
apiVersion: v1
kind: Namespace
metadata:
  name: fastschema
  labels:
    name: fastschema
    owner: devops
---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: fastschema
  namespace: argocd
spec:
  project: devops
  source:
    repoURL: https://k8sforge.github.io/fastschema-chart
    chart: fastschema
    targetRevision: 0.1.4
    helm:
      values: |
        service:
          type: ClusterIP
        persistence:
          enabled: true
          size: 10Gi
          storageClassName: ebs-sc
          accessModes:
            - ReadWriteOnce
        appBaseUrl: "http://localhost:8000"
        appDashUrl: "http://localhost:8000/dash"
        resources:
          limits:
            cpu: 500m
            memory: 512Mi
          requests:
            cpu: 100m
            memory: 256Mi
  destination:
    server: https://kubernetes.default.svc
    namespace: fastschema
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

Omit AppProject and Namespace if you already have them (e.g. via devops-apps ApplicationSet).

4. Test Access

Port-forward

kubectl port-forward svc/fastschema -n fastschema 8000:8000

Get setup token (first run)

kubectl logs -n fastschema -l app.kubernetes.io/name=fastschema --tail=30

Look for: Visit the following URL to setup the app: http://localhost:8000/dash/setup/?token=...

Setup and login

Open the URL from the logs. Complete setup to create admin account. Log in at http://localhost:8000/dash with the credentials you created.

5. Verify EBS Storage

kubectl exec -it -n fastschema deploy/fastschema -- sh

Inside the pod:

df -h /fastschema/data
ls -la /fastschema/data
echo "storage-test-$(date)" > /fastschema/data/storage-test.txt
cat /fastschema/data/storage-test.txt

Exit, then from your terminal:

kubectl delete pod -n fastschema -l app.kubernetes.io/name=fastschema

Wait for the new pod to be ready, exec in again and run:

cat /fastschema/data/storage-test.txt

If the file and content persist, the PVC is working.

6. Summary: Copy-Paste

# 1. Apply manifest
kubectl apply -f fastschema-application.yaml

# 2. Wait for sync (argocd app get fastschema)

# 3. Port-forward and get token
kubectl port-forward svc/fastschema -n fastschema 8000:8000 &
kubectl logs -n fastschema -l app.kubernetes.io/name=fastschema --tail=30

Open the setup URL from logs, create admin account, then log in at http://localhost:8000/dash.

7. Troubleshooting

Issue: No setup token in logs

Solution: Wait for pod to be ready. Token prints once on startup. If setup already done, use login at /dash with your credentials.

Issue: PVC stuck Pending

Solution: Ensure storageClassName: ebs-sc exists. Check EBS CSI driver is installed.

Issue: Service not found

Solution: kubectl get svc -n fastschema. Ensure app is synced.

8. References

FastSchema: https://fastschema.com
Configuration: https://fastschema.com/docs/configuration.html
Chart: https://github.com/k8sforge/fastschema-chart

Headlamp on EKS – Install and Test

John Ajera — Sat, 28 Feb 2026 22:21:50 +0000

Headlamp on EKS – Install and Test

Need a lightweight Kubernetes UI without the overhead of kubectl or cloud consoles? Headlamp runs in-cluster, gives you real-time visibility into workloads, logs, and events—and it's free and open source. This guide deploys it via Argo CD with Service Account token auth: no OIDC setup, no manual secrets, and access via simple port-forward. Secure, easy to run, and GitOps-friendly.

Note: This setup uses cluster-admin for simplicity—ideal for learning or individual use. For production, configure proper role-based access (RBAC) and restrict the Service Account to least-privilege roles.

1. Overview

What this guide does:

Deploys Headlamp using the official Helm chart via an Argo CD Application
Creates a Service Account headlamp with cluster-admin for token-based login
Persists data with EBS-backed PVC
Uses port-forward for local access (no ingress required)

Prerequisites:

EKS cluster running with kubectl context set
Argo CD installed

2. Prerequisites

Before starting, ensure you have:

kubectl configured with context set to your EKS cluster
Argo CD installed and syncing Applications

3. Install Headlamp

Save the manifest below as headlamp-application.yaml and apply:

kubectl apply -f headlamp-application.yaml

Argo CD will create the Application and sync Headlamp. Wait until the Application shows Synced in the Argo CD UI or CLI.

Manifest:

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: platform
  namespace: argocd
spec:
  clusterResourceWhitelist:
    - group: "*"
      kind: "*"
  destinations:
    - namespace: "*"
      server: "*"
  namespaceResourceWhitelist:
    - group: "*"
      kind: "*"
  sourceRepos:
    - "*"
---
apiVersion: v1
kind: Namespace
metadata:
  name: headlamp
  labels:
    name: headlamp
---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: headlamp
  namespace: argocd
spec:
  project: platform
  source:
    repoURL: https://kubernetes-sigs.github.io/headlamp/
    chart: headlamp
    targetRevision: 0.40.0
    helm:
      values: |
        service:
          type: ClusterIP
        config:
          inCluster: true
          oidc:
            secret:
              create: false
        serviceAccount:
          create: true
          name: headlamp
        clusterRoleBinding:
          clusterRoleName: cluster-admin
        persistentVolumeClaim:
          enabled: true
          size: 10Gi
          storageClassName: ebs-sc
          accessModes:
            - ReadWriteOnce
        resources:
          limits:
            cpu: 200m
            memory: 256Mi
          requests:
            cpu: 100m
            memory: 128Mi
  destination:
    server: https://kubernetes.default.svc
    namespace: headlamp
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

Note: The serviceAccount.create: true block creates the headlamp Service Account—this is what kubectl create token headlamp -n headlamp uses for login. Adjust storageClassName if your EKS cluster uses a different EBS storage class. Omit the AppProject and Namespace if you already have them.

4. Test Access

Port-forward

kubectl port-forward svc/headlamp -n headlamp 8080:80

Create token

kubectl create token headlamp -n headlamp

Login

Open http://localhost:8080 and paste the token when Headlamp prompts for authentication.

5. Summary: Copy-Paste

# 1. Apply manifest
kubectl apply -f headlamp-application.yaml

# 2. Wait for sync (check Argo CD UI or: argocd app get headlamp)

# 3. Port-forward and get token
kubectl port-forward svc/headlamp -n headlamp 8080:80 &
kubectl create token headlamp -n headlamp

Then open http://localhost:8080 and paste the token when prompted.

6. Troubleshooting

Issue: Application stuck in Syncing or OutOfSync

Solution: Check Argo CD logs and the Application status. Ensure the Headlamp Helm repo is reachable and storageClassName: ebs-sc exists in your cluster. If using a different storage class, update the manifest.

Issue: Token invalid or login fails

Solution: Ensure the Service Account exists: kubectl get sa headlamp -n headlamp. Re-run kubectl create token headlamp -n headlamp to generate a fresh token (tokens expire).

Issue: Argo CD ingress stuck or inaccessible

Solution: See your cluster's Argo CD troubleshooting docs.

7. References

Headlamp: https://headlamp.dev
Installation (Service Account token): https://headlamp.dev/docs/latest/installation/#create-a-service-account-token
Argo CD: https://argo-cd.readthedocs.io

Gatekeeper Disallow Default Namespace – Create, Deploy & Test Guide

John Ajera — Fri, 27 Feb 2026 21:40:15 +0000

Gatekeeper Disallow Default Namespace – Create, Deploy & Test Guide

A step-by-step guide to building and deploying a custom Gatekeeper policy that blocks workloads from being created in the Kubernetes default namespace.

1. Overview

What this guide does:

Blocks Pods, Services, Deployments, ConfigMaps, Secrets, and similar resources from being created in default
Runs in dryrun (audit-only) or deny (block) mode
Can be tuned to target specific resource types

Prerequisites:

Kubernetes cluster (1.19+)
Gatekeeper installed (Helm or manifest)
kubectl configured for your cluster

2. Prerequisites

Before starting, ensure you have:

A Kubernetes cluster (1.19+)
Gatekeeper installed (install guide)
kubectl configured for your cluster

Verify Gatekeeper is running:

kubectl get pods -n gatekeeper-system
kubectl get constrainttemplates

3. How It Works

Gatekeeper uses two resource types:

Resource	Purpose
ConstraintTemplate	Defines the policy logic (Rego) and the CRD for the constraint
Constraint	Instantiates the policy with match rules and enforcement mode

Gatekeeper's controller watches ConstraintTemplates, creates CRDs for them, and then Constraints of that kind can be applied. The order matters: ConstraintTemplate first, then Constraint.

4. Folder Structure

gatekeeper-policies/
├── constraint-templates/
│   └── disallow-default-namespace.yaml
└── constraints/
    └── disallow-default-namespace.yaml

5. Create the ConstraintTemplate

Create constraint-templates/disallow-default-namespace.yaml:

apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
  name: k8sdisallowdefaultnamespace
  annotations:
    metadata.gatekeeper.sh/title: "Disallow Default Namespace"
    metadata.gatekeeper.sh/version: 1.0.0
    description: "Disallows deploying namespaced resources to the default namespace."
spec:
  crd:
    spec:
      names:
        kind: K8sDisallowDefaultNamespace
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package k8sdisallowdefaultnamespace

        violation[{"msg": msg}] {
          input.review.object.metadata.namespace == "default"
          msg := "Deploying to the default namespace is not allowed"
        }

Key points:

metadata.name: Must be lowercase; Gatekeeper derives the constraint kind from it (K8sDisallowDefaultNamespace).
spec.crd.spec.names.kind: Defines the constraint kind created by this template.
package: Must match metadata.name (e.g. k8sdisallowdefaultnamespace).
violation: Returned when the policy fails. input.review.object is the resource being created or updated.

6. Create the Constraint

Create constraints/disallow-default-namespace.yaml:

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sDisallowDefaultNamespace
metadata:
  name: disallow-default-namespace
spec:
  enforcementAction: dryrun   # dryrun = audit only; deny = block
  match:
    kinds:
      - apiGroups: [""]
        kinds:
          - Pod
          - Service
          - ConfigMap
          - Secret
          - PersistentVolumeClaim
          - ServiceAccount
      - apiGroups: ["apps"]
        kinds:
          - Deployment
          - ReplicaSet
          - StatefulSet
          - DaemonSet
      - apiGroups: ["extensions", "networking.k8s.io"]
        kinds:
          - Ingress
      - apiGroups: ["batch"]
        kinds:
          - Job
          - CronJob

Key points:

kind: Must match the kind defined in the ConstraintTemplate.
enforcementAction:
- dryrun: Log violations only, do not block.
- deny: Block violating requests.
match.kinds: List of API groups and kinds to validate. Extend as needed.

7. Deploy

Apply in this order: template first, then constraint.

# 1. ConstraintTemplate (creates the CRD)
kubectl apply -f constraint-templates/disallow-default-namespace.yaml

# 2. Wait for the CRD to exist (a few seconds)
kubectl get crd k8sdisallowdefaultnamespace.constraints.gatekeeper.sh

# 3. Constraint
kubectl apply -f constraints/disallow-default-namespace.yaml

Verify:

kubectl get constrainttemplate k8sdisallowdefaultnamespace
kubectl get k8sdisallowdefaultnamespace

8. Test

Phase 1: Dryrun (audit only)

With enforcementAction: dryrun, violations are reported but not blocked.

Allowed:

kubectl create deployment nginx --image=nginx -n my-app

Violation (allowed but reported):

kubectl create deployment nginx --image=nginx
# Uses default namespace – creates successfully, but Gatekeeper logs a violation

Check violations:

kubectl get k8sdisallowdefaultnamespace disallow-default-namespace -o yaml

Look at status.violations.

Phase 2: Deny (block)

Switch to blocking mode:

kubectl patch k8sdisallowdefaultnamespace disallow-default-namespace --type=merge \
  -p '{"spec":{"enforcementAction":"deny"}}'

Or update the YAML and re-apply:

spec:
  enforcementAction: deny

Blocked:

kubectl create deployment nginx --image=nginx

Expected error:

Error from server ([disallow-default-namespace] Deploying to the default namespace is not allowed):
admission webhook "validation.gatekeeper.sh" denied the request: ...

Allowed:

kubectl create deployment nginx --image=nginx -n my-app

Quick test commands

kubectl run nginx --image=nginx → default namespace → Blocked
kubectl run nginx --image=nginx -n prod → prod → Allowed
kubectl create configmap foo --from-literal=k=v → default → Blocked
kubectl create configmap foo --from-literal=k=v -n staging → staging → Allowed

9. Production Considerations

Gradual rollout

Start with dryrun.
Confirm violations in status.violations or logs.
Fix or exempt workloads.
Switch to deny.

Excluding namespaces

To skip certain namespaces, add excludedNamespaces in the constraint:

spec:
  match:
    kinds: [ ... ]
    excludedNamespaces:
      - kube-system
      - gatekeeper-system
      - argocd

Argo CD

If using Argo CD:

Order: ConstraintTemplates before Constraints (e.g. sync-wave 1 vs 2).
Dry-run: Add argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true to constraints so sync succeeds when the CRD is not yet created.

Hardening the Rego

For DELETE requests, input.review.object may be empty. To avoid errors:

violation[{"msg": msg}] {
  object := input.review.object
  object != null
  object.metadata.namespace == "default"
  msg := "Deploying to the default namespace is not allowed"
}

10. Summary: Copy-Paste

# 1. Apply template
kubectl apply -f constraint-templates/disallow-default-namespace.yaml

# 2. Wait for CRD
kubectl get crd k8sdisallowdefaultnamespace.constraints.gatekeeper.sh

# 3. Apply constraint
kubectl apply -f constraints/disallow-default-namespace.yaml

# 4. Test dryrun (create resources in default, check violations)
kubectl get k8sdisallowdefaultnamespace disallow-default-namespace -o yaml

# 5. Switch to deny
kubectl patch k8sdisallowdefaultnamespace disallow-default-namespace --type=merge \
  -p '{"spec":{"enforcementAction":"deny"}}'

# 6. Test deny (confirm default namespace is blocked)
kubectl create deployment nginx --image=nginx
# Should fail with admission webhook denied

11. Troubleshooting

Issue: K8sDisallowDefaultNamespace.constraints.gatekeeper.sh not found

Solution: The constraint CRD does not exist yet. Gatekeeper creates it when it processes the ConstraintTemplate. Apply the ConstraintTemplate first, wait until kubectl get crd k8sdisallowdefaultnamespace.constraints.gatekeeper.sh succeeds, then apply the Constraint. With Argo CD, use SkipDryRunOnMissingResource=true on the constraint to avoid sync failures during this window.

Issue: ConstraintTemplate stuck or errors

Solution: Run kubectl describe constrainttemplate k8sdisallowdefaultnamespace and check status for Rego errors. Ensure package matches the template name.

Issue: No violations in dryrun

Solution: Confirm Gatekeeper admission webhook is active: kubectl get validatingwebhookconfigurations. Try a resource in the constraint's match.kinds. Inspect status on the constraint for violation details.

Issue: Policy blocks system components

Solution: Add their namespaces to excludedNamespaces (e.g. kube-system, gatekeeper-system).

12. References

Gatekeeper Installation: https://open-policy-agent.github.io/gatekeeper/website/docs/install/
Gatekeeper Documentation: https://open-policy-agent.github.io/gatekeeper/website/docs/

Grafana on EKS – Install and Test

John Ajera — Fri, 27 Feb 2026 07:06:42 +0000

Grafana on EKS – Install and Test

Deploy Grafana via Argo CD on EKS and access it locally with port-forward. The Application manifest includes persistence, Prometheus datasource, and health probes.

1. Overview

What this guide does:

Deploys Grafana using the official Helm chart via an Argo CD Application
Creates an AppProject and Application for the grafana namespace
Configures Prometheus as the default datasource (pre-configured for dashboards)
Persists data with EBS-backed PVC
Uses port-forward for local access (no ingress required)

Prerequisites:

EKS cluster running with kubectl context set
Argo CD installed
Prometheus installed

2. Prerequisites

Before starting, ensure you have:

kubectl configured with context set to your EKS cluster
Argo CD installed and syncing Applications
Prometheus running (e.g. kube-prometheus-stack-prometheus.kube-prometheus-stack.svc.cluster.local:9090—adjust the datasource URL if yours differs)

3. Install Grafana

Save the manifest below as grafana-application.yaml and apply:

kubectl apply -f grafana-application.yaml

Argo CD will create the Application and sync Grafana (Helm chart). Wait until the Application shows Synced in the Argo CD UI or CLI.

Manifest:

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: platform
  namespace: argocd
spec:
  clusterResourceWhitelist:
    - group: "*"
      kind: "*"
  destinations:
    - namespace: "*"
      server: "*"
  namespaceResourceWhitelist:
    - group: "*"
      kind: "*"
  sourceRepos:
    - "*"
---
apiVersion: v1
kind: Namespace
metadata:
  name: grafana
  labels:
    name: grafana
---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: grafana
  namespace: argocd
spec:
  project: platform
  source:
    repoURL: https://grafana.github.io/helm-charts
    chart: grafana
    targetRevision: 7.0.0
    helm:
      values: |
        deploymentStrategy:
          type: Recreate
        persistence:
          enabled: true
          type: pvc
          storageClassName: ebs-sc
          size: 10Gi
        service:
          type: ClusterIP
        adminUser: admin
        datasources:
          datasources.yaml:
            apiVersion: 1
            datasources:
              - name: Prometheus
                type: prometheus
                access: proxy
                url: http://kube-prometheus-stack-prometheus.kube-prometheus-stack.svc.cluster.local:9090
                isDefault: true
                editable: true
        resources:
          limits:
            cpu: 500m
            memory: 512Mi
          requests:
            cpu: 250m
            memory: 256Mi
        startupProbe:
          httpGet:
            path: /api/health
            port: 3000
          periodSeconds: 5
          failureThreshold: 18
        readinessProbe:
          httpGet:
            path: /api/health
            port: 3000
          periodSeconds: 10
          failureThreshold: 3
        livenessProbe:
          httpGet:
            path: /api/health
            port: 3000
          initialDelaySeconds: 30
          periodSeconds: 10
          failureThreshold: 10
  destination:
    server: https://kubernetes.default.svc
    namespace: grafana
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

Note: Adjust storageClassName if your EKS cluster uses a different EBS storage class. Ensure Prometheus URL matches your install (e.g. kube-prometheus-stack-prometheus.kube-prometheus-stack.svc.cluster.local:9090).

4. Test Access

Port-forward

kubectl port-forward svc/grafana -n grafana 3000:80

Get admin password

kubectl get secret grafana -n grafana -o jsonpath="{.data.admin-password}" | base64 -d
echo

Login

Open http://localhost:3000 and sign in with admin and the password from above.

Verify datasource

Go to Connections → Data sources. Prometheus should be configured and ready. Create a dashboard and run a query to confirm.

5. Summary: Copy-Paste

# 1. Apply manifest
kubectl apply -f grafana-application.yaml

# 2. Wait for sync (check Argo CD UI or: argocd app get grafana)

# 3. Port-forward and get password
kubectl port-forward svc/grafana -n grafana 3000:80 &
kubectl get secret grafana -n grafana -o jsonpath="{.data.admin-password}" | base64 -d && echo

Then open http://localhost:3000 and log in with admin / the password.

6. Troubleshooting

Issue: Application stuck in Syncing or OutOfSync

Solution: Check Argo CD logs and the Application status. Ensure the Grafana Helm repo is reachable and storageClassName: ebs-sc exists in your cluster. If using a different storage class, update the manifest.

Issue: Prometheus datasource connection failed

Solution: Verify Prometheus is running: kubectl get svc -n kube-prometheus-stack (or your Prometheus namespace). Update the url in the manifest to match your Prometheus service.

Issue: Argo CD ingress stuck or inaccessible

Solution: See Argo CD Ingress Stuck – Find & Fix for troubleshooting steps.

7. References

Grafana Helm Chart: https://github.com/grafana/helm-charts
Grafana Documentation: https://grafana.com/docs/grafana/latest/
Argo CD: https://argo-cd.readthedocs.io/