Forem: Omar Ahmed

Docker Production Best Practices - Complete Guide

Omar Ahmed — Fri, 06 Feb 2026 21:00:05 +0000

Part 1 : Image Selection & Architecture

1. Minimal Base Images

Containers are not Virtual Machines. They share the host kernel.

# ❌ BAD - Large attack surface (800MB+)
FROM ubuntu:22.04

# ✅ GOOD - Minimal attack surface (5-50MB) Alpine / Distroless
FROM alpine:3.19
# OR
FROM gcr.io/distroless/static-debian12

Rule:

Every package you don't include is a package that cannot have vulnerabilities.
Compiled ( Go , Rust ) Use Distroless or Scratch. Compile to a static binary, no runtime dependencies needed, only need the binary file.
Interpreted (Node, Python) Use Alpine or Slim variants. You only need the language runtime, not the OS tools.

2. Separate the Builder from the Runner - Multi-Stage Builds

We don't need the source code, compilers, dependencies, or test files in production, so we need to separate the build stage from the runtime stage. The final image discards all build tools.

FROM node:alpine AS builder
WORKDIR /app
COPY . .
RUN npm install && npm run build

FROM nginx:alpine
COPY --from=builder /app/dist /usr/share/nginx/html

┌─────────────────┐                      ┌─────────────────┐
│                 │                      │                 │
│  ┌──────────┐   │                      │  ┌──────────┐   │
│  │  Source  │   │   COPY --from=       │  │   App    │   │
│  └──────────┘   │     builder          │  │  Binary  │   │
│                 │  ─────────────────>  │  └──────────┘   │
│  ┌──────────┐   │                      │                 │
│  │ Compiler │   │                      │                 │
│  └──────────┘   │                      │                 │
│                 │                      │                 │
└─────────────────┘                      └─────────────────┘
Stage 1: Builder                        Stage 2: Runner

3. Derive Versions, Don't Guess

Always match Dockerfile base image versions to your project configuration files - Use specific version tags, not latest .

Your project config is the Source of Truth. The Dockerfile must mirror it. Mismatched versions lead to "It works on my machine" but fails in production.

  ┌─────────────┐
  │ package.json│
  │             │
  │  engines:   │
  │  node >= 20 │──────────────────────> FROM node:20-alpine
  │             │
  └─────────────┘


  ┌─────────────┐
  │   go.mod    │
  │             │
  │  go 1.21    │──────────────────────> FROM golang:1.21
  │             │
  └─────────────┘


  ┌─────────────┐
  │   Pipfile   │
  │             │
  │ python_     │
  │ version =   │──────────────────────> FROM python:3.11-slim
  │   3.11      │
  └─────────────┘

Part 2 : Optimization - Build Speed & Layer Caching

1. Combine Install and Clean Steps

RUN npm install
RUN npm cache clean --force

This creates TWO separate layers in the Docker image:

The first RUN command installs packages and creates cache files in Layer 1
The second RUN command cleans the cache in Layer 2
However, the cache files from Layer 1 are only hidden, not deleted
They still exist in the image and contribute to the total image size
This wastes disk space and increases image size unnecessarily

The Solution (Production Way)

RUN npm ci --omit=dev && npm cache clean --force

Both commands execute in a single layer
The cache files are created and then immediately cleaned within the same layer
The temporary cache files never commit to the final image history

Key Concept

Docker layers are additive. You cannot delete data from a previous layer, only mask it. When you delete files in a subsequent layer, they're hidden but still consume space in the image. To truly remove temporary files, you must clean them up in the same RUN command where they were created using && to chain commands together.

Other Example

# ❌ BAD - Two layers
RUN apt-get update
RUN apt-get install -y package && rm -rf /var/lib/apt/lists/*

# ✅ GOOD - Single layer
RUN apt-get update && \
    apt-get install -y package && \
    rm -rf /var/lib/apt/lists/*

2. Be Explicit with Files and Dependencies

No Lazy Copying - Avoid COPY . . in the final stage.

Using COPY . . copies everything from your project directory into the container, including:

.env files (sensitive credentials)
.git directory (version control history)
Test files
Documentation
Development configs
Build artifacts you don't need

The Solution:

Copy ONLY what is needed:

# ❌ BAD - Copies everything
COPY . .

# ✅ GOOD - Explicit and selective
COPY ./dist ./dist

Best Practice:

Be explicit about which files and directories you copy
Only include runtime-necessary files
Use .dockerignore to exclude unwanted files
Copy built artifacts from builder stage, not source files

3. Production Dependencies Only

Dev tools do not belong in production.

Use npm ci --omit=dev (or equivalent for your package manager)

# ❌ BAD - Installs all dependencies including dev
RUN npm install

# ✅ GOOD - Production dependencies only
RUN npm ci --omit=dev

Part 3 : Security Hardening

1. Never Run as Root

By default, Docker containers run as the root user (UID 0), which poses significant security risks. Always create and switch to a non-root user in production containers.

RUN addgroup -S appgroup && \
    adduser -S appuser -G appgroup
USER appuser

Security Risk

If an attacker compromises a container running as root, they potentially gain root privileges over the host.

This means:

Full control over the container
Possible escape to the host system
Access to sensitive host resources
Ability to compromise other containers

2. The Latest Tag is a Lie

No version control
No reproducibility
Debugging nightmares
Unexpected breaking changes
Security vulnerabilities from unknown versions

# ❌ BAD - Today you get Node 20, tomorrow you might get Node 21
FROM node:latest

# ✅ GOOD - Always Node 20.11.0 on Alpine 3.19
FROM node:20.11.0-alpine3.19

# ✅ ACCEPTABLE - Gets patch updates but stays on 20.x
FROM node:20-alpine

3. Random images from Docker Hub

Unknown Source can contain malware, crypto-miners, or backdoors.

4. Remove the Attacker's Toolkit

If an attacker gets in, don't hand them the tools to explore.

curl, wget, vim , netcat - These tools allow attackers to explore the network, download malware. If the tool isn't there, their job becomes significantly harder.

Question: If I strip all tools from my production container, how do I debug a DNS issue?

Instead of bloating your production container with debugging tools, use an Ephemeral Sidecars - temporary sidecar container that attaches to your production container's network namespace only when needed.

# Find your container ID or name
docker ps

# Run netshoot (debugging toolbox) in the same network namespace
docker run -it --rm \
  --network container:my-app-container \
  nicolaka/netshoot

# Now you have access to debugging tools:
# - curl http://api.example.com
# - dig google.com
# - nslookup database-service
# - ping redis
# - tcpdump

Simply exit the sidecar container - it's removed automatically (due to --rm flag), leaving your production container pristine.

Other Example

# network debugging
docker run -it --rm --network container:my-app nicolaka/netshoot

# basic Unix tools
docker run -it --rm --network container:my-app busybox

# alpine with tools
docker run -it --rm --network container:my-app \
  alpine sh -c "apk add --no-cache curl bind-tools && sh"

5. Root Owns, User Executes

This security pattern ensures that even if an attacker compromises your application, they cannot establish a permanent foothold by modifying the application binary itself.

If the application is compromised:

Attacker gains access to the running container as appuser
Attacker tries to inject backdoor into the application binary
Attack fails because appuser has no write permissions
Result: Attacker cannot establish persistence through binary modification

Entity	Permission	Impact
Owner (Root)	Read / Write	Can update app ✅ (during build)
User (AppUser)	Read / Execute	Can RUN app, cannot MODIFY app ❌

Implementation

FROM node:20-alpine AS builder
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
RUN npm run build

FROM node:20-alpine
# Create non-root user
RUN addgroup -S appgroup && adduser -S appuser -G appgroup
WORKDIR /app

# Copy files as root, set ownership to root
COPY --from=builder --chown=root:root /app/dist ./dist
COPY --from=builder --chown=root:root /app/node_modules ./node_modules

# Make binaries executable but not writable
RUN chmod 755 dist/

# Switch to non-root user for execution
USER appuser

CMD ["node", "dist/index.js"]

Attacker → Backdoor.sh → [×] App Binary (Read Only) → ✓ Protected

Part 4: Maintainability - Dev Experience & Documentation

1. Sort Arguments

Alphabetize multi-line installs for cleaner git diffs.

# ❌ BAD - Unsorted, harder to track changes
RUN apt-get update && apt-get install -y \
    curl \
    git \
    jq \
    python3 \
    zip

# ✅ GOOD - Alphabetically sorted (A→Z)
RUN apt-get update && apt-get install -y \
    curl \
    git \
    jq \
    python3 \
    zip

Why this matters:

Cleaner git diffs - Easy to see what was added/removed
Prevents duplicates - Sorted list makes duplicates obvious
Easier reviews - Reviewers can quickly scan alphabetically
Merge conflicts - Reduces conflicts when multiple people edit

Example git diff with sorted packages:

RUN apt-get update && apt-get install -y \
    curl \
    git \
+   htop \
    jq \
    python3 \

2. Use WORKDIR

Don't use cd. It resets every layer ( Context Reset: Layer boundary). WORKDIR sets persistent context (Set once, applies to all following).

FROM node:20-alpine
RUN cd /app
COPY package.json .
RUN npm install

Problem: The cd /app command only affects that single RUN layer. The next layer (COPY) resets back to the root directory, so files go to the wrong location.

3. OCI Labels

Tag images with metadata (source, version, description) for easier registry identification.

OCI labels are key-value pairs that provide metadata about your Docker image.

Implementation

FROM node:20-alpine

# Add OCI labels for metadata
LABEL org.opencontainers.image.title="My Application"
LABEL org.opencontainers.image.description="A production-ready Node.js application"
LABEL org.opencontainers.image.version="1.2.3"
LABEL org.opencontainers.image.authors="team@example.com"
LABEL org.opencontainers.image.source="https://github.com/myorg/myapp"
LABEL org.opencontainers.image.created="2026-02-06T12:00:00Z"
LABEL org.opencontainers.image.revision="abc123def"
LABEL org.opencontainers.image.licenses="MIT"

WORKDIR /app
COPY package*.json ./
RUN npm ci --omit=dev
COPY . .
USER node
CMD ["node", "index.js"]

Automated Label Injection (CI/CD)

# Use ARG to accept build-time variables, ARG vars Exist only during build, Can be overridden with --build-arg, Have default values (so local builds don't fail)
ARG VERSION=dev
ARG GIT_COMMIT=unknown
ARG BUILD_DATE=unknown

LABEL org.opencontainers.image.version="${VERSION}"
LABEL org.opencontainers.image.revision="${GIT_COMMIT}"
LABEL org.opencontainers.image.created="${BUILD_DATE}"

# Build with metadata from CI/CD
docker build \
  --build-arg VERSION=$(git describe --tags) \
  --build-arg GIT_COMMIT=$(git rev-parse HEAD) \
  --build-arg BUILD_DATE=$(date -u +'%Y-%m-%dT%H:%M:%SZ') \
  -t myapp:latest .

4. Handle Signals Gracefully

Always use JSON array syntax (exec form) for CMD and ENTRYPOINT in Dockerfiles or similar container configurations. This ensures your application receives system signals and can shut down gracefully, which is crucial for production reliability.

CMD npm start  # ❌ BAD

The Problem: When you use shell form syntax like CMD npm start, Node.js spawns your command through a shell (like /bin/sh). This creates a process hierarchy where:

The shell becomes PID 1 (the main process)
Your actual Node app runs as a child process (PID 2)

When a SIGTERM signal is sent (typically during shutdown, like docker stop), it goes to PID 1 (the shell). However, shells don't forward signals to their children by default, so:

the shell ( main process ) will kill, so a children process will forced kill ( not graceful shutdown )
The Node app never receives the signal
The app can't perform graceful shutdown (close connections, finish requests, cleanup)
The system eventually force-kills the app after a timeout
This can lead to data loss or corrupted state

The Solution

CMD ["npm", "start"]

When you use exec form syntax like CMD ["npm", "start"], Node.js runs your command directly without a shell wrapper. This means:

Your Node app becomes PID 1 directly
SIGTERM is sent directly to your application
Your app can handle the signal properly (using event listeners like process.on('SIGTERM'))
The app can perform graceful shutdown operations
Clean, controlled shutdown is achieved

5. Secrets Do Not Belong in Dockerfiles

If a secret goes into a Docker image, it's already leaked.

✅ DO:

Inject secrets at runtime via environment variables

# Runtime injection
docker run \
  -e DATABASE_PASSWORD=$DB_PASS \
  -e API_KEY=$API_KEY \
  myapp:latest

Use secret management tools (AWS Secrets Manager, Vault, K8s Secrets)
Mount secrets as volumes from secure storage
Use Docker secrets in Swarm mode
Keep secrets in .gitignore and .dockerignore

❌ DON'T:

Hardcode secrets in Dockerfile
COPY secret files into the image
Use ENV for sensitive data
Commit .env files with real secrets to git
Assume deleting in a later layer removes the secret

Thanks for reading! If you found this helpful, please share it with your team.

Vim

Omar Ahmed — Sat, 27 Dec 2025 12:57:40 +0000

1. Match all occurrences of the word under the cursor

2. Replace a word with another word

:%s/word/newword/gc

g: global , c: confirm

3. Visually select a word i / a (inside / outside)

viw ( Selects only the word itself )
vaw ( Selects the word plus surrounding whitespace )
vi( ( Visual texts inside () )
ci" ( change around double "" )
va( ( Visual () )
va" ( Visual "" )

4. Show all registers

:reg

5. Paste from a specific register

"<reg number>p

6. Yank (copy) the current line to the system clipboard ( + : clipboard system register)

"+yy

7. Yank the current file name to the system clipboard

:let @+=@% , then go to any file and ctrl+v

assigns the contents of the % register (current file name) to the + register (system clipboard).

8. Copy the absolute path of the current file

:let @+ = expand('%:p')

9. Record and play a macro

q<any character><action>q
@h

10. :normal Mode

When you select lines in Visual mode and press :
:'<,'>

:'<,'>normal <keys><text>

Keys:
I : go to start of lines and enter Insert mode
A : go to end of lines and append a new text

hello
world
goodbye

:'<,'>normal Ivar + space

var hello
var world
var goodbye

hello
world
goodbye

:'<,'>normal A ;

hello ;
world ;
goodbye ;

11. Number

Ctrl-a : increment number
Ctrl-x : decrement number
value = 2

list.get(0);
list.get(0);
list.get(0);
list.get(0);
list.get(0);
list.get(0);

select all lines + g + Ctrl-a

list.get(1);
list.get(2);
list.get(3);
list.get(4);
list.get(5);
list.get(6);

12. Switching Selection

select lines + o
select lines + O

13. w / W letters, digits, or underscores / whitespace characters

Lis:st.get(0) hello world

GitLab-CI

Omar Ahmed — Wed, 01 Oct 2025 18:16:35 +0000

GitLab Docs: CI/CD YAML syntax reference

gitlab cicd file name

In GitLab, the CI/CD pipeline configuration file is always named:

.gitlab-ci.yml

It must be placed in the root directory of your repository.

template .gitlab-ci.yml

pipeline
stages
job1:
    before_script
        step
    script
        step
    after_script
        step
job2:
    script
        step

workflow:
  name: Test
  rules: 
    - if: $CI_COMMIT_BRANCH == 'main'

stages:
  - test
  - deploy

test_job:
  stage: test
  tags: 
    - saas-linux-samll-amd64
  before_script:
    - chmod +x test.sh
  script:
    - ./test.sh
  after_script:
    - ls

deploy_job:
  stage: deploy
  script:
    - touch test.txt

Multi Lines in the step

workflow:
  name: Test

test_job:
  before_script:
    - chmod +x test.sh
  script:
    - ./test.txt
    - >
      this is a test pipeline
      so do not take it seriously > test.txt
  after_script:
    - echo "done"

stages

Use stages to define stages that contain groups of jobs. Use stage in a job to configure the job to run in a specific stage.
If stages is not defined in the .gitlab-ci.yml file, the default pipeline stages are:

.pre
build
test
deploy
.post

The order of the items in stages defines the execution order for jobs:

Jobs in the same stage run in parallel.
Jobs in the next stage run after the jobs from the previous stage complete successfully.

If a pipeline contains only jobs in the .pre or .post stages, it does not run. There must be at least one other job in a different stage.

Keyword type: Global keyword.

Example of stages:

stages:
  - build
  - test
  - deploy

In this example:

All jobs in build execute in parallel.
If all jobs in build succeed, the test jobs execute in parallel.
If all jobs in test succeed, the deploy jobs execute in parallel.
If all jobs in deploy succeed, the pipeline is marked as passed. If any job fails, the pipeline is marked as failed and jobs in later stages do not start. Jobs in the current stage are not stopped and continue to run.

Artifacts - Storing Job Data

Use artifacts to specify which files to save as job artifacts. Job artifacts are a list of files and directories that are attached to the job when it succeeds, fails, or always.

The artifacts are sent to GitLab after the job finishes. They are available for download in the GitLab UI

By default, jobs in later stages automatically download all the artifacts created by jobs in earlier stages. You can control artifact download behavior in jobs with dependencies.

When using the needs keyword, jobs can only download artifacts from the jobs defined in the needs configuration.

job:
  artifacts:
    name: "job1-artifacts-file"
    when: on_success
    expire_in: 1 week
    paths:
      - binaries/
      - .config

This example stores all files in binaries/, but not *.o files located in subdirectories of binaries/.

artifacts:
  paths:
    - binaries/
  exclude:
    - binaries/**/*.o

Using needs Keyword

You can use the needs keyword to create dependencies between jobs in a pipeline. Jobs run as soon as their dependencies are met, regardless of the pipeline’s stages configuration. You can even configure a pipeline with no stages defined (effectively one large stage) and jobs still run in the proper order.

Working with Variables

Variables can be defined in a CI/CD job, or as a top-level (global) keyword to define default CI/CD variables for all jobs.

1- job variables:

review_job:
  variables:
    DEPLOY_SITE: "https://dev.example.com/"
    REVIEW_PATH: "/review"
  script:
    - deploy-review-script --url $DEPLOY_SITE --path $REVIEW_PATH

2- Default variables:

variables:
  DEPLOY_SITE: "https://example.com/"

deploy_job:
  stage: deploy
  script:
    - deploy-script --url $DEPLOY_SITE --path "/"
  environment: production

deploy_review_job:
  stage: deploy
  variables:
    DEPLOY_SITE: "https://dev.example.com/"
    REVIEW_PATH: "/review"
  script:
    - deploy-review-script --url $DEPLOY_SITE --path $REVIEW_PATH
  environment: production

In this example:

deploy_job has no variables defined. The default DEPLOY_SITE variable is copied to the job and can be used in the script section.
deploy_review_job already has a DEPLOY_SITE variable defined, so the default DEPLOY_SITE is not copied to the job. The job also has a REVIEW_PATH job variable defined. Both job variables can be used in the script section.

Masking Variables using CICD Settings

masking a variable means hiding its value in the job logs so it doesn’t get exposed when printed (for example, passwords, API tokens, or other secrets).
1- Go to your project in GitLab > Settings → CI/CD → Variables > Add variable > Key: MY_SECRET_TOKEN > Value: the secret value

Enable:
- Masked → hides it in logs
- Protected (optional) → makes it available only on protected branches/tags

2- Use the variable in your .gitlab-ci.yml:

stages:
  - test

print-secret:
  stage: test
  script:
    - echo "Using secret token..."
    - echo "$MY_SECRET_TOKEN"

the logs will show something like:

Using secret token...
[MASKED]

Predefined CI/CD Variables

Predefined variables become available at three different phases of pipeline execution:

Pre-pipeline: Pre-pipeline variables are available before the pipeline is created. These variables are the only variables that can be used with include:rules to control which configuration files to use when creating the pipeline.
Pipeline: Pipeline variables become available when GitLab is creating the pipeline. Along with pre-pipeline variables, pipeline variables can be used to configure rules defined in jobs, to determine which jobs to add to the pipeline.
Job-only: These variables are only made available to each job when a runner picks up the job and runs it.
Predefined variables

print-predefined:
  stage: test
  script:
    - echo "Running on branch: $CI_COMMIT_BRANCH"
    - echo "Commit SHA: $CI_COMMIT_SHA"
    - echo "Pipeline ID: $CI_PIPELINE_ID"

GitLab Merge Requests

A Merge Request proposes changes from one branch into another (e.g., feature/foo → main).

Merge Request & rules at Job level: Use rules to include or exclude jobs in pipelines.
the keyword rules at the job level controls when a job should run (or not run) based on conditions like branch, pipeline source, variables, etc

Rules are evaluated when the pipeline is created, and evaluated in order. When a match is found, no more rules are checked and the job is either included or excluded from the pipeline depending on the configuration. If no rules match, the job is not added to the pipeline.

rules accepts an array of rules. Each rules must have at least one of:

if
changes
exists
when

merge_request_predefined_variables:
  rules:
  # CI_PIPELINE_SOURCE tells you why a pipeline was created.It’s a predefined variable that GitLab automatically sets for every job.
  # https://docs.gitlab.com/ci/jobs/job_rules/#ci_pipeline_source-predefined-variable
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
  script: |
    echo "CI_MERGE_REQUEST_LABELS - $CI_MERGE_REQUEST_LABELS"
    echo "CI_MERGE_REQUEST_TARGET_BRANCH_NAME - $CI_MERGE_REQUEST_TARGET_BRANCH_NAME"
    echo "CI_MERGE_REQUEST_ASSIGNEES - $CI_MERGE_REQUEST_ASSIGNEES"
    echo "CI_MERGE_REQUEST_SOURCE_BRANCH_NAME - $CI_MERGE_REQUEST_SOURCE_BRANCH_NAME"
    echo "CI_MERGE_REQUEST_TITLE - $CI_MERGE_REQUEST_TITLE"

job_main_branch:
  rules:
    - if: '$CI_COMMIT_BRANCH == "main"'
  script:
    - echo "I run only on main"

Use rules at Workflow Level

The rules keyword in workflow is similar to rules defined in jobs, but controls whether or not a whole pipeline is created. When no rules evaluate to true, the pipeline does not run.

Supported values: You can use some of the same keywords as job-level rules:

rules: if.
rules: changes.
rules: exists.
when, can only be always or never when used with workflow.
variables.

workflow:
  rules:
    - if: $CI_COMMIT_TITLE =~ /-draft$/ # =~: match, $: end
      when: never
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH == 'main'
       variables:
           DEPLOY_VARIABLE: PRODUCTION

In this example, pipelines run if the commit title (first line of the commit message) does not end with -draft and the pipeline is for either:

A merge request
The main branch, Then set a job-level variable DEPLOY_VARIABLE with the value "PRODUCTION"

workflow:
  name: Exploring GitLab CI Concepts
  rules:
    - if: '$CI_COMMIT_BRANCH == "main"'
      variables:
        DEPLOY_VARIABLE: "PRODUCTION"

    - if: '$CI_MERGE_REQUEST_SOURCE_BRANCH_NAME =~ /^feature/ && $CI_PIPELINE_SOURCE == "merge_request_event"' # ^: start with feature word
      changes:
        - README.md
      variables:
        DEPLOY_VARIABLE: "TESTING"

deploy-job:
  script:
    - echo "Deploying application..."
    - echo "Application successfully deployed to $DEPLOY_VARIABLE environment"

What this does:

First rule → if branch is main, it sets DEPLOY_VARIABLE=PRODUCTION.
Second rule → if source branch starts with feature and pipeline comes from a merge request and README.md changed, then DEPLOY_VARIABLE=TESTING. Job output → prints the deployment target environment (PRODUCTION or TESTING).

Resource Groups

Use resource_group to create a resource group that ensures a job is mutually exclusive across different pipelines for the same project.
For example, if multiple jobs that belong to the same resource group are queued simultaneously, only one of the jobs starts. The other jobs wait until the resource_group is free.
If two jobs have the same resource group, GitLab runs them one after the other (queue).
If they have different resource groups, they can run at the same time.
Process modes

migrate_db:
  stage: deploy
  resource_group: production
  script: ./migrate.sh

deploy_app:
  stage: deploy
  resource_group: production
  script: ./deploy.sh

Both jobs affect the same production database/app.
If two pipelines run migrations + deploy at the same time → chaos.
By sharing resource_group: production, GitLab ensures these always run one after another, never overlapping.

Resource groups control concurrency across pipelines:

Pipelines run in parallel when multiple developers push or open merge requests.
Two pipelines can both reach the deploy stage at the same time.
Stages don’t prevent different pipelines from overlapping.

Example problem:

Alice pushes → Pipeline A runs → reaches deploy.
Bob pushes → Pipeline B runs at the same time → also reaches deploy.
Both deploy jobs run together → race condition (two versions deploying to the same server).

Resource groups solve this:

If Pipeline A is deploying, Pipeline B’s deploy job is queued until A finishes.
Prevents overlapping deployments to the same environment.
Works across all pipelines, not just within one. ### Job Timeout

sleep-job:
  stage: test
  script:
    - echo "Starting long process..."
    - sleep 120   # pretend this takes 2 minutes
    - echo "Process finished!"
  timeout: 1 minute

What happens:

The job runs sleep 120 (2 minutes).
But the job has timeout: 1 minute.
After 1 minute, GitLab kills the job with a timeout error.

Parallel Matrix

parallel matrix in GitLab CI/CD lets you run the same job multiple times in parallel with different variable values

This runs 3 jobs in parallel, one for each Python version:

test_python:
  stage: test
  image: python:$PYTHON_VERSION
  script:
    - python --version
    - pytest
  parallel:
    matrix:
      - PYTHON_VERSION: ["3.8", "3.9", "3.10"]

This expands to 4 jobs:

Node 16 + Ubuntu
Node 16 + Alpine
Node 18 + Ubuntu
Node 18 + Alpine

test_node:
  stage: test
  script:
    - node --version
    - npm test
  parallel:
    matrix:
      - NODE_VERSION: ["16", "18"]
        OS: ["ubuntu", "alpine"]

3 parallel jobs → each runs tests against a different database:

integration_tests:
  stage: test
  script:
    - ./run_tests.sh --db=$DB
  parallel:
    matrix:
      - DB: ["mysql", "postgres", "sqlite"]

Skip Pipeline

If your commit message contains [ci skip] or [skip ci], GitLab won’t create a pipeline.

git commit -m "Update README [ci skip]"
git push

skip jobs if the only change is in README.md:

job_build:
  stage: build
  script:
    - echo "Building app..."
  rules:
    - changes:
        - "README.md"
      when: never   # don't run if only README changed
    - when: on_success   # run for everything else

GitOps with ArgoCD

Omar Ahmed — Sat, 13 Sep 2025 13:28:27 +0000

gitopsWithArgoCD-Docs
gitops-argocd-Repo

1. What is GitOps?

GitOps is a modern approach to managing and deploying infrastructure and applications using Git as the single source of truth.
Instead of manually applying changes to servers or clusters, you declare the desired system state in Git, and an automated tool keeps your running systems in sync with that state.

Core Principles :

Declarative configuration: All infrastructure and application configs are stored as code in Git repositories (e.g., Kubernetes manifests, Helm charts, Terraform modules).
Version-controlled: Every change is done through Git commits, pull requests, and code reviews—giving you full history, auditability, and rollback capability.
Automated reconciliation: Specialized GitOps controllers (like Argo CD or Flux) continuously watch the Git repo and the live environment. If something drifts, they automatically sync it back to match the Git state. The GitOps operator ArgoCD also makes sure that the entire system is self-healing to reduce the risk of human errors. The operator continuously loops through three steps, observe, diff, and act. In the observe step, it checks the Git repository for any changes in the desired state. In the diff step, it compares the resources received from the previous step the observe step to the actual state of the cluster ( It performs a diff (comparison) between the desired state (Git) and the actual state (cluster), Any mismatch is called “drift” ). And in the Act step, it uses a reconciliation function and tries to match the actual state to the desired state ( If a drift is found, the operator runs its reconciliation logic, It applies the necessary changes to bring the cluster back to match Git, This ensures the system self-heals from manual or accidental changes ).

Desired State: Git , Actual State: Kubernetes Cluster

Continuous deployment: Merging to the main branch becomes the trigger for deployments, replacing manual kubectl apply or ad-hoc scripts.

Benefits :

Full audit trail of all infrastructure and app changes.
Rollback is as simple as reverting a Git commit.
Strong security (no direct access to production clusters needed - pull the desired state from Git and apply it in one or more environments or clusters)
Enables collaboration and consistent workflows across teams.

Push vs Pull based Deployments :

1. Push-based Deployment :
How it works :

The CI/CD system (like Jenkins) builds the code, creates container images, pushes them to a registry, and then directly applies manifests to the Kubernetes cluster using kubectl apply.
The CI/CD system has Read-Write (RW) access to the cluster.

Key points from the image :

✅ Easy to deploy Helm charts.
✅ Easy to inject container version updates via the build pipeline.
✅ Simpler secret management from inside the pipeline.
❌ Cluster config is embedded in the CI system (tightly coupled).
❌ CI system holds RW access to the cluster (security risk).
❌ Deployment approach is tied to the CD system (less flexible).

2. Pull-based Deployment (GitOps) :
How it works :

The CI system builds and pushes images to a registry.
The desired manifests are committed to a Git repository.
A GitOps operator (like Argo CD) inside the cluster pulls the manifests from Git and syncs them to the cluster, reconciling continuously.

Key points from the image :

✅ No external user/client can modify the cluster (only GitOps operator can).
✅ Can scan container registries for new versions.
✅ Secrets can be managed via Git repo + Vault.
✅ Not coupled to the CD pipeline — independent.
✅ Supports multi-tenant setups.
❌ Managing secrets for Helm deployments is harder.
❌ Generic secret management is more complex.

2. ArgoCD Basics

What is Argo CD?
Argo CD (Argo Continuous Delivery) is a GitOps tool for Kubernetes that:

Runs inside your cluster as a controller.
Continuously monitors a Git repository that stores your Kubernetes manifests (YAML, Helm, Kustomize, etc.).
Automatically applies and syncs those manifests to the cluster.
Provides a web UI, CLI, and API to visualize and manage application deployments.

Installation :

helm repo add argo https://argoproj.github.io/argo-helm
helm repo update
kubectl create namespace argocd
helm install argocd argo/argo-cd -n argocd

# Optional: Specify custom values
helm show values argo/argo-cd > values.yaml
helm install argocd argo/argo-cd -n argocd -f values.yaml

kubectl port-forward svc/argocd-server --address=0.0.0.0 -n argocd 8080:443
# Get the Initial Admin Password
kubectl -n argocd get secret argocd-initial-admin-secret \
  -o jsonpath="{.data.password}" | base64 -d; echo

Why use Argo CD?
Argo CD solves common deployment and operations problems by embracing GitOps principles:

Problem	How Argo CD Helps
Manual `kubectl apply` steps	Automates syncing from Git
Configuration drift (manual changes)	Detects drift and self-heals by reverting to Git
Hard to audit who changed what	Uses Git history as the single source of truth
CI pipelines need cluster credentials	Removes this risk — cluster pulls from Git
Slow feedback on deployment status	Real-time UI with health, sync status, and diffs

ArgoCD Concepts & Terminology :

Term	Description
Application	A group of Kubernetes resources as defined by a manifest. An application is a Custom Resource Definition, which represents a deployed application instance in a cluster. An application in ArgoCD is defined by two key pieces of information, the source `Git repo where is the desired state of a kubernetes manifest` and the destination for your Kubernetes resources `where the resources should be deployed in a kubernetes`.
Application source type	The tool used to build the application. E.g., Helm, Kustomize, or Ksonnet.
Project	Provides a logical grouping of applications, useful when Argo CD is used by multiple teams.
Target state	The desired state of an application, as represented by files in a Git repository.
Live state	The live state of that application. What pods, configmaps, secrets, etc. are created/deployed in a Kubernetes cluster.
Sync status	Whether or not the live state matches the target state. Is the deployed application the same as Git says it should be?
Sync	The process of making an application move to its target state (e.g., by applying changes to a Kubernetes cluster).
Sync operation status	Whether or not a sync succeeded.
Refresh	Compare the latest code in Git with the live state to figure out what is different.
Health	The health of the application — is it running correctly? Can it serve requests?

What is an Argo CD Application?
An Application is the core deployment unit in Argo CD.
It represents a set of Kubernetes resources defined in a Git repository and tells Argo CD:

Where to get the manifests (Git repo + path + branch)
Where to deploy them (cluster + namespace)
How to manage them (sync policies like auto-sync, self-heal)

Creating an Application :
1.Using CLI

argocd app create color-app \
  --repo https://github.com/sid/app-1.git \
  --path team-a/color-app \
  --dest-namespace color \
  --dest-server https://kubernetes.default.svc

Explanation:
--repo: Git repository URL containing the manifests.
--path: Path inside the repo where the manifests are stored.
--dest-namespace: Target namespace in the cluster.
--dest-server: Target Kubernetes API server (usually the same cluster).

argocd login localhost:8080 --username admin --password <password>
argocd app create solar-system-app \
  --repo https://github.com/sidd-harth/gitops-argocd \
  --path ./solar-system \
  --dest-namespace solar-system \
  --dest-server https://kubernetes.default.svc

argocd app list
argocd app sync solar-system-app
argocd app get solar-system-app
# check using kubectl with the namespace that argocd deployed in it
kubectl get app -n gitops
kubectl describe app -n gitops

2.Using a YAML manifest (color-app.yaml)

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: color-app
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/sid/app-1.git
    targetRevision: HEAD
    path: team-a/color
  destination:
    server: https://kubernetes.default.svc
    namespace: color
  syncPolicy:
    automated:
      selfHeal: true
    syncOptions:
    - CreateNamespace=true

Key fields:

metadata.name: Application name shown in Argo CD UI.
spec.source: Where manifests come from (repo, branch, path).
spec.destination: Which cluster and namespace to deploy to.
syncPolicy.automated: Enables auto-sync and self-healing.
syncOptions.CreateNamespace=true: Automatically create the namespace if it doesn’t exist.

3.using UI

Click “+ NEW APP”
Application Name: A unique name (e.g. color-app)
Project: Select default (unless you created custom projects)
Sync Policy:
- Choose Manual or Automatic
- Enable Self Heal and Prune if you want Argo CD to auto-fix drift and delete removed resources
Specify the Source (Git Repo)
- Repository URL or Added it using settings => Repositories => Connect Repo => Via HTTP/HTTPS => Type: git => Name: Any Name => Project: default => Repository URL then check the Connection Status
- Revision: HEAD (or branch/tag name like main)
- Path: The path in the repo (./solar-system)
Specify the Destination (Cluster + Namespace)
- Cluster URL:
  - Use https://kubernetes.default.svc for the same cluster Argo CD is running in
- Namespace: solar-system
  - Check “Create Namespace” if it does not exist
Review and Create
- Click “Create”
Sync the Application
- After creation, go to the app page
- Click “Sync” (if using manual sync)
- Argo CD will pull the manifests and deploy them to your cluster

ArgoCD Architecture :
Argo CD is deployed as a set of Kubernetes controllers and services inside your cluster.
It continuously pulls manifests from Git and applies them to the cluster, keeping everything in sync.

Component	Role
API Server	Provides the UI, CLI (`argocd`), and REST API. Handles RBAC and authentication.
Repository Server (Repo Server)	Clones Git repositories and renders manifests (Helm, Kustomize, etc.).
Application Controller	Core GitOps reconciliation engine: compares desired vs live state and performs sync actions.
Dex (Optional)	Identity provider for SSO authentication (OIDC, GitHub, LDAP, etc.).
Redis (Optional)	Used as a cache to speed up repository and application state operations.

Create ArgoCD Project :
When you install Argo CD, it automatically creates a built-in project called default :

Has no restrictions by default
Allows any Git repository as a source
Allows deployments to any cluster and any namespace
Allows all resource kinds (cluster-scoped and namespace-scoped)

❯❯❯ argocd proj list
NAME     DESCRIPTION  DESTINATIONS  SOURCES  CLUSTER-RESOURCE-WHITELIST  NAMESPACE-RESOURCE-BLACKLIST  SIGNATURE-KEYS  ORPHANED-RESOURCES
default               *,*           *        */*                         <none>                        <none>          disabled

kubectl get appproj -n gitops # check project

we'll try to create a custom project which has some restrictions :
Creating your own AppProject lets you:

Restrict which repos can be used
Restrict which clusters/namespaces can be deployed to
Apply RBAC per project

Using UI :

Go to Settings → Projects
Click “+ NEW PROJECT”
Project Name and Description
Create
Source Repositories: only repos will be allowed.
Destinations (CLUSTER + NAMESPACE)

ClusterRole (in the rbac.authorization.k8s.io API group) is restricted.
Any Application assigned to this Project that tries to create, update, or delete a ClusterRole will be blocked by Argo CD.
The sync will fail with a permission/validation error, and the ClusterRole resource will not be applied.

Section	What it means
Cluster Resource Allow List	A whitelist of cluster-scoped resources (apply to the whole cluster, not tied to a namespace) that applications are allowed to create/update/delete. If empty → all are allowed by default.
Cluster Resource Deny List	A blacklist of cluster-scoped resources that applications are forbidden from managing. In your case, `ClusterRole` is listed here — meaning apps in this project cannot create or modify ClusterRoles.
Namespace Resource Allow List	A whitelist of namespace-scoped resources (like Deployments, ConfigMaps, Services, etc.) that applications are allowed to manage.
Namespace Resource Deny List	A blacklist of namespace-scoped resources that are forbidden.

❯❯❯ argocd proj get test-project
Name:                        test-project
Description:                 for testing
Destinations:                https://kubernetes.default.svc,*
Repositories:                https://github.com/sidd-harth/gitops-argocd
Scoped Repositories:         <none>
Allowed Cluster Resources:   <none>
Scoped Clusters:             <none>
Denied Namespaced Resources: <none>
Signature keys:              <none>
Orphaned Resources:          disabled

argocd proj get test-project -o yaml

metadata:
  creationTimestamp: "2025-09-13T18:25:20Z"
  generation: 5
  managedFields:
  - apiVersion: argoproj.io/v1alpha1
    fieldsType: FieldsV1
    fieldsV1:
      f:spec:
        .: {}
        f:clusterResourceBlacklist: {}
        f:description: {}
        f:destinations: {}
        f:sourceRepos: {}
      f:status: {}
    manager: argocd-server
    operation: Update
    time: "2025-09-13T18:43:52Z"
  name: test-project
  namespace: gitops
  resourceVersion: "30359"
  uid: 829347ce-cb3d-46bb-8b1b-8bef6f5a9a56
spec:
  clusterResourceBlacklist:
  - group: '""'
    kind: ClusterRole
  description: for testing
  destinations:
  - name: in-cluster
    namespace: '*'
    server: https://kubernetes.default.svc
  sourceRepos:
  - https://github.com/sidd-harth/gitops-argocd
status: {}

Applications in test-project can deploy anything from the specified Git repo to any namespace in the in-cluster cluster — except creating ClusterRole objects.

3. ArgoCD Intermediate

Reconciliation loop

1. timeout.reconciliation (TIMEOUT option) :
What it is:
A ConfigMap setting that defines the maximum duration of a single reconciliation loop.
A reconciliation loop is how often your ArgoCD application will synchronize from the Git repository :

In a generic ArgoCD configuration, the default timeout period is set to 3 minutes.
This value is configurable and is used within the ArgoCD repo server.
The ArgoCD repo server is responsible to retrieve the desired state from the Git repo server and it has a timeout option called as the application reconciliation timeout.
If we check the environment variable of ArgoCD repo server pod, it says that the key timeout reconciliation can be configured on an ArgoCD config map.

kubectl -n gitops describe pod argocd-repo-server-cd79f5cc4-lvvgp | grep -i "ARGOCD_RECONCILIATION_TIMEOUT:" -B1
kubectl get cm argocd-cm -n gitops -o yaml | grep -i timeout
# kubectl patch command to update the timeout.reconciliation value in the argocd-cm
kubectl -n gitops patch configmap argocd-cm --patch='{"data":{"timeout.reconciliation":"300s"}}'
kubectl -n gitops rollout restart deploy/argocd-repo-server

The argocd-repo-server takes the config value from argocd-cm :

❯❯❯ kubectl -n gitops describe pod argocd-repo-server-cd79f5cc4-lvvgp | grep -i "ARGOCD_RECONCILIATION_TIMEOUT:" -B1                                                                      
      ARGOCD_REPO_SERVER_NAME:                                      argocd-repo-server
      ARGOCD_RECONCILIATION_TIMEOUT:                                <set to the key 'timeout.reconciliation' of config map 'argocd-cm'>                                          Optional: true

❯❯❯ k get cm argocd-cm -n gitops -o yaml | grep -i timeout.reconciliation                                                                                                                 
  timeout.reconciliation: 180s

2. Webhook (GIT WEBHOOK option) :
What it is:
An event trigger that tells Argo CD to immediately start a reconciliation loop when new commits are pushed.
Where configured:
In your Git hosting platform (GitHub/GitLab/Bitbucket etc.)
You add a webhook pointing to Argo CD’s API:

https://<argocd-server>/api/webhook

kubectl -n gitops rollout restart deploy/argocd-repo-server
Effect:
Instead of waiting for the default polling interval (3 min), Argo CD instantly sees new commits and starts reconciliation.

By default, the argocd-server component only serves HTTPS (TLS) on port 443/8080.
When your Gitea webhook points to http://.../api/webhook instead of https://..., Argo CD will reject it (because it expects TLS), won’t accept it without valid HTTPS.
The --insecure flag tells argocd-server: “Serve plain HTTP instead of HTTPS.”

kubectl edit -n gitops deployments.apps argocd-server

containers:
- args:
  - /usr/local/bin/argocd-server
  - --insecure # add this line

❯❯❯ k get po -n gitops argocd-server-554fb76c44-ck57g                                                                                        ⎈ (kind-kind/solar-system)
NAME                             READY   STATUS    RESTARTS   AGE
argocd-server-554fb76c44-ck57g   1/1     Running   0          8m35s

kubectl port-forward svc/argocd-server --address=0.0.0.0 -n gitops 8080:80

Create a New App in ArgoCD
Now, when you push a new commit, Argo will show an Out of Sync status for this App.

Application health

Status	Meaning
🟢 Healthy	All resources are 100% healthy
🔵 Progressing	Resource is unhealthy, but could still be healthy given time
💗 Degraded	Resource status indicates a failure or an inability to reach a healthy state
🟡 Missing	Resource is not present in the cluster
🟣 Suspended	Resource is suspended or paused. Typical example is a paused Deployment
⚪ Unknown	Health assessment failed and actual health status is unknown

Sync Strategies

Feature	Description
Manual or automatic sync	If set to automatic, Argo CD will apply the changes then update or create new resources in the target Kubernetes cluster.
Auto-pruning of resources	Auto-pruning feature describes what happens when files are deleted or removed from Git.
Self-Heal of cluster	Self-heal defines what Argo CD does when you make `kubectl edit` changes directly to the cluster.

Declarative Setup - Mono Application

Mono-application = one Application object tracks exactly one app path.
Keep app config declarative in Git; Argo CD watches it and reconciles automatically (if automated enabled).

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: geocentric-model-app
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: default

  source:
    repoURL: http://165.22.209.118:3000/siddharth/gitops-argocd.git
    targetRevision: HEAD
    path: ./declarative/manifests/geocentric-model  # this path contains deployment and svc

  destination:
    server: https://kubernetes.default.svc
    namespace: geocentric-model

  syncPolicy:
    syncOptions:
      - CreateNamespace=true  
    automated:
      prune: true
      selfHeal: true

kubectl apply -f filename.yaml

App of Apps

What is App of Apps?

Mono-app = one Application → one set of manifests.
App of Apps = one parent/root Application points to a folder containing multiple Application YAMLs.
Argo CD applies the root app → which creates all the child apps.
You only need to manually create the root application — all other apps are bootstrapped from Git.

Example repo structure :

gitops-repo/
└─ root/
   ├─ app-of-apps.yaml            # the parent Argo CD Application
   ├─ apps/
   │   ├─ frontend.yaml           # child app definition
   │   ├─ backend.yaml            # child app definition
   │   └─ database.yaml            # child app definition
   ├─ frontend/                   # actual manifests (Helm/Kustomize/YAML)
   │   ├─ deployment.yaml
   │   └─ service.yaml
   ├─ backend/
   └─ database/

Parent Application (app-of-apps.yaml) :

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: root-app
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/your-org/gitops-repo.git
    targetRevision: main
    path: root/apps                 # folder that holds child App YAMLs
    directory:
      recurse: true                 # include all files recursively
  destination:
    server: https://kubernetes.default.svc
    namespace: argocd
  syncPolicy:
    automated:
      selfHeal: true
      prune: true
    syncOptions:
    - CreateNamespace=true

One of the child applications (frontend.yaml) :

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: frontend
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/your-org/gitops-repo.git
    targetRevision: main
    path: root/frontend
  destination:
    server: https://kubernetes.default.svc
    namespace: frontend
  syncPolicy:
    automated:
      selfHeal: true
      prune: true
    syncOptions:
    - CreateNamespace=true

How it works:
1.Apply the root app:

kubectl apply -f root/app-of-apps.yaml -n argocd

2.Argo CD syncs the root app.
3.Root app creates the child Application objects from Git.
4.Each child app manages its own manifests as if it were a standalone app.

Deploy apps using HELM Chart

1. Deploy Using My Helm Chart :

❯❯❯ cd gitops-argocd/                                                                                                     ⎈ (kind-kind/solar-system)
[I] [~/gitops-argocd] (22:34:45) main

❯❯❯ ls | grep config                                                                                                      ⎈ (kind-kind/solar-system)
📂 declarative  📂 health-check  📂 helm-chart  📂 jenkins-demo  📄 LICENSE  📂 nginx-app  📂 sealed-secret  📂 solar-system  📂 vault-secrets
[I] [~/gitops-argocd] (22:34:46) main

❯❯❯ cd helm-chart/                                                                                                        ⎈ (kind-kind/solar-system)
[I] [~/g/helm-chart] (22:34:52) main

❯❯❯ cd templates/                                                                                                         ⎈ (kind-kind/solar-system)
[I] [~/g/h/templates] (22:34:56) main

❯❯❯ pwd                                                                                                                   ⎈ (kind-kind/solar-system)
/home/poseidon/gitops-argocd/helm-chart/templates
[I] [~/g/h/templates] (22:34:58) main

❯❯❯ ls                                                                                                                    ⎈ (kind-kind/solar-system)
📄 _helpers.tpl  📄 configmap.yaml  📄 deployment.yaml  📄 NOTES.txt  📄 service.yaml
[I] [~/g/h/templates] (22:35:00) main

❯❯❯ cat deployment.yaml                                                                                                   ⎈ (kind-kind/solar-system)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ .Release.Name }}-deploy
  labels:
    {{- include "random-shapes.labels" . | nindent 4 }}
spec:
  replicas: {{ .Values.replicaCount }}
  selector:
    matchLabels:
      {{- include "random-shapes.selectorLabels" . | nindent 6 }}
  template:
    metadata:
      {{- with .Values.podAnnotations }}
      annotations:
        {{- toYaml . | nindent 8 }}
      {{- end }}
      labels:
        {{- include "random-shapes.selectorLabels" . | nindent 8 }}
    spec:
      containers:
        - name: {{ .Chart.Name }}
          image: "{{ .Values.image.repository }}"
          imagePullPolicy: {{ .Values.image.pullPolicy }}
          envFrom:
          - configMapRef:
              name: {{ .Release.Name }}-configmap
[I] [~/g/h/templates] (22:35:04) main

❯❯❯ cat service.yaml                                                                                                      ⎈ (kind-kind/solar-system)
apiVersion: v1
kind: Service
metadata:
  name: {{ .Release.Name }}-service
  labels:
    {{- include "random-shapes.labels" . | nindent 4 }}
spec:
  type: {{ .Values.service.type }}
  ports:
    - port: {{ .Values.service.port }}
      targetPort: {{ .Values.service.targetPort }}
      protocol: TCP
      name: http
  selector:
    {{- include "random-shapes.selectorLabels" . | nindent 4 }}
[I] [~/g/h/templates] (22:35:07) main

❯❯❯ cd ..                                                                                                                 ⎈ (kind-kind/solar-system)
[I] [~/g/helm-chart] (22:35:12) main

❯❯❯ ls                                                                                                                    ⎈ (kind-kind/solar-system)
📄 Chart.yaml  📂 templates  📄 values.yaml
[I] [~/g/helm-chart] (22:35:13) main

❯❯❯ cat values.yaml                                                                                                       ⎈ (kind-kind/solar-system)
replicaCount: 1
image:
  repository: siddharth67/php-random-shapes:v1
  pullPolicy: IfNotPresent
  # Overrides the image tag whose default is the chart appVersion.
  tag: ""

imagePullSecrets: []
nameOverride: ""
fullnameOverride: ""

service:
  type: ClusterIP
  port: 80
  targetPort: 80

color:
   circle: black
   oval: black
   triangle: black
   rectangle: black
   square: black    
[I] [~/g/helm-chart] (22:35:17) main

❯❯❯ cat Chart.yaml                                                                                                        ⎈ (kind-kind/solar-system)
apiVersion: v2
name: random-shapes-chart
description: A Helm chart for Random Shape App
version: 1.0.0

argocd app create helm-random-shapes \
  --repo https://github.com/sidd-harth/gitops-argocd \
  --path helm-chart \
  --helm-set replicaCount=2 \
  --helm-set color.circle=pink \
  --helm-set color.square=green \
  --helm-set service.type=ClusterIP \
  --dest-namespace default \
  --dest-server https://kubernetes.default.svc

❯❯❯ argocd app create helm-random-shapes \                                                                                ⎈ (kind-kind/solar-system)
          --repo https://github.com/sidd-harth/gitops-argocd \
          --path helm-chart \
          --helm-set replicaCount=2 \
          --helm-set color.circle=pink \
          --helm-set color.square=green \
          --helm-set service.type=ClusterIP \
          --dest-namespace default \
          --dest-server https://kubernetes.default.svc

application 'helm-random-shapes' created
[I] [~/gitops] (22:32:01) 
❯❯❯ h ls                                                                                                                  ⎈ (kind-kind/solar-system)
NAME    NAMESPACE   REVISION    UPDATED STATUS  CHART   APP VERSION
[I] [~/gitops] (22:32:09) 
❯❯❯ argocd app get helm-random-shapes                                                                                     ⎈ (kind-kind/solar-system)
Name:               gitops/helm-random-shapes
Project:            default
Server:             https://kubernetes.default.svc
Namespace:          default
URL:                https://argocd.example.com/applications/helm-random-shapes
Source:
- Repo:             https://github.com/sidd-harth/gitops-argocd
  Target:           
  Path:             helm-chart
SyncWindow:         Sync Allowed
Sync Policy:        Manual
Sync Status:        OutOfSync from  (05e2501)
Health Status:      Missing

GROUP  KIND        NAMESPACE  NAME                          STATUS     HEALTH   HOOK  MESSAGE
       ConfigMap   default    helm-random-shapes-configmap  OutOfSync  Missing        
       Service     default    helm-random-shapes-service    OutOfSync  Missing        
apps   Deployment  default    helm-random-shapes-deploy     OutOfSync  Missing        
[I] [~/gitops] (22:32:20) 
❯❯❯ argocd app get helm-random-shapes                                                                                     ⎈ (kind-kind/solar-system)
Name:               gitops/helm-random-shapes
Project:            default
Server:             https://kubernetes.default.svc
Namespace:          default
URL:                https://argocd.example.com/applications/helm-random-shapes
Source:
- Repo:             https://github.com/sidd-harth/gitops-argocd
  Target:           
  Path:             helm-chart
SyncWindow:         Sync Allowed
Sync Policy:        Manual
Sync Status:        Synced to  (05e2501)
Health Status:      Healthy

GROUP  KIND        NAMESPACE  NAME                          STATUS  HEALTH   HOOK  MESSAGE
       ConfigMap   default    helm-random-shapes-configmap  Synced                 configmap/helm-random-shapes-configmap created
       Service     default    helm-random-shapes-service    Synced  Healthy        service/helm-random-shapes-service created
apps   Deployment  default    helm-random-shapes-deploy     Synced  Healthy        deployment.apps/helm-random-shapes-deploy created
[I] [~/gitops] (22:32:51) 
❯❯❯

2. Deploy Using Bitnami Helm Charts :
Add Bitnami Helm Charts Repo :

Create App :

You can also modify all the parameters : Change service.type from LoadBalancer to ClusterIP :

❯❯❯ k get all -n bitnami                                                                                                  ⎈ (kind-kind/solar-system)
NAME                                          READY   STATUS     RESTARTS   AGE
pod/bitnami-helm-nginx-app-7bc9fc68c9-np9t7   0/1     Init:0/1   0          46s

NAME                             TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)          AGE
service/bitnami-helm-nginx-app   ClusterIP   10.96.61.98   <none>        80/TCP,443/TCP   47s

NAME                                     READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/bitnami-helm-nginx-app   0/1     1            0           46s

NAME                                                DESIRED   CURRENT   READY   AGE
replicaset.apps/bitnami-helm-nginx-app-7bc9fc68c9   1         1         0       46s

4. ArgoCD Advanced

How ArgoCD manages role-based access control.

RBAC Architecture in Argo CD :

Users or Groups
- Authenticated identities are then mapped to RBAC roles.
Roles
- Logical sets of permissions.
- Defined in the argocd-rbac-cm ConfigMap (in the argocd namespace).
- Each role can contain multiple policy rules.
Policies (Rules)
- Define what actions a role can perform on what resources.
- Written in the format:

p, <role>, <resource>, <action>, <object>, <effect>

resource → what kind of object (applications, projects, repositories, clusters…)
action → get, create, update, delete, sync, override, etc.
object → * (all) or specific names

Role Bindings
- Map users or groups to roles:

g, <username_or_group>, <role>

Give user jai permission to create clusters :

kubectl -n argocd patch configmap argocd-rbac-cm \
--patch='{"data":{"policy.csv":"p, role:create-cluster, clusters, create, *, allow\ng, jai, role:create-cluster"}}'

# Test it:
argocd account can-i create clusters '*'
# should print: yes   # (when logged in as jai)

argocd account can-i delete clusters '*'
# should print: no    # (when logged in as jai)

Give user ali permission to manage applications in kia-project :

kubectl -n argocd patch configmap argocd-rbac-cm \
--patch='{"data":{"policy.csv":"p, role:kia-admins, applications, *, kia-project/*, allow\ng, ali, role:kia-admins"}}'

# Test it:
argocd account can-i sync applications kia-project/*
# should print: yes   # (when logged in as ali)

Notes

p = policy (what a role can do)
g = group binding (who gets the role)
argocd account can-i ... tests permissions for the currently logged in user
After patching, Argo CD automatically reloads the RBAC config (no restart needed)

User Management

The default ArgoCD installation has one built-in admin user that has full access to the system and is a super user.
It is advised to only utilize the admin user for initial settings and then disable it after adding all required users.

❯❯❯ argocd account list                                                                                                   ⎈ (kind-kind/solar-system)

NAME   ENABLED  CAPABILITIES
admin  true     login

New users can be created and used for various roles.
New users can be defined using ArgoCD ConfigMap.
We edit the ArgoCD ConfigMap and add accounts.username record.
Each user can be associated with two capabilities, API key and login.

# Creates two local accounts: jai and ali
kubectl -n argocd patch configmap argocd-cm \
--patch='{"data":{"accounts.jai": "apiKey,login"}}'

kubectl -n argocd patch configmap argocd-cm \
--patch='{"data":{"accounts.ali": "apiKey,login"}}'

Grants them:

login → allows logging in via UI/CLI
apiKey → API key allows generating JSON Web Token authentication for APl access.
After this, you can bind them to RBAC roles using g, jai, role:... in the argocd-rbac-cm.

❯❯❯ kubectl -n gitops patch configmap argocd-cm \
        --patch='{"data":{"accounts.jai": "apiKey,login"}}'

configmap/argocd-cm patched

❯❯❯ kubectl -n gitops patch configmap argocd-cm \
        --patch='{"data":{"accounts.ali": "apiKey,login"}}'

configmap/argocd-cm patched

❯❯❯ argocd account list
NAME   ENABLED  CAPABILITIES
admin  true     login
ali    true     apiKey, login
jai    true     apiKey, login

❯❯❯ argocd account update-password --account jai
*** Enter password of currently logged in user (admin): 
*** Enter new password for user jai: 
*** Confirm new password for user jai: 
Password updated

ArgoCD has two predefined default roles, read-only and admin.
- The read-only role provides read-only access to all the resources, whereas the admin role provides unrestricted access to all resources.
- And by default, the admin user is assigned to the admin role.
- We can modify this and can also assign custom roles to users by editing the ArgoCD RBAC Config Map.

# set the default role in Argo CD RBAC to readonly
kubectl -n argocd patch configmap argocd-rbac-cm \
--patch='{"data":{"policy.default": "role:readonly"}}'

In this example, we are patching the ArgoCD RBAC ConfigMap by assigning a default read-only role to any user who is not mapped to a specific role.

Bitnami Sealed Secrets

Summary of Flow :

Create normal Secret
Encrypt with kubeseal → SealedSecret
Commit to Git
Argo CD syncs it
sealed-secrets-controller decrypts to a real Secret

Step 1 — Create a normal Kubernetes Secret manifest

kubectl create secret generic mysql-password \
  --from-literal=password='s1Ddh@rt#' \
  --dry-run=client -o yaml > mysql-password_k8s-secret.yaml

This produces a file like:

apiVersion: v1
kind: Secret
metadata:
  name: mysql-password
data:
  password: czF1RGRoQHJ0Iw==

Step 2 — Install the Sealed Secrets controller (via Argo CD)

argocd app create sealed-secrets \
  --repo https://bitnami-labs.github.io/sealed-secrets \
  --helm-chart sealed-secrets \
  --revision 2.2.0 \
  --dest-server https://kubernetes.default.svc \
  --dest-namespace kube-system

# then sync 
argocd app sync sealed

# or 
argocd app create sealed \
  --repo https://bitnami-labs.github.io/sealed-secrets \
  --helm-chart sealed-secrets \
  --revision 2.2.0 \
  --dest-server https://kubernetes.default.svc \
  --dest-namespace sealed \
  --sync-option CreateNamespace=true \
  --sync-policy automated

kubectl get all -n sealed
kubectl -n sealed get deploy,sealedsecret,po,svc

Step 3 — Get the public certificate from the controller

# Export the cluster’s sealing public key:
kubectl -n sealed get secret \
  -l sealedsecrets.bitnami.com/sealed-secrets-key=active \
  -o jsonpath='{.items[0].data.tls\.crt}' \
  | base64 -d > sealedSecret.crt

Step 4 — Install the kubeseal CLI locally

wget https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.18.0/kubeseal-0.18.0-linux-amd64.tar.gz -O kubeseal.tar.gz
tar -xzf kubeseal.tar.gz
sudo install -m 755 kubeseal /usr/local/bin/kubeseal

Step 5 — Seal your secret

# Use kubeseal to convert your Secret into a SealedSecret:
kubeseal -o yaml \
  --scope cluster-wide \
  --cert sealedSecret.crt \
  < mysql-password_k8s-secret.yaml \
  > mysql-password_sealed-secret.yaml

Now you have an encrypted SealedSecret manifest.

Step 6 — Commit the SealedSecret to Git (Argo CD watches it)

Push mysql-password_sealed-secret.yaml to your Git repo along with your deployment.yaml.
Argo CD will sync and apply it.
The sealed-secrets-controller pod will decrypt it and create the real Secret.

Step 7 — Verify the decrypted Secret

kubectl get secret mysql-password -o yaml

You should see the real base64-encoded password.

Hashicorp Vault

1. ArgoCD Vault Plugin CLI :
Step 1 -- Create Vault App in ArgoCD

❯❯❯ k get all -n vault-demo-gitops
NAME                                           READY   STATUS    RESTARTS   AGE
pod/vault-app-0                                0/1     Running   0          2m2s
pod/vault-app-agent-injector-67ff69f54-8dxqc   1/1     Running   0          2m3s

NAME                                   TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
service/vault-app                      ClusterIP   10.96.132.44    <none>        8200/TCP,8201/TCP   2m3s
service/vault-app-agent-injector-svc   ClusterIP   10.96.201.162   <none>        443/TCP             2m3s
service/vault-app-internal             ClusterIP   None            <none>        8200/TCP,8201/TCP   2m3s

NAME                                       READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/vault-app-agent-injector   1/1     1            1           2m3s

NAME                                                 DESIRED   CURRENT   READY   AGE
replicaset.apps/vault-app-agent-injector-67ff69f54   1         1         1       2m3s

NAME                         READY   AGE
statefulset.apps/vault-app   0/1     2m2s

❯❯❯ k get po -n vault-demo-gitops
NAME                                       READY   STATUS    RESTARTS   AGE
vault-app-0                                0/1     Running   0          2m10s
vault-app-agent-injector-67ff69f54-8dxqc   1/1     Running   0          2m11s

Step 2 --- Unsealed Process

# Access Vault UI
kubectl -n vault-demo-gitops port-forward svc/vault-app 8200:8200 --address=0.0.0.0
# From UI: 
# Key shares: 3
# Key threshold: 2
# Then Download Keys (root key and 3 sealed keys) , then UI will ask you about Unseal Keys and root key

Now: vault-app-0 will be 1/1 Ready state

❯❯❯ k get po -n vault-demo-gitops
NAME                                       READY   STATUS    RESTARTS   AGE
vault-app-0                                1/1     Running   0          10m
vault-app-agent-injector-67ff69f54-8dxqc   1/1     Running   0          10m

From UI: 
Enable new engine => KV => Next => Path: Credentials => Enable Engine => Create Secret => Path for this secret: app => Secret Data => Enter Random Vaules(username , password , apikey) => Save 
Now From Secrets, we can see credentials and from three dots => details , we can see path /credentials

Step 3 -- Create Secret with Template Syntax: vault.yaml

kind: Secret
apiVersion: v1
metadata:
  name: app-crds
  annotations:
    avp.kubernetes.io/path: "credentials/data/app"
type: Opaque
stringData:
  apikey: <apikey>
  username: <username>
  password: <password>

Step 4 -- Install ArgoCD Vault Plugin

curl -L -o argocd-vault-plugin https://github.com/argoproj-labs/argocd-vault-plugin/releases/download/v1.17.0/argocd-vault-plugin_1.17.0_linux_amd64
chmod +x argocd-vault-plugin
sudo mv argocd-vault-plugin /usr/local/bin/

# Verify installation
argocd-vault-plugin version

Step 5 -- Create vault.env

VAULT_ADDR=http://localhost:8200
VAULT_TOKEN=<root token>
AVP_TYPE=vault
AVP_AUTH_TYPE=token

argocd-vault-plugin generate -c vault.env - < vault.yaml

will see as shown below :

apiVersion: v1
kind: Secret
metadata:
  annotations:
    avp.kubernetes.io/path: credentials/data/app
  name: app-crds
stringData:
  apikey: sdfshifsdifj596211af
  password: root9289
  username: omar
type: Opaque

❯❯❯ cat vault.yaml
kind: Secret
apiVersion: v1
metadata:
  name: app-crds
  annotations:
    avp.kubernetes.io/path: "credentials/data/app"
type: Opaque
stringData:
  apikey: <apikey>
  username: <username>
  password: <password>

2. ArgoCD with ArgoCD Vault Plugin :
we need to patch argocd-repo-server deployment and argocd-cm config map :

k edit -n gitops deployments.apps argocd-repo-server

apiVersion: apps/v1
kind: Deployment
metadata:
  name: argocd-repo-server
spec:
  template:
    spec:
      containers:
      - name: argocd-repo-server
        volumeMounts:
        - name: custom-tools
          mountPath: /usr/local/bin/argocd-vault-plugin
          subPath: argocd-vault-plugin

        # Note: AVP config (for the secret manager, etc) can be passed in several ways. This is just one example
        # https://argocd-vault-plugin.readthedocs.io/en/stable/config/
        envFrom:
          - secretRef:
              name: argocd-vault-plugin-credentials
      volumes:
      - name: custom-tools
        emptyDir: {}
      initContainers:
      - name: download-tools
        image: alpine:3.8
        command: [sh, -c]

        # Don't forget to update this to whatever the stable release version is
        # Note the lack of the `v` prefix unlike the git tag
        env:
          - name: AVP_VERSION
            value: "1.7.0"
        args:
          - >-
            wget -O argocd-vault-plugin
            https://github.com/argoproj-labs/argocd-vault-plugin/releases/download/v${AVP_VERSION}/argocd-vault-plugin_${AVP_VERSION}_linux_amd64 &&
            chmod +x argocd-vault-plugin &&
            mv argocd-vault-plugin /custom-tools/
        volumeMounts:
          - mountPath: /custom-tools
            name: custom-tools

      # Not strictly necessary, but required for passing AVP configuration from a secret and for using Kubernetes auth to Hashicorp Vault
      automountServiceAccountToken: true

k edit -n gitops cm argocd-cm

data:
  configManagementPlugins: |-
    - name: argocd-vault-plugin
      generate:
        command: ["argocd-vault-plugin"]
        args: ["generate", "./"]

k rollout restart deploy  argocd-repo-server

From UI: 
Create App => URL: https://github.com/sidd-harth/gitops-argocd
path: ./vault-secrets
under the namespace option , we will find Directory Icon, switch from Directory to  Plugin => Name: argocd-vault-plugin => Env: 
VAULT_ADDR=http://vault-app.vault-demo.svc.cluster.local:8200
VAULT_TOKEN=<root token>
AVP_TYPE=vault
AVP_AUTH_TYPE=token

=> Create

# to test
k -n vault-secret get secrets -o yaml

Falco With Kubernetes

Omar Ahmed — Sat, 06 Sep 2025 20:40:33 +0000

Falco Basics

Falco is an open-source, cloud-native runtime security project designed to detect unexpected application behavior and alert on threats in real time.
By default, Falco has 5 outputs for its events: stdout, file, gRPC, shell and http.
These can be integrated with other components using falcosidekick, a daemon that extends that number of possible outputs.

Key Points about Falco:

Runtime Security: It continuously monitors your applications, containers, and hosts at runtime to detect abnormal activities.
Container Visibility: It provides complete visibility into containerized environments using a single lightweight sensor.
Rules-Based Detection: Falco uses a rich set of rules to define what is considered abnormal. When these rules are violated, alerts are triggered.

Examples of what Falco can detect by default:

A shell being run inside a container (which could indicate a breach).
A server process spawning an unexpected type of child process.
An attempt to read sensitive files, like /etc/shadow.

Falco Installation

kubectl create namespace falco

helm repo add falcosecurity https://falcosecurity.github.io/charts
helm repo update

helm install falco falcosecurity/falco \
--set falcosidekick.enabled=true \
--set falcosidekick.webui.enabled=true \
-n falco

# falco → release name
# falcosecurity/falco → chart
# -n falco --create-namespace → installs Falco in a separate falco namespace

# check that the Falco pods are running:
kubectl get pods -n falco

Test Using Falco UI

kubectl run nginx --image=nginx -n falco --restart=Never -- sleep infinity
kubectl exec -it nginx -n falco -- bash 
# user: admin , password: admin
kubectl port-forward --address=0.0.0.0 svc/falco-falcosidekick-ui 3000:2802
# go to events tab

Falco Slack Notifications

Follow these steps to create and test a Slack Incoming Webhook for Falco alerts:

Channels => Create => Create Channel => Channel Name => Falco

Create an App
- Click Create New App
- Go to Slack API Apps
- Choose From Scratch
- Enter App Name: Falco Slack App
- Pick a workspace to develop your app: Test
- Click Create App
Enable Incoming Webhooks
- From the left sidebar → click Incoming Webhooks
- Switch Activate Incoming Webhooks → ON
Add a Webhook to Workspace
- Scroll down and click Add New Webhook to Workspace
- Select a channel: #falco
- Click Allow

Copy and Test the Sample Curl URL Request

Run the sample Curl URL request in your terminal:

curl -X POST -H 'Content-type: application/json' \
--data '{"text":"Hello, World!"}' \
https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX

Open your Slack #falco channel,You should see: Hello, World!
Copy the Webhook URL Provided and use it below:

helm upgrade falco falcosecurity/falco \
--set falcosidekick.enabled=true \
--set falcosidekick.webui.enabled=true \
--set falcosidekick.config.slack.webhookurl="https://hooks.slack.com/services/XXXX" \
--set falcosidekick.config.customfields="environment:production\,datacenter:egypt" \
-n falco 

# check the revision number
helm ls -n falco

# attach a terminal in a container 
kubectl run nginx --image=nginx -n falco --restart=Never -- sleep infinity
kubectl exec -it nginx -n falco -- bash

Istio - Service Mesh

Omar Ahmed — Fri, 05 Sep 2025 15:41:45 +0000

Istio Basics

1. Download Istio CLI

curl -L https://istio.io/downloadIstio | sh -
cd istio-*
export PATH=$PWD/bin:$PATH

2. Install Istio control plane

istioctl install --set profile=demo -y
kubectl get pods -n istio-system

3. Enable auto sidecar injection on prod

kubectl create namespace prod
kubectl label namespace prod istio-injection=enabled
# Confirm:
kubectl get ns --show-labels
kubectl get namespace -L istio-injection

4. Open Kiali UI

# Kiali's UI
kubectl port-forward --address=0.0.0.0 svc/kiali 3000:20001 
# Then open: http://localhost:3000

5. Verify sidecar injection (pods should be 2/2 Ready)

kubectl -n prod create deploy node-app --image siddharth67/node-service:v1
kubectl describe pod node-app-786cb44f85-wkq99 -n prod | grep -A2 "Containers:"

❯❯❯ kubectl describe pod node-app-786cb44f85-wkq99 -n prod | grep -A2 "Containers:"                                                        ⎈ (kind-kind/prod)
Init Containers:
  istio-init:
    Container ID:  containerd://2b5ae904579ca1739120831e123b29d23ee3649c6e615f9b4fd1b24f07bbfea7
--
Containers:
  node-service:
    Container ID:   containerd://97484d714ff2c66bf40408aa674b75c049f32f0f9bfbd8d76b7a17f784bbb53b

Istio Ingress Gateway

The Istio Ingress Gateway is a specialized Envoy proxy that handles inbound traffic from outside the service mesh into your cluster.
What it does:

Acts as the entry point for external traffic
Provides load balancing, TLS termination, and protocol handling
Replaces traditional ingress controllers in Istio environments
Runs as a deployment (not a sidecar)

Virtual Service

Virtual Service defines routing rules - it tells the gateway WHERE to send the traffic once it enters the mesh.
What it does:

Defines traffic routing logic
Handles path-based routing, header matching, traffic splitting
Works with both ingress traffic (via Gateway) and internal traffic
Provides advanced routing features like retries, timeouts, fault injection

6. Bring traffic through Istio ingress (so Kiali can see it)

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: devsecops-gateway
  namespace: prod
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: devsecops-vs
  namespace: prod
spec:
  hosts:
  - "*"
  gateways:
  - devsecops-gateway
  http:
  - match:
    - uri:
        prefix: /plusone
    route:
    - destination:
        host: devsecops-svc
        port:
          number: 5000

End-to-end path:
Client → Istio ingressgateway (port 80) → VirtualService match /plusone → ClusterIP Service devsecops-svc:5000 → Pod.

kubectl -n prod apply -f - <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: devsecops
  name: devsecops
spec:
  replicas: 3
  selector:
    matchLabels:
      app: devsecops
  template:
    metadata:
      labels:
        app: devsecops
    spec:
      serviceAccountName: default
      volumes:
      - name: vol
        emptyDir: {}
      containers:
      - name: devsecops-container
        image: siddharth67/node-service:v1   
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 8080
        volumeMounts:
        - mountPath: /tmp
          name: vol
        securityContext:
          capabilities:
            drop: [ "NET_RAW" ]
          runAsUser: 100
          runAsNonRoot: true
          readOnlyRootFilesystem: true
          allowPrivilegeEscalation: false
        resources:
          requests:
            memory: "256Mi"
            cpu: "200m"
          limits:
            memory: "512Mi"
            cpu: "500m"
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: devsecops
  name: devsecops-svc
spec:
  selector:
    app: devsecops
  ports:
  - port: 8080
    targetPort: 8080
    protocol: TCP
  type: ClusterIP
EOF

7. Send traffic through the mesh

kubectl -n istio-system port-forward svc/istio-ingressgateway 8080:80
while true; do curl -s http://localhost:8080/plusone/99; echo; sleep .2; done

Kiali will not see the traffic if you're not routing through the Istio gateway, so if we want to see the workload through the Kiali UI, we should create IngressGateway and VirtualService objects.
Here's why: Kiali visualizes service mesh traffic by reading telemetry data that Istio's sidecars (Envoy proxies) collect. For this to work, your traffic needs to flow through Istio's data plane components.

Istio mTLS - Auto mutual TLS

mTLS (mutual TLS) with Istio provides automatic encryption and authentication between services in your mesh. Here's how it works:
Default Behavior:
Istio enables mTLS automatically when sidecars are injected:

Traffic between services with sidecars is automatically encrypted
Uses permissive mode by default (accepts both mTLS and plain text)
No configuration needed for basic mTLS

App A ──(plaintext)──► Sidecar A ──(mTLS)──► Sidecar B ──(plaintext)──► App B

PeerAuthentication modes

1. PERMISSIVE (Default)
Traffic accepted:
✅ mTLS from other sidecars
✅ Plain text from non-mesh services
✅ Plain text from services without sidecars

apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: prod
spec:
  mtls:
    mode: PERMISSIVE  # Accepts both mTLS and plaintext

2. STRICT
Traffic accepted:
✅ mTLS from other sidecars
❌ Plain text traffic (rejected with connection errors)

apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: enforce-mtls
  namespace: prod
spec:
  mtls:
    mode: STRICT  # Only accepts mTLS traffic

3. DISABLE
Traffic accepted:
❌ mTLS traffic (rejected)
✅ Plain text traffic only

apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: disable-mtls
  namespace: prod
spec:
  mtls:
    mode: DISABLE

❯❯❯ k get crd
NAME                                       CREATED AT
authorizationpolicies.security.istio.io    2025-09-05T13:29:53Z
destinationrules.networking.istio.io       2025-09-05T13:29:52Z
envoyfilters.networking.istio.io           2025-09-05T13:29:52Z
gateways.networking.istio.io               2025-09-05T13:29:52Z
peerauthentications.security.istio.io      2025-09-05T13:29:54Z #<<##########
proxyconfigs.networking.istio.io           2025-09-05T13:29:52Z
requestauthentications.security.istio.io   2025-09-05T13:29:54Z
serviceentries.networking.istio.io         2025-09-05T13:29:52Z
sidecars.networking.istio.io               2025-09-05T13:29:53Z
telemetries.telemetry.istio.io             2025-09-05T13:29:54Z
virtualservices.networking.istio.io        2025-09-05T13:29:53Z
wasmplugins.extensions.istio.io            2025-09-05T13:29:51Z
workloadentries.networking.istio.io        2025-09-05T13:29:53Z
workloadgroups.networking.istio.io         2025-09-05T13:29:53Z

❯❯❯ k get peerauthentications
No resources found in prod namespace.

❯❯❯ k get pa  # peerauthentications
No resources found in prod namespace.

8. MTLS Modes:

kubectl apply <<'EOF'
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: mtls
  namespace: prod
spec:
  mtls:
    mode: PERMISSIVE
EOF

Apply the Deployment below for testing:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: devsecops-client
  namespace: prod
  labels:
    app: devsecops-client
spec:
  replicas: 1
  selector:
    matchLabels:
      app: devsecops-client
  template:
    metadata:
      labels:
        app: devsecops-client
    spec:
      containers:
      - name: curl
        image: curlimages/curl:latest
        args:
          - sh
          - -c
          - |
            while true; do
              echo "$(date) -> $(curl -sS -o /dev/null -w '%{http_code}' http://devsecops-svc.prod.svc.cluster.local:5000/health || echo 'curl-failed')";
              sleep 5;
            done
        securityContext:
          runAsNonRoot: true
          runAsUser: 1000
          allowPrivilegeEscalation: false
          readOnlyRootFilesystem: true
        resources:
          requests:
            cpu: 50m
            memory: 64Mi
          limits:
            cpu: 200m
            memory: 128Mi

kubectl apply -f deploy.yaml

❯❯❯ k get pa                                                                                                                                                    
NAME   MODE         AGE
mtls   PERMISSIVE   103s

❯❯❯ k describe pa mtls                                                                                                                                  
Name:         mtls
Namespace:    prod
Labels:       <none>
Annotations:  <none>
API Version:  security.istio.io/v1
Kind:         PeerAuthentication
Metadata:
  Creation Timestamp:  2025-09-06T13:04:22Z
  Generation:          1
  Resource Version:    48804
  UID:                 1c9f7fbd-a1a3-4135-9a8a-94e7efb65e9e
Spec:
  Mtls:
    Mode: PERMISSIVE
Events:    <none>

❯❯❯ cat pa.yaml                                                                                                                                                 
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: mtls
  namespace: prod
spec:
  mtls:
    mode: DISABLE

❯❯❯ k apply -f pa.yaml                                                                                                                                          
peerauthentication.security.istio.io/mtls configured

❯❯❯ k get pa                                                                                                                                                    
NAME   MODE      AGE
mtls   DISABLE   10m

❯❯❯ k describe pa mtls                                                                                                                                          
Name:         mtls
Namespace:    prod
Labels:       <none>
Annotations:  <none>
API Version:  security.istio.io/v1
Kind:         PeerAuthentication
Metadata:
  Creation Timestamp:  2025-09-06T13:14:56Z
  Generation:          2
  Resource Version:    51468
  UID:                 17008123-2ca7-414c-9b37-136fdc9a67ad
Spec:
  Mtls:
    Mode:  DISABLE
Events:    <none>

Vault With Kubernetes 🔐

Omar Ahmed — Wed, 03 Sep 2025 22:24:46 +0000

What is HashiCorp Vault?

HashiCorp Vault is a secrets & encryption platform for securely storing, generating, and controlling access to sensitive data (API keys, DB creds, TLS certs, tokens, etc.). Teams use it to centralize secrets, issue short-lived credentials on demand, and enforce least-privilege access across apps, CI/CD, and infrastructure

Install Vault Without a DataStorage (Dev Mode)

helm repo add hashicorp https://helm.releases.hashicorp.com
helm repo update
helm repo list
helm install vault hashicorp/vault \
  --namespace vault --create-namespace \
  --set 'server.dev.enabled=true' \
  --set 'server.dataStorage.enabled=false' \
  --set 'ui.enabled=true'
kubectl config set-context --current --namespace=vault

Install Vault With a DataStorage (Raft storage)

cat > values.yaml <<'YAML'
server:
  dev:
    enabled: false
  standalone:
    enabled: true
    config: |
      ui = true
      listener "tcp" {
        address         = "[::]:8200"
        cluster_address = "[::]:8201"
        tls_disable     = 1
      }
      storage "raft" {
        path = "/vault/data"
      }
  dataStorage:
    enabled: true
    size: 1Gi

ui:
  enabled: true
  serviceType: ClusterIP   # use port-forward; change to NodePort/Ingress if you prefer
YAML

helm repo add hashicorp https://helm.releases.hashicorp.com
helm repo update
helm -n vault install vault hashicorp/vault -f values.yaml --create-namespace --version 0.16.1

Seal/Unseal

When you start a Vault server, it starts in a sealed state. In this state, Vault can access the physical storage, but it cannot decrypt any of the data on it.
Vault encrypts the data using an encryption key (in the keyring) and stores them in its storage backend. To protect this encryption key, Vault encrypts it using another encryption key known as the root key and stores it with the data.
To decrypt the data, Vault needs the root key so that it can decrypt the encryption key. Unsealing is the process of getting access to this root key. Vault encrypts the root key using the unseal key, and stores it alongside all other Vault data.
To summarize, Vault encrypts most data using the encryption key in the keyring. To get the keyring, Vault uses the root key to decrypt it. The root key itself requires the unseal key to decrypt it.

Shamir seals

Key Shares / Unseal Keys                Root Key               Encryption Key
                                                              (protected by root key)
┌─────────────────────┐
│ Jon      🔑         │
├─────────────────────┤
│ Jane     🔑         │-------------->>>🔑 ---------------------->>🔑
├─────────────────────┤                Root Key                  Encryption Key
│ James    🔑         │
├─────────────────────┤
│ Alison   🔑         │
├─────────────────────┤
│ Pam      🔑         │
└─────────────────────┘

Instead of distributing the unseal key to an operator as a single key, the default Vault configuration uses an algorithm known as Shamir's Secret Sharing to split the key into shares.
Vault requires a certain threshold of shares to reconstruct the unseal key. Vault operators add shares one at a time in any order until Vault has enough shares to reconstruct the key. Then, Vault uses the unseal key to decrypt the root key. This is the Vault unseal process.
To Summarize:

Vault never writes the master key to disk.
Only encrypted data is stored (plus an encrypted data-encryption key).
On startup Vault can’t decrypt anything by itself, so it starts sealed.
Unseal = provide a quorum of unseal key shares (Shamir) to reconstruct the master key in memory → that master key decrypts the data-encryption key → Vault becomes unsealed and can serve requests.

❯❯❯ k get po                                                                                ⎈ (kind-my-cluster/vault)
NAME                                    READY   STATUS    RESTARTS   AGE
vault-0                                 0/1     Running   0          55s
vault-agent-injector-6d97459765-4mc6b   1/1     Running   0          55s

vault-0 0/1 Ready (not 1/1) >> because vault is sealed

kubectl -n vault exec -it vault-0 -- vault status # sealed=true as shown below
Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       5
Threshold          3
Unseal Progress    0/3
Unseal Nonce       n/a
Version            1.8.3
Storage Type       raft
HA Enabled         true

❯❯❯ k exec -it vault-0 -- /bin/sh                                    ⎈ (kind-my-cluster/vault)
/ $ vault operator init 
Unseal Key 1: Q3Awl3Qw+ItIWFyLFVGFiE3OOJ1qHrHC+kQnFD2kwLbE
Unseal Key 2: OON236vYc+bf3K45J75lVHy3FmwIRdUT+mGPU3Iq4SX9
Unseal Key 3: N53iBqGPiL9RRkNNVg5DmrlD5dn6R3Vt703y6NhVUB+0
Unseal Key 4: mWPyNPitk8JTlxZByNkyYJefDo+MTpfIEcDn/lkaKuP3
Unseal Key 5: kRHm81+cYIbjO12eFJp0iJP8y3u7CR4NkOAb/4nHS5Kl

Initial Root Token: s.bQN8VBeKBga76dAUVmaGyTjJ

Vault initialized with 5 key shares and a key threshold of 3. Please securely
distribute the key shares printed above. When the Vault is re-sealed,
restarted, or stopped, you must supply at least 3 of these keys to unseal it
before it can start servicing requests.

Vault does not store the generated master key. Without at least 3 keys to
reconstruct the master key, Vault will remain permanently sealed!

It is possible to generate new unseal keys, provided you have a quorum of
existing unseal keys shares. See "vault operator rekey" for more information.
/ $

/ $ vault operator unseal Q3Awl3Qw+ItIWFyLFVGFiE3OOJ1qHrHC+kQnFD2kwLbE
Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             true ###### it is still sealed
Total Shares       5
Threshold          3
Unseal Progress    1/3
Unseal Nonce       8932e24e-a03d-59e7-a694-9c9d9d6005f3
Version            1.8.3
Storage Type       raft
HA Enabled         true
/ $ vault operator unseal OON236vYc+bf3K45J75lVHy3FmwIRdUT+mGPU3Iq4SX9
Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       5
Threshold          3
Unseal Progress    2/3
Unseal Nonce       8932e24e-a03d-59e7-a694-9c9d9d6005f3
Version            1.8.3
Storage Type       raft
HA Enabled         true
/ $ vault operator unseal N53iBqGPiL9RRkNNVg5DmrlD5dn6R3Vt703y6NhVUB+0
Key                     Value
---                     -----
Seal Type               shamir
Initialized             true
Sealed                  false ######## it is unsealed
Total Shares            5
Threshold               3
Version                 1.8.3
Storage Type            raft
Cluster Name            vault-cluster-62a5fa0a
Cluster ID              97d206e3-d4a2-2c1f-f898-6a95c034b87e
HA Enabled              true
HA Cluster              n/a
HA Mode                 standby
Active Node Address     <none>
Raft Committed Index    24
Raft Applied Index      24
/ $ vault status
Key                     Value
---                     -----
Seal Type               shamir
Initialized             true
Sealed                  false #<<<<<<<<<<<#########
Total Shares            5
Threshold               3
Version                 1.8.3
Storage Type            raft
Cluster Name            vault-cluster-62a5fa0a
Cluster ID              97d206e3-d4a2-2c1f-f898-6a95c034b87e
HA Enabled              true
HA Cluster              https://vault-0.vault-internal:8201
HA Mode                 active
Active Since            2025-09-03T20:26:58.922335892Z
Raft Committed Index    29
Raft Applied Index      29

Vault-0 becomes 1/1 Ready state (now vault is unsealed):

❯❯❯ k get po                                                        ⎈ (kind-my-cluster/vault)
NAME                                    READY   STATUS    RESTARTS   AGE
vault-0                                 1/1     Running   0          27m
vault-agent-injector-6d97459765-4mc6b   1/1     Running   0          27m

/ $ vault operator init 
Unseal Key 1: Q3Awl3Qw+ItIWFyLFVGFiE3OOJ1qHrHC+kQnFD2kwLbE
Unseal Key 2: OON236vYc+bf3K45J75lVHy3FmwIRdUT+mGPU3Iq4SX9
Unseal Key 3: N53iBqGPiL9RRkNNVg5DmrlD5dn6R3Vt703y6NhVUB+0
Unseal Key 4: mWPyNPitk8JTlxZByNkyYJefDo+MTpfIEcDn/lkaKuP3
Unseal Key 5: kRHm81+cYIbjO12eFJp0iJP8y3u7CR4NkOAb/4nHS5Kl

Initial Root Token: s.bQN8VBeKBga76dAUVmaGyTjJ

/ $ vault login s.bQN8VBeKBga76dAUVmaGyTjJ
Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key                  Value
---                  -----
token                s.bQN8VBeKBga76dAUVmaGyTjJ
token_accessor       mkdn3cQnsFcO4hmklMrV8Chq
token_duration       ∞
token_renewable      false
token_policies       ["root"]
identity_policies    []
policies             ["root"]
/ $

# Enable Kubernetes Authentication
kubectl exec -n vault -it vault-0 -- vault auth enable kubernetes
# Create Service Account for App Pods
kubectl create namespace webapps
kubectl create serviceaccount vault-auth -n webapps

# Configure Kubernetes Auth in Vault
# Extract required info:
SERVICE_ACCOUNT_NAME=vault-auth
NAMESPACE=webapps

# JWT Token
TOKEN_REVIEW_JWT=$(kubectl get secret $(kubectl get serviceaccount $SERVICE_ACCOUNT_NAME -n $NAMESPACE -o jsonpath="{.secrets[0].name}") -n $NAMESPACE -o jsonpath="{.data.token}" | base64 --decode)

# Kubernetes API Host
KUBE_HOST=$(kubectl config view --raw -o=jsonpath='{.clusters[0].cluster.server}')

# Kubernetes CA Cert
KUBE_CA_CERT=$(kubectl get secret $(kubectl get serviceaccount $SERVICE_ACCOUNT_NAME -n $NAMESPACE -o jsonpath="{.secrets[0].name}") -n $NAMESPACE -o jsonpath="{.data['ca.crt']}" | base64 --decode)

# Configure in Vault:
kubectl exec -n vault -it vault-0 -- vault write auth/kubernetes/config \
    token_reviewer_jwt="$TOKEN_REVIEW_JWT" \
    kubernetes_host="$KUBE_HOST" \
    kubernetes_ca_cert="$KUBE_CA_CERT"

Purpose of Kubernetes Auth Configuration:
This command configures Vault to trust and communicate with the Kubernetes API server so that:

Vault can validate that service account tokens are legitimate
Pods can authenticate to Vault using their Kubernetes service account tokens
Vault can verify which namespace and service account a request is coming from Breaking Down Each Parameter: token_reviewer_jwt

"$(kubectl get secret vault-token-reviewer -o jsonpath='{.data.token}' | base64 -d)"

What it is: A service account token that Vault uses to authenticate with the Kubernetes API
Why needed: Vault needs to call the Kubernetes TokenReview API to validate incoming service account tokens
What it does: Acts as Vault's "identity" when talking to Kubernetes

kubernetes_host

"https://kubernetes.default.svc:443"

What it is: The URL of the Kubernetes API server from inside the cluster
Why needed: Vault needs to know where to send TokenReview requests
kubernetes.default.svc: This is the internal DNS name for the Kubernetes API server

kubernetes_ca_cert

"$(kubectl get secret vault-token-reviewer -o jsonpath='{.data.ca\.crt}' | base64 -d)"

What it is: The Certificate Authority certificate for the Kubernetes cluster
Why needed: Vault uses this to verify the SSL certificate when connecting to the Kubernetes API
Security: Prevents man-in-the-middle attacks

The Authentication Flow:
Here's what happens when a pod tries to authenticate to Vault:

1. Pod → Vault: "Here's my service account token"
2. Vault → Kubernetes API: "Is this token valid?" (using token_reviewer_jwt)
3. Kubernetes API → Vault: "Yes, it belongs to service account X in namespace Y"
4. Vault → Pod: "Authentication successful, here's your Vault token"

Without this configuration:

Vault can't verify that service account tokens are real
Any pod could potentially forge authentication requests
You lose the security boundary that Kubernetes provides

With this configuration:

Vault cryptographically verifies each authentication request
Only legitimate pods with valid service account tokens can authenticate
You get fine-grained access control based on Kubernetes RBAC

This step essentially creates a trust relationship between Vault and your Kubernetes cluster, enabling secure, automated authentication for your workloads.

Create Vault Policy: Create a file myapp-policy.hcl:
A policy in Vault is a set of rules (paths + capabilities) that defines what a token/identity is allowed to do.
Vault is deny-by-default—nothing is allowed unless a policy explicitly grants it.

# Access to read/write secret data
path "secret/data/mysql" {  
  capabilities = ["create", "update", "read", "delete", "list"]
}

path "secret/data/frontend" {
  capabilities = ["create", "update", "read", "delete", "list"]
}

# Access to list secrets under the path
path "secret/metadata/mysql" {
  capabilities = ["list"]
}

path "secret/metadata/frontend" {
  capabilities = ["list"]
}

Apply Vault Policy:

kubectl cp myapp-policy.hcl vault/vault-0:/tmp/myapp-policy.hcl
kubectl exec -n vault -it vault-0 -- vault policy write myapp-policy /tmp/myapp-policy.hcl
vault policy read <policy name>                   # show a policy
vault policy list
# to use the app policy token 
export VAULT_TOKEN="$(vault token create -field token -policy=app)"

Create Role in Vault to Map Pod to Policy:

kubectl exec -n vault -it vault-0 -- vault write auth/kubernetes/role/vault-role \
    bound_service_account_names=vault-auth \
    bound_service_account_namespaces="webapps" \
    policies=myapp-policy \
    ttl=24h

kubectl describe clusterrolebindings vault-server-binding

Store Secrets in Vault:

# Enable KV V2 Engine at the specific path (secret)
kubectl exec -n vault -it vault-0 -- vault secrets enable -path=secret -version=2 kv
# Store Secrets in Vault
kubectl exec -n vault -it vault-0 -- vault kv put secret/mysql MYSQL_DATABASE=bankappdb MYSQL_ROOT_PASSWORD=Test@123
kubectl exec -n vault -it vault-0 -- vault kv put secret/frontend MYSQL_ROOT_PASSWORD=Test@123
kubectl exec -n vault -it vault-0 -- vault kv get secret/frontend

What makes KV v2 different from v1:

Versioning: every write creates a new version of the secret.
Soft delete & undelete: you can delete specific versions and later restore them.

Example:

/ $ vault secrets enable -path=crds kv-v2
Success! Enabled the kv-v2 secrets engine at: crds/
/ $ vault kv put crds/mysql username=root
Key              Value
---              -----
created_time     2025-09-04T12:00:14.567764837Z
deletion_time    n/a
destroyed        false
version          1
/ $ vault kv put crds/mysql username=root password=12345
Key              Value
---              -----
created_time     2025-09-04T12:00:22.119818763Z
deletion_time    n/a
destroyed        false
version          2
/ $ vault kv get crds/mysql
====== Metadata ======
Key              Value
---              -----
created_time     2025-09-04T12:00:22.119818763Z
deletion_time    n/a
destroyed        false
version          2

====== Data ======
Key         Value
---         -----
password    12345
username    root
/ $ vault kv get crds/mysql
====== Metadata ======
Key              Value
---              -----
created_time     2025-09-04T12:00:22.119818763Z
deletion_time    n/a
destroyed        false
version          2

====== Data ======
Key         Value
---         -----
password    12345
username    root
/ $ vault kv metadata get crds/mysql
========== Metadata ==========
Key                     Value
---                     -----
cas_required            false
created_time            2025-09-04T12:00:14.567764837Z
current_version         2
delete_version_after    0s
max_versions            0
oldest_version          0
updated_time            2025-09-04T12:00:22.119818763Z

====== Version 1 ======
Key              Value
---              -----
created_time     2025-09-04T12:00:14.567764837Z
deletion_time    n/a
destroyed        false

====== Version 2 ======
Key              Value
---              -----
created_time     2025-09-04T12:00:22.119818763Z
deletion_time    n/a
destroyed        false
/ $ vault kv delete crds/mysql
Success! Data deleted (if it existed) at: crds/mysql
/ $ vault kv get crds/mysql
====== Metadata ======
Key              Value
---              -----
created_time     2025-09-04T12:00:22.119818763Z
deletion_time    2025-09-04T12:02:04.56772668Z
destroyed        false
version          2

/ $

Create YAML Manifest File (With Below Configurations):
Example annotation block for a pod:

annotations:
  vault.hashicorp.com/agent-inject: "true" # this should be set to a true or false, default to false
  vault.hashicorp.com/role: "vault-role" # role name
  vault.hashicorp.com/agent-inject-secret-MYSQL_ROOT_PASSWORD: "secret/mysql" # agent-inject-secret-<the name of the secret> , this will retrieve the secret from Vault
  # agent-inject-template-<the name of the secret>: used for rendering a secret
  vault.hashicorp.com/agent-inject-template-MYSQL_ROOT_PASSWORD: |
    {{- with secret "secret/mysql" -}}
    export MYSQL_ROOT_PASSWORD="{{ .Data.data.MYSQL_ROOT_PASSWORD }}"
    {{- end }}

what to read and how to write.
vault.hashicorp.com/agent-inject-secret-MYSQL_ROOT_PASSWORD: "<vault path>"
Tells the injector which secret to fetch from Vault and that it should write a file named MYSQL_ROOT_PASSWORD under /vault/secrets/ (unless you override the filename/path).
vault.hashicorp.com/agent-inject-template-MYSQL_ROOT_PASSWORD:
Provides a custom template for what to write into that file. Without this, the injector writes a default rendering (key=value lines). With it, you control the exact content.
Template: It renders a shell line into the injected file, like: export MYSQL_ROOT_PASSWORD="supersecret123"

    {{- with secret "secret/mysql" -}}
    export MYSQL_ROOT_PASSWORD="{{ .Data.data.MYSQL_ROOT_PASSWORD }}"
    {{- end }}

The sidecar Vault Agent will:
• Auth using the service account token
• Fetch secrets from Vault
• Write them to /vault/secrets/... >>> /vault/secrets/MYSQL_ROOT_PASSWORD inside the pod

UI access

kubectl -n vault get svc vault
kubectl -n vault port-forward --address=0.0.0.0 svc/vault 8200:8200

Clean-up steps

helm uninstall vault
kubectl -n vault delete pod -l app.kubernetes.io/name=vault --force --grace-period=0 || true
kubectl -n vault delete pvc -l app.kubernetes.io/name=vault || true

Socket management and Kernel Data structures

Omar Ahmed — Mon, 12 May 2025 09:16:52 +0000

🧠 Concept of Socket and Listening Mechanism

In Linux, a socket is a special type of file file descriptor used for inter-process communication (IPC) or network communication. It acts as an endpoint for sending or receiving data between processes, either on the same system or across different systems over a network. Sockets provide a standardized way for programs to exchange data, making them a fundamental concept in networking and system programming.
When a Process listens on an IP/Port it produces a socket.

🧵 Connection Lifecycle and Queues

A socket transitions to a connection through the TCP three-way handshake: SYN, SYN-ACK, and ACK. Two key kernel-level queues are introduced:
1. Socket SYN Queue:
  - What it is:
    - Also called the incomplete connection queue.
    - Holds incoming TCP connection requests that are in the initial stage of the TCP three-way handshake.
    - Holds half-open connections awaiting handshake completion.
    - It stores connections that have received a SYN packet from a client but haven’t completed the handshake yet.
  - How it works Three Way Handshake:
    1. A client sends a SYN packet to the server to initiate a connection.
    2. The server responds with a SYN-ACK packet and places the connection in the SYN Queue.
    3. The connection remains in this queue until the client sends an ACK packet to complete the handshake.
2. Socket Accept Queue: Stores fully established connections ready for backend processes to accept using the accept() system call.
  - What it is:
    - Also called the completed connection queue.
    - Holds fully established TCP connections that have completed the three-way handshake SYN, SYN-ACK, ACK.
    - It stores complete connections and are waiting for the application to accept them via the accept() system call.
  - How it works:
    1. Once the client sends the final ACK, the connection moves from the SYN Queue to the Accept Queue.
    2. The server’s application (e.g., a web server) calls accept() to dequeue a connection and begin processing it.

⛵ Connection Queue

Once the connection is established, it's represented in the kernel by a file descriptor.
This file descriptor holds the state and metadata of the connection and lives in the accepting process’s memory space (PCB).
1. Receive Queue:
  - The receive queue is part of the kernel memory space associated with an individual connection.
  - It temporarily stores incoming data from the client before it's passed to the application (e.g., your server).
  - If the backend does not read fast enough, the receive queue fills up, leading the kernel to trigger flow control that tells the client to slow down.
2. Send Queue:
  - The send queue stores outgoing data from the server that will be transmitted to the client.
  - When the application writes data (e.g., API response), it's placed here first—not sent immediately.
  - The kernel may delay actual transmission to optimize performance, bundling small writes into larger packets before flushing them to the network.

Hi

Omar Ahmed — Mon, 12 May 2025 09:11:01 +0000

First Hello from dev.to, I'm a DevOps Engineer!