Supply Chain Security Labs

We wrote 38 hands-on labs covering every signing and supply chain verification technique we use during DevSecOps assessments. Each lab walks through a real gap in the signing chain.

Before we start, shoutout to a platform we built for YOU!

💎 Your next level in cybersecurity isn’t a dream, it’s a proactive roadmap.

HADESS AI Career Coach turns ambition into expertise:

→ 390+ clear career blueprints from entry-level to leadership

→ 490+ in-demand skill modules + practical labs

→ Intelligent AI(Not AI buzz, applied AI, promise!) tools + real-world expert coaches and scenarios

Master the skills that matter. Land the roles that pay. Build the future you want.

🔥 Start engineering your career →

https://career.hadess.io

---

We wrote 38 hands-on labs covering every signing and supply chain verification technique we use during DevSecOps assessments. Each lab walks through a real gap in the signing chain, shows you what an attacker does when that gap exists, and gives you the fix with verification steps.

The labs are grounded in real incidents: the Trivy TeamPCP attack, the tj-actions/changed-files compromise (CVE-2025-30066), the Codecov bash uploader breach, the xz-utils backdoor (CVE-2024-3094), the SolarWinds SUNBURST campaign, the 3CX cascading supply chain attack, and NotPetya. Each attack maps directly to a lab that teaches the control that would have caught it.

The labs span six areas: Sigstore ecosystem fundamentals (Cosign, Fulcio, Rekor, keyless signing), CI/CD signing pipelines (GitHub Actions, GitLab CI, AWS KMS, GCP KMS, Azure Key Vault), AWS Signer for Lambda and ECR with Terraform, artifact attestation and provenance (SLSA, in-toto, SBOM signing, GitHub artifact attestation), deployment verification (Helm chart signing, ArgoCD GPG verification, git commit signing), and admission control (Kyverno, OPA Gatekeeper, Sigstore policy controller, Notation).

A key theme throughout: use OIDC federation and workload identity instead of long-lived secrets. Every CI/CD lab shows the federated approach -- GitHub Actions OIDC to AWS, GCP Workload Identity Federation, Azure federated credentials -- so there are zero stored secrets in your pipelines.

---

Table of Contents

Sigstore Ecosystem and Cloud KMS

1. Cosign Key-Based Container Image Signing

2. Cosign Keyless Signing with Sigstore

3. Fulcio Certificate Authority and Rekor Transparency Log

4. GitHub Actions Image Signing with Cosign

5. GitLab CI Image Signing with Cosign

6. Cosign Signing with AWS KMS

7. Cosign Signing with GCP KMS and Azure Key Vault

AWS Signer

8. AWS Signer for Lambda Functions with Terraform

9. AWS Signer for ECR Container Images

Artifact Attestation and Provenance

10. GitHub Actions Artifact Attestation Signing

11. SLSA Provenance Generation and Verification

12. In-toto Attestation Framework

13. SBOM Signing and Attestation

Helm, ArgoCD, and Git Signing

14. Helm Chart Cosign Signature Verification

15. Helm Chart Provenance Files

16. ArgoCD GPG Signature Verification for Git Commits

17. ArgoCD with Cosign Image Verification

18. Git Commit GPG/SSH Signing

Package and Binary Signing

19. NPM Package Provenance and Signing

20. Python Package Signing with Sigstore

21. Go Binary Signing with Cosign and GoReleaser

22. Terraform Provider and Module Signing Verification

OIDC Federation (No Secrets)

23. GitHub Actions OIDC Federation for AWS

24. GitHub Actions OIDC Federation for GCP

25. GitHub Actions OIDC Federation for Azure

Admission Control and Policy Enforcement

26. Kyverno Admission Policies for Signature Enforcement

27. OPA Gatekeeper Signature Constraints

28. OCI Artifacts for Signatures and Attestations

29. Notation (CNCF) Container Image Signing

30. SLSA Framework Implementation (L1 through L3)

31. Policy-as-Code for Signature Enforcement

32. Sigstore Policy Controller for Kubernetes

33. Complete End-to-End Signing Pipeline

Real-World Attack Scenarios

34. GitHub Actions Supply Chain Security (tj-actions, Trivy TeamPCP)

35. CI Script and Tool Verification (Codecov Attack)

36. Multi-Party and Threshold Signing (xz-utils Backdoor)

37. Reproducible Build Verification (SolarWinds, 3CX)

38. The Update Framework / TUF (NotPetya)

---

Lab 01 — Cosign Key-Based Container Image Signing

Field Value Lab Title Cosign Key-Based Container Image Signing Risk Rating Critical MITRE ATT&CK T1525 — Implant Internal Image CIS Benchmark CIS Kubernetes 5.5.1 — Ensure Image Provenance Tools cosign, crane, docker Platform OCI Registry, CI/CD Last Updated 2026-03-27

---

Writeup

We build container images and push them to a registry. Without signing, we have no way to prove that an image came from our pipeline. An attacker who gains push access to the registry — or compromises a pull-through cache — can replace any tag with a backdoored image. Kubernetes pulls it on the next restart and nobody notices.

Cosign key-based signing attaches a cryptographic signature to the image digest using a private key. At verification time, the corresponding public key confirms the image was signed by someone holding the private key. This is the simplest signing model: generate a key pair, sign with the private key, verify with the public key.

The tradeoff is key management. The private key needs to be stored securely, rotated periodically, and never committed to version control. If the key leaks, an attacker can sign their own images and bypass verification. Key-based signing is a solid starting point, but we cover keyless signing in Lab 02 for environments where key management is a burden.

Root Cause Analysis

1. No signing step in the CI/CD pipeline — images ship unsigned

2. Images referenced by mutable tags (latest, v1.2.0) instead of immutable digests

3. No admission controller checks signatures at deploy time

4. Registry credentials have push access without audit trail on tag overwrites

5. Teams treat “it’s in our registry” as proof of provenance

Vulnerable Configuration

CI Pipeline — No Signing

# .github/workflows/build.yaml
name: Build and Push
on:
  push:
    branches: [main]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Login to registry
        run: echo "${{ secrets.REGISTRY_PASSWORD }}" | docker login ghcr.io -u ${{ github.actor }} --password-stdin
      - name: Build image
        run: docker build -t ghcr.io/acme/webapp:${{ github.sha }} .
      - name: Push image
        run: docker push ghcr.io/acme/webapp:${{ github.sha }}
      # No signing. Image is pushed and forgotten.

Kubernetes Deployment — Mutable Tag, No Verification

apiVersion: apps/v1
kind: Deployment
metadata:
  name: webapp
spec:
  replicas: 3
  template:
    spec:
      containers:
        - name: webapp
          image: ghcr.io/acme/webapp:latest  # Mutable tag, no signature check

Exploitation

Step 1: Confirm No Signatures Exist

$ cosign verify --key cosign.pub ghcr.io/acme/webapp:latest
Error: no matching signatures: no signatures found for ghcr.io/acme/webapp:latest

The image has zero signatures. Nothing ties it to our build pipeline.

Step 2: Check the Current Digest

$ crane digest ghcr.io/acme/webapp:latest
sha256:a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4e5f6a1b2c3d4e5f6a1b2

Step 3: Attacker Overwrites the Tag

# Attacker has compromised registry credentials
$ cat Dockerfile.backdoor
FROM ghcr.io/acme/webapp:latest
COPY reverse-shell /usr/local/bin/healthcheck
ENTRYPOINT ["/usr/local/bin/healthcheck"]

$ docker build -t ghcr.io/acme/webapp:latest -f Dockerfile.backdoor .
$ docker push ghcr.io/acme/webapp:latest

Step 4: Confirm the Digest Changed

$ crane digest ghcr.io/acme/webapp:latest
sha256:ff00ff00ff00ff00ff00ff00ff00ff00ff00ff00ff00ff00ff00ff00ff00ff00

Different digest. The tag now points to the attacker’s image.

Step 5: Kubernetes Pulls the Backdoored Image

$ kubectl rollout restart deployment/webapp
$ kubectl get pods -l app=webapp -o jsonpath='{.items[0].status.containerStatuses[0].imageID}'
ghcr.io/acme/webapp@sha256:ff00ff00...

No admission controller blocked it. The backdoor is running in production.

Detection

Check for Signatures on Any Image

$ cosign tree ghcr.io/acme/webapp:latest
No signatures found
No attestations found

Compare Digests Against Build Logs

$ crane digest ghcr.io/acme/webapp:latest
# Compare with the digest recorded in CI build output
# Mismatch = tag was overwritten

Registry Audit Logs

Check your registry’s audit log for push events that did not originate from CI service accounts.

Solution

Step 1: Generate a Cosign Key Pair

$ cosign generate-key-pair
Enter password for private key:
Enter password for private key again:
Private key written to cosign.key
Public key written to cosign.pub

Store cosign.key in a secrets manager (Vault, AWS Secrets Manager, GitHub Actions secret). Never commit it to the repo. Distribute cosign.pub to all verification points.

Step 2: Sign the Image in CI

# .github/workflows/build-sign.yaml
name: Build, Push, Sign
on:
  push:
    branches: [main]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Install Cosign
        uses: sigstore/cosign-installer@v3

      - name: Login to registry
        run: echo "${{ secrets.REGISTRY_PASSWORD }}" | docker login ghcr.io -u ${{ github.actor }} --password-stdin

      - name: Build and push
        id: build
        run: |
          docker build -t ghcr.io/acme/webapp:${{ github.sha }} .
          docker push ghcr.io/acme/webapp:${{ github.sha }}
          DIGEST=$(crane digest ghcr.io/acme/webapp:${{ github.sha }})
          echo "digest=$DIGEST" >> "$GITHUB_OUTPUT"

      - name: Sign image
        env:
          COSIGN_PASSWORD: ${{ secrets.COSIGN_PASSWORD }}
        run: |
          cosign sign --key env://COSIGN_KEY \
            ghcr.io/acme/webapp@${{ steps.build.outputs.digest }}
        # Signs the digest, not the tag. Tag overwrites cannot forge the signature.

Step 3: Verify Before Deploy

$ cosign verify --key cosign.pub \
    ghcr.io/acme/webapp@sha256:a1b2c3d4e5f6...

Verification for ghcr.io/acme/webapp@sha256:a1b2c3d4e5f6... --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - The signatures were verified against the specified public key

Key Management Notes

• Rotate the key pair periodically — generate new pair, sign new images with new key, update verification configs, phase out old key

• Store the private key in a hardware-backed secrets manager or KMS (see Lab 06)

• If the private key leaks, all signatures made with that key should be considered untrusted — re-sign all images with a new key

Verification

Confirm the Image Is Signed

$ cosign verify --key cosign.pub ghcr.io/acme/webapp@sha256:a1b2c3d4e5f6...

Verification for ghcr.io/acme/webapp@sha256:a1b2c3d4e5f6... --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - The signatures were verified against the specified public key

Confirm an Unsigned Image Fails Verification

$ cosign verify --key cosign.pub ghcr.io/acme/webapp:unverified
Error: no matching signatures: no signatures found

Confirm the Signature Is Attached to the Digest

$ cosign triangulate ghcr.io/acme/webapp@sha256:a1b2c3d4e5f6...
ghcr.io/acme/webapp:sha256-a1b2c3d4e5f6....sig

The .sig tag exists in the registry, confirming the signature is stored alongside the image.

---

Takeaway: Key-based signing is the first step toward image provenance. Sign every image by digest in CI, verify before deploy. But key management is a real operational cost — Lab 02 covers keyless signing to eliminate that burden entirely.

---

Lab 02 — Cosign Keyless Signing with Sigstore