Signed Container Images

ID

signed_container_image

Severity

high

Family

CI/ CD Security

Tags

cicd-sec-09, cicd-security, non-reachable, releases, security, supply-chain

Description

Container images should be cryptographically signed.

This check tries to determine if the project cryptographically signs container the images pushed to registries.

Security

Container image signing enables users to incorporate a digital "fingerprint" into an image, which can be subsequently subjected to cryptographic validation for establishing trust. This functionality empowers users of container images to authenticate the source and place confidence in the included content of such images. Compliance requirements and industry standards may demand image signing.

Image signing and verification, when done correctly, provides source authentication, integrity protection, and other services, like non-repudiation of the signer. Tampering with the image once it is signed and published in a registry is impossible (unless the signature storage, the image consumer or the registry are compromised).

Signing images needs:

  • A cryptography setup: a managed signature keypair, an external Key Management System (KMS), a hardware token with digital signature support, an ephemeral certified keypair ("keyless" signatures), or tooling like GPG,

  • a store for signatures and signer public keys or public key certificates -which could be an OCI registry, and

  • a policy enforcing verification of signatures on image consumers, so that any unsigned image, or an image with an invalid signature, is rejected. This is the most transparent option, with explicit signature verification the alternative.

There are several available tools for signing container images. Here are a few popular ones, in alphabetical order:

  • Cosign is an open-source tool designed for signing container images. It uses public-key cryptography to create signatures and verify the integrity of container images. It integrates well with container registries, and it is easy to use.

  • Docker provides built-in support for image signing through Docker Content Trust (DCT). It uses digital signatures and the Notary service to verify the integrity and authenticity of container images.

  • Podman is a container runtime and management tool that supports image signing.

  • Notary is an open-source project for trusted collections under The Update Framework specification. Provides trust and security for container image distribution. It is designed to ensure the integrity and authenticity of container images by allowing developers to sign and verify image signatures.

    The tool under Notary for image signing, Notation, is a CLI project to add signatures as standard items in the OCI registry ecosystem, and to build a set of simple tooling for signing and verifying these signatures.

  • Skopeo is a command-line utility for various operations on container images and image repositories, with support for image signing and verification.

The different runtimes can be configured to require signed container images and perform the signature verification as part of the image admission process. Read, for example, "Verifying Container Image Signatures Within CRI Runtimes" as an example of how to do this for Kubernetes.

Mitigation / Fix

Sign your container images, if not done already. The following are some examples for different tools.

Always reference and use the image digest (@sha256:…​) instead of a :tag, since digest is immutable and represents precisely the image tarball.

Cosign

Cosign tries to make cryptography transparent by using keyless signatures: the signer is authenticated via a verifiable OpenID Connect (OIDC) identity token, and generates a single-use keypair, with the public key in an ephemeral certificate that is persisted along with the image digest and signature in an immutable transparency log.

# Sign an image with your private key
cosign sign myregistry.com/myimage@digest

# Verify the image. The OIDC issuer + signer identity must be known
cosign verify myregistry.com/myimage:tag \
       --certificate-identity=subject@example.com \
       --certificate-oidc-issue=https://accounts.example.com

If you prefer to generate your keys:

# Generate a key pair (once)
cosign generate-key-pair

# Sign an image with your private key
cosign sign --key cosign.key myregistry.com/myimage:tag

# Verify an image. The public key must be distributed to verifiers
cosign verify --key cosign.pub myregistry.com/myimage:tag

More advanced usage includes using your own keys: importing key pairs or using hardware keys or KMS providers like AWS KMS, GCP KMS, Azure KeyVault, Hashicorp Vault or Kubernetes Secrets

Docker Content Trust (DCT)

Docker Content Trust (DCT) provides the ability to use digital signatures for data sent to and received from remote Docker registries. These signatures allow client-side or runtime verification of the integrity and publisher of specific image tags.

A prerequisite for signing an image with DCT is a Docker Registry with a Notary server attached (such as the Docker Hub)

# Generate keypair
docker trust key generate subject-example-com

# Add the delegation public key to the Notary server, and initiate the repository if not done
docker trust signer add \
  --key subject-example-com.key subject-example-com \
  myregistry.com/myimage

# Use the delegation private key to sign the image
# at a particular tag and push it up to the registry
docker trust sign myregistry.com/myimage:tag

# Verify the image.
# The signer public key and image signature are stored in the registry
docker trust inspect --pretty myregistry.com/myimage:tag

Alternatively, once the keys have been imported, an image can be automatically signed during push by setting the DOCKER_CONTENT_TRUST environment variable, which enables DCT in the Docker client: docker CLI commands that operate on tagged images must either have content signatures or explicit content hashes. The commands that operate with DCT are push, build, create, pull and run.

# Enable DCT
export DOCKER_CONTENT_TRUST=1

# Sign an image (happens automatically on push with DCT enabled)
docker push myregistry.com/myimage:tag

# Verify an image
docker trust inspect --pretty myregistry.com/myimage:tag

DCT is built on top of Notary project.

For full information, read Content trust in Docker.

Notation

With notation, the following assumes that a keypair is already registered with notation. The image signature is stored as an artifact in the OCI registry.

# Use the image digest instead of the tag, used here for clarity
IMAGE=myregistry.com/myimage:tag

# Sign the container image
notation sign $IMAGE
# or sign using

# List the image signature (and other OCI artifacts
# for the image repository)
notation ls $IMAGE

# At the image consumer side: first add a Notary trust policy
# Import the trust policy
notation policy import ./trustpolicy.json
# Verify the image against the policy,
# which includes signature verification
notation verify $IMAGE

For full details, read the Notation Quick Start Guide.

Podman

Podman uses GPG keys for image signing. This setup can be used to sign container images in Kubernetes environments.

To make image signing with podman work:

  1. We need a valid private GPG key on the signing machine and corresponding public keys on every system which would pull the image

  2. A web server has to run somewhere which has access to the signature storage

  3. A web server for signature storage has to be configured in any /etc/containers/registries.d/*.yaml file

  4. Every image pulling system has to be configured to contain the enforcing policy configuration via /etc/containers/policy.json. Read Verifying GPG image signatures for further information.

With this setup, podman automatically signs (by passing the GPG identity with --sign-by) and verifies image signatures (due to the enforcing policy):

# Podman uses GPG keys, list the signer keys
gpg --list-keys subject@example.com

# Sign an image. Assume gpg is configured
podman push --sign-by subject@example.com myregistry.com/myimage:tag

# Verify an image (in fact, pull will verify automatically the signature)
podman pull myregistry.com/myimage:tag

Read the tutorial "How to sign and distribute container images using Podman" for further information.

Skopeo

The Skopeo copy command can sign and push an image to a registry with the --sign-by option. Skopeo also provides stand-alone-sign and stand-alone-verify commands for signature testing and verification:

# Sign a local image and push it to OCI registry
skopeo copy --sign-by subject@example.com \
  docker-daemon:myimage:tag \
  docker://myregistry.com/myimage:tag

# Verify the image signature
skopeo stand-alone-verify myregistry.com/myimage:tag

This "Signing and Verifying Container Images" post gives additional examples on using Skopeo for signing and verifying signatures for container images.