container

Coldpatch Container Image: Quick and efficient container image patching with Project Copacetic

Phil Huang

18 Jun 2025 • 6 min read

Photo by Phil Huang @ Zhangmen Brewery, Taipei, Taiwan

Based on previous post Coldpatch Container Image: Scanning Images with Trivy , some vulnerabilities that can be patched were found in the container image. So how can we perform the patching at the lowest cost and effort?

Microsoft lauched a open source project called Project Copacentic (Copa) on 2024/9/18, which is now one of the CNCF Sandbox projects. It can perform real-time updates for fixable vulenerabilities or packages based on vulnerability reports generated by Trivy.

About Project Copacetic

Use aquasecurity/trivy or other methods to generate a vulnerability report
project-copacetic/copacetic identifies the packages that need to be updated based on the analysis of the vulnerabiltiy report
Use moby/buildkit to apply these updates to container image
project-copacetic/copacetic will automatically donwload the same OS base images as the existing container image to assist with patching.
Directly create a new patch layer on the existing container layer without needing to use a Dockerfile to build the existing image.

Investigate the number of fixable vulnerabilities

Use mcr.microsoft.com/azurelinux/base/nginx:1.25 as an exmaple

#!/bin/bash
IMAGE=mcr.microsoft.com/azurelinux/base/nginx:1.25.4-2-azl3.0.20250102
REPORT_NAME=azurelinux-base-nginx.json

docker pull ${IMAGE}

# Generate a report for the specified image
trivy image \
  --pkg-types os,library \
  --format json \
  --quiet \
  --output ${REPORT_NAME} \
  ${IMAGE}

jq '
  [ ..
    | .Vulnerabilities?
    | .[]?
  ] as $all |
  {
    "Total number of vulnerabilities":        ($all | length),
    "Number of fixable vulnerabilities":  ($all | map(select(.FixedVersion != null and .FixedVersion != "")) | length),
    "Number of non-fixable vulnerabilities":($all | map(select(.FixedVersion == null or .FixedVersion == "")) | length)
  }
' ${REPORT_NAME}

understand-your-container-image.sh

{
  "Total number of vulnerabilities": 46,
  "Number of fixable vulnerabilities": 46,
  "Number of non-fixable vulnerabilities": 0
}

understand-your-container-image.output

Until now, you can find that ther are 46 vulnerabilities that can be fixed, so will use Copa to carry out the patching operations.

Use Copa to Patch

Simply execute copa patch to proceed with the patching

#!/bin/bash
IMAGE=mcr.microsoft.com/azurelinux/base/nginx:1.25.4-2-azl3.0.20250102
REPORT_NAME=azurelinux-base-nginx.json

copa patch \
    --report ${REPORT_NAME} \
    --image ${IMAGE}

patch-by-copa.sh

...omit...
[+] Building 72.6s (17/17) FINISHED                                                                                          
 => resolve image config for docker-image://mcr.microsoft.com/azurelinux/base/nginx:1.25.4-2-azl3.0.20250102            0.4s
 => docker-image://mcr.microsoft.com/azurelinux/base/core:3.0                                                           2.4s
 => => resolve mcr.microsoft.com/azurelinux/base/core:3.0                                                               0.8s
 => => sha256:12ff10226b498639c637d7f81c12ecbd567494c4b8c26a9180edad4b500ff987 31.51MB / 31.51MB                        1.0s
 => => extracting sha256:12ff10226b498639c637d7f81c12ecbd567494c4b8c26a9180edad4b500ff987                               0.5s
 => sh -c ls /usr/bin > applications.txt                                                                                1.7s
 => CACHED docker-image://mcr.microsoft.com/azurelinux/base/nginx:1.25.4-2-azl3.0.20250102                              0.0s
 => => resolve mcr.microsoft.com/azurelinux/base/nginx:1.25.4-2-azl3.0.20250102                                         0.0s
 => tdnf install busybox dnf-utils cpio -y                                                                             39.6s
 => copy /usr/sbin/busybox /usr/sbin/busybox                                                                            0.4s
 => mkdir /copa-out                                                                                                     0.2s
 => mkdir /copa-input                                                                                                   0.2s
 => mkfile /copa-input/tool_list                                                                                        0.3s
 => mkfile /copa-input/rpm_db_list                                                                                      0.4s
 => /usr/sbin/busybox sh -c dnf -q check-update | wc -l)" -ne 0 ]; then echo >> /updates.txt; fi                        6.3s
                while IFS= read -r tool; dobelf libssh2 curl-libs libtasn1 openssl-libs xz-libs libxml2 libxml2-devel  16.1s
                    tool_path="$($BUSYBOX whi  0.7s-%{RELEASE} %{ARCH}.4-2-azl3.0.20250102, diff (docker-image://mcr.m  0.0s
 => sh -c if [ "$(/usr/bin/tdnf -q check-update | wc -l)" -ne 0 ]; then echo >> /updates.txt; fi                        6.3s
 => sh -c /usr/bin/tdnf upgrade elfutils-libelf libssh2 curl-libs libtasn1 openssl-libs xz-libs libxml2 libxml2-devel  16.1s
 => sh -c rpm -qa --queryformat "%{NAME} %{VERSION}-%{RELEASE} %{ARCH}.4-2-azl3.0.20250102, diff (docker-image://mcr.m  0.0s
" elfutils-libelf libssh2 curl-libs libtasn1 op  0.7s                                                                   2.4s
 => diff (docker-image://mcr.microsoft.com/azurelinux/base/nginx:1.25.4-2-azl3.0.20250102) -> (sh -c /usr/bin/tdnf upg  0.1s
 => merge (docker-image://mcr.microsoft.com/azurelinux/base/nginx:1.25.4-2-azl3.0.20250102, diff (docker-image://mcr.m  0.0s
 => exporting to docker image format                                                                                    2.4s
 => => exporting layers                                                                                                 1.2s
 => => exporting manifest sha256:7fd229f65ee77bf509321aafdf608d55af88a191b5fd9c7485da8e7ac54389e5                       0.0s
 => => exporting config sha256:6b9a6c8966b14113deccbd5bf1c5adfa17ae664b0e09174288fd793766d5cfcc                         0.0s
 => => sending tarball                                                                                                  1.1s
INFO[0072] Loaded image: mcr.microsoft.com/azurelinux/base/nginx:1.25.4-2-azl3.0.20250102-patched

patch-by-copa.output

Now you will get a new container image with the --patched postfix.

docker-images

Rescan the Patched Container Image

#!/bin/bash
IMAGE=mcr.microsoft.com/azurelinux/base/nginx:1.25.4-2-azl3.0.20250102-patched
REPORT_NAME=azurelinux-base-nginx-patched.json

# Generate a report for the specified image
trivy image \
  --pkg-types os,library \
  --format json \
  --quiet \
  --output ${REPORT_NAME} \
  ${IMAGE}

jq '
  [ ..
    | .Vulnerabilities?
    | .[]?
  ] as $all |
  {
    "Total number of vulnerabilities":        ($all | length),
    "Number of fixable vulnerabilities":  ($all | map(select(.FixedVersion != null and .FixedVersion != "")) | length),
    "Number of non-fixable vulnerabilities":($all | map(select(.FixedVersion == null or .FixedVersion == "")) | length)
  }
' ${REPORT_NAME}

understand-your-container-image.sh

{
  "Total number of vulnerabilities": 0,
  "Number of fixable vulnerabilities": 0,
  "Number of non-fixable vulnerabilities": 0
}

understand-your-container-image.output

✅ All done, the entire steps is very straightforward, directly patching until completion.

About Patch Layer

You can use docker history to check what changes have been made to this container image

#!/bin/bash
IMAGE=mcr.microsoft.com/azurelinux/base/nginx:1.25.4-2-azl3.0.20250102-patched

docker history ${IMAGE}

docker-history.sh

You can see Copa has added a 20.5MB patch layer to the container layer on this container image, which contains automated patching results based on the collaboration of the 3 tools: Trivy / Copa / Buildkit

Conclusion

Actually, Copa cannot solve all container patch issues. Sometimes, you may encounter situations where execution is impossible.

In this situation, you need to adopt the classic patching approach: re-build using the Patched Dockerfile method for patching.

However, be sure to remember that after completing the repairs, it is neccessary to retest to avoild functionality issues after the upgrade.

If you encounter any issues with functionality after the upgrade, I stronly recommend setting trivy image to --pkg-types os. For the Application level, please contact the development team for repairs to ensure the application function properly.

Appendix: Install Copa

#!/bin/bash
wget https://github.com/project-copacetic/copacetic/releases/download/v0.10.0/copa_0.10.0_linux_amd64.tar.gz
tar zxvf copa_0.10.0_linux_amd64.tar.gz
sudo install copa /usr/local/bin/
rm copa LICENSE README.md
copa -v

install-copa.sh

Appendix: Enable containerd snapshotter on Docker

#!/bin/bash

# Ref
# https://docs.docker.com/engine/storage/containerd/#enable-containerd-image-store-on-docker-engine

if [ "$EUID" -ne 0 ]; then
  echo "Please run as root or use sudo."
  exit 1
fi

cat <<EOF > /etc/docker/daemon.json
{
  "features": {
    "containerd-snapshotter": true
  }
}
EOF

sudo systemctl restart docker

echo "BuildKit has been enabled in Docker configuration."
echo "Result should show "[[driver-type io.containerd.snapshotter.v1]]""
docker info -f '{{ .DriverStatus }}'

enable-containerd-snapshotter.sh