Sign in Subscribe
container

Coldpatch Container Image: Re-build Image using Patched Dockerfile

Phil Huang

• 4 min read
Coldpatch Container Image: Re-build Image using Patched Dockerfile
Photo by Phil Huang @ Beigang Chaotian Temple, Yunlin, Taiwan

The previous post explained that project-copacetic/copacetic (Copa) is very useful real-time container patching tool, but it cannot completely update all container images. You might encounter the GPG errors (NO_PUBKEY, Repository not signed) issue, which prevents you from using Copa for updates no matter what you do.

[BUG] apt-get update fails in Debian Bullseye container with GPG errors (NO_PUBKEY, Repository not signed) · Issue #1108 · project-copacetic/copacetic
Version of copa 0.10.0 Expected Behavior should bypass gpg key issue, and continue install packages Actual Behavior 23:35:37 repairman@vm-tf-bastion-twn 1 ~/dockerfile/layout-3.1 $ ./hot-patch-by-c…
GitHubproject-copacetic

At this point, you can only abandon using Copa and switch to the 2nd method by writing a new Dockerfile for patching. The following will use mcr.microsoft.com/azure-cognitive-services/form-recognizer/layout-3.1:latest for explanation. This container image is a typical example of being unable to use Copa for patching.

copa-failed.output

Investigate the number of fixable vulnerabilities

Use mcr.microsoft.com/azure-cognitive-services/form-recognizer/layout-3.1:latest as an exmaple

#!/bin/bash
IMAGE=mcr.microsoft.com/azure-cognitive-services/form-recognizer/layout-3.1:latest
REPORT_NAME=form-recognizer-layout-3.1.json

docker pull ${IMAGE}

# Generate a report for the specified image
trivy image \
  --pkg-types os,library \
  --format json \
  --quiet \
  --output ${REPORT_NAME} \
  ${IMAGE}

jq '
  [ ..
    | .Vulnerabilities?
    | .[]?
  ] as $all |
  {
    "Total number of vulnerabilities":        ($all | length),
    "Number of fixable vulnerabilities":  ($all | map(select(.FixedVersion != null and .FixedVersion != "")) | length),
    "Number of non-fixable vulnerabilities":($all | map(select(.FixedVersion == null or .FixedVersion == "")) | length)
  }
' ${REPORT_NAME}

understand-your-container-image.sh

{
  "Total number of vulnerabilities": 119,
  "Number of fixable vulnerabilities": 16,
  "Number of non-fixable vulnerabilities": 103
}

understand-your-container-image.output

Check the exising User ID

#!/bin/bash
IMAGE=mcr.microsoft.com/azure-cognitive-services/form-recognizer/layout-3.1:latest

docker inspect --format '{{.Config.User}}' ${IMAGE}

check-user.sh

check-user.output

You will see that the user is www-data. However, if you use another container image and find that the USER is not specified, this field might be an empty string or null. In this case, commands will be executed with root (UID 0) identity.

Write a Patched Dockerfile

Due to the need to resolve the initial GPG NO_PUBKEY issue, we must first draft a solution for the fix.

Acquire::AllowInsecureRepositories "true";
APT::Get::AllowUnauthenticated "true";

80ssl-exceptions

Below is the actual Dockfile, with step-by-step explanations:

  1. COPY 80ssl-exceptions: This step is necessary to allow apt-get update to bypass GPG verification.
  2. USER root: Since only the root user can perform apt-get update, it is required to switch to root first.
  3. RUN apt-get: Updates are limited to security-related packages only, avoiding updates to other packages to prevent additional issues.
  4. USER www-data: As the original user is www-data, it is necessary to switch back to this user in the final layer.
FROM mcr.microsoft.com/azure-cognitive-services/form-recognizer/layout-3.1:latest

# 1. Fix apt-get ssl exceptions
COPY 80ssl-exceptions /etc/apt/apt.conf.d/80ssl-exceptions

# 2. Change root for install packages
USER root

# 3. Install security updates only
RUN apt-get update && \
    apt-get -s dist-upgrade | grep "^Inst" | grep -i securi | awk -F " " '{print $2}' | xargs apt-get install -y && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*

# 4. Change back to www-data user
USER www-data

Dockerfile

Build Patched Container Image

Below are common operaions for using docker build to create a container image.

#!/bin/bash
PATCHED_IMAGE=mcr.microsoft.com/azure-cognitive-services/form-recognizer/layout-3.1:latest-patched

docker build -t ${PATCHED_IMAGE} .

build-patched-image.sh

build-patched-image.output
docker-images.output

Rescan the Patched Container Image

#!/bin/bash
IMAGE=mcr.microsoft.com/azure-cognitive-services/form-recognizer/layout-3.1:latest-patched
REPORT_NAME=form-recognizer-layout-3.1-patched.json

# Generate a report for the specified image
trivy image \
  --pkg-types os,library \
  --format json \
  --quiet \
  --output ${REPORT_NAME} \
  ${IMAGE}

jq '
  [ ..
    | .Vulnerabilities?
    | .[]?
  ] as $all |
  {
    "Total number of vulnerabilities":        ($all | length),
    "Number of fixable vulnerabilities":  ($all | map(select(.FixedVersion != null and .FixedVersion != "")) | length),
    "Number of non-fixable vulnerabilities":($all | map(select(.FixedVersion == null or .FixedVersion == "")) | length)
  }
' ${REPORT_NAME}

understand-your-container-image.sh

{
  "Total number of vulnerabilities": 103,
  "Number of fixable vulnerabilities": 0,
  "Number of non-fixable vulnerabilities": 103
}

understand-your-container-image.ouput

Through this approach, vulnerabilities that can be patched by repair kits can be addressed, with a total of 16 vulnerabilities fixed.

About Patch Layer

Using docker history, we can find out that 4 layers were added to this container image. This is slightly different from using Copa for patching, but the overall purpose of patching is the same.

docker-history.output