Container image scanning in CI

2026-09-04 · security containers

Container image scanning is the practice of inspecting an image and its contents for known security issues before that image is promoted, signed, or deployed. In a continuous integration pipeline, the scan is less about producing a perfect security report and more about making risk visible at the point where a team can still change the artifact. A Dockerfile, base image tag, operating system package, or bundled language dependency can all introduce known vulnerabilities. If those issues are discovered only after deployment, the fix usually requires an emergency rebuild, a rushed rollout, and a difficult conversation about what is actually running in production.

The core idea is simple: treat the image as a build output with a supply chain, not as an opaque runtime unit. An image contains layers, package metadata, application files, and sometimes embedded bills of materials or signatures. Scanning tools compare what they find against vulnerability data, configuration rules, and policy. The result is only useful when it is tied to a workflow: which findings block a merge, which findings create a ticket, which findings are accepted temporarily, and which findings mean the image should be rebuilt from a newer base.

This post focuses on container image scanning in CI. It covers what scanners detect, where scans fit in a pipeline, how vulnerability databases and severity ratings affect results, how software bills of materials (SBOMs) and provenance fit alongside scanning, and why false positives and operational limits are part of normal practice rather than signs that scanning has failed.

What container image scanning actually finds

Most teams start with vulnerability scanning because it answers an urgent question: does this image contain components with known CVEs? That is important, but it is not the whole picture. Depending on the scanner and configuration, an image scan may inspect operating system packages, language-specific packages, binaries, embedded metadata, and configuration issues. Trivy, for example, describes itself as a scanner for vulnerabilities and misconfigurations across code repositories, binary artifacts, container images, and Kubernetes clusters. Its vulnerability documentation explains that it can detect issues in OS packages, language-specific packages, and certain kinds of non-packaged software.

Operating system package scanning works by identifying the distribution and package manager metadata inside the image. If the image is based on Alpine, Debian, Ubuntu, Red Hat, or another supported distribution, the scanner can look up advisories relevant to that ecosystem. This matters because the same upstream library version may be patched differently by different vendors. Trivy notes that it uses the appropriate OS vendor advisory data for packages installed through OS package managers. That design choice reduces false positives caused by version numbers alone, especially when a vendor has backported a fix without changing the upstream version string in the way a generic database might expect.

Language-specific package scanning looks at manifests and lockfiles that ecosystems such as npm, pip, Maven, RubyGems, Go modules, and others produce. In a container image, these dependencies may be present because the application was copied into the final image, because a build stage left artifacts behind, or because the runtime image includes package metadata. A scanner may report vulnerabilities in dependencies that are present in the image even if the application never calls the vulnerable code path. That does not make the finding irrelevant; it means the finding needs triage. The package is still part of the shipped artifact, and future code changes or runtime behavior may expose it.

Configuration and secret scanning are adjacent capabilities that teams often bundle under image security. A vulnerability scanner can tell you that a package has a known issue, but it cannot tell you that the container runs as root, exposes an unnecessary port, or contains a hard-coded credential unless it also performs those checks. In CI, it is useful to separate these categories in reporting. Vulnerabilities, misconfigurations, exposed secrets, and license concerns have different owners and different remediation paths. Treating every finding as the same kind of failure makes policies noisy and encourages developers to ignore the pipeline.

An important boundary is that scanners do not prove an image is secure. They can identify known issues that match available data and rules. They cannot guarantee the absence of unknown vulnerabilities, malicious behavior, logic flaws, or unsafe runtime configuration in the deployment environment. This is why image scanning belongs in a broader software supply chain practice: hardened base images, minimal packages, dependency review, SBOM generation, artifact signing, admission controls, and runtime monitoring all address different parts of the problem.

Where scanning belongs in the CI pipeline

The best time to scan is early enough that a developer can fix the issue, and late enough that the result reflects the artifact that will actually ship. Those goals can conflict, so many teams scan more than once. A useful pipeline treats scanning as a series of checks with different purposes rather than a single gate at the end.

The first scan often happens during pull request or branch builds. At this stage, the goal is fast feedback. Scanning the repository's dependency manifests can catch vulnerable libraries before an image is built. Scanning a Dockerfile or Infrastructure as Code templates can catch risky configuration before it reaches a registry. This early pass should be optimized for developer action: clear output, links to affected components, and guidance about whether the issue is in a direct dependency, a base image, or a transitive package.

The second scan happens after the image is built. This is the scan that matters most for release decisions because it inspects the final artifact. Multi-stage builds can hide or remove build-time dependencies, so a source-only scan may miss what ends up in the runtime image, or it may flag tools that were never shipped. The image scan should run against the exact tag or digest produced by the build. If the pipeline scans myapp:latest while publishing myapp@sha256:..., the report is not tied to the immutable artifact that downstream systems will pull.

A third scan can run in the container registry or on a schedule. New vulnerabilities are disclosed after images are built. A clean scan on Monday does not mean the same image is clean after a critical CVE is published on Wednesday. Registry scanning and periodic rescans help teams find vulnerable images that are still stored, still referenced by deployment manifests, or still running in clusters. This is operationally different from CI gating: CI decides whether to produce or promote a new artifact; registry and scheduled scans decide whether existing artifacts need replacement.

Placement also affects policy design. A strict "fail on any vulnerability" rule in every pull request often creates unproductive noise, especially when the vulnerability is unfixed upstream, present only in a base image the team does not control, or rated differently across data sources. A more sustainable pattern is to gate on high-confidence, fix-available findings for release branches, while still reporting lower-severity or unfixed findings for visibility. Teams can also distinguish between developer feedback and promotion controls: a pull request may warn, a staging promotion may require review, and a production promotion may block on defined thresholds.

CI systems make these patterns easy to automate, but automation should not hide the decision. A scan result is a policy input. The pipeline needs explicit rules for exit codes, severity thresholds, ignored findings, expiration dates for temporary exceptions, and ownership. Without those rules, teams either ignore the scanner or treat it as an oracle.

Vulnerability data, severity, and why results differ

Two scanners can produce different results for the same image, and that does not automatically mean one is broken. The difference often comes from data sources, package detection, version matching, severity mapping, and policy settings. Understanding these factors helps teams interpret results instead of chasing raw counts.

Vulnerability databases are not a single global list with one canonical answer. Trivy's documentation shows that OS packages are checked against vendor or distribution sources such as Alpine secdb, Debian trackers, Ubuntu CVE data, Red Hat OVAL and security data, and other ecosystem-specific feeds. Language packages use sources such as the GitHub Advisory Database, language ecosystems, and other advisory feeds. The scanner's job is to identify a component, normalize its name and version, and determine whether an advisory applies.

This is why vendor advisories matter for container images. Linux distributions often backport security fixes. A package may remain at an older upstream version number while still containing the patch that addresses a CVE. If a scanner relied only on upstream version ranges, it could report a vulnerability that the vendor has already fixed. Trivy explicitly describes this behavior: for packages installed from OS package managers, it prefers the OS vendor advisory to avoid false positives. It also provides options that change detection behavior, such as --detection-priority, which lets teams choose between more precise matching and broader coverage that may include more false positives.

Severity is another source of confusion. A CVE record, a vendor advisory, and a scanner report may label the same issue differently. Trivy documents that it prefers severity from the selected data source because vendors may evaluate impact in the context of how they package and configure software. When a source does not provide severity, Trivy can derive it from CVSS score ranges; when CVSS is unavailable, it can fall back to NVD. Teams should read severity as a prioritization hint, not as a complete risk assessment. A "critical" vulnerability in an unused package inside a non-networked build tool may be less urgent than a "medium" issue in an exposed service path, depending on the system.

Unfixed vulnerabilities also deserve careful handling. Trivy supports filtering unfixed issues with --ignore-unfixed. That option can reduce noise, but it should be used deliberately. Hiding unfixed findings can make reports easier to act on, yet it can also hide real exposure when there is no upstream patch available. A practical policy is to separate "fix now" from "mitigate or track": rebuild on a patched base image when possible, remove unnecessary packages, restrict network exposure, and document temporary risk acceptance when the artifact must ship.

The lesson for CI is to make scan configuration part of versioned engineering practice. The scanner version, database update behavior, severity threshold, ignore rules, and report format should be visible in the pipeline definition. If those settings are hidden in a UI or changed ad hoc, teams cannot explain why a build failed last week but passes today.

SBOMs, VEX, and keeping results actionable

A vulnerability scan answers "what known issues might affect this artifact right now?" An SBOM answers a different question: "what components are in this artifact?" The two are complementary. Scanning without an inventory makes recurring analysis harder. An inventory without vulnerability or exploitability context can become a static document that nobody uses during an incident.

Trivy can generate SBOMs in CycloneDX and SPDX formats, and it can also scan SBOMs as targets. This makes it possible to separate inventory generation from vulnerability analysis. A CI pipeline might generate an SBOM for every release image, store it alongside the image digest, and later rescan the SBOM when vulnerability data changes. That workflow is useful when teams need an audit trail or when rebuilding and rescanning large images repeatedly is expensive. It is not a replacement for scanning the actual image in all cases, because the SBOM is only as accurate as the generation process and the artifact it describes.

Standards matter here because SBOMs move across organizational boundaries. A development team, a platform team, a security team, and an external customer may all need to read the same inventory. CycloneDX and SPDX are common machine-readable formats for that exchange. The format choice is less important than consistency: record the exact artifact, the generation time, the tool or process used, and the relationship between the SBOM and the image digest.

Vulnerability Exploitability eXchange (VEX) adds another layer. A scanner may report a CVE because a component is present, but the product may not be affected in the way the advisory assumes. VEX is a way to communicate exploitability status, such as whether an issue is not affected, affected, fixed, or under investigation. Trivy's documentation includes VEX support in its supply chain features, which reflects a broader industry need: teams need a structured way to explain why a finding is being suppressed or why a reported CVE does not apply. Used well, VEX prevents repeated manual triage. Used poorly, it becomes a dumping ground for permanent ignores. Every exception should have an owner, a reason, and a review date.

For CI, the practical pattern is to keep machine-readable artifacts. Store the scan report, SBOM, and any attestation or VEX statement as build outputs. When a security team asks which images contain a newly disclosed library issue, the answer should come from inventory and policy records, not from memory or ad hoc shell commands on a laptop.

Signing, provenance, and trust boundaries

Scanning tells you what known issues are inside an image. It does not tell you where the image came from, whether it was modified after the build, or whether the CI system that produced it should be trusted. That is the role of signing and provenance.

Sigstore's Cosign documentation describes workflows for signing and verifying artifacts, including containers and attestations. In a CI pipeline, a common pattern is: build the image, scan it, generate an SBOM or test result, sign the image by digest, and attach attestations that describe what was produced and how. The signature helps consumers verify that the artifact came from an expected identity or key management process. Attestations can carry structured metadata such as SBOMs, provenance, or scan results.

Provenance is especially useful when teams need to answer questions after an incident. Which commit produced this image? Which workflow ran? Which builder executed it? Was the artifact promoted from staging to production, or rebuilt outside the normal path? SLSA, the Supply-chain Levels for Software Artifacts framework, provides a way to think about these integrity controls as increasing levels of assurance. It is not a scanner and does not replace vulnerability management, but it complements scanning by focusing on tamper resistance and trustworthy build information.

The trust boundary matters. If a pipeline scans an image after build but pushes an unsigned mutable tag, a later step could replace the tag contents. If a deployment system pulls by tag rather than digest, it may run an artifact that was never scanned. Good practice is to scan and sign the immutable digest, promote by digest, and verify signatures or attestations in the deployment path. CI scanning is strongest when it is connected to artifact identity.

NIST's Application Container Security Guide, SP 800-190, is broader than CI scanning, but it is a useful reference point for the underlying risk model. It explains security concerns associated with container technologies and provides recommendations for addressing them. Image vulnerabilities are one part of that picture. Registry controls, orchestration configuration, runtime isolation, and operational monitoring all affect whether a vulnerable image becomes an incident. A scan result should feed those controls rather than live in isolation.

Limitations, false positives, and operational trade-offs

Every scanner has blind spots. A team that expects perfect detection will either distrust the tool after the first noisy report or over-trust a green build. Both outcomes are dangerous.

False positives happen when a finding does not represent real risk in the scanned artifact. Common causes include package name confusion, inaccurate version mapping, backported vendor fixes, and dependencies that are present but unused. They also happen when policies treat every CVE match as equally urgent. Reducing false positives requires good data sources, careful matching, and review workflows. Trivy's preference for OS vendor advisories in OS package scanning is one example of a design choice aimed at precision. Detection-priority settings and ignore rules can tune the balance, but they should not be used to make dashboards look clean.

False negatives are harder to see. A scanner may miss an issue because the package is not supported, metadata is missing, the vulnerability has not been published in the sources the scanner uses, or the software was installed in a nonstandard way. Distroless and minimal images can reduce attack surface, but they can also change how components are detected. Binaries copied into an image without package metadata may require different analysis than packages installed through apt or apk. Teams should test scanners against representative images, including base images, application images, and hardened production images.

Performance and developer experience are operational constraints. Scanning large images on every commit can slow feedback loops, especially when databases need updates or when the registry is remote. Caching vulnerability databases, scanning by digest, and splitting fast checks from deep checks can help. The report format also matters: developers need to know the affected package, installed version, fixed version when available, path or layer context when possible, and the next action. A long table with no ownership is easy to ignore.

Another limitation is reachability. Most image vulnerability scanners report that a vulnerable component is present. They usually do not prove that the application reaches the vulnerable function with attacker-controlled input. Reachability analysis can reduce noise, but absence of reachability evidence is not proof of safety. In CI, it is reasonable to prioritize fix-available vulnerabilities in exposed services, but teams should avoid blanket assumptions that "not reachable" means "never relevant."

Policy exceptions need governance. There are legitimate reasons to allow a temporary exception: no patched version exists, the base image vendor has not released an update, or the affected component is being removed in a scheduled refactor. The exception should name the artifact or package, record why the risk is accepted, set an expiration date, and identify who approved it. Otherwise, ignore lists become permanent shadow policy.

Remediation workflows that teams can sustain

The value of scanning appears when findings lead to changes. A report by itself does not reduce risk. The workflow after detection determines whether CI scanning is a control or a ritual.

Start with ownership. Operating system package findings in a base image may belong to the platform or container team. Application dependency findings may belong to the product team. Misconfiguration findings may belong to whoever owns the Dockerfile, Helm chart, or deployment manifest. If ownership is unclear, findings bounce between teams and age out. A good CI report routes the issue to the place where it can be fixed.

Remediation usually follows a few paths. The simplest is to rebuild with updated packages. For OS packages, that may mean updating the base image tag to a patched release and rebuilding the application image. For language dependencies, it may mean updating a lockfile and verifying compatibility. For both, the CI pipeline should produce a new immutable image and a new scan result. "Fixed in source" is not enough until the shipped artifact changes.

When a fix is not available, mitigation may involve removing unnecessary packages, switching to a smaller base image, disabling an unused feature, or adding runtime controls. Removing a package is often better than suppressing a finding, because it reduces both exposure and future scan noise. This is one reason minimal images are attractive: fewer components mean fewer advisories to evaluate. The trade-off is that minimal images can complicate debugging and package detection, so teams should choose them deliberately.

Base image strategy deserves special attention. Many image vulnerabilities come from the base layer rather than the application. Teams can pin base images by digest for reproducibility, but they also need a process to refresh those digests regularly. A pinned digest that never changes will eventually accumulate known vulnerabilities. Automation can help by opening pull requests when a base image updates, running scans on the rebuilt image, and requiring review before promotion.

Finally, measure the workflow. Useful metrics include time from finding to fix, number of fix-available critical findings in release images, age of open vulnerability exceptions, and whether production deployments reference scanned digests. Counts of total vulnerabilities are less useful because they move with database updates and scanner behavior. The goal is not a flat dashboard; it is a predictable path from detection to rebuilt artifacts.

Conclusion

Container image scanning in CI works best when it is treated as an engineering control, not a checkbox. The scan should inspect the artifact that will ship, report findings in categories that developers can act on, and connect to decisions about promotion, rebuilds, and exceptions. Vulnerability databases, vendor advisories, SBOMs, signatures, and provenance each answer different questions. No single report can prove that an image is safe, but a disciplined pipeline can make known risks visible, make artifact origin verifiable, and make remediation routine.

Teams adopting image scanning should start with a narrow, reliable loop: build an image, scan the image by digest, fail or warn according to explicit policy, store the report, and rebuild when fixes are available. From there, add scheduled rescans for existing images, SBOM generation, and signing or attestations so that the CI result follows the artifact into registries and deployment systems. The long-term payoff is not fewer alerts for their own sake; it is faster, clearer decisions when the next vulnerability advisory arrives.

Sources

← back to posts