Container Vulnerability Scanning on AWS with Clair and AWS Fargate


Security and compliance of container images is a trending hot topic. The challenge discussed in this article is how to integrate a container vulnerability scanning solution within a CI/CD process. Ideally, the outcome of such a pipeline would be a secure golden image that meets the security and compliance requirements of your company and at the same time contains all the software packages that the application teams need.

  • We took a look at several options on the market and picked Clair for the following reasons:
  • Clair was developed by the CoreOS team, now part of RedHat, thus backed by a major open source contributor and vendor.
  • Clair has been out there for a while and there is a rich eco-system of tools that use the Clair API. It is quite easy to be integrated into a CI/CD process.

Solution Overview

Clair Container Scanner
Figure 1 Clair Container Scanner

The solution is shown in Figure 1. It is worth mentioning that Clair follows a traditional client-server architecture, where the Clair server is an API hosted on a container and the clients are third party tools that send commands to the API. The solution contains the following key components: [NB1]

  • Clair container – this is a container that hosts the scanning API. Clients need to make API requests to initiate any action (including image scanning). In our implementation, the Clair container has been slightly modified and hosted on AWS Fargate. It is a separate task in an autoscaling group. Details about the implementation are given below.
  • PostgreSQLDB[NB2] – this is a vulnerability database that is being regularly updated with vulnerabilities for both OS and non-OS components of the container.
  • Clients – these are tools that implement simple CLI commands to use the Clair functionality. The CLI commands trigger API requests to the Clair API. While there is a variety of options for client solutions – Clair Klar has been chosen for its simplicity and functionality. In the post, we execute Klar CLI commands from a machine in the VPC. However, it is possible to install the tool as part of the CI/CD process – as an AWS CodeBuild project for example.
  • Application [DR3] Load Balancer – an ELB is used to present a consistent DNS record in front of the Clair container hosted on AWS Fargate, it is also easy to integrate the ELB with AWS Certificate Manager, and apply a free SSL certificate to make encrypted calls to the service.

Solution Overview

1. Create an RDS database

Clair DB deployment
Figure 2 Clair DB deployment

Clair requires a functioning database instance. While the example presented in the official page creates a PostgreSQL container, it is generally recommended to have a separate PostgreSQL instance installed. In this example, we make use of a managed RDS instance.
In our experience, a small t2 instance with 10 GB of storage is sufficient for smooth operation. The database should be however monitored.
lair DB Settings
Figure 3 Clair DB Settings

2. Prepare the container image

The Clair container image is public. We recommend pulling the latest STABLE release.
NOTE: If you specify “image:latest” there is no guarantee that the release will be stable. The exact release must be explicitly specified – in our case this is v2.0.7. In the example, we use an ec2 instance (a medium t2 as some RAM is needed to test) with IAM permissions to upload an image to an ECR.
So, here’s what should be done:

2.1. Install Clair

First, let’s install docker simply by issuing the following command:
$ yum install docker
Next, we download the Clair container image. At the time of writing this article, the latest stable release is v2.0.7. Note that if you download the latest development release, the tools might not work.
$docker pull
$yum install python-pip
In order to build an image that will work on AWS Fargate, we should download and modify the docker compose file:
$ curl -L -o $HOME/docker-compose.yml
$ mkdir $HOME/clair_config
Then we download the configuration file:
curl -L -o $HOME/clair_config/config.yaml
Edit the configuration file:
$nano $HOME/clair_config/config.yaml
The important part is to update the file with the created PostgreSQL DB string. In our case this would be:
# PostgreSQLSQL Connection string
source: PostgreSQLsql://clairdb:Password123@clair_db_uri/clairdb     #clairdb:Password123 – this is username and password of the     PostgreSQL database
#clairdb after the ‘/’ is the name of the database schema.
NOTE: In this example for simplicity, we are hardcoding the password in the configuration file which is generally a bad practice. To solve this, when implementing in a production environment, the sensitive information can be retrieved from the SSM Parameter Store and passed to the container in Fargate as an environment variable.

Next, we need to install docker-compose through the following PIP command:
$ pip install docker-compose
Before running docker-compose, we should modify the docker-compose file we downloaded in the follow manner:
##########DOCKER COMPOSE############
version: ‘2’
container_name: clair_clair
restart: unless-stopped
– “6060-6061:6060-6061”
– /tmp:/tmp
– ./clair_config:/config
command: [-config, /config/config.yaml]
Finally, we launch docker compose:
$ docker-compose -f $HOME/docker-compose.yml up -d
…and check whether the container is running:
$ docker container ls
CONTAINER ID        IMAGE                         COMMAND                  CREATED             STATUS              PORTS                              NAMES
42616dda4906   “/clair -config /con…”   23 hours ago        Up 6 seconds>6060-6061/tcp   clair_clair

2.2. Install and test Clair Klar

$ find -name “*klar*” ====> ./go/bin/klar
$ export PATH=$PATH:/root/go/bin/
$ export CLAIR_ADDR=localhost
$ export CLAIR_OUTPUT=Low
Now, we can test if the Clair API works:
$ klar PostgreSQLs:9.5.1
NOTE: In this case we pull an image from In reality, any container repository can be used such as Amazon ECR.
CVE-2018-7169: [Medium]
Found in: shadow [1:4.2-3+deb8u1]
Fixed By:
An issue was discovered in shadow 4.5. newgidmap (in shadow-utils) is setuid and allows an unprivileged user to be placed in a user namespace where setgroups(2) is permitted. This allows an attacker to remove themselves from a supplementary group, which may allow access to certain filesystem paths if the administrator has used “group blacklisting” (e.g., chmod g-rwx) to restrict access to paths. This flaw effectively reverts a security feature in the kernel (in particular, the /proc/self/setgroups knob) to prevent this sort of privilege escalation.
CVE-2016-9841: [High]
Found in: zlib [1:1.2.8.dfsg-2]
Fixed By:
inffast.c in zlib 1.2.8 might allow context-dependent attackers to have unspecified impact by leveraging improper pointer arithmetic.

NOTE: The severity of the Common Vulnerabilities and Exposures

(CVEs) that Clair reports are configurable. A user can also define passing criteria and whitelists. More information about the configuration options can be found here.

2.3. Prepare the image for AWS Fargate

First, we create a new image based on the running container:

CONTAINER ID        IMAGE                         COMMAND                  CREATED             STATUS              PORTS                              NAMES
42616dda4906   “/clair -config /con…”   4 minutes ago       Up 4 minutes>6060-6061/tcp   clair_clair
$ docker commit 42616dda4906 fargate_clair_image:latest
Then we can verify whether the image is there:
$ docker image ls
REPOSITORY                 TAG                 IMAGE ID            CREATED             SIZE
fargate_clair_image        latest              67c1aac60aeb        4 seconds ago       353MB
klar                       latest              abb9667bcc83        4 days ago          353MB   latest              be456957ff54        6 days ago          52.6MB       v2.0.7              7968a46189ae        2 months ago        353MB

In order to prepare the image to run in a standalone manner on Fargate, we need to create a new image that contains the configuration file. This can be achieved in many ways. For example:

$ cd ~/clair_config
$ nano Dockerfile
FROM fargate_clair_image:latest
COPY config.yaml /config/config.yaml

Finally, we can build the image that we are going to use to launch the Fargate container service:

$ docker build -t fargate_clair:latest.

3. Create an ECR repo and upload the prepared working Clair image

Now, we are ready to create the repo:

Figure 4 ECR Repository

And … upload the image we prepared in the previous step:
$ (aws ecr get-login –no-include-email –region eu-west-1)
$ docker tag fargate_clair:latest
$ docker push

4. Create a load balancer

Figure 5 Application Load Balancer

5. Create the AWS Fargate setup

We will not go through all the details. The values given below have been tested to be working smoothly:
Prepare the Fargate task referencing the ECR URL. In our case this is: Units: 1024 (or 1 vCPU)Memory: 2048 MBTask Execution Role: ecsTaskExecutionRole which is an AWS managed role

6. Create the Fargate cluster and service

Security group – must allow all the traffic out (including non-TCP). Must allow the traffic from the LB listener IP range in. Autoscaling – normally auto-healing configuration would be enough to create a service that is reliably utilized by a CI/CD process. However, if there is a need for parallel scanning of images, then the maximum number of containers should be increased.ECS Autoscaling Service metric – CPUNetworking – in our specific setup the containers are hosted in a private security zone that is not accessible from the internet. The load balancer has a listener in the DMZ on port 6060 forwarding to port 6060 in the backend. However, the health check in the target group is on another port (by default 6061).

7. Verify that service works

7.1. Check whether the task is running

Normally, the task should be in “RUNNING” status.

Figure 6 Fargate Running Container

Indications that something is wrong include:
Service is stuck in “ACTIVATING” state. This usually means that the container is being constantly restarted. In this case if would be helpful to take a look at the CloudWatch log corresponding to the respective task. If there is no log available, this typically means that the task is unable to pull the image which is an indication that the issue is related to networking or authentication with the repository. The CloudWatch logs of the task should indicate that the container successfully connects to and updates the vulnerability database which in our case is an RDS instance. If there are any sort of errors there, it is to be expected that the scanning of images would not be successful. By default, the vulnerability database is updated every 6 hours. However, this option is configurable by the user.

7.2. Verify whether the Klar tool is still able to scan using the Fargate Clair container

First, define the environment variable that identifies the Clair API:
$ export CLAIR_ADDR=http://load_balancer_uri:6060
Second, test Klar:
$ klar PostgreSQLs:9.5.1
The result should be the same as the result we got in section 2.2.

Final Thoughts

In conclusion, Clair has been identified as a suitable open source solution that can be easily integrated within a CI/CD pipeline. The article exemplifies a simple implementation that helps to address the container security problem in an automated manner. However, there are many options and solution customizations that are outside the scope of this article. Please get in touch if we can help with AWS solution designs or any other aspect of the AWS platform.