Skip navigation
All Places > Alfresco Content Services (ECM) > Blog > Author: toni.delafuente

This blog post was originally posted here.

Dec 1st 2015: first version of this article published

 

Dec 2nd 2015: UPDATED OpenSCAP section with Atomic scan information and references

 

Dec 7th 2015: UPDATED Twistlock section, after a session/demo with the vendor. Conclusions updated.

 

Let’s suppose you are working in Security. Now, your company decides to run some applications in containers, they choose Docker, after some weeks or months testing it they want to go live, and suddenly someone says 'should we do a security audit before going to production?”, the rest of the story is you and an audit to a Docker environment.

 

You can use all your existing arsenal and procedures your are familiar to audit the application running in the containers (file permissions, logs, etc.) but what about the containers, images, dockerfiles, docker servers or even the clustering and orchestration platform? This article is about that.


Considerations for this particular audit:

  1. Check if images and packages inside images are up-to-date and are free of security vulnerabilities.
  2. Audit automatization, we must be able to automatize all checks. That will save us a precious time and we can run it as often as we require, forget about to do it manually unless you are just testing or learning.
  3. Container links and volumes. If you use read-only filesystem in your running container “docker diff” can help you to find issues.
  4. The bigger an image is the harder the audit will be, reduce as much as you can the size of your images.
  5. The host kernel is the shared point between all containers in the same server, keep that kernel up-to-date.

 

Once said that, I want to give you an overview of the existing tools I have found to achieve your duty mentioned above. I have probably missed other tools, if so, please point me to them in the comments.

 

    1. Docker Bench for Security:

 

      • Description: The Docker Bench for Security is a script that checks for dozens of common best-practices around deploying Docker containers in production. Those checks are based on all recommendations taken from the CIS Docker 1.6 Benchmark document.

 

      • Focus: mostly Docker server and few tips for images and containers.

 

      • Language: Shell script

 

      • Methodology: Run the script in the same server where Docker is running or from a container. It will create a shell report with INFO, WARN or PASS alerts.

 

      • License: Apache 2.0

 

      • Installation/usability level: Easy

 

 

      • More about audit and vulnerabilities assessment from Docker Inc:

        • Project Nautilus: presented during Docker CON 2015 in Barcelona: https://www.youtube.com/watch?v=fLfFFtOHRZQ& Project Nautilus, the new image scanning and vulnerability detection service for official repos on Docker Hub. As in @diogomonica words 'Nautilus is already working on the background on all the official images'. Nautilus looks for any suspicious piece of software. Is not depending on public vulnerabilities data bases nor based on Linux distros, instead, it looks for vulnerabilities using their own data base. We will have more information soon and probably a closer look by Q1 2016. (Thanks Diogo for the info).

 

      • My comments: From the Docker server/daemon configuration point of view this is the best tool you can use to make sure you are in the right path. Definitely I would use this tool but in conjunction with others, keep reading.

 

2. OpenSCAP Container Compliance:

      • Description: Based on the same philosophy as its parent project OpenSCAP that supports CVE scan, multiple report formats and custom policies. Specific instructions and packages for RedHat 7 are here. Note: SCAP is U.S. standard maintained by National Institute of Standards and Technology (NIST). The OpenSCAP project is an open source collection of tools for implementing and enforcing this standard.

 

      • Focus: Images and Containers

 

      • Language: Shell script

 

      • Methodology: run the oscap-docker command against an image or container and get the results on a very helpful and descriptive html report.

 

      • License: GPL v3

 

      • Installation/usability level: Easy

 

      • Demo/Presentation: N/A

 

      • My comments: If you use RedHat/Fedora/CentOS based containers this is highly recommended for you.

 

      • UPDATE (Dec 2nd 2015): If you use Atomic they have recently released a new feature that allows you to scan containers for vulnerabilities using OpenSCAP, see this blog post here and code here.

 

3. CoreOS Clair:

 

      • Description: Clair is a container vulnerability analysis service. It works as an API that analyzes every container layer to find known vulnerabilities using existing package managers such as Debian (dpkg), Ubuntu (dpkg), CentOS (rpm). It also can be used from the command line as showed here. It provides a list of vulnerabilities that threaten a container, and can notify users when new vulnerabilities that affect existing containers become known. It is being used by https://quay.io

 

      • Focus: Images and Containers

 

      • Language: Go

 

      • Methodology: Used via API or command line it extract all layers of the image, notifies if vulnerabilities are found whenever they found it because it stores all the information in a data base, it also manages its own vulnerability database updates from known vulnerability sources.

 

      • License: Apache v2

 

      • Installation/usability level: Hard

 

 

      • My comments: I couldn’t make it work in CentOS 7.1. I will add more info here as soon as I got something new.

4. Banyan Collector

 

      • Focus: Images

 

      • Language: Go

 

      • Methodology: Even though it can run in a container, banyan collector can run form command line and connect to a given Docker registry to perform its analysis. See how it works in detail here.

 

      • License: Apache 2.0

 

      • Installation/usability level: Medium-Hard

 

      • Demo/Presentation: N/A

 

      • My comments: It is very oriented to check registries more than a pure vulnerability assessment tool.

 

6. Lynis:

      • Description: Lynis is a Linux, Mac and Unix security auditing and system hardening tool that includes a module to audit Dockerfiles. It also shows some Docker server statistics and check permissions.

 

      • Focus: Dockerfile

 

      • Language: Shell script

 

      • Methodology: just run Lynis with the proper options and Dockerfile path and Lynis will take a look to the files installed and some other parameters inside the file.

 

      • License: GPL v3

 

      • Installation/usability level: N

 

      • Demo/Presentation:

 

      • My comments: You can hit two birds with one stone but not really useful for docker audit yet. I know the author is willing to add more support to Docker.

 

7. Twistlock:

 

      • Description: As in the author words: Twistlock scans container images in registries, on developer workstations, or on production servers. We detect and report vulnerabilities in the Linux distribution layer, app frameworks, and even your customer app packages. In addition to the Open Source threat feeds it uses commercial threat feeds. Their solution also offers access control to actions based in users and groups and a very interesting Runtime defense that allows to monitor and act upon security based in roles, behaviors, compliance, malicious actions and more.

 

      • Focus: images, containers, packages. Made for Docker and Kubernetes or Mesos.

 

      • Language: Shell script, Javascript and Go.

 

      • Methodology: it uses NIST to find CVEs and the Docker CIS for vulnerability assessment. It does more than just that, features like advanced access control, runtime defense, monitoring and continuous integration. A container called defender has to run in every host and a central console collect and manages all of them from a central location.

 

      • License: commercial depending on number of hosts. Free Developer Edition up to 2 hosts without support.

 

      • Installation/usability level: Not tested, I have seen a live presentation and demo run by de vendor.

 

 

      • My comments: Nothing much to say since I could’t play with it or see it in action. I will add more info once I have something else. I have have a meeting with the vendor and have a better view about what the product is, and it is the most complete solution I have seen so far. They cover enterprise grade security, they are starting and is a brand new product with just a few customer, the product has a big room to improve and add new features but it is covering in a smart way most of the requirements at this moment and with enough granularity that allow us to improve Docker security. Finally it is important to highlight that it is not just an auditing tool, it is a managed security tool for Docker.

 

    1. Bitnami Stacksmith:

 

      • Description: it is a tool to quickly generate custom Dockerfiles (as per Bitnami words: a declarative API to create containers), is not intended to be a security tool but it has that cool feature that helps you to detect outdated and vulnerable components while building your Dockerfiles or even in existing containers built in Stacksmith. It sends you an email when a compoenent has to be updated.

 

      • Focus: Dockerfiles, images and containers.

 

      • Language: unknown

 

      • Methodology: it uses an external public CVE scores https://cve.mitre.org DB to find CVEs of the given components for vulnerability assessment.

 

      • License: SaaS

 

      • Installation/usability level: Easy

 

      • Demo/Presentation: https://www.youtube.com/watch?v=4A24pD-P_N4
      • My comments: As SaaS it seems to be a very easy tool, from the security point of view it gives the user a clear view of the status of the container components which is very helpful to figure out if we have vulnerable or outdated containers.

 

8. Dockscan

 

      • Description: a brand new tool, in a very early stage, released 2 weeks ago, it was presented at BlackHat Europe Arsenal. As per the author: Dockscan is a vulnerability assessment and audit tool for Docker and container installations. It will report on Docker installation security issues as well as Docker container configurations. The tool helps both system administrator administering Docker to help them secure Docker, as well as security auditors and penetration testers who need to audit Docker installation.

 

      • Focus: Docker server

 

      • Language: Ruby

 

      • Methodology: it uses some the existing CIS Docker 1.6 Benchmark best practices. Can work in local and remote Docker installations.

 

      • License: GPL v2

 

      • Installation/usability level: easy

 

      • Demo/Presentation: N/A

 

      • My comments: It has a very short list of features yet but looks interesting, I would keep an eye on it but not to be used as a mature tool by now.

 

9. Drydock:  (do not confuse it with Dry-dock cluster)

 

      • Description: As per the author: drydock is a Docker security audit tool written in Python. It was initially inspired by Docker Bench for Security but aims to provide a more flexible way for assessing Docker installations and deployments. drydock allows easy creation and use of custom audit profiles in order to eliminate noise and false alarms. Reports are saved in JSON format for easier parsing. drydock makes heavy use of docker-py client API to communicate with Docker. It is based on CIS Docker 1.6 Benchmark.

 

      • Focus: Docker server and containers

 

      • Language: Python

 

      • Methodology: it uses some the existing CIS Docker 1.6 Benchmark best practices to check server configuration options.

 

      • License: GPL v2

 

      • Installation/usability level: Easy

 

      • Demo/Presentation: N/A

 

      • My comments: It is in a very early stage of development yet, seems to be ahead of Dockscan. Let’s see what’s next with this tool. Not mature enough to consider as a player.

 

10. Batten:

 

      • Description: Hardening and auditing tool for docker hosts and containers. It is pretty much the same as Drydock or Docker Bench for Security.

 

      • Focus: Docker server and containers

 

      • Language: Go

 

      • Methodology: run as container and check the server and containers following the  CIS Docker 1.6 Benchmark.

 

      • License: MIT

 

      • Installation/usability level: Easy

 

      • Demo/Presentation: N/A

 

      • My comments: Nothing different to what Drydock or Docker Bench for Security does.
Conclusion:

 

    • What of these tools would you use? Considering the early stage of most of them, I would use Docker Bench for Security, OpenSCAP and probably Bitnami Stacksmith. And I would keep and eye on the others. UPDATE: After the meeting I have had with @mwithrow, Director of Architecture at Twistlock, and see the product details I think that it is the most complete solution so far. See Twistlock section for details.

 

    • Most of these tools are very new with months or weeks since they were released, there is a big room to improve them and adapt them to a more enterprise scale security. It is a good starting point to address audit and vulnerabilities assessment of our container ecosystem regardless it is a production, test or development environment. Looking forward to see what have to say the big security vendors about it.

 

    • In favor of all of them, I have to say that is hard to keep them updated since Docker is growing really quick and releasing versions with a bunch of new features (including security improvements) almost every week. So it is a tough race just try to keep any of these tools up to date.  I guess is the price to pay working with emerging technology.

 

    • In other post I would like to discuss more in detail about security in orchestration and how to achieve a proper audit on solutions like Kubernetes.

 

    • What about incident response? That’s another good point to cover in a blog post.

 

    • There are more coming, I’m looking forward to see what the big fishes have to say about it (Google, MS, AWS, etc.).

Post originally posted in my personal blog here.

 

 

Well, they are 10 ideas or commands actually

 

Due to my new role at Alfresco as Senior DevOps Security Architect, I’m doing some new cool stuff (that I will be publishing here soon) and also learning a lot and helping a little bit with my knowledge on security to the DevOps team.

 

One of the goals I promised myself was to “never disable SELinux”, even if that means to learn more about it and spend time on it. I may say that it’s being a worth it investment of my time and here you go some results of it.

 

This article is not about what is or what is not SELinux, you have the Wikipedia for that. But a brief description could be: a MAC (Mandatory Access Control) implementation in Linux that prevents a process to access to other processes or files that is supposed to not to have access (open, read, write files, etc.

 

If you are here is because you want to finally start using SELinux and you are really interested on make it work, to tame this wild horse. Let me just say something, if you are really worry about security and have dozens of Linux servers in production, keep SELinux enabled, keep it “Enforcing”, no question.

 

Once said that, here is my list. It is not an exhaustive list, I’m looking forward to see your insights in the comments:

1. Enable SELinux in Enforcing mode.

 

  • In configuration files (need restart)

    • /etc/sysconfig/selinux (RedHat/CentOS 6.7 and older

    • /etc/selinux/config (RedHat/CentOS 7.0 and newer)
      • Through commands (no restart required)

        • setenforce Enforcing

        • To check the status use

          • sestatus # or command getenforce

         

        2. Use the right tools. To do cool things you need cool tools, we will need some of them:

         

          • yum install -y setools-console policycoreutils-python setroubleshoot-server

         

          • policycoreutils-python comes with the great semanage command, the lord of the SELinux commands

         

          • setools-console comes with seinfosesearch and sechecker among others

         

          • from setroubleshoot-server package we will use sealert to easily identify issues

         

        3. Get to know what is going on: Dealing with SELinux happens mostly during installation, configuration and tests of Linux services. Therefore, in case something in your system is not working properly or in the same manner as with SELinux disabled. When you are configuring and installing a service or application on a server and something is not working as expected, not starting as it should to, you always think “Damn SELinux, let’s disable it”. Forget about that, you have to check the proper place to see what is going on with it: the Audit logs. Check /var/log/audit/audit.log and look for lines with “denied”.

         

          • tail -f /var/log/audit/audit.log | perl -pe ‘s/(\d+)/localtime($1)/e’

         

          • the perl command is to convert the Epoch time (or UNIX or POSIX time) inside the audit.log file to human readable time.

         

        4. See the extended attributes in the file system that SELinux use:

         

        • ls -ltraZ # most important here is the Z
        • ls -ltraZ /etc/nginx/nginx.conf will show:
            • -rw-r–r–. root root system_u:object_r:httpd_config_t:s0 /etc/nginx/nginx.confls -ltraZ /etc/nginx/nginx.conf will show:
            • where system_u: is the user (not always a user of the system), object_r: role and  httpd_config_t: is the object type, other objects can be a directory, a port or socket and types of an object can be a config file, log file, etc.; finally s0 means the level or category of that object.

         

        5. See the SELinux attributes that applies to a running process:

         

        • ps auxZ
          • You need to know this command in case of issues.

         

        6. Who am I for SELinux:

         

        • id -Z
          • You need to know this command in case of issues.

         

         

        7. Check, enable or disable defined modes (enforcing or permissive) per deamon:

         

        • getsebool -a # list all current status
        • setsebool -P docker_connect_any 1 # allow Docker to connect to all TCP ports
        • semanage boolean -l # is another alternative command
        • semanage fcontext -l # to see al contexts where SELinux applies

         

        8. Add a non default directory or file to be used by a given daemon:

         

         

        • For a folder used by a service, i.e.: change Mysql data directory:
          • Change your default data directory in /etc/my.cnf
          • semanage fcontext -a -t mysqld_db_t “/var/lib/mysql-default/(/.*)?”
          • restorecon -Rv /var/lib/mysql-default
          • ls -lZ /var/lib/mysql-default

         

        • For a new file used by a service, i.e.: a new index.html file for Apache:
            • semanage fcontext -a -t httpd_sys_content_t ‘/myweb/web1/html/index.html’
            • restorecon -v ‘/myweb/web1/html/index.html’

           

          9. Add a non default port to be used by a given service:

           

          • i.e.: If you want nginx to listen in other additional port:

            • semanage port -a -t http_port_t -p tcp 2100
            • semanage port -l | grep  http_port # check if the change is effective

           

           

           

          10. Spread the word!

           

          • SELinux is not easy but writing easy tips make people using it and making the Internet a safer place!

          Filter Blog

          By date: By tag: