Locking down containers with Docker’s secure blocks

Containers have grown very popular for their flexibility for app development and hosting. For containers to be deployed effectively, they require a substrate to provision and manage resources, place workloads, and adapt to failures. Container orchestration tools like Docker Swarm, Kubernetes, and Marathon simplify the management of container workloads. Unfortunately, many of these systems have not been architected with security in mind and with container orchestrators, compromise of a less-privileged node can allow an attacker to gain control of the whole system or other private resources. We spoke with Diogo Mónica, Docker’s Security Lead, about how the company has been working on secure blocks to allow you to run a “least privilege” infrastructure where any participant only has access to the resources strictly necessary.

What is the thinking behind container substrates and did (security) history repeat itself when they were designed?

The shift to containers and microservices is effectively changing the way people deploy apps, and we have an opportunity to do it right this time. We want to make sure our platforms are safer, and to do that we needed the underlying substrate to be secure.

Not thinking about security by design is still a problem in the industry, but now a lot more development teams are focusing on it. If security does not come “on” by default people will not use it, so that is a big focus for us. For the past 25 years the industry has been looking at security as an afterthought, but due to high-profile attacks, cyber security is a front-and-centre issue right now.

Are other systems management tools a more secure alternative for container management?

We created SwarmKit and embedded into Docker for more secure container management. We did this as we were not satisfied with the security of the orchestration tools out here. SwarmKit takes the approach of least privilege whereas other orchestrators trust all the nodes in the cluster: if one node is compromised, the whole cluster is compromised. We have been helping others and participating in special interest groups to improve container management security and if you are building an orchestration tool, you should be thinking about security, and least-privilege from the beginning.

Tell us about Docker’s secure blocks project and what it is achieving?

The effort to design secure building blocks is part of the Moby Project: a Docker-based, but not Docker controlled community that works on all things container. Right now we have seven main pieces of software that are necessary to create a secure platform. If we bring them all together, you get a platform that provides a high-level of security guarantees: Docker EE. We are also seeing other orchestration tools take some of our ideas and use them for their projects, which is a great validation of the team’s work.

What needs to change when thinking about security architectures for containers and the cloud in general?

Two main things need to change. First, have a least-privilege focus when designing any system. There is much trust placed on zero trust networks, but we need to think about systems in a way where every component can be compromised. This applies to everything, including apps and the container orchestration that manages the whole cluster.

Secondly, instead of just thinking about the security of the runtime environment, you have to care about the whole software development lifecycle. Ensuring that the CI system does image scanning means that no known vulnerabilities will get deployed to production. Having the CI system digitally sign all the packages that are built means that no malicious attacker has injected code in the artifacts being deployed. These are both critical requirements to run a secure software development lifecycle.

How can cloud providers help with delivering more secure virtual server and container services?

Cloud providers are doing mostly the right thing by implementing centralized IAM and providing recommendations around container configurations. What they have not been doing is helping people move to secure, minimal operating systems for containers. The reality is that no one should be running general purpose operating systems on a cloud if you are only running containers. In a general OS you can write to disk and modify the state of the OS, and that should not be allowed in any production environment: everything should be immutable. Not running immutable environments allows drift in server configurations, which is one of the biggest causes of downtime. Immutability is also a major win for security, making persistence a lot harder for any attacker.

A trend in security right now is to be more data-driven and pre-emptive. How can this approach be applied to container security?

There are a few companies are up and coming in this space and are doing a great job at turning the data that containers generate into useful insight. One of these companies is StackRox right here in the US. Containers can finally generate data as close to the application as possible. For example, if you are building an IDS, you finally have data from each application, instead of data from the VM, the host or the network. This allows security products to make better decisions, since they are now monitoring every system call, and can see malicious or non-malicious behaviour at the container level. We are approaching a point where IPS systems might finally work!

What else is on the roadmap at Docker? What can we look forward to?

I am particularly excited about service identities. The idea here is to allow any application running inside of Docker to have their own x509 Certificate, so that they can use it to do mutual TLS, allowing Confidentiality Integrity and Authenticity of every single API call. This has been on the best practice list for years, but it has always been extremely difficult to deploy in practice. Creating a PKI (public key infrastructure) and managing the certificate lifecycle is an absolute nightmare and requires special expertise. It is also very prone to failure. What we are doing is removing all this extra complexity that is needed to successfully use mutual TLS between applications in your infrastructure. The only thing developers and operations people will need to be able to use mutual TLS in applications is to load the correct certificate. We are also collaborating with Spiffe.io to ensure platform independent naming for services and better interoperation with other orchestration platforms.

Overall, we realized that security is almost always bolted on, never easy to use, and never turned on by default, so, at Docker, we are trying to change this.