Having participated on and off with Kubernetes related projects since 2015, I was excited to finally go to KubeCon (officially KubeCon + CloudNativeCon) for the first time this year. Between the last KubeCon North America in Austin, Texas and this year’s in Seattle, Washington the number of attendees has doubled, reaching 8000 people, and is a testament to the adoption of Kubernetes and the Cloud Native Computing Foundation (CNCF) ecosystem.  

As the popularity of Kubernetes grows, so does the concern around the security of Kubernetes. Particularly after the discovery of a critical vulnerability in early December, which merits a score of 9.8 out of 10 in severity, there are more questions than ever on the safety and security of the container orchestration platform. Even without the arguably well timed CVE, the popularity of the container orchestration platform makes it an increasingly attractive target for malicious attacks.

My primary goal in going to KubeCon this year was to learn about what others are doing about security and to find some Kubernetes security best practices. Kubernetes is an evolving platform, and KubeCon is the most comprehensive congregation of users and experts with whom the discussion and learning on the subject can take place.

While there are many different tools dealing with Kubernetes security (not just containers, but microservices and serverless functions in general), here are a few highlighted principles that are applicable regardless of the tools you use.

Related: Interested in understanding the containers ecosystem on AWS? Download our free whitepaper.

Container Images = The Perfect Trojan Horses ( Do you know what it contains? )

Docker is currently the most common container runtime supported in Kubernetes and other orchestration platforms. One of the advantages of Docker is reusability i.e. that each image embeds within it the binary dependencies, such that the application can be built on top of those dependencies than having to build the image and dependencies from scratch. It is tempting to find a public image with Apache HTTP and place only the distinct application onto it. However, how do you know if the said Apache HTTP image does only what it advertises and has no other embedded instructions, or worse yet, whether it contains instructions that are concealed for malicious intent?

It’s important to scan through the images to ensure you know the provenance of the dependencies. Also, as the database of vulnerability keeps growing because vulnerabilities are constantly discovered by researchers, it is important to scan older images too.

There are multiple commercial and open-sourced products that support image scanning, such as Clair from CoreOS, which is used to scan quay.io too.

Role-Based Access Control (RBAC) is Good for the Soul

Role-based access control (RBAC) has been made available in Kubernetes since version 1.5, and ever since it has been part of Kubernetes security best practices as it is considerably superior to the previous standard attribute-based access control (ABAC). RBAC is somewhat tricky to set up. Namespace-specific or cluster-wide permissions (verbs) have to be deliberately associated with resources before users, groups and service accounts can be “bound” to the permissions. However, it has proven itself to be indispensable, particularly when RBAC supports permission limitations for unauthenticated users too (such as for service health checks). By implementing RBAC, the principle of minimal access can be thoroughly enforced, to limit the permissions only to specific actions in particular namespaces. That said, it is implied and advised that assignment of highly powerful roles such as cluster-admin undergo heavy scrutiny and are only applicable to very specific set of users, if at all.  

`curl | sudo bash` can be bad, so is the K8s equivalent of `kubectl apply -f`

One of the main drivers of the popularity of Kubernetes is its community support. There are many sample code and packaged solutions that an end user can leverage to install the different add-ons, dependencies, and common packages. However, while it is tempting to just look up solutions available on the internet (a good concentration thereof on github.com) and execute as is, it is important to first look into a what the solution consists of. Because of the declarative nature of Kubernetes instructions, granting and changing permissions for resources is a relatively straightforward exercise. While we all like a solution that takes care of any undifferentiated heavy lifting, we also have to make sure we understand fully what the solutions attempt to do. Any instructions we execute — especially those provided by third-parties— should be first validated to ensure it complies with the security policies of your clusters and your organization.

Admission Controls — contextual dynamic defense

Understanding the image packages, applying RBAC to limit access for service accounts and users, and scrutinizing the sample code and third-party resources before execution are all pre-emptive measures and they are all sound and valid.  However, the above enforces security on different territories of the Kubernetes workflow (image, executor permissions and execution intent respectively), other controls — such as ensuring a user actually have the permission to add a new volume as required by the deployment— need to be validated using other security measures. In these instances, the Kubernetes admission control is the perfect tool.

Per the Kubernetes docs, an admission controller is “a piece of code that intercepts requests to the Kubernetes API server prior to persistence of the object, but after the request is authenticated and authorized.” There are many admission controller plugins already shipped with the Kubernetes API server, which can be activated readily by cluster administrators. Controls such as “AlwaysPullImages” and “DenyEscalatingExec” are useful to create a cluster-wide security baseline while giving liberty to individual deployment owners within the boundary to implement based on needs.

There are primarily two types of admission — “validating” and “mutating.” Istio, a project that has been popularly known in the community as a service mesh, implements both. The validation admission control is used to make sure that the communication between two services are allowed based on custom admission policies, which are set up previously, and real-time telemetry data, while mutating admission control is used to inject the proxy sidecar (Envoy) into a pod.

Sandboxing — Prevention is even more important for containers

Sandboxing is a popular technique in computer security to isolate processes and avoid vulnerabilities from spreading, so that verification can be done and threat models can be built. Since container technology takes advantage of the shared kernel for speed and reusability, it is even more important to have a mechanism that proactively identifies risks rather than leveraging only reactive measures.

There are a number of tools and projects that support sandboxing for the containers ecosystem, including Kata, Nabla, gVisor and, most recently announced by AWS, Firecracker. While each tool has its pros and cons, this field is still fairly nascent. Container sandboxing for sure will become a more prominent topic in the months and years to come.

Kubernetes Security will Remain at the Forefront

The above are by no means the only security advices that were advertised during the myriad of discussions that took place in the Kubernetes community during KubeCon. Regardless of the specific technologies, the adages of “principle of least access”, “frequent testing”, “balancing security and utility” in computing — containers or not — are still very applicable.   

As evidenced by the popularity of the inaugural KubeSec Enterprise Summit during KubeCon, there is an ever growing interest and adoption of Kubernetes that will continue to push security even higher on the agenda of any containers-related discussions.  A lot of people are looking forward to hear more about the best practice and third-party tools, and increasingly accessible tools to help enforce Kubernetes security.

If you’d like to learn about the different container services on AWS, check our latest blogs and whitepaper.

kubernetes-containers-whitepaper-cta

Mency Woo

About Mency Woo

Mency Woo is a Containers Practice Lead at Onica with experience across healthcare, financial services and media industries. Mency loves all things containers and helping clients optimize workloads with AWS services.