Answers to your top Kubernetes security questions

With the growing popularity of container usage, a record number of organizations are turning to Kubernetes for their container orchestration needs as they look to more efficiently and quickly build and release applications.

Kubernetes brings together an application with everything it needs to run consistently in any environment. Originally built by Google and now maintained by the Cloud Native Computing Foundation, Kubernetes automates the rollout and management of containerized applications at scale, helping to remove the headaches and hurdles that accompany rapid, automated software deployments. 

As they become more common, containers and container orchestration tools such as Kubernetes are also increasingly popular targets for hackers. While Kubernetes has become widespread within the enterprise, it’s vital to familiarize yourself with the key security challenges that are likely to emerge. These are the security-related questions we hear the most often around Kubernetes.

Can containers run as root?   

If a container is running as root, attackers who evade your defenses in a container breach may be able to gain unrestricted network access across your network. After compromising the container, the attacker is free to scan the network, explore the file system, download packages, and run commands that are reserved for root users. In these “container breakout” scenarios, when container isolation mechanisms have been bypassed and additional privileges have been obtained on the host, the container can even run as root under the control of a hacker.

You can mitigate your risk by deploying these defense mechanisms to stop containers from running as root inside a cluster:

• PodSecurityPolicy is a cluster-level resource that can explicitly deny containers and pods that request to run as root at the API admission controller layer.   

• Open Policy Agent is a policy control tool which can enforce restrictions with even more flexibility.

Can containers mount sensitive volumes and directories?

Visibility into what volumes and directories are being mounted into your containers via the ‘hostPath’ configuration is critical. Be aware of the range of implications when mounting containers inside of Kubernetes clusters. You can mount certain volumes and directories from the host machine itself. But while mounting containers with everyday tasks like logging is a nonissue, granting this level of permissions for sensitive data may introduce security concerns. 

Are file systems set to read or read/write? 

Read is the preferred setting by default. If the root file system or dangerous files are mounted in read/write format, attackers might exploit the container escape vulnerability, known as CVE-2016-9962. In the event of a full container escape, the attacker will be able to remove the container from the restricted environment and abuse the underlying host server. 

Can pods run in privileged mode? 

When a container runs in privileged mode, it is granted most system capabilities available on the underlying host, which significantly increases the container’s attack surface. Consider disallowing privileged mode entirely. If additional system calls are required, you can explicitly define these in the SecurityContext of the PodSpec. 

What policies are in place and for whom? 

Take the time to audit your rules and ensure they are working as expected. DevOps tools can give security teams stricter policy control of the workloads running inside of a cluster. However, third-party tools also open the opportunity for existing policies to be either misconfigured or overwritten by another policy. Thus, it is critical to perform manual policy checks at regular intervals.

How is authentication handled? 

Kubernetes supports a number of authentication mechanisms. Authentication best practices must be in place to access a cluster. While there is no one-size-fits-all solution, make sure you have a robust authentication policy in place including multi-factor authentication (MFA), the ability to revoke credentials, and absolutely no sharing of credentials.

How does Kubernetes handle privileges?

Role Based Access Control (RBAC) is at the core of Kubernetes resource protection. The “principle of least privilege” (PoLP) dictates that each module (i.e. each user, process, device, or program) should be given only the minimal permissions which are essential to perform its intended function — and no more. Build permissions that support the PoLP to limit the blast radius of any attack. Explore tools that can help security teams audit and lock down RBAC appropriately, such as rakkess, rback, krane, and kubectl-who-can. 

How are Kubernetes Secrets being stored, retrieved, rotated, and revoked? 

Secrets must be handled with extreme caution to preserve the integrity of your environment. You can store Secrets using the Kubernetes Secrets API and etcd or elsewhere.

Where do container images come from? 

A container image is a binary dataset that encapsulates an application, plus all of its underlying files and software dependencies. Container images are dynamic and come from sources, like Docker Hub or publicly available registries, which may be hosting malicious images. When configuring clusters, ensure the image registry is trusted, secure, and checked for integrity. Use image scanning technologies and build minimal base images in order to minimize the container attack surface.

How is network security being enforced? 

Network security policy allows you to set restrictions on pod intercommunication. Network communication inside of the cluster, egress, and ingress should be as granular as possible. Limiting connection source keeps pods protected. Consider using Kubernetes Network Policies or a service mesh (e.g. Istio) to audit your network communications.

Are your hosts hardened with monitoring in place? 

Kubernetes usually relies on virtual machines or physical hardware to run workloads. These host machines need to be hardened appropriately. Close all unnecessary ports, keep systems patched, and ensure networking is locked down. Consider following the applicable CIS Benchmarks for systems hardening.

What is Kubernetes Audit?

Kubernetes Auditing provides security-relevant logs for events and activities that have taken place in your cluster. Logs should be sent to a separate log aggregation system for storage and analysis by security teams. 

How does ingress/load balancing work?

What external IPs are available? Ingress, the API object which manages external access to your cluster, can help load balance traffic. If left unchecked, the Kubernetes API has the ability to automatically create external load balancers and public IPs. Ensure your workloads are not left exposed to external attackers or curious users.

What happens if your application has a Server-Side Request Forgery (SSRF) or Remote Code Execution (RCE) bug? 

Walk through both scenarios and the incident response in detail as an exercise to ensure your defense is robust. Take security precautions to limit the blast radius in the event of a successful SSRF or RCE attack. 

What else can I do to prepare for any threats to Kubernetes environments? 

Have a threat model in place for your most common “what-if” security scenarios. Map out how your team will respond in the event of a cluster malfunction, insider threat, configuration mistake, and attack scenario. Threat modeling can be as formal or informal as you would like. We recommend Marco Lancini’s blog post, “The Current State of Kubernetes Threat Modelling,” to get started.

Additionally, ensure that all third-party components are updated. Remove any integrations which aren’t in use. Dashboards, libraries, and plugins are all sources of compromise within the Kubernetes ecosystem.

Kubernetes presents an exciting new ecosystem, but like with all tools, systems, and environments, it’s most useful if it’s secure.