Kubernetes Security Best Practices

Kubernetes is a widely used open-source platform that has gained significant popularity in the container market. The deployment or management of containers manually on a large scale would be infeasible without it.

Several enterprise organizations utilize Kubernetes as a system for managing containerized applications, in order to avoid running complicated production systems.

As reported by Cloud Native Computing Foundation (CNCF), in 2020 there was increased use of Kubernetes to 91%, up from 58% in 2018.

Kubernetes is perhaps the most widely available container orchestration platform; however, it also has some critical security flaws related to the container environment, such as misconfigurations, security events, and other vulnerabilities.

In 2018, attackers used Tesla’s cloud environment to run cryptocurrency-mining malware because its Kubernetes console was misconfigured. The console was not password-protected, and one of its pods held access credentials for Tesla’s AWS environment.

When you use a Kubernetes tool for application development or infrastructure, your development lifecycle may be quicker than ever. The process, however, may run contrary to standard security practices, especially for risk-averse companies.

Many organizations have well-established change management and security policies that conflict with the principle of “move fast and break things” in the application development process. However, these two approaches of change management and Kubernetes do not necessarily have to conflict. Kubernetes’ built-in functionality for auditing always lets you see where your apps sit at any point in time.

Down below are some important security principles related to Kubernetes Technology:

Defense in depth can be portrayed as a giant castle with thick and huge stone walls which prohibit anyone from reaching them, while high precision archers are positioned above them to pounce on any attacker who tries to break into the castle. The entrance to the gate has a bridge that crosses a canal. The bridge is only occasionally lowered to allow entry.

When using Kubernetes, you should be sure to add multiple layers of security so that even if a compromise occurs at the first layer, the attacker will be stopped on the subsequent layers. Defend with Security configuration and controls, and apply them across the following components of your Kubernetes Architecture:

• Server
• (inside and outside Cluster)
• DNS
• Storage
• Cluster
• Node
• Pod
• Container
• Application
• Control Plane

The principle of least privilege is to restrict access and only grant minimal access on a need-to-know basis, so that different components can access only the information and resources they need to operate correctly. In the event of a component being compromised, an attacker can reach only the subset of information and resources available to that component. This limits the “blast radius” of the attack.

Assume you have an e-commerce store built using a “microservice” architecture with functionality broken into small, discrete components. With product and user information held in the same database, different microservices might each be granted access to only the appropriate parts of that database. An example will be if a product-search microservice needs read-only access to the product tables, but nothing more.

If this microservice somehow gets compromised, the broken service can’t overwrite product information (because it has only read access) or extract user information (because it has no access to that data at all). You should pay particular attention to the access and permissions granted to microservices and the information and system components they access in order to operate correctly.

Applying the principle of least privilege means that we make it more difficult for an attacker to cause damage.

1. Enable Role-Based Access Control (RBAC)

Control who can access the Kubernetes API and what permissions they have with Role-Based Access Control (RBAC). RBAC is usually enabled by default in Kubernetes 1.6 and beyond (later for some managed providers), but if you have upgraded since then and haven’t changed your configuration, you’ll want to double-check your settings. Because of the way Kubernetes authorization controllers are combined, you must both enable RBAC and disable legacy Attribute-Based Access Control (ABAC).

2. Separate Sensitive Workloads

To limit the potential impact of a compromise, it’s best to run sensitive workloads on a dedicated set of machines. This approach reduces the risk of a sensitive application being accessed through a less-secure application that shares a container runtime or host. For example, a compromised node’s kubelet credentials can usually access the contents of secrets only if they are mounted into pods scheduled on that node; if important secrets are scheduled onto many nodes throughout the cluster, an adversary will have more opportunities to steal them.

3. Secure Cloud Metadata Access

Sensitive metadata, such as kubelet admin credentials, can sometimes be stolen or misused to escalate privileges in a cluster. For example, a 2018 Shopify bug bounty disclosure detailed how a user was able to escalate privileges by confusing a microservice into leaking information from the cloud provider’s metadata service. On GKE, Google now recommends Workload Identity Federation for GKE, which replaces the older metadata concealment feature. Similar countermeasures may be needed in other environments.

4. Create and define cluster Network Policies

Network Policies allow you to control network access into and out of your containerized applications. To use them, you‘ll need to make sure that you have a networking provider that supports this resource; with some managed Kubernetes providers such as Google Kubernetes Engine (GKE), you‘ll need to opt in. (Enabling network policies in GKE will require a brief rolling upgrade if your cluster already exists.) Once that’s in place, start with some basic default network policies, such as blocking traffic from other namespaces by default.

5. Enforce Pod Security Standards Cluster-wide

Pod Security Policy was removed in Kubernetes v1.25. Its replacement, the built-in Pod Security Admission controller, enforces the Pod Security Standards (Privileged, Baseline and Restricted) on each namespace and sets limits on how workloads are allowed to run in your cluster. As a start, you could require that deployments drop the NET_RAW capability to defeat certain classes of network spoofing attacks.

6. Harden Node Security

You can follow these three steps to improve the security posture on your nodes:

• Ensure the Host is secure and configured correctly. One way to do so is to check your configuration against CIS Benchmarks; many products feature an autochecker that will assess conformance with these standards automatically.
• Control Network access to sensitive Ports. Make sure that your network blocks access to ports used by kubelet, including 10250 and 10255. Consider limiting access to the Kubernetes API server except from trusted networks. Malicious users have abused access to these ports to run cryptocurrency miners in clusters that are not configured to require authentication and authorization on the kubelet API server.
• Minimize Administrative access to Kubernetes Nodes. Access to the nodes in your cluster should generally be restricted; debugging and other tasks can usually be handled without direct access to the node.

Make sure you have audit logs enabled and are monitoring them for anomalous or unwanted API calls, especially any authorization failures; these log entries will have a status message “Forbidden.” Authorization failures could mean that an attacker is trying to abuse stolen credentials. Managed Kubernetes providers, including GKE, provide access to this data in their cloud console and may allow you to set up alerts on authorization failures.

8. Deploy a Service Mesh

A service mesh tightly integrates with the infrastructure layer of the application, offering a consistent way to secure, connect, and observe microservices. A service mesh controls how various services share data in their east-west communication in a distributed system, usually with sidecar proxies.

9. Assess Container Privileges

Kubernetes security assessment should consider container capabilities, privileges, and role bindings, which all come with security risk. The least privilege that allows intended function and capabilities is the goal.

10. Secret Encryption

Since secret values protect sensitive data, we want them to be hard to access. Ideally, they should be protected at rest and in transit:

Secrets should always be stored on disk in encrypted form, so that an attacker with access to the filesystem cannot simply read them from the file. In transit, Secrets should be encrypted whenever they are sent as network traffic, so that an attacker snooping on the network cannot read them as they pass.

Encryption in transit is achieved by encrypting the traffic between the Kubernetes control-plane components using TLS. Ensure that your “etcd” cluster is also encrypted on disk. (The API server has access to the encrypted data in “etcd”, so you will also want to limit API access also).

• Scan your Images and Source Code: As with any application, implementing application security testing best practices of using various scanning tools such as SAST, DAST, IAST, or SCA will help ensure your code is as secure as possible.
• Don’t forget about Open Source: Open source code is in almost every proprietary software offering on the market. If you’re not actively managing your open source code, you’re not secure.
• Implement Workflows to remediate vulnerabilities: Automate your workflows to ensure that vulnerabilities are remediated as soon as possible and code is secure.
• Use updated Images: This one goes without saying, but make sure you’re using the most updated image available as it will likely be the most secure image available. Destroy any outdated images; it’s not really possible to patch them.

• Don’t deploy images from unknown sources: If you don’t know the provenance of an image, you can’t trust it.
• Scan images during deployment: You might find new vulnerabilities that have been disclosed since your image was last scanned in development.
• Use image admission controls: Admission controls stop an image if it violates your org’s security policies and prevent a container from running with greater privileges than it needs for its tasks.
• Limit privileges used by a container: Giving a container broad permissions can be problematic if a malicious user gets control of it. Limit your exposure by limiting a container’s privileges and never run a container with the privileged flag.
• Don’t use defaults: Enable role-based access control (RBAC) in your container orchestration to regulate access based on the roles of individual users within your organization.

• Scan Images in Production: Again, we can’t stress this enough: new vulnerabilities are disclosed all the time. Make sure you’re meticulous about continuously scanning your images.
• Secure the Network with a Container firewall: You can secure and inspect your network using a container firewall, which applies traditional network security techniques to the cloud-native Kubernetes environment.
• Implement Network Security Policies: Keep an eye on your network traffic flows by creating and defining cluster network policies that allow you to control network access in and out of your containerized applications.
• Establish security boundaries using Namespaces: Namespaces are virtual clusters in Kubernetes. They allow you to divide cluster resources between multiple users.
• Create a Security Policy for Pods: Pods, the smallest execution unit in Kubernetes, are used to organize containers. Pod Security Standards, enforced with Pod Security Admission, can be used to address issues such as a container running as root.
• Analyze Pods in the same deployment: When replicated, pods should exhibit identical behavior. Monitor and analyze pods. If you see them behaving differently, you need to investigate immediately to ensure no breach has occurred.

The following seven do’s and don’ts of Kubernetes Deployments may help you to improve the security of your applications.

#1: Do Update to the Latest Image

Very simple and easy, yet many of the development teams ignore this rule. Each update (whether for images or third-party tools used by the developing team) not only carries new features but also includes many bug fixes and security patches. Be sure to install the latest updates as they’ll most likely fix some newly discovered security loopholes.

Attackers threaten your outdated environment. Remember that Kubernetes is open-source, which means the community constantly looks for flaws and pushes updates to fix them. Don’t use any third-party tool or image that isn’t in line with the latest stable release in your environment. Guide your security team to manually check for the latest releases and schedule them for at least once a quarter.

#2: Do Implement Security Guidelines or Define Policies

The manager of the development team is responsible not only for defining the security policy and guidelines for the work environment but also for ensuring that the entire workflow follows these policies to the letter. Do not compromise on this set of rules and deal with it strictly.

Define the access control and privileges in your policy and do not give unnecessary permissions or control to an unauthorized entity. Giving the container only required privileges can help in reducing the attack radius. Additionally, define the roles for accessing Kubernetes-API and assign permissions accordingly.

#3: Do Reduce the Attack Surface

The attack surface defines the scope and possible ways a system can be attacked. If your system is more complex and detailed, you are required to put extra effort to reduce the attack surface to reduce the likelihood of an attack.

One way to implement this technique is by reducing the size of the overall system by de-coupling and minimize the amount of code. With this approach, you have to test less amount of code and secure a smaller number of vulnerabilities. Leaving unused code that you don’t need will just expand the attack surface. This will be a headache for the testing and development teams.

#4: Don’t Manage Your Cluster Manually

Automation is one of the core features of Kubernetes. Making quick changes or fixing problems manually may seem like a good idea, but it puts your security at more risk. This will disturb and create ambiguity in different parts of the deployment, especially those that need small changes like container tags/images or environment variables. Things can get even worse if a person who is totally unaware of these manual changes encounters them. This will further increase confusion and ambiguity.

#5: Don’t Follow the Default Configurations

The default configurations are targeted for general usage and are not suited for your specific lifecycle and demands. The default configurations can be used only for overviewing, not for actual implementation. With everyone aware of default configurations, it becomes easier to bypass default security checks. Even if you have to follow any of the default configurations, at least make sure to properly check and test them before implementing them into the lifecycle. For example, the default permissions allow all components to communicate with each other. This is not a good practice for access control as many components have unnecessary privileges.

#6: Don’t Carry Unnecessary Components

Remove all the unnecessary tools and components from the containers that are in production. The minimalist approach is always recommended as it will reduce the burden of managing the security for non-required tools. Any of these extra modules may carry vulnerabilities that can harm the container. Attackers target weaknesses in common tools used by the developers, so it is better to get rid of unused components.

#7: Don’t Build Secrets into Images

Building secrets into container images is a bad idea, and here are a few reasons. Anyone who has access to the image can obtain the secrets it holds. Bear in mind that the set of people who can access the image may not be the same set of people who need your production credentials.

Most of this post assumes you operate a cluster. If you run a small SaaS on a PaaS, a serverless runtime or a single container service, you do not need RBAC design, network policies or node hardening, because the provider owns that layer. Your job shrinks to image hygiene: scan images, keep base images current, keep secrets out of the build, and limit the service account the container runs as. Revisit Kubernetes when the workload outgrows the platform.

You also do not need all ten practices at once on a new managed cluster. GKE, EKS and AKS ship with RBAC on and offer audit logging and image scanning as switches. Turn those on first, then add network policies and pod security standards as you add tenants or sensitive data.

If nobody on your team can read a Kubernetes manifest, adopting the platform creates more risk than it removes. A simpler deployment model your team understands beats a cluster nobody maintains.

Does Kubernetes offer weak security? Well, it most likely depends upon how you manage this framework. Out of the box, Kubernetes does not provide the best security. You have to follow the given guidelines to get maximum security from Kubernetes. With a properly secured Kubernetes platform, you can avail yourself of exciting features like automation, deployment, and scaling while maintaining your security.

Talk to a Cybersecurity Trusted Advisor at IRM Consulting & Advisory

Check out our Marketplace