In just a few short years, Kubernetes has evolved into a mainstream technology. The open-source container orchestration project has over 1,500 contributors and is ninth in number of commits on GitHub. Most enterprises have moved beyond experimentation, and are now planning to operationalize the deployment of workloads and consumer applications to their Kubernetes clusters.
But before they do, they need to get their container security practices under control. In a Cloud Native Computing Foundation survey, organizations using Kubernetes listed security as their top container related challenge.
There are reasons for this. The orchestration platform is such a huge open-source project, with so many contributors and so much change taking place every day, it is hard to keep pace with security issues. Standard security procedures and practices that work well in traditional environments are less effective in Kubernetes, where traffic is dynamic and you need to secure a wide array of pods, containers, nodes and images.
If you do not properly secure your Kubernetes environment, you will be exposing your organization to a number of risks. Attackers could execute a “container takeover,” where they compromise controls in one container and spread their attacks to others. They could strike against Kubernetes services by bypassing traditional security controls. Or they could acquire actual Kubernetes administrative logins and use their privileges to inflict harm on the system.
To get the most out of their container environments, organizations need to understand the end-to-end picture of how to operationalize Kubernetes security. Here are best practices teams can follow to upgrade security in four key areas:
- The containers themselves
- Identity Access Management (IAM)
- Infrastructure
- CI/CD automation
Container Security
The first step in implementing tight container security is to create a pod security policy. One recommended practice is to enable a pair of built-in security measures using features in Linux: AppArmor and seccomp. AppArmor is a Linux kernel security module used to set the actual profiles. Seccomp is a computer security facility in the Linux kernel that restricts the kind of calls the file system can make. Together, these features allow administrators to put limits on users’ privileges – a must for keeping a lid on unauthorized activity after a user is allowed access to the environment.
Once you create the policies, you can tweak them to manage user access to specific container functions. Everyone does not need access to everything. Instead, it is best to implement fine-grained controls to allow users privileges based on job function, the urgency of activity and the security needs of a particular project.
Now that you have given people pathways, you need to create walls. These can be done by setting administrative boundaries between resources. Each resource can be set aside in a specific namespace, separated from other resources that others have access to. You can create new namespaces and manage each one dynamically, adding and subtracting users and resources. Keeping resources organized and properly designated lowers the risk of mistakes about malicious activities.
IAM on Container Orchestrators
Authentication and authorizations are core elements of Kubernetes security. To secure and manage access to Kubernetes APIs, administrators need to create detailed authentication and authorization policies and implement advanced, full featured screening technologies. Anything less would open up the contained orchestration process to a wide variety of risks.
One best practice is to integrate third-party identity access technologies such as dex with Kubernetes. This allows a large enterprise to create a single sign-on (SSO) based on OpenID Connect to drive authentication. This will save time while providing the requisite levels of security needed for a container environment. Rather than adding users the old way – inside the Kubernetes application using certs – the OpenID connectors allow administrators to centralize all user accounts.
By maintaining your Kubernetes user base outside your identity system, your Kubernetes administrators will not have to reconfigure certificates every time they need to add or remove a user. They can use role-based access control (RBAC) functions to bring more control to what users can have access to, ensuring that they will not have the ability to delete the node or inject unapproved container images.
Infrastructure Security
Infrastructure security focuses on how organizations build their clusters and secure their Kubernetes environments. Here are a few best practices to follow in this area:
- TLS encryption – All network traffic needs to be encrypted. This goes for all scenarios – master communicating to a slave, two applications talking to each other and master to worker nodes. All these need to be encrypted. Istio is an open source service mesh that can manage microservice deployments and perform encryptions.
- Bastion host – This is a special purpose computer on a network specifically designed and configured to withstand attacks. Access to masters should only be done through bastion hosts, which can be hardened and can monitor users who have privileged access to Kubernetes.
- Private networking – Both Kubernetes master and worker nodes need to be deployed on private subnets to ensure secure connectivity with corporate networks, prevent direct reachability from the Internet and reduce the overall attack surface.
- Network policies – Kubernetes supports network policies that control how the two parts are communicating with each other. You should start with a default setting denying everything that is not needed. In a network policy, you can manage on a fine-grain basis all the IP addresses and protocols coming in and going out.
- Cluster node images – When you are building in a Kubernetes cluster, you will be using a Linux image. It needs to be CIS benchmarked to make sure all the Linux security controls are in place. Not doing the operating system hardening process can open up the infrastructure to software vulnerabilities. When you are scaling the master and worker nodes, make sure all the Linux security controls are in place.
- Logging and monitoring – You need this to detect anomalies on the application and infrastructure levels. If there are any attacks or anomalies – high usage or potential compromises – logging and monitoring will detect them. Application performance monitoring will detect other breaches such as DDOS attacks.
Automation Security
- Image scanning – All the deployments should be controlled through an automated CI/CD environment. At a high level, everything in Kubernetes is deployed as a container image. When someone creates an application, it ends up as a container image and gets added to the container registry. The container image scanner needs to be part of the CI/CD pipeline. When someone creates a container image, it needs to be continuously scanned for vulnerabilities. You can do image whitelisting through an admission controller in Kubernetes. If your application is using certain images, those images need to be approved.
- Secret management – Your clusters also need to be integrated via secret management systems, such as HashiCorp Vault. This ensures application pods will automatically receive required passwords and secrets at runtime, based on the AppRoles attached to the pods, This is preferable to using unsecured methods to load secrets inside the containers.
- Code analysis – Code scanning and static code analysis are also integral parts of automation security. When you are working on any application code in Kubernetes, you should scan the source code to make sure it does not have any vulnerabilities or any hard-coded anomalies.
Get Started
Kubernetes security is an evolving space. If you are building an enterprise solution, you need to pursue an end-to-end holistic security posture. There is no one-size-fits-all solution, but these best practices can lighten the load on organizations trying to build up their Kubernetes resources. Understanding how best practices can enhance your application’s security will help move your container orchestration processes several steps along the track.
