Kubernetes Pod Security Admission

Kubernetes is an open-source orchestration platform for automating containerized applications' deployment, scaling, and management. With production-grade stability and features, Kubernetes has become the standard tool for many organizations deploying and managing containerized applications.

Within Kubernetes, containerized applications are managed as logical units called Pods. In any deployment environment, these Pods' security is vital. Kubernetes provides various security controls, such as Pod Security Standards (PSS) and Pod Security Admission (PSA), to efficiently manage the permissions and capabilities of Pods. These controls ensure that Pods operate with the minimum required access. This approach minimizes the risk of a compromised Pod affecting other resources.

Kubernetes version v1.21 shifted from PodSecurityPolicy to the new Pod Security Admission controls. While PodSecurityPolicy served its purpose, the new controls offer a more streamlined and accessible approach to enforcing security policies on Pods. Pod Security Admission introduces predefined security contexts and customization capabilities, enhancing flexibility, control, and ease of use.

Understanding the significance of Pod security is fundamental to managing and operating Kubernetes clusters effectively and securely. This guide will set the groundwork for anyone interested in building secure, robust, and compliant containerized environments within Kubernetes. We will explore the specifics of Pod Security Admission, including practical implementation walkthroughs, testing strategies, and seven best practices for applying Pod Security Admission controls.

The table below summarizes the Kubernetes Pod Security Admission concepts this article will explore in more detail.

Pod Security Admission is a feature introduced in Kubernetes to enforce clear and consistent isolation levels for Pods. It builds upon the Kubernetes Pod Security Standards, guidelines that govern how Pods behave and interact with other resources.

By applying security restrictions at the Kubernetes namespace level when Pods are created, Pod Security Admission provides a mechanism to ensure that Pods operate with only the necessary permissions. This enhances security and aligns with broader best practices in software deployment, minimizing the risk of unauthorized access or compromised resources.

The importance of Pod Security Admission lies in its ability to make security a fundamental and integral part of the Kubernetes ecosystem. Rather than treating security as an afterthought, Pod Security Admission ensures that it is part of the design and operation of every Pod.

This proactive approach to security helps organizations maintain compliance and protect their applications, data, and infrastructure.

Baseline, restricted, and privileged policies

To operationalize these security controls, Pod Security Admission categorizes Pods into three levels based on their security requirements. These levels, defined by the Pod Security Standards, are instrumental in aligning the security posture of Pods with organizational needs:

• Privileged policies: These policies provide the least restriction, allowing almost all Pod capabilities. They are generally used for system-level Pods that require higher privileges.
• Baseline policies: A middle ground between Privileged and Restricted; Baseline Policies block known privilege escalations while providing some flexibility. These are often suitable for general-purpose applications.
• Restricted policies: Restricted Policies significantly limit the pod's capabilities to minimize the potential attack surface. They are recommended for highly sensitive workloads or those that process critical data.

These policy levels affect specific settings in a Pod’s details, like:

• Ports used by containers (spec.containers[*].ports)
• Paths for volumes (spec.volumes[*].hostPath)
• Security settings at the pod and container levels (spec.securityContext and spec.containers[*].securityContext)

Once you know these policy levels, you can start setting up security rules for your cluster. You can do this in two ways: using namespace labels or an AdmissionConfiguration resource.

Note: Examples of how to apply these policy levels through labels and AdmissionConfiguration resources are provided in the hands-on section of this article.

Pod Security Admission labels: Bridging policy and practice

To translate high-level security policies into actionable enforcement mechanisms, Kubernetes introduces a labeling system for namespaces. Through these labels, administrators can dictate the desired behavior when a pod potentially violates a given security policy:

• pod-security.kubernetes.io/enforce: This mode is strict. A Pod is only accepted if it aligns with the defined policy.
• pod-security.kubernetes.io/audit: More lenient, this mode notes any policy violation in the audit log, but the Pod creation isn't hindered.
• pod-security.kubernetes.io/warn: Serving as a middle ground, this mode raises a warning for policy violations but otherwise permits the Pod's creation.

Administrators can flexibly apply these labels, individually or in tandem, to tailor security enforcement to specific scenarios. Moreover, these labels can be adjusted to cater to different security needs within a single namespace.

Additionally, it's worth noting that specific versions of policies can be applied to ensure compatibility and granularity in control. For instance, one could pin a specific audit version like below.

apiVersion: v1
kind: Namespace
metadata:
  name: k8s-psa-demo-ns
  labels:
    pod-security.kubernetes.io/enforce: baseline
    pod-security.kubernetes.io/enforce-version: v1.28

Ensuring adherence to the baseline policy as of Kubernetes version 1.28. This enables administrators to align security measures with specific Kubernetes versions and maintain flexibility as policies evolve.

Differences between Pod Security Admission and the deprecated PodSecurityPolicy

Kubernetes version 1.21 significantly shifted from PodSecurityPolicy (PSP) to Pod Security Admission. While PSP intended to enforce security settings on Pods, it was deprecated due to its complexity and lack of flexibility.

Pod Security Admission introduces a more streamlined approach, utilizing labels to define admission control modes at the namespace level. These labels dictate the action the control plane takes if a potential violation is detected, such as rejection (enforce), audit annotation (audit), or user-facing warning (warn).

Moreover, cluster admins can statically configure exemptions, allowing specific exceptions based on various criteria such as Usernames, RuntimeClassNames, and Namespaces.

The transition from PodSecurityPolicy to Pod Security Admission reflects a more modular and configurable design, aligning with the modernization of security controls within Kubernetes.

Practical implications include easier management, clear identification of security levels, and adaptable security according to an organization's specific requirements and policies.

We are using Amazon Elastic Kubernetes Service (AWS EKS) for this implementation. However, you can utilize other managed Kubernetes options, such as Azure Kubernetes Service (AKS) and Google Kubernetes Engine (GKE). There are alternatives for local deployments, such as minikube and K3s; if you are familiar with these options, you can also use one of these tools.

Note: Ensure you deploy Kubernetes version 1.25+ if you follow this demo on your local machine or other managed providers. For options like minikube or k3s, you can check the release pages for current versions. If you are using managed providers, there will be an option to select the Kubernetes version during deployment.

Pre-requisites

To follow this tutorial, you’ll need the below tools installed locally:

• awscli (Installed and configured with AWS credentials)
• kubectl
• helm
• eksctl (For deploying the EKS cluster)
• git

Note: Running a managed Kubernetes cluster in cloud vendors incurs hourly costs. We keep costs as low as possible in this demo by using AWS Spot instances and cleaning the resources once the demo is completed.

Deploying AWS EKS Cluster

We use eksctl tool to create an AWS EKS cluster. Start by writing a cluster.yaml file with the below configuration.

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig

metadata:
  name: pod-security-admission-demo
  region: us-east-1

iam:
  withOIDC: true

managedNodeGroups:
  - name: node-group-spot
    instanceTypes: ["t3.small", "t3.medium"]
    spot: true
    desiredCapacity: 2
    volumeSize: 8

addons:
  - name: vpc-cni
  - name: coredns
  - name: aws-ebs-csi-driver
  - name: kube-proxy

The file defines the configuration for creating an AWS EKS cluster named pod-security-admission-demo in the us-east-1 region. It also includes a managed node group called node-group-spot consisting of spot instances, a cost-effective option for running your workloads. The node group uses a mix of t3.small and t3.medium instance types, with a desired capacity of two nodes and a volume size of 8 GB for each node.

The addons section lists the necessary components for the cluster, such as

• VPC CNI plugin
• CoreDNS for service discovery
• AWS EBS CSI driver for dynamic provisioning of EBS volumes
• Kube-proxy for managing network traffic between pods and services

To apply the configuration, execute the command:

> eksctl create cluster -f cluster.yaml

This will create an EKS cluster with a node group of two nodes in the us-east-1 region. Once the cluster is ready, you should see an output similar to the one below.

EKS cluster "pod-security-admission-demo" in "us-east-1" region is ready.

Note: If you encounter any errors during the EKS cluster creation, it could be related to insufficient IAM permissions. To validate your CLI's IAM Principal, you can run the below command to ensure you're operating with the correct permissions.

> aws sts get-caller-identity

We must update the kubeconfig file with newly created cluster access to interact with the cluster. To update the kubeconfig, execute the command.

> aws eks --region us-east-1 update-kubeconfig --name pod-security-admission-demo

To confirm the cluster access, execute the command to get the Pods from all namespaces.

> kubectl get pods -A

As we've discussed the foundational concepts and guidelines around Pod Security Admission and Pod Security Standards in Kubernetes, it's time to roll up our sleeves and dive into some real-world testing.

In this hands-on section, we'll work with Kubernetes resources to understand how policies affect pod and deployment behavior. We will walk you through scenarios that progressively tighten security configurations and observe their impact.

We will use aws-samples/k8s-psa-pss-testing GitHub repository, which contains the YAML manifests and scripts we'll need for our exercises. So, let's transition from theory to practice and see these security features in action.

Initial setup

Setting the stage properly is essential before diving into the actual hands-on part. Start by cloning the GitHub repository named k8s-psa-pss-testing, which will be the backbone for our Pod Security Admission (PSA) and Pod Security Standards (PSS) testing.

> git clone https://github.com/aws-samples/k8s-psa-pss-testing.git

After cloning the repository, review the YAML manifests and scripts to get an overview of the deployment resources.

What's included in the repository:

• YAML manifests: These blueprint files define how your Kubernetes resources should look. They include configurations for deployments and pods testing various Pod Security Standards.
• tests.sh: A shell script that will apply the YAML manifests in a Kubernetes cluster. This makes it easy to automate the testing of various security policies.
• clean.sh: Another shell script that will delete the resources created by tests.sh, cleaning up the Kubernetes environment.

Kubernetes Resources in the Repository:

• policy-test namespace: This is a dedicated Kubernetes namespace where all our test resources will reside.
• Known good deployment: A Kubernetes deployment that adheres to the best practices based on documented Pod Security Standards (PSS). This will be created and then deleted as part of the tests.
• Known bad deployments and Pods: These deployments and Pods intentionally violate specific Pod Security Standards (PSS). They are categorized into different scenarios for testing:
  • Missing securityContext at both the Pod and container levels.
  • Presence of securityContext with valid but restricted settings.
  • Incorrect securityContext settings that are explicitly disallowed by specific PSS profiles.
  • Pods with disallowed settings for hostNetwork, hostPID, and hostIPC.

By running the tests.sh and clean.sh scripts in the context of various Pod Security Standards, you will gain practical insights into how PSA and PSS operate and can be enforced in a Kubernetes environment.

This initial setup ensures you have all the necessary components for the hands-on demonstration.

Scenario 1: Default configuration

The default Pod Security Admission (PSA) configuration in this scenario is set to be permissive, meaning it doesn't restrict the creation of any Kubernetes resources based on their security contexts or other settings. This is meant to be a baseline to understand what happens when no specific security policies are enforced.

Execute the tests.sh script to apply the default Kubernetes resources.

> ./tests.sh

You should see output like this:

    namespace/policy-test created

    >>> 1. Good config...
    deployment.apps/test created
    deployment.apps "test" deleted

    >>> 2. Deployment - Missing container security context element...
    deployment.apps/test created

    >>> 3. Pod - Missing container security context element...
    pod/test created

    >>> 4. Pod - Pod security context, but Missing container security context element...
    pod/test2 created

    >>> 5. Pod - Container security context element present, with incorrect settings...
    pod/test3 created

    >>> 6. Pod - Container security context element present, with incorrect spec.hostNetwork, spec.hostPID, spec.hostIPC settings...
    pod/test4 created

Observations and Discussion:

• Good config: The deployment with a known good configuration is created and immediately deleted, as expected.
• Bad configs: Notice that deployments and Pods with missing or incorrect securityContext settings are also successfully created. This happens because the default PSA policy does not enforce any restrictions.
• Spec level settings: Even the Pod with disallowed settings for hostNetwork, hostPID, and hostIPC is allowed to be created under the default, permissive PSA.

This lack of restrictions may suit specific test environments but exposes the cluster to various security risks. It's vital to understand that this is not recommended for production setups.

Cleanup:

After making your observations, it's good practice to clean up the resources you've created to leave the cluster clean.

Run the clean.sh script:

> ./clean.sh

This will remove all the resources created during the testing of this scenario, thereby restoring your Kubernetes environment to its original state. Now, you're all set to move on to the following scenario, where we'll introduce some security policies to enforce.

Scenario 2: Applying baseline policies

Concept of "Baseline" in Pod Security Standards (PSS):

The baseline policy prevents known security vulnerabilities without causing operational issues. This moderate level of security should be applicable for most workloads and provides a good balance between safety and ease of operation.

Modifying the policy-test Namespace to Include "Baseline" Labels:

Update the policy-test namespace configuration to apply the baseline policies.

> kubectl edit namespace policy-test

Add or modify the labels to match the following:

    apiVersion: v1
    kind: Namespace
    metadata:
      name: policy-test
      labels:
        pod-security.kubernetes.io/enforce: baseline
        pod-security.kubernetes.io/audit: baseline
        pod-security.kubernetes.io/warn: baseline

Execute tests.sh:

Execute the tests.sh script to apply Kubernetes resources with the new baseline policies.

> ./tests.sh

Expected Outcomes:

The test output should look like this:

namespace/policy-test created

<redacted>

>>> 6. Pod - Container security context element present, with incorrect spec.hostNetwork, spec.hostPID, spec.hostIPC settings...
Error from server (Forbidden): error when creating "policy/psa-pss/tests/6-pod.yaml": pods "test4" is forbidden: violates PodSecurity "baseline:latest": host namespaces (hostNetwork=true, hostPID=true, hostIPC=true), hostPort (container "test" uses hostPort 8080)

• 4 Pods Allowed: Pods and deployments with configurations that match the baseline security profile are allowed (1, 2, 3, and 4).
• 1 Pod Disallowed: The Pod with hostNetwork, hostPID, and hostIPC settings is forbidden. This is because it violates the baseline security standards.

Discuss the Outcomes:

• This scenario demonstrates the impact of enabling baseline policies in the namespace. Most test Pods and deployments are allowed, except for the one with disallowed host-level settings.
• The disallowed Pod (test4) proves that the baseline PSS profile effectively blocks specific configurations that could expose the system to security risks.

Cleanup:

As before, run the clean.sh script to remove the resources:

> ./clean.sh

This concludes Scenario 2. Now you understand the effects of applying a baseline security profile at the namespace level, and you're ready to explore more stringent security configurations in the following scenario.

Scenario 3: Applying restricted policies

Understanding the "Restricted" PSS Profile:

The restricted profile in Pod Security Standards (PSS) is the most stringent level of security, optimized for production-level, mission-critical workloads. This profile is designed to limit any operation that could compromise the security posture of the Kubernetes cluster. It applies multiple restrictions, such as prohibiting privileged containers, enforcing the principle of least privilege, and ensuring strong isolation between Pods.

Modifying the policy-test Namespace for "Restricted" Policies:

Update the configuration of the policy-test namespace to enforce the restricted PSS profile.

> kubectl edit namespace policy-test

The labels should look like this:

    apiVersion: v1
    kind: Namespace
    metadata:
      name: policy-test
      labels:
        pod-security.kubernetes.io/enforce: restricted
        pod-security.kubernetes.io/audit: restricted
        pod-security.kubernetes.io/warn: restricted

Execute tests.sh:

Execute the tests.sh script to try applying various Kubernetes resources under the restricted policies.

> ./tests.sh

The expected output is:

<redacted>

>>> 2. Deployment - Missing container security context element...
Warning: would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "test" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "test" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "test" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "test" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
deployment.apps/test created

>>> 3. Pod - Missing container security context element...
Error from server (Forbidden): error when creating "policy/psa-pss/tests/3-pod.yaml": pods "test" is forbidden: violates PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "test" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "test" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "test" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "test" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")

>>> 4. Pod - Pod security context, but Missing container security context element...
Error from server (Forbidden): error when creating "policy/psa-pss/tests/4-pod.yaml": pods "test2" is forbidden: violates PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "test" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "test" must set securityContext.capabilities.drop=["ALL"]), seccompProfile (pod or container "test" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")

>>> 5. Pod - Container security context element present, with incorrect settings...
Error from server (Forbidden): error when creating "policy/psa-pss/tests/5-pod.yaml": pods "test3" is forbidden: violates PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "test" must set securityContext.allowPrivilegeEscalation=false), runAsNonRoot != true (container "test" must not set securityContext.runAsNonRoot=false)


>>> 6. Pod - Container security context element present, with incorrect spec.hostNetwork, spec.hostPID, spec.hostIPC settings...
Error from server (Forbidden): error when creating "policy/psa-pss/tests/6-pod.yaml": pods "test4" is forbidden: violates PodSecurity "restricted:latest": host namespaces (hostNetwork=true, hostPID=true, hostIPC=true), hostPort (container "test" uses hostPort 8080)

Expected Outcomes:

The results of the test indicate strict enforcement of security settings:

• 1 Deployment Created: Only the good configuration (test deployment) gets created, although the underlying Pod violates the restricted PSS and is not scheduled.
• 0 Pods Allowed: None of the other test Pods are allowed, as all violate the restricted security policies.

Discussing the Outcomes:

• As expected, the restricted PSS profile is quite stringent and blocks the creation of any Pods that do not adhere to the highest security measures.
• This can be observed in the output for each failed Pod creation, where Kubernetes details the specific policy violations.
• The deployment fails to create any Pods, and the status field gives a ReplicaFailure message indicating the exact policy the Pod is violating.

Cleanup:

Finally, revert all the changes made during this test by running the clean.sh script:

> ./clean.sh

This completes Scenario 3, demonstrating the high-level security constraints imposed by the restricted PSS profile. It shows that strict security configurations, while offering robust protections, require thorough validation and testing to ensure that your workloads can operate within these constraints.

Resource clean up

To delete the EKS cluster and other AWS resources, execute the command below in the directory where you created the cluster.yaml file.

❯ eksctl delete cluster -f cluster.yaml --disable-nodegroup-eviction --force

By incorporating the below considerations into your Kubernetes security strategy, you can create a robust, secure, and efficient environment tailored to your specific operational needs.

Understand your workloads

Understanding your workload's security needs is critical before applying any Pod Security Standard (PSS) profile. Misconfiguring the PSS can result in either over-restricting or under-restricting your applications.

Over-restricting can hinder application functionality and lead to operational issues. On the other hand, under-restricting can expose your cluster to unnecessary security risks. Take time to review your workloads and tailor your security policies accordingly.

Consider using open-source tools like Kube-Bench for conducting security assessments that guide your policy decisions.

Follow the principle of least privilege

The principle of least privilege is fundamental in cybersecurity. It means giving only the permissions necessary for users (or systems) to accomplish their tasks. In the context of Kubernetes, start with the most restrictive PSS profile and only relax constraints when necessary and for the shortest time required. This minimizes the potential attack surface and reduces risk.

The open-source Kyverno tool can help you enforce and validate configurations, ensuring you adhere to this principle.

Adopt a GitOps approach for Kubernetes resource management

GitOps, a paradigm or a set of practices that empowers developers to perform tasks that typically fall under IT operations, is increasingly popular for managing Kubernetes resources. Using tools like ArgoCD, you can manage your Kubernetes configurations from your Git repositories.

This approach ensures that changes are peer-reviewed, versioned, and auditable, making your resource management more robust, consistent, and secure.

Monitor and audit

The PSA modes, such as audit and warn, offer valuable utilities for transition periods and ongoing compliance. They allow you to monitor how new policies affect your workloads without enforcing them. This provides a safety net that can help you refine your policies before you implement them, reducing the risk of operational issues.

Use the audit and warn PSA modes, open-source monitoring solutions like Prometheus, and logging solutions like Loki to keep track of your environment's security posture.

Keep up with updates

The security landscape constantly evolves with new vulnerabilities, attack vectors, and patches. Keeping your team and systems updated is not a one-time activity but an ongoing process. Regularly review Kubernetes release notes, security advisories, and other relevant publications to adjust your security configurations per the latest best practices.

Open-source tools like Trivy can scan container images for security vulnerabilities, helping you stay updated on the security front.

Document effectively

Documenting your security policies is more than a requirement; it’s a best practice. Good documentation facilitates compliance with internal and external audits and simplifies troubleshooting.

Having a documented set of policies and configurations allows you to quickly understand the security posture of your Kubernetes cluster, especially in the face of an incident or when onboarding new team members.

This is more than just the security team's responsibility; every developer, operator, and manager should understand the security best practices and how they are implemented in your Kubernetes environments.

Use test/staging environments

Always have a test or staging environment closely mimicking your production setup. First, new security features, policies, or changes should always be validated here.

This environment should be used to conduct automated tests, performance benchmarks, and even peer reviews, ensuring that new configurations don't break existing functionalities before rolling them into production.

In this journey through Kubernetes Pod Security Admission (PSA) and Pod Security Standards (PSS), we've unpacked the different layers of security configurations available to Kubernetes administrators. The PSA framework is a powerful tool in the security arsenal, offering varying degrees of control and oversight through its profiles: Privileged, Baseline, and Restricted.

Implementing these standards can significantly bolster the security posture of your Kubernetes clusters by limiting the attack surface and ensuring that workloads operate under the principle of least privilege.

By understanding your workloads, adopting the principle of least privilege, using GitOps for resource management, and diligently monitoring, documenting, and educating your team, you can pave the way for a secure, efficient, and resilient Kubernetes environment.

For more in-depth details and further exploration on Pod Security Admission, refer to the official Kubernetes documentation here.

So keep reading, keep experimenting, and most importantly, keep securing your Kubernetes clusters. The journey to Kubernetes mastery is not a sprint but a marathon, and the tools and techniques you have learned today will pave the way for a safer, more robust infrastructure tomorrow.