Kubernetes in Depth — Storage, Security, and Advanced Features

May 24

Storage in Kubernetes

Kubernetes provides flexible storage mechanisms for persistent application data. Storage management in Kubernetes encompasses volumes for temporary data and persistent volumes for long-term storage, ensuring reliability and scalability in production clusters.

Volumes and Volume Types

A Pod specification declares one or more volumes, specifying their name, type, and mount point. Volumes enable data sharing among multiple containers within a Pod, serving as a method for container-to-container communication, or across multiple Pods with defined access modes. Kubernetes supports a range of volume types, each tailored to specific use cases:

emptyDir: A temporary directory created at Pod startup and erased upon Pod termination, recreated on container restarts, ideal for transient data.
hostPath: Mounts a pre-existing resource (directory, file socket, character, or block device) from the host node’s filesystem, suitable for local testing but less reliable in production.
NFS and iSCSI: Support multiple readers, making them straightforward choices for shared read-only access.
rbd, CephFS, GlusterFS: Enable multiple writers, ideal for distributed applications requiring concurrent write access.
GCEPersistentDisk, awsElasticBlockStore: Integrate with Google Cloud and AWS storage, requiring configured accounts and privileges for cloud-based deployments.

Kubernetes Pod Volumes

A particular access mode is part of a Pod request. As a request, the user may be granted more, but not less access, though a direct match is attempted first. The cluster groups volumes with the same mode together, then sorts volumes by size, from smallest to largest. The claim is checked against each in that access mode group, until a volume of sufficient size matches.

The three access modes are:

ReadWriteOnce: Allows read-write access by a single node. Two Pods on the same node can write, but a third Pod on another node triggers a FailedAttachVolume error.
ReadOnlyMany: Permits read-only access by multiple nodes.
ReadWriteMany: Supports read-write access by multiple nodes.

When a volume is requested, the cluster matches the Pod’s access mode, grouping volumes by mode and sorting them by size (smallest to largest). The kubelet, via the kubelet_pods.go script, maps raw devices, creates mount points, and establishes symbolic links on the host filesystem to associate storage with containers. The API server interacts with StorageClass plugins to fulfill requests, with specifics varying by backend storage.

The following YAML creates a Pod named exampleA with two containers sharing an emptyDir volume:

apiVersion: v1  
kind: Pod  
metadata:  
  name: exampleA  
spec:  
  containers:  
  - name: alphacont  
    image: busybox  
    volumeMounts:  
    - mountPath: /alphadir  
      name: sharevol  
  - name: betacont  
    image: busybox  
    volumeMounts:  
    - mountPath: /betadir  
      name: sharevol  
  volumes:  
  - name: sharevol  
    emptyDir: {}  

Verify shared access:

kubectl exec -ti exampleA -c betacont -- touch /betadir/foobar  
kubectl exec -ti exampleA -c alphacont -- ls -l /alphadir  
# Output:  
total 0  
-rw-r--r-- 1 root root 0 Nov 19 16:26 foobar  

Persistent Volume Claim

Persistent Volumes (PV) abstract storage to retain data beyond a Pod’s lifecycle. Pods request storage through Persistent Volume Claims (PVCs), specifying parameters like size and StorageClass. The cluster dynamically attaches a matching PV, supporting various backend storage types.

apiVersion: v1  
kind: PersistentVolumeClaim  
metadata:  
  name: myclaim  
spec:  
  accessModes:  
  - ReadWriteOnce  
  resources:  
    requests:  
      storage: 8Gi  

In the Pod:

spec:  
  containers:  
  - name: myapp  
    image: myapp  
    volumeMounts:  
    - name: test-volume  
      mountPath: "/data"  
  volumes:  
  - name: test-volume  
    persistentVolumeClaim:  
      claimName: myclaim  

Security in Kubernetes

Kubernetes secures clusters through robust access control and data protection mechanisms. Security in production environments relies on safeguarding sensitive data and controlling API access to prevent unauthorized actions and ensure compliance.

Secrets Management

Secrets store sensitive data, such as passwords and keys, in base64-encoded format, protecting them from plain-text exposure. They are created via kubectl commands or YAML manifests and integrated into Pods as environment variables or mounted volumes.

Creating Secrets
Secrets can be generated using kubectl:

kubectl create secret generic mysql --from-literal=password=root

Alternatively, manual creation involves base64-encoding data in a YAML manifest:

echo LFTr@1n | base64  
# Output: TEZUckAxbgo=  

apiVersion: v1  
kind: Secret  
metadata:  
  name: lf-secret  
data:  
  password: TEZUckAxbgo=  

Secrets are not encrypted by default, only base64-encoded, requiring additional measures like encryption at rest for enhanced security. Each Secret is limited to 1MB, and excessive use can strain host memory, as Secrets are stored as API objects.

Using Secrets as Environment Variables
Secrets can be injected into containers as environment variables:

apiVersion: v1  
kind: Pod  
metadata:  
  name: dbpod  
spec:  
  containers:  
  - image: mysql:5.5  
    name: dbpod  
    env:  
    - name: MYSQL_ROOT_PASSWORD  
      valueFrom:  
        secretKeyRef:  
          name: mysql  
          key: password  

This configuration securely passes the password to the MySQL container without exposing it in plain text.

Mounting Secrets as Volumes
Secrets can be mounted as files within a container, providing an alternative access method:

apiVersion: v1  
kind: Pod  
metadata:  
  name: example-pod  
spec:  
  containers:  
  - name: myapp  
    image: busybox  
    command: ["sleep", "3600"]  
    volumeMounts:  
    - mountPath: /mysqlpassword  
      name: mysql  
  volumes:  
  - name: mysql  
    secret:  
      secretName: mysql  

Verify access:

kubectl exec -ti example-pod -- cat /mysqlpassword/password  
# Output: LFTr@#$  

Mounting Secrets as volumes creates files named after the Secret’s keys, readable only by the container, enhancing data isolation.

API Security

Access to the Kubernetes API is secured through authentication, authorization, and admission control. These layers protect cluster operations from unauthorized access and ensure policy compliance.

Authentication
Kubernetes supports multiple authentication methods:

X.509 certificates: Used for client authentication, typically managed by kubeadm.
Service Accounts: Provide identities for Pods to interact with the API.
OpenID Connect: Integrates with external identity providers for user authentication.

Requests failing authentication are rejected before reaching further processing.

Authorization
Authorized requests are evaluated against policies using mechanisms like:

Role-Based Access Control (RBAC): Defines permissions via Roles and RoleBindings, granting specific actions to users or Service Accounts.
Attribute-Based Access Control (ABAC): Uses attributes to define access policies, less common in modern clusters.
Node Authorization: Restricts kubelet access to node-specific resources.

RBAC is the dominant approach in Kubernetes 1.30, offering granular control over API operations.

Admission Controllers
Admission Controllers validate and modify API objects before they are persisted. They enforce policies, such as resource limits or image validation, rejecting non-compliant requests. Examples include LimitRanger and PodSecurity.

TLS Encryption
All API requests are encrypted using TLS, configured automatically by kubeadm. Without valid SSL certificates, access is restricted, ensuring secure communication between clients and the API server.

Advanced Features

Kubernetes advanced features enhance the management of complex applications. These capabilities, including configuration management and sophisticated HTTP routing, provide flexibility and scalability for production workloads.

ConfigMaps for Configuration

ConfigMaps decouple configuration data from container images for dynamic application management. ConfigMaps store non-sensitive data as key-value pairs or configuration files, enabling Pods to access settings without hardcoding. They are created via kubectl or YAML manifests and used as environment variables, command arguments, or volume mounts.

Creating ConfigMaps
ConfigMaps can encapsulate configuration files, such as config.js:

kubectl create configmap foobar --from-file=config.js

The resulting YAML:

apiVersion: v1  
kind: ConfigMap  
metadata:  
  name: foobar  
data:  
  config.js: |  
    {  
      "key": "value"  
    }  

Retrieve with:

kubectl get configmap foobar -o yaml

ConfigMaps can also be populated from literal values or directories, supporting diverse configuration needs.

Using ConfigMaps

ConfigMaps are consumed in Pods in several ways:

Environment Variables: Inject values into containers:

apiVersion: v1  
kind: Pod  
metadata:  
  name: example-pod  
spec:  
  containers:  
  - name: myapp  
    image: myapp  
    env:  
    - name: SPECIAL_LEVEL_KEY  
      valueFrom:  
        configMapKeyRef:  
          name: special-config  
          key: special.how  

Command Arguments: Pass values to container commands (not shown in the provided excerpt but supported).
Volume Mounts: Mount ConfigMaps as files:

apiVersion: v1  
kind: Pod  
metadata:  
  name: config-pod  
spec:  
  containers:  
  - name: myapp  
    image: myapp  
    volumeMounts:  
    - name: config-volume  
      mountPath: "/etc/config"  
  volumes:  
  - name: config-volume  
    configMap:  
      name: special-config  

ConfigMaps must exist in the same namespace as the Pod before deployment, unless marked optional. They support complex configurations, such as populating volumes with specific paths or access modes, enhancing application portability.

Ingress for HTTP Routing

Ingress enables centralized HTTP and HTTPS routing to cluster services. Unlike Service Types (ClusterIP, NodePort, LoadBalancer, ExternalName), Ingress routes traffic based on request host or path, reducing reliance on external load balancers. It requires an Ingress Controller to process routing rules.

Ingress Controllers
Ingress Controllers, such as Nginx or Traefik, run as Pods and monitor the /ingresses endpoint in the networking.k8s.io/v1 API group. They apply Ingress Rules to direct external traffic to Services, typically over HTTP. Multiple controllers can coexist, with annotations specifying which controller handles a given Ingress:

apiVersion: networking.k8s.io/v1  
kind: Ingress  
metadata:  
  name: example-ingress  
  annotations:  
    nginx.ingress.kubernetes.io/rewrite-target: /  
spec:  
  rules:  
  - host: example.com  
    http:  
      paths:  
      - path: /  
        pathType: Prefix  
        backend:  
          service:  
            name: my-service  
            port:  
              number: 80  

Without a matching annotation, all controllers may attempt to process the Ingress, potentially causing conflicts.

Ingress Rules

Ingress Rules define how traffic is routed based on URL paths or hostnames. The above example routes requests to example.com/ to my-service on port 80. Path types (Prefix, Exact, ImplementationSpecific) provide granular control, while annotations like rewrite-target modify request paths for compatibility.

Ingress centralizes access to multiple Services, improving efficiency in large clusters. It integrates seamlessly with cloud-native environments, supporting production-grade HTTP routing as of Kubernetes 1.30.

Kubernetes streamlines data persistence and access control for robust application deployment. Advanced routing and configuration mechanisms enhance cluster flexibility. Mastery of these capabilities drives efficient management of production systems.

Kubernetesk8sArchitectureDevOpsMLOps

Andrey Sydelov