Kubernetes in Production

After this lesson, you will be able to: Use namespaces for env separation, configure RBAC, set resource quotas, and enable horizontal pod autoscaling.

Production K8s is more than YAML. This lesson covers the four practices that turn a learning cluster into a real one.

Prerequisites:Helm

Namespaces for environment separation

Use namespaces to isolate environments inside ONE cluster (staging + production) or teams (team-a + team-b). Each namespace has its own quotas, RBAC, network policies. Pods in different namespaces still share the cluster network by default. Convention: kebab-case names (`prod`, `staging`, `team-foo`). Default namespace exists; use it only for one-off testing. Switch namespaces: `kubectl config set-context --current --namespace=prod` or use `kubens` (a one-liner switcher).

RBAC: least privilege for humans + apps

ServiceAccounts for apps, Roles for permissions, RoleBindings to glue them together.

tsx

# A read-only role for the staging namespace
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: staging
  name: read-only
rules:
  - apiGroups: [""]
    resources: [pods, services, configmaps]
    verbs: [get, list, watch]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  namespace: staging
  name: alice-read
subjects:
  - kind: User
    name: alice@example.com
  - kind: ServiceAccount
    name: cicd
    namespace: staging
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: read-only

# Cluster-wide variants: ClusterRole + ClusterRoleBinding (use sparingly)

Resource quotas: prevent one team from eating the cluster

Cap CPU, memory, object counts per namespace.

tsx

apiVersion: v1
kind: ResourceQuota
metadata:
  name: staging-quota
  namespace: staging
spec:
  hard:
    requests.cpu: "10"           # total CPU requests across all Pods in this ns
    requests.memory: 20Gi
    limits.cpu: "20"
    limits.memory: 40Gi
    persistentvolumeclaims: "10"
    services.loadbalancers: "2"  # cap public LBs
    pods: "50"

Horizontal Pod Autoscaler (HPA)

Scale Pods up/down based on CPU or custom metrics.

tsx

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: myapp-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: myapp
  minReplicas: 2
  maxReplicas: 20
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70   # scale up when avg CPU > 70%

# Verify
kubectl get hpa
kubectl top pods   # current CPU/memory (needs metrics-server installed)

# Custom metrics (requests-per-second, queue depth):
# Install Prometheus + KEDA for event-driven autoscaling.

Network policies (the underused security tool)

By default, every Pod can talk to every other Pod. NetworkPolicy locks that down. `Deny all` policy: deny ingress by default in a namespace, then explicitly allow what should flow. Massive defense-in-depth win: a compromised web Pod can't reach the DB without explicit allow. Requires a CNI that supports policies (Calico, Cilium). Not enforced by default in kind/minikube.

Common mistakes only experienced engineers avoid

Everyone has cluster-admin. Audit; most engineers need namespace-scoped Roles, not ClusterRoles. Skipping ResourceQuota. One bad team / app starves everyone. HPA without resource requests on the Pod. K8s can't compute utilization without a baseline. Setting maxReplicas too high. Autoscaling can rack up cloud bills fast on a bad metric. Forgetting NetworkPolicies. Most clusters are flat networks; compromise of one Pod = compromise of all.

Quick Check

Your HPA isn't scaling even under load. What's the most common cause?

Pick the canonical bug.

K8s is brokenThe Pods have no resource requests, so K8s can't compute % utilizationHPA is disabledCluster is too small

←Helm

Back to Kubernetes

Managed Kubernetes (EKS / GKE / AKS)→