Taming namespace disk: why ephemeral-storage quotas in Kubernetes surprise teams and how to make them behave

Namespaces and ResourceQuotas are the control knobs operators use to divide cluster resources among teams. CPU and memory quotas tend to behave the way you expect: you set a limit, workloads that request more get rejected, and the math is straightforward. Ephemeral storage (the disk used by containers for logs, scratch space, and emptyDir volumes), however, is the sneaky sibling — it often doesn’t obey quotas unless several conditions are met. This article explains the common gotchas, why they happen, and a practical checklist (plus example YAML) to get ephemeral-storage quotas working reliably at the namespace level.

Why ephemeral-storage is different (an analogy)

• Think of CPU and memory quotas like speed limits on a highway — the traffic controller (API server + scheduler) can measure vehicle speed and refuse to admit cars moving faster than allowed.
• Ephemeral storage is more like parking lot usage spread across multiple lots and cars: files can be created and deleted but still held open by a process, logs accumulate on nodes, and some storage (like CSI ephemeral volumes) is managed by drivers outside the kubelet’s direct accounting. Measuring and enforcing that usage is inherently harder.

The enforcement model you must understand

• ResourceQuota objects live inside namespaces and limit aggregates (requests and/or limits) across all pods in that namespace. But for CPU and memory, the quota machinery requires pods to set requests or limits; otherwise the control plane may reject the pod or not count it as intended. The docs explicitly note that if you enforce a quota for cpu/memory, new pods must specify requests or limits; LimitRange can inject defaults to help. (v1-33.docs.kubernetes.io)
• For ephemeral-storage, ResourceQuota only measures what is declared (requests/limits) on the Pod spec. If a Pod does not declare ephemeral-storage, it might be ignored by quota accounting — so you can end up creating pods that produce logs and consume node disk without the quota catching them. That’s the core reason why ephemeral-storage quotas appear to “not work.” (v1-33.docs.kubernetes.io)

Common gotchas and surprises

1. Pods without ephemeral-storage requests aren’t always counted
   • If pods omit ephemeral-storage requests or limits, the ResourceQuota mechanism may not count their ephemeral usage toward namespace totals. This means a poorly specified workload can silently blow past the quota. (v1-33.docs.kubernetes.io)
2. Container logs and writable layers count, sometimes unexpectedly
   • When using a CRI runtime, container logs and the writable container layers are part of local ephemeral storage and will count against the node’s and (if configured) namespace’s ephemeral-storage usage. That can produce surprising evictions or quota exhaustion when logs spike. (v1-32.docs.kubernetes.io)
3. emptyDir accurate accounting needs filesystem project quotas and kubelet feature gates
   • Accurate tracking of emptyDir sizes (without expensive directory scans) relies on filesystem project quotas (XFS or specially-built ext4) and kubelet feature gates like LocalStorageCapacityIsolationFSQuotaMonitoring and LocalStorageCapacityIsolation. If these aren’t enabled and configured, accounting will be less accurate or absent. That makes namespace-level quotas unreliable for emptyDir volumes unless the node is prepared. (kubernetes.io)
4. Some distributions have known behavior differences
   • Enterprise distributions (e.g., OpenShift) have documented cases where ephemeral-storage ResourceQuota enforcement can be incomplete unless you explicitly set requests on pods — the practical impacts are cluster-specific and worth checking for your platform. (access.redhat.com)

How to make ephemeral-storage quotas actually work — checklist

• 1) Require or inject ephemeral-storage requests/limits
  • Use a LimitRange in the namespace to apply default ephemeral-storage requests and limits so pods that omit them will still be counted. LimitRange can automatically fill in defaults at admission time. (kubernetes.io)
• 2) Make sure pods (and workload templates like Deployments) set ephemeral-storage requests/limits
  • Enforce schema/validation (e.g., ValidatingAdmissionPolicy or Gatekeeper) so that every pod or PodSpec originating from higher-level controllers includes storage requests/limits.
• 3) Prep node filesystems if you rely on emptyDir quota accuracy
  • Enable project quotas on the node filesystem (XFS supports it by default; ext4 must be built and mounted with prjquota). Also enable kubelet feature gates for LocalStorageCapacityIsolation and LocalStorageCapacityIsolationFSQuotaMonitoring to switch from slow directory walks to project-quotas-based accounting. Without this, emptyDir accounting can be blind or imprecise. (kubernetes.io)
• 4) Account for container logs and writable layers
  • Understand where your container runtime stores logs and images. Logs are ephemeral-storage; bursts in logging can suddenly consume namespace quotas and trigger evictions. Consider centralizing logs (e.g., to a stdout aggregator) and applying log rotation/retention policies at the node level.
• 5) Monitor usage from both namespace and node perspectives
  • Combine kube-apiserver quota metrics with node-level disk metrics (node-exporter, kubelet stats) to correlate quota hits with actual node pressure. Auditing and alerts for both quota usage and node free space are useful.
• 6) Test under realistic workloads
  • Create synthetic workloads that write to emptyDir, generate logs, and create ephemeral CSI volumes to verify the combined behavior of quota accounting, kubelet configuration, and node filesystem limits.

Minimal example: LimitRange + ResourceQuota to encourage correct behavior

• LimitRange (inject defaults so pods that forget requests still get counted):

apiVersion: v1
kind: LimitRange
metadata:
  name: default-storage-limits
  namespace: team-a
spec:
  limits:
  - type: Container
    defaultRequest:
      ephemeral-storage: "200Mi"
    default:
      ephemeral-storage: "1Gi"

• ResourceQuota (namespace cap on ephemeral-storage requests and limits):

apiVersion: v1
kind: ResourceQuota
metadata:
  name: team-a-storage-quota
  namespace: team-a
spec:
  hard:
    requests.ephemeral-storage: "50Gi"
    limits.ephemeral-storage: "60Gi"

Notes on the example

• The LimitRange sets a reasonable default request (200Mi) and default limit (1Gi) for containers that don’t specify ephemeral-storage. That ensures pods are counted by the quota system even if users forget to declare storage. The ResourceQuota then caps aggregated requests and limits across the namespace. Remember: these default values only apply at admission time — existing pods aren’t retroactively changed. (kubernetes.io)

Edge cases and advanced considerations

• CSI ephemeral volumes: these volumes are created through a CSI driver and may not be fully visible to kubelet’s local accounting in the same way as emptyDir, depending on how the driver reports capacity and usage. Treat CSI ephemeral storage as a separate axis and test driver behavior. (v1-32.docs.kubernetes.io)
• Deleted-but-open files: files deleted while still held open by a process continue to consume space at the filesystem level. Project quota tracking is necessary to capture those transient but real usages that simple directory-walk approaches miss. (kubernetes.io)
• Quota changes don’t affect already-running pods: if you tighten a quota, existing pods keep running (the quota logic is not retroactive); the control plane enforces quotas at creation/admission of new objects. (v1-33.docs.kubernetes.io)

A short operational recipe (quick checklist without commands)

• Add a LimitRange to inject ephemeral-storage defaults in each namespace.
• Enforce Pod spec validation so controllers and users can’t create pods without storage requests/limits.
• If you use emptyDir heavily, ensure node filesystems are XFS or ext4 with project quotas, and enable kubelet feature gates for project-quotas-based monitoring.
• Monitor both namespace quota usage and node disk pressure; instrument logs and container writable layer sizes.
• Run failure-mode tests that simulate log storms and large emptyDir writes to ensure the whole chain (requests, counting, node-level enforcement) behaves as expected.

Conclusion Ephemeral-storage is essential and useful — but its nature (local, transient, influenced by logs and runtime behavior) makes namespace-level quota enforcement trickier than CPU or memory. The good news: once you force pods to declare requests/limits (or inject them with LimitRange), prepare node filesystems for project quotas if you rely on emptyDir accuracy, and monitor both cluster and node metrics, you can get namespace-level ephemeral-storage quotas to behave reliably. Treat ephemeral-storage like a combination of parking rules and meter readers: set clear rules (requests/limits), give the system the right instruments (project quotas + kubelet feature gates), and monitor so you catch the parking violators before they fill the lot.

Sources and further reading

• Kubernetes ResourceQuota concepts and behavior. (v1-33.docs.kubernetes.io)
• Kubernetes ephemeral storage and emptyDir accounting details. (kubernetes.io)
• LimitRange admission behavior and defaults. (kubernetes.io)
• kubelet feature gates for LocalStorageCapacityIsolation and FS quota monitoring. (kubernetes.io)
• Practical notes on ephemeral-storage ResourceQuota behavior in some distributions (OpenShift). (access.redhat.com)