In the dynamic ecosystem of container orchestration—where platforms like Kubernetes, Docker Swarm, and Nomad manage the lifecycle of thousands of containers—the concept of a “restart policy” is a fundamental building block of resilience. A restart policy defines the automated behavior of an orchestrator when a container within a pod, task, or service exits or fails. It dictates the if, when, and how of re-launching a container, distinguishing between a brief hiccup (a crash loop) and a deliberate termination (a completed batch job).
Understanding restart policies is non-negotiable for Site Reliability Engineers (SREs), DevOps practitioners, and platform architects. Misconfiguring this single parameter can lead to cascading failures, runaway resource consumption, or silent data corruption. This article dissects the mechanics of restart policies, examines their implementation across major orchestrators, and explores their strategic role in designing self-healing, production-grade workloads.
The Core Mechanics: Behind the Orchestrator’s Decision
A container exists within a predefined lifecycle: it is created, starts, runs its primary process, and eventually exits. The orchestrator’s container runtime—be it Docker, containerd, or CRI-O—reports an exit code and an exit reason (e.g., OOMKilled, Error, Completed). The restart policy intercepts this event.
The decision tree is algorithmic:
- Exit Code Evaluation: Is the exit code
0(success) or non-zero (failure)? - Policy Context: Was the container intended to be long-lived (e.g., a web server) or ephemeral (e.g., a cron job)?
- Backoff Calculation: If a restart is required, how long should the orchestrator wait before attempting it to avoid a continuous restart storm?
Restart policies operate at the pod level in Kubernetes and at the service level in Swarm. They do not control the entire node or cluster but govern the immediate containerized unit. This local scope is crucial: a restart policy cannot fix underlying hardware failures or network partition events; it only handles the container process itself.
Implementation Across Major Orchestrators
Kubernetes: The Tri-Policy Model
Kubernetes offers three explicit restart policies, defined in a Pod’s spec (spec.restartPolicy). They apply to all containers within that Pod.
1. Always (Default for Deployments and StatefulSets):
The container is restarted regardless of its exit code. This is the standard for web servers, API gateways, and daemon sets. The orchestrator assumes the process must run continuously. If a Go binary panics and exits with code 1, the kubelet will restart it. If the container runs a one-shot script that exits 0, it too will be restarted—potentially causing an infinite loop. This makes Always unsuitable for Job-based workloads.
2. OnFailure:
The container is restarted only when it exits with a non-zero code (indicating an error). If the process exits cleanly (0), no restart occurs. This is ideal for batch processing jobs, data migration tasks, or any workload that should run to completion exactly once. A common pitfall: if the container crashes repeatedly, Kubernetes will increase the backoff delay (exponential backoff, capped at 5 minutes), eventually pausing restarts pending manual intervention or a Pod eviction.
3. Never:
No restart occurs, regardless of the exit code. This is used for strictly ephemeral operations, such as a container that performs a single calculation, sends a result to a message queue, and exits. It is also the default for Kubernetes Jobs. If a Pod crashes, the orchestrator marks it as CrashLoopBackOff or Error and does not attempt recovery. A higher-level controller (like a Job or CronJob) is responsible for creating a new Pod.
The CrashLoopBackOff State:
When a Pod’s containers repeatedly exit, the orchestrator transitions into CrashLoopBackOff. This is not a policy itself, but a protective state born from any restart policy. The kubelet applies an exponentially increasing delay (10s, 20s, 40s, 80s, up to 5 minutes) between restarts. This prevents a failing container from consuming CPU and memory cycles indefinitely, buying time for debugging. The state remains until the container runs for longer than the last backoff delay, at which point the backoff timer resets.
Docker Swarm: Service-Level Policies
Docker Swarm simplifies restart policies to service-level parameters. The --restart-condition flag accepts three values:
any(Default): The container is restarted under any exit condition.on-failure: Restart only on non-zero exit codes.none: Never restart.
Swarm also introduces --restart-delay, --restart-max-attempts, and --restart-window. These allow granular control: for example, a service can be instructed to attempt a restart up to 5 times, each delayed by 10 seconds, within a 5-minute window. After exhausting attempts, the service is marked as shutdown (in Swarm mode) or the task is considered failed.
HashiCorp Nomad: The restart Stanza
Nomad offers the most flexible model via its restart stanza in the job specification. It separates the restart policy from the reschedule policy—a critical distinction.
- Interval and Attempts: You define a number of restarts (
attempts) within aninterval(e.g., 3 restarts in 5 minutes). - Delay and Mode: A
delayparameter sets the time between restarts (e.g., 15 seconds). The mode can be"delay"(fixed wait) or"fail"(immediate stop if threshold exceeded).
A key Nomad feature is its reschedule policy, which is distinct from restart. If a task restarts but fails again after the restart threshold is hit, Nomad can reschedule the entire allocation onto a different client node—a higher-level recovery that Kubernetes achieves through replica sets and node controllers. This separation forces operators to think: “Do I want to fix the process in place (restart), or move it to new hardware (reschedule)?”
Restart Policies in Practice: Workload-Specific Strategies
The choice of restart policy directly correlates with a workload’s lifecycle and criticality.
Stateless Web Services (API, Frontend):
Use Always. These are designed for continuous operation. A crash is assumed to be transient (e.g., a memory spike). The exponential backoff in Kubernetes prevents rapid thrashing. Combine with a readiness probe that prevents traffic from reaching a freshly restarted container until it is healthy.
Batch Jobs and Data Processing:
Use OnFailure or Never. A job that aggregates 1TB of data and crashes mid-way should restart (to resume from a checkpoint), whereas a job that completes successfully should not be re-run. For strictly idempotent jobs, Never is safer—the orchestrator will not attempt recovery, and a higher-level controller (e.g., a Kubernetes Job) will handle retries at the Pod level.
Stateful Workloads (Databases, Message Queues):
This is the most nuanced scenario. Using Always can be dangerous. If a database container crashes due to a corrupted data file, an immediate restart may re-enter crash loop (mounting the same corrupt volume). Best practice is to combine OnFailure with a sophisticated init container or a sidecar that validates storage health before the main process starts. Some operators prefer Never for stateful workloads, relying entirely on operator-level controllers (e.g., Strimzi for Kafka, Zalando’s Postgres Operator) that perform health checks and controlled failovers.
DaemonSets (Logging, Monitoring Agents):
Use Always. These agents must run on every node. If they fail, they should be restarted immediately. However, to avoid a crash loop consuming node resources, pair the restart policy with a resource limit and a terminationGracePeriodSeconds that allows agents to flush buffers before being killed.
Common Pitfalls and Anti-Patterns
Even experienced operators misconfigure restart policies in ways that erode system reliability.
1. The Silent Infinite Loop (Always + Short Jobs):
A container that runs a quick script and exits cleanly 0 will be restarted endlessly under Always. This consumes CPU cycles and leads to confusing log volumes. The restartCount in kubectl get pods will climb indefinitely. Solution: Use OnFailure for jobs, or ensure the script exits with a non-zero code to signal completion if you must use Always.
2. Ignoring Backoff in Monitoring:
Alerts fired for CrashLoopBackOff are often ignored as “expected recovery.” However, if a container is stuck in backoff for 5 minutes (the max delay), the service experiences a 5-minute outage. Monitoring systems should track restartCount trends (e.g., 20 restarts in 1 hour) rather than just the state.
3. Overusing Always on Stateful Workloads:
A PostgreSQL container that crashes due to a disk full error will restart, immediately fail again (disk still full), and enter backoff. Meanwhile, the database is unreachable. The orchestrator will never resolve the underlying storage issue. A better approach is to use OnFailure and trigger an out-of-band alert on the first non-zero exit.
4. Nomad’s Restart vs Reschedule Confusion:
Operators new to Nomad often set high restart attempts (e.g., 10 restarts in 1 minute), thinking it improves resilience. This can hammer a failing node’s resources. Nomad’s reschedule policy is better for moving work away from a bad node; restart is for transient process failures.
Advanced Considerations: Probes, Termination, and Policy Interactions
A restart policy does not exist in isolation. It interacts critically with liveness, readiness, and startup probes in Kubernetes.
- Liveness Probe + Restart Policy: If a liveness probe fails, the kubelet kills the container and applies the restart policy. A probe failure is functionally identical to a process crash. Therefore, a faulty liveness probe (e.g., checking an endpoint that is slow but alive) can trigger unnecessary restarts.
- Termination Grace Period: When a restart is needed, the orchestrator sends a
SIGTERMand waits forterminationGracePeriodSeconds. If the process does not exit, aSIGKILLis sent. This hard kill can cause data loss for stateful workloads. A restart policy does not control this behavior; it only triggers after the process exits. - Init Containers: These are not governed by the Pod’s restart policy. If an init container fails, the Pod is restarted from the beginning (all init containers re-run). This prevents a scenario where a failed init container is left unchecked but the main container starts.
Relevancy to Container Orchestration Chaos Engineering
Restart policies are a cornerstone of chaos engineering experiments. Injecting a failure (e.g., killing a container) and observing how the orchestrator responds tests the entire recovery loop:
- Does the restart policy fire as expected?
- Does the exponential backoff delay protect the system?
- Does the downstream load balancer recognize the new container’s readiness?
A common chaos experiment is to set a container’s exit code to non-zero and verify that OnFailure triggers exactly one restart, while Always triggers continuous restarts. These tests validate that the policy aligns with the workload’s resilience model.
Ecosystem-Driven Evolution
The future of restart policies is shifting toward operator-level intelligence. Kubernetes operators (e.g., for Elasticsearch, Redis, MySQL) often override the generic Pod restart policy with application-specific logic. An operator might drain a database node before allowing a restart, or require a quorum of healthy nodes before restarting a crashed replica.
Meanwhile, serverless container platforms (AWS Fargate, Google Cloud Run) abstract restart policies entirely. They enforce a single model (restart on failure, with fixed backoff) and do not expose configuration to the user. This trade-off reduces operational complexity at the cost of flexibility.
Understanding restart policies is not merely a technical checkbox; it is a fundamental lever for controlling a system’s recovery behavior. The policy you choose defines whether a container is a phoenix (rising from any failure) or a canary (dying once to signal a deeper problem). In either case, the orchestrator executes that decision with algorithmic precision—provided you set the policy correctly.