# Kubernetes Incident Response: A Practical First-Response Checklist for DevOps and SREs

Kubernetes incidents often get worse because the first response is too fast, not too slow. A pod starts failing, someone restarts it, deletes it, or rolls something back before collecting enough evidence.

A better first response is simple: identify what is failing, collect the right evidence, and choose the safest next action. This checklist is designed for that exact moment.  
  
1\. Confirm What Is Actually Failing

Before changing anything, identify the failing object and its scope.

Start with:

```plaintext
kubectl get pods -A
kubectl get deployments -A
kubectl get events -A --sort-by='.lastTimestamp'
```

Then narrow the incident down to:

*   the affected namespace
    
*   workload or Deployment
    
*   specific Pod and container
    
*   whether the issue affects one replica or all replicas
    
*   whether the failure started after a recent deployment, configuration change, Secret update, image change, or infrastructure event
    

## 2\. Collect Evidence Before Restarting Anything

Before restarting or deleting a failing Pod, capture the current state. A restart can remove useful evidence such as previous container logs, termination details, and the exact failure conditions.

```plaintext
kubectl get pod <pod-name> -n <namespace> -o wide

kubectl describe pod <pod-name> -n <namespace>

kubectl logs <pod-name> -n <namespace> --all-containers --tail=200

kubectl logs <pod-name> -n <namespace> --all-containers --previous --tail=200

kubectl get events -n <namespace> --sort-by='.lastTimestamp'
```

Look for:

*   container exit codes and termination reasons
    
*   restart count
    
*   probe failures
    
*   image pull errors
    
*   scheduling or volume issues
    
*   `OOMKilled`
    
*   recent warning events
    
*   errors in the previous container logs  
    

## 3\. Triage `CrashLoopBackOff` Systematically

`CrashLoopBackOff` is not the root cause. It only tells you that a container is repeatedly starting, failing, and being restarted.

Start with the previous container logs:

```plaintext
kubectl logs <pod-name> -n <namespace> --previous
```

Then inspect the Pod:

```plaintext
kubectl describe pod <pod-name> -n <namespace>
```

Check for:

*   application startup errors
    
*   incorrect commands or arguments
    
*   missing ConfigMaps or Secrets
    
*   dependency or connection failures
    
*   permission errors
    
*   `OOMKilled`
    
*   failing liveness probes
    
*   non-zero container exit codes
    

## 4\. `OOMKilled`: Memory Failure or Symptom?

When a container is terminated with `OOMKilled`, confirm the memory limit and actual usage before simply increasing resources.

Start with:

```plaintext
kubectl describe pod <pod-name> -n <namespace>
```

Then check the workload’s resource requests and limits:

```plaintext
kubectl get deployment <deployment-name> -n <namespace> -o yaml
```

If metrics are available:

```plaintext
kubectl top pod <pod-name> -n <namespace> --containers
```

Check for:

*   memory limit set too low
    
*   sudden traffic or workload increase
    
*   memory leaks
    
*   large caches or in-memory processing
    
*   recent application or configuration changes
    
*   one container consuming most of the Pod’s memory
    

## 5\. Build a Repeatable First-Response Workflow

The goal during a Kubernetes incident is not to memorize every possible failure mode. It is to follow a repeatable process that reduces guesswork and protects useful evidence.

A simple workflow is:

1.  Confirm what is actually failing.
    
2.  Capture evidence before changing anything.
    
3.  Identify the failure pattern.
    
4.  Choose the smallest safe next action.
    
5.  Escalate when the evidence points beyond the workload layer.
    

For engineers who want a compact reference during an active incident, I created the free **OPSFORGED Kubernetes Incident Starter Kit** with first-response checklists, quick triage references, useful `kubectl` commands, and a simple incident decision flow.

**Free starter kit:**  
[`https://github.com/OPSFORGED/opsforged-kubernetes-incident-starter-kit`](https://github.com/OPSFORGED/opsforged-kubernetes-incident-starter-kit)

## Want the Full Visual Troubleshooting Guide?

The free starter kit is intentionally compact and designed for fast first response.

For engineers who want a deeper, more structured reference, the **OPSFORGED Kubernetes Incident Troubleshooting Playbook — Premium Visual Edition V1.3** includes broader incident coverage, visual troubleshooting flows, production-oriented guidance, and a more complete decision-making framework.

It is designed for DevOps engineers, SREs, cloud engineers, and Kubernetes learners who want a practical reference they can return to during real troubleshooting.

**Explore the full visual playbook:**  
[`https://opsforged.dev/`](https://opsforged.dev/)

**Good incident response is not about reacting faster. It is about collecting the right evidence and making the safest next decision.**