Principles
- Define the "steady state" — normal system behaviour (RPS, latency, error rate)
- Form a hypothesis: "if one Pod is killed, the system will keep running"
- Inject the failure in a controlled environment (start with staging)
- Compare the actual state with the hypothesis
Typical experiments
- Kill one microservice
- Add 2 seconds of latency to an external API
- Simulate a full database failure
- Fill the disk to 95%
Tools
Chaos Monkey (Netflix), LitmusChaos (Kubernetes), Gremlin, AWS Fault Injection Simulator.