Operator note / October 3, 2026

Leave room for the recovery path

A system sized only for steady traffic can become least reliable at the moment it needs to recover.

Retries are traffic too

When a dependency slows down, callers may retry together. Bound retries, add jitter and stop when the original deadline expires. A large queue does not create capacity; it can merely postpone the visible failure.

Memory includes the unglamorous parts

Leave headroom for connection buffers, operating-system caches, diagnostics and deployment tools. Watch sustained pressure rather than a single brief utilization spike. Resource limits should make overload controlled, not mysterious.

Document a small maintenance window

Before a change, note the rollback path and what proves success. Keep one observable request or small job as an acceptance check. In this fictional lab, those habits matter more than the attractive operational badges on the overview.