Stability: Systems and Business Practice

Overview

• Stability has both a theoretical and a practical side.
  • Theory: The concepts and methods are relatively easy to learn.
  • Practice: The approach needs to evolve across different business models and system contexts.

Four Areas of Stability

1. Service stability (the system at rest)
2. Business stability (the business process at rest)
3. Change stability (during a transition)
4. Incident stability (during an emergency)

1. Service Stability

Definition: Keep a system operational through its architecture and resource management, without relying on detailed business knowledge.

Breadth: Observe traffic distribution across clusters, data centers, and individual nodes, including CPU, memory, and JVM behavior. Monitor middleware and databases at both cluster and host level, including resource use and performance.

Depth: Follow call chains through each layer and across upstream and downstream systems. Observe traffic trends, apply rate limiting, graceful degradation, and fault tolerance where needed, and use instrumentation to quantify performance and behavior.

2. Business Stability

Definition: Understand business tables, data relationships, and dependencies across systems.

Practice: Reconcile internal data to detect and prevent errors. Analyze upstream and downstream business calls to define alerts and severity levels. Monitor logs and exceptions to protect the continuity and correctness of business workflows.

3. Change Stability

Definition: Preserve availability and compatibility while a system changes.

Practice: Use a standard workflow that includes test reports, release reports, and explanations of core design decisions. Review plans and inform affected teams to reduce rollout risk. Maintain rollback and compatibility strategies so the change remains controllable.

4. Incident Stability

Definition: Respond and contain damage quickly, with established actions that do not require devising a plan during the incident.

Practice: Locate and resolve failures promptly. Prepare alternatives: replace a failed node, or use a standby cluster or traffic segmentation when cluster traffic behaves abnormally. Afterwards, analyze root causes and improve the response plan.

Reflections on Practice

• The theory is reasonably mature, but practical capability still needs continuous improvement.
  • Service stability: Understand the system's architecture and work with different business systems.
  • Business stability: Experience workflows and dependencies across systems.
  • Change and incident stability: Learn how different teams operate. Standard processes can add overhead, but in critical situations they can make containment much faster.
• Stability work should evolve with its business context and account for differences between systems and their operating procedures.