Partial Failures in Distributed Systems

Key Insight

Unlike programs on a single computer that are typically either fully functional or entirely broken in a deterministic way, distributed systems exhibit partial failures. This means some parts of the system can fail in unpredictable ways while other parts continue to operate. This fundamental difference introduces nondeterminism, making distributed systems inherently more complex to design and manage.

The physical reality of distributed systems contrasts sharply with the idealized, mathematically perfect system model presented by a single computer. For instance, a CPU instruction always yields the same result, and data written to memory or disk remains intact. In distributed environments, however, real-world events like a hypoglycemic driver crashing into a data center's HVAC system, PDU failures, or accidental power cycles of entire racks can cause unpredictable partial failures.

These partial failures are nondeterministic: an operation involving multiple nodes and the network may sometimes work and sometimes fail without warning. A sender might not even know if a message was delivered, nor the recipient's state. This unpredictability and the possibility of partial failures are the core reasons why distributed systems are challenging to work with, requiring engineers to build in fault-tolerance mechanisms to ensure system functionality despite widespread issues.