British Airways IT Outage

British Airways' data center in London experienced a power failure. The primary power distribution system failed, causing loss of power to critical IT systems.

Backup systems worked as designed. Battery backup and diesel generators kicked in, maintaining system operations during the initial failure. For a short time, everything worked correctly.

Once the primary power was restored, the recovery sequence began. Power was gradually restored to systems in a specific order. However, the sequence was not coordinated with the recovery needs of all dependent systems.

When certain systems powered back on, they attempted to access systems that were still offline. This created cascading failures as systems tried to boot or recover in an undefined state.

Some systems developed corrupted states during the restoration sequence. Others developed file locks or resource conflicts. These corruption states persisted even after full restoration.

What should have been a few minutes of recovery took many hours. The restoration sequence itself had created new failure modes that required manual intervention to resolve.

The backup systems were tested and worked well. The restoration sequence was not tested as thoroughly under real failure conditions. The dependencies between systems during recovery were not fully mapped.

The assumption was that if power is restored and systems boot, they will return to normal. In reality, restoration is not the mirror image of failure. The sequence matters. The timing matters.

Disaster recovery testing had focused on failover. Testing had not focused on the restoration sequence itself. This sequence had never been run end-to-end under realistic conditions.

Do you test recovery sequences as thoroughly as you test failure? Do you understand the timing and ordering dependencies when systems come back online? Have you tested whether your backup and restoration systems introduce new failure modes?