Friday, October 18, 2024

The Post Office Horizon system crash caused branches to disconnect from the data center.

The Post Office experienced a system outage on the morning of August 20, leaving 3,000 branch counters unable to log into the Horizon system. This issue was caused by a disconnect between branch systems and a Fujitsu datacentre. To resolve the issue, a backup datacentre was utilized to restore operations by 9:30am.

The Post Office was made aware of the problem at 6:45am and immediately started investigating with Fujitsu, Verizon, and DXC. By 8:00am, teams were working to determine the root cause of the outage. It was discovered that certain network reverse proxies were not passing traffic to the datacentre from branches. Traffic was redirected to a secondary datacentre, restoring service by 9:30am.

Despite the quick resolution, the Post Office faced criticism for lack of communication, with some subpostmasters claiming they did not receive notifications about the issue. The Horizon system has been under scrutiny following a recent documentary highlighting the Post Office Horizon scandal. Plans to replace the system are delayed until 2030, with Fujitsu likely to receive an additional contract worth up to £180m.

The Post Office is in the process of appointing a new technology leader and awaits government funding for the Horizon replacement project. The scandal, first exposed in 2009 by Computer Weekly, has been described as one of the biggest miscarriages of justice in British history.