Third Party Software and Horizon

Using Third Party Software in an application under development offers many potential benefits: speeding up development, better quality (from incorporating a proven product), probably also reduced application cost.

On the other hand, using Third Party Software in an application under development gives away control of that part of the application, both control over future application enhancements and the ability to fix application defects (unless the source code is purchased). Lack of visibility of the inner workings of the procured component may also turn out to be a significant drawback.

NATS use of flight planning software supplied by Frequentis Comsoft worsened the impact of the UK air traffic control system outage in August 2023. NATS engineers required the assistance of the supplier to interpret the system error message and that, along with other factors, delayed the fix and adversely affected many flights and passengers.

Horizon and Riposte

The Post Office Horizon system developed by ICL/Fujitsu also included Third Party Software. ICL's bid proposed using Escher's Riposte software which was already in use by the Irish Post Office An Post. Riposte provided both a user interface and transaction handling (Riposte Desktop) along with message handling software (Riposte Message Handler) to synchronise branch accounts with a Data Centre. Escher and An Post were part of the consortium (called Pathway) led by ICL on this contract bid.

Escher today is a software and services company based in Boston USA, with offices elsewhere in the world including Dublin, specialising in Post Office systems which Escher it has sold to several countries.

Jeremy Folkes, who worked for the Post Office from 1987 to 2000, gave evidence to the Post Office Horizon IT Inquiry on 02/11/2022. Mr Folkes was involved in the evaluation of the Horizon bids, initially helping to reduce the shortlist of bidders to three (ICL Pathway, IBM, CardLink) and then undertaking more detailed assessment of elements of the ICL bid. In 2000 Mr Folkes left the Post Office and joined a company that was soon after acquired by Escher. He went on to take up several senior posts at Escher including CTO, retiring in 2021. Given that background, his evidence was of interest.

Commenting on his work for the Post Office, Mr Folkes said:

"The Riposte system, if I can just digress for a moment, involves software running on each one of the terminals in the office and these terminals, they can replicate data between each other and replicate data up to the data centre and that's the whole benefit of this and the whole way the system operates. We, I think, had come up with questions [for ICL Pathway] about failure cases, what would happen if that became disconnected, or that became disconnected?"

Given that subsequently there were several examples of accounting errors in Horizon that arose because network failures were not properly handled by Horizon, these were appropriate and prescient issues to raise; but the Post Office were not getting much help from ICL/Fijitsu as Mr Folkes revealed:

"When we started raising issues on failure conditions they [ICL] seemed to be keener [to] rubbish the specific examples we put forward rather than trying to address it."

Clearer answers were given to the Post Office when Escher staff answered some of those questions, as they obviously knew much more about the workings of the Riposte Message Handler than did ICL engineers.

Once ICL Pathway had been awarded the contract, Horizon was first released to Post Office branches in 1999. Before that initial release and during subsequent releases, the front-end software was modified by ICL/Fujitsu and by Escher to meet the specific needs of the UK Post Office. As another Inquiry witness Andrew Simpkins, who worked as a consultant for the Post Office during acceptance testing of Horizon, stated about the initial decision to base Horizon on the An Post functionality:

"But then you find that it [Riposte] is not really quite the right product that is needed for this particular clients' [Post Office's] requirements."

That was not the first time that issue has arisen when using Third Party Software in a bespoke application development for another client.

A few of the accounting errors that affected subpostmasters were attributed to defects in the underlying Riposte Message Handler software, for which ICL did not have the source code. One problem, known as the Callendar Square bug, could cause accounting errors under several circumstances such as: if a subpostmaster believed that a transaction had failed and reentered it; or if assets were being transferred between terminals at a branch. The failure symptoms were first reported in 2000 but the root cause, a failure to release a lock, was not identified as a Riposte issue until 2005 and only fixed by Escher in 2006, six years after it had first been identified as a software problem.

In his evidence to the enquiry when discussing this bug, former Fujitsu engineer Gareth Jenkins observed:

"In general, Escher would not accept any defect unless it could be reproduced, and this was one we couldn't reliably reproduce."

It would not be surprising if there were several finalised Horizon software fault report records at Fujitsu with a closure reason "Unable to reproduce" or equivalent.

From the start, Fujitsu assigned some engineers, including Mark Jarosz, to be their interface with Escher about Riposte, particularly the Message Handler component. In his evidence to the Inquiry Mr Jarosz remarked that:

"The Riposte product logged lots of error messages and there was no documentation which said what this error message means and what the consequences could be."

In a major update to Horizon in 2010 Fujitsu completely replaced the Riposte Message Handler software. Riposte was by no means the main source of Horizon accounting errors in the period up to 2010, nor did accounting issues cease after 2010, but ICL's initial decision to use Third Party Software can be seen to have had an impact on the time taken for the supplier to fix faults affecting subpostmasters' accounts.

All software could contain latent faults. Faults can also be introduced if developers do not fully understand how Third Party Software really works and how to interface to it. Using Third Party Software in a bespoke application can save developers considerable time and money, but there are potential risks.

See here for other articles on Horizon and its implications.