The Quirkiest Software Bug: A Tale of Mechanical/Magnetic Interference and Component-Level Troubleshooting

Throughout my career as a software engineer, I've encountered numerous bugs that have kept me on my toes. However, one particular incident stands out as the most peculiar and intriguing. This story revolves around a mysterious issue with an IBM ATM that seemed to be jamming even when there was no actual jam, and how an unexpected tweak resolved the problem.

A Daily Mismatch of Sensors

Back in the 1990s, an IBM ATM at a local bank had a peculiar issue. Every morning at 6 AM, the machine would report a cash jam in the escrow, even though no jam was present. At the close of the previous day, everything was normal, but as soon as the operator arrived and checked the ATM around 8 AM, no jam was found. This inconsistency was initially attributed to faulty sensors, but even after changes in sensor voltage and other standard checks, the problem remained persistent.

The anomaly was more perplexing when day and night savings time changed. The malfunction didn't just shift, it happened one hour later. It wasn't until an engineering team member observed the issue firsthand that the cause was identified. At 7 AM, the morning sunlight shining through a shop window reflected directly into the ATM's delivery slot, fooling the sensor and causing it to incorrectly report a jam. Just as surprising as the issue, was the solution - adjusting the sensor's position to block out the reflected sunlight.

The Intricacies of Document Processors (Sorters)

Let's shift our focus to a different but equally fascinating piece of software engineering history - document processors (AKA 'sorters'). These large-scale machines, which could be forty feet long, six feet high, and four feet deep, were crucial in the banking industry for decades. They read, processed, and physically sorted millions of checks daily, with large trays of checks traveling around on wire racks, processing thousands per shift for 24 hours a day.

These machines relied on magnetic ink capable of being read via both optical and electrical systems - MICR (Magnetic Ink Character Recognition) and OCR (Optical Character Recognition). One particular challenge I faced as a young field engineer was the malfunction of the spray endorser on a brand-new installation, even though it was bought directly from the factory.

For months, the document processor sat idle on the production floor as the cause couldn't be identified. Legal battles ensued over the unutilized machine, which was incredibly expensive to replace or render operational. I was the new hire tasked with investigating this issue.

A Component-Level Discovery and Fix

Multitudes of circuit boards controlled the spray endorser, part of a complex backplane housing numerous integrated circuits, PLLs, oscillators, and other vital systems. Despite everything else working perfectly, the endorser refused to function. Through meticulous troubleshooting, I found that the issue stemmed from a few incorrect wires on the backplane, which were wrapped on the wrong pins, an error that misled the engineers who worked on the machine.

To resolve this, I had to disassemble the motherboard and manually rewire the pins. After swapping the wires to their correct positions, the spray endorser functioned flawlessly. This resolution not only saved the project but also restored the company's image with the customer.

While the technical challenges were significant, the career trajectory post-resolving such a complex issue was less favorable. A global workforce reduction led to the shutdown of Burroughs, and shortly after resolving the document processor issue, I was laid off, despite the success of my endeavors.

To those interested in software engineering, these stories illustrate the importance of meticulous troubleshooting, the unexpected ways in which issues can arise (in this case, from sunlight reflection), and the critical role of physical components in software systems. The lessons learned from these unique cases are valuable for anyone working in the field.