Chris' Tale
Ah yes, we lived in interesting times back then. Would a perfectly working hard drive be marked as "failed"? Would a completely dead hard drive be left in the array? Would any data you wrote to an array still be readable the following day? These were the questions we asked ourselves nearly every day. Invariably the answers were yes, yes and occasionally (to be fair, on rare occasions these devices did actually do what they were supposed to do, just not for very long.)
How well I remember pacing the floor of a surprisingly unexciting server room for several hours on what had up until not long before been quite an enjoyable weekend, waiting for one of these devices to finish rebuilding itself (no, it couldn't be done remotely - the serial interface used to crash as well, so you had to be physically present to press buttons on the front panel.)
Why did so may drives fail? We never found out. Why did the arrays not notice the failures? We were never given a plausable reason for that either. Were we glad to see the back of these devices? Oh, yes.