Problem: A major systems manufacturer had an intermittent problem
with the audio section of a new product. During audio testing a brief burst of
noise would sometimes be heard from the on board speaker. Although the noise
event seemed to be vary rare, it created a question about whether the system
was ready to ship. More information was needed about the frequency and
conditions of occurrence of this noise burst.
Solution: Tim Black designed an audio threshold alarm and laid
out a circuit board with twelve of these alarm blocks. Twelve of these boards
were built, creating a system of 144 channels of threshold alarms. These boards
were connected to systems on burn-in racks and data was collected while the
audio test was running. Connections from each alarm to a data logger was
provided for collection of a noise burst event log.
Time needed: 10 days.
Result: After seeing the results of testing, the product software
team made changes in the firmware to correct an error in a DMA trap. The test
logger showed that the problem had been fixed.
Problem: A small manufacturer of screws needed to collect
information on the quality of their product to implement a statistical process
control system. The product was a self-drilling sheet metal screw. A mechanical
jig had been built with a controlled force drill motor and a set of
micro-switches. The test was to measure the time that passed from when the
motor started turning the screw to when the screw had cut into the plate to a
pre-set depth. The problem was that the jig only had a mechanical clock timer
and operator error was interfering with the quality of the test results.
Solution: Tim Black built a small interface device for the serial
port of a PC. This device connected the micro-switches and air solenoids of the
test jig to a standard PC computer. He then wrote a program to run the
interface, make the needed measurements, and created a printed report with the
SPC information displayed as a standard chart.
Time needed: 7 days.
Result: Before the computerized solution, the operators needed
over an hour to create a test report and there was little confidence in the
result. Afterwards, the operators could test a batch of product in about 10
minuets with high confidence that the results were repeatable.
Problem: A major manufacturer of non-PC systems was surprised to
find that a new product would not work with some third party USB mice. A quick
survey of these mice showed that the problem only existed with new (at the
time) USB devices that could also connect to a PS/2 port. (a dual mode device)
It was critical to find out what was different about these mice and why the new
product did not work with them.
Solution: The makers of chips used in mice were consulted about
the conditions that showed the difference between a PS/2 and a USB port. Using
this information, Tim Black built a test jig to create different startup
conditions for a USB port. The data collected by this test jig showed that the
mice were choosing PS/2 mode and then not responding to the USB bus after this
incorrect startup mode was selected. The new product was not initializing its
USB port fast enough for the dual mode mice to make the correct decision about
what mode to select.
Time needed: 5 days.
Result: Firmware changes were made by the product team to correct
the startup conditions and the new mice then worked correctly.
Problem: A new chip was being designed with IP from existing
vendors. Simulation of one section had shown a possible problem when a certain
sequence of events happened on the I/O pins. The real part was asynchronous, so
it was unknown if the error shown in synchronous simulation was a real problem
or an artifact.
Solution: Tim Black designed and built a discreet logic sequence
generator to create the events in real time that caused the problem in
simulation. This generator was attached to a chip with the same IP to look for
the problem.
Time needed: 13 days.
Result: The IP test silicon did not show the problem, chip
development continued without this section causing undue concern.
Problem: A customer of a very well known RTOS suspected a failure
in the scheduling function of the kernel. The fault was very rare and had
eluded detection for years. It was unknown if the fault was really in the RTOS
or hidden somewhere in the customers code.
Solution: Tim Black created a test case that used external
asynchronous stimulation to create slowly varying timing conflicts in the RTOS
scheduler. After two days of operating, an attached logic analyzer captured the
fault.
Time needed: 7 days.
Result: Analyses by the kernel team found an interrupt disable
instruction misplaced by a single cycle. This was causing a race condition on a
single memory location that could cause a task to drop out of the scheduler if
an interrupt occurred in the single cycle where the vulnerability existed.
Moving this one instruction by one location fixed the bug.
