What was the hardest problem you ever found and fixed?

[Joe L]

It wasn't a guitar amp, it was an audio amp in a two way radio.

We have about ten techs in my department and half of them had tried to fix this radio before it got sent to me. The problem was it was self generating rf which was feeding back thru the antenna input. So it was a chicken and egg thing. The signal was present all the way from the input to the audio output which was squeeling. I spent on and off a couple of hours a week for four months trying to fix that beast.

Finally, I hooked a loop of wire to an rf spectrum analyzer and "sniffed" until I found a hot spot on the board. It was in the audio section. I replaced the silicon devices and had no luck. I removed the caps and tested for value and leakage with a voltmeter. Still no luck. Finally I started replacing caps at the hottest (rf wise) spot and the first one worked!!!! WTF? I checked the one pulled with 3 different meters and found no problem. Put it back in and the oscillation came back.

Found and fixed. The other guys that worked on it bowed before me

[drewl]

Sometimes fresh eyes or a different perspective is needed for oddball problems.

Most of the hard stuff I deal with is work related telecom stuff, but back in the 80's I remember trying to repair the horizontal amp on a Tektonnix 475 oscilloscope at work that was driving me nuts.
Believe it or not, I saw the problem in a dream, woke up, went to work and fixed it.

[loudthud]

Since drewl brought up Tektronix, I remember when I worked on one of Tek's first digitizing scopes, the 7854. The scope display would randomly go into some wierd spacey mode. Nobody, not even the engineers could find the cause. This was not just one scope, many of them would do it if they got the chance. One day I was going through the calibration procedure, at the point where you had to manually change the display mode from the scope's keyboard. I entered 1 0 2 4 {shift} P/W (parts per waveform) and the scope immediately went into the spacey mode. I said "It's not supposed to do that!" So I entered a little program to loop on that command and within a minute the scope would go into that mode and all the other scopes would too. The lead tech knew how to get into the firmware and found that the interupts were not properly masked off when that command was being executed. The end of display interupt was not getting serviced so a new display cycle could not begin. The firmware had to be patched. That was when I learned the difference between something that works and something with a hidden bug.

The strangest music related fix was on a Shure PA amp. The power amp was always in protect mode. Several techs tried to fix the power amp replacing most of the transistors. The bigest obstacle was that you had to plug two or three boards together before you could test the thing and the one transistor that seemed to be the problem was deep inside this bundle of boards forming a cooling tunnel. The problem turned out to be leakage on the base of that transistor caused by solder flux trapped between one of the board to board connectors and the PCB.

[defaced]

First thing I remember fixing was our telephone as a kid. It was the old style with a bell which was hit by a hammer/lever assembly. For some reason it stopped ringing. Once we got it open, no one could figure out why it wouldn't ring. I rocked the lever/hammer assembly back and forth several times, and presto, the phone rang. I was about 4 years old at the time and was amazed that I figured out how to fix something.

Recently, I figure out how to calibrate rotational speed on a multi million dollar welding system. Everyone was telling me I was nuts that the rotational speed was off. Me being so willing to submit, kept to my guns and kept turning over rocks looking for answers. After discussing this problem with one of the techs for the equipment vendors, we decided it was a difference in the logic level of the machines. The commanding machine treated 9.4v as max rotational speed, the receiving machine treated 9v as max speed. Once we got that figured out, it was trial and error to get things dialed in by changing the gain and offset of the digital to analog converters of the commanding machine. During this, we figured out that the readout from the variable freq motor drive could be used as an accurate indication of rotation speed which saved us from doing hour long duration tests for a 2x2x3 matrix of rotational speeds. That tiny battle was easily two months of my life because no one would believe me that the speed was wrong to begin with.

[Joe L]

Since drewl brought up Tektronix, I remember when I worked on one of Tek's first digitizing scopes, the 7854. The scope display would randomly go into some wierd spacey mode. Nobody, not even the engineers could find the cause. This was not just one scope, many of them would do it if they got the chance. One day I was going through the calibration procedure, at the point where you had to manually change the display mode from the scope's keyboard. I entered 1 0 2 4 {shift} P/W (parts per waveform) and the scope immediately went into the spacey mode. I said "It's not supposed to do that!" So I entered a little program to loop on that command and within a minute the scope would go into that mode and all the other scopes would too. The lead tech knew how to get into the firmware and found that the interupts were not properly masked off when that command was being executed. The end of display interupt was not getting serviced so a new display cycle could not begin. The firmware had to be patched. That was when I learned the difference between something that works and something with a hidden bug.

In the mid 90s I started a home business writing shareware and spent many hours chasing a solution to a problem so I can understand what you were up against. I was usually pretty quick in finding solutions but there was three or four all weekenders.

The strangest music related fix was on a Shure PA amp. The power amp was always in protect mode. Several techs tried to fix the power amp replacing most of the transistors. The bigest obstacle was that you had to plug two or three boards together before you could test the thing and the one transistor that seemed to be the problem was deep inside this bundle of boards forming a cooling tunnel. The problem turned out to be leakage on the base of that transistor caused by solder flux trapped between one of the board to board connectors and the PCB.

That is why I don't work on cars. I hate with a passion having to dig deep just to get to the problem and then reassemble it just to see if its fixed. That is why I haven't done much work on modern music electronics.

[Steve Conner]

The hardest problems I've dealt with lately have all been software. I recently designed an instrument with two processors: an application processor and a DSP that talked to each other over a parallel bus. I spent ages checking the timings on the bus, and our prototype worked great. When it came time to make the first batch, I got a call from the factory saying that none of them would even boot.

I had carefully calculated the bus timings for a DSP clock rate of 300MHz, and run tests for hours with no problems, but it turned out that the DSP actually ran at 50MHz until it finished loading its firmware, so I was violating it by a factor of 6. I never noticed before because the prototype had a separate boot ROM for the DSP, so it got going and reached its final clock rate before the main processor got round to talking to it. But in the production system, the DSP firmware was to be loaded over the very same bus that wouldn't work until the DSP firmware had loaded.

Once I figured this out, the cure was cheap and nasty, use the power management API to slow the main processor down until it had finished booting the DSP.

[Mark Black]

Our company handled the installation of a sound system in a newly-rebuilt restaurant. The owner kept telling us there was a hum coming from some of the 70V system ceiling speakers when the club was closed & very quiet inside. I finally got in there during one of those quiet times and sure enough - there was a low level hum, with the amplifiers off! Turning on the amps actually decreased the hum, and disconnecting the speakers from the amp made it a little worse.

Turned out there was some HVAC equipment near the far end of a string of speakers which was vibrating the ceiling and causing that last speaker to work like a crude microphone, driving all its buddies in the same string. Turning on the amp and having its protection relay energized allowed the amp output section to act as a damper. Pretty funny actually.

Many years before that and in a previous profession I dealt with a Suzuki twin (450 I think) that would run fine one direction down a road but would do poorly coming back. That one was eventually revealed to be a missing plastic side cover on one side (which we hadn't given much thought since we saw it all the time on other bikes). The breeze blowing somewhat perpendicular to the road would disrupt the flow of air into the carbs depending on which direction you were headed. That one was just plain goofy.

[defaced]

Many years before that and in a previous profession I dealt with a Suzuki twin (450 I think) that would run fine one direction down a road but would do poorly coming back. That one was eventually revealed to be a missing plastic side cover on one side (which we hadn't given much thought since we saw it all the time on other bikes). The breeze blowing somewhat perpendicular to the road would disrupt the flow of air into the carbs depending on which direction you were headed. That one was just plain goofy.

Reminds me of one of my boss's favorite troubleshooting stories. When he got his 73' Porsche 914, he took to a shop that was known as specializing in Porsche to have it checked out. The guy gave it a good bill of health except for this annoying problem where it would want to stall when taking corners. He threw in the flag and told my boss what was going on. My boss, who like all pilots are trained to quickly recognize problems like this because there's no shoulder to pull over on when you're in the air, replied "Stalls when you take a corner? You're sucking up water in the gas tank, put some dryer in it". The Porsche guy didn't really like hearing this but sure enough, putting dryer into the gas cleaned it up. Another example that demonstrates no mater how good you think you are, we all have something to learn.

[Rhodesplyr]

I got all the oscillators and dividers (and everything else) working on Italian-made Vox Super Continental combo organ. That was probably one of the most challenging repair jobs I've undertaken. The trickiest part involved finding a bad resistor inside one of those Packaged Electronic Component modules.

[Gtr_tech]

I have a Memorymoog in for service/restoration right now. I'm sure that will be among the most challenging projects I've seen in a while.

[Prairie Dawg]

I worked in the aircraft engine field in Michigan in the eighties, and a customer with a Mitsubishi MU2 called in from Willow Run that one engine wouldn't schedule any forward pitch. You could reverse all day long but no forward pitch. So a coworker and I went down to Ypsi and ended up taking the engine off and bringing it back to the Zoo. The engine on them is tightly cowled and a real bitch to get at anything particularly on the back side of the engine. I took the prop governor off and sent it to Woodward for a bench run and they gave it a clean bill of health. Nothing on the schematics of the oiling system showed anything-until a late night with the parts manual showed a small check valve in one of the oil passages to the governor. I fabricated a piece of threaded drill rod about 3/32 diameter with a knocker on the end and pulled it out because it was swedged in place. That valve was stuck. I knew I had it knocked. So I called a friend of mine in the technical service field at Garrett and described the problem and he says "Oh yeah. Change that check valve underneath the governor-the one that isn't on any of the schematics." It went back together and worked as advertised.

Three weeks later I worked on a Fairchild Metroliner that had exactly the same problem. I'd had the parts man order three valves, so we had one in stock and repaired his airplane in about three hours, with time out for coffee. I had never seen that problem in the 10 years I worked on Garrett engines both in the field and at the manufacturer's facilities. Twice in three weeks.

[Enzo]

My favorite story was a Yamaha MT44 four track cassette deck. My junior tech was working on it, it would not record tracks 3 and 4. I joined him to help and learn. I said find out what turns on the record light. Some transistor, and it was not being driven. OK, what turns on THAT transistor? Some other one farther back. No control there either. SO what turns THAT on? We went back. Each time, I asked where the stage got its control, and he found it, and we would find it was working. FInally we got to one point, "OK< what controls THAT?" And we found it was an optical sensor. Where? On the deck itself between the reels.

We had troubleshot (troubleshooted?) the problem completely through the unit and came out the other end. We then found out that the machine acts as a normal two channel cassette deck unless you put a little reflective sticker on the back of the cassette. And that explained the mysterious sheet of reflective stickers we found in the service manual file. The sticker reflected an LED into the sensor and enabled the extra channels to record.

The tech was upset, "We just wasted an hour." I said no, we just verified our troubleshooting technique works, we found the problem. And we learned something.

[Joe L]

Seeing all these weird problems, you think it takes balls to strap on a rocket and go to space?

Ten days left until retirement .