Touching usb cable to USB connector housing produces strange effect

So I have been trying to use a Teensy LC as the controller for a dual SN76489 synth. The SN76489 is a 1980's era 3 channel soundchip - 8 bit data bus plus a chip select and write enable line. I had breadboarded a single chip and oscillator and connected up to an Arduino Uno with a program to cycle through some notes which produced the expected outcome (i.e. different notes of the correct pitch).

So I soldered up my pair of SN76489's + oscillator on some strip board, wired everything together and got cacophonous garbage coming out of the speakers rather than gently cycling notes - if I listened hard I could occasionally hear the pattern of notes that I had programmed but basically it was swamped by randomish noise.

Now as the SN76489's need 5V's so I used one of those breadboard power supplies (like >this< ) to provide the 5V whilst the Teensy LC was powered over USB from the computer. Without the Teensy LC powered the circuit does what is expected (play a default tone - when a SN76489 powers up value in the internal registers mean a nice bass note is playing by default). As soon as the Teensy is powered up sonic anarchy is unleashed - even when it is programmed to simply silence the default tone, in fact, even whilst it is still sitting in a delay(4000) before it does any code that writes anything to the SN76489s.

After plugging and unplugging the USB cable a few times I discovered that simply touching the metal tip of the USB cable (still connected to my computer) to the metal housing of the connector on the Teensy would cause this to happen. So just to be clear the USB is not plugged into the Teensy, it is just touching the casing.

I then investigated a bit more and powered my Teensy LC not from the computer but from a USB power pack attached to the same plug bar as the power pack plugged into the breadboard power supply. This worked absolutely fine, I could touch the cable to the housing with no effect and when the Teensy was powered it sent the right messages without any random noise.

So my question is: Is this an obvious thing I would have known that would happen (Teensy sending garbage down the datalines due to different power sources) if I had some Electrical Engineering education or is this a puzzler?

EDIT: SN76489 logic high is at 2V if that is a useful piece of information

I'd wager it's the switch mode power supply in your CPU creating a nasty ground loop effect.

I've seen some circuits suggested out there to break ground loops for audio equipment so I'd start there. The simplest had a capacitor and two diodes, IIRC.