Weird electrical problem after installing at client facility

Status
Not open for further replies.

jpswensen

Member
I recently delivered a consulting project I have been working on, but it isn't working as expected in the client facility. I had been testing this for months in my own shop and it was working flawlessly.

Background
This is in reference to the project I described in the blog submission page https://forum.pjrc.com/threads/62695-Laser-gates-for-measuring-sports-ball-velocity

Essentially I have a bunch of laser light gates, using the Teensy to capture the timing of the beam breaks and beam restorations to a very high accuracy. To synchronize all the gates, I have a common trigger pin on each of the four Teensy 4.1 boards that synchronizes the ARM_DWT_CYCCNT register to zero. This ensures that when the cannon fires the softball/baseball that the relative timing for each of the gates is synchronized and I can do the math/geometry to calculate the ball's incoming and rebound velocity.

This was working awesome in my own facility. However, when I moved to the customer facility, there appears to be a ton of noise/bouncing on this trigger pin and keeps resetting the count even in the middle of a shot.

Setup

I currently have a Raspberry Pi4 that is doing the computational side of things. It connects to each of the light gates over I2C and has a single pin dedicated to the common trigger. I will note that the cables from the RPI4 to the gates are each about 8 feet long, with shielded 4 strand cable. I had anticipated wanting to prevent any sort of bouncing, so basically added a debouncing capacitor, as shown in the image below. This circuit is in the main electronic box on the RPI side of the cables.

Circuit.png

I assumed that this would charge the cap when the RPI was driving the pin high, and discharge when driving the pin low, acting like a low-pass filter for any high frequency bouncing

Potential issues and attempted solutions

1. The customer facility has a ton of static electricity. They have artificial turf covering every inch of the floor and the amount of static buildup from even walking around is incredible. I have told the customer to zip-tie the cables up off of the ground, and today they are going to try and ground all of the metal frames of the equipment. They will give feedback on the addition of grounding to all adjacent metal today or tomorrow. Any other suggestions?

2. On the current firmware, I don't have that trigger pin set up as INPUT_PULLUP, but instead just INPUT. However, I would expect that with the RPI actively driving the pin high and low, that this wouldn't be necessary. Do you think this would this help?


I guess I am at a bit of a loss. In 1.5-2.0 months in my own shop, I never saw a false trigger. Now it is happening on about 50% of the shots of the cannon.

I am just trying to find a couple of potential things to try remotely before I fly out there with my scope and bag of tools to try and debug things. Any help would be greatly appreciated.
 
Any chance the lighting is interfering with the sensors?

I don't think that is the issue. The data that gets recorded looks like a long list of:
address,time,beam,bitpattern,

In a single readout I see that time (which comes straight from the ARM_DWT_CYCCNT register) get reset in the middle of a readout. This virtually guarantees the that problem is that my interrupt on the TRIGGER pin is happening more than once. The bit patterns looks correct in the sense of reporting a "1" when the beam is intact and a "0" when the beam is broken.

I suppose I could try something like a software debounce, but that doesn't give me confidence that this noise couldn't cause the trigger prematurely.
 
What exactly is it that drives this pin?

Do all of the components share a common ground?

Is the electrical circuit itself well-grounded? Is it noisy? Can you (and I'm being serious) run a long copper wire from the chassis and electrical grounds to the outside, solder it to a screwdriver, and drive it into the ground? (Okay, that's a last resort.)

What's the power supply like? Can you run it on a USB battery pack for the time being?

Is the equipment chassis plastic or metal? Can you stuff the sensitive electronics into a Faraday bag?

Do these shielded cables bring out their shielding to a pin, and if so, is that grounded?
 
Are the Raspberry Pi's GPIO pins set to maximum source and sink current drive capability?

You could put a RC filter on the teensy end.
 
What exactly is it that drives this pin?

This is driven by an RPI digital output. It is driving four different Teensy's, each of which is connected over about an 8 foot cable.

Do all of the components share a common ground?

Yes, there is a common ground bus. Each of the cables connecting the Teensy's back to the main electronics box have a ground wire connected to this ground bus. The RPi4 is also connected to this ground bus.

Is the electrical circuit itself well-grounded? Is it noisy? Can you (and I'm being serious) run a long copper wire from the chassis and electrical grounds to the outside, solder it to a screwdriver, and drive it into the ground? (Okay, that's a last resort.)
This is what I am asking the customer to do with the aluminum frame that the Teensy boards are connected to. I think the Teensy themselves are well-ground through their cables (each conductor is 22 gauge wire). However, I didn't have the aluminum frame to which those boards are attached grounded.

What's the power supply like? Can you run it on a USB battery pack for the time being?
The power supply is a high-end PC power supply. We are using the 12V to run various pneumatic valves, the 5V for the RPi4, the Teensy, and the laser emitters.

Is the equipment chassis plastic or metal? Can you stuff the sensitive electronics into a Faraday bag?
The equipment chassis is aluminum. The electronics box is about 8 feet away at a plastic table with the monitor and keyboard.

Do these shielded cables bring out their shielding to a pin, and if so, is that grounded?
I don't currently have shielded cable. I could have swore I bought shielded cable (I definitely intended to). I think I am going to try to remake the cables out of CAT6 twisted pairs. I will have the I2C twisted, the power/ground twisted, the trigger twisted with a ground, and the two remaining conductors connected to the aluminum frame and the cable shielding.
 
Have you plugged a circuit tester into the outlet to verify that the ground pin is actually grounded? If you don't have one of those, go to the panel and see if it's grounded. Typically there will be a ground wire clamped to a length of rebar (or similar) that has been driven into the ground, or the wire will be clamped to a municipal water inlet pipe, and newer construction will have more than one in different places.

Can you substitute a different RPi to see if maybe the digital pin has gone flaky?

Was all of the EMI-emitting stuff present in your lab? Is there any chance a source of EMI is inducing current in the suspect wires, possibly something unrelated to your devices? Have you tried wrapping the wire around a toroidal magnet, or some similar fix? (That last pushes the boundary of my understanding, so perhaps someone else can chime in about it.)
 
I would suggest that all the long wires be installed in standard EMT electrical conduit to shield against EMI & Static electricity. Make sure that both ends of the conduit(s) are grounded to a very low impedance ground (8' ground rod driven straight into the ground, or a piece of copper conduit layed horizontally in a 18" deep trench and buried with a #4 bare copper wire attached to one end with a brass clamp). I cannot emphasize strongly enough that when working around high EMI & Static electrical fields that proper shielding techniques such as the above should be used to protect low voltage control signals. The EMT will also shield fairly well from "H" fields as well as "E" fields which is especially helpful when using high speed triggering techniques.
 
Last edited:
This sounds an awful lot like ground loops and RF induced currents on the cabling. While shielding normally works great, you may just want to switch to something optical for the trigger. TOCSIS transmitter and receivers are fairly robust and the cabling can be purchased premade just about anywhere. (Optical head unit cabling) With shielding, you also can only ground one side, or you risk a ground loop. Sometimes a resistor is added to help drain off the excess. This used to be a really bad issue on security camera installs requiring the entire camera to be isolated to eliminate signal noise. (The joy of 1V PP signals.) Finally, if you have an idiot within a few hundred feet operating an illegally modified citizens band radio, nothing will work correctly except differential signaling or optical. (Run away from any job done near a truck stop)
 
Beside replacing the RPi, you can also try taking all but one Teensy out of the loop and seeing if the problem persists. Trying them in enough combinations may reveal that one of them has a flaky pin. It is not inconceivable that the astroturf has killed one of their pins. The rest of the advice herein might stop you from losing more in the future.

The tiniest thing can kill one of these pins. I lost an ADC pin on a 3D printer controller simply because I forgot to turn it off before disconnecting and reconnecting a thermistor. (There went 130 bucks.) Make sure you have a real ground; and wear a wrist strap connected to that real ground whenever you are handling the components, especially on site. It sounds like the place is functionally a giant Van de Graaf machine.
 
1. The customer facility has a ton of static electricity. They have artificial turf covering every inch of the floor and the amount of static buildup from even walking around is incredible. I have told the customer to zip-tie the cables up off of the ground, and today they are going to try and ground all of the metal frames of the equipment. They will give feedback on the addition of grounding to all adjacent metal today or tomorrow. Any other suggestions?

The cables are insulated, they can rest on the ground happily, seeing absolutely nothing from the static as they are shielded and insulated.

The problem from static would be if someone walks around and touches your sensor signal wires directly, otherwise its invisible to
low impedance circuitry, as there is essentially no current unless one charged body touches some other body at a different
potential.

Normally a sensor on the end of a cable would be isolated from the equipment it is installed on specifically to
prevent pickup of noise from a different ground. However if electrical noise is severe (welding equipment for
example, or heavy industrial motors), you have to defend in depth against noise, simple shielding only goes so far.

Typically industrial environments opt for 24V logic specifically as its much harder to corrupt a 24V signal than a 3V signal,
for instance.

Differential signalling is also an approach used for noisy environments (many industrial grade encoders use differential
signalling for ABZ signals). So perhaps RS485 for the trigger?
 
Status
Not open for further replies.
Back
Top