Forum Rule: Always post complete source code & details to reproduce any issue!
Page 1 of 2 1 2 LastLast
Results 1 to 25 of 26

Thread: EMI Affects Teensy 3.6 More Than 3.5 and 3.2

  1. #1
    Member randomvibe's Avatar
    Join Date
    Mar 2016
    Location
    CA
    Posts
    79

    EMI Affects Teensy 3.6 More Than 3.5 and 3.2

    My project involves a kickstarter Teensy 3.6 connected to two SPI devices: (1) an LSM6DS3 IMU sensor, and (2) the ILI9341 display. The Teensy also drives a pair of 3A stepper drivers. The drivers are powered by the very inexpensive "Dr. Meter DC Power Supply PS-305DM" (30V, 5A).

    The display SPI line is a 10 inch CAT5 internally-shielded cable. The IMU SPI line is a ~4 foot CAT6 wire-mesh-shielded cable. The stepper drivers are optoisolated. The Teensy setup ground plane is completely isolated.

    Here's the problem. When the stepper drivers are unpowered, the Teensy 3.6 setup is rock solid. No IMU dropouts. No display problems. When the drivers are powered, the display intermittently scrambles the text. This is undoubtedly an EMI problem. I traced the issue to the power supply line, which is very long (~12 ft) and rolled-up on spool just a few inches from the Teensy setup. When I unspool the supply line and move it several feet away from the Teensy 3.6, it improves the situation, but the display still scrambles ocassionally. Reducing the CPU speed does not help.

    When I swap in a Teensy 3.2 and leave all the system wiring in place, the setup is rock solid. Even if I spool-up the supply line near the setup, it's still rock solid. But my project needs more pins.

    In desperation, I bought a Teensy 3.5 at Microcenter yesterday. When I swap it in, it's rock solid when the the supply line is unspooled. It does not need to be several feet away. However, if I spool it up right next to the display, EMI scrambles the display.

    The kickstarter Teensy 3.6 appears to be very sensitive to EMI. Teensy 3.5 is definitely more stable. Teensy 3.2 is rock solid. Does this make sense to PRJC?
    Last edited by randomvibe; 01-04-2017 at 09:41 AM.

  2. #2
    Senior Member
    Join Date
    Jan 2014
    Posts
    136
    Would first look at susceptibility to power source perturbations, as the T3.5 does seem to be a bit more particular about power quality.

    Have done some IEC61326 and CISPR24 stuff on T3.1,3.2, and will try to do some regression series next month using a T3.5 when my employer sends me to the EMC lab next month. FWIW, T3.5 CE and RE are about 2.5dB less margin compared to T3.2, but this stuff is very design-dependent so may have no meaning for your widget.

  3. #3
    Senior Member+ KurtE's Avatar
    Join Date
    Jan 2014
    Posts
    3,690
    Sorry if I am completely off here, but the other thing I would wonder about is with Teensy 3.2 and 3.5 you are 5v tolerant, but with the 3.6 you are not.

    So I would also check to make sure you are not trying to feed any signals into 3.6 that are over 3.3v...

  4. #4
    Senior Member
    Join Date
    Dec 2016
    Location
    Montreal, Canada
    Posts
    2,802
    ** including spikes

  5. #5
    Senior Member
    Join Date
    Jan 2015
    Posts
    111
    @rv, what you have is a "systems-level" problem. Any time you have microcontrollers, motors, and long wires together, you have a problem that has befuddled dozens of embedded systems projects over the years. There are many many things people have done to try and bullet-proof their systems, it takes more than a single "fix" to do the job in general. You should do some background research in embedded systems design.

    For one thing, long wires make for inductive loops which act as both broadcast and pickup antennas. These things are the absolute bane of embedded systems, so you need to start there. You need small loops and lots of filter caps in strategic locations. And you should look up something called "star grounding" and/or star power distribution, and ground loop problems. This website has some really good info,
    http://www.4qdtec.com/pwm-01.html#cap

  6. #6
    Member randomvibe's Avatar
    Join Date
    Mar 2016
    Location
    CA
    Posts
    79
    Quote Originally Posted by oric_dan View Post
    @rv, what you have is a "systems-level" problem. Any time you have microcontrollers, motors, and long wires together, you have a problem that has befuddled dozens of embedded systems projects over the years.
    I agree it's a system level issue, but it's still curious that the Teensy 3.2 is rock solid when swapped-in for the 3.6 while keeping the rest of the system unchanged. So the breadboard & jumper layout was unchanged, the SPI cables were unchanged, the location of the breadboard, CAT5 lines, power supply on my desk was unchanged, etc.

    Yes, EMI has been confusing and time consuming to no end. It was especially painful when first encountering the symptoms without knowing about EMI in the first place. After lots of research and help in other forums, I found several fixes that make a dramatic difference and always use on my projects. Here's a short summary:

    • Keep breadboard jumpers as flat and short as possible.
    • Keep SPI and I2C lines as short as possible.
    • For long SPI and I2C lines, ALWAYS use shielded CAT5 or higher cables. Twisted wire pairs are a MUST.
    • SPI is more stable than I2C.
    • Tie-in ferrite beads at cable ends.
    • Minimize equivalent series resistance (ESR).
    • Do NOT use shielded conventional (not twisted) cables. Shielding increases circuit ESR.
    • Battery power supply: use low ESR capacitors with low dropout regulator.
    • If using motor or stepper drivers, ALWAYS use optoisolators.
    Last edited by randomvibe; 01-04-2017 at 08:09 PM.

  7. #7
    Senior Member
    Join Date
    Jan 2015
    Posts
    111
    Those basically sound like some good "helper" fixes, except for

    .... Keep breadboard jumpers as flat and short as possible.

    I would never use a breadboard for an app like this, :-).

    If using optoisolators, then double check there are no sneak paths for current. I never use optoisolators, but typically use separate power sources [ie, batteries] for motors and servos from the uC p/s, and pay attention to star-grounding to avoid ground loops. If you're using power supplies that run off the mains, there may be sneak pathways there.

    I use 3-row headers on my boards, signal-power-gnd, and distribute power to each peripheral, sensor, and I2C device by a separate cable, so there are no ground loops off the board. So, the uC board is the center of the "star" for everything connected to it, including the Vin supply. The 3-row headers provide individual power and gnd to every peripheral, with very small inductive loops because the power/gnd leads are parallel right next to each other - I typically use ribbon-cable here.

    In contrast, typical pcbs use long single or 2-row headers that have very few power/gnd pins, so they beg for the creation of external ground loops. Eg, by daisy-chaining power/gnds amongst the peripherals.

    It may be the new T3.6 modules are a bit more susceptible to EMI, but the "real" problem is you have to quench the EMI at its "source". It's a systems-level solution all around.

    Another thing that might be helpful is "source series-termination". Use a small R in series at the I/O pins. This helps eliminate reflections, ground-bounce, ringing, etc, for signals with fast transitions. My pcbs have series-Rs on essentially every I/O pin.
    https://www.google.com/search?q=sour...&tbm=isch&sa=X
    Last edited by oric_dan; 01-04-2017 at 09:04 PM.

  8. #8
    Senior Member PaulStoffregen's Avatar
    Join Date
    Nov 2012
    Posts
    18,655
    Quote Originally Posted by randomvibe View Post
    I agree it's a system level issue, but it's still curious that the Teensy 3.2 is rock solid when swapped-in for the 3.6 while keeping the rest of the system unchanged.
    I'm curious to know why too. But probably not curious enough to buy all this same hardware and try to precisely duplicate this test. (which would also involve quite a lot of guesswork without many detailed photos showing *exactly* what you've really done here)

    I can tell you Teensy 3.5 and 3.6 are exactly the same PCB and all the same parts, except the K64 chip is replaced by a K66, and 4 extra parts are added for the USB host port power. But those parts are powered down by default.

    No changes have been made to the design or PCB since the first Kickstarter batch. As far as I know, Robin has purchased exactly the same part numbers for every 3.5 and 3.6 build so far.

    Obviously Teensy 3.2 is a different board. I'm a bit surprised it's more solid in this test, since it has only a single ground plane on 1 of the internal layers. Teensy 3.5/3.6 have a full ground plane and another layer that's part ground plane (between the USB connector and regulator), mostly power plane (the other half of the board that gets 3.3V power from the regulator). There's also more power power supply decoupling on the newer boards, and even a ferrite bead in series with the incoming power. Maybe whatever's wrong doesn't really matter on the power lines? Maybe it's affecting the data signals?

    Is this the power supply you're using for the motor drivers?

    https://www.amazon.com/dp/B00O8DJ8QC

    If so, are you using a grounded cord and is the earth ground shorting bar present or disconnected from the negative terminal?

    When the problem occur, do you observe anything that suggest Teensy is locking up or rebooting? Or does it appear to continue running the same program, but use of the IMU and display are impacted?
    Last edited by PaulStoffregen; 01-05-2017 at 01:42 PM.

  9. #9
    Senior Member
    Join Date
    Jan 2015
    Posts
    111
    Paul, is there a way to slow down the transitions on the I/O pins?

  10. #10
    Senior Member PaulStoffregen's Avatar
    Join Date
    Nov 2012
    Posts
    18,655
    Yes, all the pins support slew rate limiting. By default pinMode() turns on the slew rate limit. Most of the libraries do too. SPI and I2C are exceptions.

  11. #11
    Senior Member+ KurtE's Avatar
    Join Date
    Jan 2014
    Posts
    3,690
    Paul maybe slightly off topic,
    Quote Originally Posted by PaulStoffregen View Post
    I can tell you Teensy 3.5 and 3.6 are exactly the same PCB and all the same parts, except the K64 chip is replaced by a K66, and 4 extra parts are added for the USB host port power. But those parts are powered down by default.
    As you mentioned the main processor chips are different. And as mentioned in another thread or two, There have been at least a few cases where the same Power supply works on T3.2 and T3.5, but not T3.6 example(https://www.amazon.com/Castle-Creati...1&keywords=BEC), so maybe the differences in these processors may impact this?

  12. #12
    Senior Member
    Join Date
    Jan 2015
    Posts
    111
    @Kurt, look at the wires on that thing. Bottom line is, when I hear people talking about "breadboards", I immediately think of this .... inductive loops and cross-talk all over the place.
    https://www.google.com/search?q=brea...&tbm=isch&sa=X

    And when I hear about "long wires", I think of this,
    https://www.google.com/search?q=grou...&tbm=isch&sa=X
    and especially this,
    http://electronicdesign.com/site-fil...74294_fig3.gif

    You cannot possibly come up with an objective measure unless you have the design built properly in the first.

  13. #13
    Senior Member PaulStoffregen's Avatar
    Join Date
    Nov 2012
    Posts
    18,655
    I'll point out again the overly obvious... so far in this thread we haven't seen even 1 picture of the actual setup.

    Yeah, I don't doubt that these extremely similar boards respond differently. But at the same time, I'm feeling like perhaps more tangible info, especially photos, could be shared to at least give us something to look at and think about while guessing.

  14. #14
    Senior Member
    Join Date
    Jan 2015
    Posts
    111
    Even 100-MHz oscilloscope probes can corrupt high-speed signals if the grounding is poor - meaning simply using the standard ground leads that come with the probes. Duh. This video shows what you get with just a simple 4" ground lead vs a good ground on the probe.

    Shortcut ... start at 8-min for the meat. Also talks about use of breadboards at the end.
    https://www.youtube.com/watch?v=zodpCuxwn_o

  15. #15
    Member randomvibe's Avatar
    Join Date
    Mar 2016
    Location
    CA
    Posts
    79
    Quote Originally Posted by PaulStoffregen View Post
    Is this the power supply you're using for the motor drivers?

    https://www.amazon.com/dp/B00O8DJ8QC

    If so, are you using a grounded cord and is the earth ground shorting bar present or disconnected from the negative terminal?

    When the problem occur, do you observe anything that suggest Teensy is locking up or rebooting? Or does it appear to continue running the same program, but use of the IMU and display are impacted?

    My power supply is "Dr. Meter DC Power Supply PS-305DM" (link below). I purchased it from Amazon, but it's no longer there. Very low quality in retrospect.

    http://www.newegg.com/Product/Produc...9SIA1PK2Z14050

    When the problem occurs, the Teensy continues running to completion. The IMU is not impacted. Only the display is impacted. I've run this condition over a 100 times.

    If I turn off the power supply, the display is perfect. I've run this condition hundreds of times.

    The Teensy is powered with the USB, and the stepper drivers are optoisolated with their own separate battery power. The stepper motor power lines are wire-mesh-shielded and include ferrite beads at both ends. The display SPI line is a 10 inch CAT5 internally-shielded cable. The IMU SPI line is a ~4 foot CAT6 wire-mesh-shielded cable with a ferrite bead. I recently added encoders, and these are also optoisolated. Picture of my setup enclosed.

    Click image for larger version. 

Name:	IMG_0544.jpg 
Views:	144 
Size:	236.5 KB 
ID:	9319
    Last edited by randomvibe; 01-05-2017 at 11:38 PM.

  16. #16
    Senior Member
    Join Date
    Jan 2015
    Posts
    111
    Very good. You've basically violated everything I've warned you about, and without having seen your ckt. Just one more comment, see figure 4, then I'm all done here.http://www.intersil.com/content/dam/...n13/an1325.pdf

    Good luck.

  17. #17
    Member randomvibe's Avatar
    Join Date
    Mar 2016
    Location
    CA
    Posts
    79
    Quote Originally Posted by PaulStoffregen View Post
    so far in this thread we haven't seen even 1 picture of the actual setup.
    I posted a picture here:
    https://forum.pjrc.com/threads/41244...l=1#post129385

    For prototyping, I typically use breadboards. Before I change the layout, I'll will hear out more opinions, then proceed with the best coarse of action.

    Is it possible that EMI or ESD could have damaged the Teensy 3.6 and/or the display to the point that it's more sensitive to EMI? This is highly subjective, but the symptoms seemed to get worse. The system would be okay one moment, then degrade the next day. Or just changing the length of the jumper for the ILI9341 RESET pin would improve or worsen the situation. Very frustrating.
    Last edited by randomvibe; 01-06-2017 at 03:49 AM.

  18. #18
    Senior Member PaulStoffregen's Avatar
    Join Date
    Nov 2012
    Posts
    18,655
    Quote Originally Posted by randomvibe View Post
    Is it possible that EMI or ESD could have damaged the Teensy 3.6 and/or the display
    Possible, maybe. Likely, not so much.

    Especially ESD tends to be the Bogeyman of the electronics world, mostly due to scare-tactic marketing material published by 3M and other manufacturers of anti-static products. Legions of troubleshooting technicians also contribute, using ESD as the convenient explanation for all sorts of problems that almost certainly aren't due to ESD, but are far beyond their skills or allotted time frames to diagnose.

    If you really do suspect a part may be somehow damaged but somehow still functioning, it's a simple matter to swap out parts. You've got spares, right?

  19. #19
    Senior Member
    Join Date
    Apr 2017
    Posts
    136
    Responding to this post as much of what's been discussed may be my issue as well.

    I'm coaching my high school teams electric race car and we've built a very elaborate data logger (measures electrical performance, speed, transmits data to the pit crew, has SPI display in car, has GPS to auto count laps, and other goodies). The test data logger is the exact same setup (cables and all) but sits safely on our work bench and works like a million bucks--it never looses a bit or pixel. The car unit is a different story, data always logged, but display works 25% of the time. I'm 99.9999% sure it's and EMI issue that causes the display to go white or get very garbled. We've been chasing that issue for 2 years. Some general details

    a. Data logger is a custom PCB board, single side, some jumpers no longer than 1 inch
    b. Cables for sensors are USB (b type)--no longer that 4 feet
    c. Cable for SPI display is serial cable (RS232) 6 feet long
    d. some cap protection in the data logger and sensors (0.1 uf on power supply lines).

    What caught my eye and promoted me to open this thread are the following tips:
    1) For long SPI and I2C lines, ALWAYS use shielded CAT5 or higher cables. Twisted wire pairs are a MUST.
    2) Tie-in ferrite beads at cable ends.
    3) Minimize equivalent series resistance (ESR).
    4) Do NOT use shielded conventional (not twisted) cables. Shielding increases circuit ESR.

    My sensor cables violate tips 4
    my display cable violates tips 1 and 4
    I don't know what #3 means.

    If feedback is such that shielded Cat5 (TWISTED PAIR) is better that serial cable, we'll redesign and rebuild. Is that what I should do? I realize you have no code, pics, diagrams, etc. to look at, but what's your gut feeling?

  20. #20
    Senior Member
    Join Date
    Dec 2016
    Location
    Montreal, Canada
    Posts
    2,802
    for what you want, differential RS485 or CANBUS is needed.

  21. #21
    Senior Member
    Join Date
    Apr 2017
    Posts
    136
    For clarification, i'm not using RS232 communication protocol to send data to the SPI display, i'm only using that type of cable, the data sent throught the cable is SPI stuff (MOSI, DS, CS) as well as Vcc, GND, and a few other lines for some buttons.

  22. #22
    Member randomvibe's Avatar
    Join Date
    Mar 2016
    Location
    CA
    Posts
    79
    Quote Originally Posted by KrisKasprzak View Post
    If feedback is such that shielded Cat5 (TWISTED PAIR) is better that serial cable, we'll redesign and rebuild. Is that what I should do? I realize you have no code, pics, diagrams, etc. to look at, but what's your gut feeling?
    I cannot stress enough the importance of using twisted-pair wire bundles for long SPI lines. I only use CAT6 or CAT7, although CAT7 cables are rather stiff.

    Before you redesign, observe the behavior of the car's datalogger with the motors disabled (no power going to the motor driver). I would also disable power to any other high power devices (e.g., transmitters). See if that makes a difference.

    #3 (equivalent series resistance) refers to the resistance produced by capacitors, which should ideally be zero, but in practice is not. High ESR components in your circuit can induce noise. I use low ESR in my power regulation (right next to the regulator) and noisy sensors such as Sharp IR range sensors (again, right at the sensor). Also, low ESR capacitors last longer. ESR is a selection criteria in digikey.

  23. #23
    Senior Member
    Join Date
    Dec 2016
    Location
    Montreal, Canada
    Posts
    2,802
    if you’re worried about SPI over long distance, I wrote a protocol for teensy 3.x to control multiple node ports over CANbus, of course, the limitation is the bandwidth of the bus (1Mbps max), by hosting another teensy (3.2 maybe) you could remotely run SPI devices over long distances. a video was posted demonstrating remote SPI over CAN accessing a MCP23S17 SPI port expander.

  24. #24
    Senior Member
    Join Date
    Apr 2017
    Posts
    136
    @randomvibe any guidance on what wires should be connected to the twisted pairs? For example GND/3v3 one one twisted pair MOSI/SCK another twisted pair, etc. We're redesigning PCB boards now and want to get it right.

    Thanks for helping out

  25. #25
    Each twisted pair should have a ground. However, managed to get a mcp23s08 SPI IOExpander running over a 6 foot CAT6 cable with only 3 pairs available. Tried multiple ways, best was CLK-GND, 3.3-MOSI, CS-MISO. At this length also had to slow the clock to 500K.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •