WIRE library I2C lockups on Teensy 3.1

B

blahfoo

Active member
In a nasty EMI environment, I've found my T3.1 based gadget needed stronger pull-up resistors on its I2C bus. Turns out the I2C driver locked up regularly, resulting in the system being pulled out by the watch dog timer. The application has a 2kW switching power supply and a 1kW VHF RF source inches from the Teensy (!) Merely sticking stronger (3.3K) pullups from 5V on SDA and SCL fixed the problem, so no complaints about the hardware. I'm anxious though about a driver that can lock the CPU if it doesn't see the right signals on its bus. It should time out instead.

I see an alternative driver from nox771 (https://forum.pjrc.com/threads/21680-New-I2C-library-for-Teensy3), which might be more robust. But since Wire is used by two other libraries that I am using, it's over my experience to swap the I2C stuff out and recompile the whole thing, is that easier than I imagine? Perhaps editing the source for the [touchscreen and I/O expander chip] libraries to "simply change the #include <Wire.h> to #include <i2c_t3.h>". That easy?

Anyone else had I2C lockups?

Thanks!
 
Not using a Teensy, but on a Sparkfun FIO V3 board with the ATMega32U4 chipset I experienced a very similar problem in a nasty EMI environment. I was using the FIO to trigger SSR's which were driving a pair high power rotary solenoids (OK, not that high power, 36V, 2A). It also had a gyroscope attached to the rotating platform that talked over I2C. Well I had forgotten my flyback diodes initially, so there were ~100V spikes in the solenoid supply lines which were millimeters away from the FIO and gyro, plus the large magnetic pulses from the solenoids themselves which were ~1 in away (though these mattered much less). It would run fine at slow speeds, but when I ramped the oscillation frequency up past about 6 Hz I was finding that the chip would lock up completely.

The problem went away when I disabled the gyroscope (and thus I2C). It also went away if I shielded the FIO, gyro, and signal wires with an aluminum foil makeshift faraday cage so I knew it was EMI. Shortly after that I realized my mistake of missing the diodes, so never tried stronger pullups, but the issue sounds very similar.

Unfortunately I no longer have that rig, so I can't go back and test it out for you.

Later on in a separate part of this project I did switch from a FIO to a Teensy 3.1 (my first Teensy experience, and hooked ever since because they are amazing). Now I only had one small part of my code using the I2C library, but using the T3 I2C library instead of Wire was as easy as changing the #include line for my code to make it function. I needed the enhanced I2C frequency control that the T3 library offered me since I had to poll an I2C sensor very fast. Overclocking the I2C bus was the only way to get it quick enough, so that library made my project possible.
 
Last edited:
No need to test anything for me, I'm after the driver bug even though I have the hardware problem fixed. Really I'm hoping someone familiar with the Wire driver code itself knows how it could lock up and might pipe in.
Thanks!


Jensen567 said:
Not using a Teensy, but on a Sparkfun FIO V3 board with the ATMega32U4 chipset I experienced a very similar problem in a nasty EMI environment. I was using the FIO to trigger SSR's which were driving a pair high power rotary solenoids (OK, not that high power, 36V, 2A). It also had a gyroscope attached to the rotating platform that talked over I2C. Well I had forgotten my flyback diodes initially, so there were ~100V spikes in the solenoid supply lines which were millimeters away from the FIO and gyro, plus the large magnetic pulses from the solenoids themselves which were ~1 in away (though these mattered much less). It would run fine at slow speeds, but when I ramped the oscillation frequency up past about 6 Hz I was finding that the chip would lock up completely.

The problem went away when I disabled the gyroscope (and thus I2C). It also went away if I shielded the FIO, gyro, and signal wires with an aluminum foil makeshift faraday cage so I knew it was EMI. Shortly after that I realized my mistake of missing the diodes, so never tried stronger pullups, but the issue sounds very similar.

Unfortunately I no longer have that rig, so I can't go back and test it out for you.
Click to expand...
 
Ah, yes, I am more of a hardware geek, so most of my software stuff is pretty hacked together, and I am constantly amazed it works at all. I'm sure someone much smarter than I am will be able to chime in and help you out on that end. Look forward to hearing a solution though since I was quite confused at the time as to how that could even happen. Good luck!
 
blahfoo said:
Really I'm hoping someone familiar with the Wire driver code itself knows how it could lock up and might pipe in.
Click to expand...

Well, I wrote it and I'm piping in right now.

Trouble is, I have no idea how to reproduce the problem. There's very little I can do until a way is found to reproduce the problem.

Perhaps the noise could be simulated? Maybe you try connecting a 1 nF to 10 nF capacitor between the SDA line and another unused digital pin (perhaps one with PWM), and do the same for SCL too. Using digitalWrite or analogWrite on those pins will cause rising and falling edges to capacitively couple to the I2C signals.

Doing too much of this could be really stressful and possibly even damaging to your Teensy, and maybe to any other I2C devices connected. If you really do wire this up and give it a good try and destroy your Teensy, please let me know and I'll be happy to replace it. Of course, my hope is you'll manage to find a way to reliably reproduce the problem which I can easily duplicate here. If a way is found, I will work on the driver code inside the Wire library.
 
OK thanks, I will see if I can make it happen by getting rid of pull-ups entirely in a low-noise situation, or by feeding a minor bit of noise in. Since raising the pull-up current fixed it I suspect it wasn't anything of damaging levels. But it's on me to find a way to reproduce it.

PaulStoffregen said:
Well, I wrote it and I'm piping in right now.

Trouble is, I have no idea how to reproduce the problem. There's very little I can do until a way is found to reproduce the problem.

Perhaps the noise could be simulated? Maybe you try connecting a 1 nF to 10 nF capacitor between the SDA line and another unused digital pin (perhaps one with PWM), and do the same for SCL too. Using digitalWrite or analogWrite on those pins will cause rising and falling edges to capacitively couple to the I2C signals.

Doing too much of this could be really stressful and possibly even damaging to your Teensy, and maybe to any other I2C devices connected. If you really do wire this up and give it a good try and destroy your Teensy, please let me know and I'll be happy to replace it. Of course, my hope is you'll manage to find a way to reliably reproduce the problem which I can easily duplicate here. If a way is found, I will work on the driver code inside the Wire library.
Click to expand...
 
Wire I2C driver lockup reproduction procedure

Allright, I ran into this I2C driver lockup-instead-of-timeout problem again in another context and I have a reproduction procedure for you to try.
1. Set up a program talking to an external I2C device, such as an MCP23008. Constantly looping, often talking to the I2C device, with some indication of continued execution of the program loop.
2. Separate the external device's power from the teensy power, and turn it off. I.E, short its power pin to ground, since it could be powered via the I2C port itself in some cases.
Teensy 3.1 running "wire" library locks up.

Another procedure I suspect may work, though I haven't tried it, simply connect the I2C pins to ground. This is effectively analogous to strong electrical noise which is what caused it for me before.

When this may be a problem - and one way to see it - is if the Teensy is set up to run on USB power or a device's internal power, but the rest of that system does not run on USB power. With a watchdog timer set up to catch lockups and reboot after 2 seconds, I see the Arduino serial monitor window blink on and off every few seconds, indicating the port disconnects, and re-establishes itself constantly. And haha, I am recording a reboot count in the EEPROM, thus rapidly working on wearing a groove in the EEPROM while this is happening.
-Phil
 
Discussing a potential signal noise problem without an Oscilloscope is really just guesswork.
Trying to manage a noise issue in software, e.g. a timeout is the wrong approach. It can help in the short term in debugging and moving the project forward but for a reliable system make sure the signal is clean before attempting to patch things in software.

A 3.3k resistor may be stronger than what you had, but still is not particularly strong. The I2C pins per spec are able to sink 3mA of current. Thus the min value for a pull-up resistor ( in a 5V system) is 1.8k. If you go lower the pins may not be able to pull down the line below logical zero.
 
Not hardware problem.

I'm not sure who you are replying to, headroom. Me, I'm not having a noise problem now, that was fixed long ago. But noise did test the driver and showed that it did not time out on an I2C comm fail. That's a software problem. I had the same issue recently due to a different condition, one that I would consider a normal error condition - talking to devices that are not powered up. That should not lock up the I2C comm driver, that would be bad form. :) Anyho it sounds like I need to check if there's a version of wire later than 4/22/15, and also, there's been some changes proposed by MPeyrega, with timeouts.

Headroom said:
Discussing a potential signal noise problem without an Oscilloscope is really just guesswork.
Trying to manage a noise issue in software, e.g. a timeout is the wrong approach. It can help in the short term in debugging and moving the project forward but for a reliable system make sure the signal is clean before attempting to patch things in software.

A 3.3k resistor may be stronger than what you had, but still is not particularly strong. The I2C pins per spec are able to sink 3mA of current. Thus the min value for a pull-up resistor ( in a 5V system) is 1.8k. If you go lower the pins may not be able to pull down the line below logical zero.
Click to expand...
 
@blahfoo I assume @Headroom's excellent observation was directed at my open guess/speculation that someone with a scope/LA and a working hardware setup might see what a $3 lighter does to a working system. If that made noise that led to a reproducible error there might be a way forward to diagnose or prevent it. My post was in response to Paul #5 wanting to find a way to make noise. It causes noise on AM 535-1605 kHz but not so much on FM 88 to 108 MHz - that could be AM.vs.FM or the freq hit.

Headroom said:
Discussing a potential signal noise problem without an Oscilloscope is really just guesswork.
Click to expand...

Just for fun running blink sketch a single lighter SNAP - with my finger over a RAW/USB teensy and the lighter spark on my finger - the Teensy restarts or hangs with serial out stopping. Same thing with a few more SNAPs of the lighter on the USB to my computer. I'm just a kid playing with fire here - not just because a lighter out of fuel is in the other room. But there is a lot of noise/energy there and it is in the general range of trouble.

To be more safe and scientifical - I got the fuel free lighter. With my hand over the Teensy and the lighter on the back of my hand it does not impair operation. But without my finger over the Teensy a similar distance doesn't impair it - but it did once at 4" and it can from non contact distance. I have I2C OLED here - but will have to delay scientificating that in any way - and I don't have a LA.

I find the Teensy is generally (dry hand) 'touch safe', but putting my finger tip on some OSH_T_3.1 bottom pads - and a single SNAP on my forearm/inner elbow/hand can also halt/restart the Teensy. I can get the same when I hold USB cable and snap my fingers. So pinching I2C wires and self zapping might direct transient noise enough to screw up a transaction that ideally the software would deal with.

Note: I don't always know what the Teensy does - the BLINK is still on - but USB goes offline, or it may reset and blink count restart. A PROGRAM BUTTON always restores it.
Note: The Piezo is contained in the factory shroud - I don't feel the shock
Note: I laid my cell on the Teensy and placed a call - and talked BLE to a unit under it too and no problem.
 
Solved, thanks nox771. i2c_t3 works, no lockup.

So I did more homework after seeing the related thread (https://forum.pjrc.com/threads/21680-New-I2C-library-for-Teensy3) from nox771. Loaded up i2c_t3 library from github, edited Adafruit ft6206 and mcp23008 libraries to include i2c_t3.h instead of Wire.h, and added a "Wire.setDefaultTimeout(10000);" call (10mS) to my initialization. Everything works as before, but no lockup on powered down devices. Good stuff!


defragster said:
@blahfoo I assume @Headroom's excellent observation was directed at my open guess/speculation that someone with a scope/LA and a working hardware setup might see what a $3 lighter does to a working system. If that made noise that led to a reproducible error there might be a way forward to diagnose or prevent it. My post was in response to Paul #5 wanting to find a way to make noise. It causes noise on AM 535-1605 kHz but not so much on FM 88 to 108 MHz - that could be AM.vs.FM or the freq hit.



Just for fun running blink sketch a single lighter SNAP - with my finger over a RAW/USB teensy and the lighter spark on my finger - the Teensy restarts or hangs with serial out stopping. Same thing with a few more SNAPs of the lighter on the USB to my computer. I'm just a kid playing with fire here - not just because a lighter out of fuel is in the other room. But there is a lot of noise/energy there and it is in the general range of trouble.

To be more safe and scientifical - I got the fuel free lighter. With my hand over the Teensy and the lighter on the back of my hand it does not impair operation. But without my finger over the Teensy a similar distance doesn't impair it - but it did once at 4" and it can from non contact distance. I have I2C OLED here - but will have to delay scientificating that in any way - and I don't have a LA.

I find the Teensy is generally (dry hand) 'touch safe', but putting my finger tip on some OSH_T_3.1 bottom pads - and a single SNAP on my forearm/inner elbow/hand can also halt/restart the Teensy. I can get the same when I hold USB cable and snap my fingers. So pinching I2C wires and self zapping might direct transient noise enough to screw up a transaction that ideally the software would deal with.

Note: I don't always know what the Teensy does - the BLINK is still on - but USB goes offline, or it may reset and blink count restart. A PROGRAM BUTTON always restores it.
Note: The Piezo is contained in the factory shroud - I don't feel the shock
Note: I laid my cell on the Teensy and placed a call - and talked BLE to a unit under it too and no problem.
Click to expand...
 
It was not directly aimed at anyone but more of an observation in connection with other threads. Often Noinse is assumed to be the culprit but when the signal is observed with an Oscilloscope it turns out not to be the problem. I am glad that the OPs initial problem was solved and I probably should have paid some better attention to the timeimgs of the posts :)

I agree that the Wire library should not just lock up but it's done that for as long as I remember. My workaround for Atmel based Arduino boards was to use the DSS Circuits Master I2C library, which has a timeout-restart feature which helped me tremendously overcome my initial problems in a I2C system.

As defragster already said, Paul has recently done some work on the library IIRC in order to get that Lidar lite sensor to work properly.
However if you want to try replacing the Wire library with the i2c-t3 library it is as simple as changing the includes.
 
Saw your post coming in as I was writing mine. Glad it works. The i2c_t3 library is really an awesome piece of work.
I use it in an I2C system with 11 I2C devices over 14 ft of CAT 5 cable at 1MHz bus frequency utilizing a PCA9600 bus buffer At the Teensy 3.0 ( the slaves are all FM+ Compatible already) and the signal is rock solid.
 
You must log in or register to reply here.
Back
Top