Teensy 3.0 goes mute when sending to Raspberry Pi

Status
Not open for further replies.
Okay, this is a strange bug. I have several Teensy 3.0's which are reading from sensors and doing Serial.println()s.

The Raspberry Pi is able to read the first five or ten lines, and then stops reading anything (using cat </dev/ttyACM0). Even if I strip the code down to the AnalogSerialOut example, it does this. Of course, this works just fine on the Teensy 2.0. It also works just fine sending to a Windows or Linux machine.
And here's a detail that may be important: it also fails when sending to a Technologics TS-7400 (ARM running Debian 5.0). The Teensy is configured for USB Type: Serial. I first noticed this under Arduino 1.0.3, but it also happens under Arduino 1.0.5 and Teensyduino 1.14.

Could this be something specific to ARM Linux?
 
Last edited:
Further, there is nothing logged in /var/log/syslog at the time it goes silent. And it goes silent after a short period of time whether you read the output or not. Closing and re-opening /dev/ttyACM0 doesn't make it start talking again.
 
That Python script looks suspiciously short. Could the forum software have helpfully "compressed" it?

Yes, the Serial2 and Serial3 fixes have nothing to do with transmitting via USB.
 
I have a couple Raspberry Pi boards buried in a box of stuff. Looks like I need to dig one out and investigate....

My first impression is to be a bit suspicious of using "cat". Sometimes cat works great, other times a gap in reception can be seen by cat as the end of file.
 
That Python script looks suspiciously short. Could the forum software have helpfully "compressed" it?

Yeah, I think you might be right. I'll check this evening, I don't have it in front of me.
It might be this at the end that's missing though.

Code:
while 1 :
   if ser.inWaiting() > 0 :
      ser.read(16384);
 
I have a couple Raspberry Pi boards buried in a box of stuff. Looks like I need to dig one out and investigate....

My first impression is to be a bit suspicious of using "cat". Sometimes cat works great, other times a gap in reception can be seen by cat as the end of file.

Thanks! I'm desperate here. We've got a sonde we have to get into the water and the customer keeps calling. I agree with what you say about cat finding EOF early. In this case, cat just sits there. "dd if=/dev/ttyACM0 bs=1 count=1" will also sit there forever once it's gone silent.

On the off chance that the Teensy 3.0 code was blocking or otherwise stopped running, I inserted a flashing LED into the AnalogReadSerial() code. No obvious change in the LED blinking rate after the silence set in.

And, out of paranoia, I tried a Teensy 2.0 doing the same thing, plugged it into the same USB hub port, ran the same 'cat' command, and it continued to read for as long as I was willing to wait.
 
Here's the code in full.

Code:
# This script flushes every byte received on the serial port                                       

import serial
import sys

ser = serial.Serial(
   port = '/dev/ttyACM0',
   baudrate = 115200,
   parity = serial.PARITY_NONE,
   stopbits = serial.STOPBITS_ONE,
   bytesize = serial.EIGHTBITS
)

if ser.isOpen() == False :
   ser.open()
   if ser.isOpen() == False :
      print "Can't open serial port"
      sys.exit(1)

while 1 :
   if ser.inWaiting() > 0 :
      ser.read(16384);
 
Hi,

Admitting up front I did not read through the entire thread...

I suspect the issue is with the ACM driver in Linux - I work around it by ensuring I have the port opened for read before letting the Teensy send anything (by having the Teensy wait for a start byte first...).

I have this issue with my Ubuntu 10.04 machine, my *recent* OpenWRT on MIPS AR9331 [suspect it wasn't an issue before...], as well as ARM boards...
 
Or it is a stty setting :) I forget... Since I never remember to set stty raw or any fancy mode, I continue to use my workaround... which works and is easy to remember...
 
Interesting! I bashed the Blink and AnalogReadSerial code together so it would blink on every Serial.println(). In the setup() function, I waited until Serial.read() returned a space. Then on the Linux side, I did "cat </dev/ttyACM0 &" and then "echo " ">/dev/ttyACM0", the LED started blinking, and .... (the envelope please) sadly, it makes no change for me. Thanks for your suggestion, though.

I tried "stty raw </dev/ttyACM0", but that didn't help.
 
Last edited:
I tried running the Python program, figuring what the hell, it's practically the same as my program, but might give me different results. I'm getting a stream of lines, but not all of them. It will just pause, as if some of the lines are going into the bit-bucket. Finally, after a minute or so, it will throw an exception, saying:
serial.serialutil.SerialException: device reports readiness to read but returned no data (device disconnected?)
 
I've seen exactly this sort of problem when 2 different programs both have the device open and are trying to read. Linux doesn't prevent a 2nd program from opening the device, but it probably should.
 
I found my Raspberry Pi board, burned a SD card with the latest Raspbian (2013-05-25). I am able to reproduce the problem.

Here's the code I'm running on the Pi:

Code:
// compile with:  gcc -O2 -Wall -o serialdump serialdump.c

#include <stdio.h>
#include <stdlib.h>
#include <sys/select.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>

#define PORT "/dev/ttyACM0"

int main()
{
        int r, port;
        fd_set rfds;
        struct timeval tv;
        char c;

        port = open(PORT, O_RDWR);
        if (port < 0) {
                printf("unable to open %s\n", PORT);
                return -1;
        }
        while (1) {
                FD_ZERO(&rfds);
                FD_SET(port, &rfds);
                tv.tv_sec = 5;
                tv.tv_usec = 0;

                r = select(port+1, &rfds, NULL, NULL, &tv);
                if (r < 0) {
                        printf("select error\n");
                } else if (r == 0) {
                        printf("no data timeout\n");
                } else if (FD_ISSET(port, &rfds)) {
                        r = read(port, &c, 1);
                        if (r == 1) {
                                printf("%c", c);
                        } else {
                                printf("read error\n");
                        }
                } else {
                        printf("select says data ready, but where?\n");
                }
        }
        close(port);
        return 0;
}

Here's the code I'm running on the Teensy 3.0:

Code:
int led = 13;
int count = 0;

void setup() {                
  Serial.begin(115200);
  pinMode(led, OUTPUT);     
}

// the loop routine runs over and over again forever:
void loop() {
  Serial.print("blink # ");
  Serial.println(count++);
  digitalWrite(led, HIGH);   // turn the LED on (HIGH is the voltage level)
  delay(500);               // wait for a second
  digitalWrite(led, LOW);    // turn the LED off by making the voltage LOW
  delay(500);               // wait for a second
}

Later today I'll get the USB protocol analyzer hooked up and look into what's actually happening....
 
Last edited:
I'm looking at the USB packets now. I believe I understand what's happening.....

The problem appears to be related to the Raspberry Pi echoing the data back to the Teensy.

On Teensy 2.0, the USB peripheral is able to buffer 2 packets per endpoint. The first 2 bytes are buffered, and then it stops accepting data. Each endpoint has it own 2 buffers, so reception can't prevent transmission.

On Teensy 3.0, the USB packets are buffered using a pool of memory. Linux transmits several packets (very inefficiently, 1 byte per packet), and Teensy 3.0 keeps accepting them until all the available memory is filled up with packets waiting to be received with Serial.available() and Serial.read(). Of course, this program never tries to receive any data, so they sit there forever/ in the buffers. After transmitting several messages, more memory is needed to keep transmitting, but all the memory is tied up with the received packages, so transmitting become impossible.

There are 3 possible solutions.

#1: Turn off character echo on Linux. Apparently this is the ECHO bit in the termios c_lflag field. See "man termios" for details. The stty program can probably also be used to disable character echo.

#2: Add code on Teensy to use Serial.read() to receive and discard any incoming data, so it can't fill up Teensy3's USB buffers.

For example, I added one line to the test code above, which makes it work perfectly with the Raspberry Pi that has character echo turned on.

Code:
int led = 13;
int count = 0;

void setup() {                
  Serial.begin(115200);
  pinMode(led, OUTPUT);     
}

// the loop routine runs over and over again forever:
void loop() {
  Serial.print("blink # ");
  Serial.println(count++);
  digitalWrite(led, HIGH);   // turn the LED on (HIGH is the voltage level)
  delay(500);               // wait for a second
  digitalWrite(led, LOW);    // turn the LED off by making the voltage LOW
  delay(500);               // wait for a second
  while (Serial.read() >= 0) ; // discard any buffed incoming data
}

#3: Deep inside the usb_dev.c code for Teensy 3.0, I anticipated this could be a problem... but actually doing something to prevent excessive reception from staving other endpoints of buffer memory is still a "TODO" item. Here's that code:

Code:
                                        // TODO: implement a per-endpoint maximum # of allocated packets
                                        // so a flood of incoming data on 1 endpoint doesn't starve
                                        // the others if the user isn't reading it regularly
                                        packet = usb_malloc();
                                        if (packet) {
                                                b->addr = packet->buf;
                                                b->desc = BDT_DESC(64, ((uint32_t)b & 8) ? DATA1 : DATA0);

Realistically, it's going to be quite a while until this gets implemented. For now, options #1 and #2 are the available solutions.
 
Last edited:
Arduinoland is gradually discovering why Dennis Ritchie's vision of what an OS should do (multiplex data streams) made UNIX so successful.
In UNIX the kernel owns a pool of i/o buffers that those bytes are consolidated into whether the user has called read yet or not. This is double buffering
that free's up the hardware (USB endpoint) buffers and saves on context switch overhead when the user finally does pull the data out.
There can be a cost to this (memory and latencies from context switches) but we now have
30 years experience on how to deal with this.
I was going to propose an API extension where you would let the user provide a function that is called
where you do the memcopy in the USB readBytes(). Then I could do my SLIP processing and buffering to make the application efficient - specially in multiple stream scenarios. I would retract this proposal if you refactor your stream buffering to share a pool of buffers. This move towards a UNIXy
way of doing this may sound daunting but actually on the Teensy 3 you have a nice lightweight way of doing this (what is called a top/bottom half
of a UNIX driver) now that you have a periodic interval timer in place.
 
Yes, I see that. I was clumsily trying to suggest that if eventually you go to the trouble of resource managing your USB pool to avoid these starvation problems between transmission and reception you might as well do it generally for all the stream devices you may have. It won't be long before there are many applications with easily 3 or 4 devices, e.g. this sort of scenario: audio i/o (I2S), USB, SPI for SD card for example, and i2c for gesture sensor.

My perspective comes from starting to explore sending OSC over various serializing devices: USB Serial, USB MIDI, SPI ethernet, SPI <-> (Rasberry PI), etc. I noticed that the Arduinos and variants have differing buffering strategies and sizes among these devices, so while stream.h gives a potentially unified API it is not obvious how to tune one's use of the API for higher performance.
 
Last edited:
I believe that you have hit the ball on the nose (or something like that). Here's what Ubuntu says about /dev/ttyACM0:
Code:
speed 57600 baud; line = 0;
eof = ^A; min = 1; time = 0;
-brkint -icrnl -imaxbel
-opost -onlcr
-icanon -echo -echoe
And here's what the Raspberry Pi running Raspbian says:
Code:
speed 9600 baud; line = 0;
-brkint -imaxbel
And ... for my logger program, when I open up a file, I have to do this:
Code:
                inf = open(os.path.join(dirname, fn))
                fd = inf.fileno()
                (iflag, oflag, cflag, lflag, ispeed, ospeed,
                    cc) = termios.tcgetattr(fd)
                lflag &= ~(termios.ICANON|termios.ECHO|termios.ECHOE|
                    termios.ECHOK|termios.ECHONL)
                termios.tcsetattr(fd, termios.TCSANOW, [iflag, oflag,
                    cflag, lflag, ispeed, ospeed, cc])
With this in place, my logger works fine. And if I turn off echoing using stty, it also works great. So I think that maybe this needs to be documented somewhere, but it's definitely a viable work-ground.

Thanks, Paul!
 
Hrm. Simply reading from Serial is not sufficient to work around this problem. If you reboot with the test Teensy code from #15 above still running, it goes mute again. Something happens during the reboot to cause it to stop transmitting. I reproduced this using the same version of Raspbian (2013-05-25). If I unplug the Teensy and plug it back in again, I get data. The whole time the light is blinking, so the Teensy hasn't crashed.

Just FYI, using the code I described in #11, I observed that the light would not start blinking by itself through an entire reboot cycle. The implication being that Linux isn't sending any characters to the Teensy (which could fill up the pool of buffers as described above).
 
Yes. Seemingly, anyway. I tried changing the permissions granted to 0654 and that's what /dev/ttyACM0 got, so I presume that the entire rules file is being applied.
 
Is there anything I can do to help fix this problem? Going to implement a relay to drop power, but that offends my engineering sensibility. Sadly, I don't have a USB protocol analyzer, so my options are limited to 1) gazing at it quizzically, 2) staring at it with furrowed brow, and 3) glaring at it with distaste.

Hey, what do you think about this theory: The Teensy3 is transmitting during the Raspberry Pi reboot. What if it is filling up the transmit buffers and the Raspberry Pi isn't pulling data out of them? Could it be that those transmit buffers never get transmitted, but consume resources needed to transmit later data that could be transmitted?

EDIT: I looked at usb_serial.c, and if the USB -> RPi communication isn't automatically restarted when the RPi comes back online, that code will definitely fill up all the transmit buffers and then start returning -1 on new writes. But that's just related to the one endpoint; there should still be resources to re-establish communication.

Would I get any useful information by un-commenting the various serial debug lines and hooking up a serial port?
 
Last edited:
I've got the Raspberry Pi on my desk again and the Teensy3 hooked up through the protocol analyzer. I'm able to reproduce the problem.

So far, I can see where it's not responding, but I'll need to do more work to figure out why. Maybe later tonight, maybe tomorrow morning....
 
Status
Not open for further replies.
Back
Top