Teensy 3.1 UART (Hardware) Serial issue data size >63 bytes

Status
Not open for further replies.
L

Lotus Pack

Member
Hi,
I am connecting to a ITT serial based module for a Flash Drive from HobbyTronics http://www.hobbytronics.co.uk/usb-host/usb-host-flash-drive and have successfully communicated to the device on Serial3. However, every time the device sends back more than 63 bytes of data I lose a few bytes. The variable flash_serial_monitor set as true in the code below attempts to determine why this is happening by monitoring the serial data stream, I think it may be the device, however I have heard that the T3.1 has a fast read buffer of 63 bytes and as all seems ok, if the data is below 63 bytes, this issue appears when there is a lot more waiting to be read. To try and pinpoint where this issue is coming from I have deliberately put a ‘#’ where the problem starts simply by counting the char’s as they arrive and pushing out a ‘#’ at byte 63.

Code: 
void serial_line_receive()
{                                            
char flash_receive[1500];              // receive buffer string must be big enough for FLASH data command results
int x1=0;                              // flash receive string pointer 
int x2=0;                              // total bytes received counter
char c=0;                              // char in from flash driver serial port
unsigned int time_out=0;               // time out compare variable
unsigned int time_wait1=0;             // timer start point
unsigned int time_wait2=0;             // timer end point

while (!FLASH.available()) if(millis()>time_out) break;     //wait for flash drive buffer to recieve data                                    
while (FLASH.available()>0 )                                // when data in buffer remember Teensy3.1 will immeadiatly fill a 63 byte packet
  { 
  if (flash_serial_monitor)  {Serial.print("[");Serial.print(FLASH.available());Serial.print("]");} // see how many bytes in buffer
    c=FLASH.read();                                          // read in data take a byte off the buffer
    flash_receive[x1++]=c;                                   // transfer to recive buffer   
  if (flash_serial_monitor) {Serial.print(c,HEX); if(c==13) Serial.println();}   //display char in stream if monitoring
  
    time_out=millis()+100;                                   //set timeout period while we have data
   time_wait1=micros();                                      // Start time for no information in buffer 
    while (!FLASH.available())                               //waiting for flash program to place data in serial buffer (on the card) 
     if(millis()>time_out) break;                            //time out if waiting too long means no more data arriving break and display data
     time_wait2=micros();                                    // wait time + overhead on small times
   delayMicroseconds(time_wait2-time_wait1);                // use variable delay to increase performence and allow the card to load the buffer if delayed 
  if (flash_serial_monitor)   {Serial.print("("); Serial.print(time_wait2-time_wait1);Serial.print(")");} //seee how long we wait for buffer to receive next byte
   x2++;                                                     // total number of bytes counted 
   if (x2==63) {Serial.print("#");
        flash_receive[x1++]='#';
           }
    }
  flash_receive[x1]=0;                                       // add a null to the end of the data for printing info
if (flash_serial_monitor)  Serial.println();                 // just keep the monitor information tidy
   Serial.println(flash_receive);                            // display all data
 if (flash_serial_monitor)  Serial.println();                // print a line to clear information from flash
 if (flash_serial_monitor)  Serial.print("Bytes read=");     // prompt on how many bytes in local receive array
 if (flash_serial_monitor)  Serial.println(x2);              // display the number of bytes received in this session  
}

Test 1 request a small amount of data (44 bytes) all seems ok.
The above DIR /F GREEN.TAB with the stream monitor on shows the following:

[44]44(1)[43]49(0)[42]52(1)[41]20(1)[40]2F(1)[39]46(1)[38]20(1)[37]47(1)[36]52(1)[35]45(1)[34]45(1)[33]4E(1)[32]2E(1)[31]54(1)[30]41(1)[29]50(1)[28]A(1)[27]D
(1)[26]47(0)[25]52(1)[24]45(1)[23]45(1)[22]4E(0)[21]2E(1)[20]54(1)[19]41(1)[18]50(1)[17]D
(1)[16]A(1)[15]55(0)[14]53(1)[13]42(1)[12]20(1)[11]46(1)[10]4C(1)[9]41(0)[8]53(0)[7]48(1)[6]20(1)[5]20(0)[4]3A(1)[3]5C(1)[2]3E(1)[1]20(100179)
DIR /F GREEN.TAP
GREEN.TAP
USB FLASH :\>

Bytes read=44
The [44] is the number of bytes in the serial buffer Serial1.available() then 44 is the HEX of the char received the (1) is the number of microseconds, this is repeated until the buffer is empty when a time out is set at 100 milliseconds. The last (100179) shows this has timed out and finishes. There is no ‘#’ in this test as we only have 44 bytes so it never reached the issue.

The teensy captures 63 bytes in its internal buffer which is why the [44] shows all of these data ready to call off.

As soon as I want to see more than 63 bytes I seem to lose characters. The next test should show 76 chars (approx) using a DIR /F *.TAP.

Test 2 request slightly more data.

[63]44(1)[62]49(1)[61]52(1)[60]20(1)[59]2F(1)[58]46(0)[57]20(1)[56]2A(1)[55]2E(1)[54]54(1)[53]41(1)[52]50(1)[51]A(1)[50]D
(1)[49]50(1)[48]4E(1)[47]33(0)[46]34(0)[45]35(1)[44]31(0)[43]2E(1)[42]54(1)[41]41(0)[40]50(1)[39]D
(1)[38]A(1)[37]4D(0)[36]49(1)[35]43(1)[34]52(0)[33]4F(1)[32]2D(1)[31]7E(1)[30]31(1)[29]2E(1)[28]54(1)[27]41(1)[26]50(1)[25]D
(1)[24]A(1)[23]47(1)[22]52(1)[21]45(0)[20]45(1)[19]4E(1)[18]2E(1)[17]54(1)[16]41(0)[15]50(1)[14]D
(1)[13]A(1)[12]42(1)[11]4C(1)[10]55(1)[9]45(1)[8]2E(1)[7]54(0)[6]41(1)[5]50(1)[4]D
(0)[3]A(1)[2]55(1)[1]53(100156)#
DIR /F *.TAP
PN3451.TAP
MICRO-~1.TAP
GREEN.TAP
BLUE.TAP
US#

Test 3 request lots of data, error happens but it continues to read after losing what seems to be a USB packet.
63]44(1)[62]49(1)[61]52(1)[60]A(1)[59]D
(1)[58]32(0)[57]30(1)[56]30(1)[55]37(1)[54]2D(0)[53]30(0)[52]38(1)[51]2D(1)[50]31(1)[49]35(1)[48]20(1)[47]30(1)[46]35(1)[45]3A(1)[44]31(1)[43]33(1)[42]3A(1)[41]34(1)[40]32(1)[39]20(1)[38]20(1)[37]20(1)[36]20(1)[35]20(1)[34]20(1)[33]20(1)[32]20(1)[31]20(1)[30]20(1)[29]20(1)[28]20(0)[27]20(1)[26]32(1)[25]36(1)[24]31(1)[23]20(0)[22]50(1)[21]4E(1)[20]33(0)[19]34(1)[18]35(1)[17]31(1)[16]2E(1)[15]54(1)[14]41(1)[13]50(1)[12]D
(1)[11]A(1)[10]32(1)[9]30(1)[8]31(1)[7]35(1)[6]2D(1)[5]30(1)[4]34(1)[3]2D(1)[2]33(1)[1]30(3238)#[4]32(1)[3]30(1)[2]31(1)[1]35(809)[4]2D(0)[3]30(1)[2]34(1)[1]2D(3248)[3]33(0)[2]30(1)[1]20(4595)[8]32(1)[7]31(0)[6]3A(1)[5]32(1)[4]33(1)[3]3A(1)[2]32(0)[1]38(3507)[4]20(1)[3]3C(1)[2]44(1)[1]49(548)[4]52(1)[3]3E(1)[2]20(1)[1]20(3511)[4]20(1)[3]20(1)[2]20(1)[1]20(538)[4]20(1)[3]20(1)[2]20(1)[1]20(3524)[4]20(1)[3]4F(1)[2]4C(1)[1]44(522)[4]46(1)[3]49(1)[2]4C(1)[1]7E(3541)[3]31(1)[2]D
(1)[1]A(8127)[8]32(1)[7]30(1)[6]31(0)[5]34(1)[4]2D(0)[3]30(1)[2]37(1)[4]2D(1)[3]32(1)[2]36(1)[1]20(6860)[9]30(1)[8]38(1)[7]3A(1)[6]31(0)[5]33(1)[4]3A(1)[3]31(1)[2]36(0)[1]20(6126)[8]20(0)[7]20(1)[6]20(1)[5]20(1)[4]20(1)[3]20(0)[2]20(1)[1]20(1985)[4]20(1)[3]20(1)[2]32(1)[1]34(2072)[4]30(1)[3]36(1)[2]31(0)[1]20(1989)[4]4D(1)[3]49(1)[2]43(1)[1]52(2067)[4]4F(1)[3]2D(1)[2]7E(1)[1]31(1986)[4]2E(1)[3]54(1)[2]41(1)[1]50(1042)[2]D
(1)[1]A(10537)[11]32(1)[10]30(1)[9]30(0)[8]37(1)[7]2D(0)[6]30(1)[5]38(1)[4]2D(1)[3]31(1)[2]35(5)[1]20(5403)[8]31(1)[7]39(1)[6]3A(1)[5]35(1)[4]39(0)[3]3A(1)[2]32(0)[1]36(638)[1]20(4375)[8]20(1)[7]20(1)[6]20(1)[5]20(1)[4]20(0)[3]20(1)[2]20(1)[1]20(3733)[4]20(1)[3]20(1)[2]20(1)[1]20(313)[4]31(0)[3]30(1)[2]37(1)[1]20(3753)[4]47(1)[3]52(1)[2]45(0)[1]45(295)[4]4E(1)[3]2E(1)[2]54(1)[1]41(3768)[3]50(1)[2]D
(1)[1]A(7926)[8]32(1)[7]30(0)[6]30(1)[5]37(1)[4]2D(1)[3]30(1)[2]38(0)[1]2D(182)[3]31(1)[2]35(1)[1]20(6707)[8]32(1)[7]30(1)[6]3A(1)[5]30(1)[4]30(1)[3]3A(0)[2]33(1)[1]32(2367)[4]20(1)[3]20(1)[2]20(1)[1]20(1688)[4]20(1)[3]20(1)[2]20(1)[1]20(2367)[4]20(1)[3]20(1)[2]20(1)[1]20(1683)[4]20(1)[3]20(1)[2]39(1)[1]37(2377)[4]20(1)[3]42(1)[2]4C(1)[1]55(1677)[4]45(1)[3]2E(0)[2]54(1)[1]41(2378)[3]50(1)[2]D
(1)[1]A(2476)[4]55(1)[3]53(1)[2]42(1)[1]20(1572)[4]46(1)[3]4C(1)[2]41(1)[1]53(2482)[4]48(1)[3]20(1)[2]20(0)[1]3A(1576)[3]5C(1)[2]3E(1)[1]20(100082)
DIR

2007-08-15 05:13:42 261 PN3451.TAP
2015-04-30#2015-04-30 21:23:28 <DIR> OLDFIL~1
2014-07-26 08:13:16 24061 MICRO-~1.TAP
2007-08-15 19:59:26 107 GREEN.TAP
2007-08-15 20:00:32 97 BLUE.TAP
USB FLASH :\>

Bytes read=267

The '#' shows in the data at point 63 then it continues to read the data in from the communication stream but has lost quite a bit of data just after this point.

Does anyone know of similar problems, I am also talking to the device vendor about this issue.

LP
 
Note, only Serial1 and Serial2 have 8 byte FIFOs in the Teensy 3.1. Serial3 does not have a FIFO. You might get better results if you switch to using either Serial1 or Serial2.

FWIW, in the Teensy 3.0, only Serial1 has the FIFO, and Teensy LC does not have any FIFO's.

I have griped in the past, that it would been nice if there was some way to have added RTS/CTS for hardware flow control to the Teensy, so that either side can signal the other when its buffers are getting full. I realize that it may not have been possible, but in the past when I used real RS-323 9/25 pin serial ports to talk to embedded devices, it helped. Otherwise, I would need to tinker with the baud rate or add explicit delays to slow down the transmission. If you can turn it on, checksums can really help at spotting data corrpution.

<edit>
On other thing that occurs to me -- are you running something else on the Teensy that make be turning off interrupts for a period of time? The standard neopixel library is one thing that turns off interrupts while it is updating the lights. But, I would also look at i2c, spi, servos, and PWM devices on general principals. Particularly when using Serial3, having interrupts disabled too long might mean losing data. With Serial1/Serial2, the 8 byte FIFO can be filled when interrupts are disabled and it can smooth things over.
 
Last edited:
either increase your buffer array->(RX_BUFFER_SIZE) size defined in seial(x).c or make sure your reading the buffer more often. The serial driver buffer defaults to 64 bytes so your most likely overrunning the buffer.

This is not the same as the hardware internal 8 byte fifo, that fifo will not protect you if you send more than 64 bytes to your teensy and do not read the buffer array fast enough. It does not give you an extra 8 bytes and actually its only set at 4 for serial1.c and serial2.c.
 
Perhaps code in another part of your program (which you didn't post) is delaying too long?

I would agree with duff, the simplest thing may be just editing serial3.c to increase the buffer size for Serial3.

Michael's point about flow control is good for cases where the other device can use flow control. This does not appear to be a device with hardware flow control.

However, it does have I2C. Perhaps you could try using I2C instead of serial?
 
Yeah, in looking at the code, I would imagine that you really should restructure the code. I think things are getting messed up when you try to read and see if a full line has been read in.

I would think about using a circular buffer. Every time through the loop, you check if a character is available, and if so, add it to the circular buffer (checking for the buffer being full first). Then independently check if you have received a full line within a timeout, and do whatever, consuming the data in the circular buffer. Do not use a delay. If you are not familiar how to avoid using delay, this tutorial might help: http://www.gammon.com.au/blink
 
Status
Not open for further replies.
Back
Top