Help with Storing incoming serial2 data in buffer at 115200 baud with teensy 3.2's

[mikeg]

I am having trouble storing incoming serial data in a Teensy 3.2 (coming from another Teensy 3.2.) in a buffer at 115200 baud.
In this code, w_time is a elapsed time variable, allowing it to time out kills the process. The incoming data length is unknown, and there is no "end of transmission" character so with this I have to have a way to exit the process.
rx_buff is a byte array ,1500 elements, I dont expect more data than about 512 bytes
cntr is the buffer pointer.
I blast in the data and only receive anywhaere from 167 to 470 bytes, its inconsistent.
This code appears to work better with Serial1, but I need both ports
I did find a post about changing the serial buffer size in Serial.c, and I increased it to 512 from 64. Any help would be appreciated.

code fragment:
[/CODE]if( Serial2.available() ) {
w_time = 0;
while( w_time < 100 ) {
while ( Serial2.available() ) {
rx_buf[cntr] = Serial2.read();
++cntr;
w_time = 0; // reset timer since data arrived
}
}
}
[[CODE/]

 

[KurtE]

if( Serial2.available() ) {
  w_time = 0;
  while( w_time < 100 ) {
    while ( Serial2.available() ) {
      rx_buf[cntr] = Serial2.read();
      ++cntr;
      w_time = 0; // reset timer since data arrived
    }
  }
}

It would probably help to have more information. Things like you say you use both Serial1 and Serial2... How?
I assume w_time is an elapsedmillis object? Is there any code that is processing interrupts...

The reason ask about multiple objects. If you have code hard spinning for 100ms, at a baud rate of 115200, you could in theory receive something like 1152 bytes of data over the serial port.

Sometimes I setup code like this to avoid multiple calls per character. Something like:

if( Serial2.available() ) {
  w_time = 0;
  int ich;
  while( w_time < 100 ) {
    while ( (ich = Serial2.read()) != -1 ) {
      rx_buf[cntr] = ich;
      ++cntr;
      w_time = 0; // reset timer since data arrived
    }
  }
}

But again not sure how much time this saves you. Again not sure what else to suggest. If your code on other side does bursts of data, maybe change timeout to something like 5 instead of 100...

 

[Frank B]

There's a function SerialX.readBytes() which reads serveral bytes at once - this is faster

 

[KurtE]

There's a function SerialX.readBytes() which reads serveral bytes at once - this is faster

Maybe, but I believe this comes from the Stream class, and this uses the setTimeout and timedRead()

int Stream::timedRead()
{
  int c;
  unsigned long startMillis = millis();
  do {
    c = read();
    if (c >= 0) return c;
    yield();
  } while(millis() - startMillis < _timeout);
  return -1;     // -1 indicates timeout
}

size_t Stream::readBytes(char *buffer, size_t length)
{
	size_t count = 0;
	while (count < length) {
		int c = timedRead();
		if (c < 0) {
			setReadError();
			break;
		}
		*buffer++ = (char)c;
		count++;
	}
	return count;
}

The one things it does in here which may help or hurt in the other code is while looping waiting for either a char or a timeout, it calls yield...

But as I mentioned, my gut tells me it is probably some interaction of things, for example if the code takes the buffer you receive and hypothetically tries to write it to Serial1, with something like: Serial1.write(rx_buf, cntr); where cntr is lets say 500 bytes, the code will write the first N bytes onto the output queue and then wait there until it has room to add the rest... But again just guessing.

 

[mikeg]

re: need more data

if( Serial2.available() ) {
  w_time = 0;
  while( w_time < 100 ) {
    while ( Serial2.available() ) {
      rx_buf[cntr] = Serial2.read();
      ++cntr;
      w_time = 0; // reset timer since data arrived
    }
  }
}

It would probably help to have more information. Things like you say you use both Serial1 and Serial2... How?
I assume w_time is an elapsedmillis object? Is there any code that is processing interrupts...

The reason ask about multiple objects. If you have code hard spinning for 100ms, at a baud rate of 115200, you could in theory receive something like 1152 bytes of data over the serial port.

Sometimes I setup code like this to avoid multiple calls per character. Something like:

if( Serial2.available() ) {
  w_time = 0;
  int ich;
  while( w_time < 100 ) {
    while ( (ich = Serial2.read()) != -1 ) {
      rx_buf[cntr] = ich;
      ++cntr;
      w_time = 0; // reset timer since data arrived
    }
  }
}

But again not sure how much time this saves you. Again not sure what else to suggest. If your code on other side does bursts of data, maybe change timeout to something like 5 instead of 100...

Yes w_time is an elapsedMillis() variable.
The routine is all alone, there is nothing else going on. when the main routine sees there is data available on Serial 1, this routine runs and and finishes when no more data arrives. In the next chunk of code not shown, it simply goes out from the buffer to the Serial USB port.
I'll try anything!
Thx

 

[Frank B]

Oh, yes, your're right Kurt !

 

[defragster]

Are those '167 to 470 bytes,' bytes contiguous - without error or missing characters - from the start of the transmission?

KurtE is right on the timing - 70 us is all it takes to get a character at 14400 Bytes/sec - so your 100 could work with elapsedMicros? Or 1 ms. If you wait too long to timeout you will miss any gap between messages.


If I were going to try this - I've had good results with serialEvent() - it is called on every exit of loop() or on any delay() { OR yield() } when there are characters. It is as good as an interrupt function - acting on input only when available and outside user code - but is not an interrupt.

In Loop just watch elapsedMillis w_time. If it is over 1 and less than 100 you just got a string completed, set w_time to 101 and deal with the string, set ich=0. If you had a terminating 'signal character' you could test and set a flag in serialEvent2().

Allow w_time to run and ignore it over 100, and when over 10,000 reset it to 101, just so it won't rollover and give a false reading

static int ich=0;  // file global, reset to 0 in loop

serialEvent2() {
  if( Serial2.available() ) {
Serial.print( "Se2 @"); // debug
Serial.print( w_time ); // debug
    w_time = 0;
    while ( (ich = Serial2.read()) ) {   [B][U]// Took off the -1 test ??????[/U][/B]
      rx_buf[cntr] = ich;
      ++cntr;
      w_time = 0; // reset timer since data arrived
    }
Serial.print( "  cntr="); // debug
Serial.println( cntr ); // debug
  }
}

 

[mikeg]

thx, I'll work on this,

 

[defragster]

This works for me - exactly as expected- when I put a wire on my T_3.6 from pin 9 to pin 10.

Note - the problem for you perhaps was the while loop watching the character value? >> while ( (ich = Serial2.read()) != -1 )

But this works as I expected/intended in post #7 and seems nice.

On a single Teensy - the Rx is wired to Tx - and the yield() results in a call to serialEvent2() - while in between the prints and prevents buffer overflow with standard buffers. This is what I was saying about serialEvent being so cool - like interrupt code. As long as your loop() cycles fast enough - or you do a yield() or delay() when you are in a busy loop somewhere in your code - without having to actually call your Serial Update code.

I have 2 yield() calls and then exit loop and it looks like the rest of the buffer must be transferring after exiting loop - as it takes 31 calls to serialEvent2() to process the 126 characters. Except for the last calls - I see FOUR characters read per call to SerialEvent2()?

Here is sample output from one pass - this repeats every 2 seconds
The @# is the value of w_time and the #=### is the count of chars in the string on leaving serialEvent2()::

@2001 #=4,@0 #=8,@1 #=12,@0 #=16,@0 #=20,@1 #=24,@0 #=28,@0 #=32,@1 #=36,@0 #=40,@0 #=44,@1 #=48,@0 #=52,@0 #=56,@1 #=60,@0 #=64,@0 #=68,@1 #=72,@0 #=76,@0 #=80,@1 #=84,@0 #=88,@1 #=92,@0 #=96,@0 #=100,@1 #=104,@0 #=108,@0 #=112,@1 #=116,@0 #=120,@0 #=124,
abcdefghijklmnopqrstuvwxyz<- 1 ->ABCDEFGHIJKLMNOPQRSTUVWXYZ_abcdefghijklmnopqrstuvwxyz<- 2 ->ABCDEFGHIJKLMNOPQRSTUVWXYZ_4328

// https://forum.pjrc.com/threads/39413-Help-with-Storing-incoming-serial2-data-in-buffer-at-115200-baud-with-teensy-3-2-s?p=122854#post122854

#define qBlink() (digitalWriteFast(LED_BUILTIN, !digitalReadFast(LED_BUILTIN) ))

static int cntr = 0; // file global, reset to 0 in loop
elapsedMillis w_time;
char rx_buf[512];


void setup() {
  // put your setup code here, to run once:
  Serial.begin(115200);
  Serial2.begin(115200);
  pinMode(LED_BUILTIN, OUTPUT);
  digitalWriteFast(LED_BUILTIN, 0);
  while (!Serial && (millis ()  <= 8000)) ;
}

void loop() {
  // put your main code here, to run repeatedly:
  if ( w_time > 100 ) {
    if ( w_time > 2000 )
    {
      w_time = 101;
      Serial2.print( "abcdefghijklmnopqrstuvwxyz<- 1 ->");
      Serial2.print( "ABCDEFGHIJKLMNOPQRSTUVWXYZ_");
      yield();
      Serial2.print( "abcdefghijklmnopqrstuvwxyz<- 2 ->");
      Serial2.print( "ABCDEFGHIJKLMNOPQRSTUVWXYZ_");
      yield();
      Serial2.print( millis() );
    }
  }
  else if ( w_time > 10 ) {
    w_time = 101;
    cntr = 0;
    Serial.println( );
    Serial.println( rx_buf );
  }
}

void serialEvent2() {
  char ich;
  if ( Serial2.available() ) {
    qBlink();
    Serial.print( "@"); // debug
    Serial.print( w_time ); // debug
    w_time = 0;
    while ( Serial2.available() ) {   // Took off the -1 test ??????
      ich = Serial2.read();
      rx_buf[cntr] = ich;
      ++cntr;
      w_time = 0; // reset timer since data arrived
    }
    Serial.print( " #="); // debug
    Serial.print( cntr ); // debug
    Serial.print( ","); // debug
    qBlink();
  }
}

<edit> Corrected a non-critical typo that compiled - to :: w_time = 101;

 

[defragster]

As you can see at most 1 ms passes between chars coming in. I could rewrite it to use microseconds but I've had enough fun with this for now

I did make this change from 10 to 1 and it still works to detect the idle time between messages when no new characters come in for 2 ms. If there was other code going on then that might need a bigger time buffer.

  else if ( [U]w_time > 1[/U] ) {

 

[mikeg]

so in reality, my problem is not that the buffer overflows after I begin to empty it, but in the main loop in which the Serial2.available() lives. The buffer has already overflowed before the loop reaches the available call. That explains the variation in the number of missing characters.
Thanks

 

[KurtE]

Hi Mike,

It is really hard to say what is wrong, with your program as we know very little about what all it is doing. There are lots of things that could influence this. Things like:

a) You have some code that processes interrupts and the interrupt handler is at a higher priority then the Serial2 interrupt handler and it takes a long time in the handler. Ways to fix this type of issue include, raise the priority of the Serial interrupt. Change the other interrupt handler to not delay in it...

b) could be your program is hanging/looping a long time in some other place: examples could be you are doing output or input from a device that for example you write a large buffer to serial port, example: Serial1.write(my_buf, 5000) This call will wait until all 5000 bytes can be at least processed. So if Serial port write buffer is lets say 100 bytes long, the code will have to hang about the time it takes for that port to output 4900 bytes out the serial port. Fixes for this include using your own large output buffer, and have your main loop not try to output more to the Serial port than there is room for, by calling Serial1lavailableForWrite(). Or have your serial input code in serialevent proceedures as the output loop will most likely be calling yield...

c) you have some delays, maybe waiting for hardware to do something. So example if you have a delay(500), you won't be checking your ports for half second... However Teensy3 core code (unlike Arduino) is setup that inside of the delay call it loops until the right time, calling yield. What the default yield code does is to check all of the Serial ports for bytes available and if there are some call off to SerialEvent...

So again like I believe defragster mentioned, if you setup your different Serial input processing to be in the functions, like: serialEvent2
Then many of these cases can be handled. However don't put delays (or only minimal delays) into the serialEvent functions, as yield is setup to not keep recursively calling the serial event code...

 

[mikeg]

Hi Mike,

It is really hard to say what is wrong, with your program as we know very little about what all it is doing. There are lots of things that could influence this. Things like:

a) You have some code that processes interrupts and the interrupt handler is at a higher priority then the Serial2 interrupt handler and it takes a long time in the handler. Ways to fix this type of issue include, raise the priority of the Serial interrupt. Change the other interrupt handler to not delay in it...

b) could be your program is hanging/looping a long time in some other place: examples could be you are doing output or input from a device that for example you write a large buffer to serial port, example: Serial1.write(my_buf, 5000) This call will wait until all 5000 bytes can be at least processed. So if Serial port write buffer is lets say 100 bytes long, the code will have to hang about the time it takes for that port to output 4900 bytes out the serial port. Fixes for this include using your own large output buffer, and have your main loop not try to output more to the Serial port than there is room for, by calling Serial1lavailableForWrite(). Or have your serial input code in serialevent proceedures as the output loop will most likely be calling yield...

c) you have some delays, maybe waiting for hardware to do something. So example if you have a delay(500), you won't be checking your ports for half second... However Teensy3 core code (unlike Arduino) is setup that inside of the delay call it loops until the right time, calling yield. What the default yield code does is to check all of the Serial ports for bytes available and if there are some call off to SerialEvent...

So again like I believe defragster mentioned, if you setup your different Serial input processing to be in the functions, like: serialEvent2
Then many of these cases can be handled. However don't put delays (or only minimal delays) into the serialEvent functions, as yield is setup to not keep recursively calling the serial event code...

Actually, the main loop in which this bit runs takes less than 100 ms to complete. There are currently no interrupts, and only the one elapsedMillis() running. When the main loop code reaches the point where it checks the Serial21.available(), that code runs until the w_time runs out. I am not writing to any other ports, or doing anything until this completes. It is blocking all other actions, thus my opinion that the buffer overflows prior to the main loop arriving at Serial2.available(). Until I redesign the rest of the code in other routines (not running at this time) that do use serial1 and serial2, I'll have to limit the transfer size.
Thanks all for your help

 

[KurtE]

You are welcome and Good Luck!

Kurt

P.S. - As one who has many times in the past debugged the wrong thing :lol: I would also always want to verify that the expected output from the other processor actually arrived at this processor. So if it were me, I would have one of my Saleae logic analyzers hooked up to the RX2 pin and verify that it sees the data that you are expecting. Example maybe there is not a proper ground between the two processors, or the wire is not making good contact...

 

[mikeg]

You are welcome and Good Luck!

Kurt

P.S. - As one who has many times in the past debugged the wrong thing :lol: I would also always want to verify that the expected output from the other processor actually arrived at this processor. So if it were me, I would have one of my Saleae logic analyzers hooked up to the RX2 pin and verify that it sees the data that you are expecting. Example maybe there is not a proper ground between the two processors, or the wire is not making good contact...

Yep, did that. good old scope and Saleae 8 channel logic pod. the data I am sending from the second teensy is exactly what I told it to send, and its getting to the input pin just fine. I am a old time troubleshooter (both age and experience) , but sort of new to coding, so I muddle by. Interrupts and threading are not things I use much, but need to spend more time with. I'm going to do that here!
Thanks for all your help.

 

[defragster]

The code in post #9 should work wired over to the other Teensy. Did you try that? That would show the data arrives.

Just take out the 7 .print and yield() statements after:

if ( w_time > 2000 )
{
w_time = 101;

100 ms is a long time - 100 us would be cool and that is longer than one character even at the slow baud rate of 14400 Bps - IIRC loop should cycle multiple thousands of times per second - less any real work and no delays.

 

[mikeg]

The code in post #9 should work wired over to the other Teensy. Did you try that? That would show the data arrives.

Just take out the 7 .print and yield() statements after:


100 ms is a long time - 100 us would be cool and that is longer than one character even at the slow baud rate of 14400 Bps - IIRC loop should cycle multiple thousands of times per second - less any real work and no delays.

I actually have to handle serial data from televisions, in which the control protocols vary from 1200 baud to 115200, and may be TTL level or true RS232. We make a set top box that lives between the TV, the cable system and the IP network, and we control all functions in the TV's (power volume, input, mute.....) over the serial control ports built into hospital and commercial grade TV's. The whole purpose of the project is to relieve the Set top box of all the details of 7 different protocols by presenting one API to the set top box, and translate that to the particular protocol the specific tv speaks. That code is all done, but we also want a serial pass thru mode that will allow large block transfers for other things some tv's do. A card based on the teensy 3.2 chip set will live inside the set top box and do the translation. it is all quite slow with these tvs, and the buffers built into the teensy now support the less than 48 byte control packets back and forth without issue. To add longer packets at higher speeds will force a rethink on the serial comm part of the code for Serial1 and serial 2. The set top box comm are on the USB serial port, which seems to be able to handle anything i send to it, and large packets coming in have not been an issue either, for some reason.

 

[Frank B]

Is it possible to use hardware-handshake ? (RTS/CTS)

 

[mikeg]

Is it possible to use hardware-handshake ? (RTS/CTS)

Nope, some TV's use a specific byte representing the number of bytes sent, some do actually use lf or cr for command termination. Some require me to decode the first part of the command to know how many bytes will follow. As with many devices, manufacturers won't use a common language, or even a common electrical protocol. It's all TIA-EIA 232 data wise, but some is TTL level, some will work with TTL or +/-232 levels, and some have to be +/-232 levels. One even has a current loop similar to MIDI!

 

[Frank B]

I don't understand. You wrote in your first post:

I am having trouble storing incoming serial data in a Teensy 3.2 (coming from another Teensy 3.2.) in a buffer at 115200 baud.

So, rts/cts is possible, if you have two free pins?

 

[mikeg]

I don't understand. You wrote in your first post:


So, rts/cts is possible, if you have two free pins?

Sorry, that was for explaining my testing setup, I was using a teensy to simulate a TV sending large blocks of data, as you have seen above, the problem is not with the sending but the receiving code.

 

[defragster]

I actually have to handle serial data from televisions, in which the control protocols vary from 1200 baud to 115200, and may be TTL level or true RS232. We make a set top box that lives between the TV, the cable system and the IP network, and we control all functions in the TV's (power volume, input, mute.....) over the serial control ports built into hospital and commercial grade TV's. The whole purpose of the project is to relieve the Set top box of all the details of 7 different protocols by presenting one API to the set top box, and translate that to the particular protocol the specific tv speaks. That code is all done, but we also want a serial pass thru mode that will allow large block transfers for other things some tv's do. A card based on the teensy 3.2 chip set will live inside the set top box and do the translation. it is all quite slow with these tvs, and the buffers built into the teensy now support the less than 48 byte control packets back and forth without issue. To add longer packets at higher speeds will force a rethink on the serial comm part of the code for Serial1 and serial 2. The set top box comm are on the USB serial port, which seems to be able to handle anything i send to it, and large packets coming in have not been an issue either, for some reason.

I'm not sure how to read this? Does the post #9 code work alone or not - as noted in post #16 - to capture the variety of output you expect to see? Once you see that work - or get it modified to work - as it did in my simulation - then you just need to add that to your other code and keep it running. If you extend the buffers as long as you empty them faster than the next message arrives there should be no loss - and the messages are seen as ended when an appropriate 'empty time' is seen - at 1200 bps that will be longer than above. But with adjustments and those prints included you'll see the timing. Of course I ran that on a T_3.6 at 180 MHz or above - I have a T_3 here I could try it on at 1200 baud and get that to work.

 

[defragster]

Running perfectly on my T_3.0 at 1200 baud - with the code posted above - using the same cross wiring of pins 9&10 on Serial2 on the single T_3.0 at 96 MHz printing to itself with standard included buffer sizes. With this change in setup():: Serial2.begin(1200);

It takes almost all of the 11 ms represented by :: else if ( w_time > 10 )

The prints in serialEvent2 show characters coming in one at a time at 8&9 milliseconds - wow 1200 baud is slow!

The good thing is the slower baud rate will be less time critical. But if a message split time of 10 ms works then both baud rates have the same tolerance, but adding yield() and removing any delay() will be more critical at 14400 Bytes/sec than 150 Bytes/sec.

I just added a loop count to this code and using elapsedMillis to keep the loop() running the cycles of:
loop() in one second at 96 MHz is :: 199,651
unless it is the pass where the internal prints to Serial2 are done once each 2 seconds then the count drops to :: 58,234 because I am printing out 126 characters to a 40 byte Transmit buffer which causes the code to block waiting for enough transmits to occur to buffer the rest and return. I just set "SERIAL2_TX_BUFFER_SIZE 140" and now the lowest count of loop() per second is :: 199,064.