Changing hardware serial buffer size

Status
Not open for further replies.
M

mborgerson

Well-known member
I suspect that this topic has been discussed before, but a forum search didn't get me a good answer.

I have a project that would be greatly simplified if I could expand the hardware serial receive buffer size for Serial1 to 32768 bytes and the transmit size for Serial2 to 2048 bytes.

The project looks like this:

1. Receive serial data from an oceanographic instrument in 36-byte packets which are sent every 10mSec at 115KBaud.
The data is sent continuously and no flow control is possible, so the Teensy (a 3.6) has to be able to keep up with the data.
2. When 500 packets have been received, (18,000 bytes), convert the data from 16-bit integers to 32-bit floating point using calibration equations.
3. Compute means, variances, and spectral data using FFTs. Stuff these results into a packet of about 1KBytes length and save the packets to SD card.
4. Transmit the packets at 9600 baud to a host system which will send them back to the real world from the wet end of nowhere via satellite.
5. Go back to Step 1 until batteries die or a research vessel retrieves the system (weeks to months later).

There is nothing too difficult in this with a loop like:

loop(){
Get500Packets();
Process500Packets(); // I've got scientists and engineers to write and debug this code
StoreResults();
SendResults();
}

but this only works if the serial buffer doesn't overflow between calls to Get500Packets(). Since processing, storing and sending the data could
take a second or two, this won't work with a small hardware serial buffer. Teensy 3.6 has lots of RAM for data arrays and packet buffers, but
expanding the buffers used by the serial interrupt handlers apparently isn't straightforward. This could probably be done with a timer callback
that checks the Serial1 for data every 10mSec and fills secondary buffer, but that seems more complex than necessary.

I've looked on other Arduino fora, and it seems that changing the serial buffer size is not as simple as changing a #define in user code space.
What is the best way to do this for the Teensy?

I could do this project with ChiBios on an STM-32 system, but I'm working with people that have much more Arduino experience than RTOS experience.
 
The only method I know to do this is to edit serial1.c and serial2.c in the Teensy hardware folder. In your arduino install directory it will be hardware/teensy/avr/cores/teensy3/serial1.c and hardware/teensy/avr/cores/teensy3/serial2.c

At the top of the file you'll see defines for the buffer sizes.
 
teensy should be fast enough to not need a 32KB buffer, but i do suggest if your receiving large bursts to start using SerialEvents function
 
tonton81 said:
teensy should be fast enough to not need a 32KB buffer, but i do suggest if your receiving large bursts to start using SerialEvents function
Click to expand...

As noted above, there won't be large bursts, but rather continuous bursts of 36 bytes every 10 milliseconds. I will look into SerialEvents and other mechanisms to synchronize the reception of the incoming packets. The packets will have a leading ID word and a terminating checksum, so that I can make sure the incoming packet handler is properly synchronized and the binary data ends up in the right place in the packet structure. I would like to avoid a finite state machine having to examine each byte read from the serial port and be able to do a Serial1.readbytes(datastructptr, sizeof(datastructptr)).

I could probably get by with a smaller buffer, but I want to keep it as an even power of 2 so that ring buffer rollover can be handled most efficiently. When I know how long the calculations will actually take to complete, I may be able to
reduce the buffer size to 16384 or 8192, but I want to make sure to have enough excess buffer capacity to handle extended cycle times when the write to the SD card takes longer than usual to complete. That will happen when the FAT has to be updated and when the SD card may have to erase a new 32KB storage block.
 
You might be overthinking things a bit for only 36 bytes every 10 ms. Teensy 3.6 can handle dramatically faster data than this. Unless you program is structured in a rather unusual way where you need to tie up the CPU for lengthy times, this is such a slow rate that the defaults are probably fine.

I would like to avoid a finite state machine having to examine each byte read from the serial port and be able to do a Serial1.readbytes(datastructptr, sizeof(datastructptr)).
Click to expand...

At the very least, you should deal with getting less than the data you wanted. It's easy if you just dedicate a variable like "bytecount", like this:

Code: 
bytecount += Serial1.readbytes(datastructptr + bytecount, sizeof(datastructptr) - bytecount)

Remember readBytes uses a timeout, so you might wish to zero the timeout to zero if you don't want to stall the rest of your program. Or maybe you can use the timeout to help you detect when the end of a packet happens.
 
Isn’t the first post saying the cpu will be tied up for lengthy periods of time?

I’m interested to hear the solution for the case described where the cpu is doing some 2 seconds of work at regular intervals.

Without interrupt based serial handling, under what conditions does serialEvent read the serial buffer.
Every loop()
Every delay()
Every serial.print()

Am I understanding the mborgersons problem correctly?
 
i do believe paul implemented it in other functions as well, but the best method of capture is just using SerialEvent itself rather than checking every loop by polling, that function is called when at least 1 byte enters the buffer so you can take care of it right away. like paul said, if your code is causing too much time “processing” and the FIFO’s get full, youll start loosing bytes, SerialEvents is there for a reason

Tony
 
I tried expanding the serial input buffer by editing serial1.c, which resides in C://Program Files(x86)/hardware/teensy/avr/cores/teensy3/serial1.c.. I changed the SERIAL1_RX_BUFFER_SIZE definition to 32768. I verified that the compiler actually gets to that code by inserting a temporary error just after the new definition. At first, I was seeing no change in the serial1 input buffer size. Serial1.c was compiling, but the new code wasn't making it to the executable.

Since I'm not familiar with the Arduino build process, it took a couple of hours to track down the problem. Being a cautious programmer, I saved a copy of the original serial1.c in the folder as original_serial1.c. After tracking down the object files, I found that original_serial1.c was being compiled. Rather than explicitly naming the files in the build, it appears that the build process compiles everything in the hardware/teensy/avr/cores/teensy3 folder. As a result I had two object files with all the serial1 routines. The linker apparently liked the original files better than my edited code and was building and linking to the original code.

When I got things straightened out by renaming original_serial1.c to original_serial1.cxx, it was immediately apparent that the serial buffer size had changed since the global variable size had increased to accommodate the larger buffer. I connected an instrument simulator sending packets (an Arduino project on an STM32F4 Nucleo card) and the code ran and an appropriate number of bytes was available in the serial1 input buffer after the simulated computation and storage delays. I'll expand the attached code to verify that no missing bytes occur after I add the code for actual SDC storage.

Code: 
/* UART Receive test to verify 32KB serial buffer enabled
   M. Borgerson   2/2/2018
*/

// set this to the hardware serial port you wish to use
#include "HardwareSerial.h"
#define HWSERIAL Serial1

uint16_t Serial1RxSize;

struct cpackettype {
  uint16_t pktid;
  uint16_t timerticks;
  uint32_t unixseconds;
  uint16_t cdata[8];
  uint16_t checksum;
};

// 36 bytes  10 for time, ticks, marker and checksum
// 26 bytes for 13 data channels

// for now, output packet is just an array of 64 floats
struct opackettype {
  float odata[64];
};

struct cpackettype  inputdata[500];
struct opackettype odata;


void Get500Packets(struct cpackettype * cptr){
static uint16_t pktcount; 
struct cpackettype *iptr;
  iptr = cptr;
  pktcount = 0;
  // later, a synchronization check will go here
  do {
    // if there's enough data in serial buffer for a packet, read it
    if(HWSERIAL.available() > sizeof(cpackettype)){
      HWSERIAL.readBytes((unsigned char *)iptr, sizeof(cpackettype)); 
      iptr++; 
      pktcount++;
    }
  } while(pktcount < 500);
  // at return 500 new packets have been stored for analysis
}


// Process the input data and fill the output data structure
// for now, just send some descriptive text to serial
void Process500Packets(struct cpackettype * cptr, struct opackettype *optr){
static uint16_t pcount;

  // just delay a bit for now--- all the math will be done as code is developed.
  delay(2000);

  pcount++;
}

void StoreResults(struct opackettype *optr){
  static  uint16_t outcount = 0;
  static  uint16_t inputavailable;
  // No storage yet, just a bit of delay
  // with an occasional longer delay to simulate the 
  // variable timing of SD card writes
  outcount++;
  if(outcount >10){ 
    delay(500);  // add an extra half second
    Serial.println("Extra SDC delay");
    outcount = 0;
  }
  delay(10); // normal SD write only about 10mSec
  // display the amount of data accumulated in serial input buffer
  // after the delay used in lieu of actual calculations sand SDC write
  inputavailable = HWSERIAL.available();
  Serial.print(inputavailable);
  Serial.println("  Bytes available in serial input buffer");
}

void setup() {
  srand(373);  // Use same random sequence for testing
	Serial.begin(9600);
	HWSERIAL.begin(115200);
  Serial.println("Starting Serial input test");
  Serial.print("Serial1 RX Buffer size is ");
  Serial.println(Serial1RxSize );
  delay(500);
}

void loop() {
  Get500Packets(&inputdata[0]);
  Process500Packets(&inputdata[0], &odata);
  StoreResults(&odata);
}
 
look at all those delays! most being 500ms, theres even one at 2secs! I hope you add SerialEvent, you might need it ;) You didnt need to re-engineer the original serial1.c file, just changing the value inside of it was all thats needed...
 
tonton81 said:
look at all those delays! most being 500ms, theres even one at 2secs! I hope you add SerialEvent, you might need it ;) You didnt need to re-engineer the original serial1.c file, just changing the value inside of it was all thats needed...
Click to expand...

Basically, all I did is change the serial1 receive buffer length in the file. Win10 makes it a bit difficult by putting the core source in the Program Files(x86) directory, which does not normally allow editing of the files.

Most of the delays will disappear in the final code, to be replaced by many calculations, which may take up to a few seconds.
 
mborgerson said:
Basically, all I did is change the serial1 receive buffer length in the file. Win10 makes it a bit difficult by putting the core source in the Program Files(x86) directory, which does not normally allow editing of the files.

Most of the delays will disappear in the final code, to be replaced by many calculations, which may take up to a few seconds.
Click to expand...

That doesn't happen using the ZIP version the IDE - get that then unzip to the chosen location. Point TeensyInstaller there and it works well. You can run multiple installs over time to unique directories if desired to test new/alternate versions of IDE or TeensyDuino.
 
do like i do, add urself permissions in the arduino IDE folder in program files folder, this allows me access to edit any file in the arduino IDE
 
Status
Not open for further replies.
Back
Top