Teensy3 freezes (1) sending >8 bytes (2) with interrupts active (3) on Serial1

Here is a problem I discovered with my (complex) Oscilloscope sketch.
http://www.mccauslandcenter.sc.edu/CRNL/tools/oscilloscope
but it is more easily demonstrated in this simple demo where the Teensy3 can send data to processing
http://www.mccauslandcenter.sc.edu/CRNL/sw/oscilloscope/bt_test.zip
In fact, you do not even need processing, as the Teensy freezes up (the light stops blinking) when you upload a sketch where
gSampleHz > 0 (interrupts running)
gBytesPerSample > 8 (large packets)
and "#define ENABLE_SERIAL1" UNcommented.
You do not even need a serial device connected to Serial1 to observe this problem. On my OSX computer, setting gSampleHz = 0 (e.g. no interrupts) I am able to transfer several thousand samples per second with either 8 or 10 bytes per sample, and with gSampleHz = 1000 and gBytesPerSample=8 I can send precisely 1kHz of 8 bit samples with either the wired or wireless. However, with gSampleHz = 100; and gBytesPerSample = 10; the system locks up. I can then comment "#define ENABLE_SERIAL1" to read "//#define ENABLE_SERIAL1"" and the Teensy will happily send 10 bytes per sample at 100 Hz (but only on the wired connection).


Since the program locks up with or without my JY_MCU attached, it seems to be independent of the serial device.


I know this is appears a pretty obscure bug, and in general the T3 is a terrific platform. However, this one has really puzzled me. I have tried changing the interrupt timer to 0,1,2 (wondering if the PIT is also used for Serial timing) to no avail.

Here is the sketch
--------------------

const int gSampleHz = 100; //set to 0 for as fast as possible, else sets samples per second, e.g. 1000 = 1kHz
const int gBytesPerSample = 8;
#define ENABLE_SERIAL1 //Teensy will freeze when gSampleHz>0, gBytesPerSample>8 and this line is UNCOMMENTED

/*
This simple sketch demonstrates that the Teensy3 locks up when trying to transmit more than 8 bytes via Serial1
To demonstrate, use PROCESSING script bt_RX_Processing to either connect to the wired Serial port, or to a Bluetooth module set up on Serial1

Serial ALWAYS Works
gSampleHz = 0; gBytesPerSample = 8; //transmits 8 bytes as fast as possible
gSampleHz = 1000; gBytesPerSample = 8; //transmits 8 bytes at 1kHz
gSampleHz = 0; gBytesPerSample = 10; //transmits 10 bytes as fast as possible
gSampleHz = 1000; gBytesPerSample = 10; //transmits 10 bytes at 1kHz
Serial FAILS when more than 8 bytes are sent with interrupts enabled
gSampleHz = 0; gBytesPerSample = 8; //transmits 8 bytes as fast as possible
gSampleHz = 1000; gBytesPerSample = 8; //transmits 8 bytes at 1kHz
gSampleHz = 0; gBytesPerSample = 10; //transmits 10 bytes as fast as possible
gSampleHz = 1000; gBytesPerSample = 10; //FAILS transmits 10 bytes at 1kHz
gSampleHz = 100; gBytesPerSample = 16; //FAILS even though 20% data transfer of 1000Hz * 8 bytes!!!!
*/

//************ no need to edit lines below here *******
const int kOutLEDpin = 13; //location of in-built light emitting diode - 11 for Teensy2, 13 for Arduino/Teensy3
#define BAUD_RATE 230400 // 230400 //921600 //460800//115200 is the max for the Uno - Teensy and Leonardo use direct for much higher speeds
byte gSerialBytes[gBytesPerSample];


void sendOsc(void) {
Serial.write(gSerialBytes, gBytesPerSample);
#ifdef ENABLE_SERIAL1
Serial1.write(gSerialBytes, gBytesPerSample);
#endif
}

//use periodic interrupt timer, PIT http://forum.pjrc.com/threads/14-Teensy-3-0-and-interrupts
// http://www.pjrc.com/teensy/datasheets.html
#define PIT_LDVAL (PIT_LDVAL##n)
#define PIT_TCTRL (PIT_TCTRL##n)
#define IRQ_PIT_CH (IRQ_PIT_CH##n)
#define PIT_TFLG (PIT_TFLG##n)
#define clearPIT {PIT_TFLG = 1;}
#define startPITFreq(n, freq) {cli(); PIT_LDVAL = (F_BUS / (freq))-1; PIT_TCTRL = 3; NVIC_ENABLE_IRQ(IRQ_PIT_CH); sei();}
#define stopPIT {cli(); NVIC_DISABLE_IRQ(IRQ_PIT_CH); PIT_TCTRL = 0; sei();}

void pit0_isr(void) { //ARM interrupt
clearPIT(0) ;
sendOsc();
}

void timer_setup() { //setup ARM interupts
startPITFreq(0,gSampleHz);
}

void timer_stop() {
stopPIT(0);
}

void setup()
{
pinMode(kOutLEDpin, OUTPUT); //turn analog status light on
for (int i = 0; i < gBytesPerSample; i++)
gSerialBytes = i;
Serial.begin(BAUD_RATE);
#ifdef ENABLE_SERIAL1
Serial1.begin(BAUD_RATE);
#endif
if (gSampleHz > 0) {
timer_setup();
}
} //setup()

void loop() {
if (gSampleHz < 1) {
sendOsc();
}
int modulo = millis() % 1000;
if (modulo == 1) digitalWrite(kOutLEDpin, HIGH);
if (modulo == 500) digitalWrite(kOutLEDpin, LOW);
} //loop()

I'm working on this now. It's definitely a bug in my serial driver code. I've got a board hooked up to my oscilloscope and I'm able to reproduce the problem and figure out where it's getting stuck, and the events leading up to that.

This is a tough one. It might take me a while. Just want to let you know I am working on it.

Thank you for finding the bug criab. I likely would have encountered it in a few weeks (and been at a complete loss).

Please give these files a try. They go into hardware/teensy/cores/teensy3.

I improved the management of when transmit interrupts are enabled and the writing of data into the transmit FIFO. I believe this fixes the bug.

I also added block write optimization, which is used automatically when you use Serial1.write(buffer, size) or Serial1.print(). This optimization reduces the CPU overhead (at least within Serial1) to approx 1.5 us per byte, which should help you use the faster baud rates.

Keep in mind the transit buffer is 64 bytes. If you transmit more data to Serial1 than will fit in the buffer, Serial1.write() waits for more space in the buffer. If your interrupt prevents the serial interrupt from running, Serial1.write() will wait forever. For this application, where you transmit a fixed size block of data at a regular interval, it should be fine. Just know that it is expected behavior to wait for space in the buffer if your write does not fit.

Confirmed, this fixed the bug.

Teensyduino 1.14 will include this fix.

Much appreciated.