Hardware or Core Issue? Serial1 UART freezes when 1kHz SPI ISR starts (Teensy 3.6)

[Peycool]

I am experiencing a hard freeze on Serial1 reception whenever an external SPI module goes active, and after trying several software fixes, I suspect it might be a hardware or low-level core conflict.

The Hardware Setup:

• Master: Teensy 3.6
• Slave: Teensy 3.6
• Comms 1 (Master to Shield): Master has an EasyCAT (EtherCAT) shield communicating via SPI. When my ROS network starts, the shield pulses a hardware interrupt on Pin 2 at 1kHz (every 1ms).
• Comms 2 (Slave to Master): Master listens to the Slave via Serial1 (Pins 0/1) at 2,000,000 baud. The Slave sends 16-byte payloads of encoder data.

The Problem: Before the ROS network is launched, the Master reads the 2Mbaud data from the Slave perfectly. However, the exact second ROS is launched, and the EasyCAT shield begins firing the 1kHz hardware interrupt on Pin 2 to trigger SPI transfers, the Master completely stops receiving updated values from Serial1. The serial buffer either overflows, gets corrupted, or completely freezes. The Slave is still transmitting perfectly.

What I Have Already Tried (Software Fixes Failed):

1. Lowering the Baud Rate: I dropped the baud rate from 2,000,000 down to 250,000. It did not solve the freeze.
2. Removing ISR Blocking (The Flag Method): I initially had the heavy SPI transfers running directly inside the Pin 2 ISR. I rewrote the code so the ISR only flips a volatile bool flag, moving all the SPI processing into the main loop() to avoid starving the CPU. This did not fix the issue.
3. Removing Handshakes: I removed all while(timeout) blocking loops from the UART parsing logic.

Here is the core structure of my Master code that is currently failing:

#include <SPI.h>
#include "EasyCAT.h"

EasyCAT EASYCAT(9, DC_SYNC);
const byte EtherCAT_InterruptPin = 2;

void setup() {
  Serial1.begin(2000000); // Slave UART
  attachInterrupt(digitalPinToInterrupt(EtherCAT_InterruptPin), EtherCAT_DC_ISR, FALLING);
}

void loop() {
  // Flag method did not solve the freeze,
  // currently returning to direct ISR calls or polling in loop.
  tickCountDownload(); // My function to read Serial1.available()
}

void EtherCAT_DC_ISR() {
  // Even if I move these to the loop() using a flag, Serial1 still freezes when 1kHz traffic starts
  EASYCAT.MainTaskP1();
  updateMotors();
  EASYCAT.MainTaskP2();
}

Since software-level ISR management didn't fix this, I am looking for hardware or low-level insights:

1. Hardware/Electrical: Could the EasyCAT shield going fully active at 1kHz be causing a current spike, ground bounce, or electrical noise that is temporarily disabling the Serial1 RX hardware?
2. Core Conflicts: Is there a known low-level conflict on the Teensy 3.6 MK66FX1M0 between the SPI clock generation and the UART1 (Serial1) interrupts or DMA channels?
3. Hardware Serial Buffers: Does a 1kHz external interrupt natively block the background UART buffer filling on Teensy 3.6, even if the ISR is incredibly short?

Any advice on how to physically or programmatically decouple these two systems would be highly appreciated.

 

[joepasquariello]

Have you tried just having an ISR that simply returns? If your serial code still works with that ISR, then the issue is not the interrupt itself, but what you do in the interrupt, which seems more likely. I suspect you'll be able to show that the problem occurs only when you execute the EasyCAT part of your program. I think it's better to keep your ISR as short as possible and execute the EasyCAT functions at task level. As for question #3, no, if your ISR is short, a 1-kHz interrupt should not get in the way of serial interrupt processing. At 2 MHz, the serial interrupt rate would be about 200 kHz, which is pretty high, so you either have to keep your EasyCAT ISR very short, or make sure the UART interrupt priority is higher than the EasyCAT interrupt priority.

 

[Peycool]

Have you tried just having an ISR that simply returns? If your serial code still works with that ISR, then the issue is not the interrupt itself, but what you do in the interrupt, which seems more likely. I suspect you'll be able to show that the problem occurs only when you execute the EasyCAT part of your program. I think it's better to keep your ISR as short as possible and execute the EasyCAT functions at task level. As for question #3, no, if your ISR is short, a 1-kHz interrupt should not get in the way of serial interrupt processing. At 2 MHz, the serial interrupt rate would be about 200 kHz, which is pretty high, so you either have to keep your EasyCAT ISR very short, or make sure the UART interrupt priority is higher than the EasyCAT interrupt priority.

With void EtherCAT_DC_ISR() { return; }, the Serial1 reception at 2,000,000 baud works perfectly even while the ROS Master is actively sending 1kHz traffic.

The UART freeze only occurs when the EASYCAT.MainTask() (SPI-based) is actually executed. I calculated that at 2Mbaud, the default 64-byte RX buffer overflows in ~256us. It appears the SPI overhead is heavily exceeding this window and starving the UART interrupt.

To combat this, I implemented Serial1.addMemoryForRead() to increase the RX buffer to 1024 bytes (which should give ~4ms of breathing room). I also moved the SPI tasks out of the ISR and into the main loop() via a volatile flag. My UART parsing logic is completely non-blocking (just continuously checking Serial1.available()).
However, this did not solve the issue. When the SPI tasks execute, the UART stream still freezes, as if the buffer expansion is being ignored or overridden.

Given the 1kHz trigger rate, are there any known issues with SPI bus contention or DMA conflicts on the Teensy 3.6 that would still block UART RX interrupts even if the buffer is enlarged? Or should the larger buffer theoretically be sufficient to 'swallow' the SPI execution time?

 

[joepasquariello]

With void EtherCAT_DC_ISR() { return; }, the Serial1 reception at 2,000,000 baud works perfectly even while the ROS Master is actively sending 1kHz traffic.

The UART freeze only occurs when the EASYCAT.MainTask() (SPI-based) is actually executed. I calculated that at 2Mbaud, the default 64-byte RX buffer overflows in ~256us. It appears the SPI overhead is heavily exceeding this window and starving the UART interrupt.

To combat this, I implemented Serial1.addMemoryForRead() to increase the RX buffer to 1024 bytes (which should give ~4ms of breathing room). I also moved the SPI tasks out of the ISR and into the main loop() via a volatile flag. My UART parsing logic is completely non-blocking (just continuously checking Serial1.available()).
However, this did not solve the issue. When the SPI tasks execute, the UART stream still freezes, as if the buffer expansion is being ignored or overridden.

Given the 1kHz trigger rate, are there any known issues with SPI bus contention or DMA conflicts on the Teensy 3.6 that would still block UART RX interrupts even if the buffer is enlarged? Or should the larger buffer theoretically be sufficient to 'swallow' the SPI execution time?

No, there is no issue with 1-kHz interrupts. That's not a high interrupt rate as your other test shows. At 2M baud, you get ~200,000 bytes/sec, or ~200 bytes/ms, so it would take 5 seconds for your buffer to overflow. When you say the UART stream freezes, do you mean that no bytes are stored? If so, then your EASYCAT stuff must be blocking interrupts. Please show your EASYCAT.MainTask() and provide a link to the library you are using. By the way, why is there no configuration of the EASYCAT object in your setup() function?

 

[Peycool]

No, there is no issue with 1-kHz interrupts. That's not a high interrupt rate as your other test shows. At 2M baud, you get ~200,000 bytes/sec, or ~200 bytes/ms, so it would take 5 seconds for your buffer to overflow. When you say the UART stream freezes, do you mean that no bytes are stored? If so, then your EASYCAT stuff must be blocking interrupts. Please show your EASYCAT.MainTask() and provide a link to the library you are using. By the way, why is there no configuration of the EASYCAT object in your setup() function?

The EasyCAT object is initialized via the constructor EasyCAT EASYCAT(9, DC_SYNC); and the hardware initialization is performed in setup() via EASYCAT.Init().
Library Info: I am using the standard EasyCAT library by AB&T Tecnologie Informatiche (V1.5).I couldn't find this version link here is my own EasyCAT.h and EasyCAT.cpp

EasyCAT.h

GitHub Gist: instantly share code, notes, and snippets.

gist.github.com

EasyCAT.cpp

GitHub Gist: instantly share code, notes, and snippets.

gist.github.com

When I say it 'freezes,' I mean Serial1.available() returns 0 and no new data enters the buffer until I stop the EtherCAT traffic. To answer the question about MainTaskP1, here is the actual implementation from my EasyCAT.cpp. The library uses SPIReadRegisterIndirect and SPIReadProcRamFifo, both of which contain polling loops:

unsigned char EasyCAT::MainTaskP1() {
  SPI.beginTransaction(SPISettings(SpiSpeed, MSBFIRST, SPI_MODE0));
  TempLong.Long = SPIReadRegisterIndirect(WDOG_STATUS, 1);
  // ... (checks watchdog and operational status)
  SPIReadProcRamFifo();
  return Status;
}

unsigned char EasyCAT::MainTaskP2() {
  SPIWriteProcRamFifo();
  SPI.endTransaction();
}

One thing I noticed is that SPI.beginTransaction starts in P1, but SPI.endTransaction isn't called until the end of P2. My motor logic and some serial processing happen in between.
Could the fact that the SPI transaction stays 'open' between P1 and P2 be causing the Teensy 3.6 to prioritize the SPI bus so heavily that the 2Mbaud UART interrupts are being dropped, even with a 1024-byte buffer?"

 

[joepasquariello]

There was no call to EasyCAT.Init() in the code you posted. beginTransation() simply sets up the SPI with the correct speed and mode for a given device, and endTransaction() is the bookend to "release" the SPI, but there is no processing associated with the SPI transaction being "open". Have you tested EasyCAT with simple programs outside the context of the objective of your larger program? I would assume it comes with examples?

 

[Peycool]

There was no call to EasyCAT.Init() in the code you posted. beginTransation() simply sets up the SPI with the correct speed and mode for a given device, and endTransaction() is the bookend to "release" the SPI, but there is no processing associated with the SPI transaction being "open". Have you tested EasyCAT with simple programs outside the context of the objective of your larger program? I would assume it comes with examples?

In fact, I was calling this library from another main codebase that I have. In that code, I have EasyCAT.Init(). Fortunately, I was able to fix the issue today by shortening my ISR and adding another interrupt in setup():

attachInterrupt(digitalPinToInterrupt(interruptComPin), tickCountDownload, RISING);      // Interruption from TEENSY SLAVE to communicate serial data

I also removed tickCountDownload(); from the loop().
I moved the motor update to the loop() and shortened the ISR function:
void EtherCAT_DC_ISR()
{
EASYCAT.MainTaskP1();
updateMotors = true;
tickCountUpload();
EASYCAT.MainTaskP2();
}