Teensy 4.1 SD card detection - why does it block my Serial1 with higher interrupt pri

[sicco]

My application does some real time control, and optionally writes data to a SD card using exFAT SdFat, SdioTeensy

The SD card may or may not be present in the Teensy 4.1 SD card socket, and it may get ejected/inserted by a user at any time. But real-time control shall always continue no matter what. And logging shall (re)commence as soon as a card is in.

I do the sd card tasks in loop(), all real-time tasks are timer interrupt driven. With a FIFO buffer in between.

My problem is that with no card inserted, when my code polls for a card, it blocks (stalls the flow for) Serial1 communication for about 1 second. I think the i/o on that serial port does not flow, but the thread that interfaces with Serial1.available(), Serial1.read(), Serial1.write() does continue ok.

Other serial port (Serial that connects to Monitor) keeps flowing without hiccups.
Also Serial2 does not seem to be affected and happily carries on.

I have set interrupt priority for LPUART6 to 32. That does not fix it.
I have also tried setting IRQ_SDHC1 priority to a much higher number than 6*16, but that also does not fix it.

My thinking is that it's related maybe to a limited number of DMA channels, and that SdioTeensy claims a DMA channel (for up to a full 100000 us while waiting for a response from a uSD card that's not there...) that Serial1 also needs. Or some other scarce Teensy hardware resource. But I cannot find it when I dig into the many library sources.

What could this be? Is it yet another interrupt that is associated with DMA?
Is there any other way to detect card present in a way that does not stop my Serial1 data flow for up to 1 second after calling sd.begin(SdioConfig(FIFO_SDIO))? Or should I refrain from repetitively doing sd.begin and sd.end calls?

[sicco]

Correction: while SdioTeensy is busy trying to detect a card that's not there, also Serial2 reception flow stops, but transmission carries on. I suspect that's also the case for Serial1. But why???

[PaulStoffregen]

Correction: while SdioTeensy is busy trying to detect a card that's not there, also Serial2 reception flow stops, but transmission carries on. I suspect that's also the case for Serial1. But why???

I don't know why. I would like to investigate, but I need a reasonably small test program to reproduce the problem. I looked briefly as your ZIP file. It is 21 files with a total of 7758 lines. Sorry, that's far too much. It also appears to depend various hardware connected to Teensy. I just don't have the time to do an investigation with so much code. If there is some bug where SD / SdFat is interfering with Serial1 receive, I need a much smaller program which doesn't depend on other hardware, except perhaps another Teensy running a tiny program to continuously transmit to RX1.

[PaulStoffregen]

Also, and hopefully this isn't too "obvious", the default Serial1 receive buffer is only 64 bytes. Unless you've used attachRts() and whatever is sending the data stops when RTS signals Teensy's buffer is nearly full, you can pretty easily get a buffer full situation if detecting the SD card takes 1 second and data arrives at any reasonably fast baud rate during that time. Other than RTS/CTS flow control, the other solution is to expand the buffer with addMemoryForRead().

Other serial port (Serial that connects to Monitor) keeps flowing without hiccups.

Keep in mind USB "Serial" is completely different from hardware "Serial1" and "Serial2". The USB protocol has flow control built in. It's always active and can't be shut off or ignored like RTS/CTS. If your PC (or Mac) tries to send more data during a time when Teensy is busy and your program isn't calling Serial.read(), the data remains in buffers on your computer until Teensy is able to receive it.

But with non-USB real hardware serial, unless you're using RTS/CTS flow control, if bytes arrive when Teensy's buffer is already full, they have to be discarded. Only proper use of RTS/CTS flow control can make the transmitter wait, as happens automatically with USB serial.


If that doesn't solve your problem, if you're convinced there is a bug in Teensy's Serial1 or Serial2 implementation, then you need to craft a small & simple test program to demonstrate the problem. I'm not necessarily saying Teensy's code is 100% bug free. It might be a bug. But a reasonable small test program is required before any investigation will begin.

[KurtE]

Likewise, I was also taking a quick look, but far to much code to navigate easily to see what might be happening...

Also a few details may be helpful, like which version of Arduino and Teensyduino are you using? Is the SDFat being used coming from the Teensyduino install directory or from your <sketches>/libraries/... directory

For example how are you detecting that the SD is inserted?

I am asking this as for example the most recent Teensyduino, Paul added the SD.mediaPresent() method which returns true or false... I search in code did not find any references to this call.
Before that I was hacking up a version of the code that at least at startup could detect if card was present by looking at the DAT3 pin, but this only worked up until we had a card inserted as once we start SDIO it uses that pin for communications...

Also unclear is how is (or is) Serial2 being blocked? That is so far looking at where you read in Serial2, you are using the SerialEvent2 to process.

void serialEvent2() 
{
    while (Serial2.available())
    {
        char ch = Serial2.read();
        {
//            Serial.print (ch);
            ParseEnc50t_character (ch);
        }
    }
}

So two possibilities:

a) Serial2 is blocked during this time and ISR not being called. If higher priority interrupt (lower value: // Cortex-M7: 0,16,32,48,64,80,96,112,128,144,160,176,192,208,22 4,240)
If the ISR is not being called, then probably some code has disabled interrupts.

b) Or the ISR is being called, and the data is retrieved and put into the software queue, but the serialEvent2 is not being called.
This one can happen very easily. The event functions like this are only ever called if the yield() function is called. Yield is called at the end of each call to loop(), and on Teensy
code it is called anytime you call a function like delay() directly or indirectly, or code explicitly call yield();

So for example if you have code like:

elapsedMillis em= 0;
while (em < 1000) {
    if (digitalRead(pin)) break;
};

For example to wait for a pin to go high to return. Or could be reading some SPI value waiting for response...
But this code hard spins and does not call yield and as such your Serial2 code is not called.

I am guessing b)...

If it were me, I would do a quick hack on Serial2 ISR, on an unused IO pin lets say pin 41, I would add something like digitalToggleFast(41);
at the start of the Serial2 ISR... (remember need to set the pin to digitalOutput). And then hook up Logic Analyzer and run the code and see if the ISR was being called...
Would probably do similar to your serial2 event function on different pin and get an idea when things are happening.

[Frank B]

Yup, b) sounds as it could be the reason.
Not sure what the events are good for - this is one of the reasons not to use them.
But of course - it could just be a disabled interrupt too. This happens very often (btw, in the "events"-code, too... )

[sicco]

I'm impressed with the number of folks responding within minutes/hours, thank you.

I've made a single file project now, just 107 lines, attached.

It replicates the issue: reception via any Serial port is frozen for 1 second when sd.begin(SdioConfig(FIFO_SDIO)) is called from a loop() thread when no card is inserted.

In the attached example, which assumed initially that there's RS485 hardware connected that responds to a '?' poll every 100 ms, it now already shows the problem using the Arduino Monitor (USB Serial) Tool: pick a Teensy 4.1, load the code, and have a uSD card at hand.

When the code runs with an SD card inserted, there are no gaps in data reception.
When the code runs without SD card inserted, every 1000 ms the serial reception is blocked for 1000 ms. I think no data is lost. It just arrives way too late. Something is blocking the expected timely serial reception flow.

(From Monitor tool I have to rapidly type anykey and <enter> with two hands so that I throw a stream of >> 1 character at a time)

[sicco]

Also, and hopefully this isn't too "obvious", the default Serial1 receive buffer is only 64 bytes. Unless you've used attachRts() and whatever is sending the data stops when RTS signals Teensy's buffer is nearly full, you can pretty easily get a buffer full situation if detecting the SD card takes 1 second and data arrives at any reasonably fast baud rate during that time. Other than RTS/CTS flow control, the other solution is to expand the buffer with addMemoryForRead().




Keep in mind USB "Serial" is completely different from hardware "Serial1" and "Serial2". The USB protocol has flow control built in. It's always active and can't be shut off or ignored like RTS/CTS. If your PC (or Mac) tries to send more data during a time when Teensy is busy and your program isn't calling Serial.read(), the data remains in buffers on your computer until Teensy is able to receive it.

But with non-USB real hardware serial, unless you're using RTS/CTS flow control, if bytes arrive when Teensy's buffer is already full, they have to be discarded. Only proper use of RTS/CTS flow control can make the transmitter wait, as happens automatically with USB serial.


If that doesn't solve your problem, if you're convinced there is a bug in Teensy's Serial1 or Serial2 implementation, then you need to craft a small & simple test program to demonstrate the problem. I'm not necessarily saying Teensy's code is 100% bug free. It might be a bug. But a reasonable small test program is required before any investigation will begin.

I did craft what you asked for - see other post from Saturday - anything else you need?
I'm using Arduino 1.8.19, and Teensyduino your latest (Dec 2021 ish).
I've now further reduced the number of code lines for an example that reproduces the issue using just a Teensy 4, a uSD card, and Monitor. In Monitor, using two hands, type on the PC keyboard just one letter and enter to create a stream of >>1 characters per second. With the (exFAT) SD card in, it responds immediately to any keystroke. Without the SD card in, it blocks reception for one full second. But why? And can it be fixed?

So it's just this code, only 71 lines:

#include <SdFat.h>
#include <SdFatConfig.h>
#include <sdios.h>

#include <TimeLib.h>

SdFs sd;
FsFile txtFile;

IntervalTimer myTimer_100ms;
volatile int Count_100ms_ticks = 0;

volatile int Serial_chars_received = 0;
volatile char last_char_from_Serial = 0;

void setup() 
{
    Serial.begin(115200);
    while (!Serial)
        delay(10);

    Serial.println ("Starting 100ms timer");
      
    myTimer_100ms.begin(timerAction_100ms, 100000);  // timerAction to run every 0.1 seconds
    myTimer_100ms.priority (3*16); //  lower number = higher priority
}

void timerAction_100ms () 
{
    char st[256] = "";

    sprintf (st, "Hello again #100msticks=%d #Serial_chars=%d last_char = %c", Count_100ms_ticks, Serial_chars_received, last_char_from_Serial);
    
    Serial.println(st);
    Count_100ms_ticks++;
}

void serialEvent() 
{
    while (Serial.available())
    {
        last_char_from_Serial =  Serial.read();
        Serial_chars_received++;
        
        Serial.write (last_char_from_Serial);
    }
}

void loop ()
{
    char str[100];
    strcpy (str, "test.log");

    char log_filename[100];
    strcpy (log_filename, "test.log");

    if (sd.begin(SdioConfig(FIFO_SDIO))) 
    {
        txtFile = sd.open(log_filename, FILE_WRITE);
        if (txtFile)
        {
            txtFile.seek(EOF);
            txtFile.write (str, strlen(str)); 
        }
    
        txtFile.close();        
        sd.end();
    }

    delay(100);
}

[KurtE]

Sorry I am confused. You mention Serial1 yet your sketch does not mention Serial1?

I ran your sketch on t4.1 with serial monitor or in this case TyCommander

And I see:

Hello again #100msticks=82 #Serial_chars=0 last_char = 
Hello again #100msticks=83 #Serial_chars=0 last_char = 
Hello again #100msticks=84 #Serial_chars=0 last_char = 
Hello again #100msticks=85 #Serial_chars=0 last_char = 
Hello again #100msticks=86 #Serial_chars=0 last_char = 
abcd
Hello again #100msticks=87 #Serial_chars=5 last_char = 

Hello again #100msticks=88 #Serial_chars=5 last_char = 

Hello again #100msticks=89 #Serial_chars=5 last_char = 

Hello again #100msticks=90 #Serial_chars=5 last_char = 

Hello again #100msticks=91 #Serial_chars=5 last_char =

Where I had typed abcd into line in monitor and hit return... So what are you expecting different?

Edit: Note the last character typed in was either CR or LF depending on line ending settings.

[sicco]

Thanks for responding. The post started with me having an issue with Serial1. I then noticed Serial2 had the same issue. And finally I noticed that also Serial (USB Serial port) shows the same problem. The last code example took all Serial1 and Serial2 references out, so that there's no need to wire something externally to the Teensy 4.1. Just so that others can see it even when they have no Teensy hardware that readily links e.g Serail1 Tx to Serial1 Rx pins (some software folks have no soldering iron/skills and no spare jumpers or just no paperclips...).

(For replicating the issue, we need something that throws >>1 byte per second to a/the Serial port. On most serial port terminal emulators (like e.g. Putty) it was enough to keep a keyboard key depressed, and then autorepeat would get me such a stream of at least 1 char every 200 ms. In Arduino's Monitor, one has to press <Enter> before something really gets sent, so that's why I use two hands, one on the enter key and one on, say, key 1. That way I can bombard the Teensy with Serial input at a rate of 1 new char every ~300 ms.)

What you did I think is type abcd<enter>. That lead to 5 chars thrown in one USB packet, with not even a ms in between each char. That's not what we want because then you cannot see the issue. I want Teensy Serial to see 'a' <gap of more than 100 but less than 500 ms> 'b' <gap of more than 100 but less than 500 ms> 'c' <gap of more than 100 but less than 500 ms> 'd' <gap of more than 100 but less than 500 ms> and so on. Then you see a steady immediate flow with SD card in, but a freeze of 1000 ms every 1000 ms of the #Serial_chars counter. And that freeze should not happen (I think...) because that's in the lowest priority loop() task.

Reason it freezes for 1 second with no SD card is that there appears to be a timeout of 1 second while waiting for any response from a card. Obviously no card is no response. But that shall not block important higher priority tasks like serial receoption (I'd say...).
With a card in, the SD card detect obviously is successful. So it doesn't need the full 1000ms because there in no timeout. But I'm worried (alarmed, sort of paranoid now) that cards that want to occasionally take some extra time for read, write (or erase!) will/can freeze and that's what my application cannot tolerate.

[KurtE]

Again I don't think it has anything at all to do with priority of interrupts or anything else like that...

You have two things going on as it pertains to Serial and your timer...

For example take your own sketch, in setup add: pinMode(13, OUTPUT);

in your IntervalTimer function add: digitalToggleFast(13);

And you will see this chugging along blinking about 5 times per second...
This should typically happen as long as no one is running an equal or higher priority interrupt OR if some code disables interrupts...

Then you have your SerialEvent code:

This will only be called in places that call yield() directly or indirectly. Yield does get called when you exit loop() or when you call functions like delay(), which we try to do in many different areas...

However if your code is like: (contained in SDFat library SdioTeensy.cpp)

static bool waitTimeout(bool (*fcn)()) {
  uint32_t m = micros();
  while (fcn()) {
    if ((micros() - m) > BUSY_TIMEOUT_MICROS) {
      return true;
    }
  }
  return false;  // Caller will set errorCode.
}

Who maybe has timeout of 1 second, then there is nothing here that yields, and as such your serialEvent() code will not be called.

[sicco]

OK, that helps. In my ignorance I had assumed that serialEvent() was (an extension of) the a UART Rx interrupt service routine and would thus be called whenever a byte has arrived through the Rx pin frontdoor.
That begs the question: do I need to write my own irq driven serial driver if I want code that responds du-moment that a byte has popped in?

[Frank B]

OK, that helps. In my ignorance I had assumed that serialEvent() was (an extension of) the a UART Rx interrupt service routine and would thus be called whenever a byte has arrived through the Rx pin frontdoor.
That begs the question: do I need to write my own irq driven serial driver if I want code that responds du-moment that a byte has popped in?

More or less, yes.
But why would this be important? What is the usecase?
Normally, an interrupt line is used for such things - if timing is important.

Edit: And remember that the serial interrupt occurs after the byte is received... after 10 bits. regardless of the bitrate, this time is ages for a 600MHz cpu... so, pretty late!

[sicco]

More or less, yes.
But why would this be important? What is the usecase?
Normally, an interrupt line is used for such things - if timing is important.

Edit: And remember that the serial interrupt occurs after the byte is received... after 10 bits. regardless of the bitrate, this time is ages for a 600MHz cpu... so, pretty late!

The use case is a sensor that has a RS485 interface, it sends a short message back when it's being polled. The Teensy polls this sensor and it runs a feedback controller using the sensor value. So the sensor signal (the contents of that response) is needed in a real-time control loop that executes deterministically at 100 iterations per second. Missing a sample is a mortal sin. It's a real-time system. Period.

Another use case, in the same system actually, is a supervisory interface to a PC for plotting data when it's connected via yet another serial (RS485) port. On that port, the UARTs are 1.5Mbaud, the packet message lengths are anything 10-1000 bytes, and the response to a poll from the PC is preferably coming less than 1 ms after the end of its poll. I don't see how else I can guarantee unconditional immediate responses than by having a UART RX interrupt service routine so that if the (last) byte of a poll does arrive, I will get the timely response delivered.

Profibus is another example where you definitely have to respond within such tight constraints.

[Frank B]

Why this tone?
Just write your isr, if you feel you need it.

have fun,
Frank

period.

[sicco]

My 'tone' here is the result of a long career as control/instrumentation engineer battling those that fail to appreciate design challenges with deterministic real-time systems.
I apologize if that's felt like / landing as an offence.

[PaulStoffregen]

I had assumed that serialEvent() was (an extension of) the a UART Rx interrupt service routine and would thus be called whenever a byte has arrived through the Rx pin frontdoor.

serialEvent() is called from yield(). It's not called from interrupt context.

I believe this problem does show that we need to improve Teensy's fork of SdFat to call yield() while it's spending time waiting. I believe Bill had yield() calls in earlier SdFat versions, but took them out for reasons probably unrelated to Teensy. Before forking, we sync'd up to his latest which added support for international chars in filenames, which I believe was also the point where yield was removed.

@sicco - Please understand the SD card libraries on Teensy have gone through some pretty major changes since version 1.53. We completely removed the ancient Arduino SD library and replaced it with a thin wrapper for SdFat. More recently, we forked SdFat to add a number of performance improvements and minor optimizations which Teensy needs to make MTP viable when accessing SD cards. We had many years of experience with the very old Arduino SD library (which only supported 8.3 filename and only up to 32GB cards) and we're still learning of these types of issues which happen with the new SdFat.

I've put yield() calls on my to-do list. But just to be realistic, it's a relatively low priority, so I can't say if it will happen for 1.57 or any other time frame. But hopefully at least understanding now the issue helps somewhat?

[Frank B]

@sicco -

two options:

- copy the existing isr code
- use interrupt chaining.

with both: keep care of the interrupt priorities.

And note that the core disables interrupts, sometimes. So really deterministic timing is not possible with teensy.
However 1ms is a long time. Probably long enough that you can ignore that.

Disable yield() by overriding it.