Collision between interrupts on Teensy 4.1

Status
Not open for further replies.
W

was-ja

Well-known member
Hello,

I have a project where I am using:

  1. SPI DMA read by external interrupt, (each interrupt occurs ca. each 8us, and I am starting to read 19 bytes, this transfer is finishing over EventResponder),
  2. DMA read from both ADC channels at maximum speed with 12 bits (each buffer is 8Kwords, each channel works at ca. 1.34MS/s),
  3. serial read at 57600, by using "void serialEvent8()" call back function,
  4. internal interrupt ca. each ms, by IntervalTimer call back function.

To check accuracy in time of each step, I am saving beginning of each step by ARM_DWT_CYCCNT.

If I run only one of mentioned above methods, then everything is perfect, I mean [1] SPI DMA receive its data in time usually without any data lost over several seconds, [2] DMA ADC channels receive data synchronously, [3] serial read and and [4] internal interrupts do their job at very accurate and precise time.

If I am running everything together, only [3] serial read still works as it should be, all other parts are loosing some data. Not always, for example, only 90% of data from [1] are captured,

channels in [2] becomes desynchronized, I mean one channels usually be about 730CPU ticks faster per 8192 data reads compared to the second channel.

I suppose that somehow the interrupt handler works not as I am expecting, but I cannot understand how to monitor it and how to check priorities of interrupts of my DMA and non DMA transfers.

Please, suggest me is there any function that can dump an information regarding to all interrupts and their priorities to check, so I can hopefully fix my problem?

Thank you!
 
I can't really comment on the problem you're describing. Even if I had the complete code, that would require a pretty deep dive into the finer details.

But I can answer this 1 question:

was-ja said:
is there any function that can dump an information regarding to all interrupts and their priorities to check, so I can hopefully fix my problem?
Click to expand...

You can use NVIC_GET_PRIORITY(irqnum) to read the priority setting for any interrupt. It's defined in imxrt.h, if you want to see it's code. It simply reads from the NVIC registers inside the ARM processor, starting at address 0xE000E400. The actual interrupt numbers are also defined at the beginning of that file, and also in the IMXRT1060 reference manual, starting on page 44.

Keep in mind the ARM processor also has 16 "exceptions" which are for all practical purposes 16 more interrupts. Normally SYSTICK is the only one used by most programs. It defaults to priority 32 (where the range is 0 to 255, with lower numbers having higher priority). Look for configure_systick() in startup.c for details.

Again, very difficult to know why you're experiencing data loss. But if you're crafting very low level code and you're trying to push to the upper limits of possible performance, it's always important to consider how much interrupt latency each part imposes on all the others, and how much interrupt latency each part can tolerate. When you test with only 1 interrupt in use, of course it will perform in a best case scenario, but a practical design needs to work in the worst case scenario based on the other code also using interrupts.
 
Yes, if you assign different interrupt priorities, you should get it working. By default, they all have (but audio + systick) the same priority.
A long time ago I posted some code (unfortunately here, no on github) that lists all available information about the interrupts. Their piorities, etc. I can't find it anymore.
Perhaps @defragster, who is good with finding old threads is able to find it... or you can try to find it..
I think it would be good for the wiki. Even better on the pjrc page, but... well, .... lol
 
Note, if all ints have the same priority, they run "round robin" - which can mean a long time - which again can cause missed interrupts (worst case).
Additionally, they all get globally disabled at various places in the teensy core files (which is a medium (for some: high) priority) issue which is - obviously - on pjrc's low priority list for > 5 years. At least I could prevent this for the systick in the T4, together with Defragster) - disabling means additional stalls.
 
Thank you very much, PaulStoffregen and Frank B for your kind information and assistance!

I printed interrupt priorities just after inicialization and during my main loop and figured out that at the setup all interrupts beside 64-th, had 128 priority and, during the main loop - 24-th (Serial4) have 64, and 110 (SDHC1) have 96, so, at least, I got some information.

Please, suggest me, is there any possibility to figure out what exactly is allocated at i-th interrupt? Preferably run-time!

For example, I do not use Serial4 (I am using corresponding pins for GPIO), but its interrupt priority was changed somehow.

Please, suggest me, is there any possibility to check interrupt queue runtime? I mean I clearly understand that my several interrupt handlers can be invoked at the same time and I would like to monitor it somehow.

I am using DMA for SPI transfer and DMA for ADC. Please, suggest me how to figure out which interrupt was allocated to which operation (and how many DMA interrupts were used for 2 channels of my DMA-ADC?).
 
was-ja said:
Please, suggest me, is there any possibility to figure out what exactly is allocated at i-th interrupt? Preferably run-time!
Click to expand...

Yes you can just read the interrupt vector and print the address. Maybe add2line then can print the line? Don't know.
Good night.
 
Thank you very much, Frank B,

I found _VectorsRam and it seems that it contain corresponding function pointers, so, probably with your suggestion regarding to add2line it will be the simplest solution!

Please, suggest me, how to disable some interrupts? Can I write 0 into appropriate position in _VectorsRam array? I understand what I am doing, I have two stages in my algorithm, where at the first stages I would like to remove all interrupts that I am not using, and at the second stages - use them as normal.
 
Frank B said:
...
Perhaps @defragster, who is good with finding old threads is able to find it... or you can try to find it..
...
Click to expand...

Frank ... More context for search? The closest thing I found related to chaining interrupts: Arduino-Events

As far as the OP - wonder the effect of the DMA #1 (125000 isr/sec) and #2 on bus/memory contention? At high rates they will bump into each other and cause delays in some fashion? And that could ripple through if the processor balks in the process?

#3 UART hardware has dedicated FIFO and some buffer on chars that should survive most disturbances - esp at 57K baud.

#4 'only' 1K interrupts sounds like it should function okay - but depending on pri setting might get delayed - and depending on what it is accessing and how long it is taking ...
 
Thank you, defragster, for your comment.

I forget to mention that SPI DMA (#1) running at normal (not DMAMEM) FASMEM. If I am using DMAMEM for 125000 isr/sec with 19 bytes, it already produce some missing entries. I played with SPI speed, 20M-32M-40M, it works the same, so, I am doing it at SPI.beginTransaction(SPISettings(40000000, MSBFIRST, SPI_MODE0)); (I did not checked, is it really 40M, or less) and normal DTCM.
 
Hello,

I played with interrupts and print them from all possible corners. I made a small print utility, may be somebody can also use it.

Code: 
class PrintInterruptsClass
{ public:
  void         *func;
  char          Name[14];
  unsigned char Priority;
  unsigned char FirstFound;
};


PrintInterruptsClass PIC[512];
int MaxPIC=0;


void CHECK_INTERRUPTS(char *name) // you should place it in some places where you are expecting to have some interrupts
{ for(int i=0; i<160; i++)
  { void *f=_VectorsRam[16+i];
    unsigned char pri=NVIC_GET_PRIORITY(i);
    for(int j=0; j<MaxPIC; j++)
      if(f==PIC[j].func && pri==PIC[j].Priority) goto next;
    PIC[MaxPIC].func=f;
    PIC[MaxPIC].Priority=pri;
    int j;
    for(j=0; j<13; j++)
      if(name[j]) PIC[MaxPIC].Name[j]=name[j]; else break;
    PIC[MaxPIC].Name[j]=0;
    PIC[MaxPIC++].FirstFound=i;
next:;
  }
  return;
}


void PrintInterrupts() // it prints the first occourance of interrupt function, its priority, its interrupt number and the name that was used by CHECK_INTERRUPTS call
{ Serial.printf("We found %d different entries of functions and its priorities:\n", MaxPIC);
  for(int i=0; i<MaxPIC; i++)
    Serial.printf("%p %d %d @ %s\n", PIC[i].func, PIC[i].Priority, PIC[i].FirstFound, PIC[i].Name);
  return;
}

Hence, I forwarded function's pointer over addr2line and got the following list:

0xc211 128 0 @ In a SetUp 0 ~/arduino-1.8.19/hardware/teensy/avr/cores/teensy4/startup.c:535
0xc3b5 0 64 @ In a SetUp 0 ~/arduino-1.8.19/hardware/teensy/avr/cores/teensy4/tempmon.c:18
0xc671 128 113 @ In a SetUp 0 ~/arduino-1.8.19/hardware/teensy/avr/cores/teensy4/usb.c:235
0xdb75 64 24 @ In a Loop ~/arduino-1.8.19/hardware/teensy/avr/cores/teensy4/HardwareSerial8.cpp:45
0x9c95 96 110 @ In a Loop ~/arduino-1.8.19/hardware/teensy/avr/libraries/SdFat/src/SdCard/SdioTeensy.cpp:288
0xdb81 128 122 @ In a Loop ~/arduino-1.8.19/hardware/teensy/avr/cores/teensy4/IntervalTimer.cpp:96
0x21 128 157 @ In a Loop ~/arduino-1.8.19/hardware/teensy/avr/libraries/SdFat/src/ExFatLib/ExFatVolume.cpp:32
0x509d 128 3 @ In an ADC 1 ~/arduino-1.8.19/hardware/teensy/avr/libraries/SPI/SPI.cpp:1500
0xdb81 240 122 @ In a RE 0 ~/arduino-1.8.19/hardware/teensy/avr/cores/teensy4/IntervalTimer.cpp:96
Click to expand...

what is really surprising for me is that I do not see any interrupt referring to DMA-ADC

Code: 
  adc->adc0->setAveraging(0); // set number of averages
  adc->adc0->setResolution(12);
  adc->adc0->setConversionSpeed(ADC_CONVERSION_SPEED::VERY_HIGH_SPEED);
  adc->adc0->setSamplingSpeed(ADC_SAMPLING_SPEED::VERY_HIGH_SPEED);

  adc->adc1->setAveraging(0); // set number of averages
  adc->adc1->setResolution(12);
  adc->adc1->setConversionSpeed(ADC_CONVERSION_SPEED::VERY_HIGH_SPEED);
  adc->adc1->setSamplingSpeed(ADC_SAMPLING_SPEED::VERY_HIGH_SPEED);

  abdma1.init(adc, ADC_0);
  abdma2.init(adc, ADC_1);
  adc->adc1->startContinuous(readPin_adc_1);
  adc->adc0->startContinuous(readPin_adc_0);

and external interrupt that I invoked by:

Code: 
  attachInterrupt(digitalPinToInterrupt(pinDR), GetADC, FALLING); // the function GetADC start SPI-DMA in my case
  event.attachImmediate(&asyncEventResponder);

Actually, both blocks are critical for me because they produce each ca. 3MBytes/s data, both use DMA, and both works perfect if used without other block and start to lost data if invoked simultaneously.

Please, suggest me:
1. is there some hidden interrupts handlers, inside for example, ~/arduino-1.8.19/hardware/teensy/avr/cores/teensy4/startup.c:535 and if yes, is there any possibility to take it into independent interrupt handler?
2. where are ADC DMA interrupts?

Thank you!
 
was-ja said:
Hello,

I have a project where I am using:

  1. SPI DMA read by external interrupt, (each interrupt occurs ca. each 8us, and I am starting to read 19 bytes, this transfer is finishing over EventResponder),
  2. DMA read from both ADC channels at maximum speed with 12 bits (each buffer is 8Kwords, each channel works at ca. 1.34MS/s),
  3. serial read at 57600, by using "void serialEvent8()" call back function,
  4. internal interrupt ca. each ms, by IntervalTimer call back function.

To check accuracy in time of each step, I am saving beginning of each step by ARM_DWT_CYCCNT.

If I run only one of mentioned above methods, then everything is perfect, I mean [1] SPI DMA receive its data in time usually without any data lost over several seconds, [2] DMA ADC channels receive data synchronously, [3] serial read and and [4] internal interrupts do their job at very accurate and precise time.

If I am running everything together, only [3] serial read still works as it should be, all other parts are loosing some data. Not always, for example, only 90% of data from [1] are captured,

channels in [2] becomes desynchronized, I mean one channels usually be about 730CPU ticks faster per 8192 data reads compared to the second channel.

I suppose that somehow the interrupt handler works not as I am expecting, but I cannot understand how to monitor it and how to check priorities of interrupts of my DMA and non DMA transfers.

Please, suggest me is there any function that can dump an information regarding to all interrupts and their priorities to check, so I can hopefully fix my problem?

Thank you!
Click to expand...

Hi,

i am not sure, but i had issues with SPI DMA together with the intervalTimer functionality. I send and forget data via SPI DMA from an intervalTimer callback. It worked first, but when i cranked up the interval to around 20ms my application stalled, it seemed to be dependant on the cycle time and the amount of data i sent via SPI DMA. Tried also TeensyThreads, same effect, application stalled immediately...
I moved back to the "normal" loop() and timing via millis/micros and voila it worked fine since then. Good i did it...i was nearly giving up my project, now i am back on track.

Torsten
 
Status
Not open for further replies.
Back
Top