Teensy 4.1 Freeze (~1786ms)

Status
Not open for further replies.
Still no freezes. Startet the game now. It is incredible fast. I can't remember that it was that fast on my 286 or 386.
I miss the sound..
 
I can confirm that the freeze in my updated TeensyMIDIPolySynth (both the full version & the pared down version) no longer occurs after making Frank B's code change (as posted/clarified by defragster) !!

Thanks & congratulations for successfully finding the apparent cause/fix . . . well done, Frank B !!

Mark J Culross
KD5RXT
 
The QSPI pads for PSRAM on the bottom are mapped to the MCU with those FlexSPI2 commands for MPU (protection ) control. Their privilege's and purpose.

From the .ld:
Code:
	FLASH (rwx): ORIGIN = 0x60000000, LENGTH = 7936K
	ERAM (rwx):  ORIGIN = 0x70000000, LENGTH = 16384K

The FlexSPI handles the address mapping and the interface - and that gives the rules for the MCU to enforce:
Code:
	SCB_MPU_RBAR = 0x60000000 | REGION(i++); // QSPI Flash
	SCB_MPU_RASR = MEM_CACHE_WBWA | READONLY | SIZE_16M;

	SCB_MPU_RBAR = 0x70000000 | REGION(i++); // FlexSPI2
	SCB_MPU_RASR = MEM_CACHE_WBWA | READWRITE | NOEXEC | SIZE_16M;
 
The QSPI pads for PSRAM on the bottom are mapped to the MCU with those FlexSPI2 commands for MPU (protection ) control. Their privilege's and purpose.
Question now is, is 1.7s freeze related with use of PSRAM or with doubling (or 'wrong'/huge size) in startup.c ?
IOW, do all freeze code use PSRAM?
Is access to PSRAM cached?, does this cache then be cleared as with DMAMEM?
 
Good Morning :)

@Mark: Glad that is works now for you!
@Tim thank you.
Question now is, is 1.7s freeze related with use of PSRAM or with doubling (or 'wrong'/huge size) in startup.c ?
In Doom, yes, it uses PSRAM (but I don't think it's used in the menu, where the freeze happened) and the doubling was problem.
IOW, do all freeze code use PSRAM?
Good question. Let's wait what Udo says.
There are a few more questionable MPU settings - maybe we have to modify them too.

My current theory is, there is no need for accesses of the FlexSPI2 memory for the 1.7 sec. freezes to occur. It's just the MPU setting. Perhaps it leads to some wrong bits, MPU-internally.
If Udo says he has *still* freezes, it may be a good Idea to review the other settings, too.

Is access to PSRAM cached?,
does this cache then be cleared as with DMAMEM?
Yes and yes. If there are accesses by the hardware (i.e. DMA) you need the same cache handling as with DMAMEM.
If you write to the flash on FlexSPI2, you need it there, too.
 
Last edited:
Walter, you were right when you said that we don't know the chip.
ARM updated a pretty hidden detail in the MPU.
It looks like the Cortex-M7 does not have the priority regions the M4 had, and uses subregion-bits instead.
 
When QSPI work first started it seems it was wrongly perceived to be working before it really was - thanks to the cache.

When the MCU is told to cache a region it does - it can be written and read back - even if not backed by any real resource. Not sure what happens when MCU needs to flush the cache for new data?

The Frank commented lines marked out a big region - was wondering if that overlapped anything - or it anything doing access there might be timing out doing I/O?
> not sure what the 256M includes starting at :: 0x70000000

Could be interesting to LA or Scope those pins if there was a 'Freeze' sketch that didn't use PSRAM with the old lines in place. Or maybe the DOOM sketch while not or before the PSRAM is used.
 
Heureka, Frank's solution seems to work :eek:

Using Frank's development IDE, including his fix, proposed and described above yesterday, my sketch runs since 16.5 hours without Freezes. I am using a version (v161) which has all features and functions re-enabled, which were one by one disabled in the sequence of fault finding over the last few days. That means all libraries and user ISRs, output is on USBSerial too.

Well, I guess we should still be a little cautious, because we have seen before that there can be many hours between Freezes and therefore I assume, we could only safely call this issue resolved, after we understood, why not only Frank's solution but also disabling any use of USBSerial are/were fixes.

I will continue to look out for Freezes during the next few days, just in case there still some "hiding". I will report back, hopefully just with good news.

Thank you Frank, well done!! Your hard work is truly appreciated!!!
 
Well, I guess we should still be a little cautious, because we have seen before that there can be many hours between Freezes and therefore I assume, we could only safely call this issue resolved, after we understood, why not only Frank's solution but also disabling any use of USBSerial are/were fixes.

I think it's because USB uses the cache-functions of the core which cause the cache-contents to be written to the RAM.
Somewhere here in this operation the MPU checks if everything is OK and it *somehow* controls the cache. But it could not do its job right because it was somehow "confused".

I whish I knew it more exactly, but there is no documentation about the inner working of the MPU.
 
Indeed I spent several hours to identify the problem.
BIG thanks to Jean-Marc Harvengt who told me that it happens in Doom. It was luck that I had the VGA board "just in time".. :)

@Udo: yes, more testing is good.
You can use your version of Arduino/Teensyduino now, if you coment out these two lines in startup.c

Paul told me that a Teensyduino Beta 8 with this fix is comming, too.

I still find it very fascinating that the freeze lasts so exactly (exact cylce-count) the same time. This can only be explained if the MPU has a counter built in somehow. But what is it for?
And, I wish to know why it - first - seemed to be fixed with GCC9 - and later not... I can't remember what I did...
 
Last edited:
I investigated this further:
It sems not to be the 2nd region.
It even freezes if I set the size of the remaining single region to a value larger then 16M

Code:
    SCB_MPU_RBAR = 0x70000000 | REGION(i++); // FlexSPI2
    SCB_MPU_RASR = MEM_CACHE_WBWA | READWRITE | NOEXEC | SIZE_32M;
freezes.

@Paul, any Idea?
 
Last edited:
This may or may not relate:

I have been fighting an issue with a program that uses the SmartMatrix and Adafruit_GFX libraries with Teensy4.1, and if you are looking at pins via a scope or equivalent, needs no other hardware or mods to the teensy4.1 (or presumably 4.0). It is very repeatable, as the (I'm assuming) DMA that is refreshing the led panel freezes (the LED panel goes blank) at pretty much the same points in a 33.9 second long loop when the problem is not seen, and one time I measured with a stopwatch 41.1 seconds when the problem is seen. I am temporarily bedridden so I have not been able to put a scope on it, but it is in a relatively simple test loop of the LED display. ANYTHING seems to make it go away, including your fix in startup.c, as well as adding even a DigitalWriteFast(14,LOW) anywhere in loop(), or most any other change in loop(), much less adding timing routines to it. Other changes, like adding in InternalTemperature.h, and initializing it but nothing else changes the places where the blanking occurs, and causes it after 20 minutes to do something (change the DMA update frequency?) so the display looks like it's shimmering a bit. Again, sorry for the lack of specificity.

Note: The problem is sudden blanking of the screen, however the program does dim out to a blank screen and holds that blank for 1 full second.

In any case, it seems to be an easier program to debug this problem on, if it is the same, because it happens so regularly. On the other hand, it might be a problem, in that it is 'delicate', in that even adding a single writeDigitalFast() in loop() makes it go away.

Here is a link to the thread on the PixelMatrix site, although it has no answers except the one you have identified above: https://community.pixelmatix.com/t/weird-blanking-problem/951

Here is a link to a zip of the source code (two files) https://www.dropbox.com/s/aosoyiqor996ebz/FeatureDemoMod.zip?dl=0

Here is a link to a video showing the problem: https://www.dropbox.com/s/iimsnh7hi3f63xp/with blanking problem.mp4?dl=0
And a link to a video showing what it should be: https://www.dropbox.com/s/bjtabbfkfsvf90l/with Serial println added.mp4?dl=0

Here is a link to the basic info on the product, with a diagram of what pins are used: http://docs.pixelmatix.com/SmartMatrix/shield-t4.html
and a link to github for source code, and schematics under extras/hardware/teensy4:https://github.com/pixelmatix/SmartMatrix

If this isn't the same issue, it certainly is an odd issue, that seems to be compiler related.
 
Last edited:
BTW: Does the 'fix' in startup.c break anything else?

@CoreyCoop:

My evidence is only circumstantial, but after adding the fix to startup.c manually (to remedy freezes on the latest version of my TeensyMIDIPolySynth that I'm tinkering with), I have not seen any detectable adverse effects from the addition/fix.

Mark J Culross
KD5RXT
 
@Paul:
Is this correct (?)
Code:
    SCB_MPU_RBAR = [COLOR=#00ff00]0x70000000[/COLOR] | REGION(i++); // FlexSPI2
    SCB_MPU_RASR = MEM_CACHE_WBWA | [COLOR=#00ff00]READONLY[/COLOR] | NOEXEC | SIZE_256M;

    SCB_MPU_RBAR =[COLOR=#00ff00] 0x70000000 [/COLOR]| REGION(i++); // FlexSPI2
    SCB_MPU_RASR = MEM_CACHE_WBWA | [COLOR=#00ff00]READWRITE [/COLOR]| NOEXEC | SIZE_16M;
The 2nd has higher priority - so the first is not active. But can the MPU handle this correctly?

Don't want produce the next false positive here...take this with care: Doom is running for 30minutes now, with the first commented.-out, and without freeze. That's unusual..

Thanks Frank! Wish I had seen this thread earlier :) Last week, I was experiencing similar issues inside a loop that "normally" took around 22ms for ~500 iterations, experiencing "freezes" every 50 or 60th iteration for around 1.6 seconds in duration (I never timed it precisely as it was the sheer size of the freeze that was of concern, plus I could never reliably isolate which piece of code caused it). My app was large and I normally use platformIO / VSCode, but I was able to move a small subset of my code and reproduce in the Arduino IDE.

What many hours of testing and attempted changes showed me was, it *only* happened if I compiled with Optimize: Fastest. At all other optimize levels, the freezes *never* occurred, but with Fastest they always occurred, immediately. I had been tempted to try GCC9, so I did - and the issue went away, I no longer experienced these freezes. Given other needs to move forward, I noted it as an anomaly to recall in the future, but moved on.

Having now seen this thread, I just went back to my test sketch, confirmed I still have the "freeze" issue under GCC5 optimize: Fastest, then made the changes mentioned here, recomplied, and everything is fine. Great job! I'm not having issues with GCC9 and am definitely liking the improved compiler, warnings, etc. but it's nice to have the option vs. being forced down that path. Thanks again for discovering the root cause!
 
I'm just surfing the thread, but: has anyone verified that those SCB_MPU register writes 1) are emitted by the compiler without any reordering, and 2) are actually executed by the CPU in the specified order?

All of the references to different behavior with different GCC versions and optimization levels make me nervous. It seems like write barriers are needed in more places than ever, as the CPU designers and compiler maintainers get more desperate in their search for optimization opportunities. Just declaring stuff volatile doesn't work so well anymore. :(
 
There were several other threads regarding this phaenomene, the fix helped in every case. A forum search will show them.


I'm just surfing the thread, but: has anyone verified that those SCB_MPU register writes 1) are emitted by the compiler without any reordering, and 2) are actually executed by the CPU in the specified order?

All of the references to different behavior with different GCC versions and optimization levels make me nervous. It seems like write barriers are needed in more places than ever, as the CPU designers and compiler maintainers get more desperate in their search for optimization opportunities. Just declaring stuff volatile doesn't work so well anymore. :(

Volatile is a sequence point. This is required by the C- standard as far as I know.
Different GCC will do the same.
I don't see any change to the Teensyduino compiler-version anytime soon or even this or next year, so even less reason to be nervous. And as PJRC can't do anything re: the compiler behaviour, you perhaps should ask the GCC Team
 
Can anyone who experienced this problem please give 1.54-beta9 a try?

https://forum.pjrc.com/threads/67252-Teensyduino-1-54-Beta-9

This latest beta includes Frank's fix. It also supports Arduino 1.8.15, though the fix applies to all supported Arduino versions.

Would be nice to just double check this bug really is fixed before the upcoming 1.54 release.

@PaulStoffregen:

I can verify that using 1.54beta9 fixes the freeze in my updated TeensyMIDIPolySynth (this version fixes/works equivalent to the manually applied fix which was initially proposed by Frank B).

My verification tests were as follows:

- completely uninstall Arduino 1.8.14 + TD 1.54beta8
- install Arduino 1.8.13 + TD 1.53 (since I'm using an RA8875 tft, I had to modify RA8875UserSettings.h to enable the capacitive TS)
- build target = T4.1 (Serial + MIDI, Smallest Code), bug is verified to be present
- install TD 1.54beta9 (since I'm using an RA8875 tft, I had to modify RA8875UserSettings.h to enable the capacitive TS)
- build target = T4.1 (Serial + MIDI, Smallest Code), bug is verified to be fixed

Mark J Culross
KD5RXT
 
Are you using serialEvents? There is an issues, at least with the Teensy 4.1 where serialEvents may recurse due to the invocation of "yield()" from within the Serial API's. This can result in lockups.

I worked around this by reimplementing the serialEvents through the following header file:
Code:
#if !defined SERIALEVENTRUN_H_DEFINED
#define SERIALEVENTRUN_H_DEFINED

void SerialEventRun(void) __attribute__((weak));

void USBHostEvent(void) __attribute__((weak));
bool USBHost_available() __attribute__((weak));

void SerialEvent() __attribute__((weak));

void SerialEvent1() __attribute__((weak));

void SerialEvent2() __attribute__((weak));

void SerialEvent3() __attribute__((weak));

void SerialEvent4() __attribute__((weak));

void SerialEvent5() __attribute__((weak));

void SerialEvent6() __attribute__((weak));

void SerialEvent7() __attribute__((weak));

void SerialEvent8() __attribute__((weak));

void SerialUSBEvent1() __attribute__((weak));

void SerialUSBEvent2() __attribute__((weak));

void SerialEventRun(void) {
  if (SerialEvent
      && Serial
      && Serial.available() > 0) {
    SerialEvent();
  }
  if (SerialEvent1
      && Serial1
      && Serial1.available() > 0) {
    SerialEvent1();
  }
  if (SerialEvent2
      && Serial2
      && Serial2.available() > 0) {
    SerialEvent2();
  }
  if (SerialEvent3
      && Serial3
      && Serial3.available() > 0) {
    SerialEvent3();
  }
  if (SerialEvent4
      && Serial4
      && Serial4.available() > 0) {
    SerialEvent4();
  }
  if (SerialEvent5
      && Serial5
      && Serial5.available() > 0) {
    SerialEvent5();
  }
  if (SerialEvent6
      && Serial6
      && Serial6.available() > 0) {
    SerialEvent6();
  }
  if (SerialEvent7
      && Serial7
      && Serial7.available() > 0) {
    SerialEvent7();
  }
  if (SerialEvent8
      && Serial8
      && Serial8.available() > 0) {
    SerialEvent8();
  }
  if (USBHostEvent
      && USBHost_available()) {
    USBHostEvent();
  }
#if !defined TEENSY_DEBUG
#if defined USB_DUAL_SERIAL || defined USB_TRIPLE_SERIAL
  if (SerialUSBEvent1
      && SerialUSB1
      && SerialUSB1.available() > 0) {
    SerialUSBEvent1();
  }
#endif
#if defined USB_TRIPLE_SERIAL
  if (SerialUSBEvent2
      && SerialUSB2
      && SerialUSB2.available() > 0) {
    SerialUSBEvent2();
  }
#endif
#endif
}
#endif

Rename your serialEvent functions to SeriaEvent, and call SerialEventRun() at the end of your loop() function.

Patrick
 
Are you using serialEvents? There is an issues, at least with the Teensy 4.1 where serialEvents may recurse due to the invocation of "yield()" from within the Serial API's. This can result in lockups.

I worked around this by reimplementing the serialEvents through the following header file:
...

Rename your serialEvent functions to SeriaEvent, and call SerialEventRun() at the end of your loop() function.

Patrick

@pmurphy - that is a problem that should not exist. Can you post a minimal example that triggers the behavior as seen on a T_4.1?
 
Wow, I found this topic after days of struggling to isolate the semi-random lockups of my T4.1 dashboard. But sometimes they lasted >2mins, sometimes 4 secs. During a lockup, all interrupts were 'frozen', but CAN msgs kept being ACKd.
After the change in startup.c, my tests ran the whole night without lockup. Thank you!
 
Status
Not open for further replies.
Back
Top