Teensy 4.1 Freeze (~1786ms)

Status
Not open for further replies.
I tested with a different program. I'm not entirely sure that the problem is in usb serial.
It's still possible that it is somewhere else and we just think it is there.

The same effect as "raw hid" has using a gcc 9.3. Without any line in the code changed. No freeze anymore.
So... IF there is a bug, GCC 9.3 hides it. Maybe it is more smart. Don't know.
Its quite possible, that the bug is inside GCC 5.4 (the one Teensyduino uses) and it produces weird code.

At the moment, I don't know how to find it. Have to think about it.... yesterday I spend another half hour to place "digitalWriteFasts" everywhere in the USB code.
I did not find any problem there.

The effect with DOOM is - the interrupt which is responsible for the SYNC signal gets stalled. The VGA Monitor switches the screen off, then. So it's easyly visible.
The code for this interrupt swicthes of all interrupts (?) <- i fixed that with removing that. But that's still not the reason. The I rised its priority to 0 (<- the highest).
Now.... tell me what that is... it STILL happens (with GCC 5.4)
The interrupt calls the often used function arm_dcache_flush_delete(). The freeze goes away with removing that. Of course the VGA monitor shows a scramnbled picture now. But... here, as we can see it's NOT USB.
Doom does not print much anyway. At least not here, on the start screen. At the first beginning a few lines are printed, but that's all. When the freeze occurs, there are no prints.

I have NO Idea what the problem is. Perhaps, its the same as with your code, Udo: We remove or change something, and the problem occurs somewhere else. Maybe be the USB code. But that does not mean that USB is the problem...

SO... I know less than before. I know that I know nothing... LOL... and I have NO Idea how to find the problem.

But one thing is good: It happens with DOOM, and it is reproducable within seconds to a few minutes.
 
Last edited:
The question is if he reads here...?

Yes, I am here & reading. I will admit I fell behind on the forum & emails since last Thursday. Many stressful things happened last week. Robin & I actually took Saturday off, and then Sunday was fully consumed with inventory counting and management.

I've just now re-read all 101 messages of this thread. I've put this at the top of my bug list.

As far as I can tell, there are 3 programs to reproduce the problem. Only 1 is short.

1: Modified UDPSendReceiveString on msg #61
2: UdoZ's privately shared code, described on msg #34 "That's a LOT of code .. :) "
3: Doom

Is there any other relatively small program which reproduces this problem?


Finally, may I ask, if we understood why the Freeze changed to 2033ms when CPU speed was set to 396MHz? I had expected we would see an increase by 600/396, but I guess that was naive thinking

This looks very close to the ratio of IPG clock.

When the CPU runs at 600 MHz, IPG is 150 MHz (CPU div by 4)
When the CPU runs at 396 MHz, IPG is 132 MHz (CPU div by 3)



Tried GCC 9
NO freeze so far - (waited for 15 minutes, now)

So, what does that mean? Seems to be a condition that is not existent anymore, with GCC9.

Wow, that is very interesting. But I want to personally spend some time this week before concluding this is a compiler bug. My gut feeling is this is somehow related to how we're using one or more peripherals and likely to return again with any compiler version.
 
My apology, if I report something which has been said before, but I don't know how to relate "USB Type: "Raw HID" to GCC x.y. All I can say, that I complied and ran the version, which had ran successfully, i.e. without Freeze for many hours, after I had replaced all Serial. with Serial5. (and read that out using a Due) today with USB Type: "Raw HID" and it failed after 68minutes. Two Freezes within one Loop(). Something I have never seen before, I mean, I never saw 2 Freezes within such short time, but then, as Frank mentioned, these Freezes seem very random, so most likely it was just a question of time, until there were two within a short period of time.

Anyway, all I wanted to say is this: the issue isn't fixed using USB Type: "Raw HID".
 
Yes, I thought that.

Not sure what you can try now.

Do you want to try GCC9? :) Again, not sure sure if that fixes the freezes. It does it for DOOM.
I'm also using the lates beta of Teensyduino.

I do not say that GCC 5 has a bug that leads to the freezes. But I don't see them with GCC 9 and DOOM.
 
Yes, I thought that.

Not sure what you can try now.

Do you want to try GCC9? :) Again, not sure sure if that fixes the freezes. It does it for DOOM.
I'm also using the lates beta of Teensyduino.

I do not say that GCC 5 has a bug that leads to the freezes. But I don't see them with GCC 9 and DOOM.

How do I get/try GCC9? I am happy to try everything that helps to find the cause of these rare but consistent Freezes.
 
If you're using Windows: I can make a zipfile for you.
Just write a email (click on my name, "send email"). German is OK:)
 
The interrupt calls the often used function arm_dcache_flush_delete(). The freeze goes away with removing that. Of course the VGA monitor shows a scramnbled picture now. But... here, as we can see it's NOT USB..
Yes, I have a arm_dcache_flush_delete in my DMA ISR
 
Walter, Doom freezes there. I'm not sure if other programs would crash there, too. But, USB uses the armd_dcache functions, too.
Udo: I have a zip file prepared for you (I' send the link via mail). Just extract to c:\Arduino and double-click Arduino.exe
Then, select "GCC 9" from the menu:
2021-03-15 18_34_42-Window.png
 
I am happy to try everything that helps to find the cause of these rare but consistent Freezes.

Any chance I can talk you into trimming your program down to as little code as possible to still reproduce the problem and then post it here on the forum?

I started working with the code on msg #61 this morning, then got distracted, but getting back to it now. Need to grab up a Windows machine to run that SocketTest program (but I did write a little C code on Linux which sends the packet but doesn't reproduce the problem).

Still, from what I've seen so far, I'm pretty sure msg #61 isn't going to turn out to be this 1786ms freeze. It's probably some other bug, probably causing a memory fault.
 
I was able to reproduce this problem from msg #61. Is it definitely NOT this 1786ms freeze. Looks like a bug inside NativeEthernet for zero length UDP.

@UdoZ - can you send me code to run here on a Teensy 4.1?
 
UdoZ / WMXZ / Frank B / Paul:

I have also been seeing a "temporary freeze" in the current version of my TeensyMIDIPolySynth that I have mostly converted from hardware control (pots / switches / pushbuttons / LEDs) to all digital control via the RA8875 touchscreen. Because my full sketch is very large (25000+ LOC) & I know that you'd really like the smallest program possible that reproduces the freeze, I have been silently attempting to pare down my sketch in an attempt to come up with another test example that might be useful for further analysis in another programmatic context from the ones that you are already using (UdoZ private code, etc.).

My sketch uses (since we still don't know exactly what all may be contributing to the freeze):
- MIDI
- USBmidi
- USBhostmidi
- compiled for Serial+MIDI
- optimization set for Smallest Code
- EEPROM
- SPI1 + I2C + INTERRUPT (RA8875)
- I2S + SPI Audio Adapter

Here's what I have found to be rock solid true so far (I don't know for certain whether the freeze that I am observing is the same or not):

1) I can reproduce my freeze condition very consistently using only my sketch + Teensy 4.1 + Audio Adapter w/ 25K pot soldered on + USB MIDI input (I use a PC running the Aria Maestosa MIDI player & playing a MIDI song, but any source of MIDI input should work)
2) The temporary freeze is seen when varying the pot on the Audio Adapter (which, in the full hardware setup, controls the backlight brightness, but the display does not have to be connected for the freeze to occur) while the sketch is running
3) The temporary freeze is detectable by noticing that the brightness will not change for approx 3-4 seconds - or if you don't have the RA8875 to connect up, during the freeze, the MIDI player is also temporarily blocked (for approx 3-4 seconds) from sending additional messages (this can be seen directly in Aria Maestosa, or by running MIDI-OX to monitor what is sent/received across MIDI)
4) Whether the temporary freeze occurs or not can be controlled by changing one line from "Serial.println("abcdefghijkl");" (with this line, the freeze does not occur) to "Serial.println("abcdefghijklm");" (by adding the one single additional character, the freeze does occur)
5) Strangely, if I also add 4096 or more additional characters in any other "Serial.println()" statement (I can do this, for example, by enabling the "#define DEBUG_EEPROM_WRITE" line, which is one of my debug capabilities), the freeze no longer occurs - note that the additional code enabled by this #define statement is not actually called by anything at runtime in the current sketch, but the additional code & therefore the additional strings are included & compiled into the sketch by enabling this particular #define statement - including them causes the freeze to no longer occur
6) When the freeze has occurred at least once, a periodic freeze continues to occur quite often, even when the pot is left unchanged - once the problem starts to occur, the problem continues to occur
7) The pared down version of my sketch is still rather large (23000+ LOC), but the fact that adding/removing a single character in a Serial.println() statement easily & directly controls whether the freeze happens or not might be useful to your troubleshooting, as well as the fact that the freeze occurs very often once it has started
8) One final observation (which is not possible to observe with the pared down version since it is so chopped up, but is certainly demonstrated with the full version of my sketch): while the freeze is active, the audio output resulting from the MIDI notes playing is interrupted by a very annoying tone, which is generally the same constant frequency, but occasionally is one octave higher in frequency. When the freeze stops, normal audio resumes in response to the MIDI notes playing, until the next freeze, when audio is once again the annoying tone for the duration of the freeze, etc, etc, etc.

Let me know if you'd like to use either my pared down sketch (freeze control with a one character change - observed by MIDI behavior and/or backlight behavior) or my full sketch (freeze control by enabling/disabling the "#define DEBUG_EEPROM_WRITE" - observed by annoying audio and/or backlight behavior) for use as another test case as the freeze problem is investigated.

Mark J Culross
KD5RXT
 
Last edited:
This looks very close to the ratio of IPG clock.

When the CPU runs at 600 MHz, IPG is 150 MHz (CPU div by 4)
When the CPU runs at 396 MHz, IPG is 132 MHz (CPU div by 3)
It therefore seems that the overrun counter is 2^28. i.e. freeze is 2^28 IPG cycles
 
I was able to reproduce this problem from msg #61. Is it definitely NOT this 1786ms freeze. Looks like a bug inside NativeEthernet for zero length UDP.

@UdoZ - can you send me code to run here on a Teensy 4.1?

Surely, but how can I do this - I can't see a PM facility attached to your profile. In addition, I made before a test without using NativeEthernetUdp.h (meaning I only used NativeEthernet.h only - my sketch uses normally both) and the Freeze remained.
 
Any chance I can talk you into trimming your program down to as little code as possible to still reproduce the problem and then post it here on the forum?
Yes, of course. Unfortunately cutting down takes time, because it always need time to check if Freezes still remain after I made changes. Having said this, I believe I was already able to show that the issue is not related to the use of analogRead(), Teensy RTC, NativeEthernetUdp, U8x8lib, SDI and user ISRs. I have also shown that Freezes are not observed if I make no use of Serial.print (and route messages via Serial5).

Well, based on your findings in NativeEthernet, I will see if I can also strip out NativeEthernet, but that will be tricky, as my code relies heavily on it.
 
My apology, if I report something which has been said before, but I don't know how to relate "USB Type: "Raw HID" to GCC x.y. All I can say, that I complied and ran the version, which had ran successfully, i.e. without Freeze for many hours, after I had replaced all Serial. with Serial5. (and read that out using a Due) today with USB Type: "Raw HID" and it failed after 68minutes. Two Freezes within one Loop(). Something I have never seen before, I mean, I never saw 2 Freezes within such short time, but then, as Frank mentioned, these Freezes seem very random, so most likely it was just a question of time, until there were two within a short period of time.

Anyway, all I wanted to say is this: the issue isn't fixed using USB Type: "Raw HID".


For completion only, here are the final results of this overnight test, using Raw HID: Test Start: 15:23, 16:31 (above mentioned "double" Freeze), then Freezes 03:27, 3:31, 7:31. Test Stop: 10:18 -> there can be many hours, or very little time between Freezes, therefore patience is required, when trying out new things.
 
@Paul I can also make a package for you, with Doom.
It's quite much code too, but the freeze happens in the VGA Library, in the interrupt, so it's easy to see. And happens withing a few minutes, sometime quite often, sometimes it take some minutes to the first freeze.
You don't need a VGA connector or monitor, it's enough to monitor the sync pin (but of course a connected monitor would be better), or to add a digitalWrite (with a LED ) to the ISR.
Note, I did not measure the duration so far But it seems to be somewhere in the range 1.5-2sec, too.
 
Measured it now. Excactly the same duration. 1.789 seconds.
Interestingly, I had to use ARM_DWT_CYCCNT to measure it. With micros() it did not happen.. (but I waited 5 minutes only)
 
No:
Code:
Bibliothek VGA_t4-master in Version 1.0 im Ordner: C:\Users\Frank\Documents\Arduino\libraries\VGA_t4-master  wird verwendet
Bibliothek USBHost_t36 in Version 0.1 im Ordner: C:\Arduino\hardware\teensy\avr\libraries\USBHost_t36  wird verwendet
Bibliothek SD in Version 2.0.0 im Ordner: C:\Arduino\hardware\teensy\avr\libraries\SD  wird verwendet
Bibliothek SdFat in Version 2.0.5-beta.1 im Ordner: C:\Arduino\hardware\teensy\avr\libraries\SdFat  wird verwendet
Bibliothek SPI in Version 1.0 im Ordner: C:\Arduino\hardware\teensy\avr\libraries\SPI  wird verwendet

Here is the output of my test code:
Code:
!!!! 2F82FE21 [COLOR=#ff0000]1073706886[/COLOR]
!!!! A0751B1 [COLOR=#ff0000]1073706886[/COLOR]
!!!! B518D7F9 [COLOR=#ff0000]1073706886
[/COLOR]!!!! 48E48025[COLOR=#ff0000] 1073706886
[/COLOR]!!!! B418879D [COLOR=#ff0000]1073706886
[/COLOR]!!!! 3C35AED9 [COLOR=#ff0000]1073706886
[/COLOR]!!!! 89AC351D [COLOR=#ff0000]1073706886
[/COLOR]!!!! 9B78BB56 [COLOR=#00ff00]1073630427[/COLOR]
!!!! B7190971 [COLOR=#ff0000]1073706886
[/COLOR][COLOR=#00ff00][/COLOR]!!!! E9E03612 [COLOR=#00ff00]1073401707[/COLOR]
[COLOR=#ff0000][/COLOR]!!!! B79B4689 [COLOR=#ff0000]1073706886[/COLOR]
[COLOR=#00ff00][/COLOR]!!!! 79C80EB2 [COLOR=#00ff00]1073153927[/COLOR]
[COLOR=#ff0000][/COLOR]!!!! 76313AE1 [COLOR=#ff0000]1073706886
[/COLOR]!!!! E23470D9 [COLOR=#ff0000]1073706886[/COLOR]
The first number is ARM_DWT_CYCCNT (as HEX) the second the duration(decimal).
Why is the duration _exactly_ the same number of cycles? That's pretty weird. Is there a bug in my added code?
Edit: However, the absolute value of ARM_DWT_CYCCNT seems pretty random, as I would expect.
1073706886 is 3FFF 7786
1.789551141 seconds.

Edit: ahhh... there was one measurement with an other duration now.
Edit: Added more numbers

Code:
 unsigned long m = ARM_DWT_CYCCNT;
[....]
#if 1  
  m = ARM_DWT_CYCCNT - m;
  if (m > 600000000) {      
      digitalWriteFast(0,1);      
      Serial.print("!!!! ");
      Serial.print(ARM_DWT_CYCCNT, HEX);
      Serial.print(" ");      
      Serial.println(m);
#endif

This is the whole interrupt code:
Code:
FASTRUN void VGA_T4::QT3_isr(void) {
  TMR3_SCTRL3 &= ~(TMR_SCTRL_TCF);
  TMR3_CSCTRL3 &= ~(TMR_CSCTRL_TCF1|TMR_CSCTRL_TCF2);
  
  
  cli();
    
  unsigned long m = ARM_DWT_CYCCNT;
  // V-PULSE
  if (currentLine > 0) {
    digitalWrite(_vsync_pin, 1);
    VSYNC = 0;
  } else {
    digitalWrite(_vsync_pin, 0);
    VSYNC = 1;
  }
  
  currentLine++;
  currentLine = currentLine % 525;


  int y = (currentLine - TOP_BORDER) >> line_double;
  // Visible area  
  if (y >= 0 && y < fb_height) {  
    // Disable DMAs
    //DMA_CERQ = flexio2DMA.channel;
    //DMA_CERQ = flexio1DMA.channel; 

    // Setup source adress
    // Aligned 32 bits copy
    unsigned long * p=(uint32_t *)&gfxbuffer[fb_stride*y];  
    flexio2DMA.TCD->SADDR = p;
    if (pix_shift & DMA_HACK) 
    {
      // Unaligned copy
      uint8_t * p2=(uint8_t *)&gfxbuffer[fb_stride*y+(pix_shift&0xf)];
      flexio1DMA.TCD->SADDR = p2;
    }
    else  {
      p=(uint32_t *)&gfxbuffer[fb_stride*y+(pix_shift&0xc)]; // multiple of 4
      flexio1DMA.TCD->SADDR = p;
    }

    // Enable DMAs
    DMA_SERQ = flexio2DMA.channel; 
    DMA_SERQ = flexio1DMA.channel; 
    arm_dcache_flush_delete((void*)((uint32_t *)&gfxbuffer[fb_stride*y]), fb_stride);
  // arm_dcache_flush((void*)((uint32_t *)&gfxbuffer[fb_stride*y]), fb_stride); //FB no change
  }  
  
  sei();  

#ifdef DEBUG
  ISRTicks++; 
#endif  
  asm volatile("dsb");
#if 1  
  m = ARM_DWT_CYCCNT - m;
  if (m > 600000000) {      
      digitalWriteFast(0,1);      
      Serial.print("!!!! ");
      Serial.print(ARM_DWT_CYCCNT, HEX);
      Serial.print(" ");      
      Serial.println(m);
#endif
  }
}
 
Last edited:
Quite possible that there are more freezes...outside the interrupt..and i just don't see them(?!?)
The freezes above occured within ~45 minutes.
 
I have upload everything needed here: https://drive.google.com/file/d/1PXLmRAM5Y-zrOPFFY3P9g9hhmYufSs-a/view?usp=sharing
It's not needed to connect something to the USB -Host.
Edit: You'll need a PSRAM.
A LED on Pin 0 is useful. It will lit when the freeze occurs.
The SD-Card contents are in the zip, the library, and even an compiled hex file.
Just insert the SD Card, start the program with serial monitor opened and wait.


the output should like this:
Code:
frequency
251760
div
10.00
USB2 PLL running
 reset waited 6
USBHS_ASYNCLISTADDR = 0
USBHS_PERIODICLISTBASE = 20013000
periodictable = 20013000
doom.wad
SD initialized, files found: 1
Current callibration params:
34
60
285
222
If it does not look like this, something is wrong.


GCC9: Now it happens with GCC9, too. :-(
SO.. forget averything I wrote for GCC9

Output with GCC9 now:
Code:
frequency
251760
div
10.00
USB2 PLL running
 reset waited 6
USBHS_ASYNCLISTADDR = 0
USBHS_PERIODICLISTBASE = 20013000
periodictable = 20013000
doom.wad
SD initialized, files found: 1
Current callibration params:
34
60
285
222
!!!! B161F2AE 1073706873
!!!! A5FA307E 1073706873
!!!! B23041DE 1073706873
!!!! B5D5BD22 1073706873
!!!! B693DA36 1073706873
!!!! 687B0FC6 1073706873
!!!! B10FC506 1073706873
!!!! 42B63E5E 1073706873
!!!! 19344BBA 1073706873
!!!! 2E0D45C2 1073706873
!!!! B44563C2 1073706873
!!!! 425F97FE 1073706873
!!!! B25697BA 1073706873
!!!! B314FF42 1073706873
!!!! B6B12C06 1073706873
!!!! A4CCC27E 1073706873
!!!! EE0FAD2A 1073706873
!!!! 7F07F116 1073706873
!!!! 47E7C8D6 1073706873
!!!! B2DA7792 1073706873
!!!! 2222F76E 1073706873
!!!! 8017040E 1073706873
!!!! EDFB82A 1073706873
!!!! 79DB6AF2 1073706873
!!!! B3F06E26 1073706873
!!!! 6ADBF4B6 1073706873
!!!! B367A5EA 1073706873
!!!! B59DCEE2 1073706873
!!!! B4F6329E 1073706873
!!!! 1FF1E566 1073706873
!!!! B69CFA32 1073706873
!!!! B6A3DDCE 1073706873
!!!! 875CC7BE 1073706873
!!!! 9FCB0AEE 1073706873
!!!! B105B6EE 1073001445
!!!! 8E9C04E2 1073706873
!!!! B77D8952 1073706873
!!!! 70989FB2 1073706873
!!!! B6464136 1073706873
!!!! 905997BA 1073706873
!!!! 8CD4363A 1073706873
!!!! B168ED96 1073706873
!!!! B2300EB6 1073706873
!!!! 8F212A76 1073706873
!!!! A78F6DA6 1073706873
!!!! 105B909E 1073706873
!!!! B613E2DE 1073706873
!!!! 26C21C12 1073706873
!!!! B4DC9AC2 1073706873
!!!! 98F89346 1073706873
!!!! B2D628C2 1073706873
!!!! 5B6BF5CA 1073706873
!!!! 7BB3959E 1073706873
!!!! 75FEA44E 1073706873
!!!! D79FEA82 1073706873
!!!! B1B38B7A 1073706873
!!!! 49D661B6 1073706873
!!!! B4D2147E 1073706873
!!!! 3C6FEBD6 1073706873
!!!! B16B8706 1073706873
!!!! 1E6C86A6 1073706873
!!!! 8968396E 1073706873
!!!! 3E2D7EAE 1073706873
!!!! B8366EEA 1073706873
!!!! B2608922 1073706873
!!!! B54E3606 1073706873
!!!! B3EF5F76 1073458885
!!!! 60D1FA66 1073706873


not sure what has changed...

Edit: Happens with the Teensy connected to a phone charger, too (No active USB) - Even faster.
 
Last edited:
@Paul:
Is this correct (?)
Code:
    SCB_MPU_RBAR = [COLOR=#00ff00]0x70000000[/COLOR] | REGION(i++); // FlexSPI2
    SCB_MPU_RASR = MEM_CACHE_WBWA | [COLOR=#00ff00]READONLY[/COLOR] | NOEXEC | SIZE_256M;

    SCB_MPU_RBAR =[COLOR=#00ff00] 0x70000000 [/COLOR]| REGION(i++); // FlexSPI2
    SCB_MPU_RASR = MEM_CACHE_WBWA | [COLOR=#00ff00]READWRITE [/COLOR]| NOEXEC | SIZE_16M;
The 2nd has higher priority - so the first is not active. But can the MPU handle this correctly?

Don't want produce the next false positive here...take this with care: Doom is running for 30minutes now, with the first commented.-out, and without freeze. That's unusual..
 
Good find Frank - hopefully that is the answer.

For others to try, this line is in : ...\hardware\teensy\avr\cores\teensy4\startup.c

About line #263. If I'm following the Freeze goes away when those lines are changes to this - with two BOLD lines commented:
Code:
	SCB_MPU_RBAR = 0x60000000 | REGION(i++); // QSPI Flash
	SCB_MPU_RASR = MEM_CACHE_WBWA | READONLY | SIZE_16M;

	[B]// SCB_MPU_RBAR = 0x70000000 | REGION(i++); // FlexSPI2
	// SCB_MPU_RASR = MEM_CACHE_WBWA | READONLY | NOEXEC | SIZE_256M;[/B]

	SCB_MPU_RBAR = 0x70000000 | REGION(i++); // FlexSPI2
	SCB_MPU_RASR = MEM_CACHE_WBWA | READWRITE | NOEXEC | SIZE_16M;

	// TODO: protect access to power supply config
 
Status
Not open for further replies.
Back
Top