Teensy 4.1 Lockup

Even TyComm overwhelms the machine at times and dumps 'system data' from somewhere instead of USB incoming data - not news.

Boring and dizzying watching and trying to see what the average is as it comes and goes.

Running the following against t_Sermon still results in quick stopping:
Code:
...
count=10041128, lines/sec=0
count=10041129, lines/sec=0
count=1004

Built as Dual Serial with stats to get some meaning from the spew against TyCommander:
Code:
...
 Grp to 50000 sum 254354 
 Grp to 100000 sum 9186332 
 Grp to 150000 sum 1167211 
 Grp to 300000 sum 135041 
 Grp to 350000 sum 1910473 
 Grp to 400000 sum 1309225 
 Grp to 500000 sum 8205257 
 Grp to 550000 sum 4543060 
 Grp to 600000 sum 6676894 
 Grp to 650000 sum 2476576 
 Grp to 700000 sum 3559980 
 Grp to 750000 sum 575597 
	 @count50000000

...
 Grp to 50000 sum 254354 
 Grp to 100000 sum 9186332 
 Grp to 150000 sum 1167211 
 Grp to 300000 sum 135041 
 Grp to 350000 sum 1910473 
 Grp to 400000 sum 1309225 
 Grp to 500000 sum 8872405 
 Grp to 550000 sum 4543060 
 Grp to 600000 sum 39819151 
 Grp to 650000 sum 110148705 
 Grp to 700000 sum 107430006 
 Grp to 750000 sum 41085060 
 Grp to 800000 sum 160157922 
 Grp to 850000 sum 3981055 
	 @count500000000

With IDE SerMon as primary it only runs to here:
Code:
...
count=16228949, lines/sec=651164
count=16228950, lines/sec=651164
count=16228951, lines/sec=651164
with stats up to there:
Code:
 Grp to 50000 sum 504584 
 Grp to 400000 sum 804025 
 Grp to 450000 sum 1329998 
 Grp to 500000 sum 1028316 
 Grp to 550000 sum 392432 
 Grp to 600000 sum 661922 
 Grp to 700000 sum 1278723 
	 @count16000000

Altered code for USB1:
Code:
// post #2626 original Paul sketch
// https://forum.pjrc.com/threads/54711-Teensy-4-0-First-Beta-Test?p=204681&viewfull=1#post204681
// https://forum.pjrc.com/threads/68199-Teensy-4-1-Lockup?p=288825&viewfull=1#post288825


// Simple test of Serial.print() speed.  How many lines/sec can your
// Arduino board print to the serial monitor?

// This combination of printing text and numbers is meant to test the
// most common way Serial printing is used with Arduino.

// On true serial boards, this test measures how much of the bandwidth
// of a high baud rate can really be used.  Efficiency copying bytes
// into the transmit buffer, and the serial transmit interrupt, are
// the main factors.

// On native USB boards, this test measures how well the USB stack can
// utilize the USB bandwidth.  USB is fast but complex protocol.  This
// test tends to measure how well optimized the USB code is.  For
// 12 Mbit/sec (full speed) USB, 36848 is the theoretical maximum
// lines/sec.  For 480 Mbit/sec (high speed) USB, 1521371 is the
// theoretical maximum lines/sec speed.  Unlike ordinary serial, your
// operating system and its drivers, the type of USB port, whether a
// USB hub is used, and CPU load on your computer can affect the speed.

uint32_t count, prior_count;
uint32_t prior_msec;
uint32_t count_per_second;

// Uncomment this for boards where "SerialUSB" needed for native port
//#define Serial SerialUSB

void setup() {
  Serial.begin(1000000); // edit for highest baud your board can use
  while (!Serial) ;
  count = 10000000; // starting with 8 digits gives consistent chars/line
  prior_count = count;
  count_per_second = 0;
  prior_msec = millis();
}

uint32_t groups[40];
void loop() {
  Serial.print("count=");
  Serial.print(count);
  Serial.print(", lines/sec=");
  Serial.println(count_per_second);
  count = count + 1;
  uint32_t msec = millis();
  if (msec - prior_msec > 1000) {
    // when 1 second as elapsed, update the lines/sec count
    prior_msec = prior_msec + 1000;
    count_per_second = count - prior_count;
    prior_count = count;
  }
[B]  groups[ count_per_second / 50000 ]++;
  if ( !(count%1000000) ) {
    for ( int ii=0; ii<40; ii++ ) {
      if ( 0!=groups[ii] ) {
        SerialUSB1.printf( " Grp to %u sum %u \n", (1+ii)*50000, groups[ii] );
      }
    }
    SerialUSB1.print("\t @count");
    SerialUSB1.println(count);
  }
[/B]}
 
@Paul
Out of curiosity I ran the usb_serial_print_speed sketch.

Code:
...
...
count=10037586, lines/sec=0
count=10037587, lines/sec=0
count=10037588, lines/sec=0
count=10037589, lines/sec=0
cou[B][COLOR="#FF0000"] <---- Stalls here[/COLOR][/B]

Opening and closing the serial monitor restarts it.
Code:
count=38956797, lines/sec=847414
count=38956798, lines/sec=847414
count=38956799, lines/sec=847414
count=3895680[B][COLOR="#FF0000"] <---- Stalls here again[/COLOR][/B]

This is on a Windows10x64 with the Locked T4.
 
@Paul
Out of curiosity I ran the usb_serial_print_speed sketch.

Code:
...
...
count=10037586, lines/sec=0
count=10037587, lines/sec=0
count=10037588, lines/sec=0
count=10037589, lines/sec=0
cou[B][COLOR="#FF0000"] <---- Stalls here[/COLOR][/B]

Opening and closing the serial monitor restarts it.
Code:
count=38956797, lines/sec=847414
count=38956798, lines/sec=847414
count=38956799, lines/sec=847414
count=3895680[B][COLOR="#FF0000"] <---- Stalls here again[/COLOR][/B]

This is on a Windows10x64 with the Locked T4.

So we both did that (p#25) and got similar results!

IDE SerMon still can't handle as before - and for Ref TyComm still works well showing it isn't the Teensy falling down ;;.
 
So we both did that (p#25) and got similar results!

IDE SerMon still can't handle as before - and for Ref TyComm still works well showing it isn't the Teensy falling down ;;.
Yep figured a good point to do that so have 2 data points win 10 and win11
 
Last edited by a moderator:
Yep figured a good point to do that so have 2 data points win 10 and win11

Indeed - very good not just tested on one machine. We got a heck of a windstorm and power was out 6 pm to 1:44 am - so running on generator and already put laptop away the other test was run on.

My adding the Dual Serial for STATS didn't alter the primary behavior ... I should have put it under #ifdef so it ran as posted on Single USB.

TyComm thrashes so hard keeping up the UI becomes ignorant of clicks and selects. If you run that w/Stats it is funny how it starts tending to be 'under 250K lps' then the higher buckets start hitting over time - then become the primary hits.

Just confirmed the RamDrive I added in 6GB of the 32GB on this machine is being used for the TyComm log files. It is noted to be 1.5X SSD speed - but that doesn't keep TyComm from showing garbage at times and getting overwhelmed.

So far Win 11 diffs are just minor UI alterations and all else seems to be unchanged as far as usual function ... except for slow Explorer rename and drag drop ... like a floppy disk instead of an SSD ?????
 
I rebooted that Macbook Air to Windows 10, and sure enough the serial monitor stalls, usually with 10 minutes.

Just as a data point - with me it was a second or two (didn't measure but was quick) running the USB test. With the original test sketch -yep ran about that amount of time.
 
Yes, it's a hard freeze and the Teensy is unresponsive. COM port is greyed out and a power cycle is required. Good idea about the crash report, thanks! So far just adding back in chunks of code and hardware to see if I can isolate the cause.

Just to bring the original hard freeze to a close - it was I2C on a cable that was causing the issues. Switching over to a protocol meant for cables (i.e. CAN bus) solved the issue.
 
Just to bring the original hard freeze to a close - it was I2C on a cable that was causing the issues. Switching over to a protocol meant for cables (i.e. CAN bus) solved the issue.

Thanks for letting us know what the original hard freeze resolution was. Have to keep that one in the back of the mind.
 
Working on this problem today. Please give this copy of teensy_serialmon.exe a try. It goes into {Arduino}/hardware/tools, replacing the copy that's already installed. The default {Arduino} location is C:\Program Files (x86)\Arduino.

Best to quit Arduino, but just closing the serial monitor window should be enough for Windows to allow you to overwrite the original file.

Does this new version fully solve the problem on your machine?
 

Attachments

  • teensy_serialmon.zip
    22.4 KB · Views: 77
No longer STALLING quickly.

See strings of 'system garbage' fly by. Goes Clear - then garbage- then clear.
> sometimes it runs GUI updates fast scrolling - then fitful pauses - and now again 800K and steady streaming.

tSM_a.png

Still running past 408M counts

<edit>: At 700M counts ... will let it run - No 'garbage' in some time - and good updates
tSM_b.png

System load not real high.
 
@Paul - seems you fixed something - not the Windows BUFFER bug - but t_sermon doing well and looking good just now.

Still running - over an hour between posts and running before that:
tSM_c.png

UPDATE: It gave up the ghost here: count=3,913,515,160
Code:
...
count=3913515160, lines/sec=573526
count=3913515161, lines/sec=573526
count=3913515

Starting with "ifdef"Dual USB using TyComm to gather stats and see if it changes anything:
Code:
 Grp to 50000 sum 205718 
 Grp to 100000 sum 438423 
 Grp to 150000 sum 2397453 
 Grp to 200000 sum 4444757 
 Grp to 250000 sum 2413710 
 Grp to 300000 sum 3603310 
 Grp to 350000 sum 534094 
 Grp to 400000 sum 962535 
	 @count25000000

...

 Grp to 50000 sum 205718 
 Grp to 100000 sum 438423 
 Grp to 150000 sum 2397453 
 Grp to 200000 sum 4743284 
 Grp to 250000 sum 2471652 
 Grp to 300000 sum 4017885 
 Grp to 350000 sum 3827966 
 Grp to 400000 sum 2453690 
 Grp to 450000 sum 4622664 
 Grp to 500000 sum 849096 
 Grp to 550000 sum 22379562 
 Grp to 600000 sum 3911706 
 Grp to 650000 sum 6020031 
 Grp to 700000 sum 6783385 
 Grp to 800000 sum 679621 
 Grp to 850000 sum 19197864 
	 @count95000000
 
Last edited:
See strings of 'system garbage' fly by. Goes Clear - then garbage- then clear.

seems you fixed something - not the Windows BUFFER bug

I only attempted to fix the "lockup" bug, where the serial monitor would stall and no longer show any data Teensy transmits.


I am aware of 3 other possible issues on Windows, all of which might be called "the Windows BUFFER bug". My hope in writing this detailed message is we can all be aware of these and talk more specifically about which problems are being seen.

1: Buffer bloat: An excessive amount of incoming data can become buffered on the PC side. This effect can be seen by crafting a program which suddenly stops transmitting, like when a button connected to a pin is pressed. If running a benchmark like lines/sec (modified to stop on command), you can estimate the size of bloated buffers by trying to measure the time from actually stopping transmission until the GUI stops updating.

2: Inconsistent performance: While running benchmarks like lines/sec, performance can vary quite a lot. Whether this correlates with other activity on the PC is unknown (at least to me).

3: Corrupted data: During times of high speed data flow, occasionally garbage data is received. It appears to be the contents of memory somewhere on the Windows machine, rather than the data Teensy transmitted.

My hope is to fix issue #1 in the future by using localhost networking rather than anonymous pipes (the way Windows implements stdin / stdout) for the connection between teensy_serialmon.exe and the IDE's serial monitor window.

I'm pretty sure #3 is a bug within Windows or the Windows-provided USBSER.SYS driver. Only Microsoft can fix that. It seems to happen with Coolterm and all other programs cable of running at these higher speeds. I currently do not have any way of reporting this problem to Microsoft.

Whether anything can be done about #2, I do not know.
 
Couldn't sleep and saw your message so figured might as well test it out.

Set up 2 T4's, one locked and 1 unlocked. Running the original tempMon sketch on one and the usb lines per second sketch.

As at defragster pointed out
defragster said:
See strings of 'system garbage' fly by. Goes Clear - then garbage- then clear.
> sometimes it runs GUI updates fast scrolling - then fitful pauses - and now again 800K and steady streaming.

The only thing I will add is that after the garbage cycle clears (as defined by Tim) the sketch does pick up again.

But neither sketch is stalling the serial monitor as before!!!!!!

In terms of item #2 Inconsistent performance, as the test is running (and still is) I was running Cura and didn't see any change (that I noticed) in lines/sec. I know that is purely anecdotal but figure its a data point.

So far been running for awhile - lost tract of time - maybe an hour or so:
Capture.PNG

Will update if and when it either serial monitor stalls.

UPDATE: at about a count of 3.5million garbage struck again but the picked cleared and serial monitor recovered. So far garbage twice in 3.5million counts. Original sketch running no issues.

Update 1: Serial Monitor still going strong:
Lines/sec:
Code:
count=4145391380, lines/sec=504740
count=4145391381, lines/sec=504740
count=4145391382, lines/sec=504740

Test Sketch in post#1
Code:
53.13°C	5891436
53.13°C	5891446
53.13°C	5891456

Going to let it keep running until morning and s
 
Last edited:
If corrupted is more technical than 'garbage' - the jest about fixing windows referred to #3 - just noting as noted as seen before - it is apparently a bug in how Windows passes buffers. TyCommander and the SerMon's both get that.

Indeed #2 is more nebulous - it certainly Surges/delays at times. There is no overt contention for resources plenty of RAM and no Core alone is above 50% - but no way of knowing where the USB load impinges on processing or gets counted in TaskMan displayed stats, if directly at all.

Got a new 'Samsung USB C' drive for transfer to laptop - quoted at up to 400MB/s - it is getting filled from SSD folder - and filling horribly slowly, so that is sharing resources - but Copy Paused doesn't change anything.

Moved T_4.1 from HUB to front port - died quickly at only 39M counts. Versus through USB 3 Hub - more variables

There is clearly a #1 Buffer bloat. Data comes in faster than the GUI can scroll it out.
 
powered up my 11th gen i7 laptop with programmed T_4.1
It doesn't run very long. Just the Explorer open to open then open IDE open with prior teensy_serialmon #1 update:

Code:
count=57222822, lines/sec=717068
count=57222823, lines/sec=717068
count=57222824, lines/sec=717068
count=572

.. wait it was long stalled after a halting start - now running again .... maybe hiding the window with browser gave it life before?
okay ran a short time, stalled ... then blank with no scroll bar Not showing (offline) and the AutoScroll check still active

Started another T_4.1 on Win 11 desktop before above - it seems still running

UI very non-responsive - at 848 M counts - but right scroll shows much below ... finally redrew on resize and checked autoscroll again - but not updating.

As I returned it had just dumped a bunch of non printable chars , then resumed.


Looking at TaskMan I see 32 bit Java? Any chance a 64 bit version would run better?
 
Last edited:
Looking at TaskMan I see 32 bit Java? Any chance a 64 bit version would run better?

Difficult to say, as Arduino doesn't publish any copies of the IDE (for Windows) using a 64 bit JRE.

If you wanted to experiment, you could try replacing C:\Program File (x86)\Arduino\java with a 64 bit JRE. Arduino uses version 8u242. This page has version 8u302. I'm not sure where to download the older 8u242 version.

https://adoptopenjdk.net/releases.html?variant=openjdk8&jvmVariant=hotspot

You'll see 4 files. You want the JRE, not the full JDK. I'm assuming the ZIP and MSI are the same files. You probably want the ZIP archive so you can extract it at a location of your choosing and then copy its contents into the {Arduino}/java folder.

I'm not entirely sure if just replacing the JRE will even work, as all the other non-Java stuff in the IDE is still 32 bits. In theory you could recompile everything, but newer versions of Arduino have stuff implemented in Google Go language...
 
Last edited:
@Paul - @defragster

Been running both sketches now for about 6 hours and still seems to be working on a Windows10x64 i7-8700K.

usb lines per second
Code:
count=940,733,502, lines/sec=503292

Test Sketch:
Code:
54.44°C	24704705

Looks like the issue of the stalled Serial Monitor is resolved - at least on my machine.

Out of curiosity what was the issue with the stalling?
 
Out of curiosity what was the issue with the stalling?

It turned out to be a deadlock condition between threads caused by code which attempts to manage buffers in a way to reduce the number of WIN32 system calls.

Or just "Paul made a mistake while trying to optimize too much" would be the simple explanation.
 
It turned out to be a deadlock condition between threads caused by code which attempts to manage buffers in a way to reduce the number of WIN32 system calls.

Got it thanks. By the way is there any other way you want us to test Serial monitor to make sure its working? Usb-lines/sec seem to be a good job of stressing.

:)
 
More serial monitor testing probably isn't worthwhile. I've already spent quite a lot of time on the 3 known issues in msg #38. They're not going to be fixed anytime soon. Unless any other problems are discovered, I'm not looking forward to rehashing those 3 known issues.

I would like to investigate the I2S lockup issue mentioned in msg #33. Especially if it was using the Wire library, all I2C problems are supposed to cause those functions to return with error code. Lockups are never supposed to happen.

Brian, if you're still reading, can you give us some detail on how to reproduce that I2C lockup?
 
More serial monitor testing probably isn't worthwhile. I've already spent quite a lot of time on the 3 known issues in msg #38. They're not going to be fixed anytime soon. Unless any other problems are discovered, I'm not looking forward to rehashing those 3 known issues.

I would like to investigate the I2S lockup issue mentioned in msg #33. Especially if it was using the Wire library, all I2C problems are supposed to cause those functions to return with error code. Lockups are never supposed to happen.

Brian, if you're still reading, can you give us some detail on how to reproduce that I2C lockup?

Got it. Just to close the SerialMonitor test out got to 2,844,425,876 before stopping. And just a side note in @brtaylor test sketch he had a delay(10) between prints. Did some experimenting and it seems the minimum delay if you you are going to do something like that is
Code:
delayMicroseconds(50);
At 50mirosecond delay it does stall the serial monitor. Didn't try anything between 50 and 100 though.

Will be interesting Brian's I2C issue especially with the stuff he works with.
 
More serial monitor testing probably isn't worthwhile. I've already spent quite a lot of time on the 3 known issues in msg #38. They're not going to be fixed anytime soon. Unless any other problems are discovered, I'm not looking forward to rehashing those 3 known issues.

I would like to investigate the I2S lockup issue mentioned in msg #33. Especially if it was using the Wire library, all I2C problems are supposed to cause those functions to return with error code. Lockups are never supposed to happen.

Brian, if you're still reading, can you give us some detail on how to reproduce that I2C lockup?

Hi Paul, let me see if I can replicate the I2C piece with a simple example.
 
Thanks. There's probably nothing that can be done to make I2C actually work over such a cable, but at least the Wire library should always return an error. It should never crash, no matter what (non-destructive) hardware is used.
 
Difficult to say, as Arduino doesn't publish any copies of the IDE (for Windows) using a 64 bit JRE.

If you wanted to experiment, you could try replacing C:\Program File (x86)\Arduino\java with a 64 bit JRE. Arduino uses version 8u242. This page has version 8u302. I'm not sure where to download the older 8u242 version.

https://adoptopenjdk.net/releases.html?variant=openjdk8&jvmVariant=hotspot

You'll see 4 files. You want the JRE, not the full JDK. I'm assuming the ZIP and MSI are the same files. You probably want the ZIP archive so you can extract it at a location of your choosing and then copy its contents into the {Arduino}/java folder.

I'm not entirely sure if just replacing the JRE will even work, as all the other non-Java stuff in the IDE is still 32 bits. In theory you could recompile everything, but newer versions of Arduino have stuff implemented in Google Go language...
Doesn't seem like a simple replace - if these steps replaced the JRE executables?
Downloaded the : OpenJDK8U-jre_x64_windows_hotspot_8u302b08.zip
and replaced in : C:\T_Drive\Arduino_1.8.16_155 - Copy\java { my Arduino is from ZIP so nothing installed in \Programs }

The "C:\T_Drive\Arduino_1.8.16_155 - Copy\arduino.exe" then shows first load image and goes away.

No Joy. Something is expecting 32b - or some other element customized or added missing or not prepared to swap 32 for 64 bit?
 
Back
Top