Arduino IDE2 - Serial Monitor sometimes does not work - continue from Upload Thread

[defragster]

Indeed, so much shorthand is one of the things that also confuses me at times.

Oppps - will look at better intro of any shorthand - but typing the same thing over and over makes long posts even longer.
That and usually stunned something odd happened {that was caught and not flushed away without a trace of no repro} in the middle of looking for something else the shorthand notes are left to get back to the task at hand.

Will reboot this machine and try again - speakers are playing scratchy Audio ... and check that plug.

[defragster]

Win 11 PC restarted and T_4.1 plugged in. Opened IDE 2.0.2 and selected Tools / Port / teensy ports / COM20 as none was selected.
{PC Sound now with clear no static - using 16GB RAM instead of 26GB}
TD is 1.57.1 with updated teensy-monitor.exe 11/25/22. Serial Monitor open and gathered output from starting Teensy.
Sketch is prior with delay(1000) active in setup() and showSome( 19 ); left from prior test.

Hit Ctrl+U and wait for upload looking at Output console window as T_4.1 LED goes solid ... then blinks some time later.

Click Serial Monitor tab to see fresh output {as expected no visible output until manual scroll}

:\T_Drive\tCode\FORUM\serialIDE2test\serialIDE2test.ino Dec  3 2022 20:07:33
run secs A=18.69

C:\T_Drive\tCode\FORUM\serialIDE2test\serialIDE2test.ino Dec  3 2022 20:07:33
run secs B=19.69
See Me: 1
 // ...

SerMon PC send : Rebooted PC then Ctrl+U - as above:

 // ...
Rebooted PC
See Me: 20
See Me: 21
See Me: 22

C:\T_Drive\tCode\FORUM\serialIDE2test\serialIDE2test.ino Dec  3 2022 20:17:14
run secs A=18.78
 // ...

... Other stuff ... - still 18+ secs to start Teensy Ports Serial Monitor
Another CtrlU, switch to IDE SerMon and Ctrl+U, Back to Teensy Ports Serial Monitor and Ctrl+U
>> No idea what caused it to go from 18 to 8 seconds before ...

Ctrl+U and switch to Teensy Ports Serial Monitor in a second or so and no blink, send 'gg' three times and no Teensy Ports Serial Monitor connect for 18.7 seconds
The three 'gg' output Messages were PC sent and T_4.1 echoed during setup() call to showSome( 19 );

Switch back to IDE SerMon and Ctrl+U twice connected in 1.21 secs and 1.02 secs and any PC output can't help but appear from loop().

[PaulStoffregen]

I'm continuing to investigate this problem. Seems like something is causing Windows WM_DEVICECHANGE messages to be delayed or sometimes not arrive at all, which plays havoc with teensy-discovery.exe and teensy-monitor.exe ability to observe arrival and removal of USB devices.

At least for the 1 case I can reliably reproduce on my Windows test machine, using Task Manager to kill the serial-discovery.exe process seems to solve the problem.



@Defragster & anyone else seeing this issue, please try killing that process and let me know if it solves all the problems, or only some, or none?

Obviously this isn't a good solution, but right now my goal is just to learn more about what's causing the problem. Feedback about how this changes things on other Windows systems would really help.

[KurtE]

Paul,

Will try that, when I reproduce this problem again.

Keep wondering what the difference is between when we first startup the IDE and you can reproduce it versus after it is up and lack of better terms "Stable" on why it reproduces early.

Some Random thoughts come to mind:

a) When the IDE starts up and reloads one or more sketches as part of the startup process. Maybe the board Serial port information is not quite right and/or maybe it is not properly loading support stuff like serial-discovery?

b) I wonder if it makes a difference in the reproducing steps if the Teensy is already connected to USB before the IDE is started or not?

c) Wonder if some of the other startup tasks might be somehow interfering? For example, at startup, the IDE may check for updated boards and libraries.
For whatever reason, on my machine, I see info message stuck on downloading Teensy boards information for a pretty long time. The others all go pretty quick. But it only does this if, a certain amount of time has elapsed since the last time they were downloaded. I believe 4 hours, but could be wrong.

But if you turn the checking off, does that make any differences in reproducing this?
(ctrl+shift+p) try typing preferences, look for Open Settings(UI), do search for update ...

Yes, I know, throwing darts!

[PaulStoffregen]

Tried turning off "Arduino: Check For Updates". At least for the problem I'm able to reproduce, made no difference.

Likewise, trying quickly after starting the IDE versus waiting a minute for it to complete "indexing" and other stuff seems to make no difference.

But killing the serial-discovery.exe task (which admittedly takes several seconds... I'm not super at switching to Task Manager and finding it in the list) does cure the problem.

[mjs513]

Tried turning off "Arduino: Check For Updates". At least for the problem I'm able to reproduce, made no difference.

Likewise, trying quickly after starting the IDE versus waiting a minute for it to complete "indexing" and other stuff seems to make no difference.

But killing the serial-discovery.exe task (which admittedly takes several seconds... I'm not super at switching to Task Manager and finding it in the list) does cure the problem.

Just want to be sure what sketch are you testing with. The one you posted in post #13?

[PaulStoffregen]

I'm running Defragster's sketch from msg #7.

To reproduce the problem, after uploading and opening the serial monitor (with Teensy Port selected), I quit and restart the IDE. When the IDE restarts, it automatically opens the serial monitor. If the sketch is already running with the LED blinking, nothing is supposed to appear. The sketch only prints stuff when the sketch starts running.

Then with the serial monitor open, I press the pushbutton on Teensy. Teensy Loader uploads the code again.

On the first pushbutton-triggered upload after starting the IDE, when Teensy starts running the sketch again, the serial monitor fails to connect to the freshly reprogrammed Teensy. Its LED stays on solid, indicating it's waiting for the serial monitor to open.

But if I first kill the serial-discovery.exe process, then press the pushbutton, after upload and Teensy begins running Defragster's sketch, the serial monitor does connect. The 2 lines appear (0.33 and 1.33 seconds on my test machine) and the LED blinks, indicating the sketch saw the serial monitor open.

[mjs513]

I'm running Defragster's sketch from msg #7.

To reproduce the problem, after uploading and opening the serial monitor (with Teensy Port selected), I quit and restart the IDE. When the IDE restarts, it automatically opens the serial monitor. If the sketch is already running with the LED blinking, nothing is supposed to appear. The sketch only prints stuff when the sketch starts running.

Then with the serial monitor open, I press the pushbutton on Teensy. Teensy Loader uploads the code again.

On the first pushbutton-triggered upload after starting the IDE, when Teensy starts running the sketch again, the serial monitor fails to connect to the freshly reprogrammed Teensy. Its LED stays on solid, indicating it's waiting for the serial monitor to open.

But if I first kill the serial-discovery.exe process, then press the pushbutton, after upload and Teensy begins running Defragster's sketch, the serial monitor does connect. The 2 lines appear (0.33 and 1.33 seconds on my test machine) and the LED blinks, indicating the sketch saw the serial monitor open.

Followed your steps can confirm it even happens with the micromod and the t36 - had to try.

With that said I did notice that with the MicroMod and the T3.6 if I cycle the IDE's serial model the sketch starts again. In other after:

On the first pushbutton-triggered upload after starting the IDE, when Teensy starts running the sketch again, the serial monitor fails to connect to the freshly reprogrammed Teensy. Its LED stays on solid, indicating it's waiting for the serial monitor to open.

if I close the SerMon and the open it again the sketch starts again

EDIT: Just a note I am on a Windows10x64 PC

[wwatson]

@Paul @Others - I recently bought a new computer with Windows 11 on it. I finally have it all setup with the latest from Arduino 2.0.2 and TD 1.58B2. Using test sketch in post#13 I can confirm that starting Arduino and compiling the sketch when the serial port is set to "COM5 Serial" there is a delay of ~18 seconds before "Begin Program" is displayed. This is without any push button presses. If I press the button several times until it scrolls to the bottom and then recompile and upload the "Begin Program" is displayed without delay but is not auto scrolled into the viewing area. The mouse wheel or scrollbar are needed to display the message. Killing "serial-discovery.exe" does eliminates the issue.

Another issue I noticed is without unplugging the Teensy and restarting Arduino 2.0.2 after shutting it down two things happen:
1 - If shut down with "COM5 Serial" selected then after restart, pressing the button gives no output until the sketch is recompiled and uploaded.

2 - If shut down and restart with plain "COM5" selected the serial monitor displays the button presses without having to recompile and reload the sketch. Killing "serial-discovery.exe" has no effect on this.

Edit: I was never able to reproduce the original problem using the straight "COM5" serial port selection.

[PaulStoffregen]

Followed your steps can confirm it even happens with the micromod and the t36 - had to try.

Does killing the serial-discovery.exe process between starting the IDE and pressing Teensy's pushbutton make any change?

[mjs513]

Does killing the serial-discovery.exe process between starting the IDE and pressing Teensy's pushbutton make any change?

Short answer is yep if you kill the serial-discovery.exe process between button pushes it will start the sketch and print:

D:\Users\Merli\Documents\Arduino\sketch_dec4a\sketch_dec4a.ino Dec  4 2022 14:53:03
run secs A=0.41

D:\Users\Merli\Documents\Arduino\sketch_dec4a\sketch_dec4a.ino Dec  4 2022 14:53:03
run secs B=1.40
See Me: 1

Edit; One thing that just started with my copy of 2.0.2 is that I had to select the board and port again after starting up ide again..

[PaulStoffregen]

... new computer with Windows 11 on it. .... there is a delay of ~18 seconds before "Begin Program" is displayed.

Looks like I might finally have to get a machine capable of running Windows 11.

But first I have a couple (ugly) ideas for getting around the unreliable arrival of WM_DEVICECHANGE messages. Really hoping that can, at least temporarily, save me from Windows 11.

[defragster]

I'm continuing to investigate this problem. Seems like something is causing Windows WM_DEVICECHANGE messages to be delayed or sometimes not arrive at all, which plays havoc with teensy-discovery.exe and teensy-monitor.exe ability to observe arrival and removal of USB devices.

At least for the 1 case I can reliably reproduce on my Windows test machine, using Task Manager to kill the serial-discovery.exe process seems to solve the problem.

...
@Defragster & anyone else seeing this issue, please try killing that process and let me know if it solves all the problems, or only some, or none?
...

Yay! It wasn't just me - Thanks @All >> run secs A=1.03

After Win 11 Update/Reboot - TaskMan UI is uglier than before {All the left edge 'subject area' icons are dark on darker} and no sign of serial-discovery.exe
> was wondering it was renamed as not found after a couple of scroll throughs ...

Installed 'MSFT Sys Internals': now a MSFT Store app. Located: Process Explorer

Has a SEARCH upper right for: "discovery"
KILLED serial-discovery.exe and IDE 2 with "PJRC Teensy Ports Serial Monitor" running "that sketch":

C:\T_Drive\tCode\FORUM\serialIDE2test\serialIDE2test.ino Dec  4 2022 12:09:01
run secs A=1.03

ALSO multiple BUTTON push uploads auto reconnect in good time!

C:\T_Drive\tCode\FORUM\serialIDE2test\serialIDE2test.ino Dec  4 2022 12:09:01
run secs A=1.04

Then Ctrl+U gave A=1.02 secs and then Button Upload gave A=0.94

Good 'discovery' @PaulStoffregen - that is an awesome purpose built sketch isn't it!

These Serial Start times match or beat on average the time the IDE_SerMon starts when 'serial-discovery.exe' is Not Running.

[wwatson]

Looks like I might finally have to get a machine capable of running Windows 11.

But first I have a couple (ugly) ideas for getting around the unreliable arrival of WM_DEVICECHANGE messages. Really hoping that can, at least temporarily, save me from Windows 11.

I hope you can too It's been an adventure getting up to speed with @defagster and the others...

One other issue I noticed with W11 is if with either Arduino 2.0.2 or 1.8.19 I use an external serial terminal program like TeraTerm or Putty and try to upload a sketch it fails with:

"C:\\Users\\wwats\\AppData\\Local\\Arduino15\\packages\\teensy\\tools\\teensy-tools\\1.57.1/teensy_size" "C:\\Users\\wwats\\AppData\\Local\\Temp\\arduino-sketch-DA06FFA9900700EA8F456FFDBEFF2142/SerMonTest.ino.elf"
Memory Usage on Teensy 4.1:
  FLASH: code:11116, data:3992, headers:8440   free for files:8102916
   RAM1: variables:4512, code:8448, padding:24320   free for local variables:487008
   RAM2: variables:12384  free for malloc/new:511904

"C:\Users\wwats\AppData\Local\Arduino15\packages\teensy\tools\teensy-tools\1.57.1/teensy_post_compile" "-file=SerMonTest.ino" "-path=C:\Users\wwats\AppData\Local\Temp\arduino-sketch-DA06FFA9900700EA8F456FFDBEFF2142" "-tools=C:\Users\wwats\AppData\Local\Arduino15\packages\teensy\tools\teensy-tools\1.57.1" "-board=TEENSY41" -reboot "-port=COM5" "-portlabel={serial.port.label}" "-portprotocol={serial.port.protocol}"
Unable to open COM5 for reboot request
  Windows Error Info: Access is denied.
  more ideas... https://forum.pjrc.com/threads/40632?p=126667&viewfull=1#post126667
Teensy did not respond to a USB-based request to enter program mode.
Please press the PROGRAM MODE BUTTON on your Teensy to upload your sketch.

This does not happen in Linux. Maybe this is just peculiar to Windows...

Edit: I do have to press thee button on the Teensy. It then uploads the sketch.

[PaulStoffregen]

One other issue I noticed with W11 is if with either Arduino 2.0.2 or 1.8.19 I use an external serial terminal program like TeraTerm or Putty and try to upload a sketch it fails with:
....
Windows Error Info: Access is denied.

Do you get the "Access is denied" error if you close the serial port in TeraTerm or Putty before attempting to upload with Arduino?

[PaulStoffregen]

KILLED serial-discovery.exe and IDE 2 with "PJRC Teensy Ports Serial Monitor" running "that sketch":

C:\T_Drive\tCode\FORUM\serialIDE2test\serialIDE2test.ino Dec  4 2022 12:09:01
run secs A=1.03

ALSO multiple BUTTON push uploads auto reconnect in good time!

C:\T_Drive\tCode\FORUM\serialIDE2test\serialIDE2test.ino Dec  4 2022 12:09:01
run secs A=1.04

Oh I'm so glad to hear this. Sounds like I might be saved from having to run Windows 11.

These Serial Start times match or beat on average the time the IDE_SerMon starts when 'serial-discovery.exe' is Not Running.

Sadly, the only workaround I can imagine involves polling, which fundamentally has a trade-off of consuming excessive CPU to get faster response. Even fast polling won't be as good as proper WM_DEVICECHANGE events.

I'm going to try an approach where the polling rate runs fast for short times when we have reason to suspect a chance may be coming, then back off to a lower CPU usage and laptop battery friendly rate.

In theory, none of this should be necessary, as WM_DEVICECHANGE is supposed to tell us when to do this stuff. But if Windows can't reliably deliver those messages under some circumstances, then I just don't see any other choice than burning CPU time to poll SetupAPI for changes.

[defragster]

...
One other issue I noticed with W11 is if with either Arduino 2.0.2 or 1.8.19 I use an external serial terminal program like TeraTerm or Putty and try to upload a sketch it fails with:

...
USB-based request to enter program mode.
Please press the PROGRAM MODE BUTTON on your Teensy to upload your sketch.

...

If an external program (as Paul noted) has the ownership of the USB Port - that prevents Upload connect.

Using TyCommander here works perfectly with IDE 2 - like IDE 1.8 - as long as 'Serial' GUI button is clicked to drop the USB connection.

And in TyComm @koromix add a --delegate cmdline param that allows interfacing through TyComm and having it drop Serial and it waits for Teensy to come back allowing Teensy.exe to upload.

And to support that on IDE 2 (at least on Windows) even found a two line local.txt add to work: github.com/Koromix/tytools/issues/103

[wwatson]

Do you get the "Access is denied" error if you close the serial port in TeraTerm or Putty before attempting to upload with Arduino?

No I do not. I also noticed that if I have SerMon open in Arduino and open up TeraTerm I will also get the Access Denied message from TeraTem which makes sense.

[PaulStoffregen]

How do TeraTerm and Putty handle unplugging the USB cable while Teensy is connected? Do they consume excessive CPU and/or get stuck?

[wwatson]

If an external program (as Paul noted) has the ownership of the USB Port - that prevents Upload connect.

Using TyCommander here works perfectly with IDE 2 - like IDE 1.8 - as long as 'Serial' GUI button is clicked to drop the USB connection.

And in TyComm @koromix add a --delegate cmdline param that allows interfacing through TyComm and having it drop Serial and it waits for Teensy to come back allowing Teensy.exe to upload.

And to support that on IDE 2 (at least on Windows) even found a two line local.txt add to work: github.com/Koromix/tytools/issues/103

Ok, got it. Turned off Serial in TyCommander uploaded the sketch without failure. Then enabled serial and "Begin Program" was displayed.

[defragster]

Oh I'm so glad to hear this. Sounds like I might be saved from having to run Windows 11.

...

You and me both!

Looking at bing.com/search?q=WIndows+11+WM_DEVICECHANGE+and+serial-discovery.exe&cvid=364ffa3d79f94551a969f92404d068b e&aqs=edge..69i57.8802j0j1&pglt=299&FORM=ANNTA1&PC =EDGEDSE

Maybe not anything new - but might be worth a couple minutes to scan?
stackoverflow.com/questions/19210421/how-to-obtain-devices-info-from-wm-devicechange << Old but shows mechanism?

That first answer links to: stackoverflow.com/questions/56649088/how-can-i-get-pid-vid-of-a-usb-when-it-is-plugged-in/57182910#57182910

And this is newer from with WIN32 and MFC examples: learn.microsoft.com/en-us/windows-hardware/drivers/kernel/processing-wm_devicechange-messages

That might have some missing MSG or tidbit? Maybe a search refinement. Not seeing any notes that 'it changed' - and if it fails on Win 10 as well it likely isn't, just something missing. Maybe The IDE is getting your messages with how it tries to discover Serial?

ADDED NOTE: Maybe if you resort to added polling that could only start after the "device is removed" given that the UPLOAD in these cases is working perfectly? That would be a 5+ second period of polling during the upload process? Maybe Teensy.exe could notify on the 'restart' for Polling?

[wwatson]

How do TeraTerm and Putty handle unplugging the USB cable while Teensy is connected? Do they consume excessive CPU and/or get stuck?

Neither one. Monitoring CPU usage shows about 0.2% CPU usage normally. Then when disconnecting or connecting CPU usage jumps briefly to 6.7% and drops back to 0.2%. In the case of Putty, the session stops and has to be restarted. TeraTerm just continues on it's merry way. It does not disconnect it seems.

[defragster]

Ok, got it. Turned off Serial in TyCommander uploaded the sketch without failure. Then enabled serial and "Begin Program" was displayed.

That works but is a pain to always remember to coordinate.

Building from the SublimeText editor with CMDLINE is why the --delegate was added to Teensy.exe could then upload to the encrypted 1062's. It allows the IDE or CMDLINE to have the TyComm auto disconnect and reconnect after upload. It is a great addition and takes out the manual coordination and timing race. The .local.txt add for IDE 2 to work is handy.

[wwatson]

That works but is a pain to always remember to coordinate.

Building from the SublimeText editor with CMDLINE is why the --delegate was added to Teensy.exe could then upload to the encrypted 1062's. It allows the IDE or CMDLINE to have the TyComm auto disconnect and reconnect after upload. It is a great addition and takes out the manual coordination and timing race. The .local.txt add for IDE 2 to work is handy.

I have run out of time to play today but my next adventure is learning Sublime Text usage with Tset. I normally use Geany in Linux as an IDE and also have it setup in W11 but do not have Geany IDE setup for compiling and uploading sketches yet. Maybe could use Tset with it

[defragster]

I have run out of time to play today but my next adventure is learning Sublime Text usage with Tset. I normally use Geany in Linux as an IDE and also have it setup in W11 but do not have Geany IDE setup for compiling and uploading sketches yet. Maybe could use Tset with it

Windows only ... See Tset. It makes a batch file that runs from Dos box or explorer or notepadd++ - anything that can run the batch file.
It is due for an update - to note the IDE 2 changes - local.txt option and it includes Arduino15 directories making it use IDE 2 install and removing those can have it use old install process instead.
Though others have worked out a linux script that can do the same. Though not ever seen it published.
>> Will update Use-Sublime-Text-as-an-Arduino-IDE-replacement when TSET is updated and can discuss there.