Teensy Qt

[defragster]

most likely not, as it also works with button pressed.

Koromix may correct,

Thanks for catching WMXZ.

I wrote that before at least once as it seemed that in bootloader (HID) mode it wasn't seen . . .

Maybe I missed another new feature - or just always use USB? Of course it couldn't be an Arduino replacement with no TeensyLoader if it couldn't handle as bootloader.

I just programmed a NEW OSH T_3.2 - which did register - but took two tries as the first time it said : "Board has disappeared"

Just hit the PGM pin low and it is (bootloader) on left pane, with Reset and Upload buttons active, so it can see when not running USB stack.

 

[WMXZ]

I wrote that before at least once as it seemed that in bootloader (HID) mode it wasn't seen . . .

Yes, when already in bootloader mode, the original cli program does not find teensy.
I modified the code so that my cli loader works also with program button previously pressed, so, I guess, also Koromix may have done this mod (I did not check). If not, its easy (only one number to add to the pid list).

 

[Koromix]

The main thing I wanted "tyc" (which was my original goal) is the auto-reboot. It's not very hard to add, you can just look around the Teensyduino core code for the calls to _reboot_Teensyduino_() and look what you need to do to have the called. Wireshark is another option, to spy on the USB traffic. Then the whole thing devolved into a real project, because I wanted to try Qt and I needed an excuse

WMXZ is correct, there is not much more to TyQt. It lists USB devices, and uses the topology information + serial number to make sure it is the same board when it disappears/reappears in a different mode/VID/PID. Serial mode, bootloader mode, the various HID modes should not be a problem.

But this means that boards programmed with Teensyduino < 1.19, that is to say "old new boards" will not behave correctly until programmed because there is a serial number 'mismatch' between running and bootloader mode. The source is the 'serial = MAC * 10' workaround added in Teensyduino 1.19 IIRC. So far, TyQt thinks it's a new board when it reappears in bootloader mode, and the previous one is considered "disappeared".

I had forgotten about that, to be honest (yeah good development practices!). Good time to add a workaround before I forget again: a warning will be shown "reprogram with recent Teensyduino for full functionality" when the problem is detected.

 

[defragster]

Cool - I corrected my 'serial' misguidance in two recent spots.

Funny it is 'OLD' - as it is a fresh OSH T_3.2.

I have three T_3.2 boards active with an ESP8266 on each. Still takes hitting the information page to find the COM port to tell the IDE for serial upload. Of course for OTA upload you need the IP. TYQT is the perfect compliment (monitor disable very useful when doing serial through the Teensy, not OTA upload to ESP)- if only the IDE were as easy to jump between the code for the two . . . Tool switches (especially going to ESP - one can get dropped that bricks OTA - thus needing serial to work) ... and a full recompile on TOOLS change - having to restart IDE when a new OTA IP comes online. With OTA I could use a VM or Visual Micro as a second clean IDE . . . still working it all out ... but TYQT is golden.

 

[defragster]

Confirmation :: TYQT can see/do HID. One Teensy went 'offline' overnight for some reason after leaving the 3 Teensy's running with attached ESP's - the ESP's are still webserving actively with 2.5 refresh of a 3K page doing fine at the same ms count (all over 10.5 hours) so none restarted. The Teensy powers the ESP so I didn't want to unplug. Not sure 'where' the Teensy was - but it was not TYQT visible - and it wasn't LED flashing as it should have been with ESP running.

Of course it is the one I top mounted the onehorse ESP unit last night so I can't touch the button, so I shorted the PGM pin and TYQT indeed then saw as 'bootloader' then I could do a 'soft' reset - and all three "custom named Teensy's" are back online and ESP's running in place! Again TYQT proves to be an awesome tool.

 

[defragster]

Koromix: My TYQT running overnight on my Win10 8GB laptop with the 3 above noted ESP/Teensy units (removed a 4th Teensy with ILI9341 after an hour or so). My machine was over 95% memory and dog slow with 20% cpu usage. I closed TYQT and the ram now at 30%. That was 5+GB of ram held - as far as I can tell - in the name of TYQT. I'm excited to see an update to the reduced lines stored per Teensy to see if it makes a difference. I still have the three IE's pointing to the ESP units (updating a collective 36 times per minute) - it is possible that IE was suddenly prompted to dump RAM - I didn't see a name on the owner of the big ram pool.

<edit update> Just to confirm it wasn't from something else (IE or closing the IDE) I let it run overnight again - I had 7.6 GB in use this AM. Looking at TaskMan and Resource Mon it wasn't clear where the RAM was??? I closed TYQT and used RAM dropped with 2.4 GB in use.

 

[Koromix]

Koromix: My TYQT running overnight on my Win10 8GB laptop with the 3 above noted ESP/Teensy units (removed a 4th Teensy with ILI9341 after an hour or so). My machine was over 95% memory and dog slow with 20% cpu usage. I closed TYQT and the ram now at 30%. That was 5+GB of ram held - as far as I can tell - in the name of TYQT.

Sounds like a leak. I'll look into it. Sorry, I actually missed your last post.

You can find a build of the current code on Dropbox:
- Win32: https://www.dropbox.com/s/yor5p4nxuuhhzyf/TyQt-0.7.0-104-g2628768-win32.zip?dl=0
- OSX: https://www.dropbox.com/s/a59bbq9hosnrqod/TyQt-0.7.0-104-g2628768-osx.dmg?dl=0

The main changes are:
- Extract hardware abstraction layer to a separate library called libhs (main cause of regressions)
- Add per-board scrollBackLimit setting
- Add general TyQt reset/restart actions (Tools menu)
- Move log to a dedicated window
- Improve support for pre-Teensyduino 1.19 boards
- Improve minimal interface / compact mode (board selection menu)
- Keep list of recently uploaded firmwares for each board
- Add 'Drop firmware association' menu action
- Improve board selection dialog

There are many other changes, especially to the 'low-level' device code... There are regressions and spurious warnings which you can safely ignore.

 

[defragster]

Sounds like a leak. I'll look into it. Sorry, I actually missed your last post.

No problem - good to see progress, and good to know I'm not always a bother.

Downloaded and running, will see how it does going forward. I have two active Teensy's with ESP's right now [RAM at 50%/4GB] - I set them to '50000' Scrollback lines, will see how memory does leaving it active rather than shutting it down.

• I see the Scrollback is working as persistent, good.
• Tools / Open Log was hard to find? That makes sense - I was just thinking Help / Verbose like Teensy.exe
• I like the Minimal device drop down - is there any reason to remove it when in full view?
• Adding a bottom option there to toggle Full/Hidden might be handy? "alt" does not re-appear the app menu - so there is no way to see how to get back - OH, Right click gives the menu.
• I like the 'Drop Association' for Auto Upload.

 

[Koromix]

No problem - good to see progress, and good to know I'm not always a bother.

Hey, you're no bother at all. And I owe you most of my users / features! As for the memory use, well if there's a leak I'm not sure how much the scrollback limit will help you, but we'll see.

I like the Minimal device drop down - is there any reason to remove it when in full view?

It is redundant with the board list, but more than that... I find it kind of ugly Actually, it's ok on Windows but on OSX it looks very alien. It is the main reason I kept the code locally for more than two weeks... But I don't have any good idea. On OSX it shown on the right of the tabs instead, to feel less strange.

I need to expose a few global application settings (the main one being the maximum number of parallel uploads/resets/reboots), so I can probably put a "Always show dropdown" setting in there.

Adding a bottom option there to toggle Full/Hidden might be handy? "alt" does not re-appear the app menu - so there is no way to see how to get back - OH, Right click gives the menu.

I liked your suggestion with Alt, and I intend to try that. The shortcut (Ctrl + H) keeps working too. Well, except on OSX which apparently already uses this shortcut so I need to find something else.

 

[defragster]

I pulled my laptop from the Teensy's - so the test was interrupted already. I just closed it and it gave back 400MB. Will see tomorrow if it grows more overnight. I'll search TaskMan and Resource Monitor to see if the memory shows against TYQT before I close it.

Redundant perhaps, but cool looking on Windows (may change if it grew with larger icons getting buttons back). "OSX ... looks very alien": say no more . . . I thought that was the look (and feel) they were going for - ick.

There are so many cool features, easily hidden - easy to forget a hot-key like (Ctrl + H). Having a clickable way out where it would be findable - once you go to Hidden you see and use the drop down naturally

BTW: when Teensy is a 'USB Type / Raw HID' TYQT Monitor works like a charm on multiple Teensy's! I have no need for it - but another poster found it broken so I tested TYQT Apparently this feature has broken in the IDE 1.6.7 - I'm not sure what it used to work like - but gives JAVA errors in 1.6.7 and the devices are not listed under ports - which is either part of the breakage - or means they can only support one?

BUG?!: I've seen this a few times (on released 0.7 and now again on the above -104 build). If I go to Settings page, change the Teeny in Hidden view with Drop Down, then click Monitor - the Monitor page stays oddly blank for what seems like an odd time? Once I found I had clicked 'Monitor Off' and thought maybe I had been doing that somehow - but that is not the case I see now. These steps may not be the critical ones - there may be others - I of course only see it after the fact when I wonder if there should be text to show. I just got it again on 'non Hidden' the second time after:
> On Settings tab
> Click other Teensy
> Click monitor and see it stay blank until the next update comes in?

ALSO: Just stay on 1 (of 2) Teensy's _ Click Settings, then Monitor: Monitor text area is empty until Teensy sends a print and then buffered data draws. Seems like the TAB change needs to push a 'dirty/refresh' event?

My second listed Teensy prints too often to catch perhaps, but my less verbose first Teensy is easy to catch with this.

 

[defragster]

Koromix: My 4GB used was up to 4.7GB - I did a compile and it was lower when I checked TaskMan later . . . value unknown
I just did a RESET on that Teensy and memory was at 4.5GB
I did a RESET on the longer unchanged Teensy and RAM now stands at 4.1GB used.

HINT: TYQT doesn't have to be closed - even a RESET returned a large amount of RAM - this is WITHOUT 'Clear on reset' checked. The above suggests UPLOAD frees RAM as well where it is Stopping Monitor for upload?

Note: In a few hours: I'll do a RAM CHECK, MONITOR STOP, RAM CHECK on one Teensy for effect, then I'll repeat that doing a RESET on the Teensy if it didn't free RAM. Then I'll do an UPLOAD to the other Teensy.

For fun I plugged in a 3rd Teensy with an ESP : current RAM free is 4.1GB

 

[Koromix]

Redundant perhaps, but cool looking on Windows (may change if it grew with larger icons getting buttons back). "OSX ... looks very alien": say no more . . . I thought that was the look (and feel) they were going for - ick.

Well I'm no fan of OSX either. I got a MacBook for free (the HDD was broken, someone abandoned it where my mother works... and it was never reclaimed), so I guess OSX often now. It's shiny looking, but the foundations are crap. Outdated and broken POSIX API, broken HID API, low care for backward compatibility, hard to cross-compile for, outdated OpenGL drivers. The list goes on.

I've seen this a few times (on released 0.7 and now again on the above -104 build). If I go to Settings page, change the Teeny in Hidden view with Drop Down, then click Monitor - the Monitor page stays oddly blank for what seems like an odd time? Once I found I had clicked 'Monitor Off' and thought maybe I had been doing that somehow - but that is not the case I see now. These steps may not be the critical ones - there may be others - I of course only see it after the fact when I wonder if there should be text to show. I just got it again on 'non Hidden' the second time after:
> On Settings tab
> Click other Teensy
> Click monitor and see it stay blank until the next update comes in?

ALSO: Just stay on 1 (of 2) Teensy's _ Click Settings, then Monitor: Monitor text area is empty until Teensy sends a print and then buffered data draws. Seems like the TAB change needs to push a 'dirty/refresh' event?

Just looked, actually what happens is the code that manages the autoscroll goes wrong when you do that, and it scrolls the text widget below that actual text. This code is more of a hack on top of QPlainTextEdit than anything else, so I'm not very surprised, guess it's time to revisit it.

Koromix: My 4GB used was up to 4.7GB - I did a compile and it was lower when I checked TaskMan later . . . value unknown
I just did a RESET on that Teensy and memory was at 4.5GB
I did a RESET on the longer unchanged Teensy and RAM now stands at 4.1GB used.

I'll look into the memory problems tonight or tomorrow.

 

[PaulStoffregen]

Teensy Loader slowly leaks resources on Macintosh also. I'm pretty sure it's a bug in Apple's HID manager.

 

[defragster]

Hi Paul - I'm on windows 10 seeing this having 4GB of RAM go away after some few hours. Just noticed it doesn't take a full TYQT close to return it. But Resource Mgr not showing it under TYQT usage.

Koromix -
The Failure to redraw seems time dependent - either on a new message it redraws or if the last message was 'recent' - coming from settings. Usually works, simple hack would do for now. Ctrl+Mouse wheel resize doesn't redraw (except a 'cursor' line) , but moving scroll bar does. Maybe it thinks the scroll position is below available text or out of bounds?

hopefully this finding will help narrow down your search - I'll do more after a longer run - but about 70 minutes since I rest all 3 Teensy's to get 4.1GB used.
RESET on #1/3: 4.8G goes to 4.7G
RESET on #2/3: 4.7G goes to 4.5G (it has more text coming from ESP)
Resource Mon shows < 250MB allocated to TYQT
Recompile (no upload) in DE does nothing
Dropped associations takes used to 4.4G
Upload to 3rd Teensy drops used to 4.2G (also more ESP driven text)
Quick reset on all 3 leaves 4.2G used
Used tyqt RAM adds up to 256MB
Close tyqt : 4.1G used
Open tyqt returns to 4.2G used.
So RESET and UPLOAD have direct affect - free almost all 'mystery memory'.
Will test stop of 'Monitor' then RESET all again for baseline before leaving for the night.

<edit> Half an hour run time? Turned OFF monitor on all three Teensies, dropped 0.5GB to baseline. Shutting down other things - Used RAM at 3.2G. Also Clearing Monitor does not show a further drop. ResMon shows 86MB used by TYQT with all clear and off.

 

[defragster]

Having reduced Scrollback to 50k lines stopped the RAM loss enough to keep the system usable for longer it seems. Pushed back to 200K lines and now up to 7.1G used and growing. This does not trigger abandoned RAM release.

Either a RESET or Monitor Off will result in that Teensy's share of abandoned RAM to be set truly free. TYQT exit/restart releases all abandoned RAM.

With three active Teensy's a 'Clear Monitor' results in the 'Resource Monitor' dropping the couple hundred MB TYQT RAM to one hundred, but does not release the abandoned RAM. So only the actual serial buffer RAM is credited to TYQT

Teensy #1 had Monitor let off - so it was not abusing any RAM.
7.1GB used : Reset Teensy #2 > 5.3GB used (has not been running VERBOSE Spew as long as #3)
5.3GB used : Monitor Off Teensy #3 > 3.1GB used

 

[defragster]

Didn't touch TYQT 23 hours ? - 50K scrollback - My machine was all but dead to me.

~15% CPU usage and 96% RAM usage : 7.6 of 7.9GB used.

I patiently waited and got some UI - couldn't see Monitor Off click work, hit a button on 1 of 3 Teensy's - still SLOW - and black screens - Ctrl+Alt+Del finally came up with Security settings.

Being impatient - I pulled my USB HUB Plug - taking all three Teensy's offline.

Machine is fine - TYQT dropped them - and RAM was at 2.3GB used when the display came back.

Monitor Off click finally processed, had I waited it probably would have dumped abandoned RAM and given me enough to go, same with button push I assume that 'bootloader' Teensy would have dropped RAM (I'll test that again later after the buffer builds)

 

[WMXZ]

Didn't touch TYQT 23 hours ? - 50K scrollback - My machine was all but dead to me.

~15% CPU usage and 96% RAM usage : 7.6 of 7.9GB used.

I patiently waited and got some UI - couldn't see Monitor Off click work, hit a button on 1 of 3 Teensy's - still SLOW - and black screens - Ctrl+Alt+Del finally came up with Security settings.

Being impatient - I pulled my USB HUB Plug - taking all three Teensy's offline.

Machine is fine - TYQT dropped them - and RAM was at 2.3GB used when the display came back.

Monitor Off click finally processed, had I waited it probably would have dumped abandoned RAM and given me enough to go, same with button push I assume that 'bootloader' Teensy would have dropped RAM (I'll test that again later after the buffer builds)

@defragster: Do I understand correctly that Serial communication of multiple teensy generates this growing RAM usage?
If yes, the following (not teensy related) observation may be relevant:
Some time ago (10 years?), I was acquiring on a PC some RS232 data with 'high' data rate (119.2 kBaud). PC was also busy with other graphic operations. At some point I realized that RS232 data were well over an hour behind real time and that without asking for any special buffer size .
My conclusion was that PC software automatically buffers serial data without limit.
OK, at that time all PC's were single core, so heavy graphic could interfere with handling of serial line and amount of unprocessed serial data could easily increase. Even if modern PCs have multi core CPUs, the operating system may run serial line on a potential busy core so same effect may happen.

OK, the situation here may be different, but sometimes, what seems to be a bug (here memory leakage) could be a side-effect of a design feature and unfortunate use.
The fact that your PC was not responsive with 15% CPU (6 cores?) could indicate that OS, serial port are locking down the same core.

On PC I noted that program allocations to cores are random but can also end up on system core. If then the maximizing memory usage paradigm kicks in with the always active "system and compressed memory task", things may get difficult to understand.

 

[defragster]

I let the Teensy's run a bit more to TYQT and indeed as soon as I pressed the button to bootloader the System freed the RAM as well.

@WMXZ: Indeed there are Teensy's on USB sending Serial data and it is depleting RAM. So far I found 6 ways to get this mystery ram to be set free on my 4 core i5 w/Win10: Restart TYQT, Unplug Teensy, TYQT Monitor OFF, Teensy Program w/IDE, Teensy Button, Teensy Reset. All of these will result in dropping the USB port so your observation that the OS is keeping the data may be relevant, maybe it isn't TYQT memory management's discarded overflow lines. Using Resource Monitor there is no item listed showing holding what could be up to 6+GB that I have seen. Note a few other things active - though right now it is at 3% with TYQT closed - fresh open of TYQT doing 2-14% (typ <5% spread over all 4 cores - but mostly all kernel time on peaks - closing TYQT doesn't change that - neither does unplugging the hub with 3 Teensy's on) with a view of the monitor open. I mentioned CPU seeing it as LOW for my use - and showing it wasn't blocking on CPU, but system RAM handling I assumed - didn't note if the disk was shown active paging. When I was purposefully writing USB as fast as I could and Koromix made a thread to read the Serial I wasn't seeing this with prolonged 1MB/sec transfers - but I was probably flashing new code regularly? Note: as bad as this is - the 1MB/sec would trash the SerMon in a couple iterations - and recently it doesn't work well at all - especially when watching more than 1 Teensy with that not being supported.

Koromix: Hopefully this gives you an idea for a resolution or at least a way to effect the release - but the obvious idea I have of periodically dropping USB could result in loss of some data at that point? I'm not sending huge data for any particular reason - just text notes as I blink a LED on the ESP - one of which I don't have the RGB wired too.

 

[defragster]

Just looking at this RAM - I tapped the MONITOR OFF and then MONITOR ON with short and long delays. The data streams in as soon as the buffer ON is clicked.

As far as I can tell the system buffers the USB Data across the OFF, at the same time it frees the abandoned RAM.

Koromix - a quick test might be to do a quick 'AUTO MONITOR OFF/ON Toggle' in the code every 'x' times the scrollback buffer is overflowed - or every 'x' MB of data received. For a test case do it after every line is received perhaps and confirm no data is lost? If this loses no data - then the only problem would be not doing it often enough and having the system somewhere maintaining unused buffers to the detriment of the system. If you email me a private build link I will test it.

 

[Koromix]

Teensy Loader slowly leaks resources on Macintosh also. I'm pretty sure it's a bug in Apple's HID manager.

The HID manager API is quite fragile, and it is easy to crash it if you look at it the wrong way. When I was porting ty to OSX I hit quite a few weird failures myself. And let's not talk about the fact that it is the only platform where HID devices cannot be accessed as device files.

hopefully this finding will help narrow down your search - I'll do more after a longer run - but about 70 minutes since I rest all 3 Teensy's to get 4.1GB used.
RESET on #1/3: 4.8G goes to 4.7G
RESET on #2/3: 4.7G goes to 4.5G (it has more text coming from ESP)
Resource Mon shows < 250MB allocated to TYQT
Recompile (no upload) in DE does nothing
Dropped associations takes used to 4.4G
Upload to 3rd Teensy drops used to 4.2G (also more ESP driven text)
Quick reset on all 3 leaves 4.2G used
Used tyqt RAM adds up to 256MB
Close tyqt : 4.1G used
Open tyqt returns to 4.2G used.
So RESET and UPLOAD have direct affect - free almost all 'mystery memory'.
Will test stop of 'Monitor' then RESET all again for baseline before leaving for the night.

<edit> Half an hour run time? Turned OFF monitor on all three Teensies, dropped 0.5GB to baseline. Shutting down other things - Used RAM at 3.2G. Also Clearing Monitor does not show a further drop. ResMon shows 86MB used by TYQT with all clear and off.

Thanks for all the information.

I've spent a few hours on this tonight and began with the "blank-monitor-on-tab-change" issue. It is a Qt bug and I implemented a way around it, but in doing so changed quite a bit how the monitor autoscroll works (and other things). I'll report it in the coming week, if nobody has reported it before.

While working this problem, I triggered many annoying heap corruption errors with "Clear Monitor". The problem comes from Qt (QTextDocumentPrivate::clear() is the culprit) and it may be another Qt bug. I'm less sure about this one though, it could be me abusing QTextDocument/QTextCursor; at the very least the documentation is lacking on what you should and should not do with it.

But I think the previous code (which you run) may have been triggering a similar bug, except instead of corruption it could have been "leaking" QTextCursor objects. I'm not sure and this part of Qt is not the most beautiful/clear code I've seen. This would be consistent with reset clearing the used memory up.

I've been dumping 50000 lines/s (with a scrollback of 200000 lines) from my LC for 30 minutes now and the memory consumption of TyQt is not changing at all. So either:
- It's Win32-specific and I'll look into that tomorrow
- Or it is fixed by the changes I made tonight

I'll send you a build in a few minutes, thank you for your help.

 

[defragster]

I got a private build with a fix Koromix he understood as needed - It worked - ran all day and RAM was in good shape. Closed TYQT and it returned ~100MB - normal app and buffer space.

 

[defragster]

Left the fixed TYQT running these past days on my Win 10 laptop - three running Teensy's - stable system - no RAM loss - about 2 dozen active tasks on my taskbar. Was sitting at 5.3 GB Used - closed TYQT is briefly dropped to 5.2 GB used - now back to 5.3 GB so RAM used was a rounding error with only 50,000 lines buffered for each Teensy - I may as well bump back up to 200,000 default lines - only takes 52MB for 3 sets of 200K lines.

 

[defragster]

Koromix: I'm seeing some jitter and oddity on line tracking on an open TYQT monitor window - on Windows when incoming lines are overfilling scroll back.

By Jitter I mean it doesn't sit stable if I scroll up and when I go down it doesn't sit at the bottom with normal keys - either a page up or page down. It seems to be when it is purging the top of the Scroll buffer with new data coming in? Some fight over what 'current' line/page/% is perhaps? And when you go to the top being scrolled away - it just dumps you to the bottom.

Question - why would you show a serial # of:: The number 4,294,967,295 is an integer equal to 2³² − 1. This is a custom Teensy somebody else made.

 

[Koromix]

Thanks for confirming that the leak is fixed.

The monitor bug is a regression I introduced when I fixed the "blank-on-tab-change" bug you reported earlier. Hopefully someday this thing will work fully as intended. I've pushed a fix to GitHub and the latest build (-117-) is available here: https://bintray.com/koromix/ty/ty/view#files

I have a working "private" branch with all the changes required to build TyQt on Visual Studio 2015. Seems to work well, so it is probably the last MinGW-compiled version. This will much improve my debugging experience on Windows (which is the main point of this change).

This build also fixes a couple of OSX bugs.

 

[defragster]

Running the new copy now - will see if the monitor cleans up, Thanks. Interesting to see VS 2015 build working. I hoped to be there but not yet.