Arduino/Teensy IDE Crashes Constantly - Java heap errors

T

Thundercat

Well-known member
Hi, the Teensyduino/Arduino IDE keeps crashing when I open files, or after I've opened files. I get the following errors:

Exception in thread "Timer-0" java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:3236)
at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)
at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
at org.apache.commons.compress.utils.IOUtils.copy(IOUtils.java:69)
at org.apache.commons.compress.utils.IOUtils.copy(IOUtils.java:49)
at org.apache.commons.compress.utils.IOUtils.toByteArray(IOUtils.java:167)
at cc.arduino.contributions.GPGDetachedSignatureVerifier.verify(GPGDetachedSignatureVerifier.java:77)
at cc.arduino.contributions.SignatureVerifier.isSigned(SignatureVerifier.java:55)
at cc.arduino.contributions.DownloadableContributionsDownloader.checkSignature(DownloadableContributionsDownloader.java:211)
at cc.arduino.contributions.libraries.LibraryInstaller.updateIndex(LibraryInstaller.java:89)
at cc.arduino.contributions.ContributionsSelfCheck.updateLibrariesIndex(ContributionsSelfCheck.java:205)
at cc.arduino.contributions.ContributionsSelfCheck.run(ContributionsSelfCheck.java:75)
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)

Does anyone know how to troubleshoot this or fix it? It's driving me mad and seriously impacting my efficiency.

Thanks,

Mike
 
Not that I may be of any specific help with this issue but what Arduino/Teensyduino versions are you talking about?
And what is your host platform: Windows, Mac or Linux?

Paul
 
Arduino 1.8.19; Teensyduino 1.58

MacOS 11.7.9

Thanks Paul

The sketch is very large - over 30,000 lines of code, so that may have something to do with it.

I can't seem to solve it though, and my productivity is in the toilet.
 
If you have not already done so, you might want to also post on the Arduino forums: https://forum.arduino.cc/latest

As hopefully one of the Arduino developers might have some better ideas on how to fix.

I am not a Java person, so not sure what the: Java Heap space error might be.

Again just throwing a dart here, but have made sure that where your temp directory is, that there is enough free disk space?
 
Thank you Kurt, I appreciate your time. I didn't know anything about a temp directory or making space available on it; there's 10% free space on the drive I'm using.

I will post the question on the Arduino forum to see if there's any fix.

Of course I've already googled this and I've just gotten some oddball comments here and there, and one Arduino dev saying they have no control over this aspect of the program. But lately it crashes the IDE so hard I have to restart it, sometimes only a few minutes after opening it; sometimes immediately. So it's become a real nuisance.

Thank you again, and I'll post back if I get any joy.

All the best,

Mike
 
Can you give a link to the Arduino forum thread? And if you haven't already given a link to this thread over there, please do so. When we get this figured out, hopefully those links will help other people who hit this same error. Nothing is more discouraging than having a tough problem and Google finds forum threads where someone had the exact same problem but there was never any solution. So please, let's plan for success and give people who later find these threads the ability to find the answer if it gets posted at the other site.

About this problem, just a couple questions first...

Does the OutOfMemoryError exception happen if you just open Arduino IDE and and your sketch, but before you actually do anything (like Verify, Upload, Serial Monitor...)? Or if you've just opened the IDE and let it sit for a few minutes without actually doing anything?

In the Tools > Ports menu, you'll probably see Teensy in 2 places. One is "Serial port" and other other is "Teensy port". You should see the same results in the serial monitor, but the way everything is implemented completely changes. The Serial port way uses a Java library to access the USB hardware and Arduino's original serial monitor implementation. The Teensy port option uses a native helper program (no Java library) to access hardware and the data storage part of the serial monitor is completely rewritten to handle the incredible speed Teensy 4.x can transmit data. If the problem happens only after having used the serial monitor, which of these options you used and how much data and how rapidly it transmits could matter. Especially the original Serial port option could consume a lot of memory and run very slowly, completely crashing the entire Arduino IDE... which is the reason I want to so much work to add the Teensy port way.

However, this OutOfMemoryError exception has stuff like "LibraryInstaller.updateIndex" and "contributions.DownloadableContributions" in the stack trace. So that makes me wonder if Arduino's official list of libraries has grown so long, and the older 1.8.19 IDE does things in such inefficient ways, that just processing the library list is now starting to cause problems on the old IDE version? That's why I asked if it gives this exception after you've just started the software but before you actually do anything.

And one final question. Is MacOS 11.7.9 "Big Sur"? I get them mixed up sometimes. I have 5 test machines here with Lion, High Sierra, Mojave, Monterey and Ventura... but no Big Sur. Maybe Monterey is closest?
 
Paul many thanks for your time and attention; you are a treasure.

This issue happens when I open a sketch and haven't actually anything. It can happen within a few seconds, or it can happen after 5 minutes of it being open. It can also happen when I open two sketches at once - that's been the main trigger in the past. If I've opened two large sketches, then it's almost a guarantee it will happen. It also seems to happen when I most need to get things done ;-)

Here is a link to the Arduino forum post: https://forum.arduino.cc/t/java-error-keeps-crashing-arduino-ide/1157150

Regarding your Tools-->Ports menu query, the Teensy Port option is grayed out for some reason. Attached is a screenshot. I'm not sure what I'm supposed to do here, or why it's grayed out.

Tools-->Ports menu.jpg

MacOS 11.7.9 is Big Sur. I'm on a 10 year old MacBook Pro that is still going strong, and that's the latest OS I can officially use.

Since the original post, I installed the latest Arduino IDE, v2.11. However within the Teensyduino environment it still lists the Arduino IDE as 1.8.19 for some reason. I have only used it since the update for a few minutes but have not had the crash yet.

Thanks for your time and hopefully a solution can be found.

Many thanks!

Mike
 
Many years ago, I had a Java crash whenever I opened a fairly large project and tracked it down to an unterminated comment. Maybe that's now been fixed but keep it in mind if all else fails ;)

Pete
 
Thank you Pete.

What do you mean by an unterminated comment?

I always use // for comments and occasionally /* and */

I don’t think it would compile if a comment was left open, would it?
 
"Teensy Ports" gray in the menu is normal. It means all the non-gray items in that part (above the divider line) are the "Teensy Ports" type. They get accessed using the faster native tools. Likewise the gray "Serial Ports", all the ones in that section are the normal type that get accessed with the Java library.

From your description, I'm feeling like my initial guess about the library list growing too long might be the case here. You're editing large code which consumes more of the Java memory pool. But that list keeps growing and the old 1.8.19 IDE isn't getting any updates, so this problem is likely to keep getting worse. Eventually I would imagine many more people who edit normal size sketches will probably also start encountering this issue.

As for what to actually do about it, redesigning the old Java-based library index download in 1.8.19 is pretty much out of the question. So the other option is to try increasing the Java memory pool. To try this, first quit Arduino IDE, since you're about to edit stuff that controls how it runs. Then make a backup copy. Control-click Teensyduino and choose "Compress Teensyduino" from the pop-up menu. This will make a zip copy which you can use to get back to where you are now (admittedly not too useful). Then Control-click and choose "Show Package Contents". You should get a new window with "Contents". Look in that folder and you should see this:

screen.jpg
(click for full size)

You need to replace the Info.plist file with one that gives Java permission to use more memory. I'll attach a copy to this message.

Please be aware changing anything inside the application bundle, even just this tiny file, will run afoul of MacOS application signing. Hopefully MacOS won't notice you've changed stuff, at least for a while so you can find out whether this actually solves the problem. But know that MacOS will discover the application's digital signature is no longer valid if you move or copy the application anywhere else on your system. The main idea right now it just to try this and figure out if it solves the problem, or diving deeper is needed?
 

Attachments

  • Info_plist.zip
    1 KB · Views: 15
el_supremo said:
Maybe that's now been fixed but keep it in mind if all else fails ;)
Click to expand...

Yeah, that was fixed a long time ago. Ancient versions of Arduino IDE uses regex in Java to parse for comments and includes and other stuff.

Eventually they switched to ctags. I believe that was somewhere in the 1.5.x betas. Later (as I recall somewhere in the 1.6.x era) running the gcc preprocessor was added, which allowed for correct parsing of #ifdef conditional compilation and discovering library dependency within other libraries.

The old regex way could fail very badly. Like so many things in Java, the bad fail cases could consume a mind boggling amount of memory for something which really should have been pretty simple.
 
Thundercat said:
Paul many thanks for your time and attention; you are a treasure.
Since the original post, I installed the latest Arduino IDE, v2.11. However within the Teensyduino environment it still lists the Arduino IDE as 1.8.19 for some reason. I have only used it since the update for a few minutes but have not had the crash yet.
Click to expand...

It doesn't sound like you're actually using Arduino IDE 2.1.1. The version shows 2.1.1 for me. Remember, there's two ways to run Teensy things with Arduino. The first is to download that "Teensyduino" release. That's a special build of Arduino IDE 1.8.19. The second is to add the Teensyduino software from within Arduino IDE 2.1.1.

The link is: https://www.pjrc.com/teensy/td_download.html
First way: Download "Macintosh Complete Software" from the "Arduino 1.8.x Software Development" section.
Second way: Download the stock Arduino IDE 2.1.1, add follow the instructions in the "Arduino 2.0.x Software Development" section.

Note that both can coexist. We still have to use the 1.8.19 version for security things and lockable Teensys.
 
PaulStoffregen said:
"Teensy Ports" gray in the menu is normal. It means all the non-gray items in that part (above the divider line) are the "Teensy Ports" type. They get accessed using the faster native tools. Likewise the gray "Serial Ports", all the ones in that section are the normal type that get accessed with the Java library.

From your description, I'm feeling like my initial guess about the library list growing too long might be the case here. You're editing large code which consumes more of the Java memory pool. But that list keeps growing and the old 1.8.19 IDE isn't getting any updates, so this problem is likely to keep getting worse. Eventually I would imagine many more people who edit normal size sketches will probably also start encountering this issue.

As for what to actually do about it, redesigning the old Java-based library index download in 1.8.19 is pretty much out of the question. So the other option is to try increasing the Java memory pool. To try this, first quit Arduino IDE, since you're about to edit stuff that controls how it runs. Then make a backup copy. Control-click Teensyduino and choose "Compress Teensyduino" from the pop-up menu. This will make a zip copy which you can use to get back to where you are now (admittedly not too useful). Then Control-click and choose "Show Package Contents". You should get a new window with "Contents". Look in that folder and you should see this:

View attachment 31736
(click for full size)

You need to replace the Info.plist file with one that gives Java permission to use more memory. I'll attach a copy to this message.

Please be aware changing anything inside the application bundle, even just this tiny file, will run afoul of MacOS application signing. Hopefully MacOS won't notice you've changed stuff, at least for a while so you can find out whether this actually solves the problem. But know that MacOS will discover the application's digital signature is no longer valid if you move or copy the application anywhere else on your system. The main idea right now it just to try this and figure out if it solves the problem, or diving deeper is needed?
Click to expand...

Paul, many kind thanks for the extremely thorough information and potential workaround. I'll follow up with this and report back what happens. Since apparently the 2.11 Arduino update didn't take (it seems I didn't install it correctly), this will be a good test.

EDIT: I have followed the steps and updated the plist. So far working correctly, but I'll let a couple days of working with it tell the tale.

Many Thanks,

Mike
 
shawn said:
It doesn't sound like you're actually using Arduino IDE 2.1.1. The version shows 2.1.1 for me. Remember, there's two ways to run Teensy things with Arduino. The first is to download that "Teensyduino" release. That's a special build of Arduino IDE 1.8.19. The second is to add the Teensyduino software from within Arduino IDE 2.1.1.

The link is: https://www.pjrc.com/teensy/td_download.html
First way: Download "Macintosh Complete Software" from the "Arduino 1.8.x Software Development" section.
Second way: Download the stock Arduino IDE 2.1.1, add follow the instructions in the "Arduino 2.0.x Software Development" section.

Note that both can coexist. We still have to use the 1.8.19 version for security things and lockable Teensys.
Click to expand...

Shawn, that's extremely helpful. Thank you so much.

EDIT: I am using the Lockable Teensys, so you gave me a great tip to stay on 1.8.19 for now.

I wonder if the Teensyduino will support 2.x with all features at some point?

Thanks again,

Mike
 
Last edited:
I use both. 1.8.19 when I need to do lockable Teensy stuff (just those external programs) and 2.1.1 for everything else. (Well, my main IDE is actually VSCode plus PlatformIO. I use 2.1.1 to verify things sometimes.) I think 2.1.1 is an improvement on 1.8.19, so I recommend that for your main development work, if you want to use the Arduino IDE at all. There’s really no need to stick with 1.8.19 except for just those external security programs.

I think the reason 2.1.1 doesn’t support Teensy secure things is because the IDE isn’t yet designed to run external programs. It’s a limitation I’m sure will be addressed in a future release. In fact, if I’m not mistaken, there’s a GitHub issue filed for that.
 
shawn said:
I use both. 1.8.19 when I need to do lockable Teensy stuff (just those external programs) and 2.1.1 for everything else. (Well, my main IDE is actually VSCode plus PlatformIO. I use 2.1.1 to verify things sometimes.) I think 2.1.1 is an improvement on 1.8.19, so I recommend that for your main development work, if you want to use the Arduino IDE at all. There’s really no need to stick with 1.8.19 except for just those external security programs.

I think the reason 2.1.1 doesn’t support Teensy secure things is because the IDE isn’t yet designed to run external programs. It’s a limitation I’m sure will be addressed in a future release. In fact, if I’m not mistaken, there’s a GitHub issue filed for that.
Click to expand...
Hi Shawn, that’s excellent info and very useful for me.

Thank you so much 🙏😊
 
As mentioned, I believe once you have your boards locked to your key, you can use IDE 2 to program your secured teensy
 
KurtE said:
As mentioned, I believe once you have your boards locked to your key, you can use IDE 2 to program your secured teensy
Click to expand...

Thank you Kurt. That workflow is a bit awkward, because it means I need to use one IDE for locking the Teensy's and another to program. I'll probably just stick with 1.8.19 for now if Paul's workaround is solid, and so far I haven't had a crash since implementing it (thank you Paul!).
 
I think the your workflow could be improved. It feels like you’re implying you need to use the security program upload frequently, which tells me you’re probably locking and programming lots of Teensys.

Are you aware that you don’t need the Arduino IDE to upload HEX or EHEX files? If you’re trying to lock and then program a bunch of Teensys, that means there’s a HEX file that locks a unit and an EHEX file that contains an encrypted form of your program.

Here’s what I’d do:
1. Generate the locking program by building it (checkmark button) without uploading it. In the Arduino output, one of the lines should contain where the locking HEX program file resides. You might need to enable verbose compile output in the Preferences.
2. Copy that file somewhere, say the desktop.
3. Generate your main program’s EHEX file and use the same procedure to find its location. I.e. just build and don’t upload.
4. Open the Teensy Loader program. It should already be open if you’ve built any program.
5. Drag the locking HEX file into Teensy Loader.
6. Ensure the “Auto” mode button is enabled and connect an unlocked lockable Teensy.
7. Press the Teensy’s program button to load the program (eg. The HEX file.)
8. Repeat steps 5-7 for your main program’s EHEX file.

Caveats and notes:
1. Because locking is permanent, you might also want to generate and use those verify and test programs. At the very least, you should watch the locking program output.
2. Because locking is permanent (I’m repeating that like for dramatic effect :)), I suggest having some serial monitor open to watch the security-related program(s) run to make sure everything is OK. For example, you can use the one in the Arduino IDE. (I personally use a program called “Serial” by Decisive Tactics. I think there’s also lots of free ones out there.)
3. Choose which of the three security programs you want to use (in addition to the one that does the actual locking; there’s two others from that Security dialog window, one of which does verification). It’s not required; use the two other programs subject to your comfort level.

There’s also a command-line version of the Teensy Loader, I believe. But strangely, I’m not sure if that version can upload EHEX files. (See: https://forum.pjrc.com/threads/73139-Teensy-Loader-command-line-version-EHEX-support)
 
shawn said:
I think the your workflow could be improved. It feels like you’re implying you need to use the security program upload frequently, which tells me you’re probably locking and programming lots of Teensys.

Are you aware that you don’t need the Arduino IDE to upload HEX or EHEX files? If you’re trying to lock and then program a bunch of Teensys, that means there’s a HEX file that locks a unit and an EHEX file that contains an encrypted form of your program.

Here’s what I’d do:
1. Generate the locking program by building it (checkmark button) without uploading it. In the Arduino output, one of the lines should contain where the locking HEX program file resides. You might need to enable verbose compile output in the Preferences.
2. Copy that file somewhere, say the desktop.
3. Generate your main program’s EHEX file and use the same procedure to find its location. I.e. just build and don’t upload.
4. Open the Teensy Loader program. It should already be open if you’ve built any program.
5. Drag the locking HEX file into Teensy Loader.
6. Ensure the “Auto” mode button is enabled and connect an unlocked lockable Teensy.
7. Press the Teensy’s program button to load the program (eg. The HEX file.)
8. Repeat steps 5-7 for your main program’s EHEX file.

Caveats and notes:
1. Because locking is permanent, you might also want to generate and use those verify and test programs. At the very least, you should watch the locking program output.
2. Because locking is permanent (I’m repeating that like for dramatic effect :)), I suggest having some serial monitor open to watch the security-related program(s) run to make sure everything is OK. For example, you can use the one in the Arduino IDE. (I personally use a program called “Serial” by Decisive Tactics. I think there’s also lots of free ones out there.)
3. Choose which of the three security programs you want to use (in addition to the one that does the actual locking; there’s two others from that Security dialog window, one of which does verification). It’s not required; use the two other programs subject to your comfort level.

There’s also a command-line version of the Teensy Loader, I believe. But strangely, I’m not sure if that version can upload EHEX files. (See: https://forum.pjrc.com/threads/73139-Teensy-Loader-command-line-version-EHEX-support)
Click to expand...

Hi Shawn, thanks for the high level review and suggestions for better workflow. I'm paranoid about it, and I watch the serial monitor outlook like a hawk. Sometimes it generates half of the text on output, so I run the 3 security programs again just to make sure.

I'll keep your ideas in mind for improved efficiencies going forward. Many thanks for the wealth of knowledge you shared!

How do you like PlatformIO?

All the best,

Mike
 
I support you in your endeavours to be extra careful with lockable Teensys. :)
It's the way to go.
 
The main place to be extra careful with lockable Teensy is making sure you have backups of your key.pem file.

Yes, today's situation requiring either the old IDE or command line usage isn't ideal. Arduino IDE 2.1.1 simply doesn't support plugins like the old versions did. Arduino developers are (slowly) working on this. So far there's no idea of a time frame, but I would be surprised if it happens in 2023.
 
Thundercat said:
I'll probably just stick with 1.8.19 for now if Paul's workaround is solid, and so far I haven't had a crash since implementing it (thank you Paul!).
Click to expand...

So the fix works? Or is it still too early to be sure?

If yes, might be worthwhile to also follow up on the Arduino forum. As the library list keeps slowly growing, I would imagine more people still using Arduino IDE 1.8.19 on MacOS will start to encounter this problem. Who knows, might become a problem on Linux and Windows too?

I'm going to try incorporating this fix into the copy of Teensyduino using IDE 1.8.19 for MacOS. Hope you'll be willing to give the next beta a test run? For fixing this sort of problem where I can't reproduce with my normal Mac test machine, I'm really depending on your help with testing...
 
Windows 10 Arduino 1.8.19 TD 1.57 java.lang.OutOfMemoryError: Java heap space

Thundercat said:
Hi, the Teensyduino/Arduino IDE keeps crashing when I open files, or after I've opened files. I get the following errors:

Exception in thread "Timer-0" java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:3236)
at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)
at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
at org.apache.commons.compress.utils.IOUtils.copy(IOUtils.java:69)
at org.apache.commons.compress.utils.IOUtils.copy(IOUtils.java:49)
at org.apache.commons.compress.utils.IOUtils.toByteArray(IOUtils.java:167)
at cc.arduino.contributions.GPGDetachedSignatureVerifier.verify(GPGDetachedSignatureVerifier.java:77)
at cc.arduino.contributions.SignatureVerifier.isSigned(SignatureVerifier.java:55)
at cc.arduino.contributions.DownloadableContributionsDownloader.checkSignature(DownloadableContributionsDownloader.java:211)
at cc.arduino.contributions.libraries.LibraryInstaller.updateIndex(LibraryInstaller.java:89)
at cc.arduino.contributions.ContributionsSelfCheck.updateLibrariesIndex(ContributionsSelfCheck.java:205)
at cc.arduino.contributions.ContributionsSelfCheck.run(ContributionsSelfCheck.java:75)
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)

Does anyone know how to troubleshoot this or fix it? It's driving me mad and seriously impacting my efficiency.

Thanks,

Mike
Click to expand...






java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:3332)
at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448)
at java.lang.StringBuffer.append(StringBuffer.java:270)
at processing.app.legacy.PApplet.join(PApplet.java:167)
at processing.app.BaseNoGui.loadFile(BaseNoGui.java:658)
at processing.app.SketchFile.load(SketchFile.java:256)
at processing.app.EditorTab.<init>(EditorTab.java:98)
at processing.app.Editor.addTab(Editor.java:1563)
at processing.app.Editor.createTabs(Editor.java:1536)
at processing.app.Editor.handleOpenInternal(Editor.java:1849)
at processing.app.Editor.<init>(Editor.java:378)
at processing.app.Base.handleOpen(Base.java:882)
at processing.app.Base.restoreSketches(Base.java:552)
at processing.app.Base.<init>(Base.java:487)
at processing.app.Base.main(Base.java:150)



Boy, I'm glad I hadn't witnessed this problem until now... Thanks everyone for your efforts. (Should I get a Mac?)

(No Teensy security involved here.)

I am trying to compile more wavetables in a simple synth to see if I can tune them up, properly.
(Some wavetables are tuned OK but many others are quite a bit out of tune.
I know that the wave table parameters need to be in RAM and have done this before.)
(Up to recently I've use close to 128 wave tables but with so much room in Flash, thought many more should not be a problem.
Flash usage was getting over half full with an extensive synth.)
The strange thing is that the sketch that the IDE starts loading is one that has never given me a problem before.
The IDE goes away in a couple seconds, so I have no chance to try different builds.

I looked to see if I can re install Java but Windows 10 does not list it as a program that I can uninstall.

(I tried an older Arduino IDE and it disappears in a couple seconds also.)

Processing 3 appears to work fine.

Weeks ago, I tried TD 1.58 and but it caused excessive RAM use.
(I DO wish I had an option to have libraries execute in FLASH, but that's not the issue now.)

I had tried Arduino 2 a few months ago, but when it messed up my Arduino 1 library I got quite upset.
(After repairing the library, I stayed with Arduino 1.)

Suggestions for fixing Java for Windows 10 ?
Is there a way to find out Java's limits?
 
You must log in or register to reply here.
Back
Top