My working sketch is broken after Arduino 2.0 and/or Teensy 1.57 upgrade

mykle

Well-known member
Hi all, i'm having a weird issue & could use some advice:

I thought it would be a harmless experiment to try installing arduino 2.0, but the firmware it compiled malfunctions on my board -- and furthermore I can't figure out how to back out of the upgrade!

Everything was working fine with Arduino 1.8.16 + Teensy 1.56, running on a Teensy 3.2 in Serial+Midi mode, all on a M1 Mac.

I followed Paul's instructions on Teensy support in Arduino 2.0. In that IDE I installed Teensy 1.57 because I figured it's probably stable by now. (There was also an option for 1.56, but I wasn't sure if that's compatible with Arduino 2.0.) Then I compiled my latest known-good version of the firmware ...

The firmware is not crashing, but not working either. USBSerial output works so I can see the main loop running in the Serial Monitor. And it's reading at least one digital pin correctly. But it's not reading its buttons or polling the IMU over SPI, so I can't control it.

Also, as I experimented with compiling earlier firmware versions I sometimes found it mysteriously lighting one LED that shouldn't be lit. So maybe Arduino has scrambled some, but not all, of its pins? Bizarre!

So then I said "get me outta here!", quit Arduino 2, fired up 1.8.16 and recompiled ... but the problems didn't go away. Urgh ... Then I notice in the (voluminous, barely legible) IDE log section that it's still compiling with Teensy 1.57? So Arduino 2.0 has overwritten at least that much of my previously-working 1.8.16 configuration.

I went to PJRC to re-locate the 1.56 download link ... but it seems to no longer be there. =/

So I re-launched Arduino 2 and used its board manager to install Teensy 1.56, then went back to Arduino 1.8.16 again and recompiled. The logs now say that files from Teensy 1.56 are being used, good ... but still, the resulting firmware is broken in the same weird way.

Probably there is some other library component or other internal file that's messed up in my Arduino directory. Maybe I need to restore from backups? But it's weird, this problem.

Have any of you seen these kinds of issues when upgrading to Arduino 2 or Teensy 1.57?
Does Arduino keep any kind of log of what it's upgraded?

Much thanks,
-mykle-
 
I believe if you uninstall Teensy board in the board manager for 2.0, then Teensy should revert back to building with the 1.8.16 stuff. There is some stuff Paul put into later Teensyduino stuff to
use the 1.8.x version code when you try to build with it. By default Arduino will build using the stuff installed in the Arduino15 directory...

Hope that makes sense.
 
Thanks for the advice. I tried it. 2.0 let me uninstall, and then 1.8.16 was still able to build ... but the bug remains.

For the record, I also tried restoring the Arduino15 directory from yesterday, and things are still screwed. So maybe it's something else?
(I wish I had done a test build before I started down this path.)

I wonder, does Arduino use compilers/tools found elsewhere on my system? Some of those versions might have shifted since I last built this sketch.
But the logs say Arduino is using its own copy of gcc, g++ and ar ...
 
So Arduino 2.0 has overwritten at least that much of my previously-working 1.8.16 configuration.

That's not what happened, but it's easy to misinterpret this way.

Arduino 1.8.x has complicated rules for where it will find files for boards and libraries. One of the places Arduino 1.8.x searches happens to be the same place Arduino 2.0.0 will install the Teensy files. Arduino 2.0.0 didn't overwrite the old files. But it did put new files in a place where Arduino 1.8.x notices them.

Teensyduino 1.57 adds a patch to Arduino 1.8.19 so it won't look in that location when using Teensy. This was done because we discovered this problem during the Arduino IDE2 beta testing. But Teensyduino 1.56 doesn't have the patch to prevent interference from Arduino 2.0.0's installs, because 1.56 was created before we were doing much testing of Teensy on IDE2 betas.

The long-term solution to your situation (other than finding and fixing whatever is wrong in the code when using 1.57) is to use Arduino Portable mode, assuming you're using Windows or Linux. Sadly, portable mode doesn't work on modern MacOS. Details here:

https://docs.arduino.cc/software/ide-v1/tutorials/PortableIDE

First extract a fresh copy of Arduino IDE 1.8.19. If using Windows, you may need the ZIP file rather than EXE installer.

Then create the "portable" folder inside it. This copy of Arduino 1.8.19 will become completely self-contained.

Download Teensyduino 1.56. It's still on the server, but without links to it. Just copy the link to the 1.57 file (on most browsers right-click to do this) and when you paste the link in the browser's address bar, look for the 3 digits "157". Edit that to "156" and press Enter or click Go, to download the 1.56 installer.

When you run the 1.56 installer, be careful to select the location of your portable mode Arduino IDE. The installer will notice it's portable mode and adjust its installation accordingly. Then you'll have a self-contained copy of Arduino 1.8.19 + Teensyduino 1.56, which will be impervious to interference from files elsewhere on your computer.
 
Thanks, I think I follow you ... but all my computers are Macs. What else can I try? Reinstall from scratch? Or can I manually delete-out the 2.0 files that 1.8.x is finding?
 
I believe on MacOS the files are stored in a "Library" folder which is in your home directory but hidden by MacOS Finder. You'll probably need to use Terminal and type "cd ~/Library" to get to it. Look for an Arduino or Arduino15 subdirectory.
 
Hi, I've been experimenting with this and found a curious clue, but still basically no success.

TLDR: I got it working just once with an older version, then I tried a newer version that failed ... and then the older version didn't work any more either.
I deleted ~/Library/Arduino* many times with no effect. Is there some other place where build components or libraries or ? could be getting cached?

Long version:
I brought over my latest firmware to my older Intel mac and built it in Teensy 1.54 + Arduino 1.8.15. (Those were relatively old versions already installed on that mac.)
I built it, installed it on the same unit that was failing before, and ... SUCCESS! Buttons working, IMU updating, everything looked good.

Next I installed the Teensyduino 1.56 download on that mac, following the link Paul posted. (FWIW, 1.56 + 1.8.16 is the last thing that was working on my M1 mac, whereas this download gives 1.56 + 1.8.19.)
Rebuilt the code, installed it ... FAIL. No buttons working.
The log messages tell me that USB works, and that SPI works to initialize the IMU, but after the MCU is not getting interrupts that it's supposed to get. It's like some (but not all?) of the pins aren't responding.

So I tried again with the 1.54 + 1.8.15 version that had worked a few minutes earlier ... now it too fails, in the same way! Super weird.

Deleted ~/Library/Arduino15 and ~/Library/Arduino and tried again ... still failing.
I delete those again and tried with Teensyduino 1.57 + 1.8.19 ... still failing.

I'm awfully puzzled. I'm wondering if it's possible that a bad object file is getting cached someplace in the build process. Where does Arduino keep those?
Is it possible that the Teensy Loader is getting jammed with an old build? Where does it keep the built firmware?
 
@mykle - the IDE Verbose console output shows the paths used for libraries and files and has a line IIRC (at least on Windows) pointing to the TEMP directory in use where the build takes place.

Deleting those files or changing build or board options or restarting the IDE (AFAIK) forces a clean build.
 
For Windows the compile data gets cached/stored in C:\Users\[user]\AppData\Local\Temp, where [user] is your user name.

Going in with Windows Explorer and delete ALL THAT YOU CAN usually gets rid of the problem.
You should close down your Arduino system before doing the above or else you will not be able to delete the files.
You may have to wait some seconds after closing your Arduino System for Windows to release the files.

In order to display that path you will have to get Windows Explorer to display HIDDEN items.
 
In Arduino, click File > Preferences, and then turn on verbose output during compilation.

It's very verbose! Buried in a mountain of output you'll see almost everything has a full pathnames. With some sifting through the many commands, you can get answers to all the questions about exactly where on your computer the files are located.

At least with Arduino 1.8.x, all the temporarily compiled stuff is supposed to be automatically deleted when you quit the Arduino IDE. On Linux and Windows, just closing the last window quits the IDE. On Mac, I believe that behavior is implemented in the later versions, but some earlier versions might have the traditional MacOS behavior where the application stays running until you use CMD-Q or click File > Quit from the menu bar. Look for the usual MacOS visual feedback on the dock to tell if the software is still running but without any windows open.
 
Often times when I hear or see issues like this, where it works on one machine but not another. And it worked before an update
but not after...

You wonder if you had edited source files in your install? That is maybe you edited some core or library files to makes things work earlier, and the new
install wrote over those.

So if you have a working copy, I would copy the files some place and then compare them to the non-working version and see if there are
differences.
 
Thanks for all the ideas! I finally figured this out, and it's not directly due to Arduino or Teensy versions at all.

In fact, the compile succeeds (on various versions) with optimization->Fast, but fails with (the default?) optimization->Faster .

I never ever touch that setting; I thought that I had been compiling with the Faster setting since forever. Perhaps I am mistaken.

The two main things that are broken by the Faster setting are:
1) The MCU is not hearing interrupts from the IMU (even though the IMU is working & communicating well over SPI)
2) The buttons aren't being read (via the Bounce2 library).

Both of these could be due to input pin signals getting garbled, but there are at least two other input pin signals that are *not* being garbled.

If Bounce2 uses interrupts, then this could also all be due to broken interrupt handling.

Anybody have a theory of why the Faster setting would be optimizing out interrupts, or confusing pins?

Using "fast" for now, but I'm still not sure how this changed ... hmm
 
Bounce2 does not use interrupts.

We've seen these sorts of problems many times where various type of code, but especially use of interrupts, works with one type of optimizations and fails with another. It's almost always a subtle bug where the code is missing "volatile" or synchronization mechanisms with respect to accessing of shared data. Without anything special in your code, the C & C++ language specs allow the compiler wide latitude to optimize code. The order that variables are written to memory can be altered, as long as the code achieves the final result. The compiler is even allowed to not write to memory at all if it can "know" other code never reads the variable. This can be really problematic with interrupts, because the compiler doesn't "know" of any functions actually calling the interrupt function. Careful use of volatile and other mechanisms is essential. It's very easy to build code that seems to work, but then fails when timing changes or the compiler optimizes a different way.

Even simple code issues are pretty much impossible to diagnose by blind guesswork. But this class of problem is among the very hardest. Still, we might be able to spot something... if you were to trim the code down to a small program which still reproduces the problem, and of course show the full program. Especially in this scenarios, it's critically important not to omit parts or show only code fragments. This sort of problem tends to be very tricky and small, seemingly unimportant details like exactly how certain variables are defined can make all the difference.
 
Back
Top