Compiler Library Resolution

Status
Not open for further replies.

visual_micro

Active member
Hi Paul,

I am authoring Visual Micro and have a question. It seems that a user has more success building his project when the Arduino IDE ignores the Teensy specific libraries. Can you please explain what libraries you expect to be used. For example we might have an Ethernet library in teensy\avr\libraries, we have an Ethernet library in ArduinoIDE\Libraries and we might have downloaded an Ethernet library into sketchbook\libraries.

Which Ethernet lib should have priority?
What do you consider to be the correct priority order for lib folders?

Sorry to ask such basic question but nothing makes complete sense :)

Tim
aka Visual Micro
 
Which Ethernet lib should have priority?
What do you consider to be the correct priority order for lib folders?

You've asked a very good question. Even though I kinda know the answer, it was difficult to find and still somewhat cryptic. Here's the info direct from Arduino. Scroll down to the "Compilation" section. See the 6 locations listed following "The include path includes:".

Screen Shot 2018-10-03 at 10.25.39 AM.png
 
Thanks very much for the reply. I think the order of compiler include paths is slightly different to the way the location of each library is established.

Unless I am confused ...

When testing with the Arduino IDE it is not possible to make use of a Teensy library if the same named library exists in either IDE\Libraries or the SketchBook\Libraries. Historically sketchbook/libraries overrides all but I would have expected a Teensy specific library to at least override the generic libraries installed with the Arduino IDE. It seems pointless having the Teensy libraries.

Therefore Visual Micro will revert to using the following library resolution order which is the same as the Arduino IDE. (We do allow project authors to specify libraries more accurately but that is too complex for most users)

If my statements are correct it might be an idea for us all to support the option of "sketchbook\hardware\[platform]\libraries" which would allow users to install libraries using library manager that are for a specific platform. In this case "sketchbook\hardware\[platform]\libraries" will take priority over "sketchbook\hardware\libraries"

I will add this functionality to Visual Micro but if Teensy or Arduino have a better idea that would be useful.

Thanks again
 
I think the order of compiler include paths is slightly different to the way the location of each library is established.

You're right. Those are search paths for #includes and sources. Note though that the last 3 places are libraries locations.

But, finally, I think I found the answer.

Scroll down to the bottom of that page:

Please note: Arduino libraries are managed in three different places: inside the IDE installation folder, inside the core folder and in the libraries folder inside your sketchbook. The way libraries are chosen during compilation is designed to allow the update of libraries present in the distribution. This means that placing a library in the “libraries” folder in your sketchbook overrides the other libraries versions.
The same happens for the libraries present in additional cores installations. It is also important to note that the version of the library you put in your sketchbook may be lower than the one in the distribution or core folders, nevertheless it will be the one used during compilation. When you select a specific core for your board, the libraries present in the core’s folder are used instead of the same libraries present in the IDE distribution folder.
Last, but not least important is the way the Arduino Software (IDE) upgrades itself: all the files in Programs/Arduino (or the folder where you installed the IDE) are deleted and a new folder is created with fresh content.
This is why we recommend that you only install libraries to the sketchbook folder so they are not deleted during the Arduino IDE update process.

So, the priorities of candidate duplicate libraries in the search path goes to:
1) Libraries in the Sketch directory
2) Libraries in the "Arduino Sketchbook" directory (set in Arduino Preferences)
3) Libraries in the Arduino IDE
 
1) Libraries in the Sketch directory
2) Libraries in the "Arduino Sketchbook" directory (set in Arduino Preferences)
3) Libraries in the Arduino IDE

Sorry, I don't think I have it quite right yet. That quote above is saying:

1) Libraries in the Sketch directory
2) Libraries inside the "Core" directories. I think the a "Core" directory could be in more than 1 place inside the Arduino IDE, in a user's Library directory (idk what to call it)(~/Library/Arduino15/packages on Mac), and/or in the "Arduino Sketchbook"/Libraries directory
3) Libraries in Arduino IDE: /Applications/Arduino 1.8.5.app/Contents/Java/libraries

This information is based on Arduino IDE 1.8.5. It might have changed in newer versions.
 
2) Libraries inside the "Core" directories. I think the a "Core" directory could be in more than 1 place inside the Arduino IDE, in a user's Library directory (idk what to call it)(~/Library/Arduino15/packages on Mac), and/or in the "Arduino Sketchbook"/Libraries directory

Arduino doesn't mention the Libraries in "Arduino Sketchbook"/Libraries, but I do. Seem like those are going to be higher priority than Libraries in "Core" directories. So, If I rewrite this once more:

1) Libraries in the Sketch directory
2) Libraries inside the "Arduino Sketchbook"/libraries directory
3) Libraries inside the "Core" directories. I think the a "Core" directory could be in more than 1 place inside the Arduino IDE, in a user's Library directory (idk what to call it)(~/Library/Arduino15/packages on Mac), and/or in the "Arduino Sketchbook"/hardware directory
4) Libraries in Arduino IDE: /Applications/Arduino 1.8.5.app/Contents/Java/libraries

I suggest you ask this question on the Arduino Forum.

Best of luck...Please let us know the response you get.

This really is important information for us to understand. Wish they'd make it easier to find.
 
When testing with the Arduino IDE it is not possible to make use of a Teensy library if the same named library exists in either IDE\Libraries or the SketchBook\Libraries. Historically sketchbook/libraries overrides all but I would have expected a Teensy specific library to at least override the generic libraries installed with the Arduino IDE.

Teensy is a "core". So, yes, the Sketchbook/libraries will override the Teensy "core" libraries. Usually this is not what a user intends but sometimes it is (example: Ethernet 2.0 is intended to replace Ethernet in Teensy and several other cores).

It seems pointless having the Teensy libraries.

It's not clear what is meant. The Teensy libraries are the ones that are specialized and optimized for the Teensy (and possibly others).
 
I'm sad to say the criteria for which library is chosen has become more complex over time as the Arduino developers have responded to all sorts of user needs.

This list is pretty close.

1) Libraries in the Sketch directory
2) Libraries inside the "Arduino Sketchbook"/libraries directory
3) Libraries inside the "Core" directories. I think the a "Core" directory could be in more than 1 place inside the Arduino IDE, in a user's Library directory (idk what to call it)(~/Library/Arduino15/packages on Mac), and/or in the "Arduino Sketchbook"/hardware directory
4) Libraries in Arduino IDE: /Applications/Arduino 1.8.5.app/Contents/Java/libraries

However, case #1 may not really treated as an install of a library (as far as the IDE's build system is concerned), just a consequence of the code being present in the program so a library located elsewhere isn't needed.

There are 2 other situations this list doesn't cover.

1: The Arduino IDE has a feature, which Teensy does *not* use, where one "core" can reference another. The idea is to allow people to make slight variations on Arduino's boards (or theoretically any board) and only include a small set of files in their "core" folder, making reference to all the rest in the original core's folder. For example, Controllino is basically Arduino Mega plus the Ethernet Shield and some other hardware. Boards like that provide a "core" with customized variant.h and boards.txt files, but not much else. When using such a board, both their core libraries folder, and the libraries folder from the original core are considered. If Controllino needed to do something different with SPI, it could have a SPI library that overrides the SPI library in Arduino's main core folder, but the norm for these boards is to minimize the code they fork & maintain. This is pretty much the opposite philosophy of Teensy, where I try to provide a very complete package that's a self-contained and as resilient to unexpected changes in other code as I can make it.

2: Starting with Arduino 1.5.x, they also added a distinction of multiple architectures. My feeling is this probably wasn't a very good design on Arduino's part (a separate core would suffice and is what everyone does anyway), but the architecture names & folders are well established now. Teensy always uses "avr" architecture, even for the 32 bit arm boards, because almost everything in the Arduino world is either "architecture=*" or "architecture=avr", regardless of how it's actually written. Originally the idea was merely that a core folder could provide multiple architecture subdirectories, and so could a library, and when you use such a library the correct subdirectory would be used and the others ignored. But later the criteria for whether a library in the core folder overrides a library in Arduino's main libraries folder (normally the lowest priority, #4 on the list above) was changed to also take a special case of the supported "architecture=" line within the library.properties file. I honestly don't really understand this, and for Teensy it has only been a problem once with the Servo library which I solved by just changing the "architecture=" line in Teensy's copy of Servo.

On top of all that, core folders can be installed in 3 places (at least 3 I know about): within the hardware folder inside the IDE's folder, or in your preferences folder (by default {AppData}\local\Arduino15\packages on Windows), and in a hardware subfolder of your sketchbook folder (by default {Documents}\Arduino\hardware on Windows).

The IDE also has 2 special modes (at least 2 I know about). There's a "portable" mode where it keeps all prefs and state inside its folder. The idea is students can put the IDE on a USB thumb drive and run it on a PC without altering any prefs stored on that PC, so it's all self contained on the portable media. There's also special stuff done for the Windows Store version. Teensy doesn't support the Windows Store version at all, so I haven't looked at this much, other than I know there's code in the IDE which change the locations of some stuff for the Windows Store version. These don't technically alter the library search order, but they may alter where the actual folders are supposed to be located.

It can be a bit exhausting to think of how complex all this stuff has become, and they're planning to add yet another layer of complexity on top of all that with the "Chainsaw" project. So far, Chainsaw looks like "only" adding a couple more levels of subdirectory structure into the core folder, maybe only header files? Personally, I'm skeptical that will end up achieving some of Massimo's goals... so I would imagine eventually it'll become even more complexity in the builder code....

As a 3rd party IDE striving to be compatible with Arduino's build system, this must be incredibly frustrating. It's quite complicated and really is a moving target even now. :(


Martino Facchin (one of the main Arduino developers) recently described the library search this way:

the precedence is a bit more complex than simply being folder based; the complete logic is this:

1. Libraries in User's Sketchbook (highest priority)
2. Libraries in Selected Board Manager Package
3. Libraries bundled in "parent" core (if the selected board is derived from another core)
3. Libraries pre-installed in Arduino IDE (lowest priority)

Inside these categories, libraries are further ordered based on the containing folder name AND the "architectures=" tag

Then there's a particular "priority inversion" that kicks in only in a special case:
- If a core package contains a library that specializes a library with higher priority, the priorities swap

With "specializes" I mean that the sketchbook lib declares as "architecture=*" while the core one reads "architecture=properArch"

In your example, SoftwareSerial is specialized by avr core (https://github.com/arduino/ArduinoC...ibraries/SoftwareSerial/library.properties#L9) while I don't know if the SoftwareSerial version contained in your core is specialized.
 
Last edited:
Would'nt it be possible to just use the arduino builder?
I remember that I called it in a shellscript to compile different versions of teensytransfer (hexfiles for github) in one single run.
 
On the libraries that come with Teensyduino...

It seems that a user has more success building his project when the Arduino IDE ignores the Teensy specific libraries.

If there are problems in any of those library, I'd like to fix them.

I mostly use Linux, not Windows (but I do have a test machine for Windows-only problems), and I really need to do the testing with the latest Arduino IDE. If you find problems that can be reproduced with Arduino, or even just the raw gcc commands, please let me know.
 
On the libraries that come with Teensyduino...



If there are problems in any of those library, I'd like to fix them.

I mostly use Linux, not Windows (but I do have a test machine for Windows-only problems), and I really need to do the testing with the latest Arduino IDE. If you find problems that can be reproduced with Arduino, or even just the raw gcc commands, please let me know.

Thanks Paul and sorry to waste time. We had a user who installed arduino into the sketchbook, "sketchbook\libraries" was the same as "ide\libraries"

This thread can be deleted. Thanks again.
 
Status
Not open for further replies.
Back
Top