I'm sad to say the criteria for which library is chosen has become more complex over time as the Arduino developers have responded to all sorts of user needs.
This list is pretty close.
1) Libraries in the Sketch directory
2) Libraries inside the "Arduino Sketchbook"/libraries directory
3) Libraries inside the "Core" directories. I think the a "Core" directory could be in more than 1 place inside the Arduino IDE, in a user's Library directory (idk what to call it)(~/Library/Arduino15/packages on Mac), and/or in the "Arduino Sketchbook"/hardware directory
4) Libraries in Arduino IDE: /Applications/Arduino 1.8.5.app/Contents/Java/libraries
However, case #1 may not really treated as an install of a library (as far as the IDE's build system is concerned), just a consequence of the code being present in the program so a library located elsewhere isn't needed.
There are 2 other situations this list doesn't cover.
1: The Arduino IDE has a feature, which Teensy does *not* use, where one "core" can reference another. The idea is to allow people to make slight variations on Arduino's boards (or theoretically any board) and only include a small set of files in their "core" folder, making reference to all the rest in the original core's folder. For example,
Controllino is basically Arduino Mega plus the Ethernet Shield and some other hardware. Boards like that provide a "core" with customized variant.h and boards.txt files, but not much else. When using such a board, both their core libraries folder, and the libraries folder from the original core are considered. If Controllino needed to do something different with SPI, it could have a SPI library that overrides the SPI library in Arduino's main core folder, but the norm for these boards is to minimize the code they fork & maintain. This is pretty much the opposite philosophy of Teensy, where I try to provide a very complete package that's a self-contained and as resilient to unexpected changes in other code as I can make it.
2: Starting with Arduino 1.5.x, they also added a distinction of multiple architectures. My feeling is this probably wasn't a very good design on Arduino's part (a separate core would suffice and is what everyone does anyway), but the architecture names & folders are well established now. Teensy always uses "avr" architecture, even for the 32 bit arm boards, because almost everything in the Arduino world is either "architecture=*" or "architecture=avr", regardless of how it's actually written. Originally the idea was merely that a core folder could provide multiple architecture subdirectories, and so could a library, and when you use such a library the correct subdirectory would be used and the others ignored. But later the criteria for whether a library in the core folder overrides a library in Arduino's main libraries folder (normally the lowest priority, #4 on the list above) was changed to also take a special case of the supported "architecture=" line within the library.properties file. I honestly don't really understand this, and for Teensy it has only been a problem once with the Servo library which I solved by just changing the "architecture=" line in Teensy's copy of Servo.
On top of all that, core folders can be installed in 3 places (at least 3 I know about): within the hardware folder inside the IDE's folder, or in your preferences folder (by default {AppData}\local\Arduino15\packages on Windows), and in a hardware subfolder of your sketchbook folder (by default {Documents}\Arduino\hardware on Windows).
The IDE also has 2 special modes (at least 2 I know about). There's a "portable" mode where it keeps all prefs and state inside its folder. The idea is students can put the IDE on a USB thumb drive and run it on a PC without altering any prefs stored on that PC, so it's all self contained on the portable media. There's also special stuff done for the Windows Store version. Teensy doesn't support the Windows Store version at all, so I haven't looked at this much, other than I know there's code in the IDE which change the locations of some stuff for the Windows Store version. These don't technically alter the library search order, but they may alter where the actual folders are supposed to be located.
It can be a bit exhausting to think of how complex all this stuff has become, and they're planning to add yet another layer of complexity on top of all that with the "Chainsaw" project. So far, Chainsaw looks like "only" adding a couple more levels of subdirectory structure into the core folder, maybe only header files? Personally, I'm skeptical that will end up achieving some of Massimo's goals... so I would imagine eventually it'll become even more complexity in the builder code....
As a 3rd party IDE striving to be compatible with Arduino's build system, this must be incredibly frustrating. It's quite complicated and really is a moving target even now.
Martino Facchin (one of the main Arduino developers) recently described the library search this way:
the precedence is a bit more complex than simply being folder based; the complete logic is this:
1. Libraries in User's Sketchbook (highest priority)
2. Libraries in Selected Board Manager Package
3. Libraries bundled in "parent" core (if the selected board is derived from another core)
3. Libraries pre-installed in Arduino IDE (lowest priority)
Inside these categories, libraries are further ordered based on the containing folder name AND the "architectures=" tag
Then there's a particular "priority inversion" that kicks in only in a special case:
- If a core package contains a library that specializes a library with higher priority, the priorities swap
With "specializes" I mean that the sketchbook lib declares as "architecture=*" while the core one reads "architecture=properArch"
In your example, SoftwareSerial is specialized by avr core (
https://github.com/arduino/ArduinoC...ibraries/SoftwareSerial/library.properties#L9) while I don't know if the SoftwareSerial version contained in your core is specialized.