Arduino CLI And IDE now Released - Teensy Supported!

Probably as Defragster mentioned and as I mentioned up above and in https://github.com/arduino/arduino-ide/issues/1039
The IDE on first run downloads several libraries into the <sketcheds>/libraries folders including SD.

And if that version has examples it would probably show up in the early section of the examples...

@KurtE - Cool - indeed, that does seem to match your post #196:
SD.h - They install SD library within <sketches>/libraries/SD ... Not compatible with our stuff. I tried to build something and it failed. I then deleted that directory, tried to build, and it still tried to use that SD directory and then erred saying file did not exist, instead of grabbing the one that TD installed which I nicely told me it did not use:

Missed reading that as such, but had seen the sketchbook issue recently when a private lib took precedence.

Wow - this is a complex process replication the old 1.x IDE to add features ... good work discovering/documenting Kurt
 
As I mentioned in previous post, There are a few issues I ran into with the SD library and IDE2, in that the IDE installs a version of the SD library
within the <sketches folder>/libraries/SD

And as things are now, it takes priority over the SD library installed by Teensyduino board manager object...

Did some comments in the issues I created, and was pointed back to the search mechanism that Arduino uses.
https://arduino.github.io/arduino-cli/dev/sketch-build-process/#dependency-resolution

So while I may not like that currently Teensyduino installs all boards as architecture = avr...

We can use that to maybe get around the systems installed library.

In particular currently the library property files in both the Arduino installer plus TD SD libraries has:
Code:
architectures=*

Now if we change the one installed by Teensyduino to have:
Code:
architectures=avr,*

The build process should pick up ours instead of the default one. I hacked it up on a MAC setup and it appeared to work.

Might be interesting if others try it as well.

Sort of a quick and dirty workaround
 
The build process should pick up ours instead of the default one. I hacked it up on a MAC setup and it appeared to work.

Might be interesting if others try it as well.

Sort of a quick and dirty workaround

Well since I have been absorbed with the PI4 lately I gave the hack a try and looks like it works:
Code:
[COLOR="#FF0000"][B]Using library SD at version 2.0.0 in folder: /home/merlin/.arduino15/packages/teensy/hardware/avr/1.56.1/libraries/SD [/B][/COLOR]
Using library SdFat at version 2.1.0 in folder: /home/merlin/.arduino15/packages/teensy/hardware/avr/1.56.1/libraries/SdFat 
Using library SPI at version 1.0 in folder: /home/merlin/.arduino15/packages/teensy/hardware/avr/1.56.1/libraries/SPI

Nice hack!
 
Good news. I've been working on arduino-cli support recently. So far only an experimental Linux x86_64 version is available. Here is the package index URL.

https://www.pjrc.com/teensy/td_156/package_teensy_index.json

I recommend the use of a single permanent URL for package indexes. The package index at that URL contains an array of all releases that have been made of the platform. Compared to a URL that changes at each version, the recommended approach has the following benefits:

  • The user only ever needs to set the URL in their preferences once.
  • It is possible to easily update to newer versions via Boards Manager, or even roll back to a previous version if necessary
  • The Arduino IDE can notify the user of an available update (not yet implemented in Arduino IDE 2.x, but it will be)

I don't know what your plans are for the Teensy package index, but the "156" in the current URL made me a bit nervous so I thought I should mention it.

It also had some cryptic message in the window console area, about to many files to keep track of what files changed... Not sure what files? Maybe ones in trash?

This issue is being tracked here:

https://github.com/arduino/arduino-ide/issues/39

When I did a search for Arduino 2.0 language server - all it was referencing was how to change the language for the IDE. Didn't find anything on the rules for setting up the hover.

Something important to understand when researching this type of subject is that the Arduino Language Server really only serves as a translator between the Arduino system and the standard clangd C++ language server. So, unless it is about something specific to Arduino, you will want to use "clangd" as the search term when looking for information on the language server and don't poison the search by limiting it to the few results that contain the "arduino" keyword.

Now if we change the one installed by Teensyduino to have:
Code:
architectures=avr,*

The build process should pick up ours instead of the default one.

Great work! I came here to suggest just this sort of change (giving the platform bundled library a higher "architecture matching priority" than the standard "SD" library).

Sort of a quick and dirty workaround

Nice hack!

I wouldn't use words like "dirty" or "hack" to describe it. This is using the system exactly as it was intended to be used.
 
Thanks @PerT

@Paul @mjs513 @defragster and all - I updated the first post in this thread to give more of these details
Feel free to edit it I missed something, or something is unclear.
 
Ok, I've put this change into Teensy's copy of SD.

https://github.com/PaulStoffregen/SD/commit/ead866570a82baab25a8c0a582b58c23718f4054

Let's give this a try in 1.57-beta3 and see if anything important breaks!

I'm working today on a script to automate releasing Teensyduino as both installers for Arduino 1.8.x and a package file for Arduino 2.0.0-rc7. Even though not a lot has changed since 1.57-beta2, when (if) I get the package index to be less manual work, going to wrap up 1.57-beta3.
 
Do we need this same hack in Ethernet, SPI, EEPROM, Servo, SoftwareSerial, Wire?


I wouldn't use words like "dirty" or "hack" to describe it.

It really feels like a hack. That's the SD library which traditionally in Arduino 1.8.x was considered built into the IDE and was supposed to be the lowest priority in the library search order. Now it's overriding Teensy's SD library. Even if there is some logical explanation involving the inter-relationship of Arduino CLI, IDE, gRPC or other components, to people who just want to depend on the tools to work properly, it really doesn't matter *why* the wrong library got used (unless of course they did something explicitly to tell the tools to use the wrong library).

Selecting the correct library in a robust way is really important. Even with Arduino 1.8.19, one of the most common end user problems is the wrong library gets used. We still see it over and over on this forum! Even though it's almost always "user error", I would not call Arduino 1.8.19 library selection particularly robust, because so many common human errors lead to wrong & confusing results. Robust software is so much more than just meeting technical requirements... it's meeting human needs when used in the ways imperfect humans work.

I'm trying to strike a balance between contributing to Arduino and enabling everyone interested in pre-release IDE 2.0 to use it with Teensy versus a ton of other non-tools work, including really urgent matters involving semiconductor shortages. Once I get 1.57-beta3 wrapped up with a package index for Arduino 2.0.0-rc7, I'm probably going to switch focus to non-tools stuff for a while.
 
Ok, I've put this change into Teensy's copy of SD.

https://github.com/PaulStoffregen/SD/commit/ead866570a82baab25a8c0a582b58c23718f4054

Let's give this a try in 1.57-beta3 and see if anything important breaks!

I'm working today on a script to automate releasing Teensyduino as both installers for Arduino 1.8.x and a package file for Arduino 2.0.0-rc7. Even though not a lot has changed since 1.57-beta2, when (if) I get the package index to be less manual work, going to wrap up 1.57-beta3.

Sounds good.

Do we need this same hack in Ethernet, SPI, EEPROM, Servo, SoftwareSerial, Wire?




It really feels like a hack. That's the SD library which traditionally in Arduino 1.8.x was considered built into the IDE and was supposed to be the lowest priority in the library search order. Now it's overriding Teensy's SD library. Even if there is some logical explanation involving the inter-relationship of Arduino CLI, IDE, gRPC or other components, to people who just want to depend on the tools to work properly, it really doesn't matter *why* the wrong library got used (unless of course they did something explicitly to tell the tools to use the wrong library).

Selecting the correct library in a robust way is really important. Even with Arduino 1.8.19, one of the most common end user problems is the wrong library gets used. We still see it over and over on this forum! Even though it's almost always "user error", I would not call Arduino 1.8.19 library selection particularly robust, because so many common human errors lead to wrong & confusing results. Robust software is so much more than just meeting technical requirements... it's meeting human needs when used in the ways imperfect humans work.

I'm trying to strike a balance between contributing to Arduino and enabling everyone interested in pre-release IDE 2.0 to use it with Teensy versus a ton of other non-tools work, including really urgent matters involving semiconductor shortages. Once I get 1.57-beta3 wrapped up with a package index for Arduino 2.0.0-rc7, I'm probably going to switch focus to non-tools stuff for a while.

Sounds good, I will try out some of the others. To see if it also hits us.

I do agree with you that they sort of changed the order of things when they moved the stuff into <sketches>/libraries instead of <IDE>/libraries

But specifying the architecture does make sense. Although the idea of them putting into the user library space, with the chance of overwriting the one I already had, has issues. Like I go ahead and put one back and then there is no default one for other architectures...

It sounds like though they could put them in with different directory names like: SPI_ARDUINO_DEFAULT and their rules might apply, and those with the exact directory name I believe again has higher priority. ...

Again I will do a few more experiments.
 
If this continues to be a problem, I'll probably write a tool we'll run with the postbuild hook which inspects the final ELF file and prominently shows warnings if wrong libraries were used. It's going to take every last ounce of self control I have to write that warning message in a way that's not cynical & snarky.

Really, I just want to find a way for the library selection to be reliable, so users don't keep suffering as they have forever with prior Arduino and we don't keep getting the same preventable problem over and over on this forum.

On a personal level, I guess I'm just frustrated that this long-standing Arduino problem of using different libraries than the user intended seems to be getting worse. I really don't care about the nitty gritty of the technical spec & details. I care about users having a good experience.
 
Last edited:
Indeed, using/adding libraries is way too common {even for the newest of 'noobs' the IDE was meant to support} and not having a clear path to using the 'desired one' is a showstopper.

The PJRC console output for "Used or Not Used libs" was a great tool for getting to a usable build after 'Humpty Dumpty Fell', but that is what requires forum attention to find and understand way too often.

Hopefully those console lines still exist with IDE 2.0? I removed prior IDE 2.0 1.56.1 install so can't test for that.
 
If this continues to be a problem, I'll probably write a tool we'll run with the postbuild hook which inspects the final ELF file and prominently shows warnings if wrong libraries were used. It's going to take every last ounce of self control I have to write that warning message in a way that's not cynical & snarky.

Really, I just want to find a way for the library selection to be reliable, so users don't keep suffering as they have forever with prior Arduino and we don't keep getting the same preventable problem over and over on this forum.

On a personal level, I guess I'm just frustrated that this long-standing Arduino problem of using different libraries than the user intended seems to be getting worse. I really don't care about the nitty gritty of the technical spec & details. I care about users having a good experience.
:D

I totally understand that this is an issue with no real good totally right answer.

As you probably could tell from my comments on the Arduino Issue, I described the older process in a simplistic way, that there was sort of 3 places where libraries could be search for in the priority:
<User library folder>, <Boards library folder>, <IDE library folder> and before the libraries we are mentioning where in the lowest priority location and it now moved to the highest priority...
Which caused (or at least exposed) the issue we are running into.

But this is far too simplistic and does not easily handle, what about you put in SD library into the sketches library folder, that supports Teensy boards, but does not support ESP32 or SAM or SAMD. So without someway to distinguish which architecture, a library is developed for, then it would break the others... So they have the checks for architecture, which makes sense. Then I want to overwrite the teensy one and the esp32 one, so then they have ways to have different names...

So I understand where they are coming from, but again I still prefer to think of it in the simplistic way.

So looking through the lists of libraries TD installs as well as which ones Arduino installs,

I think we need to look at: Ethernet, LiquidCrystal and Servo
 
For 1.57-beta3, put in code to allow CLI to do headless compile. But headless upload is more complicated, so I'm going to save that for a future day (maybe even in version 1.58). Hopefully at least headless compile will be useful for people wishing to use Arduino CLI for continuous integration tests (which usually just compile and don't actually run anything on real hardware).
 
I think we need to look at: Ethernet, LiquidCrystal and Servo

Yup, you're right, those 3 are problematic.

I can only get "architectures=avr,*" to work on Ethernet and LiquidCrystal. With Servo, it's still giving preference to home/paul/Arduino/libraries/Servo and ignoring Teensy's copy. :(
 
I'm wrapping up 1.57-beta3 now. The Servo library will have to wait for a future version. At least this will give us all the new stuff on Arduino 2.0.0-rc7, like Serial+MTP, and hopefully we can test new filesystem integration stuff on it soon.

I automated only building and coping the package and tools, so this is going to be another manual edit job on the package index. Going to just add it to the one we already have. In future versions I'll move the package index and automate updating it, but that automation is at odds with just getting this done quickly today.
 
I'm wrapping up 1.57-beta3 now. The Servo library will have to wait for a future version. At least this will give us all the new stuff on Arduino 2.0.0-rc7, like Serial+MTP, and hopefully we can test new filesystem integration stuff on it soon.

I automated only building and coping the package and tools, so this is going to be another manual edit job on the package index. Going to just add it to the one we already have. In future versions I'll move the package index and automate updating it, but that automation is at odds with just getting this done quickly today.

Very cool, Found my issue with talking to the Teensy - was all the PI4 side so in process of reinstalling and configuring it now, so good timing
 
I'm wrapping up 1.57-beta3 now. The Servo library will have to wait for a future version. At least this will give us all the new stuff on Arduino 2.0.0-rc7, like Serial+MTP, and hopefully we can test new filesystem integration stuff on it soon.

I automated only building and coping the package and tools, so this is going to be another manual edit job on the package index. Going to just add it to the one we already have. In future versions I'll move the package index and automate updating it, but that automation is at odds with just getting this done quickly today.

Sounds great. Not sure how to fix Servo... I think I/We need to comment up on the IDE issue...
As I mentioned for SD, they set the architecture=*

So not specific, so us specifying something specific took priority over it.

However when it comes to Servo library, it looks like their architecture= is not * but at least on my RPI one it shows up as:
With a long list... Something like:
"architectures": ["avr", "megaavr", "sam", "samd", "nrf52", "stm32f4", "mbed"],
So it is going to win over almost any other library version...

If I am not mistaken, this is controlled and setup by arduino-cli. I found some stuff in:
D:\GitHub\arduino-cli\arduino\libraries\librariesindex\testdata\library_index.json
and for the last version of it, it has:
Code:
    {
      "name": "Servo",
      "version": "1.1.8",
      "author": "Michael Margolis, Arduino",
      "maintainer": "Arduino \u003cinfo@arduino.cc\u003e",
      "sentence": "Allows Arduino boards to control a variety of servo motors.",
      "paragraph": "This library can control a great number of servos.\u003cbr /\u003eIt makes careful use of timers: the library can control 12 servos using only 1 timer.\u003cbr /\u003eOn the Arduino Due you can control up to 60 servos.",
      "website": "https://www.arduino.cc/reference/en/libraries/servo/",
      "category": "Device Control",
      "architectures": [
        "avr",
        "megaavr",
        "sam",
        "samd",
        "nrf52",
        "stm32f4",
        "mbed",
        "mbed_nano",
        "mbed_portenta",
        "mbed_rp2040"
      ],
      "types": ["Arduino"],
      "repository": "https://github.com/arduino-libraries/Servo.git",
      "url": "https://downloads.arduino.cc/libraries/github.com/arduino-libraries/Servo-1.1.8.zip",
      "archiveFileName": "Servo-1.1.8.zip",
      "size": 128284,
      "checksum": "SHA-256:418e1bb2e690357e37660f61cb26db6088edde15065975320fa5a3453ce78813"
    },
 
Just ran into an interesting problem when doing a clean install of ide2.0 on the PI4. Got to run the rx7 for the pi4 but then gett this error so seems can not the boards (any of them)
Screenshot at 2022-06-12 17-58-53.png

will try again later.
 
Just ran into an interesting problem when doing a clean install of ide2.0 on the PI4. Got to run the rx7 for the pi4 but then gett this error so seems can not the boards (any of them)
View attachment 28666

will try again later.

I get that message more or less every time I startup up IDE2 on my RPI4... But still get boards.

I get an empty boards list, if I edited the extra boards... and mistyped the URL... (I forgot the teensy part of the name...
Once I fixed that in preferences closed it and reopened it, then the boards came up.

Note: It also can be slow for the boards to populate
 
I get that message more or less every time I startup up IDE2 on my RPI4... But still get boards.

I get an empty boards list, if I edited the extra boards... and mistyped the URL... (I forgot the teensy part of the name...
Once I fixed that in preferences closed it and reopened it, then the boards came up.

Note: It also can be slow for the boards to populate

@KurtE
Thanks - copy and paste error with the link for teensy so nothing was loading. Deleted the incorrect one and put in correct link and now all is working again - whhat a PIA - think in old one if link was bad it just skipped it
 
Quick update did verify the CardInfo SD sketch compiled and used the correct version of the SD Library:
Code:
Multiple libraries were found for "SD.h"
  Used: /home/merlin/.arduino15/packages/teensy/hardware/avr/1.56.53/libraries/SD
  Not used: /home/merlin/Arduino/libraries/SD

Tried servo sketch and yep defaults to the arduino version.
 
The PJRC console output for "Used or Not Used libs" was a great tool for getting to a usable build after 'Humpty Dumpty Fell', but that is what requires forum attention to find and understand way too often.

Hopefully those console lines still exist with IDE 2.0?

They do, and in Arduino CLI as well. Just make sure to enable verbose output.

Note that there is currently a bug in Arduino IDE 2.x that makes its verbose output preference work differently from Arduino IDE 1.x/Teensduino:

https://github.com/arduino/arduino-ide/issues/374

gett this error so seems can not the boards (any of them)
View attachment 28666

This error indicates the Arduino IDE could not connect to Arduino's server on startup to check for the availability of a new version of the Arduino IDE to offer for auto-update. By itself, it should not cause any issues other than that you would not be notified of an available IDE update, but of course lack of Internet access in general will impact the other Arduino IDE features that rely on it (Library and Boards Manager, Firmware Updater, Certificate Uploader).

think in old one if link was bad it just skipped it

This is a known bug in Arduino IDE 2.0.0-rc7:

https://github.com/arduino/arduino-ide/issues/1036

One of the developers has already prepared a fix.



About the library overrides issue, I will bring it up with the team today to see if there are any ideas. I'll post here later if I have any news or ideas to share.
 
Hi again. After discussing the subject with the team, I have submitted our proposal for how to resolve the issue of the "built-in" libraries overriding platform bundled libraries due to the sketchbook having a higher "location priority".

The TL;DR is that Arduino IDE 2.x would simply use the same approach as Arduino IDE 1.x: storing the "built-in" libraries in a separate location that has the lowest "location priority".
 
Glad to hear future IDE 2.0 will restore the built in library override. Is there an issue or PR about this on github to follow?
 
Back
Top