Should we use BZ2 or ZST or XZ compression for Teensy downloads in Arduino IDE 2.2.x?

PaulStoffregen

Well-known member
Arduino IDE now supports ZST and XZ compression. I'm considering switching from BZ2. Downside is we lose compatibility with Arduino IDE 2.0.x and maybe even 2.1.x. So the big question is, are these better compression formats worth it?

XZ produces the smallest file sizes. ZST files are somewhat larger, but still a good improvement compared to BZ2. Both should cut download times, but we're talking modest gains like 20% to 30%.

ZST can (maybe) decompress faster. On Linux command line, XZ is slower than ZST but faster than BZ2. However, the speeds inside Arduino don't necessarily seem the same as running the command line program. You should especially be able to see the non-download overhead when switching between versions you're already installed once and Arduino has cached the downloaded files.

To give this a try, I've created alternate package index files.

Code:
Original BZ2 Compression:
https://www.pjrc.com/teensy/package_teensy_index.json

ZST Compression:
https://www.pjrc.com/teensy/package_teensy_zst_index.json

XZ Compression:
https://www.pjrc.com/teensy/package_teensy_xz_index.json

To try, open Preferences and copy 1 of these into the additional URLs. Then in Boards Manager, uninstall Teensy then install again. Or try switching between versions. Arduino IDE caches downloads, so unless you find and delete the cached files, you'll probably only get to experience the download time once for each version (and remember 1.56 & 1.57 have the old toolchain and 1.58 & 0.59 have the new toolchain, which is the lion's share of the downloaded bytes).

If you give these a try, which has the best performance on your machine? Are the slightly larger downloads of ZST offset by its (claimed) faster decompression speed? Is the performance better or worse or about the same on Linux, Mac and Windows? Does Windows 10 vs 11 show any speed difference? I'm really curious to help how it works on your machine, and if I should switch away from BZ2 to either of these?
 
Last edited:
Running on a Windows 11 home edition pc I removed Teensy before each test then did the install. Used 1.58.1 as the common denominator. Windows stopwatch was used for timing.

Original BZ2 Compression: 3 minutes 43 seconds
ZST Compression: 3 minutes 13 seconds
XZ Compression: 8 minutes 54 seconds



Notes:
Most of the time with XZ compression was use by installing teensy-compile@11.3.1

Below is the install process:
Code:
Tool teensy:teensy-discovery@1.58.0 already installed
Downloading packages
teensy:teensy-tools@1.58.0
teensy:teensy-compile@11.3.1
teensy:teensy-monitor@1.58.0
teensy:avr@1.58.1
Installing teensy:teensy-tools@1.58.0
Configuring tool.
teensy:teensy-tools@1.58.0 installed
Installing teensy:teensy-compile@11.3.1
Configuring tool.
teensy:teensy-compile@11.3.1 installed
Installing teensy:teensy-monitor@1.58.0
Configuring tool.
teensy:teensy-monitor@1.58.0 installed
Installing platform teensy:avr@1.58.1
Configuring platform.
Platform teensy:avr@1.58.1 installed
 
Wow, I saw XZ was slower on Linux too, but that's a pretty incredible difference.

Hopefully more people with Windows can give this a try and report on the overall speed....
 
For the heck of it I tried it with a secondary lower end Window's machine. An earlier Surface Pro laptop (originally with Windows 8),
Now running Windows 10. i5-4300 1.9ghz 2.5ghz 8gb of ram...

Also our network at times is not the fastest and other stuff using it as well, like I believe my wife was watchin Youtube on TV in other room.

Original: about 13.5 minutes - Probably if I run it again the time will go down.
ZST - 7min 45 seconds
XZ - about 18 minutes.

Again. these times may be somewhat skewed as I believe that it was also downloading system updates and the like.
 
Looking like ZST is the clear winner. And wow is XZ slow (as implemented by Arduino using Google Go) on Windows. It was slower when I tried on Linux, but nothing like that slow!
 
Original: about 13.5 minutes - Probably if I run it again the time will go down.
ZST - 7min 45 seconds
XZ - about 18 minutes.
@KurtE seems like you might be ready for a computer upgrade - and/or better internet?
I ran the ZST again after XZ and it was under a minute - so part of the time seems to be on the fly downloading?
When I removed 0.59.3 (as before) then installed with the current BZ2 link it seemed just over 30 seconds - so indeed the download wasn't part of the wait.
Internet here at about 40 Mbps - finally almost fast enough to be usable, versus the 6 Mbps of years past - which may be part of @KurtE's timing as AFAIK he's still on old provider used here - though maybe at up to 12 Mbps (that didn't work here thanks to intervening water reflections confounding the radio waves). 0.25 miles away (and three road connect boxes) the DSL there is 70 Mbps.
 
@KurtE seems like you might be ready for a computer upgrade - and/or better internet?
I ran the ZST again after XZ and it was under a minute - so part of the time seems to be on the fly downloading?
I already have better computer, but since I knew that @mjs513 was probably using a faster machine, I thought I would try a slower one, as mentioned:
For the heck of it I tried it with a secondary lower end Window's machine. An earlier Surface Pro laptop (originally with Windows 8),
Now running Windows 10. i5-4300 1.9ghz 2.5ghz 8gb of ram...
Net - yes could be faster. Could go with the Cell phone type connection, although 5G is somewhat spotty where I am.
 
Wow, I saw XZ was slower on Linux too, but that's a pretty incredible difference.
Decided to try this out on my RPI4 running ubuntu 22.04.3 using IDE2.2.1. Not seeing much time difference between XZ and bz2

XZ: 4 minutes 23 seconds
bz2: 4 minutes 39 seconds

Notes:
1. removed all teensy from arduino15 (staging and packages) before installing 1.58.1
2. time based on from clicking install to getting install complete message.
 
Did ZST work on Raspberry Pi?

At least in theory it's supposed to be faster on CPU limited platforms.
that will teach me. Just tried and looks like as you said - works great on CPU limited platforms:
RPI4 ZST: 1 minute 45 seconds !!!!!!

was really surprised!
 
M2 macBook air, Frontier fiber starting from 0.59.3 and doing remove first from the boards manager, got various times for each:


Code:
     ver     min:sec
zst  1.58.1  0:40
XZ   0.59.3  1:34
bz2  1.58.1  2:13
zst  0.59.3  0:17
zst  1.58.1  0:20
bz2  1.56.1  0:42
zst  0.59.3  0:50

Cached files must be messing it up.
 
M1 Mac Mini. Comcast internet: 937Mbits down, 11.3MBits up. (probably maxing Gbit switch and ethernet)
Arduino IDE 2.2.1 MacOS Sonoma 14.1.2 Installing TD 1.58.1
COMP min:sec
bz2: 0:18 0:18
zst: 0:09 0:03 0:03
xz: 1:10 1:11

It appears that the xz decompressed files are getting cached somewhere. The times stayed at ~3 seconds after Arduino restart and system restart.
xz seems to be a clear last choice for compression. About 3/4 the time is taken installing the compiler.

PC Win11 Home Ryzen 7 @3.59GHz. Comcast internet: 942Mbits down, 11.8MBits up. (probably maxing Gbit switch and ethernet) Installing TD 1.58.1
Arduino IDE 2.0.4
COMP min:sec
bz2: 0:35
zst: 0:18
xz: 3.28

xz still the clear loser. 3 times slower on PC than MI Mac Mini.
 
I tried running older IDE versions. Looks like ZST works all the way back to IDE 2.0.4, at least on Linux. Assuming Windows and MacOS are the same.

Looks like IDE 2.0.3 is the point where we lose compatibility with "extracting archive: Not a supported archive" by using ZST compression.

1701481953983.png
 
Paul,

I am assuming that you are doing this, to reduce the sizes of the downloads and speed up the installs.

Wondering if it would also make sense to break up the install some. For example, I don't remember the last time I ever did anything with the any Teensy AVR boards.... I do have one or two sitting around here somewhere, but...
 
Yes, speeding up the install time is the main goal.

Reducing bandwidth cost on the PJRC server is icing on the cake, but not my main focus. We currently have 20TB monthly allocation and we're running between 1.6TB to 1.8TB each month. Eventually we'll use a lot more, as I hope to host videos (now that YouTube puts ads on my non-monitized videos) but that's a project for another day...

I'm not really excited about the user-facing complexity of having Teensy's support be multiple packages. It could reduce the download and install time somewhat (for only 1 package) but having to add more text for people to read on the web page and in Boards Manager, and requiring a decision of which to install isn't a direction I want to do. I want it to be simple and quick install, but given a choice between simple versus quick, my feeling is simple matters most.
 
FWIW, with publishing 1.59-beta4, I changed the main package index to use ZST for all downloads. So far nobody has complained they can't download.

In hindsight, we probably should have switched to ZST earlier. I knew the older versions of Arduino IDE wouldn't support it, but I somehow had imagined it wouldn't be supported until the later versions. Was a pleasant surprise to discover the support for ZST goes all the way back to version 2.0.4.
 
I have to complain. Sloeber now says:

Installing Arduino Board Platforms
Failed to install Teensy
Archive format not supported.
 
I believe he added zstd support on github just a week or two ago. There was a github issue about this, recently closed as being resolved.
 
Back
Top