Updating sketch remotely for end user updates

R

robedney

Active member
I've searched for answers about this but haven't been able to find what I'm looking for. I'm about to publish an open source Teensy with Sound Shield project, a bone conduction programmable hearing assist device. It's all done and working well. This is a low cost alternative for the hearing impaired. The initial sketch includes a 7 channel equalizer that permits entering anyone's audiogram (the results of a hearing test). As I'm anticipating many software upgrades, adding various features (noise reduction, compression, etc.) I'd like the end-user to be able to easily download and install a new sketch.

So, my questions are:

Has anyone built a web-based solution for this? Something that downloads the new sketch, reads the teensy plugged in via USB and upgrades???

Or, is there something simpler than using the Arduino IDE, which requires multiple steps (like board selection, port, etc.)?
Many of the folks who can make good use of this device will be older, not technically savvy and even have dexterity issues -- so I'm looking for the cleanest, easiest solution.

Thanks!!!
 
Chrome has a USB API that allows you to communicate with plugged in devices.
You can use the Teensyloader CLI code or other 3rd party flashers such as TyTools as a basis to develop a small app to find the Teensy, identify it, flash it and reboot it - thus, allowing a user to update the device without installing anything on their computer, directly from their browser

I'm sure Firefox and Safari have similar APIs too
 
Thanks, I'd found that article. Unfortunately I'm just not savvy enough to put this all together in a reasonable period of time. I just can't believe that those who build devices with Arduino and particularly Teensy don't have some way to easily allow end users to update. How about a downloadable executable? Does such a thing exist?
 
As mentioned above, one of the easier ways is to use the Teensyloader CLI tool. Then you can distribute the teensyloader EXE, your firmware file, and a batch file that calls the CLI tool with the proper parameters. Voila, it updates. Now, this is complicated by a few things. For one, everyone distributes the source code for the CLI tool but not the compiled executable. Well, I did publish my version in EXE format a couple of years ago: https://drive.google.com/file/d/1-tSnrXEjOte0xtQ9-XpZSLgJ7QP03TrL/view?usp=sharing

And, my version of the source code that compiled that is here: https://github.com/collin80/teensy_loader_cli

So, in theory that EXE could work for you. However, it's not signed so Windows will complain about it. Your users would need to be able to handle this. On Windows the process isn't terrible. Technically my version also works for Linux but I'll bet you don't have a lot of Linux users. I never did get soft reset working on MacOS. So, basically your users need to use Windows.

Here is the contents of the flash.bat file from my GEVCU7 Updater:
teensy_loader_cli.exe --mcu=TEENSY_MICROMOD -s -v -w GEVCU7.hex

So, that more or less shows you how you'd call the CLI tool to write your firmware.
 
robedney said:
I just can't believe that those who build devices with Arduino and particularly Teensy don't have some way to easily allow end users to update. How about a downloadable executable? Does such a thing exist?
Click to expand...
The method that I use bypasses the Teensy bootloader entirely and has two components. One is a set of routines that you include in your Teensy sketch. This code is known as FlasherX and you can search for it on the forum. The other component is a custom EXE, for Windows in my case, that communicates with the Teensy, sending the new firmware as either Intel Hex records or as raw binary. The Teensy and the custom EXE must agree on a protocol to confirm that the entire new version has been correctly received and buffered by the Teensy, and when those conditions are met, the Teensy can erase its existing firmware, write the new firmware in its place, and reboot. The reason I use this method is to be able to do updates via UART or Ethernet, usually from within an application that a technician might use to configure the device. It works well for me, and others have used it successfully. One open question is whether it can work on a locked Teensy. For now it is only support for unlocked Teensy.
 
You didn't indicate which type of teensy this is. If it's a 4.1 then it's possible to create applications that can be upgraded by inserting an SD card with the new firmware. This is a variation on the system @joepasquariello outlined, you build the flasherX library into your application and it can then upgrade the system on the fly. You need some way to kick it off the upgrade and a way to get it a copy of the new firmware, that could be USB, serial, SD card, ethernet etc...

One of the down sides to this is that if power fails during an update it's possible for the system to end up with junk firmware. And since the firmware is what does the upgrade you can't recover without falling back to using teensyloader.
It it is also possible to create a basic bootloader system. e.g. I have a system that updates over ethernet, it's a large image and reflashing takes 30 seconds. If the power fails during this time you get corrupt main firmware. On boot the bootloader detects the firmware is invalid and instead runs a separate failsafe application, all this failsafe application does is allow updates to the main application. This way I can pull the power out mid upgrade and still recover the system over the network.
 
Web flasher got me intrigued - so started with something simple that can detect the device - will only list Teensy devices for now

First you request a USB device, then you list (it does that automatically on the first request)
Select USB Serial device from the promt.

This works only in Chrome and Edge
 
I only spent about 40 mins on this with ChatGPT
Had to refresh my memory on NodeJS project structures and how to deploy to Heroku.

I will keep playing around with it, but you can flash the teensy, just need to send the HalfKay command to get it into boot mode, then send the HEX over in chunks as the CLI does.

Will also incorporate a Serial Monitor
 
A Serial Monitor inclusion would be great for reviewing settings, etc. My problem is that I'm more of a designer than programmer, meaning that I can put stuff together, design a product and get it functioning with my limited programming skills -- but when you mention things like NodeJS, Heroku, HalfKay and CLI I'm off to Google :) This is an open source project with a budget measured in pennies. Here's a pick of the working prototype.
sparky_in_hand.jpg
box illustration.png
 
AndyA said:
It it is also possible to create a basic bootloader system. e.g. I have a system that updates over ethernet, it's a large image and reflashing takes 30 seconds. If the power fails during this time you get corrupt main firmware. On boot the bootloader detects the firmware is invalid and instead runs a separate failsafe application, all this failsafe application does is allow updates to the main application. This way I can pull the power out mid upgrade and still recover the system over the network.
Click to expand...
@AndyA can you say more about the embedded boot loader for recovery from failure during FlasherX?
All of my devices include USB and the standard Teensy boot loader chip, but I'd love to have a method that's almost foolproof.

I've been following FlasherX for a while, and am excited to see that it is so mature - including with easy SD card support. I already have a wireless way to get files to the SD card, so this could be magic! And maybe I can ditch my Windows and Mac apps designed solely to do firmware updates!
 
Davidelvig said:
@AndyA can you say more about the embedded boot loader for recovery from failure during FlasherX?
All of my devices include USB and the standard Teensy boot loader chip, but I'd love to have a method that's almost foolproof.

I've been following FlasherX for a while, and am excited to see that it is so mature - including with easy SD card support. I already have a wireless way to get files to the SD card, so this could be magic! And maybe I can ditch my Windows and Mac apps designed solely to do firmware updates!
Click to expand...
Do you mind my asking the size of your application? If it's not too large, the time for FlasherX to erase/write can be pretty short, which mitigates the risk of power outage during that critical period. The application where I'm using FlasherX is only about 200KB, and the erase/write takes less than 1 second.
 
I’ll check… but since moving from Teensy 32 to 41…. I’ve happily been able to ignore the output.
1-2 seconds to flash over USB with my Mac.
 
Davidelvig said:
can you say more about the embedded boot loader for recovery from failure during FlasherX?
Click to expand...

I posted an overview of it here: https://forum.pjrc.com/index.php?threads/teensy-4-1-dual-boot-capability.74479/#post-339243 and some code a couple of posts later.

The basic flow I ended up using was:
I define an address in EEPROM that contains a main application status field. This can be: Unknown, New, First Run, Good, Bad.
I define a constant that gives the offset of the main application from the start of flash. Depending on what you do in the bootloader either 128k or 256k is plenty. I found 128k a little tight if I wanted to allow ethernet uploads in the bootloader.
These definitions are common between both bootloader and main application.

The bootloader app can be very minimal code with flasher X and very little else. It is built like any other Teensy app.
The bootloader checks to see if header for main application is in expected location (checkForValidImage() function in code linked above) and also the current value of the status flag.
If the main application isn't found then it waits for an upload.
If the status is First Run it changes the status to Bad (this situation means it tried to run the application and it crashed).
If the status is Bad it waits for an upload.
If the status is Good it runs the application (runApp() in code linked above).
If the status is New or Unknown it changes the status to First Run and runs the application.
Once a flasherX upload of the application is complete it changes the status flag to New and reboots.

The main application is built with two changes to the Teensy library: imxrt1062_t41.ld line 6 is changed to change the FLASH origin and length to reflect the application start address. Bootdata.c line 13 is also changed to reflect the new FLASH start address. Once these files are changed the application can be built in the normal way however the result shouldn't be uploaded using the normal tool.

In the main application the last part of the startup function checks the application status flag, if it is First Run then it changes it to Good.
This means that a new application that crashes in the startup code will be flagged as bad on the next reboot, if it runs to that point that setup completes it will be assumed to be good.
When updating the main application from within the application before flasher X starts any flash operations it sets the status flag to Bad. Once the flash write is complete it sets it to New.

This also involved a couple of changes to the FlahserX library - the default is for it to automatically reboot after an update, I changed this to returning so that I could set the status flag before rebooting. After returning you can't call any functions marked as FLASHMEM, they won't be there any more. I also changed the FlasherX library to think that FLASH starts at the start of application address. This makes it so that you can't erase and overwrite the bootloader.
 
My code result in 212444 bytes. ~1 second upload.

I'm glad because, while I can follow the logic of your custom boot loader logic (mostly), it is well above my pay grade.

Thanks for your help!
 
Davidelvig said:
My code result in 212444 bytes. ~1 second upload.

I'm glad because, while I can follow the logic of your custom boot loader logic (mostly), it is well above my pay grade.

Thanks for your help!
Click to expand...
:) It's the sort of thing that is completely pointless and not even remotely worth the hassle for most applications. If you have physical access to the board then plugging in a USB cable is so much simpler.

At least it's simpler until someone has a bright idea of putting something on the end of an long ethernet cable down a tunnel that you can't easily get access to. All of a sudden being able to remotely recover from a failed upgrade is worth the hassle.
 
OK, posting here seems as right as anything for FlasherX-related stuff.

I've been able to get FlasherX working with little work!
I'd welcome input on my assumptions and process.

I have a 212K app compiled in VS Code, PlatformIO project running on a Teensy 4.1
  1. In VS Code I added a new directory to [project]\libs\ call FlasherX-main (straight from the FlasherX repo)
  2. I added 4 files in that directory from the FlasherX repo (FXUtil.cpp and .h, FlashTxx.c and .h)
  3. I modified FlashTxx.h increasing the FLASH_RESERVE from the default 4 sectors to 64 sector
    #define FLASH_RESERVE (64*FLASH_SECTOR_SIZE) // reserve top of flash
  4. I modified FXUtil.cpp to avoid asking for confirmatory user input at the terminal (commenting lines 116-120)
  5. I create 2 hex files, each calling update_firmware() with the other's filename
  6. and then in then loop() in I my code, when detecting an applicable event (toughing a button, in my case), I run the following code:
uint32_t buffer_addr, buffer_size; if (firmware_buffer_init( &buffer_addr, &buffer_size ) == 0) { Serial.printf( "unable to create buffer\n" ); for (;;) {} } File hexFile; hexFile = SD.open( "theOther.hex", FILE_READ ); if (!hexFile) { Serial.println( "SD file open failed" ); return; } Serial.println( "SD file open OK" ); update_firmware( &hexFile, &Serial, buffer_addr, buffer_size ); firmware_buffer_free( buffer_addr, buffer_size ); REBOOT;

I have more work to remove the print statements and convert them to appropriate "head-less" logging of status.

Is it this simple?

Assumptions:
- I should watch my app code size and increase the 64 in item 3 as appropriate
- It's expected by the authors that I would modify these two "library" files
- If I or FlasherX muck up, my standard teensy4.1 with its bootloader can recover using the USB port and a suitable USB flashing process.

Thanks to all involved in FlasherX! It's a big deal for me.
 
That looks good. If the update_firmware() succeeds, it won't return. If the buffer is in flash, it erases the buffer and calls REBOOT.

Regarding your assumptions:
  • The 64-sector reserve avoids FlasherX using flash that is used by T4.1 emulated EEPROM.
  • Yes, it's common for people to modify the files to match their requirements.
  • If you somehow end up with invalid program in flash, you can recover via the 15-second button hold.
Since your program is small, you could try allocating a buffer in RAM rather than Flash. To do that, simply set the RAM_BUFFER_SIZE macro in FlashTxx.h to whatever size you think will be sufficient, say 256K. FlasherX will try to allocate a block of that size from the heap. Buffering in RAM makes the process faster, so it further reduces the "critical" time of the update.
 
You must log in or register to reply here.
Back
Top