'Over the Air' firmware updates, changes for flashing Teensy 3.5 & 3.6

Hi Collin. I've been meaning to post an update with support for Teensy Micromod. I have one here, so if you can wait until tomorrow, I will post an update to FlasherX with support for Teensy MM.

There is no way to transfer/write the program in chunks. You have to send the entire program to the Teensy, which will buffer the entire program either in RAM or in Flash. If your program is small, you can buffer it in RAM. If it's too large to buffer in RAM, you must buffer in Flash. I recommend buffering in Flash because there is no real advantage to buffering in RAM. When the entire program has been sent and is in the buffer, you trigger the action of erasing the existing program in Flash and writing the new one into the erased Flash.
 
Thanks for the reply. I got it to compile (did no testing at all to actually run it) by adding || defined(ARDUINO_TEENSY_MICROMOD) where it is deciding which board to compile for. I just added that to the block that handles the Teensy 4.1 because it's basically the same thing (but actually has even more flash at 16MB!)

I was basing my idea on this comment in code at line 283 of FlashTxx.cpp:
// for T4.x, data address passed to flash_write() must be in RAM

So, it seems like you can't buffer things in FLASH on the Teensy 4.1 or MicroMod? Also, I guess I don't understand why flashing in chunks wouldn't work. I hate to be that guy that constantly seems to second guess people. I'm just honestly curious why it doesn't work.

My end goal is to load updates from an sdcard or via an over the air update from a connected ESP32 on the same board. The OTA update path would basically be able to be the exact same thing as your example program - just stream a hex file over the serial connection. Then have the program basically update itself by loading the firmware from the card then flashing it over top of itself and rebooting. The program will probably be fairly large, perhaps not so large that I could not spare enough RAM to fully buffer it. For the sdcard path it could be as simple as just plain opening the file on the card and calling the relevant routines already written to parse the hex file instead of streaming it over serial. It should be reasonably straight forward to utilize much the same code you've already got.
 
// for T4.x, data address passed to flash_write() must be in RAM

So, it seems like you can't buffer things in FLASH on the Teensy 4.1 or MicroMod?

That comment is in function flash_move(), which "moves" the entire program from the buffer, whether in RAM or Flash, to the Flash program area (begins at FlashBase). The move occurs word-by-word, and if the buffer is in Flash, each word must be read into RAM, and then written from RAM to the new location in Flash. The function that actually writes to Flash for the IMXRT1062 (T4.x and TMM) is called flash_write(), and its "source" argument must be an address in RAM. For the Kinetis chips (TLC and T3.x), the source address for flash_word() or flash_phrase() can be in Flash.

There may be a way to do something very tricky to move some of the program in chunks, but generally speaking, you don't want to erase any of the existing program until you have the entire new program in a buffer, ready to write. When it's ready, the goal is to minimize the time that the Flash does not contain a valid program, so once you have the entire new program buffered, you start the erase/write process and get it done as fast as possible and reboot. If the power fails during that erase/write process, you'll end up with an invalid program in Flash, and the Teensy won't boot. You'll still be able to recover, but only via TeensyLoader.
 
FlasherXv1 with support for Teensy Micromod

The attached files contain an update to FlasherX.ino and FlashTxx.cpp/h with support for Teensy MicroMod. The changes from v0 are very small.

- update FlashTxx.h with a set of macros for Teensy Micromod
- increase the FLASH_SIZE macro for T4.1 from 2MB to 8MB
- no changes to the code

I've tested this with Teensy Micromod and also re-tested with T4.1 to make sure the larger FLASH_SIZE value was okay, and it was.

@jonr, I haven't forgotten about the warnings that come up in Lint, and I'll get to that for the next version.
 

Attachments

  • FlashTxx.h
    5.7 KB · Views: 55
  • FlashTxx.cpp
    23.3 KB · Views: 56
  • FlasherX.ino
    12.4 KB · Views: 52
Oddly enough, so far as I can tell the attached files in the latest message are identical to the ones from message 85. I don't see any MicroMod support added and I did a diff of the files and came up with no differences. I don't know if the forum got confused or what. I've tried on two different computers and I can't seem to get the new files, just the previous version.
 
FlasherXv1 with support for Teensy Micromod - Correct Files

2nd try with files for FlasherXv1
 

Attachments

  • FlasherX.ino
    12.6 KB · Views: 53
  • FlashTxx.cpp
    23.3 KB · Views: 581
  • FlashTxx.h
    6.1 KB · Views: 56
Hi,

I am currently trying to implement updating an image from SD on a Teensy 4.1.

So I modified update_firmware(String Filename), and read_ascii_line(File* serial, char *line, int maxbytes) to make use of File.

Anything else is unmodified, first I ran into an error with the first line. I looked into the file and found that the hex lines are separated in chunks that are separated by shorter strings.

Code:
:0200000460009A
:100000004643464200000156000000000103030081
:1000100000000000000000000000000000000000E0
:1000200000000000000000000000000000000000D0
:1000300000000000000000000000000000000000C0
...
:10FFC0007047AB690C600020136070BC7047092358
:10FFD000024CF7E7D416002090DF0260C81600201C
:10FFE000C0B10822F8B5064645680F460B4C0C48D0
:10FFF00003E054F8140F50B16268AA42F9D1B16815
:02000004600199
:1000000013F0FAFB0028F4D1237C3B70F8BD6FF0AD
:100010002D00F8BD6FF02D00704700BFE0DF0260DB
:10002000DC160020012870B415D00C4C0C4D04E0F7
:100030001E7C864209D05C691D4605F11403002C24
:10004000F6D16FF02D0070BC7047AB690C600020DA
:10005000136070BC70470823024CF7E79816002025
...
:10FFA0002053002507F1FF3C01219A428CBF4FF0FE
:10FFB000000808F001082B4608FB0202501E10F84A
:10FFC000012FBB422CBF002101F001010133CCEB1A
:10FFD0000202B3F5807FB2FA82F24FEA521201FBBD
:10FFE0000255ECD1AF4214BF002508F001056B02A9
:10FFF0005B1B1F40C7EB0E03E367B9F1000F00F076
:02000004600298
...
...
...
:02000004600397
...
...
...
:02000004600496
...
...
...
:02000004600595
...
...
...
:040000056000100087
:00000001FF

When I skip that line, the loop in the method runs well until the beginning of the next chunk.

I compiled with Teensyduino and with vMicro either. The hex files have no diff.

:10FFD000024CF7E7D416002090DF0260C81600201C
:10FFE000C0B10822F8B5064645680F460B4C0C48D0
:10FFF00003E054F8140F50B16268AA42F9D1B16815
:02000004600199
:1000000013F0FAFB0028F4D1237C3B70F8BD6FF0AD
:100010002D00F8BD6FF02D00704700BFE0DF0260DB

The error is 2 (decimal) and appears when the first normal data line (ending here with 6FF0AD) goes to flash_write_block() method.

Question: Can I get rid of this if I suppress these short lines? Will it work as desired?
 
Question: Can I get rid of this if I suppress these short lines? Will it work as desired?

No, those are necessary records. They specify the "base" address for records that follow. Each of those records specifies the top 16 bits of a 32-bit address. For example, the first one specifies address 6000 0000, followed by 6001 0000, 6002 0000, etc. The records in between contain the 64K of data for the given base address.

I see addresses up to 6005 0000, so the code is larger than 256K, and you must buffer in FLASH, not in RAM.

Have you tried sending this hex file to FlasherX via USB serial? If that works okay, then you know that your problem is the code for reading/processing from the SD, and not the file itself. If you attach your hex file to a message, I can try it here.
 
Last edited:
Hi, thank you for your response.

Some about processing the stream from the file. First I ran into probs because the read_ascii_file() gave me an empty line on the next run after every correctly processed string. This was because the hex file not only contains a CR character but additionally a LF (or vice-versa?). Maybe the terminal transfer fixes that (?)

I fixed it for me by peeking the stream for the next byte, and eating it if necessary, before returning to the main routine.

The Strings are identical to those in the file, as far as I see in the serial console.



Now my problem.

I just wrote some Serial.println inbetween the code and I see that it is currently writing to Flash.

The error now occurs at
Code:
				int error = flash_write_block(addr, hex.data, hex.num);

where addr = 1611147304
and num = 16

edit:
In the hex file it is line 9095, directly at the beginning of a new block
:0438200075D8000057
:1038280080C00F40000000031900000010000000D5
/edit

My board is a Teensy 4.1 with Ethernet connected, a 32 G SD, a BMP280, and a memory expansion of 8Mb which is tested to be good by EXTMEM

I attached both, the source code and the hex file, thank you for your help. I yust wrote Database Apps through all my life and this is all really new to me.

(Source code features)

This program is written to get temperature, humidity and pressure measure from a BMP280 (breakout by watterott) and send the values to a MQTT-broker via ethernet. (In my case a mosquitto/streamsheets installation on a raspi). If SD Card is present, it can log to a local file and it will overwrite its default configuration with a config file if exists.

The base topic for rx and tx are configurable, to trigger a measure just send a raw string "tt" to inputbasetopic/trigger, to trigger multiple measurements send <tt,num_measures,sleeptime> (without brackets). <tt,10,100> means do ten measures and wait 100 ms after each measure. MQTT Explorer can help there.

To initiate the update process put a SD card in, put a hex file in the subfolder /update, and send "update" via mqtt to inputbasetopic/system. The transfer process to the final location on flash is currently commented out, so no real update will happen.
 

Attachments

  • Sketch1.zip
    15.5 KB · Views: 50
  • Sketch vom Teensy.zip
    387.9 KB · Views: 47
Last edited:
In the hex file it is line 9095, directly at the beginning of a new block

:0438200075D8000057
:1038280080C00F40000000031900000010000000D5

Thank you for the info. The record that begins with :04 contains 4 bytes for address 3820. The next record begins with :10 and contains 16 bytes for address 3828. This means there is no data for the 4 bytes beginning at 3824. This is unusual, but it should be okay. I need to do some testing, which I can do this evening.
 
FlasherXv2 -- bug fix in FlashTXX.cpp

Your hex file had something I hadn't seen before, and it exposed a bug in the buffering logic in flash_write_block(). The hex file contained a record with only 4 bytes, followed by a record with a start address that did not immediately follow the end address of the previous record.

These lines in flash_write_block():

Code:
    if (buf_count < FLASH_WRITE_SIZE) {			//   if buf not complete
      ((char*)&buf)[buf_count++] = data[data_i++];	//   copy a byte to buf
      continue;						        //     continue while()
    }

needed to be modified to this:

Code:
    ((char*)&buf)[buf_count++] = data[data_i++];	//   copy a byte to buf
    if (buf_count < FLASH_WRITE_SIZE) {			//   if buf not complete
      continue;						        //     continue while()
    }

With this change, I was able to send your entire hex file to a T4.1 running the updated FlasherX sketch. However, after sending the hex file, it failed the check for the FLASH_ID string. You should add a line to your sketch that calls printf( FLASH_ID ), to ensure the FLASH_ID string is embedded in your hex file.

Here are the modified files for FlasherXv2:
 

Attachments

  • FlasherX.ino
    12.7 KB · Views: 58
  • FlashTxx.cpp
    23.3 KB · Views: 55
  • FlashTxx.h
    6.1 KB · Views: 53
Hello. I am using a 3.6 board and I uploaded the flasherx version. As far as software versions, I am using arduino 1.8.16 and Teensyduino 1.55. I am using the blink.ino.hex as the file to be sent via TeraTerminal. Down below, is the error code I am having. Exactly " abort - new code missing string fw_teensy3.6" Is there anyway to find what this is happening? Thanks for your help


Code:
#elif defined(__MK66FX1M0__) 
  #define FLASH_ID "fw_teensy3.6"		// target ID (in code)
  #define FLASH_SIZE		(0x100000)		// 1Mb program flash
  #define FLASH_SECTOR_SIZE	(0x1000)		// 4Kb sector size
  #define FLASH_WRITE_SIZE	(8)			// 8-byte/64-bit writes
  #define FLASH_RESERVE		(2*FLASH_SECTOR_SIZE)	// reserve top of flash
  #define FLASH_BASE_ADDR	(0)			// code starts here


hex file: 545 lines 8676 bytes (00000000 - 000021E4)
new code contains correct FSEC value FFFFF9DE
abort - new code missing string fw_teensy3.6
erase FLASH buffer / free RAM buffer...

FlasherX OTA firmware update v1 Nov 19 2021 21:39:05
WARNING: this can ruin your device
target = fw_teensy3.6 (1024K flash in 4K sectors)
buffer = 968K FLASH (0000C000 - 000FE000)
waiting for hex lines...
 
I am using the blink.ino.hex as the file to be sent via TeraTerminal. Down below, is the error code I am having. Exactly " abort - new code missing string fw_teensy3.6"

If you are running FlasherX on a T3.6, the application does a check to make sure that your hex file was built for the same target. If you send a hex file built for T4.0 to a T3.6, you could brick your T3.6.

As a first test, send the hex file for FlasherX (FlasherX.ino.hex), so after the reboot, the T3.6 will once again be running FlasherX. You can repeat that a few times to get comfortable with how it works. After that, you can modify your blink program to #include "FlashTXX.h", and in the blink sketch, add println(FLASH_ID). This will cause your blink application to contain the string "fw_teensy36" that FlasherX requires.
 
It worked!!!. Couple questions. I added to the blink sketch "#include "FlashTXX.h" and Serial.println(FLASH_ID).Is it ok to place Serial.println(FLASH_ID) inside de VoidSetup()?. I also noticed that after successfully loading and running blink.ino.hex I need to upload again FlasherX if I want to upload a different HEX file. Is that the case?

Thanks. I am still doing lots of testing and so far it works perfect. Thanks for your help./
 
Is it ok to place Serial.println(FLASH_ID) inside de VoidSetup()?

Yes, as long as FLASH_ID is included somewhere in your program, it will work.

I need to upload again FlasherX if I want to upload a different HEX file. Is that the case?

Yes, that's true. Think of FlasherX as just a demonstration of the capability to send new firmware to a Teensy, with these steps: (a) create a buffer to hold the new firmware, (b) receive the firmware and save it in the buffer, (c) verify that the firmware is complete and okay, (d) erase the old firmware and write the new firmware in its place, (e) reboot. If you go back and read earlier messages, particularly message #85, you will find some discussion of how you might use the code in FlashTXX.cpp/h to build "Flasher" capability into your own programs.
 
Great! It works!!!

Thank you very much, I just changed those lines, added a println(FLASH_ID) to the code and it worked instantly.

Maybe the unusual structure might come from vMicro, a plugin to Visual Studio that makes use of the Teensyduino toolchain. It adds some debugging like Breakpoints and watch expressions. It seems to inject some code into the sketch. But its all too obscure to me. I am happy to have it running now. So again many thanks to you and all others who are involved.
 
Maybe the unusual structure might come from vMicro, a plugin to Visual Studio that makes use of the Teensyduino toolchain. It adds some debugging like Breakpoints and watch expressions. It seems to inject some code into the sketch. But its all too obscure to me. I am happy to have it running now. So again many thanks to you and all others who are involved.
Just change from Debug setting to Release, then no debug code is added.
 
Just wanted to say thank you to Joe and everyone else who worked on these OTA routines. I successfully implemented them in my project in order to allow firmware to be updated from an SDCard. I think I'll add support to use them from CANbus as well. My project is automotive and will be in a waterproof enclosure. It will be a really PITA to get to the sdcard. So, while that functionality is very useful, it will be even better to be able to use CAN bus and not have to open the locked case.

Anyway, in the spirit of open source and giving back, all of this will be in my open source project. So, if anyone else needs to load Teensy firmware over sdcard or CAN, just go ahead and snag it from my code if you want. I didn't modify the files found here too much. I did break out all the places where it wanted to use Serial and turned them into FsFile instead. Because I did that I might make my CAN based loader actually just write files to the sdcard. I also have an ESP32 on-board and code to update that from sdcard as well. So, then the CAN loader could be used basically as an sdcard interface instead of a direct bootloader. This is useful to me also because I'll be writing logs to the sdcard so having an sdcard to can interface would be cool. Anyway, that's getting off-topic. The on-topic part is that my code is open and it includes a version that directly reads from the sdcard to flash a Teensy. I tested it to work with Teensy MicroMod.

https://github.com/collin80/GEVCU7
 
Thanks for letting us know. Several people have asked about updating from SD, so now they have a working example. I'm very glad to see you didn't have to make any changes to FlashTXX.cpp/h. My objective was for those to be independent of the method of communication. It's great news that you were able to use them when updating via SD, CAN, etc.
 
Note that the most elegant way to use the FlasherX with a new input device is to implement a minimal Stream class for the device. Then update_firmware() works without any modifications.

On the other hand, it's easy to just replace read_ascii_line() with a version that inputs a line from the device.
 
Here are the modified files for FlasherXv2:

First, thanks for so much detailed work on this. I am developing a remote IOT device that will be quite inaccessible, so this capability is very important.

One suggestion... I hate to ask for more work, but perhaps you could put the current code on github or somewhere. It took wandering through a number of threads here, and then working backwards from the last posting to find the 'latest' code.

Thanks!
 
Back
Top