OTA through Ethernet with Teensy 4.1

Shuptuu

Active member
Hi,
I've been able to implement OTA through Ethernet on a Teensy 4.1.
Based and using the FlasherX Library.
It's using also QNEthernet and Async_AdvancedWebServer (that I had to correct - see my post regarding a bug on Upload here: https://forum.pjrc.com/threads/7222...1-bug-onUpload?p=321229&viewfull=1#post321229 )
So be sure to use the fixed version I posted
My demo code below.
Works like a charm for me.
Tell me what you think about it!!

Code:
/****************************************************************************************************************************
  OTA through Ethernet demo code for Teensy41
  by Olivier Merlet

  WARNING: You can brick your Teensy with incorrect flash erase/write, such as
  incorrect flash config (0x400-40F). This code may or may not prevent that.

  I, Olivier Merlet, give no warranty, expressed or implied for this software and/or
  documentation provided, including, without limitation, warranty of
  merchantability and fitness for a particular purpose.

  based on:
  - FlasherX Library (https://github.com/joepasquariello/FlasherX)
  - QNEthernet 
  - Async_AdvancedWebServer !!!! be sure to use the modified version fixing Upload bug 
  see (https://forum.pjrc.com/threads/72220-AsyncWebServer_Teensy41-bug-onUpload?p=321229&viewfull=1#post321229)
 
  
 *****************************************************************************************************************************/

#if !( defined(CORE_TEENSY) && defined(__IMXRT1062__) && defined(ARDUINO_TEENSY41) )
  #error Only Teensy 4.1 supported
#endif


#define USING_DHCP            true
//#define USING_DHCP            false

#if !USING_DHCP
  // Set the static IP address to use if the DHCP fails to assign
  IPAddress myIP(192, 168, 2, 222);
  IPAddress myNetmask(255, 255, 255, 0);
  IPAddress myGW(192, 168, 2, 1);
  //IPAddress mydnsServer(192, 168, 2, 1);
  IPAddress mydnsServer(8, 8, 8, 8);
#endif

#include "QNEthernet.h"       // https://github.com/ssilverman/QNEthernet
using namespace qindesign::network;

#include <AsyncWebServer_Teensy41.h>

AsyncWebServer    server(80);

#include "FXUtil.h"		// read_ascii_line(), hex file support
extern "C" {
  #include "FlashTxx.h"		// TLC/T3x/T4x/TMM flash primitives
}


//******************************************************************************
// hex_info_t	struct for hex record and hex file info
//******************************************************************************
typedef struct {	// 
  char *data;		// pointer to array allocated elsewhere
  unsigned int addr;	// address in intel hex record
  unsigned int code;	// intel hex record type (0=data, etc.)
  unsigned int num;	// number of data bytes in intel hex record
 
  uint32_t base;	// base address to be added to intel hex 16-bit addr
  uint32_t min;		// min address in hex file
  uint32_t max;		// max address in hex file
  
  int eof;		// set true on intel hex EOF (code = 1)
  int lines;		// number of hex records received  
} hex_info_t;

//******************************************************************************
// hex_info_t	struct for hex record and hex file info
//******************************************************************************
void read_ascii_line( Stream *serial, char *line, int maxbytes );
int  parse_hex_line( const char *theline, char *bytes, unsigned int *addr, unsigned int *num, unsigned int *code );
int  process_hex_record( hex_info_t *hex );
void update_firmware( Stream *in, Stream *out, uint32_t buffer_addr, uint32_t buffer_size );


static char line[96];					// buffer for hex lines
int line_index=0;
static char data[32] __attribute__ ((aligned (8)));	// buffer for hex data
hex_info_t hex = {					// intel hex info struct
  data, 0, 0, 0,					//   data,addr,num,code
  0, 0xFFFFFFFF, 0, 					//   base,min,max,
  0, 0						//   eof,lines
};
bool ota_status=0; // 1=running
bool ota_final=0;
uint32_t buffer_addr, buffer_size;



void OTAend(AsyncWebServerRequest *request){
  AsyncWebParameter* p = request->getParam(0);
  Serial.printf("FILE[%s]: %s, size: %u\n", p->name().c_str(), p->value().c_str(), p->size());
  if (ota_final){
    Serial.printf( "\nhex file: %1d lines %1lu bytes (%08lX - %08lX)\n",
		hex.lines, hex.max-hex.min, hex.min, hex.max );

    // check FSEC value in new code -- abort if incorrect
    #if defined(KINETISK) || defined(KINETISL)
    uint32_t value = *(uint32_t *)(0x40C + buffer_addr);
    if (value != 0xfffff9de) {
      out->printf( "new code contains correct FSEC value %08lX\n", value );
    }
    else {
      out->printf( "abort - FSEC value %08lX should be FFFFF9DE\n", value );
      ota_final=false;
    } 
    #endif
  }

  if (ota_final){
    // check FLASH_ID in new code - abort if not found
    if (check_flash_id( buffer_addr, hex.max - hex.min )) {
      Serial.printf( "new code contains correct target ID %s\n", FLASH_ID );
    }
    else {
      Serial.printf( "abort - new code missing string %s\n", FLASH_ID );
      ota_final=false;
    }
  }

  AsyncWebServerResponse *response = request->beginResponse((!ota_final)?500:200, "text/plain", (!ota_final)?"OTA Failed... Going for reboot":"OTA Success! Going for reboot");
  response->addHeader("Connection", "close");
  response->addHeader("Access-Control-Allow-Origin", "*");
  request->send(response);

  if (ota_final){ 
    Serial.printf( "calling flash_move() to load new firmware...\n" );
    flash_move( FLASH_BASE_ADDR, buffer_addr, hex.max-hex.min );
    Serial.flush();
    REBOOT;
  } else {
    Serial.printf( "erase FLASH buffer / free RAM buffer...\n" );
    firmware_buffer_free( buffer_addr, buffer_size );
    delay(15000);
    Serial.flush();
    REBOOT;
  }
}

void OTA(AsyncWebServerRequest *request, String filename, size_t index, uint8_t *data, size_t len, bool final){
  if (!ota_status){
    Serial.println("Starting OTA...");
    if (firmware_buffer_init( &buffer_addr, &buffer_size ) == 0) {
        Serial.println( "unable to create buffer" );
    } else {
      Serial.printf( "created buffer = %1luK %s (%08lX - %08lX)\n",
      buffer_size/1024, IN_FLASH(buffer_addr) ? "FLASH" : "RAM",
      buffer_addr, buffer_addr + buffer_size );
      ota_status=true;
    }      
  }
  if (ota_status) {
    if(len){
      size_t i=0;
      while (i<len){
        if (data[i]==0x0A || (line_index==sizeof(line)-1)){ // '\n'
          line[line_index] = 0;	// null-terminate
          //Serial.printf( "%s\n", line );
          if (parse_hex_line( (const char*)line, hex.data, &hex.addr, &hex.num, &hex.code ) == 0) {
            Serial.printf( "abort - bad hex line %s\n", line );
            return request->send(400, "text/plain", "abort - bad hex line");
          }
          else if (process_hex_record( &hex ) != 0) { // error on bad hex code
            Serial.printf( "abort - invalid hex code %d\n", hex.code );
            return request->send(400, "text/plain", "invalid hex code");
          }
          else if (hex.code == 0) { // if data record
            uint32_t addr = buffer_addr + hex.base + hex.addr - FLASH_BASE_ADDR;
            if (hex.max > (FLASH_BASE_ADDR + buffer_size)) {
              Serial.printf( "abort - max address %08lX too large\n", hex.max );
              return request->send(400, "text/plain", "abort - max address too large");
            }
            else if (!IN_FLASH(buffer_addr)) {
              memcpy( (void*)addr, (void*)hex.data, hex.num );
            }
            else if (IN_FLASH(buffer_addr)) {
              int error = flash_write_block( addr, hex.data, hex.num );
              if (error) {
                Serial.printf( "abort - error %02X in flash_write_block()\n", error );
                return request->send(400, "text/plain", "abort - error in flash_write_block()");
              }
            }
          }
          hex.lines++;
          line_index=0;         
        } else if (data[i]!=0x0D){ // '\r'
          line[line_index++]=data[i];
        }
        i++;
      }
    }
    if (final) { // if the final flag is set then this is the last frame of data
        Serial.println("Transfer finished");
        ota_final=true;
    }else{
        return;
    }
  } else {
    return request->send(400, "text/plain", "OTA could not begin");
  }
}

void handleNotFound(AsyncWebServerRequest *request)
{
  request->send(404, "text/plain", "Not found");
}

void setup()
{
  Serial.begin(115200);

  while (!Serial && millis() < 5000);

  delay(200);

  Serial.print("\nOTA through Ethernet demo code for ");
  Serial.println(BOARD_NAME);
  

  delay(500);

#if USING_DHCP
  // Start the Ethernet connection, using DHCP
  Serial.print("Initialize Ethernet using DHCP => ");
  Ethernet.begin();
#else
  // Start the Ethernet connection, using static IP
  Serial.print("Initialize Ethernet using static IP => ");
  Ethernet.begin(myIP, myNetmask, myGW);
  Ethernet.setDNSServerIP(mydnsServer);
#endif

  if (!Ethernet.waitForLocalIP(10000))
  {
    Serial.println(F("Failed to configure Ethernet"));

    if (!Ethernet.linkStatus())
    {
      Serial.println(F("Ethernet cable is not connected."));
    }

    // Stay here forever
    while (true)
    {
      delay(1);
    }
  }
  else
  {
    Serial.print(F("Connected! IP address:"));
    Serial.println(Ethernet.localIP());
  }

#if USING_DHCP
  delay(1000);
#else
  delay(2000);
#endif

  server.onNotFound(handleNotFound);

  server.on("/", HTTP_GET, [](AsyncWebServerRequest * request) {
    String html = "<body><h1>OTA through Ethernet demo code for Teensy41</h1><br \><h2>select and send your binary file:</h2><br \><div><form method='POST' enctype='multipart/form-data' action='/'><input type='file' name='file'><button type='submit'>Send</button></form></div></body>";
    request->send(200, "text/html", html);
    });

  server.on("/", HTTP_POST, OTAend, OTA);

  server.begin();

  Serial.print(F("HTTP EthernetWebServer is @ IP : "));
  Serial.println(Ethernet.localIP());
  Serial.print(F("Visit http://"));
  Serial.print(Ethernet.localIP());
  Serial.println("/");
}

void loop()
{
}

the file: View attachment Teensy41_OTA.ino
 
Exactly. As you can do with esp32. Very usefull for boards not in a very accessible place. And I'm surprised nobody did it before!
 
Cool. Will look at it tomorrow.

Sorry Joe, I didn't realize you're the FlasherX author.
Please to meet you! ;)
You did a very good job with your library, thank you!!
Could be maybe nice to include in the library what I've done. I can help if you want.
Well, up to you.
 
Sorry Joe, I didn't realize you're the FlasherX author.
Please to meet you! ;)
You did a very good job with your library, thank you!!
Could be maybe nice to include in the library what I've done. I can help if you want.
Well, up to you.

Thank you for posting your program, yes, it would be good to have examples for all of the possibilities. So far we have USB serial, UART, SD, and Ethernet. Still to go are I2C and SPI, but I think those are not as necessary.
 
Hi Joe,
I found a small issue in my code: I realized the http response is not sent at the end of the process.
During my tests, I was not doing the final flash_move() but firmware_buffer_free() instead, just to not destroy my board with mistakes I could have done. And like that, the http response was correctly sent. So making a new test like that, I discovered the response was sent AFTER the firmware_buffer_free() (but initiated before)!
So I guess the buffer allocation use an address used by AsyncWebServer, disabling the send feature if by chance the firmware you try to upload overwrite this address with a bad value for AsyncWebServer?
Any idea?
 
So I guess the buffer allocation use an address used by AsyncWebServer, disabling the send feature if by chance the firmware you try to upload overwrite this address with a bad value for AsyncWebServer?

You didn't say, but I'm guessing you must be buffering the new firmware in RAM. Check the size of your firmware. If it is smaller than the size of the RAM buffer, then you could try making the RAM buffer smaller. If the code is too large, you can buffer in FLASH, which works very well, and I would recommend you try that. The risk of bricking is really for Teensy 3.x, which has critical FLASH locations which must be written correctly. T4.x has the blink recovery program you can access by holding the program button down for 15 seconds.

Also, I want to acknowledge that Flasher was created by user @jonr for Teensy 3.2, and work was also done by others as you can see in the notes in the source and in the forum threads. I picked up where they left off, resolved some issues in T3.x and extended to LC, T4.x, and MicroMod.
 
I took this bit of code from FlasherX.ino:
Code:
serial->printf( "created buffer = %1luK %s (%08lX - %08lX)\n",
		buffer_size/1024, IN_FLASH(buffer_addr) ? "FLASH" : "RAM",
		buffer_addr, buffer_addr + buffer_size );

And in my case (a teensy 4.1), it returns "Created buffer = 7828K FLASH (60057000 - 607FC000)"

To force RAM usage, you have to change #define RAM_BUFFER_SIZE (0 * 1024) in FlashTxx.h ?

My HEX file is about 980Kb
later it's written: "hex file 22163 lines 354304 bytes (60000000 - 60056800)"
Should not be starting at 60057000?
 
Okay, going back to your problem, you were speculating that the buffer was using an address used by AsyncWebServer, but if the buffer is in FLASH, that can't be. Since you are already using a buffer in FLASH, keep it that way, and debug the other issue. It could be related to something you're doing with FlasherX, but I don't think it's a conflict with the FlasherX firmware buffer.
 
In my demo code, in orde to find when exaclty the system is not able to send a response any more before the buffer to be free, I added a line after "hex.lines++;":
Code:
if (hex.lines==12270){request->send(200, "text/plain", "line");Serial.println(hex.lines;}
And played with the hex.lines number.
The winner is 12270! when hex.lines=12269, I receive the response straight away. When hex.lines=12270, I don't receive the response.
I made the same tests using an other sketch with this OTA update feature included. This skect is bigger. Same behavior but not with the same hex.lines value.
So you're right, this is not linked to the address but something else overloaded somewhere...
But don't know where to search...
At least, it's not blocking nor corrupting the firmware update. So that's not critical.
Just annoying, cause when you need to do a firmware update through ethernet, it's because you don't have access to the usb and a monitoring console. So the http response is the only way to know if the update is ok...
 
I'm getting an odd result when using this.

Everything works, it upgrades the firmware as expected. The odd bit is that it upgrades it twice.
I'm using the example code with minimal changes to wrap it in a class, functionally it's the same. Uploading a local file from a windows 11 desktop using chrome.

Ultimately it's not a critical issue, just unexpected. Anyone have any idea why this is happening?

Oh and thanks to Shuptuu for a very helpful library.
 
I'm getting an odd result when using this.

Everything works, it upgrades the firmware as expected. The odd bit is that it upgrades it twice.
I'm using the example code with minimal changes to wrap it in a class, functionally it's the same. Uploading a local file from a windows 11 desktop using chrome.

Ultimately it's not a critical issue, just unexpected. Anyone have any idea why this is happening?

Oh and thanks to Shuptuu for a very helpful library.

Do you have the same thing using the example without any change?
If not, there's something wrong in your changes.
If yes, read my next post and try with the new code I will post.
 
OK, I got it!
Well, at least I found a workaround for the issue I had.

So the issue has nothing to do with FlasherX. It's a bug in the AsyncWebServer library.
And the response is not sent after the firmware_buffer_free() but when the function returns. (I've been confused cause the firmware_buffer_free() was the last step in the function, but if you had a delay after the firmware_buffer_free(), the response is sent after the delay. So no link at all)

Well, there's a very easy workaround: moving the flash_move() and firmware_buffer_free() out of that function so the response is sent before those final steps.
And I just added a new flag, checked in the loop. If true, we apply the firmware update.

The new demo including this workaround:
Code:
/****************************************************************************************************************************
  OTA through Ethernet demo code for Teensy41
  by Olivier Merlet

  WARNING: You can brick your Teensy with incorrect flash erase/write, such as
  incorrect flash config (0x400-40F). This code may or may not prevent that.

  I, Olivier Merlet, give no warranty, expressed or implied for this software and/or
  documentation provided, including, without limitation, warranty of
  merchantability and fitness for a particular purpose.

  based on:
  - FlasherX Library (https://github.com/joepasquariello/FlasherX)
  - QNEthernet 
  - Async_AdvancedWebServer !!!! be sure to use the modified version fixing Upload bug 
  see (https://forum.pjrc.com/threads/72220-AsyncWebServer_Teensy41-bug-onUpload?p=321229&viewfull=1#post321229)
 
  
 *****************************************************************************************************************************/

#if !( defined(CORE_TEENSY) && defined(__IMXRT1062__) && defined(ARDUINO_TEENSY41) )
  #error Only Teensy 4.1 supported
#endif


#define USING_DHCP            true
//#define USING_DHCP            false

#if !USING_DHCP
  // Set the static IP address to use if the DHCP fails to assign
  IPAddress myIP(192, 168, 2, 222);
  IPAddress myNetmask(255, 255, 255, 0);
  IPAddress myGW(192, 168, 2, 1);
  //IPAddress mydnsServer(192, 168, 2, 1);
  IPAddress mydnsServer(8, 8, 8, 8);
#endif

#include "QNEthernet.h"       // https://github.com/ssilverman/QNEthernet
using namespace qindesign::network;

#include <AsyncWebServer_Teensy41.h>

AsyncWebServer    server(80);

#include "FXUtil.h"		// read_ascii_line(), hex file support
extern "C" {
  #include "FlashTxx.h"		// TLC/T3x/T4x/TMM flash primitives
}


//******************************************************************************
// hex_info_t	struct for hex record and hex file info
//******************************************************************************
typedef struct {	// 
  char *data;		// pointer to array allocated elsewhere
  unsigned int addr;	// address in intel hex record
  unsigned int code;	// intel hex record type (0=data, etc.)
  unsigned int num;	// number of data bytes in intel hex record
 
  uint32_t base;	// base address to be added to intel hex 16-bit addr
  uint32_t min;		// min address in hex file
  uint32_t max;		// max address in hex file
  
  int eof;		// set true on intel hex EOF (code = 1)
  int lines;		// number of hex records received  
} hex_info_t;

//******************************************************************************
// hex_info_t	struct for hex record and hex file info
//******************************************************************************
void read_ascii_line( Stream *serial, char *line, int maxbytes );
int  parse_hex_line( const char *theline, char *bytes, unsigned int *addr, unsigned int *num, unsigned int *code );
int  process_hex_record( hex_info_t *hex );
void update_firmware( Stream *in, Stream *out, uint32_t buffer_addr, uint32_t buffer_size );


static char line[96];					// buffer for hex lines
int line_index=0;
static char data[32] __attribute__ ((aligned (8)));	// buffer for hex data
hex_info_t hex = {					// intel hex info struct
  data, 0, 0, 0,					//   data,addr,num,code
  0, 0xFFFFFFFF, 0, 					//   base,min,max,
  0, 0						//   eof,lines
};
bool ota_status=0; // 1=running
bool ota_final=0;
bool ota_apply=0;
uint32_t buffer_addr, buffer_size;

void OTAapply(){
  delay(100);
  if (ota_final){ 
    Serial.printf( "calling flash_move() to load new firmware...\n" );
    flash_move( FLASH_BASE_ADDR, buffer_addr, hex.max-hex.min );
    Serial.flush();
    REBOOT;
    for (;;) {}
  } else {
    Serial.printf( "erase FLASH buffer / free RAM buffer...\n" );
    firmware_buffer_free( buffer_addr, buffer_size );
    Serial.flush();
    REBOOT;
    for (;;) {}
  }
}

void OTAend(AsyncWebServerRequest *request){
  AsyncWebParameter* p = request->getParam(0);
  Serial.printf("FILE[%s]: %s, size: %u\n", p->name().c_str(), p->value().c_str(), p->size());
  if (ota_final){
    Serial.printf( "\nhex file: %1d lines %1lu bytes (%08lX - %08lX)\n",
		hex.lines, hex.max-hex.min, hex.min, hex.max );

    // check FSEC value in new code -- abort if incorrect
    #if defined(KINETISK) || defined(KINETISL)
    uint32_t value = *(uint32_t *)(0x40C + buffer_addr);
    if (value != 0xfffff9de) {
      Serial.printf( "new code contains correct FSEC value %08lX\n", value );
    }
    else {
      Serial.printf( "abort - FSEC value %08lX should be FFFFF9DE\n", value );
      ota_final=false;
    } 
    #endif
  }

  if (ota_final){
    // check FLASH_ID in new code - abort if not found
    if (check_flash_id( buffer_addr, hex.max - hex.min )) {
      Serial.printf( "new code contains correct target ID %s\n", FLASH_ID );
    }
    else {
      Serial.printf( "abort - new code missing string %s\n", FLASH_ID );
      ota_final=false;
    }
  }

  AsyncWebServerResponse *response = request->beginResponse((!ota_final)?500:200, "text/plain", (!ota_final)?"OTA Failed... Going for reboot":"OTA Success! Going for reboot");
  response->addHeader("Connection", "close");
  response->addHeader("Access-Control-Allow-Origin", "*");
  request->send(response);

  ota_apply=true;
}

extern "C" char* sbrk(int incr);
int freeRam() {
  char top;
  return &top - reinterpret_cast<char*>(sbrk(0));
}

void OTA(AsyncWebServerRequest *request, String filename, size_t index, uint8_t *data, size_t len, bool final){
  if (!ota_status){
    Serial.println("Starting OTA...");
    if (firmware_buffer_init( &buffer_addr, &buffer_size ) == 0) {
        Serial.println( "unable to create buffer" );
    } else {
      Serial.printf( "created buffer = %1luK %s (%08lX - %08lX)\n",
      buffer_size/1024, IN_FLASH(buffer_addr) ? "FLASH" : "RAM",
      buffer_addr, buffer_addr + buffer_size );
      ota_status=true;
    }      
  }
  if (ota_status) {
    if(len){
      size_t i=0;
      while (i<len){
        if (data[i]==0x0A || (line_index==sizeof(line)-1)){ // '\n'
          line[line_index] = 0;	// null-terminate
          //Serial.printf( "%s\n", line );
          if (parse_hex_line( (const char*)line, hex.data, &hex.addr, &hex.num, &hex.code ) == 0) {
            Serial.printf( "abort - bad hex line %s\n", line );
            return request->send(400, "text/plain", "abort - bad hex line");
          }
          else if (process_hex_record( &hex ) != 0) { // error on bad hex code
            Serial.printf( "abort - invalid hex code %d\n", hex.code );
            return request->send(400, "text/plain", "invalid hex code");
          }
          else if (hex.code == 0) { // if data record
            uint32_t addr = buffer_addr + hex.base + hex.addr - FLASH_BASE_ADDR;
            if (hex.max > (FLASH_BASE_ADDR + buffer_size)) {
              Serial.printf( "abort - max address %08lX too large\n", hex.max );
              return request->send(400, "text/plain", "abort - max address too large");
            }
            else if (!IN_FLASH(buffer_addr)) {
              memcpy( (void*)addr, (void*)hex.data, hex.num );
            }
            else if (IN_FLASH(buffer_addr)) {
              int error = flash_write_block( addr, hex.data, hex.num );
              if (error) {
                Serial.printf( "abort - error %02X in flash_write_block()\n", error );
                return request->send(400, "text/plain", "abort - error in flash_write_block()");
              }
            }
          }
          hex.lines++;
          line_index=0;
        } else if (data[i]!=0x0D){ // '\r'
          line[line_index++]=data[i];
        }
        i++;
      }
    }
    if (final) { // if the final flag is set then this is the last frame of data
        Serial.println("Transfer finished");
        ota_final=true;
    }else{
        return;
    }
  } else {
    return request->send(400, "text/plain", "OTA could not begin");
  }
}

void handleNotFound(AsyncWebServerRequest *request)
{
  request->send(404, "text/plain", "Not found");
}

void setup()
{
  Serial.begin(115200);

  while (!Serial && millis() < 5000);

  delay(200);

  Serial.print("\nOTA through Ethernet demo code for ");
  Serial.println(BOARD_NAME);
  

  delay(500);

#if USING_DHCP
  // Start the Ethernet connection, using DHCP
  Serial.print("Initialize Ethernet using DHCP => ");
  Ethernet.begin();
#else
  // Start the Ethernet connection, using static IP
  Serial.print("Initialize Ethernet using static IP => ");
  Ethernet.begin(myIP, myNetmask, myGW);
  Ethernet.setDNSServerIP(mydnsServer);
#endif

  if (!Ethernet.waitForLocalIP(10000))
  {
    Serial.println(F("Failed to configure Ethernet"));

    if (!Ethernet.linkStatus())
    {
      Serial.println(F("Ethernet cable is not connected."));
    }

    // Stay here forever
    while (true)
    {
      delay(1);
    }
  }
  else
  {
    Serial.print(F("Connected! IP address:"));
    Serial.println(Ethernet.localIP());
  }

#if USING_DHCP
  delay(1000);
#else
  delay(2000);
#endif

  server.onNotFound(handleNotFound);

  server.on("/", HTTP_GET, [](AsyncWebServerRequest * request) {
    String html = "<body><h1>OTA through Ethernet demo code for Teensy41</h1><br \><h2>select and send your binary file:</h2><br \><div><form method='POST' enctype='multipart/form-data' action='/'><input type='file' name='file'><button type='submit'>Send</button></form></div></body>";
    request->send(200, "text/html", html);
    });

  server.on("/", HTTP_POST, OTAend, OTA);

  server.begin();

  Serial.print(F("HTTP EthernetWebServer is @ IP : "));
  Serial.println(Ethernet.localIP());
  Serial.print(F("Visit http://"));
  Serial.print(Ethernet.localIP());
  Serial.println("/");
}

void loop()
{
  if (ota_apply){OTAapply();}
}

the file: View attachment Teensy41_OTA.ino
 
If yes, read my next post and try with the new code I will post.

Applying the new changes seems to have cleared the issue, not sure what exactly was going on but it is happy now.

I've made a couple of changes as part of merging this into my existing project. I'm not saying these make it better or worse, just a better fit for my use case or more in my style of coding:

I removed ota_final and ota_apply and changed ota_status into an enum. This then tracks the transfer status through all the stages.

I added the function
Code:
void SendStatusPage(AsyncWebServerRequest *request, const char *Message, const int code=400);

void SendStatusPage(AsyncWebServerRequest *request, const char *Message, const int code) {
   int len = snprintf(PageOut,PageOutBufferSize,"<html><head><meta http-equiv=\"refresh\" content=\"10\"></head>");
   len += snprintf(PageOut+len,PageOutBufferSize-len,"<body><h1>%s</h1></body></html>", Message);
  return request->send(code, "text/html", PageOut);          
}
PageOut is a buffer of PageOutBufferSize bytes that is also used when creating the main upload page which includes things like the current version number.

I then use that function to send any error messages so that it automatically returns to the upload page.
I did something similar for the completion page, it also sends the message as HTML with a refresh meta tag but a larger timeout to allow for rebooting.

In the OTA function I added a check that line_index != 0 before trying to parse the hex line. This prevents an empty line on the end of the file causes errors.

I can post more of my final code if people want, I haven't because it's a little intermixed with my other network handling (I have a raw TCP and raw UDP servers running too)
 
I'm a bit stuck tracking down a semi-related bug.

If on the upload page you hit the submit button without setting a filename the system crashes with a null pointer exception. It looks to be related to an invalid string data type. I'm guessing it doesn't like no filename being sent but having poked around in the library I can't find where to cleanly catch this situation.

As a work around I would recommend changing the main web page to be:
Code:
  server.on("/", HTTP_GET, [](AsyncWebServerRequest * request) {
    String html = "<body><h1>OTA through Ethernet demo code for Teensy41</h1><br \><h2>select and send your binary file:</h2><br \><div><form method='POST' enctype='multipart/form-data' action='/'><input type='file' name='file' required='required'><button type='submit'>Send</button></form></div></body>";
    request->send(200, "text/html", html);
    });
Adding the required attribute file entry box prevents the form being submitted without a file being selected. Ideally the system should still cope with invalid data but at least this makes it harder to send bad data in the first place.
 
Thanks AndyA,
Good ideas for the enum, the function, ...
It's always interesting to see people implemantations, so feel free to post your full code!
For the web page, for sure there's improvements to be done for a better security. On my example, this is just the most simple implemantation for a file transfer. No security at all.
This "required" option is good. Using java, there's other things which can de done like sending first the filename and the filesize to the teensy board for a file check before starting the transfer: you can check if the filename respect some predefined rules and if the size won't be to big. But it's then really project dependant.
 
The guts of the code aren't too different to your code. As mentioned, it's mostly changes to fit it into my code structure.

The NetServer class is a singleton (a way of ensuring only one instance can ever exist of a given class, NetServer::Instance() is used as a pointer to that class in external code, a bit like a global pointer but always valid and read only). In addition to the web upgrade it contains a TCP server and some UDP listening used for the main purpose of the system.
I've removed the code / functions that are unrelated to the web server side so some of the hardware initialisation code and initial network connection is missing but that's all standard.

It also adds a check that an extra line has been added to the start of the .hex file with a specific header. Not something that would be hard to get around if you tried but enough to prevent accidentally loading some random .hex file. This also checks the version number is an upgrade, again doesn't stop you going backwards since it's just checking the text line at the start, but it prevents accidentally loading an old version.

int the main loop() function I call NetServer::Instance()->BackgroundTask();

Added to FXUtil.h (was in the .cpp but if you need it for external code it should be in the header for the library not my own code.)
Code:
typedef struct {	// 
  char *data;		// pointer to array allocated elsewhere
  unsigned int addr;	// address in intel hex record
  unsigned int code;	// intel hex record type (0=data, etc.)
  unsigned int num;	// number of data bytes in intel hex record
 
  uint32_t base;	// base address to be added to intel hex 16-bit addr
  uint32_t min;		// min address in hex file
  uint32_t max;		// max address in hex file
  
  int eof;		// set true on intel hex EOF (code = 1)
  int lines;		// number of hex records received  
} hex_info_t;

void read_ascii_line( Stream *serial, char *line, int maxbytes );
int  parse_hex_line( const char *theline, char *bytes, unsigned int *addr, unsigned int *num, unsigned int *code );
int  process_hex_record(hex_info_t *hex );
void update_firmware( Stream *in, Stream *out, uint32_t buffer_addr, uint32_t buffer_size );

My class header file (with unrelated functions removed)
Code:
#pragma once
#include <Arduino.h>
#include "QNEthernet.h" 
#include "Teensy41_AsyncTCP.hpp"
#include "AsyncWebServer_Teensy41.hpp"
#include "FXUtil.h"		// read_ascii_line(), hex file support
extern "C" {
  #include "FlashTxx.h"		// TLC/T3x/T4x/TMM flash primitives
}


namespace NetServerName{

const int WEB_SERVER_PORT = 80;
const int MaxHTTPMessageLen = 512;
const int HexLineLen = 96;
const int PageOutBufferSize = MaxHTTPMessageLen;
};

class NetServer { 
public:
inline static NetServer* Instance() { if (!m_instance) m_instance = new NetServer; return m_instance;};

// sends any waiting background data and parses recently received packets
void BackgroundTask();

private:
static NetServer* m_instance;
NetServer();

void startNetwork(bool usingDHCP,int port);

AsyncWebServer* webServer = NULL;

void handleNotFound(AsyncWebServerRequest *request);
void handleFrontPage(AsyncWebServerRequest *request);
void OTAend(AsyncWebServerRequest *request);
void OTA(AsyncWebServerRequest *request, String filename, size_t index, uint8_t *data, size_t len, bool final);
inline static void handleNotFoundStatic(AsyncWebServerRequest *request) { m_instance->handleNotFound(request);};
inline static void OTAendStatic(AsyncWebServerRequest *request) {m_instance->OTAend(request);};
inline static void OTAStatic(AsyncWebServerRequest *request, String filename, size_t index, uint8_t *data, size_t len, bool final) {m_instance->OTA(request, filename, index, data, len, final);};
inline static void handleFrontPageStatic(AsyncWebServerRequest *request){m_instance->handleFrontPage(request);};

void SendStatusPage(AsyncWebServerRequest *request, const char *Message, const int code=400);

char line[NetServerName::HexLineLen];					// buffer for hex lines
char PageOut[NetServerName::PageOutBufferSize]; // buffer for output pages
int line_index=0;
char data[32] __attribute__ ((aligned (8)));	// buffer for hex data
hex_info_t hex = {					// intel hex info struct
  data, 0, 0, 0,					//   data,addr,num,code
  0, 0xFFFFFFFF, 0, 					//   base,min,max,
  0, 0						//   eof,lines
};
enum {Idle, WaitingHeader, Active, Complete, Apply, Aborted, Error} ota_status = Idle; 
uint32_t buffer_addr, buffer_size;
};

My NetServer.cpp file (with unrelated functions removed)
Code:
#include "NetServer.h"
#include "EthernetToParallel.h"

using namespace NetServerName;
using namespace qindesign::network;

NetServer* NetServer::m_instance = NULL;

NetServer::NetServer() {
  m_connected = false;
}

void NetServer::startNetwork(bool usingDHCP,int port) {

  webServer = new AsyncWebServer(WEB_SERVER_PORT);
  webServer->onNotFound(handleNotFoundStatic);
  webServer->on("/", HTTP_GET, handleFrontPageStatic);
  webServer->on("/", HTTP_POST, OTAendStatic, OTAStatic);

  webServer->begin();

  m_connected = true;
}


void NetServer::BackgroundTask() {
  if (ota_status == Apply){ 
    delay(500);    
    Serial.printf( "calling flash_move() to load new firmware...\n" );
    flash_move( FLASH_BASE_ADDR, buffer_addr, hex.max-hex.min );
    Serial.flush();
    REBOOT;
    while(true) {};
  } else if (ota_status == Aborted) {
    if (buffer_addr) {
      Serial.printf( "erase FLASH buffer / free RAM buffer...\n" );
      delay(100);   
      firmware_buffer_free( buffer_addr, buffer_size );
    }
    ota_status = Idle;
  }
}

void NetServer::handleNotFound(AsyncWebServerRequest *request)
{
  SendStatusPage(request,"Not found",404);
}

void NetServer::handleFrontPage(AsyncWebServerRequest *request)
{
   int len = snprintf(PageOut,PageOutBufferSize,"<body><h1>MyDevice %s</h1>", SerialNumber);
   len += snprintf(PageOut+len,PageOutBufferSize-len,"<h2>Device info:</h2>Network interface firmware: %u.%u.%u</h2><br>", _Version, _SubVersion,_Build);
   len += snprintf(PageOut+len,PageOutBufferSize-len,"System firmware: %s<br>", FWVersion);
   len += snprintf(PageOut+len,PageOutBufferSize-len,"FPGA: %s<br>", FPGAVersion);
   len += snprintf(PageOut+len,PageOutBufferSize-len,"Hardware: %s<br>", HardwareVersion);
   len += snprintf(PageOut+len,PageOutBufferSize-len,"<h2>Firmware update:</h2><div><form method='POST' enctype='multipart/form-data' action='/'><input type='file' name='file' required='required'><button type='submit'>Send</button></form></div></body>");
   request->send(200, "text/html", PageOut);
}


void NetServer::OTAend(AsyncWebServerRequest *request){
  AsyncWebParameter* p = request->getParam(0);
  Serial.printf("FILE[%s]: %s, size: %u\n", p->name().c_str(), p->value().c_str(), p->size());
  if (!buffer_addr) {
    ota_status = Error;
  }

  if (ota_status == Complete){
    Serial.printf( "\nhex file: %1d lines %1lu bytes (%08lX - %08lX)\n", hex.lines, hex.max-hex.min, hex.min, hex.max );

    // check FSEC value in new code -- abort if incorrect
    #if defined(KINETISK) || defined(KINETISL)
    uint32_t value = *(uint32_t *)(0x40C + buffer_addr);
    if (value != 0xfffff9de) {
      out->printf( "new code contains correct FSEC value %08lX\n", value );
    }
    else {
      out->printf( "abort - FSEC value %08lX should be FFFFF9DE\n", value );
      ota_status = Error;
    } 
    #endif
  }

  if (ota_status == Complete){
    // check FLASH_ID in new code - abort if not found
    if (check_flash_id( buffer_addr, hex.max - hex.min )) {
      Serial.printf( "new code contains correct target ID %s\n", FLASH_ID );
    }
    else {
      Serial.printf( "abort - new code missing string %s\n", FLASH_ID );
      ota_status = Error;
    }
  }

  int len = snprintf(PageOut,PageOutBufferSize,"<html><head><meta http-equiv=\"refresh\" content=\"20\"></head>");
  len += snprintf(PageOut+len,PageOutBufferSize-len,"<body><h1>%s</h1></body></html>",  (ota_status != Complete)?"OTA Failed...":"OTA Success! Rebooting");
  AsyncWebServerResponse *response = request->beginResponse((ota_status != Complete)?500:200, "text/html",PageOut);
  response->addHeader("Connection", "close");
  response->addHeader("Access-Control-Allow-Origin", "*");
  request->send(response);
  if (ota_status == Complete){
    ota_status = Apply;
  } else {
    ota_status = Aborted;
  }
}

void NetServer::SendStatusPage(AsyncWebServerRequest *request, const char *Message, const int code) {
   int len = snprintf(PageOut,PageOutBufferSize,"<html><head><meta http-equiv=\"refresh\" content=\"10\"></head>");
   len += snprintf(PageOut+len,PageOutBufferSize-len,"<body><h1>%s</h1></body></html>", Message);
  return request->send(code, "text/html", PageOut);          
}

void NetServer::OTA(AsyncWebServerRequest *request, String filename, size_t index, uint8_t *data, size_t len, bool final){
  if (ota_status == Idle){
    Serial.println("Starting OTA...");
    Serial.printf("File name %s, len %u\r\n",filename.c_str(), len);
    ota_status=WaitingHeader;
    line_index=0;
    buffer_addr = 0;
  }
  if (ota_status != Idle) {
    if(len){
      size_t i=0;
      if (ota_status != Error) {
      while (i<len){
        if (data[i]==0x0A || (line_index==HexLineLen-1)){ // '\n'
          line[line_index] = 0;	// null-terminate 
          if (ota_status == WaitingHeader) {
            unsigned int VersionIn, SubVersionIn, BuildIn;
            int count = sscanf((const char*)line,"MyDevice %u_%u_%u",&VersionIn,&SubVersionIn, &BuildIn);
            if (count < 3) {
              Serial.printf( "abort - Invalid file format %d\r\n",count);
              ota_status = Error;
              SendStatusPage(request, "abort - Invalid file formal");
              return;          
            } else {
              Serial.printf( "Upgrade to %u,%u,%u\r\n%s\r\n",VersionIn, SubVersionIn, BuildIn,(const char*)line);
            }
            if ((VersionIn > _Version) || 
                ((VersionIn == _Version) && (SubVersionIn > _SubVersion)) ||
                ((VersionIn == _Version) && (SubVersionIn == _SubVersion) && BuildIn > _Build)) {
                  Serial.println("Header good...");
                  if (firmware_buffer_init( &buffer_addr, &buffer_size ) == 0) {
                    Serial.println( "unable to create buffer" );
                    ota_status = Error;
                  } else {
                    Serial.printf( "created buffer = %1luK %s (%08lX - %08lX)\n",
                    buffer_size/1024, IN_FLASH(buffer_addr) ? "FLASH" : "RAM",
                    buffer_addr, buffer_addr + buffer_size );
                    ota_status = Active;
                }
            } else {
              ota_status = Error;
              Serial.println( "abort - Downgrade not allowed");
              SendStatusPage(request, "abort - Downgrade not allowed");
            }
          } else {
            if (line_index>0) {
              if (parse_hex_line( (const char*)line, hex.data, &hex.addr, &hex.num, &hex.code ) == 0) {
                Serial.printf( "abort - bad hex line %s\n", line );
                ota_status = Error;
                SendStatusPage(request, "abort - bad hex line");
              } else if (process_hex_record( &hex ) != 0) { // error on bad hex code
                Serial.printf( "abort - invalid hex code %d\n", hex.code );
                ota_status = Error;
                SendStatusPage(request, "abort - invalid hex code");
              } else if (hex.code == 0) { // if data record
                uint32_t addr = buffer_addr + hex.base + hex.addr - FLASH_BASE_ADDR;
                if (hex.max > (FLASH_BASE_ADDR + buffer_size)) {
                  ota_status = Error;
                  Serial.printf( "abort - max address %08lX too large\n", hex.max );
                  SendStatusPage(request, "abort - max address too large");
                } else if (!IN_FLASH(buffer_addr)) {
                  memcpy( (void*)addr, (void*)hex.data, hex.num );
                } else if (IN_FLASH(buffer_addr)) {
                  int error = flash_write_block( addr, hex.data, hex.num );
                  if (error) {
                    ota_status = Error;
                    Serial.printf( "abort - error %02X in flash_write_block()\n", error );
                    SendStatusPage(request, "abort - error in flash_write_block()");
                  }
                }
              }
              hex.lines++;
            }
          }
          line_index=0;
        } else if (data[i]!=0x0D){ // '\r'
          line[line_index++]=data[i];
        }
        i++;
      }
      }
    }
    if (final) { // if the final flag is set then this is the last frame of data
        Serial.println("Transfer finished");
        ota_status = Complete;
    }else{
        return;
    }
  } else {
    SendStatusPage(request,  "OTA could not begin");
    return;
  }
}
 
One additional tweek - if you use the EEPROM emulation library then that will make use of the top area of the flash as detailed in https://www.pjrc.com/store/teensy41.html#memory

This causes the FlashTxx library to think the flash is full and allocate a buffer size of 0. Unsurprisingly this causes the upgrades to fail.

Changing the value of FLASH_RESERVE in Flashtxx.h to reserve the area required by the EEPROM library fixes this.

The relevant section of Flashtxx.h (line 60 onwards) becomes:
Code:
#elif defined(__IMXRT1062__) && defined(ARDUINO_TEENSY41)
  #define FLASH_ID		"fw_teensy41"		// target ID (in code)
  #define FLASH_SIZE		(0x800000)		// 8MB
  #define FLASH_SECTOR_SIZE	(0x1000)		// 4KB sector size
  #define FLASH_WRITE_SIZE	(4)			// 4-byte/32-bit writes    
  #define FLASH_RESERVE		(0x40 * FLASH_SECTOR_SIZE)	// reserve top of flash - 256k used for eeprom emulation
  #define FLASH_BASE_ADDR	(0x60000000)		// code starts here
 
Sorry if this is a dumb question that has been answered...but where this the OTA part of this example coming from? The Teensy is running ethernet, but how is it connected to a different wifi? network? I have a Adafruit AirLift co-processor(basically an ESP32), and I would love to use it to connect the Teensy to the WiFi and do my updates from there.

thanks,
-Mark
 
I'm not sure I am answering your question, but it's important to understand that the ONLY way to connect to the Teensy bootloader is through a hard connection to the Teensy USB port. OTA updates using FlasherX can be done over any type of connection (USB, UART, Ethernet, etc.) but you have to write your own "receiver" code and build it into your Teensy program. The FlasherX example shows how to do this for UART, and users @shuptuu and @AndyA have written their own Ethernet code to receive a hex file over Ethernet.

If you're hoping that you can just connect to your Teensy over wifi and do your updates the same as with a hard USB connection, that is not possible.
 
Thanks for the reply, Joe!

Before I found this thread, the broad strokes of what I was thinking was:
1) Connect to a WiFi network
2) Receive the firmware update via a UDP port, write it someplace
3) Once the entire firmware update has been received and stored, do the update.

Lots of hand waving at this point, but if I have the new firmware binary stored someplace can't I use FlasherX to update the firmware?

Seeing this thread, that's why I was asking about how it actually did an update "OTA" using the ethernet since I know the T4.1 has built-in ethernet. But if all it takes is storing a binary file that can be transmitted via a UDP port, I can do that today. I just want to use WiFi instead of a direct connection. I'm not too worried about security, and I don't think I need it to be running a server (just open a UDP port). I want to create this method of firmware update because I am creating a smaller robot and having to get it tethered every time I want to perform an update is cumbersome. This update method would greatly increase productivity when writing code for the robot. And it would be cool. :cool:

-Mark
 
Back
Top