Thanks Joe, that works with the FlasherX hex indeed!
So I just need to make sure that the "fw_teensy40" is set in my hex, I guess I can use #define FLASH_ID "fw_teensy40" in my sketch.
Thanks again
OTA through Ethernet with Teensy 4.1
- Thread starter Shuptuu
- Start date
Thanks Joe, that works with the FlasherX hex indeed!
So I just need to make sure that the "fw_teensy40" is set in my hex, I guess I can use #define FLASH_ID "fw_teensy40" in my sketch.
Thanks again
joepasquariello
Well-known member
So I just need to make sure that the "fw_teensy40" is set in my hex, I guess I can use #define FLASH_ID "fw_teensy40" in my sketch. Thanks again
You're welcome, and yes, the idea of FlasherX is to build it into your own application, so that as you release new versions, each one can be updated via FlasherX to the new version.
Hey Joe,
There are these lines of code in FXUtil.cpp :
I understand that it is detecting a case where the USB serial is being used as a source for the update, but is it important that both in and out are both Serial? What if I have a case where in is Serial but out is Serial5? Is it important to perform the print/flush? Should print/flush always be performed on in if it is USB serial, regardless of what the value of out is?
-Mark
There are these lines of code in FXUtil.cpp :
Code:
// reliability of transfer via USB is improved by this printf/flush
if (in == out && out == (Stream*)&Serial) {
out->printf( "%s\n", line );
out->flush();
}I understand that it is detecting a case where the USB serial is being used as a source for the update, but is it important that both in and out are both Serial? What if I have a case where in is Serial but out is Serial5? Is it important to perform the print/flush? Should print/flush always be performed on in if it is USB serial, regardless of what the value of out is?
-Mark
joepasquariello
Well-known member
Hi Mark. In my original testing, both in and out were USB Serial. I don't think the print/flush is needed if your output stream goes to a UART, but it's probably worth testing.
I would like to submit this new version of read_ascii_line. I was looking at handling UART the same way I handled TCP, but ran into issues with data not always being available and then getting bad hex lines as a result. This version handles it by processing until the max bytes read, a CR (or CRLF) is encountered after at least the first character has been read, or a timeout (10 ms) occurs waiting for the stream to become available. I am curious also if this would solve the reliability/integrity issues you were seeing with USBSerial, Joe.
Let me know what you think.
-Mark
Code:
void read_ascii_line( Stream *serial, char *line, int maxbytes, bool* streamHasCRLF )
{
int c;
int count = 0;
int crlfCount = 0;
uint32_t lastReadTime = millis();
while(count < (maxbytes - 1) && millis() < (lastReadTime + 10)) {
if (serial->available()) {
c = serial->read();
lastReadTime = millis();
} else {
continue;
}
// if c is a CR or LF
if (c == '\r' || c == '\n') {
if (count == 0) {
// use this as indication that lines have CRLF
*streamHasCRLF = true;
} else {
if (*streamHasCRLF) {
crlfCount++;
if (crlfCount == 2) {
// reached end of line
break;
}
} else {
// reached end of line
break;
}
}
} else {
line[count++] = c;
}
}
// null terminate
line[count] = 0;
}Let me know what you think.
-Mark
joepasquariello
Well-known member
Hi Mark. What is "maxbytes"? Is it the size of the line array? What are you using to send the hex file to the Teensy?
Hey Joe,
The signature of the method hasn't changed except for the addition of the bool* to hold the streamHasCRLF value between calls. maxbytes is the total number of bytes available to the char* line.
As far as my setup, I have two Teensy 4.0s. The first one is connected to WiFi and is listening for data on a TCP port. When data is detected, it starts reading the data as bytes in 512 byte chunks and then writing them to Serial5. The second Teensy is receiving the sent data on its Serial5. When it detects incoming data, it starts processing by sending Serial5 as a Stream* to FlasherX update_firmware method (after setting up the firmware buffer, etc). Serial5 is initialized to 115200 on both Teensys and have 16384 byte buffers for RX and TX. The issue I was seeing was that I would end up with a bunch of 'invalid hex line' errors where the line string was empty. I think it was easily possible to reach the 'end' of incoming data and have cases where data was not available. It could be at the end of a line or it could be in the middle of a line. So, I reworked the code to keep checking for incoming data and only give up when 1) the buffer limit was reached, 2) a CR or CRLF was encountered for the end of the line, or 3) a time threshold was breached since the last char was read. The 10ms threshold has not been a problem in my limited testing. If the threshold were to be breached, then most likely a bad line would be generated.
With this new code, I don't have any issues with bad hex line errors and everything is read correctly on my above circuit setup. I was going to look at replicating something for the USB serial as input next, as soon as I can figure out how to send data to the USB port. I am not sure what issues you had seen with the USB serial input, and I was wondering if this code would also alleviate those issues as well. Then you wouldn't need the print/flush code if the input was equal to Serial.
Let me know if I can answer any more questions!
-Mark
The signature of the method hasn't changed except for the addition of the bool* to hold the streamHasCRLF value between calls. maxbytes is the total number of bytes available to the char* line.
As far as my setup, I have two Teensy 4.0s. The first one is connected to WiFi and is listening for data on a TCP port. When data is detected, it starts reading the data as bytes in 512 byte chunks and then writing them to Serial5. The second Teensy is receiving the sent data on its Serial5. When it detects incoming data, it starts processing by sending Serial5 as a Stream* to FlasherX update_firmware method (after setting up the firmware buffer, etc). Serial5 is initialized to 115200 on both Teensys and have 16384 byte buffers for RX and TX. The issue I was seeing was that I would end up with a bunch of 'invalid hex line' errors where the line string was empty. I think it was easily possible to reach the 'end' of incoming data and have cases where data was not available. It could be at the end of a line or it could be in the middle of a line. So, I reworked the code to keep checking for incoming data and only give up when 1) the buffer limit was reached, 2) a CR or CRLF was encountered for the end of the line, or 3) a time threshold was breached since the last char was read. The 10ms threshold has not been a problem in my limited testing. If the threshold were to be breached, then most likely a bad line would be generated.
With this new code, I don't have any issues with bad hex line errors and everything is read correctly on my above circuit setup. I was going to look at replicating something for the USB serial as input next, as soon as I can figure out how to send data to the USB port. I am not sure what issues you had seen with the USB serial input, and I was wondering if this code would also alleviate those issues as well. Then you wouldn't need the print/flush code if the input was equal to Serial.
Let me know if I can answer any more questions!
-Mark
joepasquariello
Well-known member
Hi Mark. Am I correct that you have this: host <----wifi----> T4(1) <----UART----> T4(2) ?
Are you sending the actual hex file produced by the IDE from the host to T4(1)? How are you sending it? I don't understand why there is an issue related to LF versus CR/LF. The hex file has only LF, right?
Regarding the issue I had, I don't understand USB well enough to be sure, but the print/flush seemed to be necessary to slow things down. That's probably not entirely correct, and possibly completely wrong, but it works.
Yes, you're picture is my thousand words.
Yes. I am sending the .hex file generated by the IDE, slightly modified to have the file size and CRC32 value at the beginning of the file. I'm using linux, so I don't entirely understand why the .hex file has CRLF instead of just CR, but I can verify that it does. I'm sending the contents of the file using the netcat command. Here is a script I use to prepend the size and CRC32 values and then send the file via netcat.
And just to be clear, the current code in FlasherX handles CR vs CRLF just fine. It just ignores any CR or LF that it finds at the beginning of a line until it reaches a non-CR or LF character. My earlier issue was that the final LF was never read. And this would throw off any CRC32 check I wanted to include. So I had modified the original read_ascii_line to ensure that it read all of the stream contents (including the final LF).
But that is not the issue I was encountering now. I was encountering times when read_ascii_line would return an empty line, and looking at the current code it seems that the only time that could really happen is when the stream->available() returns false each time it was called. That is why I added the check for a timeout threshold, giving the system a chance to wait for more data to arrive before returning a partial or empty line. Between the need to read all CRs and LFs and to wait for a timeout, it was easier to rewrite the method, and I was just offering it back since I like FlasherX and have been using it.
I have packaged the FlasherX code (with my changes) with the size and CRC32 check, and my support code to process updates via USB, UART, or TCP into a FlasherXUpdater library. That way I can reuse more code across my projects. It's not 'pure' in the sense that it has no other dependencies, it is dependent on some of my other libraries I use, but it works for my situation.
I've tested it across USB, UART, and TCP with Teensy 4.0s on my linux system, and it seems to be working very well. I removed the print/flush code for Serial, and at 115200 baud it doesn't seem to drop any bytes. I do have to create a larger read/write buffer though. You can see the examples that I included. And I also made the timeout threshold configurable because with TCP I still get the occasional bad hex line. But I have read_ascii_line instrumented to print when it encounters a boundary error (max bytes or timeout), so hopefully I can have a better idea of what is going on when it happens next.
Anyways, please feel free to pull out anything you find useful back into FlasherX. Like I said, just trying to give back.
thanks,
-Mark
joepasquariello
Well-known member
Hi Mark. I included read_ascii_line() in FXUtil.cpp because I thought it was "generic" enough that it would be useful to anyone transferring via UART. I think in your case, the gaps appear because of what is happening on the wifi side, i.e how the file is broken up into chunks, and where the breaks occur. Could you modify that code so the breaks occur after whole lines? Or, could you put something in front of read_ascii_line() that does the waiting? In my own applications, and I think this should be true for you, too, I want to control the transfer, so what I do is:
(1) on the host side, read the hex file and parse it into an array holding the entire image
(2) send the image to Teensy in relatively large chunks via a packetized protocol
(3) on the Teensy side, receive and parse packets and write the data to the flash buffer
The core of FlasherX is creating the buffer, writing to the buffer, reading from the buffer, and erasing/writing the new code to flash. Everything related to how the transfer occurs should probably be outside the library.
The question I'm trying to ask about the wifi transfer is _how_ does the file get transferred? In other words, _why_ are there gaps at random points in the hex file? Do you not control what is in each TCP packet?
The other thought that I have is since there seems to be a difference between sending from linux and windows hosts, I would define the CRC not on the file, but on the contents after parsing the hex records, because that would be independent of platform.
Ultimately, I don't think I have control over how or how large the packets are being sent. I could try to write code to bundle into packets, but the TCP layer will have more control than I do, I think. It may try to optimize packets and contents. The introduction of the timeout threshold doesn't slow anything down when using USB or UART, the data will always be read as fast as it is received. It just allows some flexibility if the content happens to be slightly delayed or hits some kind of bottleneck. I will say that the timeout has only been needed for the TCP transfer. The only time I have encountered a bad hex line using UART or USB was when I had it at 115200 baud but did not allocate a larger read/write buffer (you can see it in my examples). In that case I think there was a data overwrite that was happening in the receiving buffer (I think the buffer is implemented as a ring buffer). But once I allocated a large enough buffer (8192 seemed to do the trick), everything worked great.
My testing of the UART situation is somewhat unique. Sending the data over TCP and then sending the data over UART is probably not a common case. But it actually replicates the current situation on my robot where one Teensy is connected to the WiFi and is acting as a data conductor to/from the second non-WiFi Teensy. I'd be willing to bet that if one replaced the TCP part with USB, there wouldn't be much of a timeout/available issue.
Also, I didn't want to be required to write code to package the data in a certain way so that the data or the code matches up. That seems brittle. I'm already prepending the .hex file size and crc32 value, but that data has to be included in some manner. I like that I can cobble together the needed bytes just using bash scripts and no specialized C/Python code.
The great thing about your code is that once I have a Stream that contains the .hex file data, I can just hand it off to perform the update. As long as it can be managed as a Stream, everything just works. I just had to generalize the read_ascii_line code to allow for possible slight delays. And adding the size and CRC32 check before allowing the update to happen seemed natural since I could calculate the CRC32 value in parallel with the hex lines being processed. No extra buffer or time required.
There isn't a platform issue with the CRC. As long as the bytes that are sent are the same as the bytes in the original file (whether CR or CRLF), then the CRC will work correctly. The act of sending the bytes is not converting the contents between CR or CRLF. The original code handled CR vs CRLF just fine (and my new version does too), it just didn't read the last LF because it would stop at the last CR (it got the last hex line...so why continue reading data). There is a CRC32 tool on linux, and I imagine that there is some equivalent tool for Windows. It's a pretty well known algorithm. But sending a "Windows" version or a "Linux" version to the same destination, both will be processed correctly as far as CRLF is concerned. As long as the CRC32 is run on the actual contents that are sent, it should work fine.
Anyways, like I said, take what you like, ignore the rest or ignore it all. I have to say that I have been using it all day to update the two sketches used on my robot (one for the Teensy connected to WiFI, one for the other Teensy connected via UART to the first Teensy), and it has been awesome. No glitchy brick updates. Every now and then there is a dropped TCP packet or something, but it is correctly detected and the update is aborted. Now if I could just remember to not send an updated sketch that has the wrong WiFi password, I would be golden.
That is the one problem that no timeout threshold is going to fix. -Mark
And with that comment, I did not mean to imply that the FlasherX was doing gltichy brick updates before. Just my case of doing stuff over TCP was being gltichy for the reasons we've been discussing. Which was not an original use case for the code. Just to be clear.
FlasherX is great work.-Mark
joepasquariello
Well-known member
And with that comment, I did not mean to imply that the FlasherX was doing gltichy brick updates before. Just my case of doing stuff over TCP was being gltichy for the reasons we've been discussing. Which was not an original use case for the code. Just to be clear.
No worries, Mark. I'm glad that FlasherX is helping you. I couldn't do my projects without it. I get what you're saying about being able to do what is necessary with scripts, etc. It's interesting to me that there is no standard UI in the Arduino world. Some people are using Python. I use Windows development tool C++ Builder, so I have always been able to share C/C++ modules between the UI and the target.
Hi,
great that you created this possibility to update a teensy...
I have a question:
When i am using this method first time i have to use the code you posted here and download it to the teensy, Okay
Then i am able to download my application via OTA(OTW), Okay
Then the teensy will boot up again with my application, Okay
Is now still the possibility to do a OTA, because i assume the OTA code is now overwritten with my application? Is the OTA code still "there" somewhere?
Thank you
Torsten
great that you created this possibility to update a teensy...
I have a question:
When i am using this method first time i have to use the code you posted here and download it to the teensy, Okay
Then i am able to download my application via OTA(OTW), Okay
Then the teensy will boot up again with my application, Okay
Is now still the possibility to do a OTA, because i assume the OTA code is now overwritten with my application? Is the OTA code still "there" somewhere?
Thank you
Torsten
joepasquariello
Well-known member
You must add the FlasherX utility functions into your own application. Think of FlasherX as a set of functions that provide the following:
- create a buffer in flash or RAM to store the new firmware
- write the new firmware into the buffer
- erase the existing firmware and write the new firmware into flash
What you need to decide for your own application is how you want to transfer the new firmware from your host to the Teensy. The example application for FlasherX allows you to send the hex file containing new firmware from host to Teensy using a terminal emulator (such as TeraTerm), or to read the hex from built-in SD card on T3.5, T3.6, or T4.1. Neither of these is likely what you will want to do in a "real-world" application, but they do provide working examples to get you started.
- create a buffer in flash or RAM to store the new firmware
- write the new firmware into the buffer
- erase the existing firmware and write the new firmware into flash
What you need to decide for your own application is how you want to transfer the new firmware from your host to the Teensy. The example application for FlasherX allows you to send the hex file containing new firmware from host to Teensy using a terminal emulator (such as TeraTerm), or to read the hex from built-in SD card on T3.5, T3.6, or T4.1. Neither of these is likely what you will want to do in a "real-world" application, but they do provide working examples to get you started.
Hi,
thank your for this explenation....very good...
Will experiment with this...my plan is to update via ethernet..and browser....i think it is the most convinient way...and everyone has a browser and ethernet...
Regarding flash or RAM, who decides this? Is this "automatic" dependant of the size, or is this a define to set before?
Torsten
Last edited:
Nice work. Got me thinking.
If FlasherX simply accepts buffer additions using
I could get this implemented pretty easily with some of our systems to enable CAN-based updating. It would be horrendously slow but it would allow for firmware updates to be executed with our scan tool similar to a car which would be nice for our physically inaccessible modules.
Might also be interesting to try to do it via one of our I2C busses. That might even be a more appropriate use for this.
If FlasherX simply accepts buffer additions using
Code:
flash_write_block()Might also be interesting to try to do it via one of our I2C busses. That might even be a more appropriate use for this.
joepasquariello
Well-known member
FlasherX can work with any method of transmitting the new firmware. If you can get the new firmware into a buffer in Flash or RAM, you can then do the update.
I am wondering why Shuptuu is placing the null character if data == 0x0A (line feed). If the data portion of the line contains 0x0A, then the parse_hex_line() call will fail, won't it?
Edit: Okay nevermind. We are not reading a binary file. We are reading a text file...
Code:
if (data[i]==0x0A || (line_index==sizeof(line)-1)){ // '\n'
line[line_index] = 0; // null-terminate
//Serial.printf( "%s\n", line );
if (parse_hex_line( (const char*)line, hex.data, &hex.addr, &hex.num, &hex.code ) == 0) {
Serial.printf( "abort - bad hex line %s\n", line );
return request->send(400, "text/plain", "abort - bad hex line");
}
.
.
.
}Edit: Okay nevermind. We are not reading a binary file. We are reading a text file...
Last edited:
Okay, implemented it and works like a charme with an ethernet connection and the async web service...
Thank you very much for this feature!
Thank you
Torsten
joepasquariello
Well-known member
Glad to hear. If you can share your code, it will help others trying to do the same thing.
Yes you i are right, i did it exactly like in the example in the beginning of this thread. Only i modified it that it fits into my application "statemachine" => when the OTA transfer is initiated i stop all my "other" flowing IP communication streams and change into the OTA state and let it go until the reboot is happening. Additionally i show the progress of the hexlines coming in on the screen which is taken from the OTA function, that you can see what is going on.
I think the progress from OTAend is not visable...not really sure if the "old code" is still there, in OTAend the new code is moved in....
Thank you
Torsten
Thanks to the discussions on this thread, I forked joepasquariello's branch and added a Class that manages an upload request from a client and performs the update. I've added the example .ino in the forked branch. It's available at https://github.com/ssaenger/FlasherX-Ethernet_Support.
I'm having an issue with a 720kb FW file. (note: FW update works fine when using a small file like blink)
Based on the messages I'm getting, I guess the buffer is not large enough.
It doesn't matter if I start with a small file, or my big firmware file, I always get the same 96K buffer size.
target = fw_teensy41 (8192K flash in 4K sectors)
created buffer = 96K FLASH (607E4000 - 607FC000)
reading hex lines...
abort - max address 60018010 too large
erase FLASH buffer / free RAM buffer...
The firmware file is stored on a separate flash memory chip, and I'm using LittleFS and MTP to load the file in memory for now.
Based on the following, is there anything I change in FlashTxx.h to allow flashing a bigger file?
"FlasherX buffers the new code in flash by default, and automatically defines the largest possible flash buffer, from the top of existing code to the bottom of FLASH_RESERVE (settable in FlashTxx.h). "
#elif defined(__IMXRT1062__) && defined(ARDUINO_TEENSY41)
#define FLASH_ID "fw_teensy41" // target ID (in code)
#define FLASH_SIZE (0x800000) // 8MB
#define FLASH_SECTOR_SIZE (0x1000) // 4KB sector size
#define FLASH_WRITE_SIZE (4) // 4-byte/32-bit writes
#define FLASH_RESERVE (4*FLASH_SECTOR_SIZE) // reserve top of flash
#define FLASH_BASE_ADDR (0x60000000) // code starts here
Based on the messages I'm getting, I guess the buffer is not large enough.
It doesn't matter if I start with a small file, or my big firmware file, I always get the same 96K buffer size.
target = fw_teensy41 (8192K flash in 4K sectors)
created buffer = 96K FLASH (607E4000 - 607FC000)
reading hex lines...
abort - max address 60018010 too large
erase FLASH buffer / free RAM buffer...
The firmware file is stored on a separate flash memory chip, and I'm using LittleFS and MTP to load the file in memory for now.
Based on the following, is there anything I change in FlashTxx.h to allow flashing a bigger file?
"FlasherX buffers the new code in flash by default, and automatically defines the largest possible flash buffer, from the top of existing code to the bottom of FLASH_RESERVE (settable in FlashTxx.h). "
#elif defined(__IMXRT1062__) && defined(ARDUINO_TEENSY41)
#define FLASH_ID "fw_teensy41" // target ID (in code)
#define FLASH_SIZE (0x800000) // 8MB
#define FLASH_SECTOR_SIZE (0x1000) // 4KB sector size
#define FLASH_WRITE_SIZE (4) // 4-byte/32-bit writes
#define FLASH_RESERVE (4*FLASH_SECTOR_SIZE) // reserve top of flash
#define FLASH_BASE_ADDR (0x60000000) // code starts here