Zmodem transfer from Teensy to PC

el_supremo

Well-known member
Zmodem file transfer from Teensy to PC

Last year I started trying to convert the rz and sz programs to allow zmodem transfers between Teensy (and Arduino) and a PC. I got the 'sz' code (from Teensy to PC) more or less working but then something distracted me and I only started looking at it again last week. The sz code now works but in a rather restricted way.
- It can send one file or the contents of an entire directory from a uSD card on the Teensy to the PC. It works when sending to TeraTerm but I can't get it to work with Hyperterminal. TeraTerm only works on COM1 2, 3 and 4.
- TeraTerm and Hyperterminal have the problem/feature that all directory paths MUST exist before transferring a file. E.g. with TeraTerm, if you have set its receive directory as C:\fred and send the file: /logs/2014/06/23/171002.csv then the path C:\fred\logs\2014\06\23\ must already exist.
- it will not work in a 'hostile' environment. Any timeouts or other transmission errors will cause it to fail.

In one test, it transferred 10 files with a total of 2853852 bytes in 266 secs at 115200baud. That's a throughput of about 107000baud which isn't too bad.
I've decided to post it as-is so that if someone finds it useful they can give it a try. Maybe someone with more patience than I have can improve it or even get rz to work as well :)

Pete
P.S. it uses about 6kB of sram so it isn't going to work on Teensy2 or 328-based Arduinos. I did have it running on a Teensy++2 and currently it runs on a Teensy3.
 

Attachments

  • zmodem_test_z3.zip
    37 KB · Views: 560
Last edited:
Last year I started trying to convert the rz and sz programs to allow zmodem transfers between Teensy (and Arduino) and a PC. I got the 'sz' code (from Teensy to PC) more or less working but then something distracted me and I only started looking at it again last week. The sz code now works but in a rather restricted way.

Thanks for your work.

I recall reading that hyperterminal doesn't do orthodox ZMODEM, but usually works because it can quietly fall back to YMODEM and XMODEM, which sz/rz will negotiate. I never tested that but maybe you did w/o knowing.

I've always been a ZMODEM fan so I have to try this!

I've got stock XMODEM sorta working (hostile environment).

If I succeed in improving anything, I'll post a patch.
 
I didn't do any more with it after I posted that zip file.
Let me know how it goes. Maybe I'll play with it some more.

Pete
 
I didn't do any more with it after I posted that zip file.
Let me know how it goes. Maybe I'll play with it some more.

Pete

I figured I would make this my first stop in announcing the sketch that I put together in order to pay the proper respect to Pete (el_supremo) for getting this started!

The sketch that I have built forms a rudimentary Arduino (and Teensy I suppose?) SDCard file manager via a DOS/shell like Serial interface. I completed the code for SZ including auto-start and crash recovery. Most importantly, I completed RZ which was the bulk of the remaining effort!

On my Arduino Mega 2560 R3, I can receive files on the Arduino at up to 57600 baud reliably; maybe you guys with those much faster Teensy boards can crank out more bps on receive, it's easy to configure and try.

I tried HyperTerminal, SyncTERM, and TeraTerm all successfully, send and receive. While SyncTERM is available for any platform you're likely to be using, I'm hopeful that this sketch will operate with other popular terminal applications on Mac and Linux platforms given that I got a perfect run on anything that still reasonably runs on Windows.

Video demonstration here:


Sketch available for download at GitHub here:

https://github.com/ecm-bitflipper/Arduino_ZModem

More information is available on the Wiki page of that GitHub repository.

I hope someone finds this sketch useful! Thanks again to Pete for seeding this idea!
 
I've just finished transferring a [EDIT]10MB WAV from Teensy3.0 to the PC at 115200 which took nearly 16 minutes for an average data throughput of 10700 cps.
If anyone else tries this a couple of bits of info.
- If you use TeraTerm and need a COM port higher than the default COM4, you can edit the teraterm.ini file and change the MaxComPort entry to whatever number is useful. You can change these other defaults too, as required:
ComPort=8
BaudRate=115200
- I edited zmodem_fixes.h to set ZSERIAL to SERIAL3 and have a USB/Serial adapter on pins 7 and 8 of the Teensy.


@bitflipper: Thanks very much for doing this.

Pete
 
Last edited:
I've just finished transferring a 1GB WAV from Teensy3.0 to the PC at 115200 which took nearly 16 minutes for an average data throughput of 10700 cps.
If anyone else tries this a couple of bits of info.
- If you use TeraTerm and need a COM port higher than the default COM4, you can edit the teraterm.ini file and change the MaxComPort entry to whatever number is useful. You can change these other defaults too, as required:
ComPort=8
BaudRate=115200
- I edited zmodem_fixes.h to set ZSERIAL to SERIAL3 and have a USB/Serial adapter on pins 7 and 8 of the Teensy.


@bitflipper: Thanks very much for doing this.

Pete

Hi Pete,

Thanks for trying my sketch! I was curious as to whether it would work right on the Teensy. If I'm interpreting your results correctly, the sketch is running faster with the optimizations that I made? It would be really cool to hear if you can achieve higher speeds (> 57600) than I did on the Arduino Mega 2560 R3 when sending a file to the Arduino. You'll know you've gone too high if either the transfer doesn't start at all, or TeraTerm seems to get into an infinite loop retrying from the same location over and over (SyncTERM gives up really fast whereas TeraTerm keeps trying and trying...)

After I posted the code to GitHub I did think that maybe I should have moved the definitions for ZSERIAL and DSERIAL over to zmodem_config.h rather than leaving them in zmodem_fixes.h - they're not really fixes per say, they really should be considered user configurable. The other thing that needs moving to zmodem_config.h is the SD_SEL pin definition for sd.open(). Next release. :)

Anyway, you're welcome, I hope you enjoy the sketch and find it useful.
 
The Teensy 3.2 can do 6 Megabit/s. But i'd choose Serial1 with this speed, because it has the largest FIFO.
 
Correction: that file wasn't 1GB. It was only 10MB! (I've corrected the original message)

It would be really cool to hear if you can achieve higher speeds
That test was at 115200 but TeraTerm won't go any higher. I can't get Hyperterminal to work at 115200.

Pete
 
Correction: that file wasn't 1GB. It was only 10MB! (I've corrected the original message)


That test was at 115200 but TeraTerm won't go any higher. I can't get Hyperterminal to work at 115200.

Pete

I was hoping you could try sending a file TO the Teensy at 115200. Your test shows the other direction - to the PC. The piece that remains outstanding is knowing if a faster board like the Teensy can keep up with the incoming file transfer or if it gets overwhelmed past 57600 like the Arduino Mega does.
 
bitflipper, may i ask why you need zmodem ?
And why not usb ?

The T3 is so fast, that *for sure* it can go really much faster than 115200.
 
Frank B,

I don't quite understand your question, maybe I'm missing something? The USB port, at least on the Arduino Mega, presents only a virtual serial port to the host PC, there is no USB other than that.

I wanted ZModem in order to ensure that I am sending/receiving files Arduino<->PC that are complete and error free. Without a protocol like ZModem to do error checking and correction, there is a risk of a buffer overrun which would lead to file corruption.

Unfortunately, the way that the protocol was specified and code originally written many years back, it would require significant re-work to avoid the byte-at-a-time read that it does from Serial. I think that's going to ultimately be the limiting factor for overall speed. I did what I could to peel away the layers of embedded function calls the code was doing just to read a byte, but because it watches each byte for a set of control/escape characters, there's not a lot else I'm seeing that can be done given the protocol specs.
 
Yes, the USB presents a virtual Serial port (on the Teensy it can do HID or other types, too). But the Teensy has inbuilt USB Hardware and does not have a Serial<->USB Converter like the MEGA. It does not need it. And it is much faster...
Then, USB is by design error free (uses CRC Checks as far as i know), so there will be no errors.
That's why i asked...

p.s. did you see the benchmark?
 
Last edited:
Ah, okay. I'm not familiar with the Teensy - I'm using an Arduino Mega because my project needs a boat load of digital pins. That's why ZModem makes sense in my case. Your question is better suited for Paul (el_supremo). I just stumbled on this forum from a web search and found this thread that Paul started. And so I came back here to thank Paul and show what became of the code he started.

Hand off! Paul - why did you bother with ZModem on a Teensy?

Hopefully there is in fact a use case on the Teensy platform because it would keep my questions about ZModem receive speed relevant. :)
 
Yes, if you use one of the three hardware-serials.

Though I suspect you would want to only use Serial1 or Serial2 on Teensy 3.2/3.1 (and only Serial1 on the Teensy 3.0). This is due to the 3.1/3.2 having 8 byte FIFO queues for Serial1/Serial2 that are implemented in hardware. If your serial line supports hardware flow control, then you want to only use Serial1, and setup the serial line to use the dedicated RTS/CTS signals.
 
Hand off! Paul - why did you bother with ZModem on a Teensy?
It's Pete.
I started working on it on an Arduino Nano but rapidly ran into problems with insufficient ram. So I decided that first I would get it working on a Teensy (Teensy++2) and then work backwards. Then other things got in the way and I never got back to it.

Pete
 
I see, makes some sense. Getting it under 3K I think can be done. 2K, that's a tough challenge. I used this cool online .elf analyzer (http://sparks.gogo.co.nz/avr-ram-use.html) that shows how much dynamic memory each global variable is consuming. Here's a partial output:

RAM: Global Variables/Objects
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1025 Bytes secbuf
1024 Bytes Txb
591 Bytes sd
257 Bytes Pathname
157 Bytes Serial
36 Bytes fout
36 Bytes SFEMP3Shield::track
33 Bytes Attn
17 Bytes zputhex(int)::digits
16 Bytes vtable for HardwareSerial
16 Bytes intFunc
8 Bytes vtable for SdFile
8 Bytes vtable for SdFat
6 Bytes __iob
4 Bytes vpos
..etc.. the rest is very minor.

secbuf and Txb might be able to be combined as it's quite possible (would have to study the code carefully) that they are each used exclusively for either RZ (secbuf) or TZ (Txb).

- Can't get rid of sd.
- Pathname could be trimmed down, saving around 200 bytes
- Can't get rid of Serial

Not much else is going to make a material difference.

After getting all the easily dealt with un-F()'d strings out of dynamic RAM (I could get about 64 more bytes with a bunch more work) I come up with this:

Sketch uses 42,788 bytes (16%) of program storage space. Maximum is 253,952 bytes.
Global variables use 3,458 bytes (42%) of dynamic memory, leaving 4,734 bytes for local variables. Maximum is 8,192 bytes.


So if I get rid of one 1024 buffer, 200 bytes from Pathname, and 64 bytes in strings, 36 bytes for not using a SparkFun MP3 shield, I could save another 1324 bytes. 3458 - 1324 = 2134. Close, but not close enough. It might work then with one of those weird boards that sports 2.5K of dynamic memory. But to get under 2K I think it would mean hacking back the 1K buffer.

Argh, why couldn't the Arduino designers give us 4K of memory on the smaller boards?!

Anyway, I should probably get out of your guys hair, this is a Teensy forum where you don't need to worry about these things!
 
Last edited:
I have a similar problem - teensy datalogger buried in an airframe but with a serial port on the outside. I'm recording to a micro-SD on the datalogger, but would like to be able to dump the record out the serial port between flights. Data is ASCII and it can be written reliably to a terminal at 115,200. I wrote a routine to copy incoming serial data (the file) to a Nano micro-SD, but best it can do with 99% accuracy is 9600 bps. I don't need every bit, but it would be good to have at least 95%.

Files get big so it would be better to run more than 9600 bps throughput.

I'm thinking of trying xmodem on both sides of this. It looks like it would fit on the nano but haven't tried yet. the nano doesn't have to do anything but trigger the transfer and write the transferred file to an SD.

does this sound plausible?

john
 
does this sound plausible?

john

Hi John,

In concept, I think you could use the ZModem sketch I built to transfer your files from your Arduino to your PC. If your log files grow over time (between transfers), you could even lessen the time it takes to transfer because of the Crash Recovery feature that I added to the sketch; it will continue transfers from where it left off last time it transferred the same file.

The trouble is the very small amount of memory available in the Nano. With only 32K of flash, you'd have to cut out the ZModem receive half of the code just to make it small enough to flash. That would be simple enough, but then there is the tougher problem of only 2K of dynamic memory available on the Nano. As I discussed above, I could get the usage down to just around 2K, but not safely enough for the sketch to run reliably without hacking down the transfer buffer which means modifying some pretty cryptic bits of code.

With the low price of the Teensy these days, I'd have to recommend switching out your Nano for one of those if that's possible. Then you'd have enough space to even blend your sketch with the ZModem sketch (or any other file transfer technique you want to use) and transfer files on demand!
 
I already have another teensy and had expected that it would perform better than the Nano in a number of ways. downside is I need to provide space for a buck regulator to get the 9 volt battery supply down to the less than 7 volts the teensy can handle. there is some tail wagging the dog to all of this, but system must include the 9 volt battery, the buck regulator, a small oled, a push-button, the SD card, the micro-controller, an on/off switch and fit in a box i can make on my small 3d printer.

but I guess I'm stuck. I'll keep the Nano Box and do another one, hopefully with fewer of the mistakes I made on the first one. None of this is meant to be critical of the teensy - it's wonderful. My guess is a regulator which could handle voltages up to say 14 volts would have taken up too much real estate.

FWIW, I got into this stuff in 1983 in Chicago, was a patron of CBBS, the bulletin board system run by Ward Christiansen and Randy Suess. Ward was the guy who brought cyclic redundancy check to modem transfer software. I think his work was the original Xmodem although Chuck Forsberg may have been the source of the name.

thanks for your observations,

john
 
Last edited:
You know what John, I just thought of a way I MIGHT be able to relatively easily build a Nano compatible version that has SZ (send-to-PC) only and RZ disabled, and with a trimmed down send buffer. The receive buffer is the one I most worried about slimming down, but if I'd be disabling receive anyway then it just might work. The way I'd do it is to put a configuration macro in zmodem_config.h that would just need to be uncommented to disable RZ and use a slimmer send buffer.

Question is, if I build it will you commit to testing it on your Nano? I can only easily test on a Mega which won't crap out if > 2K of memory is used.

In your application, are you okay to flash the entire ZModem sketch temporarily in order to get your files and then put your actual sketch back, or does your sketch need to be running even when you want to be pulling files? I want to make sure this is actually a practical approach for someone willing to test it out before I go ahead.
 
One more question, which version of the Nano are you using? I think there's only a real hope with the ATmega328 one with 32KB of flash and 2KB of RAM.

Specifications: ....
Flash Memory 16 KB (ATmega168) or 32 KB (ATmega328) of which 2 KB used by bootloader
SRAM 1 KB (ATmega168) or 2 KB (ATmega328)...
 
Hi BF,
I'm sending from the teensy's SD and receiving and writing to an SD on the Nano. let me check my nano for capacities. thx much. and yes, I will test it and try my best to make it work. but it would be over several weeks. possibly quicker, but i lead a complicated life and am not the only controller of my time (marriage it's called.)

my Nano (the reciever) is the 328 version - 32kb and the 2kb memory
 
Last edited:
John,

Now I'm not sure I've understood your desired functionality correctly. Is the remaining problem you are trying to solve only involving conveniently sending files from your Arduino Nano to some sort of PC or laptop, or do you need some sort of bi-directional data transfer problem solved? I just want to make sure we're talking about the same thing. I'm only hopeful I can solve the SZ (send files from Arduino to PC) problem when it comes to a small board like Nano or Uno.
 
Back
Top