MicroPython for Teensy 3.1

[dhylands]

I've got the initial port of MicroPython for the Teensy 3.1 working (just the REPL, no I/O - i.e. GPIO/ADC etc)

I made a fork of the micropython repository here:
https://github.com/dhylands/micropython

All of my changes are in the teensy directory, with the exception of an edit to py/misc.h which is required for main.cpp to be a C++ file.

I'm not sure I actually need main.cpp to be C++ anymore, but I'll wait until some I/O ha been added to verify if I can rename it to main.c.

Teensy device connected @/dev/ttyACM0 (serial 21973)
Micro Python for Teensy 3.1
Type "help()" for more information.
>>> x = 3
>>> y = 7
>>> x + y
10
>>> print("This is a test")
This is a test
>>> help()
NameError: name 'help' is not defined
>>>

Current compiled .elf size:

arm-none-eabi-size "micropython.elf"
   text	   data	    bss	    dec	    hex	filename
 129932	   1636	   2440	 134008	  20b78	micropython.elf

[stevech]

Looking forward to this mini-Python. Big deal.

Can you distribute a .hex file for the T3.1?

I recall you were looking at printf().
Here's a link to using the intrinsic printf() with T3, with user-defined stream redirection.
http://forum.pjrc.com/threads/24795-...ction-Teensy-3

[dhylands]

Hi Steve,

Looking forward to this mini-Python. Big deal.

Can you distribute a .hex file for the T3.1?

Sure. The initial version is here: http://www.davehylands.com/Misc/micropython.hex

You can't do much with this version (yet).

I recall you were looking at printf().
Here's a link to using the intrinsic printf() with T3, with user-defined stream redirection.
http://forum.pjrc.com/threads/24795-...ction-Teensy-3

I noticed that there was a version of printf included with the STM port of MicroPython (this is the one which will go with the MicroPython board).
So I decided to use that version. At least then the behaviour will be the same. This neatly sidesteps all of the libc issues.

[stevech]

rats. Teensy's loader says "unreadable". I re-downloaded the .hex again. Same.
last few records are as below.
loader's verbose log says
Open File event
ihex: parse error line 8195

... line 8195 to the end of the file, is:

:04FFFC009488FF1FC7
:020000022000DC
:100000009C88FF1F9C88FF1FA488FF1FA488FF1FD8
:10001000AC88FF1FAC88FF1FB488FF1FB488FF1F88
:10002000BC88FF1FBC88FF1FC488FF1FC488FF1F38
:10003000CC88FF1FCC88FF1FD488FF1FD488FF1FE8
:10004000DC88FF1FDC88FF1FE488FF1FE488FF1F98
:10005000EC88FF1FEC88FF1FF488FF1FF488FF1F48
:10006000FC88FF1FFC88FF1F0489FF1F0489FF1FF6
:100070000C89FF1F0C89FF1F1489FF1F1489FF1FA4
:100080001C89FF1F1C89FF1F2489FF1F2489FF1F54
:100090002C89FF1F2C89FF1F3489FF1F3489FF1F04
:1000A0003C89FF1F3C89FF1F4489FF1F4489FF1FB4
:1000B0004C89FF1F4C89FF1F5489FF1F5489FF1F64
:1000C0005C89FF1F5C89FF1F6489FF1F6489FF1F14
:1000D0006C89FF1F6C89FF1F7489FF1F7489FF1FC4
:1000E0007C89FF1F7C89FF1F8489FF1F8489FF1F74
:1000F0008C89FF1F8C89FF1F9489FF1F9489FF1F24
:100100009C89FF1F9C89FF1FA489FF1FA489FF1FD3
:10011000FFFFFFFF0000020035000000CEFBFFFFE5
:10012000CB030000010000000000000034000000CC
:10013000CEFBFFFFCB030000010000000000000029
:1001400040F801004153434949000000000000000D
:10015000000000000000000000000000000000009F
:100160000000000001000000415343494900000025
:10017000000000000000000000000000000000007F
:100180000000000000000000D8F901004CF7010059
:100190004CF701004CF701004CF701004CF701004F
:1001A0004CF701004CF701004CF701004CF701003F
:1001B000FFFFFFFFFFFFFFFFFFFFFFFFFFFF00004D
:0401C000819C01001D
:00000001FF

[PaulStoffregen]

Yup, it was set for 128k max. I'm testing a new version now.....

[PaulStoffregen]

Ok, I just posted 1.18-rc2 which should fix this bug.

[dhylands]

I can confirm that the loader 1.18rc2 works great with large hex files.

I've rewritten the MicroPython port to be purely in C. I created a new branch here:
https://github.com/dhylands/micropython/tree/teensy-3.1
and I've done a pull request on the official micropython port.

Incidently, the C only version is around 70K

Newly support features:

help() is now hooked up
pyb.led(1) turns the onboard LED on
pyb.led(0) turns the onboard LED off
pyb.Led(0) creates a Led object with on() and off() attributes. Currently only LED 0 (the onboard LED) is suppported.
pyb.gpio(pin) configures the pin as input and returns the current value.
pyb.gpio(pin, val) configures the pin as output and sets the value to val.
pyb.delay(msec)

So you can now:

>>> for i in range(10):
...     x=pyb.led(1)
...     pyb.delay(100)
...     x=pyb.led(0)
...     pyb.delay(500)
...

and the onboard LED will blink 10 times.

Now I need to figure out some type of on-board storage so a script can be stored and executed.

The STM version of MicroPython seems to be using an open source FAT file system.
We can probably spare 32K or so for a filesystem.

[dhylands]

Err we have 256K of flash and are currently using 70K.

So we can probably go upto 128K (although I'd rather start out smaller and leave room for more code features). 64K or 96K seems like a better size to start with.

[stevech]

I ran the work-in-progress MicroPython on a T3.1, downloaded > 128KB with the new loader.
Will try the newer smaller one, C-only.

This is an impressive beginning. It needs, of course, some means to import from SD card or LAN/SMB or other medium. If no ethernet, need some way to let PC write the SD card via USB - kind of like mbed does.

I'll be happy to help test. Thanks for the effort.

Please post a .hex - or is it easy for me (windows user) to convert something (.elf?) in the posting to .hex for the T3 uploader?

[PaulStoffregen]

Now I need to figure out some type of on-board storage so a script can be stored and executed.

What if example code to write to the unused flash memory existed?

Of course, no such example exists today.... but another project to build a Basic interpreter needs it, so this is on my tentative to-do list.

[dhylands]

What if example code to write to the unused flash memory existed?

Of course, no such example exists today.... but another project to build a Basic interpreter needs it, so this is on my tentative to-do list.

I think that any approach that has persistence would be good. Personally, I'm impartial to the actual implementation. I need to figure out what exactly the MicroPython board will be doing and then decide if I should go that route or go the more Teensy influenced route makes the most sense.

This is very early days, and Damien (the author of MicroPython) is busy gettiing stuff ready for the release and adding extra ports is just a distraction. I think that support for persistence is probably the single most important next thing to do. Being able to blink LEDs and control servos, etc is not particularly useful if you can't save your scipts.

So I'd love to get your thoughts on what APIs you might be looking to use, and possibly even help out on the implementation and/or testing. I'm a professional SW developer by trade, with a passion for robotics. I've got lots of experiencee working on embedded platforms and a bent towards linux.

[stevech]

My votes - first cut:

1. Wiznet 820 non-solder plugin. Use if/when needed
2. socket and select() libraries from Python
3. make select() work for UARTS so no special code is needed for non-blocking or blocking reads
4. watchdog timer restart for unattended operation
5. zip file file system as Python has, so pyc can be in one zip file sent to device, and seen as a file system, so no app source code on device. PC Python compiler->pyc's -> zip -> Teensy
6. python import statement can use pyc's in the one zip file
7. Python 2.7 and 3.x support byte arrays, as would microPython (instead of the kludge of using strings for byte arrays)
8. mechanism to state which py or pyc to run at startup
9. mechanism to say what to do when python app exits: reboot, restart, nothing

I'd prefer to see microPython on Teensy 3.1 so there's enough flash/RAM so that onerous compromises aren't needed. The price difference is nill.
Not to forget about RPi, Beagleboard - run LInux/Python without limits. Noted: some apps do need add-on cards, more cost. But these are generic - no matter Python, C++, Javascript being used. This is a veiled suggestion that a Python-centric board has to compete with the likes of RPi and equivalents.

[dhylands]

Just an FYI:

My changes have been merged into the official micropython repository

I will continue to work in my fork and submit PRs (pull requests) to the micropython project.

[PaulStoffregen]

I checked out the code, switch to the teensy3.1 branch and got it to compile to 70k. I connected with a terminal, and...

About to add functions()
Micro Python for Teensy 3.1
Type "help()" for more information.
>>> help()
Welcome to Micro Python!

This is a *very* early version of Micro Python and has minimal functionality.

Specific commands for the board:
    pyb.info()             -- print some general information
    pyb.gc()               -- run the garbage collector
    pyb.delay(<n>)         -- wait for n milliseconds
    pyb.Led(<n>)           -- create Led object for LED n (n=0)
                              Led methods: on(), off()
    pyb.gpio(<pin>)        -- read gpio pin
    pyb.gpio(<pin>, <val>) -- set gpio pin
>>>

Now if only I actually knew a little Python !!

So, for on chip non-volatile storage, there are 3 options. I'd like to help get at least one of these working.

#1: Use the remaining 186K of flash memory. This is a bit dangerous, but with some checks in the code, it ought to be able to avoid overwriting the 70K that's actually used. It also had a huge downside that the entire flash memory becomes unresponsive while writing. All interrupts must be disabled. The memory is writeable in 4 byte chunks and erasable in 2K blocks. Writing turns 1s into 0s, and erasing sets an entire block back to all 1s.

#2: Use the 2K EEPROM. This is the simplest way, but it's limited to only 2K. The code also already exists. Memory is writeable on 1, 2 or 4 byte sizes and can be rewritten easily.

#3: Use the 32K data flash. The 2K EEPROM actually is a hardware layer that writes to this 32K area, so it can't be used when EEPROM is configured. It works pretty much the same way as normal flash, writes are 32 bits and erasing is 1K blocks. The big upside to using the 32k data flash is it doesn't interfere with the normal flash memory (or risk overwriting micropython itself), so interrupts don't need to be disabled and normal can be run, as long it doesn't access the data flash memory.

EEPROM (#2) is probably the only option that can support interactive editing. Either type of flash probably needs to have the user compose their code in RAM, then "burn" it to the flash with a command. But what if they continue editing? Really, that's a rhetorical question.... my involvement will probably be limited to just writing up some example code that erases and writes. Let me know which of these schemes you'd like, and I'll try to assist on the C side. Who knows, maybe I'll even learn a tiny bit of Python somehow?!

[MichaelMeissner]

Well depending on your serial speed, how much data you are down loading, etc. it might be just simpler to use the old fashioned sneakernet approach, i.e. remove the SD card, move it to the PC with a card reader, transfer the files, and move it back to the Teensy.

[stevech]

Well depending on your serial speed, how much data you are down loading, etc. it might be just simpler to use the old fashioned sneakernet approach, i.e. remove the SD card, move it to the PC with a card reader, transfer the files, and move it back to the Teensy.

OMG no. <grin>.
I've done a large embedded system with Python, months of work. Target is an embedded ARM but has an small RTOS kernel. The method is to edit/compile with Eclipse on Windows/linux, yielding pyc files that are automatically put in a zip file. Then the zip file gets copied to a tiny file system like FAT32 on the embedded device. Then the bytecode interpreter is run. It uses the Python zip.py library, to make the pyc files inside the one zip file appear to be files in another file system ("drive"). This new "drive" is added to the search path for the Python import function.

This has worked well for developing on a PC, push the zip file to the target and run. In some kinds of apps or functions within an app, one can run/debug on the PC then target the embedded device.

sneaker-net SD card juggling would get too tiresome in about 10 minutes. Ideal situation is use SMB on ethernet to share py or pyc files with PC on which edit/compile is done. If ethernet isn't needed in final system, just unplug that LAN module.

[dhylands]

...snip...

Now if only I actually knew a little Python !!

So, for on chip non-volatile storage, there are 3 options. I'd like to help get at least one of these working.

#1: Use the remaining 186K of flash memory. This is a bit dangerous, but with some checks in the code, it ought to be able to avoid overwriting the 70K that's actually used. It also had a huge downside that the entire flash memory becomes unresponsive while writing. All interrupts must be disabled. The memory is writeable in 4 byte chunks and erasable in 2K blocks. Writing turns 1s into 0s, and erasing sets an entire block back to all 1s.

#2: Use the 2K EEPROM. This is the simplest way, but it's limited to only 2K. The code also already exists. Memory is writeable on 1, 2 or 4 byte sizes and can be rewritten easily.

#3: Use the 32K data flash. The 2K EEPROM actually is a hardware layer that writes to this 32K area, so it can't be used when EEPROM is configured. It works pretty much the same way as normal flash, writes are 32 bits and erasing is 1K blocks. The big upside to using the 32k data flash is it doesn't interfere with the normal flash memory (or risk overwriting micropython itself), so interrupts don't need to be disabled and normal can be run, as long it doesn't access the data flash memory.

EEPROM (#2) is probably the only option that can support interactive editing. Either type of flash probably needs to have the user compose their code in RAM, then "burn" it to the flash with a command. But what if they continue editing? Really, that's a rhetorical question.... my involvement will probably be limited to just writing up some example code that erases and writes. Let me know which of these schemes you'd like, and I'll try to assist on the C side. Who knows, maybe I'll even learn a tiny bit of Python somehow?!

And what I'm considering is option number #4. I've used this method on previous projects and basically you create a super-simple read-only file-system. You set it up to that it goes at the high end of flash. You use teensy loader to write it (this assumes that I can write to arbitray locations without erasing all of flash).

The file system is basically flat (i.e. no directories). You have an array of directory entries which each contains a filename, offset, and length to get to the data. At the very end you put a simple header which has some signature bytes, and an offset to get to the directory header.

With NOR flash, you can access the memory directly, no need to even read into RAM.

Very simple to implement, read-only file system. Not quite as convenient as the writable version, but its something, and because its so simple, you can make the files be contiguous in memory. With any block based file system, you'd need to read stuff into RAM

Actually, I don't see any reason why multiple approaches couldn't be used. Have a read-only filesystem, and also have a writable filesystem. It really all boils down to implementing something simple for "mount-points" i.e. have an array of mount-points. Each mount point has a name, perhaps some information about where its located, and some function pointers to the actual code which implements that particular file system.

Is wear-leveling a concern for the flash? What about bad-block handling?

I guess that you could do a wear-leveling layer which maps logical blocks to physical ones and then even use FAT on top of that (just thinking out loud here).

Does the onboard flash have any "extra" bytes per block? (maybe this is only for NAND - its been a while since I was messing around with this stuff).

[stevech]

With a read-only file system, what does a Python program do that needs to write a small amount of non-volatile data (a frequent need)?
SD card?
T3's EEPROM (is use of that available via Teensyduino?

wear leveling: I'd say no. Despite lots of re-flashing due to flash-breakpoints (J-Link on ARM), I never had an issue. I reasoned that if/when I do, I just toss the (development) board. Fielded units: get reflashed remotely but not often.

No bad-block handling that I've ever seen.
My loader did do one semi-trick: If any new block matches the existing block (flash sector) the erase/rewrite is skipped. I've found that often this saves a lot of needless block erases.

[MichaelMeissner]

As we used to say in the 1970's, don't under-estimate the bandwidth of a station wagon filled with magnetic tapes (presumably now it is of a jumbo jet filled with blu-ray disks).

You might look at EyeFi and Transcend, which are wifi cards that look like SD cards to a camera. Perhaps they will work in a Teensy, perhaps not.

In terms of SMB, and so forth, I tend to think that by the time you get that far up the network stack, that it is better to go with something like Linux, rather than trying to import and shoe-horn code written for a time-sharing threaded system to a system without the OS infrastructure. Besides, if you go with a Linux box, somebody else has written the code. But if you have the time and energy, go for it.

[PaulStoffregen]

You set it up to that it goes at the high end of flash. You use teensy loader to write it (this assumes that I can write to arbitray locations without erasing all of flash).

Unfortunately no, this is impossible. The bootloader implements a security measure where the entire flash is erased upon the first write. You can't add more data or make any changes to the flash. You'd have to rewrite the entire memory, even if you want to change just 1 byte.

However, Teensy Loader is pretty fast, so maybe rewriting the entire flash is an option?

Is wear-leveling a concern for the flash?

Yes, it most certainly is, if you're going to be writing frequently. The recommended write endurance is 10k cycles, so if you're going to write data regularly, some form of wear leveling is needed.

The 2K EEPROM is really just the 32K data flash memory, with a wear leveling algorithm implemented in the chip's hardware. That's why you can have 2K of EEPROM, or 32K of data flash, but not both.

If RAM is used for interactive editing and the user's "finished" Python program is burned to flash, the 32K data flash might work very nicely.

What about bad-block handling?

No. Well, I suppose you could go to a lot of trouble to build bad block detection and remapping. But if you're written the flash so much with a wear leveling algorithm that effectively spreads the wear, then by the time you're detecting bad blocks there's probably not much point.

The on-chip flash is definitely not like high density NAND chips where you need to worry about bad blocks. It's fabricated in a 90nm process.

I guess that you could do a wear-leveling layer which maps logical blocks to physical ones and then even use FAT on top of that (just thinking out loud here).

This sound really complicated. But then a bytecode interpreter isn't exactly simple either. Still, I have to wonder if this is really worthwhile?

Perhaps a simpler mechanism that lets the user just store one Python program onto the non-volatile memory might be a good first step? Programming hours can always go into extending the feature set at some future point. Seems like so many hours would be better spent mapping the many APIs for all the chip's functionality and maybe even some of the commonly used Arduino libraries onto Python libraries.

Does the onboard flash have any "extra" bytes per block? (maybe this is only for NAND - its been a while since I was messing around with this stuff).

No. Well, I believe the flash controller does have some ECC bits internally, but they're not made available.

There's also some other misc flash stuff, which is separate from the 256K of program flash and 32K of data flash. There's a config register which determines how the 32K of data flash works. There's also 64 bytes of "write once" memory. The last 8 of those bytes are initialized by PJRC with a serial number and ethernet mac address. The other 56 are blank.

[dhylands]

ok - here's how I think I'll approach it.

1 - Append the python source script to the micropython.hex file and burn it along with micropython.
2 - Add support for zip files and allow the zip file to be appended to the micropython .hex. You could probably just use the zip as a storage container, storing uncompressed data (which allows for direct access ala XIP). In fact this would allow XIP for imported modules written in C, as long as they were compiled using PIC.
3 - Possibly allow precompiled bytecode files to be substituted for source files.

After this point, you really need a way to transfer files in and out of whatever storage method you choose to use.

My first choice hails back to the days of dialup - use zmodem. Most serial terminals still support this, and it works over any serial-like interface (usb serial, UART serial, network sockets).

Another approach would do do something using USB HID, but that has the disadvantage of not working over UART or network.

3 - Add support for EEPROM/dataflash - storing plain text and/or bytecodes.
4 - Add support for EEPROM/dataflash - storing a zip file. This should approx double the apparent capacity.
5 - Add support for FatFs stored in dataflash.
6 - Add support for FatFs stored in regular flash.

Of course, this is just a plan, so its subject to change at any time

[PaulStoffregen]

Sounds like a good plan.

Maybe a script could be made to compile the Python source to bytecodes and append to the already-built .hex file? The script could run the hardware/tools/teensy_post_compile program after producing the new .hex, which causes Teensy Loader to automatically open it, and then hardware/tools/teensy_reboot to reboot the board. That's how Arduino does things when you click the Upload button.

[stevech]

I'd not want py source on the embedded device. Instead, pyc produced on a PC. Then placed in a zip file which, on the teensy, becomes a virtual read-only mounted file system, for Python imports.

I've never understood how the pyc can be/is totally standardized for machine type and python version numbers, so it can cross-compile with assurance that, say, an interpreter on the Teensy can use the same pyc files as an interpreter on a a PC (Windows, MAC, etc.). I'll guess it has to do with the python release numbers (of which there are too many!).

[dhylands]

I do know that for MicroPython, the byte-codes are NOT compatible with the regular full python.

This has to do with wanting to minimze heap allocations. See dpgeorge's comment here (2nd to the last comment when I wrote this email).

Obviously, using bytecode would be preferred to using source code, but I think that using source code is easier to start with.

There is a unix port of MicroPython already, so creating the tool which emits appropriate bytecodes and Thumb assembler shouldn't be too difficult (IIRC MicroPython allows inline assembler to be used).

[dhylands]

Sounds like a good plan.

Maybe a script could be made to compile the Python source to bytecodes and append to the already-built .hex file? The script could run the hardware/tools/teensy_post_compile program after producing the new .hex, which causes Teensy Loader to automatically open it, and then hardware/tools/teensy_reboot to reboot the board. That's how Arduino does things when you click the Upload button.

So, if you go into micropython/teensy and do:

make upload

then it already calls teensy_post_compile and teensy_reboot

However, I'd much rather use a purely command line tool like load_linux_only.c

The only thing missing from load_linux_only.c is that it doesn't reboot the teensy. How difficult is it to add that functionality?