Teensy 4.1 : dynamic load of code (EXE files for my FW)

[tjaekel]

An invitation to brainstorm, implement ... - or if you have already an implementation...

Goal
I want to run my FW. It should be possible to load code, during runtime and let it be executed, without to touch the running system.
Load different code files, without to flash, without to reset MCU - like an EXE file or a "plug-in".

Use Case
I have a GPIO interrupt. The actions to do (e.g. a SPI transfer, check the data, do some stuff), should be "programmable".
I do not want to modify the entire project, instead: I want to load a piece of code, generated (separately) and loaded when my MCU runs my FW already.
The system code is extended during runtime (without to recompile, flash, restart ... the entire MCU and FW, extend with this "plug-in").
It like to "load and run an EXE file".

Usage
My users run the board with my FW. They can write and generate a "plug in" (exe FILE), load it to the running system (e.g. via network, from SD card).
It extends the functionality without to flash, without to reset the running MCU. And they can change the "plug-in" on the fly, load another one...

Ideas
Most obvious: I reserve space in ITCM (or on FlashROM) where I can load code later (or flash the reserved FlashROM sectors).
Q: how to reserve a fix, known region (it should have fix, constant start and end address) in ITCM (or FlashROM, ITCM is preferred due to speed).

My users should use Arduino IDE with my FW LIB files (to resolve and use existing functions, e.g. UART print, SPI transaction).
They just create a "sketch" for "their user code" on "my sketch".
This user code can call existing functions in my MCU FW.
(OK, just provide a linker reference file to resolve externals).

There is this "SVC n" assembly function (Software Vector Call, Supervisor Call).
If I would use this - the generated code is "location independent" (relocatible) - cool: I do not need to tell where all the functions are. The reloaded code would do
SVC 3, e.g. for "UART_print()" and all fine.
Q: but as I understand: SVC is like an Interrupt: how can I call other functions which need also INTs, but not "blocked" due to SVC call?
SVC would be like an interrupt, other interrupts blocked.
And biggest problem: how to return from SVC to caller like a "synchronous function call" (with a return value)?

OK, I could implement a "BIOS vector table":
A linker file provides which address "to call" for a specific function (like a vector table). Just tell the user code where "the BIOS vector table" is and call
now the "BIOS vectors" (easier, just a bit more space and not really "relocatible", e.g. I move the vector table around).

So, my users can now use Arduino IDE, with my "FW LIB file" (resolving external calls), edit code, compile and create an EXEcutable.
(and just this user code, no insights to my MCU code, much shorter code and faster compile time).
The MCU FW running can load such EXE-file, activate and let it be executed (e.g. it replaces my GPIO INT handler code).

The main questions:

• how to reserve well known, fix space on ITCM?
• How to reserve fix, well known, space in FlashROM?
• Would you use SVC or a "BIOS Vector Table"?

OK, it needs to modify the linker script (to reserve the space).
Assume, I want to "load this code" on ITCM (and space is reserved) - can I read as data from a file (SD card) and load/write to ITCM?
Can I do a data write on ITCM code memory?

As I understand: all the code sits in an (external) flash memory. It is loaded during boot time into internal ITCM RAM (with attributes from compile time).
When I flash a sector in (external) flash memory - I assume it would be executed from there (slow). How to force to copy this code
into internal ITCM RAM, for faster execution - without to reboot?
Is there a helper function to load code from external ROM Flash to ITCM during runtime?
(actually nonsense: space must be available in ITCM for it, load directly, e.g. from SD card to ITCM)

Let's see.
Just to share idea, invite you for this project, brainstorm with you... It should be possible (I am sure), just to cope with the "devils details and traps". ;-)

 

[shawn]

Does a scripting language count? Lua works just fine on the Teensy, including file access, script storage, etc., plus some Teensy-specific functions.

 

[tjaekel]

Thank you. Pointing to Lua was great. I appreeciate.
I have checked a bit the Lua code - looks promising and I might try.

FYI,
I run a C-code interpreter ("Pico-C", similar to Lua, using hashing, like a JIT compiler).
The problem is:
I need the EXTMEM (external QSPI RAM), it is pretty RAM hungry for large scripts and not enough internal SRAM in MCU left.
It works.
But it is slow due to EXTMEM (4-bit lane access).
Therefore, the idea was to load the code into internal ITCM (or for scripts on DTCM), using "plug in" in order to "program" INT handlers.

Let's see.
I might give Lua a try (and compare with "Pico-C"), but when also EXTMEM needed - potentially also too slow.

Thank you for the great hint.

 

[PaulStoffregen]

Let's see.
Just to share idea, invite you for this project, brainstorm with you...

This idea has been discussed at least few times previously on this forum. Some of those threads had a *lot* of details. Maybe with some searching you might find those lengthy messages about the messy linker script issues and other thorny details.

As far as I know, nobody has gone to the considerable effort to actually make this work. All prior conversations have ultimately ended with the same conclusion, that it would have so many limitations that the huge effort just wouldn't be worthwhile. But who knows, maybe you'll feel differently and put in the work? If you do, I really hope you'll share, because this question does seem to come up once or twice a year. Would be wonderful if anyone actually got it working and shared the code, even if it's usefulness is limited.

Can I do a data write on ITCM code memory?
...
It should be possible (I am sure), just to cope with the "devils details and traps". ;-)

To answer this specific question, by default the MPU is configured to disallow writing to ITCM and to disallow executing code from DTCM and other regions used for data, as a proactive security measure which (hopefully) makes exploiting buffer overflow bugs more difficult.

ITCM is indeed writable at runtime if the MPU doesn't restrict your access. To play with MPU settings, look for configure_cache() in startup.c.

https://github.com/PaulStoffregen/c...ed00fe8b7e8adf0ea27bd2/teensy4/startup.c#L280

 

[jmarsh]

To answer this specific question, by default the MPU is configured to disallow writing to ITCM and to disallow executing code from DTCM and other regions used for data, as a proactive security measure which (hopefully) makes exploiting buffer overflow bugs more difficult.

I don't believe this is the case for ITCM, its region in the MPU is marked READWRITE and is writable by default.