View Full Version : Teensy3 PROGMEM compatibilty issues

08-09-2013, 12:03 AM
It really makes we want to leave AVR behind for micro-controllers with better architectures....
Like ARM and PIC32.

Currently the Teensyduino for Teensy3 has some AVR progmem compatibility macros
in hardware/teensy/cores/teensy3/avr/pgmspace.h
that allow compiling AVR code that used PROGMEM on teensy3
Depending on how people declared their data, it may end up putting the data in RAM
vs in the flash.

People are sloppy about whether they use "const" on progmem data/variables
because it didn't' matter on AVR since the AVR pgmspace.h used PROGMEM to put the data
into a section that got linked into flash.
The issue with Teensy3 is that the compatibility file avr/pgmspace.h
file changes PROGMEM to nothing.
While that makes the code compile, there is an issue with that.
When users don't explicitly use "const" on their PROGMEM data/variables, the variable data
ends up in RAM vs flash because initialized variables end up in
section .data which ends up being linked into region 'RAM'.

I ran into this issue on the openGLCD code as I was adding teensy3 support.
It was not immediately obvious since there is so much more RAM on Teensy3 than the AVR chips,
but eventually if there is enough progmem data, you get a RAM overflow error from the linker.

Of course I can fix my library code,
but I've seen lots of user code floating around out there that doesn't use const when using PROGMEM.
If you look at the Arduino progmem page:
it uses lots of prog_xxx typedefs but none of the examples use const, so arduino progmem users
will not use const, if they go by those examples.
This will cause all their PROGMEM data to end up in RAM instead of flash
when the code is compiled for Teensy3.

The fonts and bitmaps shipped with the openGLCD library can be fixed but it's
more difficult to get all the other generation tools fixed.
Because of this, users may have applications that work on AVR
that won't work on the ARM because the RAM gets blown out.

I think this issue will need to be resolved anyway since it isn't an issue just related
to the openGLCD library. It is a generic Teensy3 PROGMEM issue.
The real solution is make PROGMEM on teensy3 work the way it worked on AVR.
(put the variable data into a section that gets linked into flash)

For the time being I am using this definition for PROGMEM:
#define PROGMEM __attribute__((section(".rodata")))

I'm not sure what the ARM linker script does but from reading about sections for the ARM,
.rodata seems to be the section name for read only data.
The above definition seems to put the data into the flash when const is not used.

however, it does create a warning:
420: Warning: setting incorrect section attributes for .rodata

So it isn't really a proper solution.
I haven't dug any deeper to find a proper solution.
I'm sure it is trivial, I just don't know much about gcc sections or linker scripts.

--- bill

08-09-2013, 01:37 AM
This is the sort of change that makes me nervous. It certainly needs to be very thoroughly tested.

Harmless warnings during compile aren't good. They really distract people from important messages. There's probably some way to do this without generating warnings.

Right now, I just can't dedicate time to this. Later this year, I'll be getting back to porting and documenting libraries. I've been building up a very long list of newer libraries. Most are already ported, many scattered across thread on this forum, others by their authors or some by users.

Earlier this year I did quite a lot of work on an automated regression testing system. It's still not very mature. But it already tests some of the libraries, like OneWire. I do plan to bring lots of the libraries into it. This sort of change to PROGMEM is exactly the type of thing automated regression testing would be great to verify.

Please do not delay openGLCD release for this. It's definitely not going to happen quickly. In the sort term, let's try to update everything that reasonably can be updated to use "const". Newer versions of avr-libc require const, and other non-AVR platforms will also need it, so there's a pretty compelling advantage to updating old code to const, even without Teensy3.

Also, I opened an issue on Arduino's github, requesting that old PROGMEM page be updated.

08-09-2013, 03:33 AM
I agree about the warnings, particularly with the latest compiler options in 1.0.5 arduino IDE since
it kicks out the full include stack so even a small little warning ends up kicking up a full page of red text.

I've been pulling my hair out with various AVR PROGMEM issues on the library ever since I started working on it.
I hate that stuff because it is so fragile particularly with C++ since it never really worked.

Changing the PROGMEM define makes me nervous too, but I think it is just matter of understanding
the default linker script and what section names and attributes it uses.
I do think the final solution will involve having to define PROGMEM as some kind of section/attribute
rather than just NOPing it out.

I would assume that testing this should be fairly easy since after the link
either the data is only in the flash or you can see it taking up RAM space.

Eventually I'll get back to this and figure out the real solution.

For now, I'm a bit torn on what to do.
I might include my "hack" for Teensy3 that redefines PROGMEM for openGLCD sketches
because without it, a few openGLCD demo sketches won't build because the RAM is blown out.
I could fix the data declarations for the fonts and bitmaps that I ship, but
I don't have control over some of the font and bitmap header file generation tools out
there that people are using.

I'll mull it around and figure some thing out.

--- bill

08-09-2013, 04:00 AM
Note, another issue that I've read about is starting with GCC 4.5, they have changed the implementation of PROGMEM to use the named address support I wrote for a different processor (Cell/SPU), and there are some syntax changes that were needed due to the way named addresses are specified. Named address come from the ISO technical report ISO/IEC TR 18037, which by the way also defines fixed point arithmetic that came up recently.

Another issue seems to be the old progmem used attributes on type definitions, and that just does not work with the new compilers (you can only have attributes on real variables).

Most Arduino users won't see the changes for the time being, since the Arduino project is sticking with the older compiler, but some people are moving to the new compiler, and so you may have some code targeted to the new compiler, and a lot of code targeted to the old.

08-09-2013, 11:28 AM
Bill, please be sure to use const on everything that ships with the library. Even though Arduino ships avr-gcc 4.3.2, people who install Arduino through a linux distro tend to get a recent compiler which requires const. So even without Teensy3, you'll start seeing users who will have trouble if const is missing.

08-09-2013, 06:09 PM
I can already tell that I'm going to have to dig down into this and fully understand
how all this attribute and section stuff really works, in C vs C++, where it is broken, and how it has changed between
different versions of the AVR tools.

There are so many HACKs out there that Arduino people are using to try to avoid C++ warnings by
redefining PROGMEM.
For example, the one that Oliver uses in his m2tklib breaks the GLCD lib if a user
tries to include some other font.
And there are many other ones out there that people are using with similar type issues which is why
I've come to hate it so much.

The Arduino IDE in the linux Mint repo is 1.0 which uses the avr-gcc and bintools from the repo.
The current linuxmint13 avr-gcc in the repo is avr-gcc 4.5.3
So if using the repo Arduino IDE you get IDE 1.0 and avr-gcc 4.5.3

However, starting with linux IDE version 1.0.1 the IDE now includes its own compiler and all the bintools.
When you install (post 1.0) linux IDE from the Arduino site it includes the compiler and bin tools.
Linux Arduino IDE 1.0.5 includes avr-gcc 4.3.2
So unless you go out of your way to delete the compiler tools from the installation tree,
you get the compiler that ships with the IDE (4.3.2) and not the one that is installed on your machine from your repo.

All that said, I just went and built some sketches using IDE 1.0 and avr-gcc 4.5.3 and IDE 1.0.5 also patched to use avr-gcc 4.5.3
and I see no progmem issues when not using const.

So from what I can tell "const" being there/not-there isn't a compile issue.
(code still compiles and works)

The 4.3.2 avr-gcc that ship with the recent IDEs define PROGMEM as:


The 4.5.3 in the linux MINT repo also defines PROGMEM as:


with the default PROGMEM definition you do get this warning when you use it in C++ code

warning: only initialized variables can be placed into program memory area

That warning will be there regardless of whether you use const or not and
occurs in both avg-gcc 4.3.2 and 4.5.3
(It shows up twice in 4.5.3 for each use)
Even placing the declartions of the PROGMEM data inside a "C" external block
will not eliminate the warning.

However, if you do some HACK like this to modify the definition of PROGMEM

#ifdef PROGMEM
#undef PROGMEM
#define PROGMEM __attribute__((section(".progmem.data")))

The warnings are eliminated, however under certain (not typical) situations you can get a link error
which is what can happen when m2tklib is used on top of GLCD.

If anybody has some further reading or some magic for how to really resolve this, I'd love
to see it.
Eventually, I will figure out something that "just works" for all situations.

--- bill

08-09-2013, 09:51 PM
So from what I can tell "const" being there/not-there isn't a compile issue.
(code still compiles and works)

I don't know exactly which version of the toolchain "const" becomes an issue, but it definitely is with some newer versions.

09-12-2013, 02:23 AM
ok, so i've been digging around quite a bit on this progmem & const issue.
Here is some additional information:

Being an old-time C guy I didn't realize the difference between "const" in C and in C++.
In C++ the const declaration with an initializer will not create a global. In C it will.
C++ requires "extern" even on the declaration to get a global.

With respect to all the AVR PROGMEM stuff there are many issues, bugs, and subtlties depending
on how the declaration is done and whether C or C++ is being used and even which version
of the compiler you may be using.

Some of the progmem issues in the GLCD library were due to my lack of understanding of
C++ const declarations at the time and also at the time the fonts and bitmaps were
being shared between C++ and C code.

Long story short for a variety of reasons, "static" was added to all the font and bitmap
declarations in the GLCD library to prevent certain use cases for creating a link error
with duplicate data. The choice instead was made to just duplicate the flash usage instead
when those edge case usages occurred.

So here is how "static", "const" and PROGMEM and the various progmem workarounds
all interact with each other:

If you use "static" on your declaration of your data in Arduino/C++ files you will get a warning like
'xxx" defined but not used
if the var isn't used
If you add "const" the warning will go away.

"const" without "static" in C++ behaves like static in C.
i.e. you get a local copy of the data and if multiple compilation units
do this with the same variable, you end up with multiple copies of the same data
in the linked image.

If you use the default PROGMEM definition in AVR, then you get a warning for every single
declaration when using C++ (which is all arduino sketches)
but you can use or not use const on the PROGMEM declarations.

If you redefine the AVR PROGMEM definition to be something like:
#define PROGMEM __attribute__((section(".progmem.data")))
which eliminates the C++ progmem warnings,
then all PROGMEM variables must either use const or none of them can use const. Since mixing these in same section
cause you to get a linker error
i.e. if some PROGMEM vars are const and some aren't then it won't link.
the dreaded: "varname causes a section type conflict"

On ARM if PROGMEM is defined to be nothing for compatibility,
then things compile and there is no link error but if const was not
used then vars end up in RAM.

On ARM if you override the PROGMEM definition with some thing like:
define PROGMEM __attribute__((section(".rodata")))
It forces the variable's data into FLASH regardless of whether const was used.
If const was not used, then there will be single link warning about
a section type override/mismatch

__________________________________________________ _________________________
So here is the boiled down issues:
On AVR people were sloppy and inconsistent with their PROGMEM declarations because
pretty much any way you used PROGMEM pretty much worked.
const or not, it still worked.

C++ gives a warning for every single declaration that uses PROGMEM regardless of whether
const is used. (I think this is fixed in very new avr g++ but it isn't what is shipped with Arduino)

If the default AVR PROGMEM definition is overridden to eliminate the warnings, then all the code compiles
but may not link depending on how people declared their progmem variables.
The user will see section type conflict errors.

On non AVR processors that created a PROGMEM compatibility by simply turning "PROGMEM" to nothing,
AVR code that did not use const on its PROGMEM declarations, will end up with the data in RAM.

On ARM if you override the PROGMEM definition to force the data into a read-only section,
it works as long as all the variables use const or none of them use const.
(Can't mix const and non const in the same section)
If const is not used when forcing the data into a read only section,
then there will be a single warning at link time (with no indication as to what caused it or where in the code)
that that there is a section type mismatch if PROGMEM is ever used on a declaration without "const".

__________________________________________________ _________________________
Summary is that there is no PROGMEM compatibility solution for ARM that works
without a warning for all the crazy ways people used PROGMEM on AVR.
However, I believe that it is worth considering using a PROGMEM definition that forces all the
data initialized with PROGMEM to be in a read only section over a null definition of PROGMEM.

Here is my reasoning:
- Forcing the data to a read only section makes it behave much closer to the way it worked on AVR.
- If they misuse/abuse PROGMEM by mixing const and non const data, they will get an error
and will be forced to fix the code rather than silently gobbling up RAM.
- in the big picture, I think PROGMEM compatibility should be, well, compatible with existing code.

Here is the approach I'm taking in openGLCD.
Fonts and Bitmaps will have a new way of declaring and initializing data that works
across all platforms.
It will work as it "should". in that all data will be declared const.
On AVR, a PROGMEM attribute will be used and on non AVR this will not be used.
However, I will not be using the standard AVR PROGMEM in the initializations.
openGLCD will have its own attribute that is hidden and wrapped inside a simple font and bitmap declaration macro to hide this nastiness.
This will put all the openGLCD data into its own section on AVR to avoid any potential
const issues with other data from other libraries or other parts of the user sketch code.
i.e. I don't want anything in openGLCD to break because of code somewhere else.

For compatibility with code that does not ship with openGLCD, things start
to get complicated, just like for ARM, because I really want to create as much
backward compatibility as possible for users that are transitioning existing GLCD code to
openGLCD running on Teensy3.

Right now I'm not seeing a way to avoid having to override the Teensy3 PROGMEM definition
for a definition that forces the initialized data to read only section.

After I get all the font and bitmap data converted over, I'll come back with some further comments
on PROGMEM compatibility on Teensy3.

--- bill

09-12-2013, 03:43 AM
If you use "static" on your declaration of your data in Arduino/C++ files you will get a warning like
'xxx" defined but not used
if the var isn't used
If you add "const" the warning will go away.

You can also add __attribute__((__unused__)) which should silence the warning.

09-12-2013, 06:54 AM
I agree, PROGMEM is a mess.

I've been crazy busy the last few days, unable to get any development time in. Just started to catch up again now. I dug out the GLCD test board and I'm going to solder a display to it tonight and give this a try.....

09-12-2013, 08:37 AM
Thanks for the tip. That may come in handy.

The resistors work fine in place of the level shifter.
Make sure to use the pre-packaged download image on the openGLCD downloads page vs
the download link on the "Overview page".
As of right now, v0.6a is the latest.
I've got a big update that I'll be pushing to the repo either tonight or tomorrow with a matching
zip image for the downloads page.
It changes the way fonts, bitmaps and progmem are handled as mentioned above.
It is much cleaner and simpler for the user as now they will no longer have to ever mess with using PROGMEM
(even on AVR) when declaring their own font and bitmap data.
They also don't ever have to include font or bitmap header files in their sketches.

--- bill