Flash strings and RAM

Quick question about the use of flash strings (F("This is a flash string")) and RAM usage. When outputting these strings via serial.print or ethernet client.print, is the whole string copied to RAM? I'm asking this because I have a large header string (1,2kB) for a CSV file which I have to send through an Ethernet connection and serial output for debugging purposes. I'm a bit tight on my T3 RAM and want to know if I will have to switch to a T3.1 to avoid running out of free RAM. Thx!

On the ARM platforms, the F("...") does not involve actual code. The string automatically goes into read-only memory (flash memory). The ARM is a so-called Von-Nuemann machine where the program code space and the data space are in the same address space. The AVR's are so-called Harvard machines where the program code space is a different address space from the data space. On those machines, F("...") puts the string into program space, and it passes a special pointer type to the print function. I don't know what the implementation of the print function taking the special pointer type on AVR to say whether it copies the whole string into data, or does a byte by byte conversion.

If your code is ARM only, you don't need to use F("...") at all.

The actual macro in arduino will copy it character by character, so it will not use up memory the size of the flash string (which defeats the purpose). I'm not sure if it works the same way in teensy 3 though, since F macro is just a wrapper used for compatibility in teensy, as teensy will store constant strings in flash. Sorry this does not really answer your question, but I think it is a very good question, and if it turns out teensy uses the actual full size of the flash string in ram, then this is a design problem, as it defeats the purpose of conserving ram.

No doughboy, due to the combined address space of the ARM, the compiler/linker will put strings and const data into flash memory, and pass a normal pointer. Anything using the pointer just does normal pointer indirection. No copy of the string is made. No special pointer that involves doing special instructions needs to be done. The F("") macro is just there to help people port code from the AVR. It is not needed on the ARM. Here is what the F() macro looks like on the Teensy 3.0/3.1/LC:


In the print class you have:


As far as I can tell the class __FlashSTringHelper is never defined. Pointers to it are created, and then when used get converted back to const char *.

This is a bit redundant with the prior post, but hopefully worded for the newbie.
In some (most?) compilers, there is a default for processors like ARM Cortex where constants and C strings are stored in flash and not copied to RAM during initialization. In processors like the AVR that have dual address spaces, the default is usually to copy constants/strings to RAM (very wasteful), because special memory access functions are needed to read data from flash on the AVR. Not so on the ARM as it has a single address space containing flash for code and constants, RAM, I/O, etc.

Perhaps I should rephrase my question: (on ARM) are strings that are stored in flash (which I already knew) copied to RAM when calling the serial.print() or client.print() macro? Or do the macros just read it from the flash memory? I would suspect the latter of course, but I'd rather be sure.

They read it from flash memory.

Epyon said:

Perhaps I should rephrase my question: (on ARM) are strings that are stored in flash (which I already knew) copied to RAM when calling the serial.print() or client.print() macro?

Click to expand...

No.

Well, not as part of the serial.print() or client.print(). No extra RAM, other than a small number of bytes of temporary stack usage associated with the function call and local variables in the print() function is used. A complete copy of whatever you print is absolutely not copied from Flash to RAM, for the sake of the print() operation. The print() function accesses your data directly from the Flash, on 32 bit Teensy.

Internally, the USB port and the W5200 Ethernet chip involve complex buffering and other features to optimize bandwidth utilization of packets... but that happens after the print() function gives the data to those complex objects. For Ethernet, RAM inside the W5200 ethernet chip is involved.

Or do the macros just read it from the flash memory?

Click to expand...

Yes

I would suspect the latter of course, but I'd rather be sure.

Click to expand...

Yes, confirmed.

Also, it was mentioned earlier that data is copied 1 byte at a time on AVR. That's true on regular Arduino. On Teensy 2.0 and Teensy++ 2.0, it's actually copied in small blocks (as I recall, 16 bytes) which really reduces the CPU time for the copy, at the cost of only a very small temporary buffer on the stack.

Thx for the confirmation everyone!

PaulStoffregen said:

For Ethernet, RAM inside the W5200 ethernet chip is involved.

Click to expand...

On a related note, I found something peculiar about this too. I'm using the W5100 for this particular project, which should have 8Kb of TX/RX buffer. However when sending the ~1,2kB string at once (through client.print()) made the Wiznet drop the last 1/3th of it. Client.print() statements after that (e.g. some HTTP stuff) weren't dropped. Client.write(), which I thought should send the string a little bit slower resulted in the same problem. I then split up the string in two, that seemed to work.

Epyon said:

However when sending the ~1,2kB string at once (through client.print()) made the Wiznet drop the last 1/3th of it. Client.print() statements after that (e.g. some HTTP stuff) weren't dropped. Client.write(), which I thought should send the string a little bit slower resulted in the same problem.

Click to expand...

It's probably a bug in the Ethernet library... maybe even a long-standing problem few if any people have ever encountered, since normal Arduino boards are based on smaller AVR chips.

Please start a new thread in the Suggesting & Bug Reports section. Remember the "forum rule", a complete program demonstrating the problem is required. Some info about what to run on the other side is also needed for the problem to be reproducible.

I probably can't tear into the Ethernet library right away... but if you're willing to post a reproducible test case, I'll put this on my list of bugs to investigate and fix.

Note, a standard ethernet frame can only send at most 1,500 bytes in a frame, and sometimes between two hosts, it is smaller than that. I would imagine somebody is not properly handling fragmented packets. I know in the past, I would at times need to tune mtu sizes manually if I was getting too much disconnnects (http://en.wikipedia.org/wiki/Maximum_transmission_unit). Now, this was in a hosted environment where you could use commands to change the parameters. Perhaps you just need to send smaller amounts of data.

This might be MTU related, but since more Arduino projects deal with such small data on such low-power chips, I'd suspect it's a simple Ethernet library bug in a seldom-used case.

PaulStoffregen said:

It's probably a bug in the Ethernet library... maybe even a long-standing problem few if any people have ever encountered, since normal Arduino boards are based on smaller AVR chips.

Please start a new thread in the Suggesting & Bug Reports section. Remember the "forum rule", a complete program demonstrating the problem is required. Some info about what to run on the other side is also needed for the problem to be reproducible.

I probably can't tear into the Ethernet library right away... but if you're willing to post a reproducible test case, I'll put this on my list of bugs to investigate and fix.

Click to expand...

Thanks, I'll do that. I have a lot of rather complex projects relying on the standard Ethernet library and I can say it contains a lot of quirks. Especially the stability after long uptimes can be problematic. It's like the SPI communication sometimes hangs: the led on the SPI clock pin stays on and the Teensy isn't responding anymore, only a power cycle can get things going again.

MichaelMeissner said:

Note, a standard ethernet frame can only send at most 1,500 bytes in a frame, and sometimes between two hosts, it is smaller than that. I would imagine somebody is not properly handling fragmented packets. I know in the past, I would at times need to tune mtu sizes manually if I was getting too much disconnnects (http://en.wikipedia.org/wiki/Maximum_transmission_unit). Now, this was in a hosted environment where you could use commands to change the parameters. Perhaps you just need to send smaller amounts of data.

Click to expand...

But why did it send the client.print() statements after that then? Shouldn't the W5100 be able to split up the data into chunks that fit the MTU size configured on my router?