Teensyduino 1.55 Beta #1

Status
Not open for further replies.
Malloc() works - but like the solution posted in #345 - the address is subject to change between builds.
? no.
First, it should survive a reset - not a rebuild. That makes no sense anyway and is not intended to do that. Your solution will not survive a rebuild, too if things change. See:
Second, no, it will not change. Unless you add a malloc or other memory consuming things

Third, most important, an "offset" or similar destroys the heap/malloc system. It does not know about it. Indeed, a malloc is the only valid solution.

The Great thing about the RAW Address is : it is always there - safe ABOVE the 32KB without wasting it for other use
It kills the heap. My code does not waste anything.

Tim, Paul, others: I think you all *need* a vacation. ;-)
 
? no.
First, it should survive a reset - not a rebuild. That makes no sense anyway and is not intended to do that. Your solution will not survive a rebuild, too if things change. See:
Second, no, it will not change. Unless you add a malloc or other memory consuming things

Third, most important, an "offset" or similar destroys the heap/malloc system. It does not know about it. Indeed, a malloc is the only valid solution.

Noted and agreed that over use of malloc() would hit the undefined DMAMEM.

Tested and observed - AFAIK - that fixed offset from _heap_end; as shown does survive rebuild while powered, or warm restart, as long as the 'unknown allocation' stays above the HAB altered 32KB region like in the example.
 
Yup, you see my solution hass less disadvantages than yours :) A rebuild that uses additional memory can kill both. But that's not needed - it should survive a reset - not a new build with different sourcecode.

A "perfect" solution would be to tell the linker about it (i.e. modify linker scripts)
I think that's not wanted.
 
Yup, you see my solution hass less disadvantages than yours :) A rebuild that uses additional memory can kill both. But that's not needed - it should survive a reset - not a new build with different sourcecode.

A "perfect" solution would be to tell the linker about it.
I think that's not wanted.

Nope - agree to disagree :) Unless the code grows or the power fails the RAW address from the END won't fail - unless it runs out of malloc space - then that fails for everything. ZZzzzz ...
 
what's your point?


Of course you don't need to use malloc if you want forum questions , later, why "malloc() fails" or why "Memory gets currupted" or .. "Bug in malloc()" .. or ..."filesystem destroyed"
"But I use the code from the example? Is it wrong?"

It's an official example.


Edit: Tested with MemFS - works. Something like MemFS would be a better example - usecase than a full filesystem anyway..
Edit: No.. even that is too much :) Just a malloc as shown above. No FS.
 
Last edited:
Attention, I slightly modified the code in the post #347 to make it interrupt safe.

GCC Question: Is it useful to add __attribute__((malloc)) ?
 
Last edited:
Indeed the RAW address is UGLY - but prevents WASTE of that 32KB - though it DOES assure that any leftover DMAMEM is blocked form dynamic allocations.

You could use malloc() and allocate enough memory that the address exceeds the 32kb. - and then free the unused memory. This is also not pretty, but cleaner, and nothing would be wasted.

With this talk of what is "clean" versus "ugly", maybe it's time (before 1.55 release) to reconsider whether RAM filesystem persistence is useful or even a reasonably good idea at all.

My concern is something that's fun from an academic perspective may actually be a fundamentally bad design. The main problem is LittleFS doesn't make any strong checks on data integrity, like CRC checks or a duplicate allocation table (as FAT does). While this conversation has a laser-like focus on behavior across code updates, that probably isn't the most important consideration. In nearly all practical applications, code updates are infrequent. What does happen often during realist usage is power cycling. Software bugs where a fault or watchdog reboot recovers are also an important consideration.

The possible problem is RAM tends to retain data for a short time after power loss. But not reliably. Each bit has higher chance of initializing in a different state the longer power is lost. With even a short power loss, odds of all bits correct are low, but odds of some or even many bits retaining their original state are pretty surprisingly high.

If we reboot due to some form of automatic recovery from an unknown software bug, trusting the prior contents of RAM also seems like a rather risky idea.

LittleFS initialization looks for a relatively small amount of metadata to recognize a filesystem. The entire design of LittleFS is built on top of the assumption of reliable non-volatile memory. LittleFS does very little to check whether the storage media is faulty.

Perhaps we should completely remove RAM persistence support?

If it remains, at the very least I want to see the begin() function do better checking. If the prior reboot was due to power on reset, the RAM definitely needs to be wiped clean. Maybe we should also wipe if the reboot was cause by any sort of fault?

But a better question is why do we even have such a risky feature?!
 
Perhaps we should completely remove RAM persistence support?
It may be useful in some cases, but - I agree - not for a FS.
But a better question is why do we even have such a risky feature?!
Well... that is a bit inconsistent. You'd had to remove CrashReport, too. No, that's not wanted.
Perhaps the best way is to just leave it "undocumented" or "reserved" (as NXP calls such things sometimes)
 
CrashReport's use of persistent RAM differs in 2 important ways.

1: CRC check is done over all data. Odds of mistakenly using corrupted data are very low.

2: Data is used once and then wiped. We automatically recover after a single (very unlikely due to CRC) wrong result. A single corruption event can not cause all future use to fail. Only the next use is at risk, because the corrupted data is used once and then automatically destroyed.

The problem is LittleFS wasn't designed for persistent RAM like CrashReport is. There are ways corrupted RAM could could cause a broken filesystem to remain forever in use.

I'm specifically asking about LittleFS. If we keep RAM persistence in LittleFS for 1.55, it needs to be much more cautious. But whether it is ever worth the risk is a question I want to discuss. At this moment, I'm leaning towards reverting the RAM persistence patch.
 
If we do keep persistent RAM support in LittleFS, at a minimum I believe we need to change the code to enlarge each block to hold a CRC32 check. The begin() function would need to check the CRC on every block. If any are corrupt, we start over fresh.
 
I don't care, but its some work and as the real use cases for littlfs in RAM are pretty rare I'd think twice before doing that.
 
I'm specifically asking about LittleFS. If we keep RAM persistence in LittleFS for 1.55, it needs to be much more cautious. But whether it is ever worth the risk is a question I want to discuss. At this moment, I'm leaning towards reverting the RAM persistence patch.
LittleFS RAM persistence is not something that seems particularly useful at least from my perspective. I really don't see a great need for it especially if you have flash available that can do that for you. Prefer to use the memory for other purposes like graphics for temporary storage or framebuffers.
 
LittleFS RAM persistence is not something that seems particularly useful at least from my perspective. I really don't see a great need for it especially if you have flash available that can do that for you. Prefer to use the memory for other purposes like graphics for temporary storage or framebuffers.
I second this!

my 2 cents which is probably inflated.

Yes we can all use a vacation :D Unfortunately unsafe to go anywhere.

RAMFS - I unlikely to use this for much of anything... Although maybe something I would use with MTP... As a quick and dirty way to allow the sketch to transfer data back and forth to PC. Would never expect it to survive a reboot, let alone after reprogram.

KISS - Would not add anything special to library to enhance this usage case, except maybe some notes some place... And the notes maybe are not specific to LittleFs, but simply a mention somewhere that the lower 32-64K of the RAM2 are used by the processor during the boot process... But again maybe confusing than helpful....

Now back to infusing coffee and figuring out more about SDFat32.... Again
 
... Deja Vu ...

At a minimum the comment in "integrity/RAM.ino" needs changed as it is known to break w/DMAMEM now as written when the HAB overwrite area is incorporated in the media.
> odd how easy repro is now when it didn't show up before TD 1.54 release - The "D"ir Verify in functions.ino should be updated.

I can put forth a PR for the RAM.INO - or others can: change the comment, separate "Integrtiy/DMAMEM.INO", or alter the allocation in a fashion the fits the DMAMEM comment: "typically survives Restart/Upload"

There were some number of folks looking for 'static' RAM across restarts - not necessarily for LittleFS usage - but like CrashReport to have a record of a prior run that got interrupted with warm restart.

This demonstrates a usable idiom for persistence capabilities on 1062 {where the HAB area must be avoided } : 4.0, 4.1, MMod: Sketch can log large amounts of data or debug analytics at high speed { versus 4 DWORDS of NVRAM and slow speed }.
As usual there are caveats:
> as long as it manages a flush before restart
> as long as other dynamic or static allocations don't use the same area a second time
> power is maintained
 
Just to mention.. there are few opens bug I reported (last one here, which is a serious issue), and a fix that I did for a other user (PR unmerged).. would be really nice if that could happen before beta 2.
 
Last edited:
... Deja Vu ...

At a minimum the comment in "integrity/RAM.ino" needs changed as it is known to break w/DMAMEM now as written when the HAB overwrite area is incorporated in the media.
> odd how easy repro is now when it didn't show up before TD 1.54 release - The "D"ir Verify in functions.ino should be updated.

Yes, Deja Vu indeed. How many messages do we have now on this and other threads about the details of trying to get LittleFS RAM data to persist across code updates? I haven't counted, but it sure seems like a lot!

I'm on the cusp of reverting the persistent patch. My feeling at this moment is RAM persistence for LittleFS, as it is currently implemented, was a mistake. If you feel it should be kept for 1.55, now is the time to make a clear and well reasoned argument why.


This demonstrates a usable idiom for persistence capabilities on 1062 {where the HAB area must be avoided } : 4.0, 4.1, MMod: Sketch can log large amounts of data or debug analytics at high speed { versus 4 DWORDS of NVRAM and slow speed }.
As usual there are caveats:
> as long as it manages a flush before restart
> as long as other dynamic or static allocations don't use the same area a second time
> power is maintained

I'm skeptical about this being a usable idiom.

As it exists today (no CRC checks, no cache management, etc), my feeling it's at best a poorly designed hack which doesn't belong in a software release we call "stable".


I can put forth a PR for the RAM.INO - or others can: change the comment, separate "Integrtiy/DMAMEM.INO", or alter the allocation in a fashion the fits the DMAMEM comment: "typically survives Restart/Upload"

Yes, when / if I remove LittleFS RAM persistence, this needs to be updated.
 
Support for initialized EXTRAM is missing.
Has there already been a decision to (not) implement this?

Maybe one could check some T4 registers if there was a "power on"n and copy the data in this case only? I'd say that would need some testing.

Otherwise it would be nice to see a warning or errormessage for the case where EXTMEM const data exist.. via size-tool, or linker can do that too(?), perhaps.
 
Last edited:
Support for initialized EXTRAM is missing.
Has there already been a decision to implement this?

I tried but could not make it work, at least not with the old toolchain we're currently using. I could not find a way to cause the compiler to put the initialization data for variables in a specific linker section into their own section. Every attempt I made had the initialization values mixed together in the normal .data segment, with no way for the startup code to know which parts to copy to internal vs external memory.

If you can find a good way, please share. If it can work on the current toolchain, I would love to merge this before 1.55. If a newer toolchain is needed, that would be a good reason to risk updating the toolchain in 1.56.
 
Ah..ok
I can try a bit on the weekend, maybe.
The issue with a newer toolchain is, they are all a little bit slower :-(
 
It would work perhaps to use a special name const with the same attributes as PROGMEM

So.. would translate to PROGMEM _EXTMEMINIT = {1,2,34} and then just copy it from flash to PSRAM on power-on.
The name would have to be fix.

Would be a ugly workaround, but at least a way.. but the user can do this, too so no need for support by TD.
We could perhaps add a helper function that (EDIT: checks for power-on reset and then) does the copy?


by the way:
EXTMEM int a[] = {1,2,3,4} even crashes ;)
With added "const " it does something.. at least the data seems to be in the cache, without any mods.
 
Installed 1.55 Beta1 over a fresh installation of Arduino IDE 1.8.15. Installation worked fine w/o any issues.

However, as the 1.55 Beta1's change log mentioned an updated NativeEthernet, I checked it behaviour with one of my programs. Unfortunately, NativeEthernet still creates "Socket Event Err: -20" and "SendErr: -20" errors if a program acts both as server and client. My program uses client.close() and client.stop(), as recommended, and it make sure that all incoming data has been read before these two closing commands are executed. The sketch does not connect to an outside server, whilst it serves an Ethernet request, and vice versa. My program receives 3 - 6 short Ethernet requests per minute from a Windows Server and it makes one MODBus tcp call per minute to a photovoltaic inverter. These are all short events of typcially less than 35ms.

The time between these "Socket Event Err: -20" can be seconds or hours, but so far I have only (except once, see below) seen "Socket Event Err: -20" followed by 7 times "SendErr: -20", making it always 8 messages in total. Does thhis tell us anything? As mentioned, there was one exception when there was "Socket Event Err: -20" + 7 times "SendErr: -20" very frequently for a period of approx. 35 minutes, which stopped (to return to the infrequent occurence) when "Socket Event Err: -20" was follwed by 15(!!!) times "SendErr: -20".

The sketch run otherwise fine for now approx. 12 hours, particularly getSocketNumber() seems now stable, i.e. the sketch did not run out of sockets (as before with 1.54+1.8.15).

This leads me to this question: are these "Socket Event Err: -20" and "SendErr: -20" meaningful messages, as it seems a sketch cannot prevent or act on them? They seem not to affect the reception or sending of data. And why are they almost always 8 errors in a row. Surely that must mean something to the experts - I hope.
 
Yes, Deja Vu indeed. How many messages do we have now on this and other threads about the details of trying to get LittleFS RAM data to persist across code updates? I haven't counted, but it sure seems like a lot!

I'm on the cusp of reverting the persistent patch. My feeling at this moment is RAM persistence for LittleFS, as it is currently implemented, was a mistake. If you feel it should be kept for 1.55, now is the time to make a clear and well reasoned argument why.


I'm skeptical about this being a usable idiom.

As it exists today (no CRC checks, no cache management, etc), my feeling it's at best a poorly designed hack which doesn't belong in a software release we call "stable".



Yes, when / if I remove LittleFS RAM persistence, this needs to be updated.

Just coming back and finding this post.

As far as utility - just like CrashReport survives restart to enable system debug - exposing a way to have larger user areas persist could be useful for continued operation or restart or After Issue debugging where log/data of prior run persists for review. LittleFS is a good place - and the code does the dCache_Flush on change { though I edited it in as flush_delete and should just be _flush }

This "remove LittleFS RAM persistence" affects DMAMEM usage as well as QSPI PSRAM EXTMEM drives? Not sure PSRAM has issues like the 32KB clobber by HAB that are showing?
> LittleFS_RAM myfs; :: Using just a memory buffer pointer covers RAM, EXTMEM and DMAMEM
> When _RAM is used for media - it is subject to the same checks on consistency as all other Media supported by 'lfs' core code.

A T_4.0 without the onboard option of PSRAM ( or FLASH or SD } could easily call some portion of DMAMEM into use for some usable 10's to 100's of KB of data - maybe just a temp holding area until slower SPI devices could take the data without interrupting some other 'process'

RE: 'idiom' :: If a user stores sketch data in a piece of DMAMEM ( or EXTMEM ) expecting persistence then proper use of dcache_flush would apply and if required some CRC check like CrashReport uses could be applied for the data to survive. - But that isn't directly related to LittleFS.

-> I'll fork the RAM ( like PSRAM ) to a DMAMEM.ino integrity example.
-> should also offer the updated 'function.ino' were 'D'ir Verify tests file contents ( when created by the example } for each Integrity example.
-> and update the sync to just flush, not flush_delete from cache
 
Status
Not open for further replies.
Back
Top