Serialize large objects for storage on SD card?

[JKIM]

I have been researching on the best way to serialize large objects for storage on the SD card of Teensy 4. Specifically ones that don't have to store a DOM into a temporary variable, before serialization.

I have tried Boost, Cereal, ArduinoJson and ThorsSerializer. I think for my purposes, ThorsSerializer looks like the right choice, but then the trouble is getting it streamed onto the SD card, which I have another thread open on: https://forum.pjrc.com/threads/60666-C-streams-with-ChaN-FatFS

Most of the serialization, except for ArduinoJson, seem to be aimed at desktop applications.
Is this not a common requirement? I didn't find many resources other than ArduinoJson (which has wonderful documentation!)

JK

 

[PaulStoffregen]

Is this not a common requirement?

I'm going to answer no, this is not commonly done, at least judging by how infrequently this question comes up.

Most libraries and code people write with Teensy involves a strong dependency on hardware. Maybe there are some cases where serialization makes sense in the context of objects dealing pretty directly with hardware devices? The few times object serialization has been discussed, it's usually about creating a communication protocol. Typically people go with a more traditional approach like parsing arrays of bytes or overlaying a C-style packed struct onto incoming data.

The other not-as-strong trend is we're gradually moving towards more use of constexpr constructors in Teensy's libraries. This usually fits nicely with Arduino's style guidelines, specifically using a begin() function to do actual hardware initialization. Use of C++ constexpr constructors allows the compiler to perform much better optimizations. While you can design your own objects any way you like, the impressive performance gains of constexpr are pretty compelling.

Maybe if you gave us a little more explanation of what you're trying to accomplish, it could add context to better understand your question and answer in ways most likely to actually help you.

 

[JKIM]

Thanks Paul. I am trying to accomplish the serialization of a large amount of application data (MIDI) from an external RAM chip onto an SD card for long term storage and data exchange. Preferably in a human readable/portable format, like JSON.

The external RAM is larger than the RAM banks in Teensy4, so I need to either serialize in chunks, or serialize as a stream. I expected that storing volatile RAM data onto nonvolatile media would be common.

Previously I used the direct "parsing of arrays" approach, but I have progressed to using vectors of objects, so that the memory can be dynamically allocated at runtime. This precludes using a straight binary dump/restore process.

 

[wangnick]

Hi JKIM,

can you not traverse your application data down to a reasonable depth, and then on-by-one emit the objects on that depth as a reasonably-sized JSON substring onto the file stream? I don't fully understand why you are aiming for a library to do that for you ...

Kind regards,
Sebastian

 

[JKIM]

Hi JKIM,

can you not traverse your application data down to a reasonable depth, and then on-by-one emit the objects on that depth as a reasonably-sized JSON substring onto the file stream? I don't fully understand why you are aiming for a library to do that for you ...

Kind regards,
Sebastian

Yes, that is definitely an option. My member types are fairly simple, ints, pointers and vectors, so a library is maybe a bit of overkill in most cases. But I wanted to check for recommendations before I go too far down that rabbit hole.
By the time I handle the various data types and the deserialization, I may be close to the functionality of a library like ArduinoJson.

JK

 

[JKIM]

OK, I have gotten somewhat further.

I was able to get ThorsSerializer working with SdFat (Teensy4 beta, using SDIO) for serialization at least.

I had to change SdFat to use custom class names so that it didn't conflict with stdlib, and then adapt ThorsSerializer to use the SdFat streams. Using this method I was able to serialize about 4MB of raw data from RAM to about 26.5MB of verbose JSON on the SDcard in ~29.5 seconds. I haven't really looked at optimization yet, this was just getting it to work initialliy.

For deserialization, it's a different story. ThorsSerializer uses stream iterators, which aren't implemented in SdFat... so I need to figure out how to move forward there.

JK