Storing large amounts of configuration data

turbo2ltr

Active member
My project will have a lot of configuration data. It's designed to be very generic, but be able to be used in over 100 different locations and each location has site-specific configuration. The units need to be movable to any location without needing to be reconfigured (other than some dip switches)

So while not all the data will need to be loaded into RAM at the same time, I do need (ok, I want) to have it all stored in one file for ease of management.

So the amount of data is over the 4K EEPROM size, so it will need to be stored on the SD card, or the flash I put on the audio card. But most likely the SD card for a reason you will soon see. Now if that was the only requirement, We'd be done here. But it's not..

One of the my requirements is that the configuration file be human readable. This really throws a wrench in the mix. The reason for this is so that the file can easily be edited in the field if needed

So the challenge is now instead of just storing a struct, I have to have string names for everything. As if the amount of data wasn't big enough... Then I'd have to do the slow process of parsing it all. There really wouldn't be any quick access to any of the data.

I started looking at database and dictionary libraries. None of them really is what I need. My data consists of all kinds of types. My data isn't really rows of the same data so databases aren't really useful. A lot of the libraries are locked to a specific type or at least make it difficult to handle structs. I considered using JSON libraries, but that also seemed to suffer from similar issues. Most of my programming I do is php so I'm pretty spoiled with loose types.

Another factor is, at least in the beginning stages, is not locking myself into a fixed length. I would much prefer some sort of arbitrary key/value type setup that is not so rigid. I was hoping not to have to explicitly move each variable into RAM but I don't think there's going to be way around that.

In a former life I did PalmOS programming, so I considered a linked list sort of method Palm used in their PDB file types. This wouldn't lock me into a fixed length, but also isn't human editable.

Other than the writing on the wall of having to make some compromises, anyone have any suggestions?

Thanks!
 
A Teensy can chew through arrays of chars pretty reliably as long as you can define max lengths at compile time, which in this case sounds like you can since you know the max length of a line and wouldn't be trying to loop back. A lot would depend on how robust you want the error handling/validating of the files to be, which can be a bit of a vortex.

I've successfully used https://arduinojson.org/ for a project and like the way the website has tools to test your strucs/parseing but using JSON really pushes you into providing a PC side tool to manage JSON file since it's not really human editable.
 
Well I managed to delete my reply by pressing the "+ Reply to thread" button when I was done instead of the "Post quick reply"... <sigh>

Anyway, long story short:
I considered that library. I was planning on having a web-based config tool built in so using JSON would allow me to let the client javascript do any heavy lifting.
As you mentioned, though, you aren't going to edit large JSON files with a text editor.

Feeling like the human editable requirement will be the first to get dropped.
 
"Human readable"? How does a human read a Teensy? Well, the Teensy formats the data and pumps it out as Serial or whatever, to a terminal that the human can read. So store compressed or optimised data and produce the full-size flat file only on demand.

The SD card is the best way to store this kind of stuff. Then you can pop the card out of the Teensy and put it in your PC to use a text editor or whatever. It does leave it open to user-abuse, like if someone takes the card and forgets to replace it. So it does need to have a "safe" configuration that it can use without the card.

Lots of devices like cameras and dataloggers use a special config file on the SD card.

I've used the Windows .ini format a few times. That's infinitely extendible.
 
A few more questions:

- How much data are we talking about? 5kB? 50kB? 50MB?
- How fast do you need the parsing to be?
- Can the parsing happen once at boot time and then be cached in a faster format?

Might be helpful for you to post a sample of the configuration data for one of these devices. Obfuscate it if it's sensitive...
 
- How much data are we talking about? 5kB? 50kB? 50MB?
- How fast do you need the parsing to be?
- Can the parsing happen once at boot time and then be cached in a faster format?

I haven't actually finished compiling all the variables I'll need, I only have a basic POC done at this point. It is working well but I'm already having issues managing the configuration for the single location. So before I continue development, I wanted to get this issue out of the way so I have a solid base to handle config as I go forward.

I have started putting a spreadsheet together of all the variables and back-of-the-napkin says I'm just over 4k at this point. Each location can have a couple different sub-configurations, and we may add more locations going forward, so that is sure to grow.

I'm guessing parsing does not need to be super fast.. I expect I'll load the relevant configuration into RAM on boot. The real fun is going to be trying to write it back out if needed.

"Human readable"? How does a human read a Teensy?
Not sure if you are trying to be funny. I am referring to the configuration file on the SD card. But it seemed you knew that.

I've used the Windows .ini format a few times. That's infinitely extendible.

You've used the INI format with a teensy? If so can you expand on how you went about it?
 
Based on your requirements, one possible approach is to use a configuration file format that supports human readability and flexible key/value pairs, such as YAML (YAML Ain't Markup Language). YAML is a lightweight data serialization format that is commonly used for configuration files.

You can store your lead data enrichment in a YAML file on the SD card. YAML allows you to represent data using a combination of lists, dictionaries, and scalar values (strings, numbers, booleans, etc.). This flexibility should accommodate the various types of data you mentioned.

Here's an example of how your configuration file in YAML format might look:
location: "Site A"
units:
- id: 1
name: "Unit 1"
dip_switches: [true, false, true]
settings:
- parameter: "temperature"
value: 25.0
- parameter: "humidity"
value: 60.0
- id: 2
name: "Unit 2"
dip_switches: [false, false, true]
settings:
- parameter: "temperature"
value: 22.5
- parameter: "humidity"
value: 55.0
- parameter: "light_level"
value: 80

In this example, the configuration file includes a "location" field specifying the site name and a list of "units," each with their specific settings. You can have multiple units with different configurations.

To work with YAML in your code, you can find YAML parsing libraries for your specific programming language. Most popular programming languages have YAML libraries available that can parse the YAML file into a data structure you can work with.

By using YAML, you can achieve human-readable, flexible, and easily editable configuration files while storing all the data in one file on the SD card.
 
Last edited:
Back
Top