Forum Rule: Always post complete source code & details to reproduce any issue!
Results 1 to 7 of 7

Thread: Storing large amounts of configuration data

  1. #1

    Storing large amounts of configuration data

    My project will have a lot of configuration data. It's designed to be very generic, but be able to be used in over 100 different locations and each location has site-specific configuration. The units need to be movable to any location without needing to be reconfigured (other than some dip switches)

    So while not all the data will need to be loaded into RAM at the same time, I do need (ok, I want) to have it all stored in one file for ease of management.

    So the amount of data is over the 4K EEPROM size, so it will need to be stored on the SD card, or the flash I put on the audio card. But most likely the SD card for a reason you will soon see. Now if that was the only requirement, We'd be done here. But it's not..

    One of the my requirements is that the configuration file be human readable. This really throws a wrench in the mix. The reason for this is so that the file can easily be edited in the field if needed

    So the challenge is now instead of just storing a struct, I have to have string names for everything. As if the amount of data wasn't big enough... Then I'd have to do the slow process of parsing it all. There really wouldn't be any quick access to any of the data.

    I started looking at database and dictionary libraries. None of them really is what I need. My data consists of all kinds of types. My data isn't really rows of the same data so databases aren't really useful. A lot of the libraries are locked to a specific type or at least make it difficult to handle structs. I considered using JSON libraries, but that also seemed to suffer from similar issues. Most of my programming I do is php so I'm pretty spoiled with loose types.

    Another factor is, at least in the beginning stages, is not locking myself into a fixed length. I would much prefer some sort of arbitrary key/value type setup that is not so rigid. I was hoping not to have to explicitly move each variable into RAM but I don't think there's going to be way around that.

    In a former life I did PalmOS programming, so I considered a linked list sort of method Palm used in their PDB file types. This wouldn't lock me into a fixed length, but also isn't human editable.

    Other than the writing on the wall of having to make some compromises, anyone have any suggestions?


  2. #2
    Senior Member
    Join Date
    Apr 2013
    A Teensy can chew through arrays of chars pretty reliably as long as you can define max lengths at compile time, which in this case sounds like you can since you know the max length of a line and wouldn't be trying to loop back. A lot would depend on how robust you want the error handling/validating of the files to be, which can be a bit of a vortex.

    I've successfully used for a project and like the way the website has tools to test your strucs/parseing but using JSON really pushes you into providing a PC side tool to manage JSON file since it's not really human editable.

  3. #3
    Well I managed to delete my reply by pressing the "+ Reply to thread" button when I was done instead of the "Post quick reply"... <sigh>

    Anyway, long story short:
    I considered that library. I was planning on having a web-based config tool built in so using JSON would allow me to let the client javascript do any heavy lifting.
    As you mentioned, though, you aren't going to edit large JSON files with a text editor.

    Feeling like the human editable requirement will be the first to get dropped.

  4. #4
    Member MorganS's Avatar
    Join Date
    Apr 2015
    Northern Nevada
    "Human readable"? How does a human read a Teensy? Well, the Teensy formats the data and pumps it out as Serial or whatever, to a terminal that the human can read. So store compressed or optimised data and produce the full-size flat file only on demand.

    The SD card is the best way to store this kind of stuff. Then you can pop the card out of the Teensy and put it in your PC to use a text editor or whatever. It does leave it open to user-abuse, like if someone takes the card and forgets to replace it. So it does need to have a "safe" configuration that it can use without the card.

    Lots of devices like cameras and dataloggers use a special config file on the SD card.

    I've used the Windows .ini format a few times. That's infinitely extendible.

  5. #5
    Senior Member blackketter's Avatar
    Join Date
    May 2015
    San Francisco
    A few more questions:

    - How much data are we talking about? 5kB? 50kB? 50MB?
    - How fast do you need the parsing to be?
    - Can the parsing happen once at boot time and then be cached in a faster format?

    Might be helpful for you to post a sample of the configuration data for one of these devices. Obfuscate it if it's sensitive...

  6. #6
    Quote Originally Posted by blackketter View Post
    - How much data are we talking about? 5kB? 50kB? 50MB?
    - How fast do you need the parsing to be?
    - Can the parsing happen once at boot time and then be cached in a faster format?
    I haven't actually finished compiling all the variables I'll need, I only have a basic POC done at this point. It is working well but I'm already having issues managing the configuration for the single location. So before I continue development, I wanted to get this issue out of the way so I have a solid base to handle config as I go forward.

    I have started putting a spreadsheet together of all the variables and back-of-the-napkin says I'm just over 4k at this point. Each location can have a couple different sub-configurations, and we may add more locations going forward, so that is sure to grow.

    I'm guessing parsing does not need to be super fast.. I expect I'll load the relevant configuration into RAM on boot. The real fun is going to be trying to write it back out if needed.

    "Human readable"? How does a human read a Teensy?
    Not sure if you are trying to be funny. I am referring to the configuration file on the SD card. But it seemed you knew that.

    I've used the Windows .ini format a few times. That's infinitely extendible.
    You've used the INI format with a teensy? If so can you expand on how you went about it?

  7. #7

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts