Forum Rule: Always post complete source code & details to reproduce any issue!
Page 2 of 4 FirstFirst 1 2 3 4 LastLast
Results 26 to 50 of 84

Thread: SPI Flash Allocation Table

  1. #26
    Senior Member+ Frank B's Avatar
    Join Date
    Apr 2014
    Location
    Germany NRW
    Posts
    5,679
    I would vote for the linked-list, too.

    But, sometime ago,(i hope i find it again!) i found a paper,describing a new algorithm (but simple) for wear-levelling which provided more flashing-cyclyes in most cases.
    For wear-levelling, a fixed FAT is better than the linked list. Plus, you can delete data in a better way.

    @iwanders: you know that the "medium" is soldered and not changeable ?
    Last edited by Frank B; 01-10-2015 at 11:16 AM.

  2. #27
    Yes I know that's the case for the SPI memory chip on the audio board, but that's not necessary if the system is hardware independent.

    For example; one could use an I2C flash memory chip which is connected with a header. I thike it would be useful if you can detect whether the memory is already formatted or not.

  3. #28
    Senior Member
    Join Date
    Jun 2013
    Location
    So. Calif
    Posts
    2,828
    the 16 or 32MB SPI flash chips (8 pin) are $1 or less.
    If there's an I2C version, it would avoid all the issues with the T3 CPU having but one SPI port.

  4. #29
    Senior Member
    Join Date
    Jun 2013
    Location
    So. Calif
    Posts
    2,828
    Some ARM vendors have NAND flash wear-leveling code in their open source ARM HAL libraries. Normally used with 8 bit parallel flash chips. Maybe can work with serial flash too.

  5. #30
    Senior Member PaulStoffregen's Avatar
    Join Date
    Nov 2012
    Posts
    20,299
    Quote Originally Posted by stevech View Post
    If there's an I2C version, it would avoid all the issues with the T3 CPU having but one SPI port.
    Yes, but the SPI sharing issues are minor compared to the lack of bandwidth on I2C.

  6. #31
    Senior Member PaulStoffregen's Avatar
    Join Date
    Nov 2012
    Posts
    20,299
    Ok, sounds like there's a pretty good consensus forming around using a linked list approach.

    My main concern with the proposal in msg #24 is the lack of ability to locate all of a file's data aligned to 4K sector boundaries. Some types of files, like MP3 and wavetables, will never need to be written. We can pack those tightly. But files intended for recording will want to be aligned onto erasable sectors. A tightly packed linked list format can't do that.

    The linked list concept could still be used, if more bytes were allocated to indicate the data location and length, rather than requiring the data to immediately follow the header info, and requiring it's length to be implied by the space between headers.

  7. #32
    Senior Member+ Frank B's Avatar
    Join Date
    Apr 2014
    Location
    Germany NRW
    Posts
    5,679
    Hmm..
    This all let me think: Why don't we look at existing Flash - Filesystems ? We don't need to use one of them, but mabe we get ideas..

    http://en.wikipedia.org/wiki/List_of...id_state_media

    And with a bit luck, we find a good one in this list, find existing sourcecode and don't have to re-invent the wheel.

  8. #33
    Junior Member
    Join Date
    Oct 2014
    Posts
    14
    Quote Originally Posted by Frank B View Post
    Hmm..
    This all let me think: Why don't we look at existing Flash - Filesystems ? We don't need to use one of them, but mabe we get ideas..

    http://en.wikipedia.org/wiki/List_of...id_state_media

    And with a bit luck, we find a good one in this list, find existing sourcecode and don't have to re-invent the wheel.
    Thats why i posted spiffs library.

  9. #34
    Senior Member PaulStoffregen's Avatar
    Join Date
    Nov 2012
    Posts
    20,299
    I was really hoping to keep this very simple, basically a descriptive partition table rather than a "file system".

  10. #35
    Senior Member+ Frank B's Avatar
    Join Date
    Apr 2014
    Location
    Germany NRW
    Posts
    5,679
    Quote Originally Posted by Relampago View Post
    Thats why i posted spiffs library.
    Looks good, from the description. and it has wear-leveling :-)
    He states that is not good for >128MB, but maybe this can be changed ?

  11. #36
    Senior Member PaulStoffregen's Avatar
    Join Date
    Nov 2012
    Posts
    20,299
    I just spent some time reading about SPIFFS. This is exactly the type of complexity I do NOT want.

    For the audio library, we need files to be stored as continuous data. When an audio library object accesses a file on the flash, it will need only 2 numbers: the beginning address and the allocated length. As an object plays samples, or reads a wave table for synthesis, or pulls in more of an MP3 stream, we NEVER want to incur extra overhead of reading index sectors or other metadata, like SPIFFS requires.

    When 128 samples need to be read from the flash, the most important goal is keeping our flash access to only those 256 bytes, for consistent and predictable timing. Remember, this will be embedded within some object like a sample player. Users will drag any number of them onto the GUI and click Export to get a sketch that might try to read 12 simultaneous streams. In such a case, 12 instances will end up pulling a total of 3072 bytes from the flash every 2.9 ms, which happens to be about 1/3rd of the total SPI bandwidth, not including slight protocol overhead required by the SPI flash chip.

    Continuous, non-fragmented file allocation is essential. Every file's storage must be described by only 2 numbers, beginning address and total data length.

    Tools that format the entire flash and decide what space each file gets can have complexity. Most files can be allocated and packed byte-wise, and others meant for recording or writing can be allocated to 4k sector boundaries. The allocation table can provide plenty of metadata about the file, though data type and a text name are probably all we need.

    However, being able to quickly locate a file, with a minimum number of SPI transfer bytes, is valuable. An expected use case will be the main program searching for the file based on its text filename, while the library is accessing many other files. One or more SPI transactions will delay the audio update interrupt, so we want to minimize the time we're preventing the audio library from updating.

    I fear this conversation has wandered pretty far from focusing on the real-time requirements of the audio library. Indeed there are lots of great general purpose file systems, like SPIFFS, that would be wonderful if the goal was file storage for general purpose computing.

    Please remember, the goal here is CD quality audio, possibly a dozen or more polyphonic streams, on a low power microcontroller.
    Last edited by PaulStoffregen; 01-10-2015 at 09:07 PM.

  12. #37
    Senior Member+ Frank B's Avatar
    Join Date
    Apr 2014
    Location
    Germany NRW
    Posts
    5,679
    Yes, it's the question what the goal is.

    Ok, another idea, the other "extreme" :-)
    I wrote a very simple sketch which copys files to the SPIFlash. Very simple to use. I could extend it a little bit to output a small piece of c-sourcecode (some #defines or const array. If wanted, with filenames..).
    Then, all the user has to do is copy&paste this to his sketch, and voilą.

    https://forum.pjrc.com/threads/27409...1750#post61750

    Ok, since it will take some time until we have our FAT, i'll do it this way. The sketch can flash every file and is not restricted to audio.
    Last edited by Frank B; 01-10-2015 at 09:22 PM.

  13. #38
    Senior Member PaulStoffregen's Avatar
    Join Date
    Nov 2012
    Posts
    20,299
    Quote Originally Posted by Frank B View Post
    Ok, another idea, the other "extreme" :-)
    I wrote a very simple sketch which copys files to the SPIFlash. Very simple to use. I could extend it a little bit
    Yes, I looked at your code briefly yesterday. Extending that program sounds like a great path forward.

    to output a small piece of c-sourcecode (some #defines or const array. If wanted, with filenames..).
    Yes, something like this. I have a bit of C++ trickery in mind for the defines, to carry both the address and length in a single name and allow the compiler to optimally inline both.

    I do also want the sketch to store filenames and allocation data in the flash. It can be a fixed table, a linked list, or any number of other structures. Later I want to build other sketches that have a GUI front end, so the allocation info and filenames need to be stored in the flash to later allow a beautiful front end.

  14. #39
    Senior Member+ Frank B's Avatar
    Join Date
    Apr 2014
    Location
    Germany NRW
    Posts
    5,679
    Great.
    What c++ trickery do you have in mind ?
    It's on Github.. erverybody is free to submit patches.
    But i can modify it too. Just tell me what do you want... :-)

  15. #40
    Junior Member
    Join Date
    Oct 2014
    Posts
    14
    Paul, i totally agree with you. I currently do that Kind of adress + length for reading from sdcard in my drum computer prototype.

    If we focus on streams, simplicity and performance, index and length is sufficient for me, as i know what kind of data I've written and want i need to read.

  16. #41
    Senior Member
    Join Date
    Jun 2013
    Location
    So. Calif
    Posts
    2,828
    The ARM HALs (some anyway) have libraries for parallel and SPI NAND flash I/O, wear-leveling and bad-block remapping. Source for that is open, so perhaps that source can be adapted for the Arduino environment. Perhaps Freescale has one too.
    The step after that is 8KB for FATfs that mates with the APIs in the above - such as open source
    http://elm-chan.org/fsw/ff/00index_p.html
    or
    http://elm-chan.org/fsw/ff/00index_e.html

    BUT the goal here may have been DIY to make it tiny. Wheras the above is less frugal since many current low cost ARM M4's have 512 or 1MB of flash and 256KB or more of RAM.
    Last edited by stevech; 01-11-2015 at 12:55 AM.

  17. #42
    Senior Member PaulStoffregen's Avatar
    Join Date
    Nov 2012
    Posts
    20,299
    Steve, we're talking specifically about supporting the optional W25W128FV (and equivalent part numbers) NOR Flash chip on the audio board.

    This thread has already gotten side-tracked pretty badly!

    If you want to talk about NAND Flash chips, or *any* chip other than 8-pin SPI NOR Flash compatible parts, and any usage case other than the specific real-time media requirements of the audio library, please start a new thread.

    The audio board is already made. A couple thousand have already shipped, so the hardware is absolutely fixed with that 8-pin location to add optional SPI NOR Flash. Future boards might get designed with all sorts of other chips, but here any now, we're working with hardware that's already made. Other types of chips simply aren't up for discussion right now (except in a new thread, where anything is fair game)

    What's needed is software. Frank and Pete have both made a start, writing directly to the chip. It's time to finalize some basic allocation scheme, so as we make several different object that can use this SPI Flash, they can work together nicely.

  18. #43
    Senior Member
    Join Date
    Jun 2013
    Location
    So. Calif
    Posts
    2,828
    Oh. I misread thinking the goal was more generalized on storage.

    I skim-read too much.

  19. #44
    Senior Member
    Join Date
    Feb 2013
    Posts
    563
    Quote Originally Posted by Relampago View Post
    Paul, i totally agree with you. I currently do that Kind of adress + length for reading from sdcard in my drum computer prototype.

    If we focus on streams, simplicity and performance, index and length is sufficient for me, as i know what kind of data I've written and want i need to read.
    wanted to second that. opening by filename should be avoided or at least be made optional.

    as an aside -- so if you reading from SDcard by address and length, i take it you pulled uint8_t open(SdFile* dirFile, uint16_t index, uint8_t oflag) into SD/the audio lib? i'd be curious: are there any performance gains to be had from that? or is it mostly to define the start/end points of the sample?

  20. #45
    Senior Member+ Frank B's Avatar
    Join Date
    Apr 2014
    Location
    Germany NRW
    Posts
    5,679
    For me, it does not really matter how exactly the files are stored. The only thing is, i need to know, how you want it. Then i can do something. Or you do it. Then i need a lib. But without knowing...

    It would be much more interesting (for me) to have the possibility to stick the teensy in an usb-port and have 2 or 3 drives..(sd-card, spiflash, maybe teensyflash)...
    Unfortunately, i don't have the time and knowlage to do it.

    Edit: Or one drive with 3 directories. Or MTP.
    Last edited by Frank B; 01-11-2015 at 11:32 AM.

  21. #46
    Senior Member PaulStoffregen's Avatar
    Join Date
    Nov 2012
    Posts
    20,299
    I'm currently working on the bed-of-nails tester for the upcoming low-cost Teensy board (Cortex M0+), so I can't do much at the moment. I'll get back to this in a few days, with some "final" decisions.

  22. #47
    Senior Member+ Frank B's Avatar
    Join Date
    Apr 2014
    Location
    Germany NRW
    Posts
    5,679
    Great.
    If you decide to work on the USB-side (MTP?) i offer my help. I know nothing about usb, but maybe i could help with the file/memory/flash/spi accesses..
    Hey, you don't need a GUI with that :-) the operating-systems are the GUI :-)

  23. #48
    Senior Member PaulStoffregen's Avatar
    Join Date
    Nov 2012
    Posts
    20,299
    Later this year - after the low-cost board is out, after another bootloader upgrade, and after a couple months of more audio work, I'm planning to implement 3 more USB types: audio, MTP and UDP-only network (mainly for E1.31/Artnet lighting and low-latency Open Sound Control).

    But first, I need to get the low-cost boards released, and in the process, make some decisions and fiddle with APIs a bit on this SPI flash stuff, so we can all make progress within a simple allocation scheme that will let many different ways to use the SPI flash work together nicely and allow beautiful GUI tools to someday be built, of course in my spare time!
    Last edited by PaulStoffregen; 01-11-2015 at 10:32 PM.

  24. #49
    Senior Member+ Frank B's Avatar
    Join Date
    Apr 2014
    Location
    Germany NRW
    Posts
    5,679
    Quote Originally Posted by PaulStoffregen View Post
    Later this year - after the low-cost board is out, after another bootloader upgrade, and after a couple months of more audio work, I'm planning to implement 3 more USB types: audio, MTP and UDP-only network (mainly for E1.31/Artnet lighting and low-latency Open Sound Control).

    But first, I need to get the low-cost boards released, and in the process, make some decisions and fiddle with APIs a bit on this SPI flash stuff, so we can all make progress within a simple allocation scheme that will let many different ways to use the SPI flash work together nicely and allow beautiful GUI tools to someday be built, of course in my spare time!

    When will the M0 Board be released? Can you give technical info..( which processor? )
    I would like to adapt my public libs as soon as possible :-) I buy two.. (Is there a way to reduce the shipping cost ( > 10 $) to Germany? ;-)
    And, to stay on topic: Is it possible to use the audioboard or is there an adapter for spiflash ?

  25. #50
    Senior Member Pensive's Avatar
    Join Date
    Aug 2014
    Location
    Basingstoke, UK
    Posts
    561
    The coder knows where everything is in the FLASH when they write it.

    but the flash persists a reboot, the variables storing that information do not.

    Therefore some kind of filename will be essential, otherwise you have to erase and rewrite the flash each time you boot, and collect the file offset positions.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •