Forum Rule: Always post complete source code & details to reproduce any issue!
Page 21 of 21 FirstFirst ... 11 19 20 21
Results 501 to 513 of 513

Thread: Many TLAs: MTP MSC FS SD SDFat LittleFS UsbMSCFat to work with each other 8)

  1. #501
    Senior Member+ KurtE's Avatar
    Join Date
    Jan 2014
    Posts
    9,576
    Quote Originally Posted by wwatson View Post
    Alright guy's. Will hang tough

    Thanks.
    I hear you, but we are slowly getting the pieces into place and things are coming along. Been having some fun trying to make some simpler to create MTP sketches and include things like detecting when SD card inserted on T4.x... Maybe T35/6 as well... Have not tested in awhile.
    Also with external SD readers that have an IO pin for card detection... Still WIP...

    Right now trying to figure out why: trying to get the Free space on FAT32 external volume is REAL REAL REAL slow. I know one of the reasons is we have to read in every ... FAT sector and count the bits... But I also through we had option turned on to cache the free after the first time we call it... And that is not working...

    In my SD wrap class I enabled the Free Space callback to my code (like we do for MSC drives) so I see what is happening...

    Debug Run:
    Code:
    :\GitHub\MTP_t4\examples\SD_MTP-logger\SD_MTP-logger.ino Sep  1 2021 17:40:06
    Initializing SD ...
    Date: 1/9/2021 17:41:17
    addFSToStorage: Added FS
    sd_addFilesystem: 0 200040ac SDIO 20004568 200040ac
    Total Size: 66002944 Used Size: 21889024
    Trying to open SPI config: 10 255 0 16000000
    addFSToStorage: Added FS
    sd_addFilesystem: 1 2000458c SPI10 20004a48 2000458c
    Total Size: 32014073856 Used Size: 524288
    SD initialized.
    
    Menu Options:
    	[0-9] - Set Active Storage
    	l - List files on disk
    	e - Erase files on disk
    	s - Start Logging data (Restarting logger will append records to existing log)
    	x - Stop Logging data
    	d - Dump Log
    	h - Menu
    
    CMD: 1002(OPEN_SESSION)l: 16 T:0 : 1
    RESP:2001(RSP:OK)l: 16 T:0 : 1
    CMD: 1001(GET_DEVICE_INFO)l: 12 T:1
    RESP:2001(RSP:OK)l: 12 T:1
    CMD: 1014(GET_DEVICE_PROP_DESC)l: 16 T:2 : d402
    RESP:2001(RSP:OK)l: 16 T:2 : d402
    CMD: 1004(GET_STORAGE_IDS)l: 12 T:3
    RESP:2001(RSP:OK)l: 12 T:3
    CMD: 1005(GET_STORAGE_INFO)l: 16 T:4 : 10001
    
    
    }}}}}}}}} SDMTPClass::usedSizeCB called 20004a6c 0 536887468 cs:254 ft:16
    2
    65537 0 name:SDIO
    
    
    }}}}}}}}} SDMTPClass::usedSizeCB called 20004a6c 0 536887468 cs:254 ft:16
    1
    65537 0 name:SDIO
    RESP:2001(RSP:OK)l: 16 T:4 : 10001
    CMD: 1005(GET_STORAGE_INFO)l: 16 T:5 : 20001
    
    
    }}}}}}}}} SDMTPClass::usedSizeCB called 20004a6c 1 536888716 cs:10 ft:32
    10016
    131073 1 name:SPI10
    
    
    }}}}}}}}} SDMTPClass::usedSizeCB called 20004a6c 1 536888716 cs:10 ft:32
    7143
    131073 1 name:SPI10
    RESP:2001(RSP:OK)l: 16 T:5 : 20001
    Two SD drives one is internal the other with CS=10...

    When I try to enumerate the top level Teensy Object, the used size is called like 2 or 3 times for each one of these.
    The internal small FAT16 one takes like: 1 ms

    The larger 32GB Fat32 external drive is taking like: 10 seconds, 7 seconds, 10 seconds... Yes seconds...

    So will debug to see why the values are not cached... And/Or may try to do some stuff where I will tell MTP a lie and when it asks for free space... If I suspect it will slow... (Size > X Fat32...) then say 50% free... To see if I can quickly get the drives to work in MTP.

    MTP will timeout in many of these cases.

  2. #502
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    15,096
    @KurtE not sure if this is maintained - certainly not without 'work' ? But is part of the design.

    FS Information Sector :: wikipedia.org/wiki/Design_of_the_FAT_file_system
    The "FS Information Sector" was introduced in FAT32[42] for speeding up access times of certain operations (in particular, getting the amount of free space). It is located at a logical sector number specified in the FAT32 EBPB boot record at position 0x030 (usually logical sector 1, immediately after the boot record itself).
    Code:
    Byte offset	____ Length (bytes)	______ Contents
        0x1E8	____ 4	    _____________ Last known number of free data clusters on the volume, or 0xFFFFFFFF if unknown. Should be set to 0xFFFFFFFF during format and updated by the operating system later on. Must not be absolutely relied upon to be correct in all scenarios. Before using this value, the operating system should sanity check this value to be less than or equal to the volume's count of clusters.
    So it can be 'cached' and stored On Disk

  3. #503
    Senior Member wwatson's Avatar
    Join Date
    Aug 2017
    Posts
    538
    Quote Originally Posted by defragster View Post
    For some reason the TINY partition is FAT16 - not supported?

    Cleaned that for fresh use to 18GB FAT32 and 100GB NTFS.

    Can't build now as the WIP FS updates have broken the DiskIO build it seems?
    As far as I know NTFS is not supported with SdFat only Fat16, Fat32 and EXFat. I never tried using Fat16. Going to wait until things settle and stabilize as per Paul and Mike. There are plenty of other things to work with like speeding up read and writes in MSC. @KurtE has already given clues as how to do this with the msReadSectorsWithCB() method. But still is fun to forget about work and play

  4. #504
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    15,096
    Quote Originally Posted by wwatson View Post
    As far as I know NTFS is not supported with SdFat only Fat16, Fat32 and EXFat. I never tried using Fat16. Going to wait until things settle and stabilize as per Paul and Mike. There are plenty of other things to work with like speeding up read and writes in MSC. @KurtE has already given clues as how to do this with the msReadSectorsWithCB() method. But still is fun to forget about work and play
    Opps - had the list right in front of me ... and let it default to NTFS on PC formatting ... will go back to ExFAT as it SHOULD be

    <EDIT>: FIXED for future testing :: 18GB FAT32 and 100GB ExFAT

  5. #505
    Senior Member wwatson's Avatar
    Join Date
    Aug 2017
    Posts
    538
    Quote Originally Posted by defragster View Post
    Opps - had the list right in front of me ... and let it default to NTFS on PC formatting ... will go back to ExFAT as it SHOULD be

    <EDIT>: FIXED for future testing :: 18GB FAT32 and 100GB ExFAT
    Thanks for testing

  6. #506
    Senior Member+ KurtE's Avatar
    Join Date
    Jan 2014
    Posts
    9,576
    Quote Originally Posted by defragster View Post
    @KurtE not sure if this is maintained - certainly not without 'work' ? But is part of the design.

    FS Information Sector :: wikipedia.org/wiki/Design_of_the_FAT_file_system
    The "FS Information Sector" was introduced in FAT32[42] for speeding up access times of certain operations (in particular, getting the amount of free space). It is located at a logical sector number specified in the FAT32 EBPB boot record at position 0x030 (usually logical sector 1, immediately after the boot record itself).
    Code:
    Byte offset	____ Length (bytes)	______ Contents
        0x1E8	____ 4	    _____________ Last known number of free data clusters on the volume, or 0xFFFFFFFF if unknown. Should be set to 0xFFFFFFFF during format and updated by the operating system later on. Must not be absolutely relied upon to be correct in all scenarios. Before using this value, the operating system should sanity check this value to be less than or equal to the volume's count of clusters.
    So it can be 'cached' and stored On Disk
    Thanks, earlier I was experimenting with at least reading this... Through MSC stuff Also as mentioned, I had/have speed ups in the MSC case to reading in the FAT sectors, by read me in N sectors or the like and call me with each one, such that we can get the data as quickly as possible...

    I will probably play with using some of these techniques. in this SD case as well.

    However I first want to see why the asking for it multiple times the option in SD Library to keep this data is not...
    That issSdFatConfig file has:
    Code:
    //------------------------------------------------------------------------------
    /**
     * Set MAINTAIN_FREE_CLUSTER_COUNT nonzero to keep the count of free clusters
     * updated.  This will increase the speed of the freeClusterCount() call
     * after the first call.  Extra flash will be required.
     */
    #ifdef __arm__
    #define MAINTAIN_FREE_CLUSTER_COUNT 1
    #else
    #define MAINTAIN_FREE_CLUSTER_COUNT 0
    #endif
    Maybe need to check if I somehow overwrote this?

  7. #507
    Senior Member+ KurtE's Avatar
    Join Date
    Jan 2014
    Posts
    9,576
    Quote Originally Posted by KurtE View Post
    That issSdFatConfig file has:
    Code:
    //------------------------------------------------------------------------------
    /**
     * Set MAINTAIN_FREE_CLUSTER_COUNT nonzero to keep the count of free clusters
     * updated.  This will increase the speed of the freeClusterCount() call
     * after the first call.  Extra flash will be required.
     */
    #ifdef __arm__
    #define MAINTAIN_FREE_CLUSTER_COUNT 1
    #else
    #define MAINTAIN_FREE_CLUSTER_COUNT 0
    #endif
    Maybe need to check if I somehow overwrote this?
    Aargh...

    My Sublime project was looking at Arduino1.8.13, but I am using Arduino1.8.15 ... Double Argh!

    The default one just has: #define MAINTAIN_FREE_CLUSTER_COUNT 0


    @Paul - Again looks like our current version of SDFat(beta) is somewhere between your library version and the current one up at: https://github.com/greiman/SdFat-beta
    Yours is 13 commits behind... But the stuff I saw in the build was neither version...

    The current one up on greiman

    Has stuff in it now like:
    Code:
    #ifndef MAINTAIN_FREE_CLUSTER_COUNT
    #define MAINTAIN_FREE_CLUSTER_COUNT 0
    #endif  // MAINTAIN_FREE_CLUSTER_COUNT
    I don't see anywhere in this config file where it has something like a __has_include.... to override, so my assumption (and from previous conversations with him), is that he is
    expecting overrides to be done things like Makefiles.

    Not sure what approach yoi may want to take...

    But before I shoot myself in foot again, I think I will some big warning message to building certain files if that option is not set...

  8. #508
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    15,096
    Quote Originally Posted by KurtE View Post
    Aargh...

    My Sublime project was looking at Arduino1.8.13, but I am using Arduino1.8.15 ... Double Argh!

    ...
    Opps ... I just started using projects and now having new TD installs ... I was alert and copied the file "Teensy_18.15_154.sublime-project" for beta TD 1.55 and edited the path stored inside so I could open it with: "Teensy_18.15_155.sublime-project"

    Now Beta #2 and IDE 1.8.16 pending ...

  9. #509
    Senior Member+ KurtE's Avatar
    Join Date
    Jan 2014
    Posts
    9,576
    Secondary note in our current sets of code base

    I now have in the code:
    Code:
    #include <SDMTPClass.h>
    #include <mscFS.h>
    
    #if defined(MAINTAIN_FREE_CLUSTER_COUNT) && (MAINTAIN_FREE_CLUSTER_COUNT == 0)
    #warning "###############################################################"
    #warning "# MAINTAIN_FREE_CLUSTER_COUNT is 0 in SdFatConfig.h           #"
    #warning "# Large Fat32 SD cards will likely fail to work properly      #"
    #warning "###############################################################"
    #endif
    So hopefully I (and others) wont chase it around again and again and again ...

  10. #510
    Senior Member+ KurtE's Avatar
    Join Date
    Jan 2014
    Posts
    9,576
    Quote Originally Posted by KurtE View Post
    Thanks, earlier I was experimenting with at least reading this... Through MSC stuff Also as mentioned, I had/have speed ups in the MSC case to reading in the FAT sectors, by read me in N sectors or the like and call me with each one, such that we can get the data as quickly as possible...

    I will probably play with using some of these techniques. in this SD case as well.

    However I first want to see why the asking for it multiple times the option in SD Library to keep this data is not...
    That issSdFatConfig file has:
    [CODE]//------------------------------------------------------------------------------
    /**
    * Set MAINTAIN_FREE_CLUSTER_COUNT nonzero to keep the count of free clusters
    * updated. This will increase the speed of the freeClusterCount() call
    * after the first call. Extra flash will be required.
    */
    Quick follow up here. As I mentioned we experimented some with the Information sector.
    The code is in the MSC stuff: UsbMscFat\src\PFsLib\pfsvolume.cpp

    Code:
    uint32_t PFsVolume::getFSInfoSectorFreeClusterCount() {
      uint8_t sector_buffer[512];
      if (fatType() != FAT_TYPE_FAT32) return (uint32_t)-1;
    
      // We could probably avoid this read if our class remembered the starting sector number for the partition...
      if (!m_blockDev->readSector(0, sector_buffer)) return (uint32_t)-1;
      MbrSector_t *mbr = reinterpret_cast<MbrSector_t*>(sector_buffer);
      MbrPart_t *pt = &mbr->part[m_part - 1];
      BpbFat32_t* bpb;
      if ((pt->type != 11) && (pt->type != 12))  return (uint32_t)-1;
    
      uint32_t volumeStartSector = getLe32(pt->relativeSectors);
      if (!m_blockDev->readSector(volumeStartSector, sector_buffer)) return (uint32_t)-1;
      pbs_t *pbs = reinterpret_cast<pbs_t*> (sector_buffer);
      bpb = reinterpret_cast<BpbFat32_t*>(pbs->bpb);
      
      //Serial.println("\nReadFat32InfoSectorFree BpbFat32_t sector");
      //dump_hexbytes(sector_buffer, 512);
      uint16_t infoSector = getLe16(bpb->fat32FSInfoSector); 
    
      // I am assuming this sector is based off of the volumeStartSector... So try reading from there.
      //Serial.printf("Try to read Info sector (%u)\n", infoSector); Serial.flush(); 
      if (!m_blockDev->readSector(volumeStartSector+infoSector, sector_buffer)) return (uint32_t)-1;
      //dump_hexbytes(sector_buffer, 512);
      FsInfo_t *pfsi = reinterpret_cast<FsInfo_t*>(sector_buffer);
    
      // check signatures:
      if (getLe32(pfsi->leadSignature) !=  FSINFO_LEAD_SIGNATURE) Serial.println("Lead Sig wrong");
      if (getLe32(pfsi->structSignature) !=  FSINFO_STRUCT_SIGNATURE) Serial.println("struct Sig wrong");    
      if (getLe32(pfsi->trailSignature) !=  FSINFO_TRAIL_SIGNATURE) Serial.println("Trail Sig wrong");    
      uint32_t free_count = getLe32(pfsi->freeCount);
      return free_count;
    
    
    }
    
    bool PFsVolume::setUpdateFSInfoSectorFreeClusterCount(uint32_t free_count) {
      uint8_t sector_buffer[512];
      if (fatType() != FAT_TYPE_FAT32) return (uint32_t)false;
    
      if (free_count == (uint32_t)-1) free_count = freeClusterCount();
    
      // We could probably avoid this read if our class remembered the starting sector number for the partition...
      if (!m_blockDev->readSector(0, sector_buffer)) return (uint32_t)-1;
      MbrSector_t *mbr = reinterpret_cast<MbrSector_t*>(sector_buffer);
      MbrPart_t *pt = &mbr->part[m_part - 1];
      BpbFat32_t* bpb;
      if ((pt->type != 11) && (pt->type != 12))  return (uint32_t)-1;
    
      uint32_t volumeStartSector = getLe32(pt->relativeSectors);
      if (!m_blockDev->readSector(volumeStartSector, sector_buffer)) return (uint32_t)-1;
      pbs_t *pbs = reinterpret_cast<pbs_t*> (sector_buffer);
      bpb = reinterpret_cast<BpbFat32_t*>(pbs->bpb);
      
      //Serial.println("\nReadFat32InfoSectorFree BpbFat32_t sector");
      //dump_hexbytes(sector_buffer, 512);
      uint16_t infoSector = getLe16(bpb->fat32FSInfoSector); 
    
      // OK we now need to fill in the the sector with the appropriate information...
      // Not sure if we should read it first or just blast out new data...
      FsInfo_t *pfsi = reinterpret_cast<FsInfo_t*>(sector_buffer);
      memset(sector_buffer, 0 , 512);
    
      // write FSINFO sector and backup
      setLe32(pfsi->leadSignature, FSINFO_LEAD_SIGNATURE);
      setLe32(pfsi->structSignature, FSINFO_STRUCT_SIGNATURE);
      setLe32(pfsi->freeCount, free_count);
      setLe32(pfsi->nextFree, 0XFFFFFFFF);
      setLe32(pfsi->trailSignature, FSINFO_TRAIL_SIGNATURE);
      if (!m_blockDev->writeSector(volumeStartSector+infoSector, sector_buffer)) return (uint32_t) false;
      return true;
    }
    Not sure if we have any place trying to use these currently...

  11. #511
    Senior Member wwatson's Avatar
    Join Date
    Aug 2017
    Posts
    538
    @KurtE - I actually was playing with this this weekend before all of the recent updates. On a FAT32 formatted USB or SDIO drive I was getting the delay when using MSCFormatter or the Linux FAT32 formatting. Then I did all of the new updates with TD1.55B2 and no matter what I do including setting '#define MAINTAIN_FREE_CLUSTER_COUNT 0' I cannot duplicate the delay. Probably missing something. Tested with diskIOMB and diskIOTesting which actually would show the delay but now there is no delay, Really confused here

    Edit: Not sure where all of you are with this but am quietly working a non-blocking 'MassStorageDriver.cpp'. Made some Interesting advancements today. Testing is next

  12. #512
    Senior Member+ KurtE's Avatar
    Join Date
    Jan 2014
    Posts
    9,576
    Thanks will start playing/testing with the MSC stuff soon. I am seeing these issues with SD cards plugged in with SPI. Yes with MSC we played with a few different ways. Will see how hard it will be to try it here as well.

    at a minimum will try deferring while the system is enumerating the drives.

  13. #513
    Senior Member wwatson's Avatar
    Join Date
    Aug 2017
    Posts
    538
    Quote Originally Posted by KurtE View Post
    Thanks will start playing/testing with the MSC stuff soon. I am seeing these issues with SD cards plugged in with SPI. Yes with MSC we played with a few different ways. Will see how hard it will be to try it here as well.

    at a minimum will try deferring while the system is enumerating the drives.
    Cool - Wait and watch

    Edit: Help where I can...
    Last edited by wwatson; 09-08-2021 at 02:42 AM.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •