Forum Rule: Always post complete source code & details to reproduce any issue!
Page 48 of 48 FirstFirst ... 38 46 47 48
Results 1,176 to 1,184 of 1184

Thread: Teensyduino File System Integration, including MTP and MSC

  1. #1176
    Senior Member+ mjs513's Avatar
    Join Date
    Jul 2014
    Location
    New York
    Posts
    8,119
    Ok guess its my turn. while @KurtE was working on all that I started looking at formatting again. Unfortunately a lot of whats in fatFormat and exFatFormat is dependent on the final API and where needed data elements are can be found. For instance, just as a start I did have to
    1. Modify findPartitions function to return a couple of extra data elements - makes it equivalent to what was in getPartitionInfo in the pfsLib library.
    2. Also remembered that we exposed a few extra data elements in FsVolume than what Bill has currently in SdFat. Specifically,
    Code:
    fatLength.
    sectorsPerClusterShift
    clusterHeapStartSector.
    Not sure how to handle this. Do we want to see if Bill is willing to add it or we keep it as a delta?

    This is really preliminary and probably wont make it into beta2 as things are changing too much to finalize.

  2. #1177
    Senior Member wwatson's Avatar
    Join Date
    Aug 2017
    Posts
    687
    @Paul, @KurtE, @mjs513 - Decided to test the latest with USBhost_t36. I tested with TD1.57B1 first Paul's latest and then Kurt's latest. found no issues.

    Here is output from Kurt's active branch of USBHost_t36. Tested with two thumb drives one with a single EXFAT partition and another with two partitions, EXFATP1 and FAT32P2.

    Here is the output:
    Code:
     === Drive index 0 found ===
           connected: 1
         initialized: 1
       USB Vendor ID: 0951
      USB Product ID: 1666
          HUB Number: 1
            HUB Port: 1
      Device Address: 2
    Removable Device: YES
            VendorID: Kingston
           ProductID: DataTraveler 3.0
          RevisionID:     
             Version: 6
        Sector Count: 241660915
         Sector size: 512
       Disk Capacity: 123730388480 Bytes
    
    Partition Table
            part,boot,bgnCHS[3],type,endCHS[3],start,length
    exFAT:  1,0,0x4,0x1,0x4,0x7,0xFE,0xC2,0xFF,2048,241657856
    pt_#0:  2,0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0,0
    pt_#0:  3,0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0,0
    pt_#0:  4,0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0,0
             < unused area starting at: 241659904 length 1011 >
    
    === File system (0) ===
            Manufacturer: Kingston
            Product: DataTraveler 3.0
            Serial Number: 4CEDFB74A418F4A0D96A029
    Volume name: 128GEXFAT
    Volume type: FAT64
    Cluster Size: 131072 bytes
    Volume size: 123723579392 bytes
     Space used: 419168256 bytes  (91 ms to compute)
    
    Files:
    2019-01-01 00:00     56863276 armaged.wav
    2019-01-01 00:07     43025728 DeffLepardAnimal.wav
    2019-01-01 00:07      7502713 StatusQuoWhateverYouWant.mp3
    2019-01-01 00:09     35639166 ThePaperboysMolinos.wav
    2019-01-01 00:10     36381082 ThePaperboysMolinosI'veJustSeenaFace.wav
    2019-01-01 00:56     94519514 YouareLazyJimmyBarnesJoeBonamassa.wav
    2022-01-01 00:00     32768000 32MEGfile.dat
    2019-01-01 00:00     13166660 YouareLazyJimmyBarnesJoeBonamassa.mp3
    2019-01-01 00:52     32768000 bench.dat
    2022-01-01 00:00     32768000 test.txt
    2022-01-01 00:00     32768000 copy.txt
    2022-01-01 00:00         2043 datalog.txt
    
    
     === Drive index 1 found ===
           connected: 1
         initialized: 1
       USB Vendor ID: 0951
      USB Product ID: 1666
          HUB Number: 1
            HUB Port: 4
      Device Address: 3
    Removable Device: YES
            VendorID: Kingston
           ProductID: DataTraveler 3.0
          RevisionID:     
             Version: 6
        Sector Count: 30218841
         Sector size: 512
       Disk Capacity: 15472046592 Bytes
    
    Partition Table
            part,boot,bgnCHS[3],type,endCHS[3],start,length
    exFAT:  1,0,0x4,0x1,0x4,0x7,0xFE,0xC2,0xFF,2048,17928192
    FAT32:  2,0,0xFE,0xC2,0xFF,0xB,0xFE,0xC2,0xFF,17930240,12288000
    pt_#0:  3,0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0,0
    pt_#0:  4,0,0x0,0x0,0x0,0x0,0x0,0x0,0x0,0,0
             < unused area starting at: 30218240 length 601 >
    
    === File system (1) ===
            Manufacturer: Kingston
            Product: DataTraveler 3.0
            Serial Number: D067E516489AF1B0B60F06F
    Volume name: EXFATP1
    Volume type: FAT64
    Cluster Size: 32768 bytes
    Volume size: 9176088576 bytes
     Space used: 103710720 bytes  (34 ms to compute)
    
    Files:
    2022-02-21 14:13        16976 pathname
    2022-02-21 14:13          974 pathname.c
    2022-02-21 13:23         2352 pathname.o
    2021-11-20 14:13        14916 playFSwav.cpp
    2021-11-20 14:11         3659 playFSwav.h
    2022-03-05 14:58        17184 qsortStructArray
    2022-03-05 15:01         1384 qsortStructArray.c
    2022-03-05 14:58         3224 qsortStructArray.o
    2022-03-05 14:20        17352 qsortTest
    2022-03-05 11:16         4418 qsortTest.c
    2022-03-05 09:35         5776 qsortTest.o
    2021-11-23 18:25       319806 Screenshot at 2021-11-23 17-25-06.png
    2021-12-05 18:39    102879394 YoureLazyJimmyBarnesJoeBonamassa.wav
    
    === File system (2) ===
    Volume name: FAT32P2
    Volume type: FAT32
    Cluster Size: 4096 bytes
    Volume size: 6279168000 bytes
     Space used: 334946304 bytes  (0 ms to compute)
    
    Files:
    2019-01-01 00:03   56862764 armaged.wav
    2019-01-01 00:02   43025216 DeffLepardAnimal.wav
    2019-01-01 00:03   60148386 StatusQuoWhateverYouWant.wav
    2019-01-01 00:03   35638654 ThePaperboysMolinos.wav
    2019-01-01 00:03   36380570 ThePaperboysMolinosI'veJustSeenaFace.wav
    2019-01-01 00:02  102878370 YoureLazyJimmyBarnesJoeBonamassa.wav
    
    
     *** enter any key to run again ***
    Looks like things are progressing

  3. #1178
    Senior Member+ KurtE's Avatar
    Join Date
    Jan 2014
    Posts
    10,323
    Good morning Paul and all:

    Status update for MSC integration into USBHost:

    I currently have A Pull request: https://github.com/PaulStoffregen/USBHost_t36/pull/89
    Which added in a lot of the functionality we had in the earlier integration attempt: Details from the PR:
    @mjs513 and myself have been extending the beta1 code to add some additional stuff. We still need to go through and rework some of the examples and test more, but thought I would go ahead and issue a PR with current WIP as to give you a head start on seeing what has changed so far:
    First pass to add the functionality we were using with the previous
    integration pass and work toward beta 2...
    Still retaining some support for the Beta1 way of starting up a filesystem:
    but added USBFilesystem::end() which closes the file system allows you to reuse the object.
    Added bool operator to USBFilesystem know if object is valid.
    Added USBFSBase class that works like hid class or the like, that USBFilesystem derives from. Which allows us to have
    multiple Filesystem implementations.
    Added code to create list of these, with ini() method added to Filesystem object to add it to a free list.
    When these objects are claimed, I remove them from this list and add them to list connected to drive. When drive
    goes away, I end() these objects and then put them back on free list. Note: I am contemplating not linking and unlinking
    but simply when walk whole lists each time.
    Added (maybe temporary) version of DriveInfo that tries to walk through partitions of a drive and print out information.
    Extended USBDrive::findPartition, including:
    a) support extended partitions
    b) reworked the GPT walking to not bail on first not known GUID but instead all GUIDS, until empty entry found
    b1) The enumeration does not check the guid. Instead returns it and lets caller decide if they understand the guid
    c) returns a little more information needed for format.
    d) return code was updated from bool to int, where 0 is no FS found, and other values to know if MBR, GPT, EXT partition
    Enumeration: Added method to USBDrive::startFilesystems
    Also added USBDrive::filesystemsStarted() method as ran into issues like, the DriveInfoAll sketch would enumerate all of the disks and Filesystems, and then wait to see if you want to run it again. Well suppose you unplug one disk and plug in another, we would still end up with same number of drives... So caching which disk we enumerated would know that something had changed. Now the enumeration code sets state and drive going way unsets it...
    Which first does begin() to make sure drive is ready, it then keeps calling findPartition until it receives
    back a return of 0.
    It then calls off to claim for each of the free USBFilesystem objects until one claims it or none...
    If FS claimed it removes the item from list and adds to the drives list.
    When drive goes away, it walks it's list and calls another member of the new FS base class which the current
    one calls end(). It also then removes the item from drive list and puts back on free.
    Since I issued the above Pull Request we (mostly Mike) have been working on format support. I have now pushed up new branch based on the branch associated with the PR:
    https://github.com/kurte/USBHost_t36...n_part2_format

    This includes a drop of a new class that Mike developed that contains the Format code. This code is still being updated and tested. Yesterday I added it as itís own individual class into the USBHost library (in that branch), I then added the Format method to the USBFilesystem class. It was defaulting to the base classes empty return false method.

    This implementation, creates an instance of the format class and then calls off to the a simplified format member which will try to format the volume to the same type (if possible), there may be some limitations depending on size of volume.

    I did run into some issues, with this, in that like the SD format code, I malloc the buffer to use. Ran into issue that the ReadSectors/WriteSectors methods did not handle this. And it corrupted the disks. So added in the check and calls to arm_dcache_ Ö Which is now working.

    As I mentioned we have a couple of sketches updated to use the newer enumeration, including with MTP_Teensy, and these now call off to use the format without changes to these sketches.

    Still being tested, and fixed.

    For example the next thing to fix, is an issue, that Suppose the Format does change the Filesystem information. I am currently holding a state variable in the USBDrive object that says the drive was enumerated. So my user code will not go through and enumerate again and for example won't in test cases show updated Partition information. So I probably need to update that state, when we enumerate again, I need to know that I am
    already holding onto a USBFilesystem object for that Partition. So probably don't try to claim another one. Probably want to add new method that says updatePartition, which will give it a chance to change things.

    Note: in this case I had already updated the USBFilesystem object at the completion of the format... So will see what all needs to be done.

  4. #1179
    Senior Member+ mjs513's Avatar
    Join Date
    Jul 2014
    Location
    New York
    Posts
    8,119
    Good after noon all.
    Using Kurt's latest branch with formatting decided to test the changes with my MTP_TEST_INTEGRITY sketch (not pushed yet with the updates)> Here is a dump of the storage list and what it looks like windows explorer
    Code:
    Dump Storage list(13)
    store:0 storage:10001 name:PgmIndx fs:2001210c pn:    (In program mem)
    store:1 storage:20001 name:BUILTIN fs:20014b98 pn:    (BUILTIN sd Card)
    store:2 storage:30001 name:EXT1 fs:2001506c pn:		  (Card on audio shield)
    store:3 storage:40001 name:RAM0 fs:20014990 pn:EXTMEM
    store:4 storage:50001 name:RAM1 fs:20014a5c pn:EXTMEM
    store:5 storage:60001 name:PROGM fs:200147ec pn:PROGRAM
    store:6 storage:70001 name:QSPI fs:200148b8 pn:W25Q128JV*M (DTR)
    store:7 storage:80001 name:MSC0 fs:200122ac pn:		  (MSC0/1 on USB: MBR fat32+exFAT)
    store:8 storage:90001 name:MSC1 fs:20012754 pn:
    store:9 storage:a0001 name:MSC2-GPTFAT32 fs:20012bfc pn:   (MSC2/3 on USB: GPT fat32+exFAT)
    store:10 storage:b0001 name:MSC3 fs:200130a4 pn:
    store:11 storage:c0001 name:MSC4-Part1 fs:2001354c pn:     (MS4/5 on USB SD Card: MBE fat32+exFAT)
    store:12 storage:d0001 name:MSC5-Part2 fs:200139f4 pn:
    Click image for larger version. 

Name:	Capture.jpg 
Views:	0 
Size:	36.3 KB 
ID:	28389

    Have not done exhaustive testing but it looking good - for several drives I tried the updated formatting code and seems to work (note changes to FsVolume no longer needed). Copying files and directories seem to be working to usb drives as well. Have also tested play back of audio files in windows from USB drives and that seem to be working.

  5. #1180
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    15,849
    Nice work! So, to catch up using TD 1.57b1:
    MAIN :: github.com/KurtE/MTP_Teensy
    and
    github.com/kurte/USBHost_t36/tree/fs_integration_part2_format

    With optional edit to CORES to support SERIAL still required under the prior overloaded 'everything' option? { will have to dig that up }

  6. #1181
    Senior Member+ KurtE's Avatar
    Join Date
    Jan 2014
    Posts
    10,323
    Quote Originally Posted by defragster View Post
    Nice work! So, to catch up using TD 1.57b1:
    MAIN :: github.com/KurtE/MTP_Teensy
    and
    github.com/kurte/USBHost_t36/tree/fs_integration_part2_format

    With optional edit to CORES to support SERIAL still required under the prior overloaded 'everything' option? { will have to dig that up }
    Yes - all of the above.
    Assumes Beta 1 install
    Plus the trees you mentioned.

    Note: I have a couple of USB examples converted. Both my CircuitMicroPython example plus under simplified... examples I have example 5...

    Still debugging some trying to tell user list was updated... code.

    Paul - In all of your not free time! also at some point would be interesting to talk events/notification... Like when a drive is inserted, can other sub-systems and user sketch get some form of notification.

    Like New USBFilesystem object is claimed, have ability for MTP to optionally automatically add the MTP storage list....

    But first would be good to get this level of functionality working properly and migrated back into the master branch.

  7. #1182
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    15,849
    Quote Originally Posted by KurtE View Post
    Yes - all of the above.
    Assumes Beta 1 install
    Plus the trees you mentioned.
    ...
    Thanks Kurt, it builds.
    Will have to get back to what was in progress in Feb and start with some simple Xfer examples for confirming valid file transfer.

    Browsing simple folders on T_4.1 SD works.
    Opening a TXT file of 244 or 756 bytes is FAST.

    Three folders with a 500 byte file show that file to take some LONG time 'Calculating' before the file opens: Measures 30 seconds delay
    >> "This PCTeensyMakeFiles 22048JanSRC_o222\bbb.txt"

  8. #1183
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    15,849
    Does this mean anything as a complete issue indication:
    Code:
    $$$ Failed to read Index record: 15 bytes Read: 0
    $$$ Failed to read Index record: 16 bytes Read: 0
    $$$ Failed to read Index record: 15 bytes Read: 0
    On current over large SD tried to copy what I thought was a smaller subdirectory and this over filled TyComm 24 MB buffer repeating those type lines ... and kept going long after for many non stop minutes.
    Total 107999 files of Size 1088487493 Bytes
    Bytes Used: 28699525120, Bytes Total:128144375808
    > Had to restart the Teensy to stop it as I could not get Edge Browser to respond while that was running.

    Will start with clean SD card and just the files from MakeFiles.ino to verify and test for base function and perhaps any simple repro's.

  9. #1184
    Senior Member+ mjs513's Avatar
    Join Date
    Jul 2014
    Location
    New York
    Posts
    8,119
    @defragster
    Hi Tim
    A little confused are you transfering files from the PC to the Builtinsd Card over MTP. I just transferred 104MB of audio files to the BUILTIN SDCard in about 14 seconds?

    Writing you 'B'ig File:
    Code:
    Start Big write of 25124864 Bytes............................................................
    Big write /0_bigfile.txt took 24.38 Sec for 25124864 Bytes : file3.size()=25122816
    	Big write KBytes per second 1030.55 
    
    Bytes Used: 43122688, Bytes Total:63831015424
    
    Delete with read verify all #bigfile's
    	Verify /0_bigfile.txt bytes 25122816 : ..................................................	GOOD! >>  bytes 25122816
    	Big read&compare KBytes per second 901.68
    and writing Kurt's Index file:
    Code:
    	Big write KBytes per second 360.94

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •