Forum Rule: Always post complete source code & details to reproduce any issue!
Results 1 to 17 of 17

Thread: MTP dependency issues w/Teensy 3.6 and SDFat 2.0.4

  1. #1
    Junior Member
    Join Date
    Feb 2021
    Posts
    12

    MTP dependency issues w/Teensy 3.6 and SDFat 2.0.4

    Dear community,

    I'm successfully using a Teensy 3.6 as a Unicode-compatible native MP3 player, and would now like to add MTP.
    However, I am running into several dependency issues. There seem to be two libraries: MTP and MTP-t4.
    Despite its name, MTP-t4 says it supports Teensy 3.x - and I would prefer to use it, as I read it supports MTP+Serial.

    With MTP I am running into the problem, that it requires SdFatSdioEX, which does not seem to exist in SDFat 2.0.4,
    which I am using in my main program. So I changed SdFatSdioEX into SdExFat, as in my main program, and File into
    ExFile. This seems to do it, however, the implementation of MTP does not seem to support SdFAT's UNICODE feature,
    which I am using in my main program (I do have a lot of foreign script files, which otherwise would not load and play
    correctly). I could go further this road, by implementing UTC2-UTF8 conversion methods for the filenames, but I am
    not sure if this has perspective.

    With MTP-t4 I am running into the problem that the "FS" type is not found (Storage.h). It seems to be implemented
    in the LittleFS library, which apparently is obligatory (IMHO not completely clear from the README, which I originally read
    as suggesting that LittleFS support is optional). I installed LittleFS, but the FS type is still not found, until I add an
    #include <LittleFS.h> into Storage.h. When I do this, I get the following:

    Code:
    C:\Users\tobia\Documents\Arduino\libraries\LittleFS-main\src/LittleFS.h: In member function 'virtual File LittleFSFile::openNextFile(uint8_t)':
    C:\Users\tobia\Documents\Arduino\libraries\LittleFS-main\src/LittleFS.h:164:51: error: no matching function for call to 'File::File(LittleFSFile*)'
         return File(new LittleFSFile(lfs, f, pathname));
    And other issues related to "File".
    I assume this is a version incompatibility with my SDFat 2.0.4. Do I need the SDFat-beta to run this?
    I installed SDFat-beta, but then my main program would not compile as the ExFile type is not found.
    Can I safely change ExFile into File without losing my long filename & Unicode functionality?
    If so, won't "LittleFS" etc. break sooner or later due to char16_t* vs char8* incompatibility, just like the original MTP?

    Not sure which path I should take. Thank you for any input.

    Thank you

  2. #2
    Senior Member+ KurtE's Avatar
    Join Date
    Jan 2014
    Posts
    9,563
    If it were me, I would install the latest Teensyduino beta from the thread: https://forum.pjrc.com/threads/66182...no-1-54-Beta-6
    This has some better support for SD and the like... Plus some of the stuff built into core. Hopefully more soon.
    But it does have things like updated FS.h, LittleFS a new version of SD library that is using a newer version of SD stuff...

    Not sure if the MTP_t4 up on @WMXZ works currently on T3.5? Maybe?

    The current branch that I have been playing with is in my own fork/branch: https://github.com/KurtE/MTP_t4/tree...d_object_large
    It has been awhile since I tried a T3.5, but a few days ago I did try a T3.6.

    I am not sure if that helps or not.

  3. #3
    Junior Member
    Join Date
    Feb 2021
    Posts
    12
    Great adviсe, thank you. With the beta Teensyduino the code actually compiles
    Serial is working, and there is even an MTP device showing up in Windows.
    Its contents are empty, but that is probably because I still need to implement something.
    I will have a closer look at the examples.
    Thanks again!
    Last edited by mcleinn; 02-20-2021 at 09:25 PM.

  4. #4
    Junior Member
    Join Date
    Feb 2021
    Posts
    12
    Ok, the example in MTP_t4 is working, but only if #define USE_EXFAT_UNICODE_NAMES 0 in ExFatConfig.h
    In this case, the filenames displayed in Windows are displaying all non-American letters as questionmarks.
    In other words, it looks like Unicode is not yet supported.

    This is the error when compiling mtp_test with USE_EXFAT_UNICODE_NAMES 1

    Code:
    Linking everything together...
    "C:\\Program Files (x86)\\Arduino\\hardware\\teensy/../tools/arm/bin/arm-none-eabi-gcc" -O2 -Wl,--gc-sections,--relax,--defsym=__rtc_localtime=1613869118 "-TC:\\Program Files (x86)\\Arduino\\hardware\\teensy\\avr\\cores\\teensy3/mk66fx1m0.ld" -mthumb -mcpu=cortex-m4 -mfloat-abi=hard -mfpu=fpv4-sp-d16 -fsingle-precision-constant -o "C:\\Users\\tobia\\AppData\\Local\\Temp\\arduino_build_650259/mtp-test.ino.elf" "C:\\Users\\tobia\\AppData\\Local\\Temp\\arduino_build_650259\\sketch\\mtp-test.ino.cpp.o" "C:\\Users\\tobia\\AppData\\Local\\Temp\\arduino_build_650259\\libraries\\SD\\SD.cpp.o" "C:\\Users\\tobia\\AppData\\Local\\Temp\\arduino_build_650259\\libraries\\SdFat\\FreeStack.cpp.o" "C:\\Users\\tobia\\AppData\\Local\\Temp\\arduino_build_650259\\libraries\\SdFat\\MinimumSerial.cpp.o" "C:\\Users\\tobia\\AppData\\Local\\Temp\\arduino_build_650259\\libraries\\SdFat\\ExFatLib\\ExFatDbg.cpp.o" "C:\\Users\\tobia\\AppData\\Local\\Temp\\arduino_build_650259\\libraries\\SdFat\\ExFatLib\\ExFatFile.cpp.o" "C:\\Users\\tobia\\AppData\\Local\\Temp\\arduino_build_650259\\libraries\\SdFat\\ExFatLib\\ExFatFilePrint.cpp.o" "C:\\Users\\tobia\\AppData\\Local\\Temp\\arduino_build_650259\\libraries\\SdFat\\ExFatLib\\ExFatFileWrite.cpp.o" "C:\\Users\\tobia\\AppData\\Local\\Temp\\arduino_build_650259\\libraries\\SdFat\\ExFatLib\\ExFatFormatter.cpp.o" "C:\\Users\\tobia\\AppData\\Local\\Temp\\arduino_build_650259\\libraries\\SdFat\\ExFatLib\\ExFatPartition.cpp.o" "C:\\Users\\tobia\\AppData\\Local\\Temp\\arduino_build_650259\\libraries\\SdFat\\ExFatLib\\ExFatVolume.cpp.o" "C:\\Users\\tobia\\AppData\\Local\\Temp\\arduino_build_650259\\libraries\\SdFat\\ExFatLib\\upcase.cpp.o" "C:\\Users\\tobia\\AppData\\Local\\Temp\\arduino_build_650259\\libraries\\SdFat\\FatLib\\FatDbg.cpp.o" "C:\\Users\\tobia\\AppData\\Local\\Temp\\arduino_build_650259\\libraries\\SdFat\\FatLib\\FatFile.cpp.o" "C:\\Users\\tobia\\AppData\\Local\\Temp\\arduino_build_650259\\libraries\\SdFat\\FatLib\\FatFileLFN.cpp.o" "C:\\Users\\tobia\\AppData\\Local\\Temp\\arduino_build_650259\\libraries\\SdFat\\FatLib\\FatFilePrint.cpp.o" "C:\\Users\\tobia\\AppData\\Local\\Temp\\arduino_build_650259\\libraries\\SdFat\\FatLib\\FatFileSFN.cpp.o" "C:\\Users\\tobia\\AppData\\Local\\Temp\\arduino_build_650259\\libraries\\SdFat\\FatLib\\FatFormatter.cpp.o" "C:\\Users\\tobia\\AppData\\Local\\Temp\\arduino_build_650259\\libraries\\SdFat\\FatLib\\FatPartition.cpp.o" "C:\\Users\\tobia\\AppData\\Local\\Temp\\arduino_build_650259\\libraries\\SdFat\\FatLib\\FatVolume.cpp.o" "C:\\Users\\tobia\\AppData\\Local\\Temp\\arduino_build_650259\\libraries\\SdFat\\FsLib\\FsFile.cpp.o" "C:\\Users\\tobia\\AppData\\Local\\Temp\\arduino_build_650259\\libraries\\SdFat\\FsLib\\FsNew.cpp.o" "C:\\Users\\tobia\\AppData\\Local\\Temp\\arduino_build_650259\\libraries\\SdFat\\FsLib\\FsVolume.cpp.o" "C:\\Users\\tobia\\AppData\\Local\\Temp\\arduino_build_650259\\libraries\\SdFat\\SdCard\\SdCardInfo.cpp.o" "C:\\Users\\tobia\\AppData\\Local\\Temp\\arduino_build_650259\\libraries\\SdFat\\SdCard\\SdSpiCard.cpp.o" "C:\\Users\\tobia\\AppData\\Local\\Temp\\arduino_build_650259\\libraries\\SdFat\\SdCard\\SdioTeensy.cpp.o" "C:\\Users\\tobia\\AppData\\Local\\Temp\\arduino_build_650259\\libraries\\SdFat\\SpiDriver\\SdSpiArtemis.cpp.o" "C:\\Users\\tobia\\AppData\\Local\\Temp\\arduino_build_650259\\libraries\\SdFat\\SpiDriver\\SdSpiChipSelect.cpp.o" "C:\\Users\\tobia\\AppData\\Local\\Temp\\arduino_build_650259\\libraries\\SdFat\\SpiDriver\\SdSpiDue.cpp.o" "C:\\Users\\tobia\\AppData\\Local\\Temp\\arduino_build_650259\\libraries\\SdFat\\SpiDriver\\SdSpiESP.cpp.o" "C:\\Users\\tobia\\AppData\\Local\\Temp\\arduino_build_650259\\libraries\\SdFat\\SpiDriver\\SdSpiParticle.cpp.o" "C:\\Users\\tobia\\AppData\\Local\\Temp\\arduino_build_650259\\libraries\\SdFat\\SpiDriver\\SdSpiSTM32.cpp.o" "C:\\Users\\tobia\\AppData\\Local\\Temp\\arduino_build_650259\\libraries\\SdFat\\SpiDriver\\SdSpiTeensy3.cpp.o" "C:\\Users\\tobia\\AppData\\Local\\Temp\\arduino_build_650259\\libraries\\SdFat\\common\\FmtNumber.cpp.o" "C:\\Users\\tobia\\AppData\\Local\\Temp\\arduino_build_650259\\libraries\\SdFat\\common\\FsCache.cpp.o" "C:\\Users\\tobia\\AppData\\Local\\Temp\\arduino_build_650259\\libraries\\SdFat\\common\\FsDateTime.cpp.o" "C:\\Users\\tobia\\AppData\\Local\\Temp\\arduino_build_650259\\libraries\\SdFat\\common\\FsStructs.cpp.o" "C:\\Users\\tobia\\AppData\\Local\\Temp\\arduino_build_650259\\libraries\\SdFat\\common\\PrintBasic.cpp.o" "C:\\Users\\tobia\\AppData\\Local\\Temp\\arduino_build_650259\\libraries\\SdFat\\common\\SysCallBareUno.cpp.o" "C:\\Users\\tobia\\AppData\\Local\\Temp\\arduino_build_650259\\libraries\\SdFat\\iostream\\StdioStream.cpp.o" "C:\\Users\\tobia\\AppData\\Local\\Temp\\arduino_build_650259\\libraries\\SdFat\\iostream\\StreamBaseClass.cpp.o" "C:\\Users\\tobia\\AppData\\Local\\Temp\\arduino_build_650259\\libraries\\SdFat\\iostream\\istream.cpp.o" "C:\\Users\\tobia\\AppData\\Local\\Temp\\arduino_build_650259\\libraries\\SdFat\\iostream\\ostream.cpp.o" "C:\\Users\\tobia\\AppData\\Local\\Temp\\arduino_build_650259\\libraries\\SPI\\SPI.cpp.o" "C:\\Users\\tobia\\AppData\\Local\\Temp\\arduino_build_650259\\libraries\\MTP_t4-master\\MTP.cpp.o" "C:\\Users\\tobia\\AppData\\Local\\Temp\\arduino_build_650259\\libraries\\MTP_t4-master\\Storage.cpp.o" "C:\\Users\\tobia\\AppData\\Local\\Temp\\arduino_build_650259\\libraries\\LittleFS\\LittleFS.cpp.o" "C:\\Users\\tobia\\AppData\\Local\\Temp\\arduino_build_650259\\libraries\\LittleFS\\LittleFS_NAND.cpp.o" "C:\\Users\\tobia\\AppData\\Local\\Temp\\arduino_build_650259\\libraries\\LittleFS\\littlefs\\lfs.c.o" "C:\\Users\\tobia\\AppData\\Local\\Temp\\arduino_build_650259\\libraries\\LittleFS\\littlefs\\lfs_util.c.o" "C:\\Users\\tobia\\AppData\\Local\\Temp\\arduino_build_650259\\libraries\\Time\\DateStrings.cpp.o" "C:\\Users\\tobia\\AppData\\Local\\Temp\\arduino_build_650259\\libraries\\Time\\Time.cpp.o" "C:\\Users\\tobia\\AppData\\Local\\Temp\\arduino_build_650259/core\\core.a" "-LC:\\Users\\tobia\\AppData\\Local\\Temp\\arduino_build_650259" -larm_cortexM4lf_math -lm -lstdc++
    C:\Users\tobia\AppData\Local\Temp\arduino_build_650259\sketch\mtp-test.ino.cpp.o: In function `FsVolume::mkdir(char const*, bool)':
    
    C:\Program Files (x86)\Arduino\hardware\teensy\avr\libraries\SdFat\src/FsLib/FsVolume.h:185: undefined reference to `ExFatVolume::mkdir(char const*, bool)'
    
    C:\Users\tobia\AppData\Local\Temp\arduino_build_650259\sketch\mtp-test.ino.cpp.o: In function `FsBaseFile::getName(char*, unsigned int)':
    
    C:\Program Files (x86)\Arduino\hardware\teensy\avr\libraries\SdFat\src/FsLib/FsFile.h:233: undefined reference to `ExFatFile::getName(char*, unsigned int)'
    
    C:\Users\tobia\AppData\Local\Temp\arduino_build_650259\sketch\mtp-test.ino.cpp.o: In function `FsVolume::rmdir(char const*)':
    
    C:\Program Files (x86)\Arduino\hardware\teensy\avr\libraries\SdFat\src/FsLib/FsVolume.h:232: undefined reference to `ExFatVolume::rmdir(char const*)'
    
    C:\Users\tobia\AppData\Local\Temp\arduino_build_650259\sketch\mtp-test.ino.cpp.o: In function `FsVolume::rename(char const*, char const*)':
    
    C:\Program Files (x86)\Arduino\hardware\teensy\avr\libraries\SdFat\src/FsLib/FsVolume.h:220: undefined reference to `ExFatVolume::rename(char const*, char const*)'
    
    C:\Users\tobia\AppData\Local\Temp\arduino_build_650259\sketch\mtp-test.ino.cpp.o: In function `FsVolume::remove(char const*)':
    
    C:\Program Files (x86)\Arduino\hardware\teensy\avr\libraries\SdFat\src/FsLib/FsVolume.h:202: undefined reference to `ExFatVolume::remove(char const*)'
    
    C:\Users\tobia\AppData\Local\Temp\arduino_build_650259\sketch\mtp-test.ino.cpp.o: In function `FsVolume::exists(char const*)':
    
    C:\Program Files (x86)\Arduino\hardware\teensy\avr\libraries\SdFat\src/FsLib/FsVolume.h:103: undefined reference to `ExFatVolume::exists(char const*)'
    
    C:\Program Files (x86)\Arduino\hardware\teensy\avr\libraries\SdFat\src/FsLib/FsVolume.h:103: undefined reference to `ExFatVolume::exists(char const*)'
    
    C:\Users\tobia\AppData\Local\Temp\arduino_build_650259\sketch\mtp-test.ino.cpp.o: In function `FsVolume::remove(char const*)':
    
    C:\Program Files (x86)\Arduino\hardware\teensy\avr\libraries\SdFat\src/FsLib/FsVolume.h:202: undefined reference to `ExFatVolume::remove(char const*)'
    
    C:\Users\tobia\AppData\Local\Temp\arduino_build_650259\libraries\SdFat\FsLib\FsFile.cpp.o: In function `FsBaseFile::open(FsVolume*, char const*, int)':
    
    C:\Program Files (x86)\Arduino\hardware\teensy\avr\libraries\SdFat\src\FsLib/FsFile.cpp:95: undefined reference to `ExFatFile::open(ExFatVolume*, char const*, int)'
    
    collect2.exe: error: ld returned 1 exit status

  5. #5
    Senior Member+ KurtE's Avatar
    Join Date
    Jan 2014
    Posts
    9,563
    Sorry, I am not sure if the main owner @WMXZ of the library has looked at Unicode stuff yet.

    I will try to take a quick look, but busy playing with other stuff

  6. #6
    Junior Member
    Join Date
    Feb 2021
    Posts
    12
    WMXZ library in storage.h seems to be char* focused, yes. I guess this would need to be changed.
    At the very least it is not compatible with the Unicode mode of SdFat.
    In the example, it gets an SDClass from the SD wrapper (in the Teensy libraries), and that also does not seem to support Unicode.
    And FS.h in Cores?
    Last edited by mcleinn; 02-20-2021 at 11:18 PM.

  7. #7
    Senior Member
    Join Date
    Jul 2014
    Posts
    3,317
    Quote Originally Posted by mcleinn View Post
    WMXZ library in storage.h seems to be char* focused, yes. I guess this would need to be changed.
    At the very least it is not compatible with the Unicode mode of SdFat.
    In the example, it gets an SDClass from the SD wrapper (in the Teensy libraries), and that also does not seem to support Unicode.
    And FS.h in Cores?
    I'm using SDFat as supplied via Teensyduino, which has in ExFatConfig.h as default "#define USE_EXFAT_UNICODE_NAMES 0 "
    It works for me and I'm sorry for non-latin charactersets.
    However, if someone develops the code further to include characters that I cannot read and verify, he will be welcome.
    However, it must also work with char * to be compatible with my programs.

  8. #8
    Junior Member
    Join Date
    Feb 2021
    Posts
    12
    Ok, I see. Rewriting all the mentioned libraries (at least SD wrapper and MTP-t4) for additional char16_t support seems quite some work.
    But I am considering writing a small patch for SdFat, which would implement the eight missing char* functions in Unicode mode, so it would at least compile.
    The functions could do transformation between UCS-2 and UTF-8. Ideally the existing libraries would work that way, but I need to give it a try
    Last edited by mcleinn; 02-21-2021 at 04:58 PM.

  9. #9
    Senior Member
    Join Date
    Nov 2012
    Posts
    318
    I wrote SdFat with the intention of better Unicode support but the underlying Arduino system make this difficult. Even supporting 8-bit as opposed to 7-bit ASCII is difficult since this require ANSI code pages.

    exFAT defines so much of how 16-bit characters work with it's Up-case Table so I have not added more features.

    I planned to redo FAT LFN to support Unicode but now that is low priority. FAT has a somewhat different view of Unicode than exFAT.

    Yes I never went back to implement the missing char* functions in Unicode mode. I have worked a bit with users to try some things in Unicode but they gave up. There are not Arduino libraries for Unicode on display devices, no Unicode Strings...

    Chan's FatFs makes a better attempt at Unicode support than SdFat.

    Few if any embedded RTOSs have good Unicode support so I can't fault Arduino.

  10. #10
    Senior Member PaulStoffregen's Avatar
    Join Date
    Nov 2012
    Posts
    25,049
    Arduino kinda does support UTF8. If you type or copy non-latin characters into strings, the Arduino IDE does end up putting UTF8 encoded strings into memory. And if you transmit UTF8 encoded data to the serial monitor, it does properly display non-latin characters.

    Very few libraries do any sort of UTF8 specific processing, but much of the idea of UTF8 is that you can usually just treat it like a stream of 8 bit bytes.

    Teensy's USB Keyboard.print() does parse UTF8 and translates characters to their mapping on various non-US keyboard mappings.

    If you'd like to support UTF8 strings in SdFat, I could contribute that UTF8 to Unicode code.

  11. #11
    Junior Member
    Join Date
    Feb 2021
    Posts
    12
    Thank you for the offer, Paul!

    In the last hour, I already implemented the missing eight functions in SdFAT(Core), by calling UTF8<->UCS2 conversion methods.
    The code compiles. Good, this is a step forward.

    The MTP device is displayed in the host system. However, Windows only shows the Latin letters of the filenames - all non-Latin (non default charset) letters are written with two erroneous characters. In other words, Windows does not recognize the strings sent by MTP as UTF-8 and just displays them assuming a regular 8-bit charset.

    I would have to look more closely at the point where the strings are sent to the host system, if they can somehow be marked as UTF-8 (or UCS-2)
    Or maybe you would know? Unlike with Serial, it seems not enough just to send UTF8 characters.

    If you tell me where you keep your SdFat branch for Core on Github, I could branch it, so you can see my additions.

    Unicode conversion used based on
    unicode.c: https://searchcode.com/codesearch/raw/19128073/
    unicode.h: https://searchcode.com/file/19128066...lib/unicode.h/
    (GNU license, not MIT; so maybe better to rewrite?)

    I am already using this successfully to convert UCS-2 from SdFat to UTF-8 for Serial.println etc. - so the routines themselves are OK.

    Code:
    /*      Function Name:  UCS2toUTF8
     *      Description:    Conversion of an UCS2 coded string into UTF8.
     *      Arguments:      str16     - The UCS2 coded string
     *      Returns:        The UTF8 coded result string. 
     */
    void UCS2toUTF8(ExChar16_t *str16, char *str8)
    {
        char *p8, *p;
        ExChar16_t *p16;
    
        p8  = str8;   // reset pointers
        p16 = str16;
    
        while(*p16 > 0) {
            p = (char*)p16;
            if(*p16 < 0x0080) {
                *p8 = p[0];
                p8++;
            }
            else if(*p16 < 0x0800) {
                *p8 = 0xc0 | ((p[1] & 0x7) << 2) | ((p[0] & 0xc0) >> 6);
                p8++;
                *p8 = 0x80 | (p[0] & 0x3f);
                p8++;
            }
            else {
                *p8 = 0xe0 | ((p[1] & 0xf0) >> 4);
                p8++;
                *p8 = 0x80 | ((p[1] & 0xf) << 2) | ((p[0] & 0xc0) >> 6);
                p8++;
                *p8 = 0x80 | (p[0] & 0x3f);
                p8++; 
            }
            p16++;
        }
        *p8 = 0;  // terminate UTF8 string
    }
    Code:
    /*	Function Name:	UTF8toUCS2
     *	Description: 	Conversion of an UTF8 coded string into UCS2/UNICODE.
     *			If the encoding of the character is not representable
     *			in two bytes, the tilde sign ~ is written into the
     *			result string at this position.
     *			For an illegal UTF8 code an asterix * is stored in
     *			the result string.
     *	Arguments:	str	- The UTF8 coded string
     *	Returns:	The UCS2 coded result string. The allocated memory
     *			for this string has to be freed by the caller!
     *			The result string is stored independent of the
     *			architecture in the high byte/low byte order and is
     *			compatible to the XChar2b format! Type casting is valid.
     *			ExChar16_t is used to increase the performance.
     */
    ExChar16_t *UTF8toUCS2(const char *str)
    {
       ExChar16_t *str16, *p16, testINTEL = 0, c16;
       int    clen, cInString;
       char   *p;
    
      /* In the first step we try to determine the string
       * length in characters.
       */
       cInString = mbStrLen(str);
       cInString++;		/* For the terminating null */
    
       /* Now we need memory for our conversion result */
    
       str16 = (ExChar16_t *)malloc(cInString * sizeof(ExChar16_t));
       if (str16)
       {
         /* Start the conversion: Determine the number of bytes
          * for the next character, decode it and store the
          * result in our result string
          */
          p   = str;
          p16 = str16;
          while ((clen = mbCharLen(p)) > 0)
          {
    	 switch (clen)
    	 {
    	    case 1: *p16 = (ExChar16_t)*p;
    		    break;
    
    	    case 2: c16 = (p[1] & 0x3f) + ((p[0] & 0x1f) << 6);
    		    *p16 = ((c16 > 0x7f) && ((p[1] & 0xC0) == 0x80)) ? c16 : '*';
    		    break;
    
    	    case 3: c16 = (p[2] & 0x3f) + ((p[1] & 0x3f) << 6) + ((p[0] & 0xf) << 12);
    		    *p16 = ((c16 > 0x7ff) && ((p[1] & 0xC0) == 0x80) && ((p[2] & 0xC0) == 0x80)) ? c16 : '*';
    		    break;
    
    	    default: *p16 = '~';	/* character code is greater than 0xffff */
    	 }
    	 p16++;				/* Jump to the next character */
    	 p += clen;
          }
          *p16 = 0;				/* String termination */
    
          /* Swap the bytes, if we are on a machine with an INTEL architecture */
    
          if (*((char *)&testINTEL))
          {
    	 char *src, *dest, c;
    
    	 src = dest = (char *)str16;
    	 src++;
    	 while (*src || *dest)
    	 {
    	    c = *dest; *dest = *src; *src = c;
    	    src += 2; dest += 2;
    	 }
          }
          return str16;
       }
       return NULL;
    }
    ExFatVolume.h
    Code:
    #if  USE_EXFAT_UNICODE_NAMES
      // Not implemented when Unicode is selected.
      bool exists(const char* path) {
    	  ExChar16_t* path16 = UTF8toUCS2(path);
    	  bool rv = exists(path16);
    	  free(path16);
    	  return rv;
      }
      bool mkdir(const char* path, bool pFlag = true){
    	  ExChar16_t* path16 = UTF8toUCS2(path);
    	  bool rv = mkdir(path16, pFlag);
    	  free(path16);
    	  return rv;
      }
      bool remove(const char* path){
    	  ExChar16_t* path16 = UTF8toUCS2(path);
    	  bool rv = remove(path16);
    	  free(path16);
    	  return rv;
      }
      bool rename(const char* oldPath, const char* newPath){
    	  ExChar16_t* oldPath16 = UTF8toUCS2(oldPath);
    	  ExChar16_t* newPath16 = UTF8toUCS2(newPath);
    	  bool rv = rename(oldPath16, newPath16);
    	  free(oldPath16);
    	  free(newPath16);
    	  return rv;
      }
      bool rmdir(const char* path){
    	  ExChar16_t* path16 = UTF8toUCS2(path);
    	  bool rv = rmdir(path16);
    	  free(path16);
    	  return rv;
      }
    #endif  //  USE_EXFAT_UNICODE_NAMES
    ExFatFile.h
    Code:
    #if USE_EXFAT_UNICODE_NAMES
      // Not Implemented when Unicode is selected.
      bool exists(const char* path) {
    	  ExChar16_t* path16 = UTF8toUCS2(path);
    	  bool rv = exists(path16);
    	  free(path16);
    	  return rv;
      }
      size_t getName(char *name, size_t size8) {
    	  size_t size16 = size8 / 3 - 1;
    	  ExChar16_t name16[size16];
    	  size_t size = getName(name16, size16);  
    	  UCS2toUTF8(name16, name);	  
    	  return size;
      }
      bool mkdir(ExFatFile* parent, const char* path, bool pFlag = true) {
    	  ExChar16_t* path16 = UTF8toUCS2(path);
    	  bool rv = mkdir(parent, path16, pFlag);
    	  free(path16);
    	  return rv;
      }
      bool open(ExFatVolume* vol, const char* path, int oflag) {
    	  ExChar16_t* path16 = UTF8toUCS2(path);
    	  bool rv = open(vol, path16, oflag);
    	  free(path16);
    	  return rv;
      }
      bool open(ExFatFile* dir, const char* path, int oflag) {
    	  ExChar16_t* path16 = UTF8toUCS2(path);
    	  bool rv = open(dir, path16, oflag);
    	  free(path16);
    	  return rv;
      }
      bool open(const char* path, int oflag = O_RDONLY) {
    	  ExChar16_t* path16 = UTF8toUCS2(path);
    	  bool rv = open(path16, oflag);
    	  free(path16);
    	  return rv;
      }
      bool remove(const char* path) {
    	  ExChar16_t* path16 = UTF8toUCS2(path);
    	  bool rv = remove(path16);
    	  free(path16);
    	  return rv;
      }
      bool rename(const char* newPath) {
    	  ExChar16_t* newPath16 = UTF8toUCS2(newPath);
    	  bool rv = rename(newPath16);
    	  free(newPath16);
    	  return rv;
      }
      bool rename(ExFatFile* dirFile, const char* newPath) {
    	  ExChar16_t* newPath16 = UTF8toUCS2(newPath);
    	  bool rv = rename(dirFile, newPath16);
    	  free(newPath16);
    	  return rv;
      }
    #endif  // USE_EXFAT_UNICODE_NAMES
    Last edited by mcleinn; 02-21-2021 at 07:23 PM.

  12. #12
    Junior Member
    Join Date
    Feb 2021
    Posts
    12
    I just checked MTP.cpp, and it seems to me (haven't worked with C or C++ in a while), that the writestring function blows back the 8bit strings into a 16bit string (which I guess is to be expected, MTP being a Microsoft protocol; and they switched to UCS2 internally back in '93)

    Code:
      void MTPD::write8 (uint8_t  x) { write((char*)&x, sizeof(x)); }
      void MTPD::write16(uint16_t x) { write((char*)&x, sizeof(x)); }
      void MTPD::write32(uint32_t x) { write((char*)&x, sizeof(x)); }
      void MTPD::write64(uint64_t x) { write((char*)&x, sizeof(x)); }
    
    #define Store2Storage(x) (x+1)
    #define Storage2Store(x) (x-1)
    
      void MTPD::writestring(const char* str) {
        if (*str) 
        { write8(strlen(str) + 1);
          while (*str) {  write16(*str);  ++str;  } write16(0);
        } else 
        { write8(0);
        }
      }
    whereas, to correctly transform UTF8 back into UCS2, it should call something like the unicode UTF8toUCS2() function I posted above, to correctly recognize 2 and 3 byte sequences:

    Code:
    	 switch (clen)
    	 {
    	    case 1: *p16 = (ExChar16_t)*p;
    		    break;
    
    	    case 2: c16 = (p[1] & 0x3f) + ((p[0] & 0x1f) << 6);
    		    *p16 = ((c16 > 0x7f) && ((p[1] & 0xC0) == 0x80)) ? c16 : '*';
    		    break;
    
    	    case 3: c16 = (p[2] & 0x3f) + ((p[1] & 0x3f) << 6) + ((p[0] & 0xf) << 12);
    		    *p16 = ((c16 > 0x7ff) && ((p[1] & 0xC0) == 0x80) && ((p[2] & 0xC0) == 0x80)) ? c16 : '*';
    		    break;
    
    	    default: *p16 = '~';	/* character code is greater than 0xffff */
    	 }
    I can see something similiar in the Android MTP code (granted, they are not converting to UCS2, but to variable-length UTF16 - which we might leave for later): https://www.androidos.net.cn/android...ringBuffer.cpp

    Code:
     
    void MtpStringBuffer::writeToPacket(MtpDataPacket* packet) const {
        std::u16string src16 = utf8ToUtf16(mString);
        int count = src16.length();
    
        if (count == 0) {
            packet->putUInt8(0);
            return;
        }
        packet->putUInt8(std::min(count + 1, MTP_STRING_MAX_CHARACTER_NUMBER));
    
        int i = 0;
        for (char16_t &c : src16) {
            if (i == MTP_STRING_MAX_CHARACTER_NUMBER - 1) {
                // Leave a slot for null termination.
                ALOGI("Mtp truncating long string\n");
                break;
            }
            packet->putUInt16(c);
            i++;
        }
        // only terminate with zero if string is not empty
        packet->putUInt16(0);
    }
    Last edited by mcleinn; 02-21-2021 at 11:24 PM.

  13. #13
    Junior Member
    Join Date
    Feb 2021
    Posts
    12
    Ok, I implemented it, and I can confirm it works. The directory listing on the host PC now shows Unicode characters correctly.

    MTP.cpp:

    Code:
      void MTPD::writestring(const char* str8) {
        if (*str8) 
        { 
    	  ExChar16_t* str16 = UTF8toUCS2(str8);
    	  write8(str16len(str16) + 1);
          for (int i=0; i<str16len(str16); i++) 
    	    write16(str16[i]); 
    	  write16(0);
    	  free(str16);
        } else 
        { 
    		write8(0);
        }
      }
    Code:
      void MTPD::readstring(char* buffer8) {
        int len = read8();
        if (!buffer8) {
          read(NULL, len * 2);
        } else {
    	  ExChar16_t buffer16[len];
          for (int i = 0; i < len; i++) {
            int16_t c2;
            buffer16[i] = c2 = read16();
          }
    	  UCS2toUTF8(buffer16, buffer8);
        }
      }
    Currently, I am including the unicode.cpp and the char16 type from MTP.cpp like this:
    Code:
    #include "ExFatLib/unicode.h"
    #include "ExFatLib/ExFatTypes.h"
    The upload.cpp I added to SdFat, as I mentioned above.
    Obviously, this can be done cleaner, if the unicode library is moved to some other place (in core), so we can access it from both SdFat and MTP. We might also want to move the generic type for the 16 bit characters.
    I took ExChar16_t from ExFatTypes.h, which says:

    Code:
    #if __cplusplus < 201103
      #warning no char16_t
      typedef uint16_t ExChar16_t;
    //  #error C++11 Support required
    #else  // __cplusplus < 201103
      typedef char16_t ExChar16_t;
    #endif  // __cplusplus < 201103
    Last edited by mcleinn; 02-22-2021 at 01:22 AM.

  14. #14
    Senior Member
    Join Date
    Jul 2014
    Posts
    3,317
    Quote Originally Posted by mcleinn View Post
    Ok, I implemented it, and I can confirm it works. The directory listing on the host PC now shows Unicode characters correctly.

    MTP.cpp:

    Code:
      void MTPD::writestring(const char* str8) {
        if (*str8) 
        { 
    	  ExChar16_t* str16 = UTF8toUCS2(str8);
    	  write8(str16len(str16) + 1);
          for (int i=0; i<str16len(str16); i++) 
    	    write16(str16[i]); 
    	  write16(0);
    	  free(str16);
        } else 
        { 
    		write8(0);
        }
      }
    Please note that there is no garbage collection on Teensy, so malloc/free combo may create problems in the long run.

  15. #15
    Junior Member
    Join Date
    Feb 2021
    Posts
    12
    Please correct me, if I am wrong, but I think C++ and C generally do not have garbage collection.
    But yes, I am reading that dynamic allocation is considered "bad style" on embedded systems, as they do not have a lot of memory and one can quickly run out of it, if not using it carefully.
    My particular use is very short-time (memory freed right after sending the data), so it won't be an issue - but I can remove the dynamic memory from the library - no problem.
    It is not crucial, we can easily do without it. Still, maybe Paul could suggest how we can go on from here? (as to placing the unicode conversion functions and the type)
    Last edited by mcleinn; 02-22-2021 at 11:47 AM.

  16. #16
    Senior Member
    Join Date
    Nov 2012
    Posts
    318
    Quote Originally Posted by PaulStoffregen View Post
    Arduino kinda does support UTF8. If you type or copy non-latin characters into strings, the Arduino IDE does end up putting UTF8 encoded strings into memory. And if you transmit UTF8 encoded data to the serial monitor, it does properly display non-latin characters.

    Very few libraries do any sort of UTF8 specific processing, but much of the idea of UTF8 is that you can usually just treat it like a stream of 8 bit bytes.

    Teensy's USB Keyboard.print() does parse UTF8 and translates characters to their mapping on various non-US keyboard mappings.

    If you'd like to support UTF8 strings in SdFat, I could contribute that UTF8 to Unicode code.
    Paul,
    I stopped development of 16-bit Unicode when I discovered how hard it was to use. I already have decided that the next attempt would be UTF8 based.

    The case problem is next. FAT16/FAT32 LFN is ambiguous. Windows has it's way of handling case. Microsoft doesn't clearly define the rules for comparing filenames. You see stuff like this:

    Note that CharLower always maps uppercase I to lowercase I ("i"), even when the current language is Turkish or Azerbaijani. If you need a function that is linguistically sensitive in this respect, call LCMapString.

    The lstrcmp and lstrcmpi functions do not perform byte comparisons like their ANSI namesakes, for example, strcmp. Instead, they compare strings according to the rules of the locale.
    I will start by offering the two options for comparing file names. Either use the standard SD exFAT Up-case table or a user Up-Case function.

    I don't plan to keep the 16-bit wide character API in the future.

  17. #17
    Junior Member
    Join Date
    Feb 2021
    Posts
    12
    IMHO UTF8 in SdFat would be perfectly fine. If this is coming, we can leave the UTF8-UTC2 transfer functions for SdFat as a patch.
    But MTP needs to convert properly to 16bit (UTC2/UTF16) strings when communicating with the host system, in any case.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •