Teensy 4.1 - Adafruit Airlift Featherwing Co-Processor FTP Server not opening Port 21

[wwatson]

@mjs513 - Decided to check out this version of FTPServer using QNEthernet on the the T4.1:
https://github.com/khoih-prog/FTP_Server_Teensy41

It works in passive mode but still has file transfer problems that we are seeing.

Status:	Connecting to 192.168.0.110:21...
Status:	Connection established, waiting for welcome message...
Response:	220--- Welcome to FTP_Server_Teensy41 ---
Response:	220 -- FTP_Server_Teensy41 v1.0.0 --
Status:	Plain FTP is insecure. Please switch to FTP over TLS.
Command:	USER teensy4x
Response:	331 Ok. Password required
Command:	PASS ********
Response:	230 Ok
Status:	Server does not support non-ASCII characters.
Status:	Logged in
Trace:	Measured latency of 248 ms
Status:	Retrieving directory listing...
Command:	PWD
Response:	257 "/" is your current directory
Status:	Directory listing of "/" successful
Status:	Disconnected from server
Status:	Connecting to 192.168.0.110:21...
Status:	Connection established, waiting for welcome message...
Error:	Connection timed out after 20 seconds of inactivity
Error:	Could not connect to server
Status:	Disconnected from server
Status:	Delaying connection for 5 seconds due to previously failed connection attempt...
Status:	Connecting to 192.168.0.110:21...
Status:	Connection established, waiting for welcome message...

The section marked in red is where I initiated the transfer from the T4.1 to Filezilla. For some reason it disconnects from the Teensy and tries to reconnect to it and hangs waiting for the welcome message. Don't know if this is normal for a file transfer or not. Either way the server is not disconnecting, reconnecting and sending the welcome message. Also you can connect and disconnect from the Teensy without having to reset the Teensy.

Got to take walk away from this for a while and checkout whats going on with USBHost_t36

[wwatson]

@mjs513, @Khoih-prog - Was working with FTP_Server_Teensy41 today to try and figure out why I was only getting partial transfers when downloading from the T4.1 to the PC using gFTP or Filezilla. Turns out the failure was in the doRetrieve() function:

bool FtpServer::doRetrieve()
{
  if ( ! dataConnected())
  {
    file.close();

    return false;
  }

  int16_t nb = file.read( buf, FTP_BUF_SIZE );

  if ( nb > 0 )
  {
    data.write( buf, nb );
    bytesTransfered += nb;
    return true;
  }

  closeTransfer();

  return false;
}

The data.write() buffer was being overwritten before the transfer was complete. That's why the files were coming up smaller on the client side than the original transfer size. I changed the doRetrieve() function to this:

bool FtpServer::doRetrieve()
{
  if ( ! dataConnected())
  {
    file.close();
    return false;
  }
  // Find available space in data.write() buffer.
  int spaceLeft = data.availableForWrite();
  // Get remaining bytes to read from file.
  int32_t leftToXfer = file.available();

  if (spaceLeft <= 0) {
    return true; // Return true if no space available.
  }

  // Calculate read size.
  // Base the amount to read on the space available in the
  // data.write() buffer. 
  if(leftToXfer) {
    int32_t nb = file.read( buf,(spaceLeft <= FTP_BUF_SIZE) ? spaceLeft : FTP_BUF_SIZE);
    data.write(buf, nb);
    bytesTransfered += nb;
    return true;
  }
  closeTransfer();
  return false;
}

The size of the transfer will always be equal to spaceLeft if it is less than or equal to FTP_BUF_SIZE.
Tested this with QNEthernet and NativeEthernet. FTP_BUF_SIZE gave the best transfer speeds. ~2MBs.

@Khoih-prog will issue a PR.

[mjs513]

@wwatson
Sorry been distracted with USBMSC stuff. Sounds like you are making progress. Also probably applicable SimpleFTPServer as well. Will have to play again soon with this.

[KurtE]

Sorry all, keep meaning to try this stuff out...

But been distracted also with the MSC stuff. More details on that soon.

[wwatson]

@wwatson
Sorry been distracted with USBMSC stuff. Sounds like you are making progress. Also probably applicable SimpleFTPServer as well. Will have to play again soon with this.

I have SimpleFTPServer so messed up I will have to start over with it. I think now that I have FTP_Server_Teensy41 working, I'll try to add WiFiNINA to it as a base to work from. It is easier to adapt to and debug. Use it as a control. Then I'll start fresh with SimpleFTPServer.

I did a little testing with the current state of USBHost_t36. Everything seems to work so far I'm sure you guys are up to your eyebrows with changes. Will watch and test were I can...

Even tested multiple partitions on a thumb stick. Worked good

[khoih-prog]

Hi @wwatson

The new FTP_Server_Teensy41 releases v1.1.0 has just been published. Your contribution is noted in Contributions and Thanks

Best Regards,

---

Releases v1.1.0

1. Fix bug incomplete downloads from server to client. Check Incomplete downloads from server to client. #2

[wwatson]

Hi @wwatson

The new FTP_Server_Teensy41 releases v1.1.0 has just been published. Your contribution is noted in Contributions and Thanks

Best Regards,

---

Releases v1.1.0

1. Fix bug incomplete downloads from server to client. Check Incomplete downloads from server to client. #2

Sorry for a delayed reply. Down 4 people at work so not having to much time to play. I have however had moments to integrate WiFiNINA to FTP_Server_Teensy41. It works fine if not transferring files. From what I am seeing the problem is with WiFiNINA if I add delays in doRetrive or doStore I can initiate a transfer but it never closes the file on completion. Also when the transfer is in progress data.available() never returns more than 1500 bytes at a time and during the transfer it sometimes returns 0 bytes. It never closes the transfer when done. Just keeps looping through the doRetrieve and doRestore functions. Not sure yet what is going on but I think it has to do with WiFiNINA and the 8MHz SPI clock rate.

Hopefully will have time to play with the client side as well

[Juraj]

1500 is a size of a TCP packet. it is normal that available() returns 0 between packets, even it shouldn't be so slow that only one packet is available at time.

[beermat]

I am having the same issue as wwatson above in #126, even using the latest version of the FTP Server 1.1.0 . Using FileZilla (3.59.0) on Windows, I can connect in passive or active mode and (usually, but not always) get a directory listing back. But if I try to transfer a file from the Teensy 4.1, I get the same status in FileZilla as mentioned above:

Status:	Connecting to 192.168.1.91:221...
Status:	Connection established, waiting for welcome message...
Status:	Plain FTP is insecure. Please switch to FTP over TLS.
Status:	Server does not support non-ASCII characters.
Status:	Logged in
Status:	Retrieving directory listing...
Status:	Directory listing of "/" successful
Status:	Connecting to 192.168.1.91:221...
Status:	Connection established, waiting for welcome message...
Error:	Connection timed out after 20 seconds of inactivity
Error:	Could not connect to server
Status:	Disconnected from server
Status:	Delaying connection for 5 seconds due to previously failed connection attempt...

The line highlighted in red is the status given in FileZilla when I request the file

[wwatson]

1500 is a size of a TCP packet. it is normal that available() returns 0 between packets, even it shouldn't be so slow that only one packet is available at time.

Thanks for the input> I am still learning this stuff What is the proper way to determine that the transfer is complete? Does the client close the connection?

[wwatson]

Alright, Went back to Native Ethernet and QNEthernet using FTP_Server_Teensy41 latest version and had no problems transferring files back and forth between the client and server using Filezilla and gFTP. All transfers completed. Really not sure what is the problem with WiFiNINA but it does not see whatever it is that signals transfer completion from a FTP client

That is the only thing that does not work...

[khoih-prog]

Hi All,

Wow, I just found out and fixed the bug in WiFiNINA + WiFiNINA_Generic libraries. Already tested OK using Nano_RP2040_Connect with FTPClient_Generic.

The very bad bug affects sending TCP data consecutively, leading to very short or corrupted uploading files.

Will finalize and update the WiFiNINA_Generic library shortly.

[khoih-prog]

This is the directory listing, showing correct octocat.jpg file size of 51695 bytes, after running FTPClient_UploadImage example. The server is VSFTP on RPi

pi@raspberrypi-02:/home/ftp_test $ ls -la
total 80
drwxrwxrwx 3 ftp_test ftp_test  4096 May 22 18:40 .
drwxr-xr-x 4 root     root      4096 May 14 16:55 ..
-rw-r--r-- 1 ftp_test ftp_test   220 May 14 16:55 .bash_logout
-rw-r--r-- 1 ftp_test ftp_test  3523 May 14 16:55 .bashrc
-rw------- 1 ftp_test ftp_test    11 May 22 18:40 hello_world.txt
drwx------ 2 ftp_test ftp_test  4096 May 16 20:19 NewDir
-rw------- 1 ftp_test ftp_test 51695 May 22 18:40 octocat.jpg
-rw-r--r-- 1 ftp_test ftp_test   807 May 14 16:55 .profile

The octocat.jpg is displayed normally without corruption.

[wwatson]

Hi All,

Wow, I just found out and fixed the bug in WiFiNINA + WiFiNINA_Generic libraries. Already tested OK using Nano_RP2040_Connect with FTPClient_Generic.

The very bad bug affects sending TCP data consecutively, leading to very short or corrupted uploading files.

Will finalize and update the WiFiNINA_Generic library shortly.

Wow is right. I have spent the last two weekends working on FTP_Server_Teensy41 and WiFiNINA. The problem is with data.connected() always shows 1. Using the same library with QNEthernet would show data.connected() going to 0 from one when the transfer was complete. The only way could complete the transfer was with this hack:

bool FtpServer::doStore()
{
 uint32_t tmout = millis();
  int na = 0;
while((millis() - tmout) != 50) {
  na = data.available();
  if(na != 0)
    break;
}
  if(na == 0) {
    closeTransfer();
    return false;
  } 
  if ( na > FTP_BUF_SIZE )
    na = FTP_BUF_SIZE;

  int16_t nb = data.read((uint8_t *) buf, na );
  int16_t rc = 0;

  if ( nb > 0 )
  {
     //FtpDebug << millis() << " " << nb << endl;
    rc = file.write( buf, nb );
    bytesTransfered += nb;
  }

  if ( nb < 0 || rc == nb  )
    return true;

  FtpOutCli << F("552 Probably insufficient storage space") << endl;
  file.close();
  data.stop();
  closeTransfer();
  return false;
}

The other problem is when I disconnected Filezilla or gFTP the sever will not disconnect until it times out. I went through WiFiNINA all the way to nina-fw.
This is where it is failing:

uint8_t WiFiClient::connected() {
  if (_sock == 255) {
    return 0;
  } else if (available()) {
//Serial.printf("\navailable() = %d\n", available());
    return 1;
  } else {
    uint8_t s = status();

    uint8_t result =  !(s == LISTEN || s == CLOSED || s == FIN_WAIT_1 ||
                      s == FIN_WAIT_2 || s == TIME_WAIT ||
                      s == SYN_SENT || s== SYN_RCVD ||
                      (s == CLOSE_WAIT));
//Serial.printf("\nresult = %d, s = %d\n", result, s);

    if (result == 0) {
      WiFiSocketBuffer.close(_sock);
      _sock = 255;
    }

    return result;
  }
}

s always == 4 and result always == 1.
From that point on it's above my pay grade
Will check out what you found...

[khoih-prog]

HI All,

Please use the new WiFiNINA_Generic releases v1.8.14-5

I've been testing many times and still uploading OK, without losing data.

Will update the forked WiFiNINA later, if necessary.

---

Releases v1.8.14-5

1. Fix bug causing data lost when sending large files. Check Thread: Teensy 4.1 - Adafruit Airlift Featherwing Co-Processor FTP Server not opening Port 21. This is just a kludge for temporary use, waiting for better and final fix, dealing with SPI driver, from Arduino.
2. Update `Packages' Patches`

[khoih-prog]

Testing OK with FTP_Server_Teensy41

Starting FTP_Server_SDFAT2 on TEENSY 4.1 with QNEthernet
FTP_Server_Teensy41 v1.1.0
Initializing SD card...
Wiring is correct and a card is present.

Card type: SDHC

Volume type is FAT32

Volume size (Kbytes): 31452672
Volume size (Mbytes): 30715

===============================
SDCard Initialization : done.
index1.htm                                       3810    23:14  April 29, 2022
foo.txt                                            13    00:00  January 1, 1980
index2.htm                                       3714    00:39  January 1, 2019
index.htm                                        3810    22:50  April 29, 2022
mydatalog.txt                                      10    22:24  April 9, 2022
edit.htm.gz                                      4116    20:15  March 13, 2022
CanadaFlag_1.png                                41214    20:15  March 13, 2022
CanadaFlag_2.png                                 8311    20:15  March 13, 2022
CanadaFlag_3.jpg                                11156    20:15  March 13, 2022
favicon.ico                                      1150    00:38  January 1, 2019
graphs.js.gz                                     1971    20:15  March 13, 2022
CanadaFlag_2_1.png                               8311    22:52  April 29, 2022
ESP_AT_WM_Lite.txt                               1547    22:51  April 29, 2022
helloworld.txt                                    318    00:00  January 1, 2019
myNewDir/
my_new_dir/
  octocat.jpg                                   51695    00:11  January 1, 2019
  hello_world.txt                                  11    00:11  January 1, 2019
octocat.jpg                                     51695    00:01  January 1, 2019
hello_world.txt                                    11    00:01  January 1, 2019
done!
=========== USE_QN_ETHERNET ===========
Initialize Ethernet using static IP => IP Address = 192.168.2.241
FTP Server Credentials => account = teensy4x, password = ftp_test
[FTP]  Ftp server waiting for connection on port  21
[FTP]  Client connected!
USER teensy4x
PASS ftp_test
[FTP]  Authentication Ok. Waiting for commands.
CWD /
PASV
[FTP]  Connection management set to passive
[FTP]  Listening at  192.168.2.241 : 55600
Type I
STOR octocat.jpg
[FTP]  Receiving  octocat.jpg
[FTP]  Transfer completed in (ms)  12448 , Speed (kbytes/s)  4
PASV
[FTP]  Connection management set to passive
[FTP]  Listening at  192.168.2.241 : 55600
Type A
STOR hello_world.txt
[FTP]  Receiving  hello_world.txt
[FTP]  Transfer completed in (ms)  217 , Speed (kbytes/s)  0
QUIT
[FTP]  Disconnecting client
[FTP]  Ftp server waiting for connection on port  21

[mjs513]

HI All,

Please use the new WiFiNINA_Generic releases v1.8.14-5

I've been testing many times and still uploading OK, without losing data.

Will update the forked WiFiNINA later, if necessary.

---

Releases v1.8.14-5

1. Fix bug causing data lost when sending large files. Check Thread: Teensy 4.1 - Adafruit Airlift Featherwing Co-Processor FTP Server not opening Port 21. This is just a kludge for temporary use, waiting for better and final fix, dealing with SPI driver, from Arduino.
2. Update `Packages' Patches`

@khoih-prog and @wwatson
Sorry been away from this for a while been distracted as I mentioned before but been following whats going on. Its great that you found the bug !!!!

Anyway, I just downloaded the updated Generic library and ran ScanNetworksAdvanced to make sure it would work and as expected it worked without an issue.

Next step I updated SimpleFTPServer to use the Generic library instead of the standard arduino WiFiNina library. Reran the SimpleFtp server sketch that we had been using, the Teensy made the connection to wifi and started the server (NOTE: I am using active mode).

Ran fileZilla again and it getting stuck on waiting for response from server. Maybe should try Passive mode

[khoih-prog]

Hi @mjs513

I suggest you first try using another known-working FTP Server, such as VSFTP, ftpd, or FTP_Server_Teensy41 to be sure your Adafruit Featherwing board is working OK (hardware, nina-fw, etc). Then moving on to use the board with possibly-still-buggy SimpleFTPServer.

I haven't tried that SimpleFTPServer on Nano_RP2040_Connect, or any other board, to know if it's OK.

[khoih-prog]

Hi @wwatson

I have however had moments to integrate WiFiNINA to FTP_Server_Teensy41. It works fine if not transferring files.

Could you please post the code or make a PR to add WiFiNINA support to FTP_Server_Teensy41, so that we can save duplication and time.
I'll try to have a look to see if there is any more bug when WiFiNINA working as FTP Server.

[wwatson]

Hi @wwatson



Could you please post the code or make a PR to add WiFiNINA support to FTP_Server_Teensy41, so that we can save duplication and time.
I'll try to have a look to see if there is any more bug when WiFiNINA working as FTP Server.

Below are my versions of FTP_Server_Teensy41 and WiFiNINA. This was tested with a T4.1 and the Adafruit ESP32 Airlift Featherwing. With my hack to 'DoStore()' I am able upload large and small files to the T4.1. I know that 'doRetrieve()' DOES NOT work. Probably do to the same problem with 'doStore()'.

All other FTP functions are working ok at this point Just the two do's that don't do.

WiFiNINA_Generic will need to be modified as it does not support the setPins() function and possibly other functions peculiar to Adafruit's version.
I really think the problem lies in nina-fw. Could be wrong...

EDIT: Again the problem is with data.connected(). Always returns 1. Never shows 0 when the transfer is done. See p#139.

[khoih-prog]

Hi @wwatson

Thanks for the code. It'll take some time later to merge the Adafruit WiFiNINA to WiFiNINA_Generic library, as I currently don't have the Featherwing to test.

To test the new mod, which fixes the file transfer, you can change your version of WiFiNINA library, file WiFiClient.cpp, line #143-168, to

size_t WiFiClient::write(const uint8_t *buf, size_t size)
{ 
  if (_sock == NO_SOCKET_AVAIL)
  {
    setWriteError();
        
    return 0;
  }

  if (size == 0)
  {
    setWriteError();
        
    return 0;
  }

  size_t written = ServerDrv::sendData(_sock, buf, size);
  
  uint8_t timesResent = 0;
  
  while ( (written != size) && (timesResent++ < 100) )
  {
    // Don't use too short delay so that NINA has some time to recover
    // The fix is considered as kludge, and the correct place to fix is in ServerDrv::sendData()
    delay(100);
    
    written += ServerDrv::sendData(_sock, buf + written, size - written);
  }
   
  if (!written && _retrySend) 
  {
    written = retry(buf, size, true);
  }
  
  if (!written)
  {
    // close socket
    ServerDrv::stopClient(_sock);
    setWriteError();
       
    return 0;
  }

  if (!ServerDrv::checkDataSent(_sock))
  {
    setWriteError();
       
    return 0;
  }
  
  return written;
}

from

size_t WiFiClient::write(const uint8_t *buf, size_t size) {
  if (_sock == NO_SOCKET_AVAIL)
  {
	  setWriteError();
	  return 0;
  }
  if (size==0)
  {
	  setWriteError();
      return 0;
  }

  size_t written = ServerDrv::sendData(_sock, buf, size);
  if (!written)
  {
	  setWriteError();
      return 0;
  }
  if (!ServerDrv::checkDataSent(_sock))
  {
	  setWriteError();
      return 0;
  }

  return written;
}

PS:

BTW, I don't think now the nina-fw can fix the issue, as I tested using Nano_RP2040_Connect with original nina-fw, and still OK.

[khoih-prog]

HI @wwatson,

You can also help test using WiFiNINA_Generic with Featherwing, just by modifying file WiFiNINA_Pinout_Generic.h, line #368-381

to

  #if defined(__IMXRT1062__)
    // For Teensy 4.x
    #warning You have to modify pin usage according to actual connection for Teensy 4.0/4.1
    #define PINS_COUNT           (60u)
    // T4.1 SPI pin defs for WiFiNINA AirLift.
    #define NINA_GPIO0  (255u)                           //6
    #define NINA_RESETN (6u)
    #define NINA_ACK    (9u)

    // T4.1 SPI pin defs for WiFiNINA AirLift.
    #define SPIWIFI_SS       5   //PIN_SPI1_SS           //10
    #define SPIWIFI_ACK      9   //NINA_ACK               //7
    #define SPIWIFI_RESET    6   //NINA_RESETN            //5

  #elif ( defined(__MKL26Z64__) || defined(ARDUINO_ARCH_AVR) )

from

  #if defined(__IMXRT1062__)
    // For Teensy 4.0
    #warning You have to modify pin usage according to actual connection for Teensy 4.0
    #define PINS_COUNT           (60u)
    //NINA
    #define NINA_GPIO0  (6u)                             //6
    #define NINA_RESETN (2u)
    #define NINA_ACK    (5u)


    #define SPIWIFI_SS       10   //PIN_SPI1_SS           //10
    #define SPIWIFI_ACK      5   //NINA_ACK               //5
    #define SPIWIFI_RESET    2   //NINA_RESETN            //7


  #elif ( defined(__MKL26Z64__) || defined(ARDUINO_ARCH_AVR) )

then remove from the example the line

WiFi.setPins(SPIWIFI_SS, SPIWIFI_ACK, ESP32_RESETN, ESP32_GPIO0, &SPIWIFI);

and change from #include "WiFiNINA.h" to #include "WiFiNINA_Generic.h".

I can compile OK here, but no Featherwing to test yet.

[mjs513]

HI @wwatson,

You can also help test using WiFiNINA_Generic with Featherwing, just by modifying file WiFiNINA_Pinout_Generic.h, line #368-381

to

  #if defined(__IMXRT1062__)
    // For Teensy 4.x
    #warning You have to modify pin usage according to actual connection for Teensy 4.0/4.1
    #define PINS_COUNT           (60u)
    // T4.1 SPI pin defs for WiFiNINA AirLift.
    #define NINA_GPIO0  (255u)                           //6
    #define NINA_RESETN (6u)
    #define NINA_ACK    (9u)

    // T4.1 SPI pin defs for WiFiNINA AirLift.
    #define SPIWIFI_SS       5   //PIN_SPI1_SS           //10
    #define SPIWIFI_ACK      9   //NINA_ACK               //7
    #define SPIWIFI_RESET    6   //NINA_RESETN            //5

  #elif ( defined(__MKL26Z64__) || defined(ARDUINO_ARCH_AVR) )

from
....

then remove from the example the line

WiFi.setPins(SPIWIFI_SS, SPIWIFI_ACK, ESP32_RESETN, ESP32_GPIO0, &SPIWIFI);

and change from #include "WiFiNINA.h" to #include "WiFiNINA_Generic.h".

I can compile OK here, but no Featherwing to test yet.

Gave this a try with your Generic library that I downloaded this morning. Made the changes to WiFiNINA_Pinout_Generic.h and client says again about waiting for response from server.



Maybe @wwatson will have better luck than me. See next post for mods to WiFiNINA

[mjs513]

Hi @wwatson

Thanks for the code. It'll take some time later to merge the Adafruit WiFiNINA to WiFiNINA_Generic library, as I currently don't have the Featherwing to test.

To test the new mod, which fixes the file transfer, you can change your version of WiFiNINA library, file WiFiClient.cpp, line #143-168, to

size_t WiFiClient::write(const uint8_t *buf, size_t size)
{ 
  if (_sock == NO_SOCKET_AVAIL)
  {
    setWriteError();
        
    return 0;
  }

  if (size == 0)
  {
    setWriteError();
        
    return 0;
  }

  size_t written = ServerDrv::sendData(_sock, buf, size);
  
  uint8_t timesResent = 0;
  
  while ( (written != size) && (timesResent++ < 100) )
  {
    // Don't use too short delay so that NINA has some time to recover
    // The fix is considered as kludge, and the correct place to fix is in ServerDrv::sendData()
    delay(100);
    
    written += ServerDrv::sendData(_sock, buf + written, size - written);
  }
   
  if (!written && _retrySend) 
  {
    written = retry(buf, size, true);
  }
  
  if (!written)
  {
    // close socket
    ServerDrv::stopClient(_sock);
    setWriteError();
       
    return 0;
  }

  if (!ServerDrv::checkDataSent(_sock))
  {
    setWriteError();
       
    return 0;
  }
  
  return written;
}

from
....

PS:

BTW, I don't think now the nina-fw can fix the issue, as I tested using Nano_RP2040_Connect with original nina-fw, and still OK.

Made the suggested change but also had to copy over the retry function and while I was at it I did copy over the setRetry function as well.

Two tests were run. Transferring from the PC to T4.1 and then from the T4.1 to PC using Filezilla.

1. From PC (client) to T4.1 (server) coping a 4.8MB file was not an issue:


2. From the T4.1 to the PC however failed, see screen shot below - eventually it says time out, and get a message in the file transfer window that could not start the transfer:

[khoih-prog]

HI @mjs513

That's good news. As I suspect, the xfer bug affects 2-ways in WiFiNINA. We already fixed just 1-way, now still have to find out and fix the other way.

Sorry, have no time now. Can you use FTPClient_Generic as Client to test? FileZilla still has elusive issue with FTPServer

Could you test the WiFiNINA_Generic as simple WebServer / WebClient to see if OK to xfer large files? We have to fix from here first.