Forum Rule: Always post complete source code & details to reproduce any issue!
Results 1 to 10 of 10

Thread: Arduino/Teensy. Would like ability to retrieve server client that hasn't sent data.

  1. #1
    Junior Member
    Join Date
    Sep 2016
    Posts
    13

    Arduino/Teensy. Would like ability to retrieve server client that hasn't sent data.

    Title says it all. Current Arduino API will not return a socket (wrapped in a Ethernet Client class) unless it's received data on that socket. I'm trying to implement an FTP server with passive mode on a Teensy 3.5. The passive data socket will never receive data from the FTP client, only send. Further, the client will hang if the data socket is not closed by the server.

    So first attempts at just using server.write do send the data, but since I have no way to close the socket, the FTP client hangs.

    The base Arduino Ethernet library does make EthernetClient::status() and EthernetClass::_server_port public members. So I believe I could iterate through all sockets in userspace and find my open server socket to solve this problem. However the teensy version EthernetClass::socketStatus() is private and EthernetServer::server_port is commented that it will become private soon. So I can't iterate through sockets in userspace to find my established server connection.

    For now, I may try to just use server.writes and the public EthernetServer::server_port to iterate through and close all connections that match the server port.

    Open to other suggestions as well.

    (Looks at code again...)

    OK, EthernetClient::status() provides public access to EthernetClass::socketStatus(). So i was able to get this implemented using non-documented API calls.

    Code:
    for (int sock = 0; sock < MAX_SOCK_NUM; sock++) {
            EthernetClient data(sock);
            if (dataServer.server_port[sock] == FTP_DATA_PORT_PASV) {
              uint8_t s = data.status();
              if (s == 0x17 || s == 0x1C) {
                //assume its the first established one we find with the server port number
                  //data = myclient;
                  pasvDataSocket = sock;
                  return true;
                }
            }
            
          }
    So this is working for me, i iterate through all the sockets till i find one that matches my saved port number and shows as TCP established in status. But since this is undocumented API the Teensy Ethernet library does not match the Arduino Ethernet library (where the port numbers are saved) so this isn't portable. Would be nice if we could establish a documented call to retrieve connected clients that haven't sent data.

    Ok, now on to figure out why every TCP packet is taking 200ms to ACK and severely slowing down the file transfer....

  2. #2
    Junior Member
    Join Date
    Sep 2016
    Posts
    13
    Quote Originally Posted by madsci1016 View Post

    Ok, now on to figure out why every TCP packet is taking 200ms to ACK and severely slowing down the file transfer....
    And in case anyone else finds this:

    EthernetServer.cpp line 9

    Code:
    sockindex = Ethernet.socketBegin(SnMR::TCP, _port);
    to

    Code:
    sockindex = Ethernet.socketBegin(SnMR::TCP | SnMR::ND, _port);
    Sped that up. Though it sends 5 acks for every frame now. Guess that's the tradeoff.

  3. #3
    Quote Originally Posted by madsci1016 View Post
    Would be nice if we could establish a documented call to retrieve connected clients that haven't sent data.

    Ok, now on to figure out why every TCP packet is taking 200ms to ACK and severely slowing down the file transfer....
    Agreed. Providing a library that only works one way (a client connection must talk first) when there are defined protocols (FTP) that simply don't work that way shows someone wasn't thinking much about what their library actually provides. I've just run into this issue too, though I assumed it was just a bug until I read Arduino does it this way too. Saves me posting a request for a fix that isn't going to happen.

    But I'm not seeing slow file-read transfers via my W5500 (line 9 being changed to what you show doesn't seem affect anything). File-writes are slow, but I assume that's the SDCard/FAT library I'm using. I see 275KB/s read and 22KB/s write speeds both ways line 9 is set here. A 10 times improvement in write speed would be good, I'm not sure how useful it is as it stands because I want to write to it far more often than read from it.

  4. #4
    Senior Member PaulStoffregen's Avatar
    Join Date
    Nov 2012
    Posts
    16,228
    Let's talk about an extension for the EthernetServer class.

    It's looking likely that Arduino will soon accept Teensy's Ethernet lib (fully replacing theirs, which only supports W5100) and I'll take over most Ethernet lib maintenance. So now's a good time to think about how to permanently fix this for all of Arduino....

    Also, please use 1.41-beta3, if you're not already. Several important updates to Ethernet have happened since 1.40 was released.

  5. #5
    I didn't seem to have as many issues as madsci1016 wrote about (closing sockets on the server side when needed) but that could be because I changed the library to what I thought it should be while developing my solution. I basically changed ESTABLISHED to give me the socket regardless, and leave CLOSE_WAIT as it was:

    Code:
    EthernetClient EthernetServer::available()
    {
    	bool listening = false;
    	uint8_t sockindex = MAX_SOCK_NUM;
    
    	for (uint8_t i=0; i < MAX_SOCK_NUM; i++) {
    		if (server_port[i] == _port) {
    			uint8_t stat = Ethernet.socketStatus(i);
                if (stat == SnSR::ESTABLISHED) {
    				sockindex = i;
                } else if (stat == SnSR::CLOSE_WAIT) {
    				if (Ethernet.socketRecvAvailable(i) > 0) {
    					sockindex = i;
    				} else {
    					// remote host closed connection, our end still open
    					Ethernet.socketDisconnect(i);
    					// status becomes LAST_ACK for short time
    				}
    			} else if (stat == SnSR::LISTEN) {
    				listening = true;
    			} else if (stat == SnSR::CLOSED) {
    				server_port[i] = 0;
    			}
    		}
    	}
    	if (!listening) begin();
    	return EthernetClient(sockindex);
    }
    I thought it was a bug because the example WebServer solution (that I initially used for guidance) should still work because it calls client.available() later, before doing anything with data.

    While I'm happy with my current change, leaving EthernetServer.available as it is but adding a new method with the code above (called connected?) would be all I am after and shouldn't affect any existing users.

    Maybe madsci1016 has some more ideas though?

  6. #6
    Quote Originally Posted by Jasoroony View Post
    and 22KB/s write speeds both ways line 9 is set here. A 10 times improvement in write speed would be good, I'm not sure how useful it is as it stands because I want to write to it far more often than read from it.
    Just a quick update on my slow SD writes..

    Taking out the write-to-file call shows the little W5500 module brings in ~3MB/s (not bad at all, given a 100Mb/s link does ~10MB/s max).

    That got me thinking, so I tried a "better rated" SD card (the first one was an 8GB freebie with an old phone I bought) and the new one did 220KB/s. Changing the write file routine to cache incoming data to cluster size before writing improved it to 270KB/s. It's now "very useful".

    I can't seem to use an SPI clock any higher than 25MHz without my sketch locking up though, attempting to see if >25MHz improves anything.

  7. #7
    Thinking more... Odd that a sketch actually locks up on a Teensy 3.5 with a too high (not *that* high though) clock on a peripheral...

  8. #8
    Senior Member PaulStoffregen's Avatar
    Join Date
    Nov 2012
    Posts
    16,228
    Quote Originally Posted by Jasoroony View Post
    Odd that a sketch actually locks up on a Teensy 3.5 with a too high (not *that* high though) clock on a peripheral...
    Indeed that would be odd, if the lockup were only due to overclocking the SPI.

    But apparent lockups are surprisingly common for all sorts of other reasons. To illustrate, let me describe for you a problem we heard repeatedly in the early days of Teensy 3.0 and 3.1. People would report that Teensy wasn't retaining their code after power cycling. They would upload from Arduino and everything would work fine. Then when they power cycled the board, it would be unresponsive. The behavior so strongly implanted the idea that Teensy had not retained their code that it was nearly impossible to get people who experienced this problem to do almost anything to actually investigate the problem. They were absolutely convinced Teensy had not retained the program they uploaded, and virtually nothing could shake they belief once it had taken hold.

    Of course Teensy was retaining their code. Some cases were a stray "while (!Serial) ;" left over in setup(), but the more common cause was their code or a library they used initializing some other chip, usually a motion sensor. Those chips had a brief startup time before they can respond. When uploading, the chip has already been powered on for quite some time. But on a cold boot, it would tend to be in that startup phase, unable to communicate. Many of those motion sensors start in a low power mode, and begin running when sent a command to start up. Most of the programs people used were libraries developed for regular Arduino boards which have a lengthy bootloader step before they start running. They would sent the hardware init command, and never do any checking whether the part actually responded, never retry or wait for the hardware to actually be ready. Then the rest of their program would be unresponsive, because it depended on getting good data from the sensor. Teensy was indeed running their program, but the program was never designed to properly handle the fault cases.

    My guess is this lockup is very likely something similar, where the SPI running fast is giving wrong results, and the rest of the code isn't designed to deal with whatever isn't happening as expected. The challenging part is diagnosing the problem. Like with the rapid startup issue, there's a very natural human tendency to quickly form conclusions that fit a theory based on the parts that are known, without considering the larger system effects (especially undefined software behavior) which aren't as well known. Once those theories are formed, it's incredibly difficult to focus effort on investigating the unknowns...

    In any case, the "Teensy didn't retain my code after power cycle" is now a thing of the past, due to a 400 ms delay that was added to the startup code, before C++ constructors and setup() are run. Simple as that sounds, it actually took a couple years to fully understand the issue and add that delay. That's how tricky these sorts of "lockup" problems can be!

  9. #9
    Senior Member PaulStoffregen's Avatar
    Join Date
    Nov 2012
    Posts
    16,228
    About the EthernetServer class limitation, detecting clients which do not transmit data upon connecting, let's call this problem "EthernetServer can't do FTP".

    Even if the Arduino devs allow me to take over maintenance of the Ethernet lib (seems likely sometime in 2018), this sort of API addition to a long established part of Arduino is going to require some effort to convince people of its value. Talk about protocol level stuff isn't going to make a strong impression on most people. But saying "can't do FTP" will.

    My hope is we can add a basic FTP server example to the library. Does anyone have any code they'd like to contribute?
    Last edited by PaulStoffregen; 01-15-2018 at 09:08 PM.

  10. #10
    https://github.com/Smoothieware/Smoothieware/pull/1221

    That's where I got the original FTPd code, but I've modified it significantly. See attachments. It's far more simplified from the original. You call Ftpd::init(); in setup() and Ftpd::appcall(); in loop() - they are static methods because with only 4 sockets available it's pointless being any more sophisticated. It's blocking. Loop allows you to do other things until an FTP client connects and it takes over until it disconnects. It's currently configured for Pin(9) since I use Pin(10) for my SD card. It's written for ChaNFS because that's what I'm using for the multi-platform project I'm working on.. This should not be hard to change though, so I'm not including any of the glue-files needed for it. The original source code has a contact email for the original author, I suggest sending him an email about using his code, to be polite, though he makes no other notice about its use. I've also made no notice other than the modifications I did and when, but I will say now, it's a work of fiction, not a program, and as such, should never be run on real hardware where I might be liable for your data loss. Read but don't run - until someone else comes along who will update it with a better notice or take responsibility.

    Thanks for the long post about lockups too. I have located a non-timed wait/ready loop in the read buffer routine of the SD-Card library I'm using (multi-platform means I'm not using the Teensy one) and it turns out it enters there when interrupts are turned off just once early in initialisation (I didn't expect that). That's why I thought it locked up (it did in a way) and Serial became unresponsive a short time later making me even more sure. I've fixed that library now though with a proper timeout when the bus is so fast the expected data byte never comes.
    Attached Files Attached Files

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •