Forum Rule: Always post complete source code & details to reproduce any issue!
Results 1 to 23 of 23

Thread: Teensy 4.1: two ETH servers - FAILS

  1. #1

    Teensy 4.1: two ETH servers - FAILS

    I have one ETH server (for Web Browser, as HTTPD server) running. - Fine
    I want to add a second, similar ETH server, but with TCP port 8080.
    I have added code and I have two instances of server:

    EthernetServer server(80);
    EthernetServer serverNETCMD(8080);


    But this FAILS!
    After a while, I get "No data available, closing socket".
    I saw also an error message spit out (from deep inside the LIB), with an error code.

    Just one single server at a time runs fine. Two servers at the same time does not.
    WHY?
    This Arduino LIB is such a nightmare.

    I assume it does not work because of not enough DMA descriptors for ETH controller. It might run out
    of memory, out of DMA descriptors or the LwIP stack is confused about two parallel TCP sessions.

    How to instantiate TWO TCP servers, one on port 80 (for WebBrowser access), another one on port 8080
    (used for Python scripts)?

  2. #2
    Senior Member
    Join Date
    Mar 2017
    Location
    Oakland, CA, USA
    Posts
    693
    Which Ethernet library are you using? Are you able to post your whole program? What error message are you seeing? Is that “no data available” message being printed on the Teensy side or on your PC?

    I suspect that client connections aren’t being managed properly, or not closed correctly.

  3. #3
    I use #include <NativeEthernet.h>
    I am sure, client connection handling is correct (for one server): I see "new client" and "client disconnected" in pairs, one server runs forever fine.
    (both servers send HTTP response with "Refresh: 5", so, auto-refreshing in WebBrowser)

    I get this error:
    StateErr: -28

    The entire project is here:
    https://github.com/tjaekel/Teesny_4_1

    I tried to use a Mutex (lock) - it does not solve the problem!

    The code for the ETH servers (as HTTPD daemons):

    Code:
    
    ////#include <SPI.h>
    #include <NativeEthernet.h>
    #include <TeensyThreads.h>
    
    void thread_HTTPD(void);
    void thread_NETCMD(void);
    
    Threads::Mutex serverLock;
    
    // Enter a MAC address and IP address for your controller below.
    // The IP address will be dependent on your local network:
    byte mac[] = {
      0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED
    };
    IPAddress ip(192, 168, 0, 196);
    
    // Initialize the Ethernet server library
    // with the IP address and port you want to use
    // (port 80 is default for HTTP):
    EthernetServer server(80);
    EthernetServer serverNETCMD(8080);
    
    void HTTPD_setup(void) {
      // You can use Ethernet.init(pin) to configure the CS pin
      //Ethernet.init(10);  // Most Arduino shields
      //Ethernet.init(5);   // MKR ETH shield
      //Ethernet.init(0);   // Teensy 2.0
      //Ethernet.init(20);  // Teensy++ 2.0
      //Ethernet.init(15);  // ESP8266 with Adafruit Featherwing Ethernet
      //Ethernet.init(33);  // ESP32 with Adafruit Featherwing Ethernet
    
    #if 0
      // Open serial communications and wait for port to open:
      Serial.begin(9600);
      while (!Serial) {
        ; // wait for serial port to connect. Needed for native USB port only
      }
    #endif
      Serial.println("Ethernet WebServer Example");
    
      // start the Ethernet connection and the server:
      Ethernet.begin(mac, ip);
    
      // Check for Ethernet hardware present
      if (Ethernet.hardwareStatus() == EthernetNoHardware) {
        Serial.println("Ethernet shield was not found.  Sorry, can't run without hardware. :(");
        while (true) {
          delay(1); // do nothing, no point running without Ethernet hardware
        }
      }
      if (Ethernet.linkStatus() == LinkOFF) {
        Serial.println("Ethernet cable is not connected.");
      }
    
      // start the server
      server.begin();
      Serial.print("server is at: ");
      Serial.println(Ethernet.localIP());
    
      //run as a RTOS thread
      threads.addThread(thread_HTTPD);
      threads.addThread(thread_NETCMD);
    }
    
    uint32_t HTTPD_GetIPAddress(void) {
      return Ethernet.localIP();
    }
    
    void thread_HTTPD(void) {
      static uint32_t cnt = 0;
    
      while(1) {
        // listen for incoming clients
        EthernetClient client = server.available();
        if (client) {
    
          serverLock.lock();
    
          Serial.println("new client");
          // an http request ends with a blank line
          boolean currentLineIsBlank = false;
          while (client.connected()) {
            if (client.available()) {
              char c = client.read();
    #if 0
              //this prints all received characters on Serial
              Serial.write(c);
    #endif
              // if you've gotten to the end of the line (received a newline
              // character) and the line is blank, the http request has ended,
              // so you can send a reply
              if (c == '\n' && currentLineIsBlank) {
                // send a standard http response header
                client.println("HTTP/1.1 200 OK");
                client.println("Content-Type: text/html");
                client.println("Connection: close");  // the connection will be closed after completion of the response
                client.println("Refresh: 5");         // refresh the page automatically every 5 sec
                client.println();
                client.println("<!DOCTYPE HTML>");
                client.println("<html>");
                // avoid to request all the time a favicon - otherwise, we get always two requests!
                client.println("<head><link rel=\"icon\" href=\"data:,\"></head><body>");
    
    #if 0
                // output the value of each analog input pin
                for (int analogChannel = 0; analogChannel < 6; analogChannel++) {
                  int sensorReading = analogRead(analogChannel);
                  client.print("analog input ");
                  client.print(analogChannel);
                  client.print(" is ");
                  client.print(sensorReading);
                  client.println("<br />");
                }
    #else
                client.print("<h2>Welcome on Teensy: ");
                client.print(cnt);
                Serial.print("XXXX: "); Serial.println(cnt);
                cnt++;
                client.println("</h2>");
    #endif
                client.println("</body></html>");
                break;
              }
    
              if (c == '\n') {
                // you're starting a new line
                currentLineIsBlank = true;
              } else if (c != '\r') {
                // you've gotten a character on the current line
                currentLineIsBlank = false;
              }
            }
          }
          // give the web browser time to receive the data
          //delay(1);
          threads.delay(1);
          // close the connection:
          client.stop();
          Serial.println("client disconnected");
    
          serverLock.unlock();
        }
        threads.delay(1);
        threads.yield();
    
      }
    }
    
    void thread_NETCMD(void) {
      static uint32_t cnt = 0;
    
      while(1) {
        // listen for incoming clients
        EthernetClient client = serverNETCMD.available();
        if (client) {
    
          serverLock.lock();
    
          Serial.println("new client");
          // an http request ends with a blank line
          boolean currentLineIsBlank = false;
          while (client.connected()) {
            if (client.available()) {
              char c = client.read();
    #if 0
              //this prints all received characters on Serial
              Serial.write(c);
    #endif
              // if you've gotten to the end of the line (received a newline
              // character) and the line is blank, the http request has ended,
              // so you can send a reply
              if (c == '\n' && currentLineIsBlank) {
                // send a standard http response header
                client.println("HTTP/1.1 200 OK");
                client.println("Content-Type: text/html");
                client.println("Connection: close");  // the connection will be closed after completion of the response
                client.println("Refresh: 5");         // refresh the page automatically every 5 sec
                client.println();
                client.println("<!DOCTYPE HTML>");
                client.println("<html>");
                // avoid to request all the time a favicon - otherwise, we get always two requests!
                client.println("<head><link rel=\"icon\" href=\"data:,\"></head><body>");
    
    #if 0
                // output the value of each analog input pin
                for (int analogChannel = 0; analogChannel < 6; analogChannel++) {
                  int sensorReading = analogRead(analogChannel);
                  client.print("analog input ");
                  client.print(analogChannel);
                  client.print(" is ");
                  client.print(sensorReading);
                  client.println("<br />");
                }
    #else
                client.print("<h2>Teensy CMD: ");
                client.print(cnt);
                Serial.print("XXXX: "); Serial.println(cnt);
                cnt++;
                client.println("</h2>");
    #endif
                client.println("</body></html>");
                break;
              }
    
              if (c == '\n') {
                // you're starting a new line
                currentLineIsBlank = true;
              } else if (c != '\r') {
                // you've gotten a character on the current line
                currentLineIsBlank = false;
              }
            }
          }
          // give the web browser time to receive the data
          //delay(1);
          threads.delay(1);
          // close the connection:
          client.stop();
          Serial.println("client disconnected");
    
          serverLock.unlock();
        }
        threads.delay(1);
        threads.yield();
    
      }
    }
    
    My impression: two parallel servers are not supported by the "NativeEthernet" LIB
    (I do not see that I have QNEthernet LIB installed - I would like use this instead)
    Last edited by defragster; 05-25-2023 at 12:23 AM. Reason: Added [CODE] marking: # on post toolbar

  4. #4
    with the Mutex::Lock around I get now this (on my UART console):
    now "SendErr: -20"


    > ipaddr
    IP address: 192.168.0.196

    > new client
    XXXX: 0
    client disconnected
    new client
    XXXX: 1
    client disconnected
    new client
    XXXX: 2
    client disconnected
    Socket Event Err: -20
    new client
    SendErr: -20
    SendErr: -20
    SendErr: -20
    SendErr: -20
    SendErr: -20
    SendErr: -20
    SendErr: -20
    SendErr: -20
    SendErr: -20
    SendErr: -20
    SendErr: -20
    SendErr: -20
    SendErr: -20
    SendErr: -20
    SendErr: -20
    SendErr: -20
    SendErr: -20
    XXXX: 0
    SendErr: -20
    SendErr: -20
    SendErr: -20
    SendErr: -20
    client disconnected
    new client
    XXXX: 3
    client disconnected
    new client
    XXXX: 1
    client disconnected

  5. #5
    With a lock used this way:


    Threads::Mutex serverLock;

    void thread_HTTPD(void) {
    static uint32_t cnt = 0;

    while(1) {
    // listen for incoming clients

    serverLock.lock();

    EthernetClient client = server.available();
    if (client) {


    it works better, but I see randomly still:
    "No data available, closing socket"

    I guess, the Arduino LIB is not designed, not tested, for two parallel TCP servers running.

  6. #6
    Senior Member
    Join Date
    Oct 2016
    Posts
    1,045
    Quote Originally Posted by tjaekel View Post
    I guess, the Arduino LIB is not designed, not tested, for two parallel TCP servers running.
    Try using QNEthernet. NativeEthernet is no longer maintained, and QNEthernet is now preferred.

  7. #7
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    17,109
    Seeing: #include <TeensyThreads.h>

    Doesn't inspire confidence. Didn't read the code to understand usage - but interrupt task switching on a timer tick might be the problem if it affects code not designed for it and not used with switching in the middle of a critical process.

    There was another recent user of 'some' process that encountered such problems.

  8. #8
    Senior Member
    Join Date
    Oct 2016
    Posts
    1,045
    Quote Originally Posted by defragster View Post
    Seeing: #include <TeensyThreads.h>

    Doesn't inspire confidence. Didn't read the code to understand usage - but interrupt task switching on a timer tick might be the problem if it affects code not designed for it and not used with switching in the middle of a critical process.

    There was another recent user of 'some' process that encountered such problems.
    Oh, you're right. I didn't even read the code. As soon as I saw "NativeEthernet", I thought he should switch to QNEthernet. I'm pretty sure that Shawn just recently mentioned that QNEthernet is designed to run in one thread, so perhaps NativeEthernet is, too.

  9. #9
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    17,109
    Quote Originally Posted by joepasquariello View Post
    Oh, you're right. I didn't even read the code. As soon as I saw "NativeEthernet", I thought he should switch to QNEthernet. I'm pretty sure that Shawn just recently mentioned that QNEthernet is designed to run in one thread, so perhaps NativeEthernet is, too.
    No doubt QNEthernet with @Shawn more active - been playing with parts here.

    Since that was covered the next line was as far as I read. TeensyThreads is a very nice development - but with limited applications - or with extreme care to perhaps use it cooperatively with long switch times and then having thread yield when when safe.

  10. #10
    Senior Member
    Join Date
    Mar 2017
    Location
    Oakland, CA, USA
    Posts
    693
    Pro tip: when pasting code, use the "[CODE]" tags (or the "#" button), and that will keep the spacing and formatting. It's hard to follow when all the leading spacing is missing. (Yes, cut and paste into an IDE that reformats will work, but it's much easier to read the code if it's formatted well.)

  11. #11
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    17,109
    Quote Originally Posted by shawn View Post
    Pro tip: when pasting code, use the "[CODE]" tags (or the "#" button), and that will keep the spacing and formatting. It's hard to follow when all the leading spacing is missing. (Yes, cut and paste into an IDE that reformats will work, but it's much easier to read the code if it's formatted well.)
    That is a good tip - Edited p#1 above ...

  12. #12
    Senior Member
    Join Date
    Mar 2017
    Location
    Oakland, CA, USA
    Posts
    693
    In theory, QNEthernet can run from a single thread, but it's not designed for being used from multiple threads or contexts (one example is some access from an ISR and some from somewhere else). I also find that performance improves greatly without threads. I usually have a bunch of separate "mini/sub programs" that are called in sequence from the main loop. I've so far not had a problem with that approach. You can even check if a program is available to run and run it more than once in that loop. For example, for programs A, B, and C: loop{ABC} or: loop{ABAC}.

    I'm 100% sure that QNEthernet runs well with multiple servers because I've done it in multiple commercial projects.

    Try these two things (switching to QNEthernet and then no threads) and see what happens. (I'm also curious how your program will behave with just a switch to QNEthernet while keeping the threads.)

    Note: the NativeEthernet library uses a 1ms timer tick (i.e. from an ISR) to execute Ethernet things, so it uses FNET's (the underlying stack) "multi-context" features. Doing _anything_ with multithreading, especially inside a library that needs to work within a very large number of different types of programs is very hard to get right. This is one of the reasons I designed QNEthernet to be run in a single-threaded fashion. One of my other main reasons is because I often prefer the single loop style: check a list of "things" to see if each is ready to process something, and then process each "thing".

    Some related links:
    See this conversation, especially these two posts: https://github.com/ssilverman/QNEthe...ent-1550481187 and https://github.com/ssilverman/QNEthe...ent-1551014434

    I also advise against using `print` or `println` when outputting to connections because it's possible, if, say, some buffer is full, that not everything will get sent, and you won't know about it. See some notes here: https://github.com/ssilverman/QNEthe...to-connections
    Last edited by shawn; 05-25-2023 at 12:44 AM.

  13. #13
    Thank you all.
    Changing to QNEthernet and using AsyncWebServer works much better (two web browsers connected to Teensy).
    TFTP needs a modification to use with QNEthernet.
    And TFTP runs in parallel with the two web servers - COOL.

    Working project is here:
    https://github.com/tjaekel/Teesny_4_1

    BTW:
    there is potentially an issue with TCP in general:
    if web browser, or my Python script, opens all the time a new connection (and closing after one request) - we can get "out of TCP ports", on the host side (as source port),
    not a FW issue, an issue with assigning source ports and having "zombie ports" for a while.
    So, for Python scripts, firing very often and fast requests on server - actually, I keep the connection open.
    Not tested yet with Python, just a heads up: if always connect and close and very fast - we can run out of source TCP ports (on the host side).

  14. #14
    Senior Member
    Join Date
    Mar 2017
    Location
    Oakland, CA, USA
    Posts
    693
    Note that AsyncWebServer (and its AsyncTCP dependency) doesn’t really use QNEthernet; it uses the library’s included lwIP stack (it’s what QNEthernet uses under the covers). My opinion is that those “Async” libraries should make their own distribution on top of lwIP.

  15. #15
    I think it is not true: AsyncWebServer seems to use/need QNEthernet:
    https://github.com/khoih-prog/AsyncWebServer_Teensy41

    It does not work without to include "QNEthernet.h" (compile errors).

    It needs this:

    #include "QNEthernet.h" // https://github.com/ssilverman/QNEthernet
    using namespace qindesign::network;

    #include <AsyncWebServer_Teensy41.h>

  16. #16
    Senior Member
    Join Date
    Mar 2017
    Location
    Oakland, CA, USA
    Posts
    693
    It uses the initialization.

  17. #17
    "rrrrr" - I could cry:
    running 2x AsyncWebServer plus TFTP server - it works fine for approx. half an hour, maximum one hour.
    The CPU resets after while. (I hear the disconnect on USB and I see it has booted by itself)

    No idea why, no idea how to debug and fix.
    Not acceptable as a reliable, professional implementation or industrial platform (so sad, because the HW is nice,
    just the IDE, Arduino, the LIB code... lacks)

    It brings me again to: "rrrrr - this Arduino stuff drives me crazy..."
    (why do they promise that Arduino and their boards would be for industrial, professional use? Even the Portenta H7 was already a nightmare,
    now this Teensy 4.1 again and as well...).

  18. #18
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    17,109
    Quote Originally Posted by tjaekel View Post
    "rrrrr" - I could cry:
    ...
    No idea why, no idea how to debug and fix.
    ...
    Is there a check and Serial.print(CrashReport) in setup() with a wait for Serial when if(CrashReport) is found?

    That is the first thing to check for a clue/guide.

  19. #19
    Cool, not knowing this feature.
    I have added and a new run (after flashing and run again) gives me this:


    CrashReport:
    A problem occurred at (system time) 18:28:2
    Code was executing from address 0x1B838
    CFSR: 82
    (DACCVIOL) Data Access Violation
    (MMARVALID) Accessed Address: 0x10 (nullptr)
    Check code at 0x1B838 - very likely a bug!
    Run "addr2line -e mysketch.ino.elf 0x1B838" for filename & line number.
    Temperature inside the chip was 58.16 C
    Startup CPU clock speed is 600MHz
    Breadcrumb #3 was 808126556 (0x302B085C)
    Breadcrumb #4 was 1897883742 (0x711F685E)
    Breadcrumb #5 was 2014081909 (0x780C7375)
    Breadcrumb #6 was 3997603853 (0xEE46980D)


    I am still running to hit the error again (this log is potentially from previous run).
    I have to see where to find this *.ELF file (in TEMP directory?)

  20. #20
    I found the ELF in C:/users/<user>/AppData/Local/Temp/arduino=sketch-B8......
    But I do not have "addr2line" (launched on command line).

    And I saw now new crash report (after few minutes):


    > CrashReport:
    A problem occurred at (system time) 19:10:21
    Code was executing from address 0x1B828
    CFSR: 82
    (DACCVIOL) Data Access Violation
    (MMARVALID) Accessed Address: 0xC (nullptr)
    Check code at 0x1B828 - very likely a bug!
    Run "addr2line -e mysketch.ino.elf 0x1B828" for filename & line number.
    Temperature inside the chip was 59.63 C
    Startup CPU clock speed is 600MHz
    Reboot was caused by auto reboot after fault or bad interrupt detected


    I have checked the generate *.lst file:
    It shows me these lines of code:

    sgmtype = (fnet_uint8_t)(FNET_TCP_FLAGS(insegment));
    1b824: 9904 ldr r1, [sp, #16]
    fnet_uint32_t tcp_ack = fnet_ntohl(FNET_TCP_ACK(insegment));
    1b826: 4683 mov fp, r0
    switch(cb->tcpcb_connection_state)
    1b828: f898 2080 ldrb.w r2, [r8, #128] ; 0x80
    sgmtype = (fnet_uint8_t)(FNET_TCP_FLAGS(insegment));
    1b82c: 68cb ldr r3, [r1, #12]
    switch(cb->tcpcb_connection_state)


    assuming inside this function:

    0001b730 <_fnet_tcp_input>:


    So, deep inside the FNET LIB function.
    But what is the reason?

    I assume: this QNEthernet is not designed for multi-threading:
    I run it in a TeensyThread thread (also the TFTP is a thread).

  21. #21
    Senior Member
    Join Date
    Mar 2017
    Location
    Oakland, CA, USA
    Posts
    693
    QNEthernet is explicitly not designed for asynchronous use.
    See here: https://github.com/ssilverman/QNEthe...ent-1550481187
    And here: https://github.com/ssilverman/QNEthe...ent-1551014434

    Also, FNET is used by NativeEthernet (which does things asynchronously every millisecond via an ISR), not QNEthernet. QNEthernet uses lwIP under the covers and doesn’t support being called from multiple contexts (eg. threads). It appears there are things being included that aren’t compatible. I use multiple servers all the time, and they’re quick and reliable.

    My suggestion is to not use threads nor anything asynchronous for the networking parts. Also, be sure you’re only building in one networking library.

  22. #22
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    17,109
    Quote Originally Posted by tjaekel View Post
    Cool, not knowing this feature.
    ...
    I am still running to hit the error again (this log is potentially from previous run).
    I have to see where to find this *.ELF file (in TEMP directory?)
    As long as power is maintained the prior CrashReport data is maintained in a small piece of RAM2/DMAMEM that is not cleared on Warm Restart.

    You likely didn't add breadcrumbs either - but some 32 bit values are also stored at program request - and the PJRC code does some verification before printing 'breadcrumbs' - somehow they end up being showed as stored when they have not been.

    Quick scan it looked like TeensyThreads was dropped - but the #define just must have moved. @shawn provided the QNEthernet code so not using threading as noted in prior posts seems the next step.

  23. #23
    Meanwhile, I think, the problem comes from using 'real threads', via TeensyThreads:
    - not doing so, all done inside loop() (even tricky to make all sub-functions non-blocking there) - it seems to work fine

    My TFTP (launched on UART command line via "tftp 1") is still a real thread, but the only one.
    It works still fine, except: when I run my Pico-C interpreter (via UART command line "picoc") - this crashes all, immediately.

    So, I guess:
    - TeensyThreads does not work together with AsyncWebServer
    - TeeesyThread has a small default stack size (and sub-functions in a thread "eat up" the stack space for a thread and corrupts other memory)

    So far OK, except: "tftp 1" launched and "picoc" - it crashes still (also a stack_size/memory corruption issue?).

    Latest version of project:
    https://github.com/tjaekel/Teesny_4_1

    BTW: for the Pico-C - you need external QSPI RAM soldered (2x): all the scripts (the Pico-C heap) are located there.
    If you do not have - do not launch Pico-C (or add a cross-check if 16 MB QSPI RAM (EXTMEM) is there)

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •