Teensy 4.1: two ETH servers - FAILS

tjaekel

Well-known member
I have one ETH server (for Web Browser, as HTTPD server) running. - Fine
I want to add a second, similar ETH server, but with TCP port 8080.
I have added code and I have two instances of server:

EthernetServer server(80);
EthernetServer serverNETCMD(8080);


But this FAILS!
After a while, I get "No data available, closing socket".
I saw also an error message spit out (from deep inside the LIB), with an error code.

Just one single server at a time runs fine. Two servers at the same time does not.
WHY?
This Arduino LIB is such a nightmare.

I assume it does not work because of not enough DMA descriptors for ETH controller. It might run out
of memory, out of DMA descriptors or the LwIP stack is confused about two parallel TCP sessions.

How to instantiate TWO TCP servers, one on port 80 (for WebBrowser access), another one on port 8080
(used for Python scripts)?
 
Which Ethernet library are you using? Are you able to post your whole program? What error message are you seeing? Is that “no data available” message being printed on the Teensy side or on your PC?

I suspect that client connections aren’t being managed properly, or not closed correctly.
 
I use #include <NativeEthernet.h>
I am sure, client connection handling is correct (for one server): I see "new client" and "client disconnected" in pairs, one server runs forever fine.
(both servers send HTTP response with "Refresh: 5", so, auto-refreshing in WebBrowser)

I get this error:
StateErr: -28

The entire project is here:
https://github.com/tjaekel/Teesny_4_1

I tried to use a Mutex (lock) - it does not solve the problem!

The code for the ETH servers (as HTTPD daemons):

Code:
[FONT=Courier New]
////#include <SPI.h>
#include <NativeEthernet.h>
#include <TeensyThreads.h>

void thread_HTTPD(void);
void thread_NETCMD(void);

Threads::Mutex serverLock;

// Enter a MAC address and IP address for your controller below.
// The IP address will be dependent on your local network:
byte mac[] = {
  0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED
};
IPAddress ip(192, 168, 0, 196);

// Initialize the Ethernet server library
// with the IP address and port you want to use
// (port 80 is default for HTTP):
EthernetServer server(80);
EthernetServer serverNETCMD(8080);

void HTTPD_setup(void) {
  // You can use Ethernet.init(pin) to configure the CS pin
  //Ethernet.init(10);  // Most Arduino shields
  //Ethernet.init(5);   // MKR ETH shield
  //Ethernet.init(0);   // Teensy 2.0
  //Ethernet.init(20);  // Teensy++ 2.0
  //Ethernet.init(15);  // ESP8266 with Adafruit Featherwing Ethernet
  //Ethernet.init(33);  // ESP32 with Adafruit Featherwing Ethernet

#if 0
  // Open serial communications and wait for port to open:
  Serial.begin(9600);
  while (!Serial) {
    ; // wait for serial port to connect. Needed for native USB port only
  }
#endif
  Serial.println("Ethernet WebServer Example");

  // start the Ethernet connection and the server:
  Ethernet.begin(mac, ip);

  // Check for Ethernet hardware present
  if (Ethernet.hardwareStatus() == EthernetNoHardware) {
    Serial.println("Ethernet shield was not found.  Sorry, can't run without hardware. :(");
    while (true) {
      delay(1); // do nothing, no point running without Ethernet hardware
    }
  }
  if (Ethernet.linkStatus() == LinkOFF) {
    Serial.println("Ethernet cable is not connected.");
  }

  // start the server
  server.begin();
  Serial.print("server is at: ");
  Serial.println(Ethernet.localIP());

  //run as a RTOS thread
  threads.addThread(thread_HTTPD);
  threads.addThread(thread_NETCMD);
}

uint32_t HTTPD_GetIPAddress(void) {
  return Ethernet.localIP();
}

void thread_HTTPD(void) {
  static uint32_t cnt = 0;

  while(1) {
    // listen for incoming clients
    EthernetClient client = server.available();
    if (client) {

      serverLock.lock();

      Serial.println("new client");
      // an http request ends with a blank line
      boolean currentLineIsBlank = false;
      while (client.connected()) {
        if (client.available()) {
          char c = client.read();
#if 0
          //this prints all received characters on Serial
          Serial.write(c);
#endif
          // if you've gotten to the end of the line (received a newline
          // character) and the line is blank, the http request has ended,
          // so you can send a reply
          if (c == '\n' && currentLineIsBlank) {
            // send a standard http response header
            client.println("HTTP/1.1 200 OK");
            client.println("Content-Type: text/html");
            client.println("Connection: close");  // the connection will be closed after completion of the response
            client.println("Refresh: 5");         // refresh the page automatically every 5 sec
            client.println();
            client.println("<!DOCTYPE HTML>");
            client.println("<html>");
            // avoid to request all the time a favicon - otherwise, we get always two requests!
            client.println("<head><link rel=\"icon\" href=\"data:,\"></head><body>");

#if 0
            // output the value of each analog input pin
            for (int analogChannel = 0; analogChannel < 6; analogChannel++) {
              int sensorReading = analogRead(analogChannel);
              client.print("analog input ");
              client.print(analogChannel);
              client.print(" is ");
              client.print(sensorReading);
              client.println("<br />");
            }
#else
            client.print("<h2>Welcome on Teensy: ");
            client.print(cnt);
            Serial.print("XXXX: "); Serial.println(cnt);
            cnt++;
            client.println("</h2>");
#endif
            client.println("</body></html>");
            break;
          }

          if (c == '\n') {
            // you're starting a new line
            currentLineIsBlank = true;
          } else if (c != '\r') {
            // you've gotten a character on the current line
            currentLineIsBlank = false;
          }
        }
      }
      // give the web browser time to receive the data
      //delay(1);
      threads.delay(1);
      // close the connection:
      client.stop();
      Serial.println("client disconnected");

      serverLock.unlock();
    }
    threads.delay(1);
    threads.yield();

  }
}

void thread_NETCMD(void) {
  static uint32_t cnt = 0;

  while(1) {
    // listen for incoming clients
    EthernetClient client = serverNETCMD.available();
    if (client) {

      serverLock.lock();

      Serial.println("new client");
      // an http request ends with a blank line
      boolean currentLineIsBlank = false;
      while (client.connected()) {
        if (client.available()) {
          char c = client.read();
#if 0
          //this prints all received characters on Serial
          Serial.write(c);
#endif
          // if you've gotten to the end of the line (received a newline
          // character) and the line is blank, the http request has ended,
          // so you can send a reply
          if (c == '\n' && currentLineIsBlank) {
            // send a standard http response header
            client.println("HTTP/1.1 200 OK");
            client.println("Content-Type: text/html");
            client.println("Connection: close");  // the connection will be closed after completion of the response
            client.println("Refresh: 5");         // refresh the page automatically every 5 sec
            client.println();
            client.println("<!DOCTYPE HTML>");
            client.println("<html>");
            // avoid to request all the time a favicon - otherwise, we get always two requests!
            client.println("<head><link rel=\"icon\" href=\"data:,\"></head><body>");

#if 0
            // output the value of each analog input pin
            for (int analogChannel = 0; analogChannel < 6; analogChannel++) {
              int sensorReading = analogRead(analogChannel);
              client.print("analog input ");
              client.print(analogChannel);
              client.print(" is ");
              client.print(sensorReading);
              client.println("<br />");
            }
#else
            client.print("<h2>Teensy CMD: ");
            client.print(cnt);
            Serial.print("XXXX: "); Serial.println(cnt);
            cnt++;
            client.println("</h2>");
#endif
            client.println("</body></html>");
            break;
          }

          if (c == '\n') {
            // you're starting a new line
            currentLineIsBlank = true;
          } else if (c != '\r') {
            // you've gotten a character on the current line
            currentLineIsBlank = false;
          }
        }
      }
      // give the web browser time to receive the data
      //delay(1);
      threads.delay(1);
      // close the connection:
      client.stop();
      Serial.println("client disconnected");

      serverLock.unlock();
    }
    threads.delay(1);
    threads.yield();

  }
}
[/FONT]

My impression: two parallel servers are not supported by the "NativeEthernet" LIB
(I do not see that I have QNEthernet LIB installed - I would like use this instead)
 
Last edited by a moderator:
with the Mutex::Lock around I get now this (on my UART console):
now "SendErr: -20"


> ipaddr
IP address: 192.168.0.196

> new client
XXXX: 0
client disconnected
new client
XXXX: 1
client disconnected
new client
XXXX: 2
client disconnected
Socket Event Err: -20
new client
SendErr: -20
SendErr: -20
SendErr: -20
SendErr: -20
SendErr: -20
SendErr: -20
SendErr: -20
SendErr: -20
SendErr: -20
SendErr: -20
SendErr: -20
SendErr: -20
SendErr: -20
SendErr: -20
SendErr: -20
SendErr: -20
SendErr: -20
XXXX: 0
SendErr: -20
SendErr: -20
SendErr: -20
SendErr: -20
client disconnected
new client
XXXX: 3
client disconnected
new client
XXXX: 1
client disconnected
 
With a lock used this way:


Threads::Mutex serverLock;

void thread_HTTPD(void) {
static uint32_t cnt = 0;

while(1) {
// listen for incoming clients

serverLock.lock();

EthernetClient client = server.available();
if (client) {


it works better, but I see randomly still:
"No data available, closing socket"

I guess, the Arduino LIB is not designed, not tested, for two parallel TCP servers running.
 
Seeing: #include <TeensyThreads.h>

Doesn't inspire confidence. Didn't read the code to understand usage - but interrupt task switching on a timer tick might be the problem if it affects code not designed for it and not used with switching in the middle of a critical process.

There was another recent user of 'some' process that encountered such problems.
 
Seeing: #include <TeensyThreads.h>

Doesn't inspire confidence. Didn't read the code to understand usage - but interrupt task switching on a timer tick might be the problem if it affects code not designed for it and not used with switching in the middle of a critical process.

There was another recent user of 'some' process that encountered such problems.

Oh, you're right. I didn't even read the code. As soon as I saw "NativeEthernet", I thought he should switch to QNEthernet. I'm pretty sure that Shawn just recently mentioned that QNEthernet is designed to run in one thread, so perhaps NativeEthernet is, too.
 
Oh, you're right. I didn't even read the code. As soon as I saw "NativeEthernet", I thought he should switch to QNEthernet. I'm pretty sure that Shawn just recently mentioned that QNEthernet is designed to run in one thread, so perhaps NativeEthernet is, too.

No doubt QNEthernet with @Shawn more active - been playing with parts here.

Since that was covered the next line was as far as I read. TeensyThreads is a very nice development - but with limited applications - or with extreme care to perhaps use it cooperatively with long switch times and then having thread yield when when safe.
 
Pro tip: when pasting code, use the "
Code:
" tags (or the "#" button), and that will keep the spacing and formatting. It's hard to follow when all the leading spacing is missing. (Yes, cut and paste into an IDE that reformats will work, but it's much easier to read the code if it's formatted well.)
 
Pro tip: when pasting code, use the "
Code:
" tags (or the "#" button), and that will keep the spacing and formatting. It's hard to follow when all the leading spacing is missing. (Yes, cut and paste into an IDE that reformats will work, but it's much easier to read the code if it's formatted well.)[/QUOTE]

That is a good tip - Edited p#1 above ...
 
In theory, QNEthernet can run from a single thread, but it's not designed for being used from multiple threads or contexts (one example is some access from an ISR and some from somewhere else). I also find that performance improves greatly without threads. I usually have a bunch of separate "mini/sub programs" that are called in sequence from the main loop. I've so far not had a problem with that approach. You can even check if a program is available to run and run it more than once in that loop. For example, for programs A, B, and C: loop{ABC} or: loop{ABAC}.

I'm 100% sure that QNEthernet runs well with multiple servers because I've done it in multiple commercial projects.

Try these two things (switching to QNEthernet and then no threads) and see what happens. (I'm also curious how your program will behave with just a switch to QNEthernet while keeping the threads.)

Note: the NativeEthernet library uses a 1ms timer tick (i.e. from an ISR) to execute Ethernet things, so it uses FNET's (the underlying stack) "multi-context" features. Doing _anything_ with multithreading, especially inside a library that needs to work within a very large number of different types of programs is very hard to get right. This is one of the reasons I designed QNEthernet to be run in a single-threaded fashion. One of my other main reasons is because I often prefer the single loop style: check a list of "things" to see if each is ready to process something, and then process each "thing".

Some related links:
See this conversation, especially these two posts: https://github.com/ssilverman/QNEthernet/issues/39#issuecomment-1550481187 and https://github.com/ssilverman/QNEthernet/issues/39#issuecomment-1551014434

I also advise against using `print` or `println` when outputting to connections because it's possible, if, say, some buffer is full, that not everything will get sent, and you won't know about it. See some notes here: https://github.com/ssilverman/QNEth...2a/README.md#how-to-write-data-to-connections
 
Last edited:
Thank you all.
Changing to QNEthernet and using AsyncWebServer works much better (two web browsers connected to Teensy).
TFTP needs a modification to use with QNEthernet.
And TFTP runs in parallel with the two web servers - COOL.

Working project is here:
https://github.com/tjaekel/Teesny_4_1

BTW:
there is potentially an issue with TCP in general:
if web browser, or my Python script, opens all the time a new connection (and closing after one request) - we can get "out of TCP ports", on the host side (as source port),
not a FW issue, an issue with assigning source ports and having "zombie ports" for a while.
So, for Python scripts, firing very often and fast requests on server - actually, I keep the connection open.
Not tested yet with Python, just a heads up: if always connect and close and very fast - we can run out of source TCP ports (on the host side).
 
Note that AsyncWebServer (and its AsyncTCP dependency) doesn’t really use QNEthernet; it uses the library’s included lwIP stack (it’s what QNEthernet uses under the covers). My opinion is that those “Async” libraries should make their own distribution on top of lwIP.
 
"rrrrr" - I could cry:
running 2x AsyncWebServer plus TFTP server - it works fine for approx. half an hour, maximum one hour.
The CPU resets after while. (I hear the disconnect on USB and I see it has booted by itself)

No idea why, no idea how to debug and fix.
Not acceptable as a reliable, professional implementation or industrial platform (so sad, because the HW is nice,
just the IDE, Arduino, the LIB code... lacks)

It brings me again to: "rrrrr - this Arduino stuff drives me crazy..."
(why do they promise that Arduino and their boards would be for industrial, professional use? Even the Portenta H7 was already a nightmare,
now this Teensy 4.1 again and as well...).
 
"rrrrr" - I could cry:
...
No idea why, no idea how to debug and fix.
...

Is there a check and Serial.print(CrashReport) in setup() with a wait for Serial when if(CrashReport) is found?

That is the first thing to check for a clue/guide.
 
Cool, not knowing this feature.
I have added and a new run (after flashing and run again) gives me this:


CrashReport:
A problem occurred at (system time) 18:28:2
Code was executing from address 0x1B838
CFSR: 82
(DACCVIOL) Data Access Violation
(MMARVALID) Accessed Address: 0x10 (nullptr)
Check code at 0x1B838 - very likely a bug!
Run "addr2line -e mysketch.ino.elf 0x1B838" for filename & line number.
Temperature inside the chip was 58.16 °C
Startup CPU clock speed is 600MHz
Breadcrumb #3 was 808126556 (0x302B085C)
Breadcrumb #4 was 1897883742 (0x711F685E)
Breadcrumb #5 was 2014081909 (0x780C7375)
Breadcrumb #6 was 3997603853 (0xEE46980D)


I am still running to hit the error again (this log is potentially from previous run).
I have to see where to find this *.ELF file (in TEMP directory?)
 
I found the ELF in C:/users/<user>/AppData/Local/Temp/arduino=sketch-B8......
But I do not have "addr2line" (launched on command line).

And I saw now new crash report (after few minutes):


> CrashReport:
A problem occurred at (system time) 19:10:21
Code was executing from address 0x1B828
CFSR: 82
(DACCVIOL) Data Access Violation
(MMARVALID) Accessed Address: 0xC (nullptr)
Check code at 0x1B828 - very likely a bug!
Run "addr2line -e mysketch.ino.elf 0x1B828" for filename & line number.
Temperature inside the chip was 59.63 °C
Startup CPU clock speed is 600MHz
Reboot was caused by auto reboot after fault or bad interrupt detected


I have checked the generate *.lst file:
It shows me these lines of code:

sgmtype = (fnet_uint8_t)(FNET_TCP_FLAGS(insegment));
1b824: 9904 ldr r1, [sp, #16]
fnet_uint32_t tcp_ack = fnet_ntohl(FNET_TCP_ACK(insegment));
1b826: 4683 mov fp, r0
switch(cb->tcpcb_connection_state)
1b828: f898 2080 ldrb.w r2, [r8, #128] ; 0x80
sgmtype = (fnet_uint8_t)(FNET_TCP_FLAGS(insegment));
1b82c: 68cb ldr r3, [r1, #12]
switch(cb->tcpcb_connection_state)


assuming inside this function:

0001b730 <_fnet_tcp_input>:


So, deep inside the FNET LIB function.
But what is the reason?

I assume: this QNEthernet is not designed for multi-threading:
I run it in a TeensyThread thread (also the TFTP is a thread).
 
QNEthernet is explicitly not designed for asynchronous use.
See here: https://github.com/ssilverman/QNEthernet/issues/39#issuecomment-1550481187
And here: https://github.com/ssilverman/QNEthernet/issues/39#issuecomment-1551014434

Also, FNET is used by NativeEthernet (which does things asynchronously every millisecond via an ISR), not QNEthernet. QNEthernet uses lwIP under the covers and doesn’t support being called from multiple contexts (eg. threads). It appears there are things being included that aren’t compatible. I use multiple servers all the time, and they’re quick and reliable.

My suggestion is to not use threads nor anything asynchronous for the networking parts. Also, be sure you’re only building in one networking library.
 
Cool, not knowing this feature.
...
I am still running to hit the error again (this log is potentially from previous run).
I have to see where to find this *.ELF file (in TEMP directory?)

As long as power is maintained the prior CrashReport data is maintained in a small piece of RAM2/DMAMEM that is not cleared on Warm Restart.

You likely didn't add breadcrumbs either - but some 32 bit values are also stored at program request - and the PJRC code does some verification before printing 'breadcrumbs' - somehow they end up being showed as stored when they have not been.

Quick scan it looked like TeensyThreads was dropped - but the #define just must have moved. @shawn provided the QNEthernet code so not using threading as noted in prior posts seems the next step.
 
Meanwhile, I think, the problem comes from using 'real threads', via TeensyThreads:
- not doing so, all done inside loop() (even tricky to make all sub-functions non-blocking there) - it seems to work fine

My TFTP (launched on UART command line via "tftp 1") is still a real thread, but the only one.
It works still fine, except: when I run my Pico-C interpreter (via UART command line "picoc") - this crashes all, immediately.

So, I guess:
- TeensyThreads does not work together with AsyncWebServer
- TeeesyThread has a small default stack size (and sub-functions in a thread "eat up" the stack space for a thread and corrupts other memory)

So far OK, except: "tftp 1" launched and "picoc" - it crashes still (also a stack_size/memory corruption issue?).

Latest version of project:
https://github.com/tjaekel/Teesny_4_1

BTW: for the Pico-C - you need external QSPI RAM soldered (2x): all the scripts (the Pico-C heap) are located there.
If you do not have - do not launch Pico-C (or add a cross-check if 16 MB QSPI RAM (EXTMEM) is there)
 
Back
Top