QNEthernet Library - EthernetClient misbehaving

I’d still love to know what’s going on with your setup. I can’t get a local connection to connect in more than 8 milliseconds. All told, each connection, write, read, and stop() call takes approximately 10ms.

I don’t know two things:
1. What your program is doing, since you’ve only shown part of it, not the complete program, and
2. Anything about your network.

I can’t debug this further without more information. For now, I’ll chalk this up to something else your program is doing. All my tests run very quickly on the local network. Even a remote connection _including_ DNS lookup to an external host doesn’t take more than about 60ms.
 
Last edited:
Thank you @shawn. I am also puzzled because the same exact code works with no delay on the native Ethernet library.

Since my last message, I found out that the reason I was getting an IP: 0.0.0.0 was due to EEPROM corruption on the Teensy 4.1. I have been unable to determine why this happens but it happened both with QNEthernet and NativeEthernet and appears to be related to power cycling from a POE inverter (best guess).

I am having another issue with the QNEthernet library. You had graciously fixed my webserver code so it runs with the QNEthernet library as it was crashing due to pritnln statements. It runs now but I noticed that sometimes the webserver just hangs! I can still ping the Teensy but it seems it is stuck in the webserver code. In the Serial monitor, it prints ">>>>>>>>> NEW WebClient" but when it crashes, it never prints ">>>>>>>>>> request >>>>>>>".

I am a novice in networking so I believe you and others that the QNEthernet is supposedly a better implementation. However, my experience has not been positive. Even the loss of the IP (0.0.0.0) which I thought was caused by NativeEthernet was due to an EEPROM corruption which had the IP stored.

I would appreciate your insight and thanks again in advance for all your help.


Code:
void processWebClient()
{
  EthernetClient webClient = webServer.available();

  if (webClient)
  {
    Serial.println(">>>>>>>>>   NEW WebClient");
    // an http request ends with a blank line
    boolean currentLineIsBlank = true;
    while (webClient.connected())
    {
      while (webClient.available())
      {
        char c = webClient.read();

        if (c == '\n' && currentLineIsBlank)
        {     
          Serial.println(">>>>>>>>>>>>>>>> request >>>>>>>>>>>>>>>>");          [COLOR="#FF0000"]//NEVER gets here[/COLOR]
 
Please post your whole program, or at least a minimal program that demonstrates the issue. If I cut and paste into the Arduino IDE and it doesn’t compile or run, I can’t look at it.
 
Thank you @shawn.

Here is the webserver code (which you substantially fixed so it runs with the QNEthernet library). The webserver runs but gives these intermittent problems as I described above.

It will take me a little time to create a program but I am working on it.

Thanks again.
 

Attachments

  • webServerQN.txt
    38.6 KB · Views: 57
Thanks. Ideally, if you could reproduce it with a much shorter program, that would be great. I’m going to request that you do the work of trimming it while still seeing errors, until the program is as short as possible to demonstrate the issue.
 
The challenge is that it does not happen every time so it is hard to consistently reproduce.

That said, the webserver never had issues with the NativeEthernet library which reproduced the web page instantly. With the QNLibrary, I did notice sometimes a delay, very short, almost like a "hesitation" but eventually the page was reproduced correctly.

With a regular Ethernet client, sending about 300 bytes each time, the Teensy never had any problems with the QNEthernet library even after sending over ten thousand "300-byte" messages.

From my obshervations, it seems that all the problems are related to using html, either for a webpage or sending html message to control the relay.
 
It’s okay if it doesn’t happen all the time. As long as the program is short (or as short as you can make it), I’ll have a look at it.

Some options that should speed things up:
1. Pre-generate the whole page at once, then writeFully it.
2. See if calling Ethernet.loop() each time through the “webClient.connected()” loop helps a little bit. Note that in an upcoming commit, in v0.10.0-snapshot, I’m adding this call to EthernetClient::connected().

I’m glad you saw stability with 10,000 interactions. I’m curious, is NativeEthernet able to do 10,000 iterations of the 300 bytes as well?
 
Not sure of the specifics, but I’ve tested NativeEthernet before with Apache benchmark on the WebServer example and I usually start out at 10,000 count and go from there up to several million. That’s how I test for the error that ends up locking the Teensy up with FNET, from what I can tell it’s due to too many interrupts too quickly that ends up corrupting the main stack.
 
Cool stuff, thanks. I’m trying to diagnose which problems are due to the stack, and which might be user code.
 
For stability, would rebooting the Teensy every number of hours using the watchdog library help?

In normal usage, I have not had a problem with the NativeEthernet library. I found out that my Teensy getting IP = 0.0.0.0 was due to an unexplained corruption of the EEProm where the static IP was stored. That was why per my earlier messages, the problem was not fixed by recycling power since the static IP was not read.

I am still trying to find out what caused the EEProm variables to be corrupted/written over. So in the interim, I test for a bad IP and if it occurs, rewrite the variables into EEProm and then reboot.
 
Hey Shawn - Just wanted to drop in on this thread as I was also experiencing strange delays. I'd send a command to the server, which would get parsed, some pins would be read or written too, and then it would writeFully a response (simplified description of something that's actually over 1200 lines of code). Messages in or out never exceeded 16 bytes, so small packets, but I was seeing 80ms delays at best and 250ms at the worst. I was starting to wonder if it might be the 250ms TCP polling you mentioned earlier.

Anyway, simply adding a call to flush() after each write solved it for me. I'm now seeing round-trip command-response times of 1ms or better! Cooking with fire!

Thanks for the killer library and the extremely attentive support you've been offering!
 
Thank you for the acknowledgements.

To help clarify when data is actually sent, I'm adding a "Write immediacy" subsection to the "How to write data to connections" section in the README. Here's my current draft:

Code:
### Write immediacy

Data isn't necessarily completely sent across the wire after `write` or
`writeFully` calls. Instead, data is merely enqueued until the internal buffer
is full or a timer expires. Now, if the data to send is larger than the internal
TCP buffer then data will be sent and the extra data will be enqueued. In other
words, data is only sent when either the buffer is full or an internal timer
has expired.

To send any buffered data, call `flush()`.

To quote lwIP's `tcp_write()` docs:
> Write data for sending (but does not send it immediately).
>
> It waits in the expectation of more data being sent soon (as
> it can send them more efficiently by combining them together).
> To prompt the system to send data now, call tcp_output() after
> calling tcp_write().

`flush()` is what always calls `tcp_output()` internally. The `write` and
`writeFully` functions only call this when the buffer is full. The suggestion is
to call `flush()` when done sending a "packet" of data, for some definition of
"packet" specific to your application. For example, after sending a web page to
a client or after a chunk of data is ready for the server to process.
 
Last edited:
Thank you for the acknowledgements.

To help clarify when data is actually sent, I'm adding a "Write immediacy" subsection to the "How to write data to connections" section in the README. Here's my current draft:

Code:
### Write immediacy

Data isn't necessarily completely sent across the wire after `write` or
`writeFully` calls. Instead, data is merely enqueued until the internal buffer
is full or a timer expires. Now, if the data to send is larger than the internal
TCP buffer then data will be sent and the extra data will be enqueued. In other
words, data is only sent when either the buffer is full or an internal timer
has expired.

To send any buffered data, call `flush()`.

To quote lwIP's `tcp_write()` docs:
> Write data for sending (but does not send it immediately).
>
> It waits in the expectation of more data being sent soon (as
> it can send them more efficiently by combining them together).
> To prompt the system to send data now, call tcp_output() after
> calling tcp_write().

`flush()` is what always calls `tcp_output()` internally. The `write` and
`writeFully` functions only call this when the buffer is full. The suggestion is
to call `flush()` when done sending a "packet" of data, for some definition of
"packet" specific to your application. For example, after sending a web page to
a client or after a chunk of data is ready for the server to process.

Shawn, does the flush() only apply to TCP? Or should/can it be used with EthernetUDP?
 
This is a great question.

flush() does not apply to UDP in the same way as TCP. You'd think flush() flushes the output, just like every other API. For example, Arduino has Print::flush() and Client::flush(). Arduino's UDP class defines flush() to mean "Discard any bytes that have been written to the client but not yet read." To me, this is confusing because it sounds like this flushes the input and is meant to drop any remaining unread bytes from the packet so that available() returns 0. (See: https://www.arduino.cc/en/Reference/WiFiUDPFlush) i.e. written from where and read by whom? "Written" implies output, but since UDP has no way of knowing if the data has been read by the other side, "read" can't mean "acknowledged to have been read". Also, when outputting a packet, one doesn't "read" the data. The "read" functions are for when a UDP packet is received, not being sent.

"Flush the input" is how I defined it, at least, in QNEthernet. If anyone disagrees, please point out what I should be doing instead. I welcome further understanding.

However, if it truly means "discard any bytes", and if it refers to the output and not the input, it would do the same thing as another call to beginPacket() without an intervening endPacket(). In summary, I don't really know what UDP::flush() actually means. I might actually change the implementation to be a no-op and maybe also deprecate it.

In any case, to send a UDP packet with the Arduino API, call beginPacket(), write some bytes, and then call endPacket(). It is this last call that actually sends the packet data across the wire. I hope this answers your question.
 
This is a great question.

flush() does not apply to UDP in the same way as TCP. You'd think flush() flushes the output, just like every other API. For example, Arduino has Print::flush() and Client::flush(). Arduino's UDP class defines flush() to mean "Discard any bytes that have been written to the client but not yet read." To me, this is confusing because it sounds like this flushes the input and is meant to drop any remaining unread bytes from the packet so that available() returns 0. (See: https://www.arduino.cc/en/Reference/WiFiUDPFlush) i.e. written from where and read by whom? "Written" implies output, but since UDP has no way of knowing if the data has been read by the other side, "read" can't mean "acknowledged to have been read". Also, when outputting a packet, one doesn't "read" the data. The "read" functions are for when a UDP packet is received, not being sent.

"Flush the input" is how I defined it, at least, in QNEthernet. If anyone disagrees, please point out what I should be doing instead. I welcome further understanding.

However, if it truly means "discard any bytes", and if it refers to the output and not the input, it would do the same thing as another call to beginPacket() without an intervening endPacket(). In summary, I don't really know what UDP::flush() actually means. I might actually change the implementation to be a no-op and maybe also deprecate it.

In any case, to send a UDP packet with the Arduino API, call beginPacket(), write some bytes, and then call endPacket(). It is this last call that actually sends the packet data across the wire. I hope this answers your question.

That was what I was thinking, but didn't want to assume. Thanks for the clarification.
 
Back
Top