Inherent Latency in Ethernet Modbus over Serial with Teensy 4.1?

[milesg]

Hi All,
I've been working on a project where the circuitry allows for either Ethernet or Serial connectivity based on a jumper position. The code is essentially identical between either mode of communication, the only difference being some ternary operators set up to make sure I'm addressing the correct server:

I'm using QNEthernet and ArduinoModbus libraries.

// Here's an example of how I'm getting data from the chosen connection to set my output:
  for (uint8_t i=0; i<outputChannelsNum; i++) {
    boolean state = (jumperMode==2)?modbusTCPServer.coilRead(i+1):ModbusRTUServer.coilRead(i+1); // Get target state from coils 1-24
    if (digitalRead(outputPins[i])!=state) digitalWrite(outputPins[i], state); // write new value if changed
  }

Everything works great! No issues with the circuit or function of the modbus server with the jumper set in either position. The concerning bit is that my code responds to modbus calls over Serial in about 25ms (that's at 9600 Baud), and about 240ms when using Ethernet.

Here's the loop I'm using:

void loop() {
  if (jumperMode==2) { // Ethernet loop
    qn::EthernetClient client = EthServer.available();
    if (client) {
      modbusTCPServer.accept(client);
      if (client.connected()) { // Examples shows while() here. This prevents the loop from recovering when the cable is disconnected. if() resolves the issue.
        if (modbusTCPServer.poll()) { // poll() returns true when a request is present
          if (modbusTCPServer.holdingRegisterRead(44)) saveDefaults();
          if (modbusTCPServer.holdingRegisterRead(45)) closeAll();
          if (modbusTCPServer.holdingRegisterRead(46)) FSVAll();
          updateOutput();
        }
      }
    }
  }

  if (jumperMode==3 && ModbusRTUServer.poll()) { // Serial loop
    if (ModbusRTUServer.holdingRegisterRead(44)) saveDefaults();
    if (ModbusRTUServer.holdingRegisterRead(45)) closeAll();
    if (ModbusRTUServer.holdingRegisterRead(46)) FSVAll();
    updateOutput();
  }
}

You can see that I've copied the example code nearly 1:1 (the only difference in structure is that I changed a 'while' to an 'if' as noted, maybe there's a better way to deal with the issue I had there.) Of course there are the extra things that need to be done to work with the Ethernet server, but the extra latency seems excessive.

Is this normal behavior, or am I doing something wrong? I believe I've posted the relevant code, but let me know if you want to see more details.
Thanks for any insight that can be given on this matter.

 

[shawn]

If you can share a complete, minimal, example that shows the issue, maybe I could have a look. Also describe how to test it.

 

[milesg]

Alright, so here's a simpler version:

It just sets up the server with only one coil and the ability to switch between communication modes based on the jumper position at startup (the jumper is made for configuration at the time of assembly, not changing while the code is running or anything fancy like that). I have a test PCB with a MAX3232 chip on the Uart port and the T4.1 Ethernet kit for the TCP comms. So I'm testing, where this device is plugged into my Gigabit router and PC's COM port. Not sure what the easiest way for someone else to test this might be, but I think the intent is pretty clear here. Note that for my PCB I edited RS485.cpp to use Serial5, and I've bumped up the serial speed to 19200, but See the screenshots below to see the results of running this code.

#include <QNEthernet.h> // Most optimized Ethernet library for Teensy 4.1 Seems to have resolved memory leak in other libraries
#include <ArduinoModbus.h> // Dependency ArduinoRS485\RS485.cpp must have line 209 edited to use Serial5 and associated pins.
#include <EEPROM.h> // Enable access to EEPROM on Teensy board
namespace qn = qindesign::network;

qn::EthernetServer EthServer(502);
ModbusTCPServer modbusTCPServer;

void updateOutput(); // Updates all output pins to high/low based on Modbus coil values

// Global variables
byte mac[] = { 0x9C, 0x52, 0x69, 0x62, 0x65, 0x72 }; // Default MAC address
byte ip[] = { 192, 168, 1, 50 }; // Default IP address
byte subnet[] = { 255, 255, 255, 0 }; // Default Subnet mask
uint8_t jumperMode = 0; // Set during setup() to select boot mode. 1=Load defaults, 2=Ethernet, 3=Serial

void setup() {
  for (uint8_t i=2; i<5; i++) { // Set up input pins
    pinMode(i, INPUT_PULLUP); // Set all 3 jumper positions as inputs
  }
  pinMode(13, OUTPUT);

  delay(1); // Wait for inputs to stabilize. Stops false jumperMode=1 on cold start

  // Set mode based on jumper position
  if (digitalRead(2)==LOW) {
    jumperMode=1; // unused in this example
  } else if (digitalRead(3)==LOW) { // Ethernet Connection
    jumperMode=2;
  } else if (digitalRead(4)==LOW) { // Serial Connection
    jumperMode=3;
  }

  if (!jumperMode) { // No jumper installed
    while(1); // Halt program
  }

  if (jumperMode==2) { // Ethernet mode
    qn::Ethernet.begin(mac, ip, subnet); // Create object
    EthServer.begin(); // Start listening for client connections
    modbusTCPServer.begin(); // Start modbus service
    modbusTCPServer.configureCoils(0, 1);
  }

  if (jumperMode==3) { // Serial mode
    ModbusRTUServer.begin(1, 19200); // Start modbus service
    ModbusRTUServer.configureCoils(0, 1); // Coils for turning all shutters off, and individual control of each channel
  }
}

void loop() {
  if (jumperMode==2) { // Ethernet loop
    qn::EthernetClient client = EthServer.available();
    if (client) {
      modbusTCPServer.accept(client);
      if (client.connected()) { // Examples shows while() here. This prevents the loop from recovering when the cable is disconnected. if() resolves the issue.
        if (modbusTCPServer.poll()) { // poll() returns true when a request is present
          updateOutput();
        }
      }
    }
  }

  if (jumperMode==3) { // Serial loop
    if (ModbusRTUServer.poll()){
      updateOutput();
    }
  }
}

void updateOutput() {
  boolean state=(jumperMode==2)?modbusTCPServer.coilRead(0):ModbusRTUServer.coilRead(0);
  digitalWrite(13, state);
}

In the attached images you can see how my Modbus testing software (QModMaster) is sending a request to read the value of the coil and the time it takes to get a response. The results are a bit faster in this simplified code, but still pretty slow on the ethernet front.

EDIT: made code blocks cpp for clarity. Also, I can ping the Teensy and always get <1ms response

Thanks for taking the time to look at this!

 

[shawn]

A couple of things I noticed at first glance:
1. There's no need to specify a MAC address on the Teensy 4.1 because there's a built-in one. May I suggest using the begin(ip, netmask, gateway) function instead? As well, the one you're actually using in the code you pasted is begin(mac, ip, dns). See: https://www.arduino.cc/reference/en/libraries/ethernet/ethernet.begin/ (I don't love the Arduino Ethernet API for these things.) (This is the one you want if you really must specify your own MAC address: Ethernet.begin(mac, ip, dns, gateway, subnet))
2. I would use if (client.available() > 0) instead of if (client.connected()). I think that's the real intent. (You could even experiment with while (client.available() > 0) because that will be false when there's no data.) The Boolean check just checks if connected. The connected() function checks if connected OR if there's still data available, or both, so it will always be true as long as the connection exists OR there's data, or both. The available() function checks how much data is available, even after disconnect (similar to connected()).

This link may also help: https://github.com/ssilverman/QNEth...ce/README.md#how-to-write-data-to-connections
Basically, QNEthernet's client send functions don't send data until either a short timer expires, until there's enough data for a packet, or until flush() is called on the connection. It looks like the place where that Modbus library is calling write() is here: https://github.com/arduino-librarie...3a4f1331dc8/src/libmodbus/modbus-tcp.cpp#L202

Try changing to something like this:

size_t retval = ctx_tcp->client->write(req, req_length);
ctx_tcp->client->flush();
return retval;

(Note: I haven't tested this.)

Let me know if these tips help.

 

[shawn]

I’ll add: it’s not correct to always assume some write() call (on, say, the Arduino Print interface, which many types of objects implement) completes without buffering. Unfortunately, many programs and libraries assume this.
See:

• https://www.arduino.cc/reference/en/language/functions/communication/print/
• https://www.arduino.cc/reference/en/language/functions/communication/wire/write/

Maybe I need to add some QNEthernet option that performs a flush() on every write() call? I’m hesitant to do this just yet, though, because it disagrees with how write() is supposed to work. But it may help to not have to modify some libraries that make this unfortunate assumption. It might also affect TCP throughput in some cases, though.

 

[shawn]

I just realized that all the bytes still may not get out. Any write() calls (with any library, really) should have their return values checked and then any unsent bytes should be re-sent. write() doesn't always send all the bytes it's given. A better QNEthernet equivalent would be:

size_t retval = ctx_tcp->client->writeFully(req, req_length);
ctx_tcp->client->flush();
return retval;

This uses writeFully() instead of write(). (You could accomplish the same thing yourself by looping until complete. See also: https://github.com/ssilverman/QNEth...ce/README.md#how-to-write-data-to-connections)

In summary, just using a write() call isn't correct usage, in most cases.

 

[milesg]

Wow Shawn thanks! This is a lot of info. I get a little bewildered with the lower level stuff, I tried my best and here are the results...

1. You're right, I didn't need to specify Mac address, so I updated the variables and call to begin() such that it uses the fields you suggested. I'll have to change some things in my full program to use the IPAddress datatype, but no big deal. This didn't help the speed, but it does make the IDE complain less about deprecated overloads. I was just copying the examples from the modbus library, so I'm happy to bring things up to date. Thanks!

2. I changed to if (client.available() > 0) in my loop. Again, this didn't help with the speed, but I do like to know I'm doing things in a more preferable way.

3. Updating the call to the write() function in modbus-tcp.cpp#L202 made a HUGE difference. I was getting an average of 150ms reply time in this test-code, but only about 2ms average after the change. Unfortunately I'm getting a [ Class "Client" has no member "writeFully" ] error when trying to implement the further correction you suggested... I think I get the intent here, but it seems that the call isn't looking in the right place. "Go to Definition" takes me to Ln 256 of QNEthernetFrame.cpp, but I don't see a writeFully() in that file.)
I'm a little nervous about trying this fix with just write() because of the potential problems you mentioned. But it does work with this single coil example, so I'll try it on my full program just for kicks.

Here's the current code, and the change to modbus-tcp.cpp is exactly as you suggested above with write():

#include <QNEthernet.h> // Most optimized Ethernet library for Teensy 4.1 Seems to have resolved memory leak in other libraries
#include <ArduinoModbus.h> // Dependency ArduinoRS485\RS485.cpp must have line 209 edited to use Serial5 and associated pins.
#include <EEPROM.h> // Enable access to EEPROM on Teensy board

using namespace qindesign::network;
EthernetServer EthServer(502);
ModbusTCPServer modbusTCPServer;

void updateOutput(); // Updates all output pins to high/low based on Modbus coil values

// Global variables
IPAddress ip{ 192, 168, 1, 50 }; // Default IP address
IPAddress subnet{ 255, 255, 255, 0 }; // Default Subnet mask
IPAddress gw{ 192, 168, 1, 1}; // Gateway
uint8_t jumperMode = 0; // Set during setup() to select boot mode. 1=Load defaults, 2=Ethernet, 3=Serial

void setup() {
  for (uint8_t i=2; i<5; i++) { // Set up input pins
    pinMode(i, INPUT_PULLUP); // Set all 3 jumper positions as inputs
  }
  pinMode(13, OUTPUT);

  delay(1); // Wait for inputs to stabilize. Stops false jumperMode=1 on cold start

  // Set mode based on jumper position
  if (digitalRead(2)==LOW) {
    jumperMode=1; // unused in this example
  } else if (digitalRead(3)==LOW) { // Ethernet Connection
    jumperMode=2;
  } else if (digitalRead(4)==LOW) { // Serial Connection
    jumperMode=3;
  }

  if (!jumperMode) { // No jumper installed
    while(1); // Halt program
  }

  if (jumperMode==2) { // Ethernet mode
    //qn::Ethernet.begin(mac, ip, subnet); // Create object
    Ethernet.begin(ip, subnet, gw);
    EthServer.begin(); // Start listening for client connections
    modbusTCPServer.begin(); // Start modbus service
    modbusTCPServer.configureCoils(0, 1);
  }

  if (jumperMode==3) { // Serial mode
    ModbusRTUServer.begin(1, 19200); // Start modbus service
    ModbusRTUServer.configureCoils(0, 1); // Coils for turning all shutters off, and individual control of each channel
  }
}

void loop() {
  if (jumperMode==2) { // Ethernet loop
    EthernetClient client = EthServer.available();
    if (client) {
      modbusTCPServer.accept(client);
      if (client.available()>0) { // Updated to Shawn's suggestion
        if (modbusTCPServer.poll()) { // poll() returns true when a request is present
          updateOutput();
        }
      }
    }
  }

  if (jumperMode==3) { // Serial loop
    if (ModbusRTUServer.poll()){
      updateOutput();
    }
  }
}

void updateOutput() {
  boolean state=(jumperMode==2)?modbusTCPServer.coilRead(0):ModbusRTUServer.coilRead(0);
  digitalWrite(13, state);
}

 

[shawn]

You’re right about writeFully(). I didn’t look at that Modbus library closely. They’re using “Client” and not “EthernetClient”. Ignore the writeFully() note for now. (But write() calls should still technically be called repeatedly until they’re done with the input. See that README link I posted for how. If you like.)

And yes, the first two notes won’t affect the speed.

 

[milesg]

... See that README link I posted for how. If you like...

Alright, new day! Yep I was looking at that, but couldn't quite figure out how to fit it into the code last night. Again, I'm not great with this low level stuff. But I tinkered with it this morning and I think I was able to get the same functionality cobbled into modbus-tcp.cpp#L202. Here's what I changed it to:

    // return ctx_tcp->client->write(req, req_length); // Original line 202
    size_t retval=0;
    while (req_length > 0 && ctx_tcp->client) {
        retval = ctx_tcp->client->write(req, req_length);
        req_length -= retval;
        req += retval;
    }
    ctx_tcp->client->flush();
    return retval;

This seems to work just as well as your original suggestion (no errors, and <3ms return time with my full program running 40+ registers and 20+ coils.)
I'm assuming the above code is doing the job of writeFully(), but I'm a little unsure of how I declared retval=0 in case the while() never happens. Does this look like the return will always be correct, or should I declare/order things differently?

I've actually played with several modbus libraries and they all seem to have some shortcomings, but this one seems to be simple to use and fairly up-to-date. I looked at Paul's fork, but it seems to be behind now even though it's probably more optimized for teensy. I wish there was a fork of ArduinoModbus that uses QNEthernet directly, but if this little fix doesn't break anything else, I'm happy to keep it in place.

Thanks again for your help!

 

[shawn]

Yeah, I expect it to be about the same speed. It’s just the proper way because write() may not send all the bytes requested of it. Since you’re using something like Modbus, I’m guessing that reliability is important to you.

Yes, it looks similar to what writeFully() does under the covers. Check out the library source code to confirm.

As for the library using QNEthernet, because I made EthernetClient conform to the Client interface included with Teensyduino, it shouldn’t matter too much. It’s just that that Modbus library could stand to be a little more robust when using Client::write() (actually Print::write()). It’s in the Arduino API docs that write() returns the number of bytes actually written; that’s why it needs to be checked.

 

[milesg]

Thanks Shawn,
That was my concern... Since write() may be called more than once in the loop but only returns once I couldn't quite gather if the way I have it is returning the correct value. It does seem to work, but yes indeed reliability is important, so I didn't want to jump to conclusions. I did do my best to duplicate the source of writeFully() in this new location.

I'll keep playing with it, but I think I'm up and running. Ethernet is running faster than Serial(9600) again. All is right in the world

 

[shawn]

I just added an option to flush after all write calls. Other than the issue of write() not actually sending all the bytes (eg. because of internal buffers and such), just setting this new option in qnethernet_opts.h should duct tape the original issue (again, assuming the write() packets are small enough). Note that it might not matter for, say, the Modbus use case, it might affect TCP performance for other use cases. The new option: QNETHERNET_FLUSH_AFTER_WRITE

 

[PedroMatos]

Hi all,

First time commenting/posting in this forum.
I am trying to build an ModbusEthernet communication between teensy 4.1 as a Server and a host computer as a Client. The shield I got for the teensy is the following one:

PJRC Store

www.pjrc.com

I am trying to use QNEthernet Library and the code that I am running on the teensy is based on yours code since any of the examples were usefull for me.

4.1 Teensy code:

#include <QNEthernet.h> // Most optimized Ethernet library for Teensy 4.1 Seems to have resolved memory leak in other libraries
#include <ArduinoModbus.h> // Dependency ArduinoRS485\RS485.cpp must have line 209 edited to use Serial5 and associated pins.
#include <EEPROM.h> // Enable access to EEPROM on Teensy board

using namespace qindesign::network;
EthernetServer EthServer(502);
ModbusTCPServer modbusTCPServer;

void updateOutput(); // Updates all output pins to high/low based on Modbus coil values

// Global variables
IPAddress ip{ 192, 168, 1, 50 }; // Default IP address
IPAddress subnet{ 255, 255, 255, 0 }; // Default Subnet mask
IPAddress gw{ 192, 168, 1, 1}; // Gateway


void setup() {

  pinMode(13, OUTPUT);

  delay(1); // Wait for inputs to stabilize.

  // Open serial communications and wait for port to open:
  Serial.begin(9600);
   while (!Serial) {
    ; // wait for serial port to connect. Needed for native USB port only
  }
 
  //qn::Ethernet.begin(mac, ip, subnet); // Create object
  Ethernet.begin(ip, subnet, gw);
  EthServer.begin(); // Start listening for client connections
  Serial.print("Modbus server address:");
  Serial.println(Ethernet.localIP());
  // start the Modbus TCP server
  if (!modbusTCPServer.begin()) {
    Serial.println("Failed to start Modbus TCP Server!");
    while (1);
  }
  modbusTCPServer.configureCoils(0, 1);
}

void loop() {
  EthernetClient client = EthServer.available();
  if (client) {
    Serial.print("Client Connected\n");
    modbusTCPServer.accept(client);
    if (client.available()>0) { // Updated to Shawn's suggestion
      if (modbusTCPServer.poll()) { // poll() returns true when a request is present
        updateOutput();
      }
    }
  }
}

void updateOutput() {
  boolean state=modbusTCPServer.coilRead(0);
  digitalWrite(13, state);
}

And on the host computer I am running a simple modbus Client on python:

#!/usr/bin/env python3


import time
from pyModbusTCP.client import ModbusClient

# init
c = ModbusClient(host='192.168.1.50', port=502, auto_open=True, debug=True)
bit = True

# main loop
while True:
    # read 10 bits (= coils) at address 0, store result in coils list
    coils_l = c.read_coils(0, 10)

    # if success display registers
    if coils_l:
        print('coil ad #0 to 9: %s' % coils_l)
    else:
        print('unable to read coils')

    # sleep 2s before next polling
    time.sleep(2)

Anyway, I am very beginner in this matter, feel free to correct me and point me in the right direction.
Thanks in advance,

Pedro

 

[milesg]

Hi Pedro,
I'm no expert either, but my project did end up working out nicely. A big help in troubleshooting was to have a known-good modbus program on my PC to test my Teensy code. I used QModMaster (https://sourceforge.net/projects/qmodmaster/) but there are probably a dozen such programs that you could use.

By using a software like this, you take out any guess work as to which side you might need to troubleshoot. Once you've successfully tested your Teensy code, you can develop your Python code with more confidence.

Also, I changed my RS485.cpp to use Serial5 for Modbus communication. This keeps the Serial.print() commands from interfering. Not sure if you've done the same, or if that's even an issue, but I always try to keep my diagnostics communications separated.

You can also remove the delay(1); since you're not setting up any inputs in your code.

 

[PedroMatos]

Hi Milesg,

Thank you for the fast reply. I really appreciate your help.

I did read that you changed it to Serial5 but I cannot figure out how it can be done. Can you provide help over here ?

Meanwhile I'll download the program and try to debug better my code.

Thank you,
Pedro

 

[milesg]

There are a couple ways, but the easiest is to just hard-code it:

line 209 of RS485.CPP was changed to the following

RS485(Serial5, 20, 255, 255);

In my case, I was actually using a MAX3232 (RS-232 doesn't transmit/receive on the same lines) so I didn't need to define my RX & TX Enable pins (just set to 255 to effectively disable that function.) And I used the default TX pin of 20 for Serial5. You can use Serial, Serial1, etc... Just make sure you choose the appropriate pin numbers for your circuit.