New lwIP-based Ethernet library for Teensy 4.1

Yeah, I decided to do two things: rely on the now-supported printf, and split stdPrint into stdoutPrint and stderrPrint, along with a new QNETHERNET_ENABLE_CUSTOM_WRITE macro.
 
I just released 0.20.0. The changes:

Code:
* Added a `BroadcastChat` example that implements a simple chat over UDP.
...
Highlights:
* Two new examples: AltcpTemplate and BroadcastChat
...
Link: https://github.com/ssilverman/QNEthernet/releases/tag/v0.20.0

Tested the BroadcastChat example amongst three T_4.1's on home network and it works effortlessly!

They start up see who they are and any 'chat' from one appears on the others SerMon output! :cool:

Left them on overnight and they are still awake and responsive.

For fun two of them are actually devices on T_4.1 USB Host's (one as Serial the other Raw Hid) - so desk it a mess with 5 T_4.1's, two usb Host cables and three Ethernet adapters and cables and three micro usb cables to this computer.
 
I just released v0.21.0. The changes:

Code:
### Added
* Added `EthernetClass::linkIsCrossover()` for checking if a crossover cable
  is detected.
* Added entropy-based random number functions, `entropy_random()` and
  `entropy_random_range(range)`. The second uses an unbiased algorithm.

### Changed
* Renamed TRNG tests to test_entropy.
* Added calling file, line, and function information to
  `LWIP_ASSERT_CORE_LOCKED()`.
* Un-deprecated `EthernetClass::MACAddress(mac)` and `setDnsServerIP(ip)`.
* Optimized byte-swapping by using GCC built-ins.

### Fixed
* Reset the PHY in a much more conservative way. Hopefully this helps with
  restarting Ethernet causing packets to not be received.
* Fixed dhcp struct assignment to be done each time the netif is added. This
  addresses `netif_add()` clearing all the client data.
* Fixed `LWIP_PLATFORM_ASSERT()` to flush stdout's underlying `Print` object.
  This ensures all output for an assertion failure gets sent out before the call
  to `abort()`.
* Fixed the link status values changing after setting the link up.

Highlights:
* A way to check for a crossover connection.
* Bug fixes.
* Other small improvements.

Link: https://github.com/ssilverman/QNEthernet/releases/tag/v0.21.0
 
@shawn - Is my understanding correct that your QNEthernet library does not support TLS 'out of the box', but provides the hooks to do so, if someone wished to try? If so, do you know of anyone that has successfully done this with a third party TLS library?

I'm currently using NativeEthernet, which includes mbedtls and provides this support out of the box (after minor fixes to the code), and I use this to connect to various services that require TLS (MapBox, National Weather Service, Philips Hue API, SensorPush API). I recently had interest in using a DTLS-based service, but NativeEthernet seems to be effectively abandoned by the original author and the included mbedtls version is significantly behind the latest release, so it feels like a very painful an effort to get that to where I want it to be, so I was looking for alternatives that aren't as painful :) Any thoughts or guidance would be appreciated!
 
@beermat - Thank you for the question. I'll begin by summarizing my mental model of the library:
1. It's an Arduino API-style layer on top of lwIP, with the assumption that all network functions are performed from the same context (same thread or similar).
2. lwIP is configured as single-threaded, with certain features enabled and configured a certain way.

There's a few ways a TLS layer could be accomplished:
1. Use the Arduino-style API as a base layer with some TLS library's functions built on top of that. i.e. you'll write a layer to call the Arduino-style API.
2. Use the lwIP layer as the base layer. Similar to #1, you'd use the appropriate lwIP functions to provide a data source and sink to the TLS library.
3. lwIP provides a layer that pretends to be a regular TCP layer, but behaves as a "decorator" to that API so that calls can be intercepted. It's called "Application Layered TCP", or "altcp". Intercepted calls can be rerouted to do proxy things or connection-related things such as TLS. QNEthernet provides a way to interface with the altcp layer, but you need to write your own implementation of whatever you're trying to intercept (eg. a proxy or TLS layer).

The advantage to #3 is that your program doesn't have to change; everything is handled under the covers by the exact same TCP calls. They're just intercepted to go to a TLS driver instead of directly to the TCP machinery of lwIP.

There already exists an altcp-based driver for mbedtls in the lwIP distribution (not included with the QNEthernet library; it's easy to add yourself), but it's based on an older version of that library. I looked at it a little bit, but didn't want to invest the time creating a new or modified driver. There's also wolfSSL, and I started to write a driver for that, just to get a feel for it, but haven't finished. It's unlikely I'll finish it in the near- or even medium-term future, or if I do, I'll likely put it under the GPL because that's what wolfSSL uses for its non-commercial licence.

To complete a new, or modify an existing, mbedtls or wolfSSL driver, it might take a few weeks of work.

For anything faster than "I might or might not finish something this year," contact me privately. Alternatively, you could try to modify the existing mbedtls driver yourself.
 
I just released v0.22.0. The changes:

Code:
### Added
* `EthernetClass::setDHCPEnabled(flag)` enables or disables the DHCP client. May
  be called either before or after Ethernet has started.
* `EthernetClass::isDHCPEnabled()` returns whether the DHCP client is enabled.
  Valid whether Ethernet has been started or not.
* New `LinkWatcher` example.
* Added support for building for unsupported boards via a new bare lwIP driver.

### Changed
* Limit the number of times `enet_proc_input()` can loop to twice the ring size.
* Limit UDP output size to the maximum possible (65535 - 28(total header)).
* It's now possible to know when adding or removing a MAC address filter failed.
* Make it possible to disable and exclude DHCP, DNS, IGMP, TCP, and UDP.
* Changed `EthernetClass::setMACAddress(mac)` to use the built-in MAC address if
  the given array is NULL.
* Changed `EthernetClass::begin(mac)` to wait for an IP address. The default is
  a 60-second timeout. There's also a new, optional, `timeout` parameter for
  specifying that timeout. This change makes the API match the Arduino
  Ethernet API.
* Renamed `enet_getmac(mac)` to `enet_get_mac(mac)`.
* Better NULL argument checking.
* Simplified `ServerWithListeners` example.
* Changed `enet_init(...)` to return a `bool` for detecting init. failure.

### Fixed
* Fixed how `EthernetClient` functions work when there's a pending connect
  triggered by `connectNoWait()`.
* Fixed how raw frame size limits are checked. Padding is already handled by
  the MAC.
* Fixed compilation if `LWIP_IGMP` is disabled, including making
  `LWIP_MDNS_RESPONDER` dependent on `LWIP_IGMP` (in addition to `LWIP_UDP`).
* Improved `trng_is_started()` by adding an additional check for the "OK to
  stop" bit. It now works at system startup if the clock is already running.

Highlights:
* Some bug fixes
* Better `Ethernet.begin(mac)` Arduino API compatibility
* The ability to selectively remove certain features from the build

Link: https://github.com/ssilverman/QNEthernet/releases/tag/v0.22.0
 
The lwIP project just released v2.2.0-rc1, so I made a QNEthernet branch that uses it. I could use some help testing it in your projects, if anyone is willing. I'll be testing with it too, but I think the release could use a wider pool of people that already use QNEthernet.

Source: https://github.com/ssilverman/QNEthernet/tree/lwip-2.2.0-rc1
PlatformIO `lib_deps` entry: https://github.com/ssilverman/QNEthernet.git#lwip-2.2.0-rc1

In short, they based this new release off of their `master` branch. Before, that branch and their "stable release" branch diverged quite a bit.
 
The lwIP project just released v2.2.0-rc1, so I made a QNEthernet branch that uses it. ...

pulled the ZIP as github desktop was giving weird help on branches ...: Using library QNEthernet at version 0.23.0-snapshot in folder: T:\T_Drive\tCode\libraries\QNEthernet

As far as the BroadcastChatUDP code in progress - it built and runs fine across 3 T_4.1's.
 
I just released v0.23.0. The changes:

Code:
## [0.23.0]

### Added
* Added `qindesign::security::RandomDevice`, a class that conforms to the
  _UniformRandomBitGenerator_ C++ named requirement. This makes it a little
  easier to generate platform-independent entropy.
* `EthernetUDP::receivedTimestamp()` for retrieving a packet's approximate
  arrival time.
* Added a "`writeFully()` with more break conditions" subsection to the README.
* `EthernetFrame::receivedTimestamp()` for retrieving a frame's approximate
  arrival time.
* `EthernetClient::connectionTimeout()` for getting the current timeout value.

### Changed
* Changed `EthernetUDP::localPort()` to be `const`.
* Changed `entropy_random_range()` to return zero if `EAGAIN`.
* Changed `entropy_random_range()` to use Daniel Lemire's nearly-
  divisionless algorithm.
* Updated lwIP to v2.2.0-rc1.
* Made `StdioPrint` single-argument constructor `explicit`.
* Updated `EthernetClass::linkStatus()` to return `EthernetLinkStatus::Unknown`
  if the hardware hasn't yet been probed.
* Updated `AltcpTemplate` example to print proxy information.
* Updated `EthernetUDP::availableForWrite()` to return the amount remaining
  before hitting the maximum possible payload size.
* Add 5 to `MEMP_NUM_SYS_TIMEOUT` option for mDNS instead of 1.

### Removed
* Removed TTL concept from `MDNSClass`. This enables it to compile with the
  latest lwIP.

### Fixed
* Fixed `SNTP_SET_SYSTEM_TIME_US(sec, us)` definition to set the RTC directly
  because `settimeofday()` doesn't exist here.
* Fixed `AltcpTemplate` example so that it compiles when `LWIP_ALTCP` isn't set.

Highlights:
1. Update to lwIP 2.2.0-rc1

Link: https://github.com/ssilverman/QNEthernet/releases/tag/v0.23.0
 
I just released v0.24.0. The changes:

Code:
## [0.24.0]

### Added
* The new `QNETHERNET_MEMORY_IN_RAM1` configuration macro indicates that
  lwIP-declared memory should go into RAM1.
* New `EthernetFrameClass::receiveQueueSize()` function.
* New `EthernetUDP::receiveQueueSize()` and `setReceiveQueueSize(size)`
  functions.
* Enabled external definition of macros `LWIP_NETIF_LOOPBACK` and
  `LWIP_LOOPBACK_MAX_PBUFS`.
* Sprinkled some more `Ethernet.loop()` calls where pcb and pbuf
  allocations fail.
* Added `EthernetClass::setLinkState(flag)` for manually setting the link state
  when a link is needed, such as when using the loopback feature. Network
  operations will usually fail unless there's a link.
* Added more unit tests:
  * test_ethernet:
    * test_setLinkState
    * test_udp_receive_queueing
    * test_udp_receive_timestamp
    * test_udp_state
    * test_client_connectNoWait
    * test_client_timeout
    * test_client_state
    * test_server_state
    * test_other_state
  * test_entropy:
    * test_randomDevice
* Added commented-out `LWIP_STATS_LARGE` option to _lwipopts.h_.

### Changed
* Changed memory declaration macro, `LWIP_DECLARE_MEMORY_ALIGNED()`, to use the
  `MEM_ALIGNMENT` value.
* Now calling `shrink_to_fit()` on the UDP and Ethernet frame queues when
  changing their size.
* Add 6 to `MEMP_NUM_SYS_TIMEOUT` option for mDNS instead of 5. Timeout
  exhaustion was still observed with 5.
* There's now a single _lwip_driver.h_ header for interfacing with the stack.
* Changed all `EthernetClass::begin(mac, ...)` functions to be consistent. If
  the MAC address is NULL then the MAC will be set to the internal one. Also,
  if starting Ethernet fails, the MAC address will not be changed.
* Improved Ethernet tests to do proper object destruction when tests fail. The
  Unity test framework makes use of _longjmp_ for failing tests, and that
  doesn't work well with object destructors.
* Unit test updates.
* Made single-argument `EthernetClass` and `EthernetClient` constructors
  explicit.

### Fixed
* Now using the correct name when adding an mDNS service.
* Pre-reserving memory for raw frames and UDP packets prematurely exhausts the
  heap when a larger number of them are reserved in the queue. These buffers are
  no longer reserved; they only grow appropriately when data comes in.
* Fixed closing `EthernetClient`s to remove the connection state if not already
  connected. Restarting an `EthernetClient` via one of the `connectXXX()`
  functions calls `close()` first. If there was no connection, then closing
  never removed the internal connection object, causing a leak.
* Fixed `Ethernet.loop()` to also poll the netif if loopback is enabled. This
  allows loopback to work.
* `EthernetServer::end()` now sets the port to -1.

Highlights:
1. Fixed some potential memory issues
2. Loopback support was fixed
3. More and improved unit tests

Link: https://github.com/ssilverman/QNEthernet/releases/tag/v0.24.0
 
As I'm sitting here trying to debug why my RaspberryPIW running the Apache web server is seizing up every few days to a few weeks, it dawned on me that my T4.1 based server running your QNEthernet hasn't crashed in a few years !! I have been installing your latest versions so it hasn't been continuously running, but that is still pretty impressive.

Thanks for making such a stable and manageable program!
 
@bicycleguy thank you for the compliment; it means a lot. I use it extensively myself and most of the features are dictated by my own needs.

I'm curious, do you have a wish list or suggestions for the current library or future features? I keep telling myself, "This is just about ready to be a 1.0.0 release," but in the great tradition of ICQ, I seem to be asymptotically approaching it instead. :)
 
I just released v0.25.0. The changes:

Code:
## [0.25.0]

### Added
* New "Heap memory use" section in the README that discusses memory
  allocation functions.
* New _RandomNumbers_ example.
* Added a README subsection that talks about the `RandomDevice` class.
* Added `RandomDevice::instance()`.
* Added `EthernetClient::status()` for getting the TCP connection state.

### Changed
* Enabled the `MEM_LIBC_MALLOC` option by default to make use of the system-
  defined malloc functions instead of the lwIP-defined ones.
* Moved around where the netif gets its address set.
* Changed the license to "AGPL-3.0-or-later".
* Made `RandomDevice` constructor and destructor private.
* Improved unit tests.
* Updated lwIP to v2.2.0.

### Removed
* Removed the `extern qindesign::security::RandomDevice randomDevice` instance
  from _QNEthernet.h_ because the way to access the device now is via its
  `instance()` function (the constructor is also private now). (It had been
  added in v0.23.0.)

### Fixed
* Fixed `enet_init()` to always initialize the internal MAC address on
  first init.
* Fixed execution error when running _main.cpp_ (`MAIN_TEST_PROGRAM` macro
  defined) by removing `build_type = debug` from _platformio.ini_.
* Fixed static initialization order for `Ethernet`, `EthernetFrame`, and `MDNS`
  singletons by using the Nifty Counter idiom.

Highlights:
1. Now using the system-defined malloc functions instead of the lwIP-defined ones
2. New `EthernetClient::status()` function for getting the TCP connection state
3. Updated lwIP to v2.2.0

Link: https://github.com/ssilverman/QNEthernet/releases/tag/v0.25.0
 
I just pushed a bunch of changes that add Mbed TLS support. I'm still working on the design, but I'd love some opinions about what I have so far. I updated the README and added an MbedTLSDemo example.

The design relies on lwIP's "altcp" feature to route all TCP calls through an intermediate layer. I've tried to make this at least semi-easy to work with.

Anyone up for helping and trying it out? I'd love some feedback before I roll another release.
 
Anyone up for helping and trying it out? I'd love some feedback before I roll another release.
I'm using IDE2.2.1 on macOS with the latest QNEthernet it installs. I'm not great at GitHub Desktop but do use it (probably incorrectly).
Should I clone the master into ~/Documents/Arduino/libraries replacing the existing QNEthernet folder ?
 
I'm using IDE2.2.1 on macOS with the latest QNEthernet it installs. I'm not great at GitHub Desktop but do use it (probably incorrectly).
Should I clone the master into ~/Documents/Arduino/libraries replacing the existing QNEthernet folder ?
Yes, you could do that.
 
The new 0.59.4 core seems to be messing with QNEthernetv0.25 and v0.26:
C++:
~/Documents/Arduino/libraries/QNEthernet/src/QNEthernet.cpp: In member function 'bool qindesign::network::EthernetClass::begin(const IPAddress&, const IPAddress&, const IPAddress&, const IPAddress&)':
~/Documents/Arduino/libraries/QNEthernet/src/QNEthernet.cpp:186:11: error: ambiguous overload for 'operator!=' (operand types are 'const IPAddress' and 'const IPAddress')
  186 |   if (dns != INADDR_NONE) {
      |       ~~~ ^~ ~~~~~~~~~~~
      |       |      |
      |       |      const IPAddress
      |       const IPAddress
~/Documents/Arduino/libraries/QNEthernet/src/QNEthernet.cpp:186:11: note: candidate: 'operator!=(uint32_t {aka long unsigned int}, uint32_t {aka long unsigned int})' (built-in)
  186 |   if (dns != INADDR_NONE) {
      |       ~~~~^~~~~~~~~~~~~~
In file included from ~/Documents/Arduino/libraries/QNEthernet/src/QNEthernet.h:15,
                 from ~/Documents/Arduino/libraries/QNEthernet/src/QNEthernet.cpp:7:
~/Library/Arduino15/packages/teensy/hardware/avr/0.59.4/cores/teensy4/IPAddress.h:82:14: note: candidate: 'bool IPAddress::operator!=(const IPAddress&) const'
   82 |         bool operator!=(const IPAddress& addr) const {
      |              ^~~~~~~~
In file included from ~/Documents/Arduino/libraries/QNEthernet/src/QNEthernet.h:33,
                 from ~/Documents/Arduino/libraries/QNEthernet/src/QNEthernet.cpp:7:
~/Documents/Arduino/libraries/QNEthernet/src/util/ip_tools.h:27:6: note: candidate: 'bool qindesign::network::operator!=(const IPAddress&, const IPAddress&)'
   27 | bool operator!=(const IPAddress &a, const IPAddress &b);
      |      ^~~~~~~~
 
Yep, that's fixed in the latest push. It's currently at v0.26.0-snapshot and there's a good chance I'll make a new release soon.

I had added my own const IPAddress operators for comparison operations because previous Teensyduinos didn't have the const version. The latest code only adds those if the Teensyduino version is <= 1.58.
 
I'm preparing a v0.26.0 release shortly. Would anyone here be willing to get the latest code and try the new Mbed TLS feature out? I'd love it if someone other than me went through the instructions to verify that it works.
 
I just released v0.26.0. The Changelog:
Code:
## [0.26.0]

### Added
* Added `EthernetClient::localIP()`.
* Added `EthernetClass::hostByName(hostname, ip)` convenience function.
* Added `EthernetClass::setDNSServerIP(index, ip)` and `dnsServerIP(index)`.
* Added some support for Mbed TLS v2.x.x. There's four new adapter functions
  for assisting integration (see _src/altcp_tls_adapter.cpp_), included if the
  `QNETHERNET_ALTCP_TLS_ADAPTER` option is set:
  1. `qnethernet_altcp_is_tls`
  2. `qnethernet_altcp_tls_client_cert`
  3. `qnethernet_altcp_tls_server_cert_count`
  4. `qnethernet_altcp_tls_server_cert`
* Added _MbedTLSDemo_ example.
* Added printing the message size in _LengthWidthServer_ example.
* Added `QNETHERNET_ENABLE_ALTCP_DEFAULT_FUNCTIONS`-gated default
  implementations of the altcp interface functions.
* Added `EthernetClass::macAddress()` for returning a pointer to the current
  MAC address.

### Changed
* Updated lwIP to the latest master (5e3268cf).
* Made the driver non-blocking:
  1. TX: if there's no available buffer descriptors (returns ERR_WOULDBLOCK)
  2. Link checks via MDIO
* Updated the tests:
  * Added a 10s connection timeout to `test_client_addr_info()`
  * Added SNTP retries to `test_udp()`
  * Updated and added some messages
* Made `MDNSClass::Service::operator==()` `const`.
* Completely revamped PHY and pin initialization.
* Gated PHY and Ethernet shutdown in `EthernetClass::end()` with a macro; the
  default behaviour is to not execute these blocks. This and the previous are
  part of the quest to figure out why performance drops off a cliff when
  Ethernet is restarted via first calling `end()`.
* Changed return type of `qnethernet_get_allocator` to `bool`.
* Renamed `qnethernet_get_allocator` and `qnethernet_free_allocator` to
  `qnetheret_altcp_get_allocator` and `qnethernet_altcp_free_allocator`,
  respectively.
* Changed `qnethernet_get_allocator` and `qnethernet_free_allocator` `allocator`
  parameter to be a reference.
* Renamed `QNETHERNET_MEMORY_IN_RAM1` to `QNETHERNET_LWIP_MEMORY_IN_RAM1`.
* Updated the _AltcpTemplate_ example.
* Fixed `IPAddress`-related build problems with the new Teensyduino 1.54-beta4.
* Updated the _RawFrameMonitor_ example with information about how to
  disable DHCP.
* Disabled waiting in `EthernetClient::close()` for altcp clients because it's
  not defined.
* Changed configuration macro checks to use value instead of definedness.

### Fixed
* Fixed a `printf` conversion specifier in the _RandomNumbers_ example. This was
  causing a compile warning.
* Fixed location of STATIC_INIT_DECL() for `RandomDevice` by putting it into
  the header. It needs to be in a place where users of the class see it.
* Fixed `EthernetClass::setMACAddress()` for when the interface is up.

Highlights:
  • Added some more convenience functions.
  • Added support for Mbed TLS v2.x.x.
  • The driver is now completely non-blocking.
  • Completely revamped PHY and pin initialization.
Link: https://github.com/ssilverman/QNEthernet/releases/tag/v0.26.0
 
Here's some updates:
  1. I made a driver for the W5500 that plugs into QNEthernet. You can enable it in the new qnethernet_opts.h file (after overwriting using the ZIP file about to be mentioned). I haven't put it in the latest `master`, but you can find it here: https://github.com/ssilverman/QNEthernet/discussions/62. Simply update to the latest QNEthernet from git and unzip the latest w5500_driver.zip file (from that discussion) inside the library directory (i.e. wherever you put QNEthernet). It will create some new files and overwrite a few others.
  2. There's a revamped driver API so it's a little easier to write new drivers.
  3. There's a new qnethernet_opts.h file where you can specify QNEthernet options without having to specify them project-wide. This should make changes easier if you're using the Arduino IDE.
  4. I've also made an effort to make the library compilable for other systems.
 
I just released v0.27.0. Here's the Changelog:
Code:
## [0.27.0]

### Added
* New `QNETHERNET_FLUSH_AFTER_WRITE` configuration macro for flushing after
  every call to `EthernetClient::write()`. This may reduce TCP efficiency.
* Added a W5500 driver.
* Added a new `EthernetHardwareStatus::EthernetTeensy41` enum value.
* New `QNETHERNET_DEFAULT_HOSTNAME` option.

### Changed
* Made it easier to add other low-level drivers.
* Redesigned the driver interface.
* Un-deprecated `EthernetClass::begin(mac, timeout)` and
  `begin(mac, ip, dns, gateway, subnet)`.
* Un-deprecated `EthernetClass::init(sspin)` and added a driver function for
  setting the chip select pin. The type of `sspin` was also changed to `int`.
* Removed dependency on `elapsedMillis`. This might help with compiling for
  other platforms.
* Made the library easier to compile for other platforms.
* Replaced all `#define` guards with `#pragma once`.
* All `QNETHERNET_*` configuration macros can now be defined in a new
  `qnethernet_opts.h` file in addition to the project build.
* Removed test dependencies on Teensy-specific things.
* Renamed _t41_ driver source files to use _teensy41_ in the name instead.
* Changed return type of `EthernetUDP::beginWithReuse()` and
  `beginMulticastWithReuse()` to `bool`.
* Using better randomness for `LWIP_RAND()`.
* Changed attributes to use the C++ style.
* Changed the default hostname to "qnethernet-lwip".
* Renamed `enet_get_mac(mac)` to `enet_get_system_mac(mac)`.

### Fixed
* Added missing `hostByName` to keywords.txt.
* Fixed `enet_output_frame()` to return false if there's no output buffer.
* Fixed `EthernetClient::write(buf, size)` to re-check the state after a call
  to `loop()`.
* Fixed `util::writeMagic()` `mac` parameter to be `const`.
* Fixed `test_ethernet`'s `tearDown()` to remove listeners before calling
  `Ethernet.end()`. This ensures no out-of-scope variables are accessed.
* Updated `StdioPrint` to not use `errno` because stdio doesn't set this.

Highlights:
* Revamped driver interface
* New W5500 driver
* Can now specify library options in qnethernet_opts.h
* Easier to compile the library for other systems

Link: https://github.com/ssilverman/QNEthernet/releases/tag/v0.27.0
 
Back
Top