New lwIP-based Ethernet library for Teensy 4.1

shawn said:
I realized after it was uneditable that the UDP send speed is a value > 100Mbps. I’m sure there’s a good reason for that…
Click to expand...

since UDP is unreliable (packets can be damaged ,lost, duplicated, arrive out of order) and the OS may buffer output, it is best to measure UDP performance at the receiver.

attached are some C test programs from the 1980's -- you'll get lots of warnings from gcc. Despite the warnings, programs still build on linux. udpsrc is a rate-controlled UDP sender. ttcp is vintage 1984

I fixed my udp.parsePakcet() problem by adding a 1 second timeout and delaying less than 10 seconds in my udp_ntp() test. Often the first UDP packet is not sent as the lwIP stack awaits ARP resolution (observe with wireshark).

With udp_ntp() test you can measure crystal drift of your T4.1 (rollover at 4295 seconds)
Code: 
ntp 3841126089.265207  rtt 466 us   -5.61 ppm over 440 s
ntp 3841126099.265624  rtt 345 us   -5.62 ppm over 450 s
ntp 3841126109.266149  rtt 462 us   -5.61 ppm over 460 s
...
 

Attachments

  • etest.zip
    11.7 KB · Views: 109
Last edited:
@manitou: The remoteIP() was zero because when that was called, the client was already disconnected. I assume that once a client disconnects, things like remoteIP() shouldn't be valid; they go along with `operator bool()` and with `connected()`.
 
I just released v0.7.0. The changes:

Code: 
## Added
* The Boolean-valued link state is now `EthernetClass::linkState()`.
* Added a weak `_write()` definition so that `printf` works and sends its output
  to `Serial`. Parts of lwIP may use `printf`.
* Now powering down the PHY in `enet_deinit()`.
* Added calls to `loop()` in `EthernetServer::accept()` and `available()` to
  help avoid having to have the caller remember to call loop() if checking
  connectivity in a loop.
* Added a call to `end()` in the `QNMDNS` destructor.
* Added a new externally-available `Print *stdPrint` variable for `printf`
  output, both for lwIP and optionally for user code.
* Added the ability to set the SO_REUSEADDR socket option when listening on a
  port (both TCP and UDP).
* Added four examples and a "note on the examples" section in the README.
  1. FixedWidthServer
  2. LengthWidthServer
  3. ServerWithAddressListener
  4. SNTPClient

## Changed
* `EthernetClass::linkStatus()` now returns an `EthernetLinkStatus` enum. The
  Boolean version is now `EthernetClass::linkState()`.
* The `EthernetLinkStatus` enum is no longer marked as deprecated.
* Updated `EthernetClient` output functions to flush data when the send buffer
  is full and to always call `loop()` before returning. This should obviate the
  need to call `flush()` after writes and the need to call `loop()` if writing
  in a loop. (`flush()` is still useful, however, when you've finished sending a
  "section" of data.)
* Changed `EthernetUDP::parsePacket()` to always call `loop()`.
* Changed `_write()` implementation to be non-weak and updated the output to
  direct to the new `Print *stdPrint` variable. It has a default of NULL, so
  there will be no output if not set by user code.

## Fixed
* Restarting `Ethernet` (via `begin()` or via `end()`/`begin()`) now works
  properly. DHCP can now re-acquire an IP address. Something's weird about
  `EventResponder`. It doesn't look like it's possible to `detach()` then
  `attach()`, or call `attach()` more than once.
* Fixed `QNMDNS` to only call `mdns_resp_init()` once. There's no corresponding
  "deinit" call.

There are now a few examples and a note about them in the README.

Link: https://github.com/ssilverman/QNEthernet/releases/tag/v0.7.0
 
Has anyone gotten this library to work with VisualTeensy?

I was hoping to try it out.

If I create a new project in VisualTeensy 1.2.0 and only add in the QNEthernet (0.7.0), it won't compile the demo blink program. Using Teensyduino 1.55 core.

Code: 
CORE [ASM] memset.S             
CORE [ASM] memcpy-armv7m.S 
CORE [CC]  sm_util.c 
CORE [CC]  sm_calloc.c 
CORE [CC]  sm_free.c 
C:\PROGRA~2\Arduino\hardware\teensy\avr\cores\teensy4/sm_free.c: In function 'sm_free_pool':
C:\PROGRA~2\Arduino\hardware\teensy\avr\cores\teensy4/sm_free.c:15:3: error: 'errno' undeclared (first use in this function)
   errno = EINVAL;
   ^
C:\PROGRA~2\Arduino\hardware\teensy\avr\cores\teensy4/sm_free.c:15:3: note: each undeclared identifier is reported only once for each function it appears in
C:\PROGRA~2\Arduino\hardware\teensy\avr\cores\teensy4/sm_free.c:15:11: error: 'EINVAL' undeclared (first use in this function)
   errno = EINVAL;
           ^
make: *** [makefile:183: .vsteensy/build/core/sm_free.c.o] Error 1      
make: *** Waiting for unfinished jobs....
CORE [CC]  clockspeed.c 
CORE [CC]  usb_keyboard.c 
CORE [CC]  usb_seremu.c 
CORE [CC]  usb_serial.c 
CORE [CC]  sm_malloc.c 
C:\PROGRA~2\Arduino\hardware\teensy\avr\cores\teensy4/sm_malloc.c: In function 'sm_malloc_pool':
C:\PROGRA~2\Arduino\hardware\teensy\avr\cores\teensy4/sm_malloc.c:17:3: 
error: 'errno' undeclared (first use in this function)
   errno = EINVAL;
   ^
C:\PROGRA~2\Arduino\hardware\teensy\avr\cores\teensy4/sm_malloc.c:17:3: 
note: each undeclared identifier is reported only once for each function it appears in
C:\PROGRA~2\Arduino\hardware\teensy\avr\cores\teensy4/sm_malloc.c:17:11: error: 'EINVAL' undeclared (first use in this function)
   errno = EINVAL;
           ^
C:\PROGRA~2\Arduino\hardware\teensy\avr\cores\teensy4/sm_malloc.c:101:10: error: 'ENOMEM' undeclared (first use in this function)
  errno = ENOMEM;
          ^
make: *** [makefile:183: .vsteensy/build/core/sm_malloc.c.o] Error 1    
CORE [CC]  usb_joystick.c 
CORE [CC]  delay.c 
CORE [CC]  usb.c 
CORE [CC]  analog.c 
CORE [CC]  sm_szalloc.c 
C:\PROGRA~2\Arduino\hardware\teensy\avr\cores\teensy4/sm_szalloc.c: In function 'sm_szalloc_pool':
C:\PROGRA~2\Arduino\hardware\teensy\avr\cores\teensy4/sm_szalloc.c:14:3: error: 'errno' undeclared (first use in this function)
   errno = EINVAL;
   ^
C:\PROGRA~2\Arduino\hardware\teensy\avr\cores\teensy4/sm_szalloc.c:14:3: note: each undeclared identifier is reported only once for each function it appears in
C:\PROGRA~2\Arduino\hardware\teensy\avr\cores\teensy4/sm_szalloc.c:14:11: error: 'EINVAL' undeclared (first use in this function)
   errno = EINVAL;
           ^
make: *** [makefile:182: .vsteensy/build/core/sm_szalloc.c.o] Error 1   
CORE [CC]  sm_alloc_valid.c 
C:\PROGRA~2\Arduino\hardware\teensy\avr\cores\teensy4/sm_alloc_valid.c: 
In function 'sm_alloc_valid_pool':
C:\PROGRA~2\Arduino\hardware\teensy\avr\cores\teensy4/sm_alloc_valid.c:14:3: error: 'errno' undeclared (first use in this function)
   errno = EINVAL;
   ^
C:\PROGRA~2\Arduino\hardware\teensy\avr\cores\teensy4/sm_alloc_valid.c:14:3: note: each undeclared identifier is reported only once for each function it appears in
C:\PROGRA~2\Arduino\hardware\teensy\avr\cores\teensy4/sm_alloc_valid.c:14:11: error: 'EINVAL' undeclared (first use in this function)
   errno = EINVAL;
           ^
make: *** [makefile:182: .vsteensy/build/core/sm_alloc_valid.c.o] Error 
1
CORE [CC]  usb_serial3.c 
CORE [CC]  interrupt.c 
CORE [CC]  sm_zalloc.c 
CORE [CC]  startup.c 
C:\PROGRA~2\Arduino\hardware\teensy\avr\cores\teensy4/startup.c: In function '_sbrk':
C:\PROGRA~2\Arduino\hardware\teensy\avr\cores\teensy4/startup.c:643:25: 
error: 'errno' undeclared (first use in this function)
                         errno = ENOMEM;
                         ^
C:\PROGRA~2\Arduino\hardware\teensy\avr\cores\teensy4/startup.c:643:25: 
note: each undeclared identifier is reported only once for each function it appears in
C:\PROGRA~2\Arduino\hardware\teensy\avr\cores\teensy4/startup.c:643:33: 
error: 'ENOMEM' undeclared (first use in this function)
                         errno = ENOMEM;
                                 ^
make: *** [makefile:182: .vsteensy/build/core/startup.c.o] Error 1      
CORE [CC]  rtc.c 
The terminal process "C:\Program Files\PowerShell\7\pwsh.exe -Command .../GitHub/VisualTeensy_v1_2_0/make.exe all -j -Otarget" terminated with exit code: 1.
 
This library contains its own copy of the system file errno.h. For some reason the the core includes the library file instead of the correct system one. Probably some lookup order issue in the makefile. I'll have a look. BTW: might be better to discuss that kind of issues in the VisualTeensy thread. I'll answer there if I found a solution.
 
Glad you got it working. See the new SNTPClient example in the QNEthernet release for another approach.
 
Jumbo frames are on my list to test. IP fragmentation is enabled (see lwipopts.h and lwip/opt.h), so in theory they should work. Have you tried them yet?

By “mpu” do you mean “MTU”? If so, that can’t be changed since it’s a hardware limit. IP fragmentation/reassembly should do what you want. (I think, since I haven’t played with it yet.)
 
IP fragmentation can do what you want, though for TCP both sides must agree to use a larger size through the MSS setting.
 
vjmuzik said:
IP fragmentation can do what you want, though for TCP both sides must agree to use a larger size through the MSS setting.
Click to expand...

Thanks Shawn, yes, MTU, not mpu. Perhaps fragmentation will work for me, I'll test this today.

Still learning this ethernet stuff. Do I need to just make sure my buffers are large enough & simply send it, letting LWIP handle all the fragmenting?
 
It should handle the fragmenting, though after thinking about it with TCP you don’t have to use ip fragmentation. Due to the way TCP works as long as you have big enough buffers to store it it will split the 3-5k packet into multiple packets of size up to the MSS setting already. Also take into consideration that TCP is a streaming protocol meaning you shouldn’t design your code to expect to receive all of a large packet at the same time. It can happen, but it’s not a guarantee like it would be with protocols like UDP. Though depending on what kind of transfer speeds you are trying to achieve it can help to increase the MSS and buffer size since it’s able to transfer more data between acknowledgments.
 
shawn said:
By “mpu” do you mean “MTU”? If so, that can’t be changed since it’s a hardware limit. IP fragmentation/reassembly should do what you want. (I think, since I haven’t played with it yet.)
Click to expand...

I need to correct myself here. The Ethernet frame size is limited by hardware. The MTU is not.
 
shawn said:
Here’s some interesting links on MTU path discovery and fragmentation:

@TeensyWolf what is your use case for the jumbo frames? I’m curious about it.
Click to expand...

I'm interfacing with some legacy hardware over serial and providing an Ethernet interface that typically exists with the newer models. I don't analyze the data or know much about the control protocol, just pushing bits back & forth with minimal reverse-engineering. Once I had it up and running, every once in a while, it saw some data being requested that I hadn't seen before, which can get rather large. Once it started to overflow my buffer (I was 64 bytes initially), I then saw 700 bytes. On one unit, it was over 3000. I wiped that unit, as the history isn't needed, but I should have kept it for test. So I haven't seen that happen since. I'm not sure if Jumbo will work here or fragmentation, the pc application is a given as well as the device. For now we have a procedure (wipe the unit) to get beyond this issue.

It's possible the fragmentation actually worked once I got the buffer sizes up from 2K to 8K, but it was late in the day and I was assuming I needed a jumbo frame to solve the issue, that whole "a little bit of knowledge..." conundrum.

Sorry I'm unable to talk much more about the specific hardware and protocols, but that's the project in a nutshell. As I think about it now, Jumbo is probably not needed here.

This is my first time diving deep into TCP, deep for me anyway. Learning something new everyday. Thanks for the links and the library. It's been running solid.
 
Are you sure jumbo frames were being used, though? Eg. That it’s not just a TCP stream containing data larger than a few frames?

Which “buffer” was overflowing? I’m not 100% convinced this is related to jumbo frames, per your description.
 
shawn said:
Are you sure jumbo frames were being used, though? Eg. That it’s not just a TCP stream containing data larger than a few frames?

Which “buffer” was overflowing? I’m not 100% convinced this is related to jumbo frames, per your description.
Click to expand...

My Serial RX buffer, which I then send to TCP. The buffer was undersized, so my serial checksums were off.

I'm unsure about jumbo frames, I never knew what they really were until I googled "max size of tcp packet". Then I figured I had a problem as I had 3000 bytes and Jumbo seemed to be the obvious answer, hence the question here. But these things are 100mbit only, and I believe jumbo is for 1gb-E.

When I get a chance to Wireshark a non-legacy device, I'll see what's actually being sent.
 
Without knowing more, my first guess is that the processing code isn’t properly handling the input data. I could be wrong, but I wouldn’t start looking at TCP frame sizes; I’d look at how your application is reading (or sending) bytes.

In my experience, most applications don’t need to worry about packet size; that’s a different layer.

Another question: why do you think the data is being sent in only a single packet?
 
I just released v0.8.0. The changes:

Code: 
### Added
* Added a check that `Entropy` has already been initialized before calling
  `Entropy.Initialize()`.
* Added a "How to write data to clients" section to the README that addresses
  how to fully send data to clients.
* Added `EthernetClient::writeFully()` functions that might help address
  problems with fully writing data to clients.
* Added a new "Additional functions not in the Arduino API" section to
  the README.
* Added `EthernetClient::closeOutput()` for performing a half close on the
  client connection.

### Changed
* Updated the `ServerWithAddressListener` example. It's more complete and could
  be used as a rudimentary basis for a complete server program.
  1. Added a "Content-Type" header to the response,
  2. It now looks for an empty line before sending the response to the client,
  3. Added the ability to use a static IP,
  4. Added client and shutdown timeouts, and
  5. Added a list to the description at the top describing some additional
     things the program demonstrates.
* In `EthernetClass::end()`, moved setting the DNS to 0 to before DHCP is
  released. This ensures that any address-changed events happen after this.
  i.e. the DNS address will be 0 when an address-changed event happens.

Direct link: https://github.com/ssilverman/QNEthernet/releases/tag/v0.8.0
 
I just released v0.9.0. The changes:

Code: 
### Added
* Added example that uses `client.writeFully()` to the "How to write data to
  connections" README section.
* Added `EthernetClient::close()` for closing a connection without waiting. It's
  similar to `stop()`.
* Added `DNSClient` class for interfacing with lwIP's DNS functions.
* Added a "DNS" section to the README.

### Changed
* Renamed the "How to write data to clients" README section to "How to write
  data to connections".
* Increased the maximum number of UDP sockets to 8.
* Updated `EthernetClass`, `EthernetClient`, and `EthernetUDP` to use the new
  `DNSClient` class for DNS lookup and DNS server address setting.

Direct link: https://github.com/ssilverman/QNEthernet/releases/tag/v0.9.0
 
Shawn,

My last TCP project has been working very reliably, thanks for your help and this library.

I see you have a reference to Dan Brown's implementation of IEEE1588 on your Github readme. Is this something QNEthernet supports now, or is on the roadmap to support one-day?

I'm looking a project that could use some tight time stamping of data for separate devices, shooting them over ethernet to a collector/recorder. I'm going to try it with NTP, but I'd like it to be within a half-millisecond, maybe less, to remove that uncertainty. So kinda eyeballing that IEEE1588 as a plan-B.

Thanks.
 
It’s been on my mind the last few days. When I have some time, I’ll maybe look into what’s required to implement it from an API perspective. What would a good API look like to you? I know what 1588 is, and even how it’s implemented, I’ve just never used it, so an external perspective from someone who isn’t as close to the internal plumbing would be nice.

P.S. Thank you for the nice words. :)
 
Think this may help solve an issue I am fighting with the old Ethernet lib?

If ethernet cable not plugged in and try to fire up

Code: 
Ethernet.begin(macAddr, IPAddr);

Freezes up things. I wait A few minutes and still stuck on that line of code running.
As soon as I plug A cable in, few seconds later, it recovers and moves on and is happy.

When I can get A chance i'll try this out.

Thank you for doing the hardwork!
 
You must log in or register to reply here.
Back
Top