T4.1 conflict between two libraries

L

LaVoxxx

Member
Hello Teensy members,

When developing a sepcific multi IO PCB for a project, I have to process different signals, control actuators and communicate with a master controller via EtherCAT.

In detail, the following peripherals are connected to a Teensy 4.1:
- 2 x load cell
- 2 x analogue microphone
- 10 digital combination inputs and outputs
- 6 x RC servo control
- 1 x WS2812 LED strip with max. 255 pixels
- 1 x EtherCAT slave shield with SPI interface (EasyCAT Pro)

During development, I noticed the following problems, some of which I was able to solve, but some of which I was not.

github.com

EasyCAT/libraries/EasyCAT/EasyCAT.h at master · dslemusp/EasyCAT

Repository containing libraries and configuration gui for the EasyCAT shield board - dslemusp/EasyCAT
github.com github.com

In order to be able to use the EtherCAT Salve Shield library (link above) without errors, I had to reduce the Teensy clock to 12-24MHz.
With higher clock rates, especially the standard 600MHz, I constantly received corrupt data from the shield via SPI or the cyclic method MainTask() hung up completely. SPI runs with 8MHz.

I also had to insert a delay of 125us after each CS Write command in the EasyCAT library, otherwise I had similar problems.

I found both ‘solutions’ in various forum threads, but they had nothing to do with my actual problems.
With these adjustments everything runs stable, except the WS2812 is causing me big problems.

I use the WS2812serial.h library from Paul, which basically works fine for what I need it for.
However, as soon as I call the show() method, I get the same problems with the SPI interface of the EtherCAT Shield.

Unfortunately, I lack the necessary in-depth knowledge to judge for myself to what extent the two libraries can influence each other and am therefore dependent on your help.

All libraries I use are - if available - those from Paul and adapted for the Teensys.
The EtherCAT communication between shield and master is 100% stable and error-free, so this can be ruled out as a cause.

I will be happy to provide further information if you need it.

Many thanks in advance and best regards from Switzerland
LaVoxxx
 
Really no one who has an idea or an approach as to how I can solve or at least narrow down the problem?

The EasyCAT library only uses native SPI commands, which should basically run on the Teensy with the corresponding SPI library, or am I wrong?

As this is my first point of contact with the Teensy forum, please let me know if I can provide you with further information or if I am in the wrong thread or something similar.
Also, I didn't want to flood you with more details and code snippets. :)

I would be very grateful if someone would take pity on me...
Best regards
 
I think the issue is that you've detailed a problem that only exists with a fairly uncommon piece of hardware. It's hard to debug an issue if you can't reproduce it.

WS2812 require some fairly accurate timings, I can see dropping the CPU speed that much and adding lots of waits causing issues.

I don't have the ethercat shield so can't say for sure but as a general rule if a library only works when the CPU speed is reduced to 1/20th of the normal speed then there are either some serious issues with that library or with the hardware it's talking to.
The first step should be to get each library to work independently without having to cripple the CPU speed and then worry about getting them to work together.
 
Is there a reason to use the EasyCAT over SPI versus the T_4.1's native ethernet with a $4 adapter? For general use the QNEthernet library offers great functionality and doesn't take fiddly adjustments to work at 600 MHz and no delays.

The PJRC forum has slow and active days - to see a response takes the right person - online - to recognize an issue. Slow days and uncommon or involved situations won't get fastest turn around.

@AndyA has good advice to find all the things that work alone then together at 600 MHz. Slowing the processor down limits the ability to be responsive with many active elements needing timely attention.

WS2812serial as non-blocking is the right library to use - but it will require some timely background processing overhead.
 
defragster said:
Is there a reason to use the EasyCAT over SPI versus the T_4.1's native ethernet with a $4 adapter? For general use the QNEthernet library offers great functionality and doesn't take fiddly adjustments to work at 600 MHz and no delays.
Click to expand...
While there are a lot of similarities EtherCAT is different to standard ethernet, I'm guessing he has a specific hardware requirement that the built in ethernet can't handle.
If it can be done using standard 100 Mbit ethernet then yes the built in device is the way to go.
 
Thank you for your quick response.

@AndyA - We have decided in favour of the ‘unusual’ hardware (EasyCAT), as this eliminates the effort of implementing our own EtherCAT slave, which should not be underestimated.
This requires specific controllers on which the software stack itself has to be integrated and, in the case of commercial use, results in complex approval tests by the EtherCAT org..

As you suggested, I will do some tests again, in which I will add the libraries one after the other and record which changes are necessary and when, in order to avoid errors.
I'll get back to you on this...


@defragster - The reason for using EtherCAT is the existing infrastructure.
We are machine builders and use an IPC from Beckhoff as the higher-level PLC controller, all of whose slaves communicate via EtherCAT and offer functions such as hard real-time and frame synchronisation across different slaves.
Furthermore, 100Mbit EtherCAT is much more stable than native Ethernet, over which we originally worked using Modbus TCP, which in turn led to frequent faults (cables run through an industrial robot arm with many large servomotors).
 
I have been carrying out various endurance tests with different software configurations the whole time and have tried to categorise when critical problems occur.

Below are the configurations and their stability:
  1. EasyCAT.h and counters only - stable at 600 MHz
  2. + analogRead() incl. threshold value evaluation for 2 microphones - stable at 600 MHz
  3. + GyverHX711.h for 2 x load cell evaluation - stable at 200 MHz
  4. + Servo.h for 6 x KST servo motor - stable at 20 MHz
  5. + WS2812erial.h - never stable as soon as show()-method is called in cyclic intervals (e.g. for 20ms running light)

Although reducing the main clock seems unattractive, I have no problem with it as long as everything runs stably and reliably.
However, something in the WS2812 library seems to influence the SPI communication (only when calling the show() method).

Because of the Servo.h library I am forced to use the WS2812.h.
Unfortunately Adafruit and FastLED both influence the control behaviour of the servo motors.

Can you tell anything from this data?

I have looked through the libraries (EasyCAT.h and WS2812Serial.h) again in detail - as I said, I am absolutely no specialist in this area.
For me, the only thing they have in common is that they both work directly or indirectly with DMA in some way, but that's just a guess on my part.
 
There is a shorter way to say stable with a CPU speed of 200 MHz but not at 600 MHz. Broken.

As before, either fix or don't use the broken libraries.

Two things both using DMA could potentially explain issues using the two together. I don't know enough about the teensy DMA system to know whether this would be an easy fix, it may be as simple as changing which DMA channels are used for one of the two libraries.
 
LaVoxxx said:
+ Servo.h for 6 x KST servo motor - stable at 20 MHz
Click to expand...
Have you tried PWMServo library instead? Assuming you can use PWM pins for the servos,
This library is installed as part of Teensy board installation either through Teensyduino (for IDE 1.x) or the board manager for 2.x
 
LaVoxxx said:
  1. + WS2812erial.h - never stable as soon as show()-method is called in cyclic intervals (e.g. for 20ms running light)
Click to expand...

Can you reproduce the WS2812Serial problem without the EtherCat hardware connected?

Or to ask more specifically, can you share a (hopefully small) program which I or anyone else with WS2812 LEDs (but not the other special hardware) can run on a Teensy 4.1 to reproduce this WS2812Serial issue?

I'd really like to investigate why it's not working. But without a program to reproduce the problem, no way to know what's really going wrong.
 
KurtE said:
Have you tried PWMServo library instead? Assuming you can use PWM pins for the servos,
This library is installed as part of Teensy board installation either through Teensyduino (for IDE 1.x) or the board manager for 2.x
Click to expand...

I have used just about every library I could find for my peripherals, including ServoPWM.h.
Unfortunately, this resulted in other ‘problems’, which is why I refrained from using it again:
  • No native method for specifying the pulse width in ms, therefore less resolution of the motors (could certainly be supplemented by rewriting the library)
  • In the end, the use has led to more SPI problems than the Servo.h library
  • The possible use of FastLED or the Adafruit library has massively worsened the overall performance of my application (e.g. the cyclic update rate using analogRead() was many times slower)


PaulStoffregen said:
Can you reproduce the WS2812Serial problem without the EtherCat hardware connected?
Click to expand...

Unfortunately I cannot reproduce the error using only the WS2812Serial, not even in combination with other libraries (EasyCAT excluded).

In the end, the error only manifests itself in the fact that when receiving the SPI data (using EasyCAT.MainTask()) the byte order is shifted and/or truncated, as a result of which I have incorrect values in the buffer, or the Methods freezes the Teensy completely).

Perhaps the error could be reproduced with an alternative SPI hardware, but unfortunately I don't have anything suitable to hand.


Current status

I have spent the last two days again with intensive trials, tests and ‘optimisations’ and have now reached a stable status for me, in which all functions work without errors in the required cycle time of 1ms, including the WS2812Serial. :)

I have made the following adjustments for this, in part purely empirically:
  • Reduced the clock rate of the Teensy to 10MHz (the higher, the more likely erroneous buffer data occurred)
  • Increased the SPI clock to 30MHz (this corresponds to the maximum SingleSPI clock rate of the EasyCAT Pro - less also led to erroneous buffer data)
  • Increased the time between the call cycles of the individual objects in my application (only EasyCAT.MainTask() is now called every cycle, the rest only every 1-10ms)
  • Shortened the cabling of the overhead lines on my ‘test board’ as much as possible and separated signal lines from each other as much as possible
  • The supply voltages of the individual consumers are separated from each other by means of several power supply units.

The optimisation of the cabling and the power supply proved to be an important step.
Although I still have to massively reduce the Teensy clock, the sporadic and previously non-reproducible SPI errors have completely disappeared.

It also seems to me (just a guess) that the SPI clock must be higher than the Teensy clock, at least in combination with the rest of the libraries.
Any other combination will result in the known errors.

Even though the road has been rocky so far, I am completely satisfied with the current result in terms of stability and functionality.
Although @AndyA would say it's broken. ;)

In any case, castrating the powerful Teensy 4.1 to 10MHz leaves a bitter aftertaste, but for me at the end of the day what counts is whether it meets my application requirements, and this now seems to be the case.

I will now start developing my PCB and in the meantime observe the stress test of my application in continuous operation.
In any case, I will be back with status updates, hopefully no more new problems.

For anyone interested, I have attached my current test code (neither complete nor final).
 

Attachments

  • EasyCAT_PRO_TestApp.ino
    13.1 KB · Views: 14
You must log in or register to reply here.
Back
Top