Synchronize many teensy boards over multiple meters underwater

weygoldt

New member
Hi everybody,

The project:

I am looking for some hardware advice with the following situation: We currently have multiple teensy-based data loggers that we want to arrange in a grid. This arrangement is used to gain knowledge about the spatial distribution of the logged signal. We currently use an 8 channel audio board to record the voltage on 8 electrodes for each microcontroller unit. Using at least 8 of those 8-electrode units, should in the end result in a grid of at least 64 electrodes, spaced approximately 50 cm apart, resulting in a grid of about 3.5 x 3.5 meters. In summary: 8 microcontrollers with 8 data inputs each. The controllers and audio boards will be enclosed in a waterproof housing and submerged. Recording takes several days, problably resulting in timing differences in the datasets of the loggers when combined.

The problem:

One of the biggest challenges is thus keeping the 8 units in sync. Wireless is not an option since they will be submerged. The current approach we came up with sounds simple: Use a master teensy that is connected to the 8 slave loggers using some kind of bus, which stops and starts them all at the same time in a regular interval. The master unit should be above the surface to be able to indicate if all of the loggers are still active and to synchronize them. Hence we need a wired connection to all slave units across at least 10 meters. It should also provide the power for all units. Hence we need at least 3 cables, ground, power and data.

Our approaches thus far:

(1) The DIY-bus
First idea we had was to use a simple data wire that switches state (i.e. input to output and vice versa) according to the current status. Every slave could send a "finished" signal to the master after x time has passed and once the master receives the signals of all slaves, it could switch the input to an output pin and send the reset signal to all units. This would be simple but limited in distance and very sensitive to noise. Additionally, every unit would need its own cabled connection to the master.

(2) The I2C bus
The second idea we came up with would be to use the I2C bus (we are aware that the distance we are talking about are much too large for the official use cases of this bus). This should (?) theoretically work if we dramatically decrease the frequency. But we have not tested it yet over long distances. The advantage would be, that we could daisy-chain the units, hence reducing wiring, which is desperately needed at the scale of this project.

The question:

We are looking for advice on how to achieve this as simply and reliably as possible. I already came across e.g. the RS485 bus to transmit data over longer distances but did not find any useful examples that implements this using a teensy. We would be very grateful for any advice or hints about this issue.
 
I like differential signals over (maybe shielded) twisted pair, for noise immunity. Any signals could be converted to differential signals, really, but they all require a little extra hardware (transceivers). Something like RS-485 is easily accessed from a Teensy via any of its serial ports; it just needs that conversion hardware.

For example: https://www.sparkfun.com/products/10124

The hardware (and cabling) needs to be rated for the speeds you need, but it’s not as much of a concern for lower speeds.
 
You didn't indicate how well synchronised you need things to be. I'm going to assume you are aiming for millisecond levels of synchronisation rather than nanoseconds.
By far the easiest is a broadcast based signal. The master sends a single signal that goes to all devices.

The options that come to mind are:
Ethernet - A UDP broadcast packet would do the trick, due to the variability in the network processing there may be a little bit of difference between units but it should be minimal. This does require a board with ethernet support but also provides a very high speed and flexible way to get data to and from the units.

RS232 - Simple and easy if you only want to send and not receive but if you want to get data back from 8 different nodes it gets messy.

RS485 - Similar to RS232 from a firmware perspective but gives you the option for sending data back.

CAN - Gives you multi-drop packet based communication with automatic prioritisation and bus arbitration.

Personally if all that is needed is in effect a tick to indicate when to start I'd be tempted to go with a uni-directional RS232 connection. You could either then send a single character to indicate start or use the pins as GPIO and use a pulse to indicate start. Very simple and easy.
If bi-direction data is needed the I'd probably go with CAN, that way everyone can talk to everyone else but the sync signal gets top priority. Worst case you'd get a timing jitter of 1 packet length (someone had just started sending data back when the next sync was due) but it would be the same for all devices.

Ethernet has the advantages of higher data rates and easier than the others to plug into a PC to monitor but has a significant firmware overhead.

All of these will require external driver ICs to produce the signal levels needed for longer distances (Teensy4.1 includes the ethernet Phy chip but still needs the magnetics and socket).
 
Thank you for your advice @shawn and @AndyA. Since bidirectional communication would be beneficial, I will look into using the CAN bus then. From what I read, this provides the distance and reliability that I need and should hopefully be fairly simple if we get a transceiver module.
 
If the devices have common ground (perhaps grounded to the surrounding water) then any should work.

If they don't have the same ground, ethernet might be a good choice, since the signals are isolated via the ethernet magnetics. But ethernet as implemented on Teensy 4.1 needs connection to an ethernet switch to talk to more than 1 other device.
 
Hi Paul and thanks for your reply. Since we plan on also supplying the units with power using other cords in the same cable, they will have a common ground. I am still gravitating towards CAN for that reason. I have no experience working with either, but CAN seems easier to get into (tell me if I'm wrong). There is another issue coming up though: We would connect all 8 recorders along the same CAN bus. If they finish their recording interval, they should send a 'finished' signal to the master, which then restarts the recording on all units. But how would a can bus handle the 'finished' signal to the master, which should ideally all be send at the same time?
 
Thank you for your advice @shawn and @AndyA. Since bidirectional communication would be beneficial, I will look into using the CAN bus then. From what I read, this provides the distance and reliability that I need and should hopefully be fairly simple if we get a transceiver module.

It's very basic - Rx and Tx from the processor to the transceiver. Connect all the CAN high lines together, connect all the CAN low lines together. Add terminating resistors between the CAN high and CAN low wires. It is designed to be purely differential, unless you are using the same cable to remotely supply power there is not normally a need for a ground connection between two units on the same bus.

Two things that will stop it working are no terminating resistors and not having anything on the bus acknowledging packets. Either of these issues will cause constant errors. Make sure your devices are set to acknowledge packets, this is normally the default but sometimes people switch on silent mode without realising this can break things. It's OK to have multiple things acknowledge at the same time.

Ideally the terminating resistors are two 120 ohm resistors, one at each end of the bus. But it's generally fairly tolerant to this being a little off, especially at lower speeds. I regularly run busses with 1 or 3 resistors without any issues.
If you have a star shaped bus (as I suspect you may end up with given your situation) then you could possibly go with 120 ohms in the middle and higher value resistors on each point of the star aiming for a total parallel resistance of around 60 ohms.

The CAN hardware will automatically give priority to messages with lower IDs, make use of this when deciding what IDs to use for different messages.
 
Weygoldt:
I implemented a system sort of like yours about ten years ago using the original Teensy3. It uses RS485 (which can have multiple talkers and listeners on the same bus) full duplex. The data coming and going used DMA to move the data over the RS485 bus so the Teensy did not have to concern itself with that part of the hardware. It worked at 500k bits per second. Each packet has a time stamp, CRC, address, and size info so each chunk of data basically looks a lot like ethernet packetized networking. The MAX14854 RS485 transceiver I believe is rated at 10Mbytes/second.

Here is the picture from their data sheet:

Screen Shot 2023-03-29 at 2.55.39 PM.jpg

This system has been working very reliably for about 10 years 24 hours a day.

My .02 cents worth,
Ed
 
Back
Top