Shuffling RTCM3 correction data through a Micromod Teensy

[ninja2]

My hardware:
Sparkfun Main Double board, fitted with:
Processor board : Micromod Teensy
Function board 0: Micromod ZED-F9P GNSS
Function board 1: Micromod ESP-32 WiFi/Bluetooth

My recent interest has been developing a reliable feed of an RTCM3 correction data to the ZED-F9P, so it can maintain RTK levels of GPS accuracy (~cms)

I have a pair of programs doing this task reasonably well, one on the ESP32, the other on the Teensy. A public correction data stream is received by the ESP32 and forwarded over Serial into the Teensy, which then forwards it via another Serial port to the ZED-F9P.

A bit more detail: the ESP32 connects via its built-in WiFi (and my router) to a public NTRIP Caster (Networked Transport of RTCM via Internet Protocol) that provides the feed. So this is my data pipeline:

NTRIP Caster -> ESP32 -> Teensy -> ZED-F9P GNSS

The average data rate of the feed is not particularly high, around 2.4kB/s. However it comes in bursts of one, several or many RTCM correction messages. So a burst can be anywhere from a few bytes containing a single short message, to over 16kB in one burst containing as many as 50 messages, on occassion. A typical burst has around 5-8 messages contained in ~2,500 bytes.

To handle the data bursts the ESP32 reads the data into a 16kB buffer. The buffer contents are processed between bursts, dividing the burst into individual messages. The CRC of each message is checked to filter out any duds. Verified messages are forwarded one at a time to the teensy.

The teensy receives each message as a 'package' consisting of 3 header bytes + the message bytes + 3 CRC bytes. These packages are typically between 50 and 600 bytes in length. Unlike on the ESP32, a dedicated buffer is not currently used. Instead the main program loop uses

while (Serial2.available())

to process a whole package, including CRC checking (again) before forwarding the package (if healthy) on to the ZED-F9P.

As mentioned I have a working solution, but it was not always so. At first the teensy was dropping about 50% of the packages. At that stage both Serial2 in and Serial1 out were set to 115,200 baud. I thought that would be sufficient since that = 14.4kB/s, comfortably more than the average RTCM data rate of < 2.5kB/s. But it only started working after I inserted

delay(50)

into the ESP32 processing loop, effectively adding a 50ms break between package transmissions. At last, the teensy hardware serial ports could keep up!

But that workaround didn't sit right, so I kept fiddling. I increased the Serial1 & Serial2 to 921600 baud (i.e an 8-fold increase) and found I could reduce

delay(50)

down to

delay(5)

with very few dropped messages. If I increased back up to 10 the dropped packages were almost zilch.

Improved but still not ideal. For now the teensy only has the task of echoing this data stream on to the ZED-F9P, but it will soon have several other major tasks, so I want my solution to add minimal load for the teensy to do the data echo function.

I've started looking into/contemplating how the Serial buffers work. There is helpful info at Teensyduino Hardware Serial Ports, but I could't find any info on the standard buffer sizes. The Arduino Serial reference suggests only 64 bytes but I think it may be 128 bytes since Serial.availableForWrite() always seems to report values < 128.

I guess I'm looking for ideas on how to make the serial data echo more efficient on the teensy, and specifically if I increase the buffer size on the incoming, or outgoing data (or both) is that likely to further free up the serial pipeline? If so, how to do that ... ?

TIA

 

[thebigg]

Look at these functions:

    // Increase the amount of buffer memory between reception of bytes by the
    // serial hardware and the available() and read() functions. This is useful
    // when your program must spend lengthy times performing other work, like
    // writing to a SD card, before it can return to reading the incoming serial
    // data.  The buffer array must be a global or static variable.
    void addMemoryForRead(void *buffer, size_t length);
    // Increase the amount of buffer memory between print(), write() and actual
    // hardware serial transmission. This can be useful when your program needs
    // to print or write a large amount of data, without waiting.  The buffer
    // array must be a global or static variable.
    void addMemoryForWrite(void *buffer, size_t length);

 

[ninja2]

I did read about them. But addMemoryForRead() &r addMemoryForWrite() seem to use ordinary memory for the extra buffer, is using either (or both) really any better then adding buffer(s) myself, in my code ?

 

[thebigg]

Yes. It is integrated into the hardware interrupts that handle the read / write of the UART.

 

[ninja2]

OK I'll have to give them a try ... hopefully there's a good example of how to use them properly

 

[ninja2]

FWIW these instruction worked like a charm:

#define RX_BUFFER_SIZE 1024               // expanded ESP serial incoming buffer 
#define TX_BUFFER_SIZE 1024               // expanded ZED serial outgoing buffer
unsigned char RX_buffer[RX_BUFFER_SIZE];
unsigned char TX_buffer[TX_BUFFER_SIZE];
ESP.addMemoryForRead(RX_buffer,RX_BUFFER_SIZE); // add 1K buffer to reduce risk of incoming bottlenecks
ZED.addMemoryForWrite(TX_buffer,TX_BUFFER_SIZE); // ditto for outgoing bottlenecks

 

[AndyA]

Two things to note:

1) GNSS receivers will expect RTCM to be arriving via radio link which may have questionable reliability. This means they have good packet validity checking and data sanitisation. In other words while all of your checking of packet structure and checksums is good it's also completely unnecessary. The GNSS is more than capable of handling any bad data that may arrive on that port.
If you are going to do some sanity checking on the data and are seeing bursts of 50 messages then you must be getting multiple epochs worth of data in the same burst. In that situation discard any messages with the same type but an older timestamp.
Also the station information (station location, glonass biases etc...) are only needed once by the receiver. Once you are getting RTK you don't need to forward that data again.

2) You've hit a standard issue with uarts (it actually makes for a great interview question, obscure enough that they may not have seen it but not so abstract that it's unrealistic to expect them to be able to think it through with some prompting). The data rate is set by the transmitter and can vary by 5% for a given baud rate. Which means if forwarding the data on from one port to the other you may end up receiving data faster than you are sending it and so run out of buffer if there is a long burst of data.
The generic solutions to this is to either add flow control, add more buffering (which implies calculating the maximum size data burst to calculate how much you may need), or to set the output port to use a higher baud rate than the input port.
If however the data flow is purely in one direction the sneaky solution is to set your data transmitter to use 2 stop bits while the receiver is set to 1 stop bit. This will slow the data rate down by around 10%, even with worst case tolerances on the uarts the output will still keep up with the input. This only works for single direction data since the unit set to 2 stop bits will fail to receive data sent by the unit on 1 stop bit unless there is a pause between each byte. This lets you keep the same baud rates and doesn't require adding flow control or buffering.

 

[ninja2]

Very interesting @AndyA.

I started with simple byte by byte forwarding from the ESP32 into Serial2 on Teensy, and again on the Teensy Serial1 to the ZED-F9P. This sort of worked but was unreliable and the RTK lock would drop out a lot of the time. So I added buffering, parsing and CRC checking on the ESP32 that resulted in transmission of messages one at a time (rather than bursts up to 50). I also added on-the-fly CRC checking for each message package received by the Teensy, before forwardng to the ZED. At this stage only about 50% of the packages were passing CRC checks on the Teensy (possibly due to that 5% variation you mention?). Upping the baud rates from 115200 to 921600 reduced the dropouts, and finally expanding the Serial buffer using those

addMemoryForWrite()
addMemoryForRead()

instructions eliminated almost all dropouts.

With these in place I also found that CRC checking on the Teensy is no longer needed, I can revert to simple byte-by-byte forwarding to the ZED and it still works fine. This is good - the Teensy has lots of other important tasks to do.

Yesterday I also found the ZED doesn't work too well as a GNSS over 921600 baud, and reducing it back to 115200 didn't hinder the RTCM3 stream. I've left the buffering, parsing and CRC checking on the ESP32 and the serial path to Teensy at 921600 as this is all the ESP32 has to do, and it's working well.

All this may well have been due to those 5% baud rate variations. (and I've still got your nifty 2 stop bits trick up my sleeve)
In any event the whole exercise was a great introduction to the RTCM3 protocol, and Serial buffering.

 

[AndyA]

Yesterday I also found the ZED doesn't work too well as a GNSS over 921600 baud, and reducing it back to 115200 didn't hinder the RTCM3 stream.

As you mentioned in your first post the average rate on RTCM is tiny, higher rates help with bursts but if you buffer enough then you don't need much of an average rate.
Generally I avoid running the ublox serials over around 230400, I've hit issues with reliable communication at higher rates. Admittedly that was on older parts so the F9 may be better.

In any event the whole exercise was a great introduction to the RTCM3 protocol, and Serial buffering.

Entertaining isn't it. They take bit packing to the extreme, if something needs 13 bits then it gets 13 bits, no padding, not a single bit wasted on boring things like byte aligning fields. And to add to the fun some fields are variable lengths depending on the contents of previous fields.

If you use an F9 as a base station the ublox RTK correction output has some oddities that mean their corrections only work intermittently with most other brands of GNSS receiver. I ended up having to create something to unpack and rewrite the correction messages on the fly in order to make them work reliably on other receivers. That was fun.