Teensy 4.1 NTP server

[ddrown]

I ported the embedded NTP stratum 1 server that I originally wrote for the stm32f407. It uses the 1588 ethernet hardware for RX and TX timestamps. I'm also using the 1588 event input capture to measure the reference PPS in hardware.

My first task was to get the clock synchronized with my reference GPS module's PPS. I'm using a PID controller to keep them in sync.




Once I have the local clock synchronized, I can then start serving the time to clients. I compared the teensy's clock to my stm32mp1-based NTP stratum 1 server. The green and blue lines represent the request and response latency, and the purple line is the offset between the two clocks. This is a 23 microsecond round trip time and an offset of 570 nanoseconds. There's a small change in the graph at the end where I turned on adjustments for the published latency figured for the PHY (105ns TX delay, 350ns RX delay).



This is better than my original platform of the stm32f407 (below), because the Teensy's crystal is much better.




Code is here: https://github.com/ddrown/teensy-ntp

And I'm using my own version of the Teensy lwip library, with 1588 timestamping support: https://github.com/ddrown/teensy41_ethernet

[ddrown]

I wanted to use the teensy_loader_cli but the soft reboot option wasn't working for me. I didn't dig very far into why it wasn't working, but I put together a workaround. If you send the character "r" to the serial, the teensy reboots into the bootloader.

static void bootloader_poll() {
  if(Serial.available()) {
    char r = Serial.read();
    if(r == 'r') {
      Serial.println("rebooting to bootloader");
      delay(10);
      asm("bkpt #251"); // run bootloader
    }
  }
}

[manitou]

Wow, nice work on the NTPD server using GPS and 1088. I haven't quite figured out all the components, but I boldly reconfigured my sparkfun GPS to run at 115200 baud and emit ZDA sentences. (I did manage to screw that up by getting the GPS into binary mode, then i had to find the binary data that would put it back in ASCII/NMEA mode ... sigh). I think it's running on my T4.1 with GPS Tx to pin 0 and GPS PPS to pin 35.

Ethernet 1588 NTP Server
------------------------

netif status changed: ip 192.168.1.19, mask 255.255.255.0, gw 192.168.1.1
waiting for link
enet link status: up
S 56627443 3802693683
81627251 0.000007680 0.000007680 0.000000000 8455 3802693684
106627058 -0.000001510 0.000007700 0.166664958 7555 3802693685
131626865 -0.000001107 0.000007720 0.750002265 7614 3802693686
156626672 -0.000001016 0.000007720 0.800003231 7622 3802693687
181626480 -0.000000960 0.000007710 0.593748987 7617 3802693688
206626287 -0.000000855 0.000007710 0.607158124 7626 3802693689
231626095 -0.000000803 0.000007707 0.875059485 7627 3802693690
256625902 -0.000000710 0.000007707 0.888899207 7636 3802693691
281625710 -0.000000668 0.000007704 1.199987769 7637 3802693692
...
  eticks       offset      drift       chisq   drift     secs
3338400888 -0.000000000 0.000007958 1.893251061 7958 3802771912
3388400490 0.000000006 0.000007958 1.517588139 7958 3802771914
3463399893 0.000000013 0.000007958 1.293152332 7959 3802771917
3538399296 0.000000018 0.000007958 1.258479357 7959 3802771920
3613398699 0.000000022 0.000007958 1.313638568 7960 3802771923
3688398102 0.000000025 0.000007958 1.611920714 7960 3802771926
3813397108 -0.000000013 0.000007958 1.605589628 7956 3802771931
3863396710 -0.000000004 0.000007958 1.631969333 7957 3802771933
3963395915 -0.000000030 0.000007958 2.179639101 7954 3802771937
...

I assume the 2nd column is offset between GPS clock and T41 clock, your first and third graphs above. How do i get data to reproduce your frequency offset plot. I have a GPS PPS sketch using the GPT timer that lets me measure T4.1 crystal drift for either 24MHz crystal or 32KHz RTC crystal, so I know what ppm to expect (-8.1 ppm).

Should your ntpd sketch not respond to NTP queries? I have a simple (old NTP v1) ubuntu C program that does an ntp query, but I get no reply from T4.1. I can ping the T4.1, and with tcpdump I can see the UDP ntp query packet go out on the wire, but no response. I haven't tried adding the T4.1 as an NTP server to /etc/ntp.conf on my ubuntu box.

Just an observation (not your problem), the sparkfun GPS is supposed to flash its LED with a good GPS lock, but with the T4.1 running the NTP/GPS server, it sometimes stops blinking for a while. The GPS is still emitting NMEA but it may be using its internal clock ? I actually had one mbed eval board (NUCLEO F446RE) that would never work with that GPS, because of high frequency interference maybe?? Any how it looks like the T4.1 running native ethernet affects GPS signal ???

i don't have a way to test 1088 with other boxes.

EDIT 1: i added the T41 host as a server to ntp.conf and that appears to be working

12:52:57.041744 IP 192.168.1.10.123 > 192.168.1.19.123: NTPv4, Client, length 48
12:52:57.041991 IP 192.168.1.19.123 > 192.168.1.10.123: NTPv4, Server, length 48
12:54:01.041860 IP 192.168.1.10.123 > 192.168.1.19.123: NTPv4, Client, length 48
12:54:01.041984 IP 192.168.1.19.123 > 192.168.1.10.123: NTPv4, Server, length 48

ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 manitou         .CDMA.           1 u   37   64   17    0.493    2.413   0.132
*192.168.1.19    .PPS.            1 u   37   64   17    0.200    2.423   0.182

Using sntp 192.168.1.19 to query the T41 works
2020-07-02 13:05:09.845665 (+0500) -0.001066 +/- 0.000726 192.168.1.19 s1 no-leap

EDIT 2:
offset of GPS clock to disciplined T4.1 clock (1088 ether clock, 25MHz)

[ddrown]

Wow, nice work on the NTPD server using GPS and 1088. I haven't quite figured out all the components, but I boldly reconfigured my sparkfun GPS to run at 115200 baud and emit ZDA sentences. (I did manage to screw that up by getting the GPS into binary mode, then i had to find the binary data that would put it back in ASCII/NMEA mode ... sigh). I think it's running on my T4.1 with GPS Tx to pin 0 and GPS PPS to pin 35.

Ethernet 1588 NTP Server
------------------------

netif status changed: ip 192.168.1.19, mask 255.255.255.0, gw 192.168.1.1
waiting for link
enet link status: up
S 56627443 3802693683
81627251 0.000007680 0.000007680 0.000000000 8455 3802693684
106627058 -0.000001510 0.000007700 0.166664958 7555 3802693685
131626865 -0.000001107 0.000007720 0.750002265 7614 3802693686
156626672 -0.000001016 0.000007720 0.800003231 7622 3802693687
181626480 -0.000000960 0.000007710 0.593748987 7617 3802693688
206626287 -0.000000855 0.000007710 0.607158124 7626 3802693689
231626095 -0.000000803 0.000007707 0.875059485 7627 3802693690
256625902 -0.000000710 0.000007707 0.888899207 7636 3802693691
281625710 -0.000000668 0.000007704 1.199987769 7637 3802693692
...

I assume the 2nd column is offset between GPS clock and T41 clock, your first and third graphs above. How do i get data to reproduce your frequency offset plot.

The columns from that output:

• T41 raw counter value at PPS (25MHz)
• seconds offset between GPS PPS and virtual T41 clock (which is the first graph). The third graph is the measurement over the network, which will combine that offset with the network measurement noise
• frequency measurement in seconds per second. multiply by 1 million to get parts per million (the frequency offset plot)
• chisq fit of frequency measurement
• used frequency, in parts per billion (includes phase adjustments to reduce the offset in column 2)
• seconds since 1900, NTP's timebase. Subtract by 2208988800 to get unix epoch time (seconds since 1970)

I have a GPS PPS sketch using the GPT timer that lets me measure T4.1 crystal drift for either 24MHz crystal or 32KHz RTC crystal, so I know what ppm to expect (-8.1 ppm).

Should your ntpd sketch not respond to NTP queries? I have a simple ubuntu C program that does an ntp query, but I get no reply from T4.1. I can ping the T4.1, and with tcpdump I can see the UDP ntp query packet go out on the wire, but no response. I haven't tried adding the T4.1 as an NTP server to /etc/ntp.conf on my ubuntu box.

It's very picky about what NTP queries it will respond to.

• NTP version must be either 3 or 4
• NTP mode must be client (3)
• UDP data length must be exactly 48 bytes long

If you show your tcpdump -vv output, I can help with that.

Just an observation (not your problem), the sparkfun GPS is supposed to flash its LED with a good GPS lock, but with the T4.1 running the NTP/GPS server, it sometimes stops blinking for a while. The GPS is still emitting NMEA but it may be using its internal clock ? I actually had one mbed eval board that would never work with that GPS, because of high frequency interference maybe?? Any how it looks like the T4.1 running native ethernet affects GPS signal ???

All sorts of things can interfere with GPS signal. It's a very weak signal when it gets to your receiver. The satellites overhead are moving very quickly, so if your antenna can only see part of the sky, there will be times that the module will lose lock and stop blinking till a new satellite moves into view.

For an example of noise, I have an SBC based on the rk3328 processor with USB3. When I plug something into that USB port, it makes a lot of RF noise around the GPS frequencies:



Normally the noise is much lower:

i don't have a way to test 1088 with other boxes.

I'm using chrony with hardware timestamps turned on to test NTP on my other systems. Not every NIC supports them, but my Intel NICs do. To check if you have support (the SOF_TIMESTAMPING_TX_HARDWARE and SOF_TIMESTAMPING_RX_HARDWARE are important):

$ sudo ethtool -T em1
Time stamping parameters for em1:
Capabilities:
        hardware-transmit     (SOF_TIMESTAMPING_TX_HARDWARE)
        software-transmit     (SOF_TIMESTAMPING_TX_SOFTWARE)
        hardware-receive      (SOF_TIMESTAMPING_RX_HARDWARE)
        software-receive      (SOF_TIMESTAMPING_RX_SOFTWARE)
        software-system-clock (SOF_TIMESTAMPING_SOFTWARE)
        hardware-raw-clock    (SOF_TIMESTAMPING_RAW_HARDWARE)
PTP Hardware Clock: 0
Hardware Transmit Timestamp Modes:
        off                   (HWTSTAMP_TX_OFF)
        on                    (HWTSTAMP_TX_ON)
Hardware Receive Filter Modes:
        none                  (HWTSTAMP_FILTER_NONE)
        all                   (HWTSTAMP_FILTER_ALL)
        ptpv1-l4-sync         (HWTSTAMP_FILTER_PTP_V1_L4_SYNC)
        ptpv1-l4-delay-req    (HWTSTAMP_FILTER_PTP_V1_L4_DELAY_REQ)
        ptpv2-l4-sync         (HWTSTAMP_FILTER_PTP_V2_L4_SYNC)
        ptpv2-l4-delay-req    (HWTSTAMP_FILTER_PTP_V2_L4_DELAY_REQ)
        ptpv2-l2-sync         (HWTSTAMP_FILTER_PTP_V2_L2_SYNC)
        ptpv2-l2-delay-req    (HWTSTAMP_FILTER_PTP_V2_L2_DELAY_REQ)
        ptpv2-event           (HWTSTAMP_FILTER_PTP_V2_EVENT)
        ptpv2-sync            (HWTSTAMP_FILTER_PTP_V2_SYNC)
        ptpv2-delay-req       (HWTSTAMP_FILTER_PTP_V2_DELAY_REQ)

[ddrown]

I setup a second teensy with an Adafruit GPS. I added the GPS_USES_RMC define to settings.h to support it. Originally I took the timestamp of the GPRMC message, but it would often be delayed into the next second, which caused the software to think the clock was off by 1s. So now it's taking the timestamp of the GPGGA message and uses that instead.



I put that change into place at around 01-00:30, and you can see the difference it made on the yellow line in the graph. The two Teensies agree on the time pretty closely. The Archmax's measurement over the network has the most noise, and the clock "cheese" has a static offset of around 500ns.

[manitou]

I also experimented with Adafruit ultimate GPS. I found how to enable ZDA sentences and changed baud to 115200, BUT when I cycle power on the Adafruit GPS, it reverts back to 9600, and ZDA is no longer enabled. Of course, after program upload it will retain the 115200/ZDA setting, so additional code would have to be added to NTP sketch to start in 9600 and then enable ZDA and set to 115200 , then Serial1.end() and then start up Serial1 again at 115200. Messy, plus I observed the same intermittent GPS lock with the T4.1 running the NTP sketch, and I never could get all those Serial1 settings to work (LAG errors or B errors). T4.1 and adafruit GPS does OK with simple GPS sketches. My GPS is just sitting in a bedroom and no external antenna. The adafruit has an option for an external antenna and that's why i decided to experiment with it. I'll probably order the external antenna.

So were your adafruit tests at 9600 and no ZDA?

I was getting sub-microsecond offsets (i added data to my post #3) with the sparkfun GPS, but there were spikes when GPS lock was lost. I looked with scope and ZDA message comes about 340 ms after PPS on sparkfun

[ddrown]

I also experimented with Adafruit ultimate GPS. I found how to enable ZDA sentences and changed baud to 115200, BUT when I cycle power on the Adafruit GPS, it reverts back to 9600, and ZDA is no longer enabled. Of course, after program upload it will retain the 115200/ZDA setting, so additional code would have to be added to NTP sketch to start in 9600 and then enable ZDA and set to 115200 , then Serial1.end() and then start up Serial1 again at 115200. Messy, plus I observed the same intermittent GPS lock with the T4.1 running the NTP sketch, and I never could get all those Serial1 settings to work (LAG errors or B errors). T4.1 and adafruit GPS does OK with simple GPS sketches. My GPS is just sitting in a bedroom and no external antenna. The adafruit has an option for an external antenna and that's why i decided to experiment with it. I'll probably order the external antenna.

So were your adafruit tests at 9600 and no ZDA?

Yes, I'm using the firmware defaults for the adafruit GPS, which is 9600 and no ZDA message. I'm also using an external antenna on a windowsill. Without an external antenna, I get bad reception.

I was getting sub-microsecond offsets (i added data to my post #3) with the sparkfun GPS, but there were spikes when GPS lock was lost. I looked with scope and ZDA message comes about 340 ms after PPS on sparkfun

Yeah, 300ms ish is pretty typical. As long as it is within 950ms, this code will accept it.

I haven't done anything special for hold-over while there's no signal, so I'd expect it to drift in those cases.

My measurements are on a system that's also synchronized to GPS directly, so that improves the measurements as well.

[ddrown]

The RMC message taking more than 1s would generate those lag error messages. The LAG messages can also be caused by no pps due to signal lock loss.

I'm surprised you're getting B (bad time) though. Is your GPS module resetting itself somehow?

[manitou]

The RMC message taking more than 1s would generate those lag error messages. The LAG messages can also be caused by no pps due to signal lock loss.

I'm surprised you're getting B (bad time) though. Is your GPS module resetting itself somehow?

I think i only got B message in one experiment trying to get ZDA and 115200 work on adafruit GPS ... not to worry.

I added an offset and drift plot to post #3

[JBeale]

This is amazing work, thanks for posting this! I noticed the 1588 hardware features listed for the T4.1, and I wondered if they would be supported in some way.
On the first graph in your first post "Phase offset: Teensy vs PPS" ...if you are taking a PPS signal direct from a GPS, especially with a non-ideal antenna placement, is it possible most of that jitter is actually from the GPS PPS output granularity plus its receiver slipping in/out of phase lock as each sat signal fades in/out? I assume it might look quieter if you took the reference PPS from a stable GPSDO ?

[ddrown]

The phase offsets happen at the same time as the frequency changes, which has me thinking that my synchronization code needs some tweaking. I suspect PPS from a GPSDO wouldn't change much unless the system clock also came from the GPSDO.

[JBeale]

I would expect almost any un-corrected crystal oscillator to have better short term jitter than a raw GPS PPS signal, although of course worse over some longer term, but when the oscillator is controlled in a PLL then I don't know. I have a GPSDO fed from a roof-mount antenna, and a Teensy 4.1 and am curious about this. To try it out I guess I also need some kind of ethernet adaptor for the T4.1; anything else?

[ddrown]

Sure, the short term phase noise from the crystal will be much lower than the GPS. But I think what is causing the larger offsets are the frequency change due to temperature changes. It would be worth testing covering the teensy's crystal to lower the convection. Looks like it normally wanders around 100 parts per billion over an hour in my setup.

If you only want to test the offsets, you don't even need Ethernet. Just pps and nmea on the serial.

[manitou]

Sure, the short term phase noise from the crystal will be much lower than the GPS. But I think what is causing the larger offsets are the frequency change due to temperature changes. It would be worth testing covering the teensy's crystal to lower the convection. Looks like it normally wanders around 100 parts per billion over an hour in my setup.

If you only want to test the offsets, you don't even need Ethernet. Just pps and nmea on the serial.

You could print T4 temperature (tempmonGetTemp() float degrees C) in your table printing in updateTime() to track drift and temperature.

Here is diurnal NTP drift variation (ppm) for my home Dell desktop (Ubuntu with CDMA time source) over 5 days


Peaks are at about 4 in the afternoon.

[manitou]

Yes, I'm using the firmware defaults for the adafruit GPS, which is 9600 and no ZDA message. I'm also using an external antenna on a windowsill. Without an external antenna, I get bad reception.

I purchased active antenna and placed it on windowsill attached to my adafruit ultimate GPS, and i no longer lose lock. Settings were 9600 with GPS_USES_RMC. offset/drift look good with GPS PPS and disciplined T4.1 clock and 1088 NTP server. I also tested a u-blox NEO-M8U GPS with the external antenna and the 1088 NTP server. I added T4 temperature to table output. Offset and drift plots below


On u-blox GPS I enabled ZDA, disabled some other NMEA sentences, and set baud to 115200. Fix with 12 satellites and HDOP 0.8

Below is drift during early morning hours probably showing 17 minute HVAC cycles. Used u-blox PPS interrupts with T4.0 GPT @ 150MHz.

38553 secs 149998303 ticks  -11.313  -11.319 ppm  58.8 C
38554 secs 149998303 ticks  -11.313  -11.319 ppm  58.8 C
38555 secs 149998303 ticks  -11.313  -11.319 ppm  58.8 C
38556 secs 149998306 ticks  -11.293  -11.319 ppm  58.8 C
38557 secs 149998303 ticks  -11.313  -11.319 ppm  58.8 C
38558 secs 149998304 ticks  -11.307  -11.319 ppm  58.8 C

I can't really see the temperature cycles in the MCU temperature data, but a separate I2C temperature sensor (Si7021) does reveal the HVAC cycles. Roughly if temperature changes 1.0 C then MCU frequency changes 0.067 ppm

Here are two plots showing change in drift versus temperature. Samples are about every 3 seconds.



(The active antenna also fixed the GPS signal problems that I was seeing on mbed board)

[ddrown]

I purchased active antenna and placed it on windowsill attached to my adafruit ultimate GPS, and i no longer lose lock. Settings were 9600 with GPS_USES_RMC. offset/drift look good with GPS PPS and disciplined T4.1 clock and 1088 NTP server. I also tested a u-blox NEO-M8U GPS with the external antenna and the 1088 NTP server. I added T4 temperature to table output. Offset and drift plots below

That looks good, thanks for trying it out!

[ddrown]

You could print T4 temperature (tempmonGetTemp() float degrees C) in your table printing in updateTime() to track drift and temperature.

I've added this line to updateTime:
Serial.printf("PPS: %u %u %.2f\r\n", gpstime, lastPPS, tempmonGetTemp());

I've collected this type of data in the past for other crystals, so I'd like to redo the plots below for the teensy:

32khz RTC crystal:


12MHz TCXO:

[ddrown]

I let this run for 5 days, and these are interesting results. I have two different Teensy 4.1 boards running this code.

Temperature vs frequency over 256 second intervals:




Just using a linear fit gives a pretty good result for this temperature range. If the room these are in had a larger temperature range, a quadratic equation might fit better.

They're within a foot of each other and running the same code, but one is showing +10C from the other. They're plugged into different USB hosts, so maybe some heat is coming from the hosts. But I also assume there's some temperature measurement offset error from tempmonGetTemp() that varies from teensy to teensy.

This suggests that adding frequency estimation from temperature measurements to the control loop might be worthwhile. At the very least, it'd be good for keeping the time while there's no PPS from the GPS.

[JBeale]

Is it correct to say that this code acts as a PTP server only, and not a client? For example, if I have two T4.1 boards and I want board #2 to receive time over a LAN from board #1 and blink a LED at the exact top of each NTP second, is that possible?

To get up to speed with this, first I tried doing this with two Raspberry Pi boards since the NTP client and server software is easily available (I'm using 'chrony') and I found with my setup that a 1-PPS signal output from a Raspberry Pi based NTP LAN client has a 150 usec offset and about 10 usec RMS jitter relative to the original GPS PPS input to the NTP server. That is using purely software sync on both client and server, running non-realtime Linux. With hardware timestamping (and no OS) I expect a system using Teensy 4.1 on both ends could do rather better, and I'm curious to find out how much better. I see there is something called "UdpNtpClient" in my Arduino "Ethernet" library folder but it simply sends one NTP packet, waits for a reply for 1 second (!) and then displays the previously received time and date. I think what I was looking for was the time (ns) offset and frequency (ppm) offset between some running T4.1 timer, and NTP / UTC time, so I could use that to schedule some future I/O event accurately aligned to NTP time.

  sendNTPpacket(timeServer); // send an NTP packet to a time server
  // wait to see if a reply is available
  delay(1000);
  if (Udp.parsePacket()) { ...

[ddrown]

Is it correct to say that this code acts as a PTP server only, and not a client? For example, if I have two T4.1 boards and I want board #2 to receive time over a LAN from board #1 and blink a LED at the exact top of each NTP second, is that possible?

This code is only built as an NTP (not PTP) server. But I do have different code for the esp8266: https://github.com/ddrown/esp8266-clock

It uses an NTP client to keep the local clock in sync, and displays the time on a SPI based display. Wifi has a lot of extra latency and jitter, but I was still able to keep that clock within +/-10ms. A teensy on Ethernet would be able to do better than that, but would need some work porting the code over.

To get up to speed with this, first I tried doing this with two Raspberry Pi boards since the NTP client and server software is easily available (I'm using 'chrony') and I found with my setup that a 1-PPS signal output from a Raspberry Pi based NTP LAN client has a 150 usec offset and about 10 usec RMS jitter relative to the original GPS PPS input to the NTP server. That is using purely software sync on both client and server, running non-realtime Linux. With hardware timestamping (and no OS) I expect a system using Teensy 4.1 on both ends could do rather better, and I'm curious to find out how much better.

150us seems like the right ballpark for offset on a rpi. Unfortunately, the rpi does not have hardware timestamps for PPS capture or for Ethernet packets timestamps. So interrupt latency and jitter will be the largest sources of timing error on it.

I have some other systems with timestamp hardware on them, and these are my results: https://blog.dan.drown.org/stm32mp1-ntp-server-part-3/ The summary: "The median offset between these two systems is now 41ns! Standard deviation is 87ns, min -503ns max +416ns."

I see there is something called "UdpNtpClient" in my Arduino "Ethernet" library folder but it simply sends one NTP packet, waits for a reply for 1 second (!) and then displays the previously received time and date. I think what I was looking for was the time (ns) offset and frequency (ppm) offset between some running T4.1 timer, and NTP / UTC time, so I could use that to schedule some future I/O event accurately aligned to NTP time.

  sendNTPpacket(timeServer); // send an NTP packet to a time server
  // wait to see if a reply is available
  delay(1000);
  if (Udp.parsePacket()) { ...

Yup, the Arduino NTP client is very basic and is lacking in features.

[JBeale]

I've got a start, at least. The code runs on my T4.1, and ping to it from another host shows a smaller RTT than anything else on my LAN (avg = 0.186 msec). I have a Trimble 57964-80 GPSDO with roof antenna, providing a stable PPS signal. Unfortunately, that model speaks its own UCCM protocol, and does not have any NMEA mode. Therefore I am also using an Adafruit Ultimate GPS module for the serial NMEA data. I see the T4.1 monitor window updates every 3rd second. The second column (offset) stays fairly small, but the 4th column (chi-square) bounces around a lot. I don't know what values to expect for that.

  eticks      offset     drift     chi-sq   drift(ppb)   secs
472272546 0.000000054 0.000006338 2.233723879 6342 3805069694
547272070 0.000000075 0.000006338 3.055933952 6344 3805069697
647271435 0.000000104 0.000006339 6.246305466 6348 3805069701
722270959 0.000000105 0.000006340 8.608863831 6350 3805069704
797270483 0.000000097 0.000006340 12.498611450 6349 3805069707
847270166 0.000000079 0.000006341 12.661604881 6349 3805069709
972269373 0.000000055 0.000006343 12.250881195 6348 3805069714
1047268898 0.000000012 0.000006343 10.769866943 6344 3805069717
1097268582 -0.000000035 0.000006343 8.412799835 6340 3805069719
1197267949 -0.000000079 0.000006343 11.882316589 6335 3805069723
1272267474 -0.000000083 0.000006343 18.039707184 6335 3805069726
1322267158 -0.000000112 0.000006342 24.708629608 6330 3805069728
1422266525 -0.000000116 0.000006341 34.249813080 6329 3805069732
1497266050 -0.000000102 0.000006339 38.252040863 6328 3805069735
1572265575 -0.000000085 0.000006337 37.948085785 6328 3805069738
1622265258 -0.000000061 0.000006335 33.235595703 6328 3805069740
1697264783 -0.000000044 0.000006333 26.158643723 6328 3805069743
1797264150 -0.000000035 0.000006333 12.825462341 6329 3805069747
1822263992 -0.000000044 0.000006332 7.780921936 6327 3805069748
1897263517 -0.000000028 0.000006332 4.204079628 6328 3805069751
1972263042 -0.000000009 0.000006331 2.795843363 6329 3805069754
2072262409 -0.000000002 0.000006331 1.701416016 6330 3805069758
2172261776 -0.000000000 0.000006331 1.505235434 6330 3805069762
2222261460 -0.000000020 0.000006331 1.326510429 6328 3805069764
2322260827 -0.000000013 0.000006331 1.661483765 6329 3805069768
2372260511 -0.000000030 0.000006331 2.637467384 6327 3805069770
2447260037 -0.000000053 0.000006331 5.042006493 6324 3805069773
2522259563 -0.000000067 0.000006330 7.685133934 6322 3805069776
2597259090 -0.000000114 0.000006329 15.834042549 6317 3805069779
2672258616 -0.000000109 0.000006328 19.417743683 6316 3805069782
2747258143 -0.000000138 0.000006327 26.917015076 6312 3805069785
2822257670 -0.000000157 0.000006324 33.382953644 6307 3805069788
2922257039 -0.000000149 0.000006322 41.053100586 6306 3805069792
2972256723 -0.000000120 0.000006320 42.935028076 6306 3805069794
3047256249 -0.000000078 0.000006319 34.806823730 6310 3805069797
3122255775 -0.000000043 0.000006317 25.022006989 6311 3805069800
3222255144 -0.000000045 0.000006316 15.631460190 6310 3805069804
3272254828 -0.000000025 0.000006315 9.222945213 6311 3805069806
3347254354 0.000000004 0.000006315 6.383878708 6313 3805069809
3422253880 0.000000027 0.000006315 4.052414894 6316 3805069812
3522253249 0.000000007 0.000006314 4.030448914 6314 3805069816
3597252776 -0.000000014 0.000006314 4.060725212 6311 3805069819
3647252460 0.000000004 0.000006314 3.975740910 6313 3805069821
3722251987 -0.000000012 0.000006315 3.571488857 6312 3805069824

[ddrown]

I've got a start, at least. The code runs on my T4.1, and ping to it from another host shows a smaller RTT than anything else on my LAN (avg = 0.186 msec). I have a Trimble 57964-80 GPSDO with roof antenna, providing a stable PPS signal. Unfortunately, that model speaks its own UCCM protocol, and does not have any NMEA mode. Therefore I am also using an Adafruit Ultimate GPS module for the serial NMEA data. I see the T4.1 monitor window updates every 3rd second. The second column (offset) stays fairly small, but the 4th column (chi-square) bounces around a lot. I don't know what values to expect for that.

chisquare values are in counts (25MHz) so 42.9 counts = 1.7us. Those values look reasonable to me.

I'm considering dropping old samples when the chisq value is high. This usually happens when the local clock is changing frequencies quickly.

For instance, when this dropped from 8600ppb to 8532ppb in about 30 seconds, the chisq values went up quickly:

count       offset      frequency   chisq         ppb  ntpseconds
4070505216 -0.000000268 0.000008600 101.986709595 8573 3804156066
4145504574 -0.000000309 0.000008599 184.083648682 8567 3804156069
4220503934 -0.000000410 0.000008596 285.031646729 8554 3804156072
535999 -0.000000514 0.000008589 369.453765869 8536 3804156075
75535361 -0.000000605 0.000008582 493.047149658 8519 3804156078
150534724 -0.000000685 0.000008573 628.721679687 8502 3804156081
225534088 -0.000000753 0.000008562 759.418212891 8483 3804156084
300533453 -0.000000806 0.000008554 924.698669434 8469 3804156087
400532607 -0.000000844 0.000008544 1062.952758789 8454 3804156091
475531973 -0.000000834 0.000008532 1200.365600586 8442 3804156094

but this clock has a mean chisq value of 11.8:

$ grep '^[0-9]' pps.log | stats -f4
          Field:  4
          lines:  319041
           mean:  11.853469343780
       variance:  479.423481311380
        std dev:  21.895741168350
            sum:  3781742.712908904534
            min:  -0.000000105000
            max:  1200.365600586000

[JBeale]

Thank you for that reply. Plotting the drift and chi-sq values over two hours, I see the drift is nothing like the random-walk noise I expected. Instead it is dominated by sudden dramatic shifts, as if my GPSDO has far too little low-pass filtering to smooth out the timing shifts as individual satellites come into and out of view. At any rate it is a very interesting result, so I will have a look into that next.

[ddrown]

Thank you for that reply. Plotting the drift and chi-sq values over two hours, I see the drift is nothing like the random-walk noise I expected. Instead it is dominated by sudden dramatic shifts, as if my GPSDO has far too little low-pass filtering to smooth out the timing shifts as individual satellites come into and out of view. At any rate it is a very interesting result, so I will have a look into that next.

I'm curious why you assume the frequency changes are coming from the GPSDO and not from the Teensy's crystal. Could you explain why you suspect the one but not the other?

[JBeale]

I have done no investigation yet, so I certainly don't know the cause of that behavior. Normally you expect a GPSDO to be much more stable than a simple SMT oscillator that isn't even temperature stabilized. I'd expect the main contribution to frequency drift for a simple crystal source to be temperature, and I have a ballpark idea of how much ppm change to expect from that, what the peak rate of change should be, and what plots of drift rate generally look like over time. For a span of a few hours, they should look similar to the ambient temperature vs time, unless you've got real problems with your power supply stability. That plot is unexpected based on what I know about how the ambient temperature changes in my office and what I've measured before with frequency drift of various boards, so I'm interested to find out what's going on. I'm sure I will learn something.