Teensy 4.1 JackTrip client, sync issues

hatchjaw

Member
Hello,

I'm working on setting up multiple Teensy 4.1's as JackTrip clients for audio spatialisation (e.g. distributed WFS). Using NativeEthernet, I have a collection of ethernet-shield-equipped Teensies all receiving UDP packets of encoded audio from a JackTrip hub server running on a laptop, and sending packets back to the server. All that's required for this to work is that the sample rate and buffer size reported in the JackTrip packet headers match between server and client(s). So far so good.

Not so good, is trying to keep these clients in sync with the server. My JackTripClient object inherits from AudioStream (and EthernetUDP), and its update method is as follows:

Code:
void JackTripClient::update(void) {
#ifdef USE_TIMER
    // If using TeensyTimer, just do audio output here, as the timer's callback calls updateImpl()
    doAudioOutput();  
#else
    updateImpl();
#endif
}

void JackTripClient::updateImpl() {\
    // Read as many UDP packets as are available right now; place the contents in a circular buffer
    receivePackets(); 
#ifndef USE_TIMER
    // Copy the latest unread packet from the circular buffer to this object's audio outputs
    doAudioOutput(); 
#endif
    // Get the current audio input block, put it into a UDP packet and send that to the server
    sendPacket(); 

    if (showStats && connected) {
        packetStats.printStats();
        udpBuffer.printStats();
    }
}

As the name suggests receivePackets() sometimes finds more than one packet. That, plus sendPacket(), and doAudioOutput() look like this:

Code:
void JackTripClient::receivePackets() {
    if (!connected) return;

    int size;

    // Check for incoming UDP packets. Get as many packets as are available.
    while ((size = parsePacket()) > 0) {
        lastReceive = 0;

        if (size == EXIT_PACKET_SIZE && isExitPacket()) {
            // Exit sequence
            Serial.println("JackTripClient: Received exit packet");
            Serial.printf("  maxmem: %d blocks\n", AudioMemoryUsageMax());
            Serial.printf("  maxcpu: %f %%\n\n", AudioProcessorUsageMax());

            stop();
            return;
        } else if (size != UDP_PACKET_SIZE) {
            Serial.println("JackTripClient: Received a malformed packet");
        } else {
            // Read the UDP packet and write it into a circular buffer.
            uint8_t in[size];
            read(in, size);
            udpBuffer.write(in, size);

            // Read the header from the packet received from the server.
            memcpy(serverHeader, in, sizeof(JackTripPacketHeader));

            if (showStats && packetStats.awaitingFirstReceive()) {
                Serial.println("===============================================================");
                Serial.printf("Received first packet: Timestamp: %" PRIu64 "; SeqNumber: %" PRIu16 "\n",
                              serverHeader->TimeStamp,
                              serverHeader->SeqNumber);
                packetHeader.TimeStamp = serverHeader->TimeStamp;
                packetHeader.SeqNumber = serverHeader->SeqNumber;
                Serial.println("===============================================================");
            }

            packetStats.registerReceive(*serverHeader);
        }
    }

    if (lastReceive > RECEIVE_TIMEOUT_MS) {
        Serial.printf("JackTripClient: Nothing received for %.1f s. Stopping.\n", RECEIVE_TIMEOUT_MS / 1000.f);
        stop();
    }
}

void JackTripClient::sendPacket() {
    // Might have received an exit packet, so check whether still connected.
    if (!connected) return;

    // Get the location in the UDP buffer to which audio samples should be
    // written.
    uint8_t packet[UDP_PACKET_SIZE];
    uint8_t *pos = packet + PACKET_HEADER_SIZE;

    // Copy audio to the UDP buffer.
    audio_block_t *inBlock[NUM_CHANNELS];
    for (int channel = 0; channel < NUM_CHANNELS; channel++) {
        inBlock[channel] = receiveReadOnly(channel);
        // Only proceed if a block was returned, i.e. something is connected
        // to one of this object's input channels.
        if (inBlock[channel]) {
            memcpy(pos, inBlock[channel]->data, CHANNEL_FRAME_SIZE);
            pos += CHANNEL_FRAME_SIZE;
            release(inBlock[channel]);
        }
    }

    packetHeader.SeqNumber++;
    packetHeader.TimeStamp += packetInterval;
    packetInterval = 0;

    // Copy the packet header to the UDP buffer.
    memcpy(packet, &packetHeader, PACKET_HEADER_SIZE);

    // Send the packet.
    beginPacket(serverIP, serverUdpPort);
    size_t written = write(packet, UDP_PACKET_SIZE);
    if (written != UDP_PACKET_SIZE) {
        Serial.println("JackTripClient: Net buffer is too small");
    }
    auto result = endPacket();
    if (0 == result) {
        Serial.println("JackTripClient: failed to send a packet.");
    }

    packetStats.registerSend(packetHeader);
}

void JackTripClient::doAudioOutput() {
    if (!connected) return;

    // Copy from UDP inBuffer to audio output.
    // Write samples to output.
    audio_block_t *outBlock[NUM_CHANNELS];
    uint8_t data[UDP_PACKET_SIZE];
    // Read a packet from the input UDP buffer
    udpBuffer.read(data, UDP_PACKET_SIZE);
    for (int channel = 0; channel < NUM_CHANNELS; channel++) {
        outBlock[channel] = allocate();
        // Only proceed if an audio block was allocated, i.e. the
        // current output channel is connected to something.
        if (outBlock[channel]) {
            // Get the start of the sample data in the packet. Cast to desired
            // bit-resolution.
            auto start = (const int16_t *) (data + PACKET_HEADER_SIZE + CHANNEL_FRAME_SIZE * channel);
            // Copy the samples to the output block.
            memcpy(outBlock[channel]->data, start, CHANNEL_FRAME_SIZE);
            // Finish up.
            transmit(outBlock[channel], channel);
            release(outBlock[channel]);
        }
    }
}

Using `if` instead of `while` in receivePackets() means the client drops a packet every so often. The time between dropouts varies a lot, perhaps more related to conditions on the server rather than the Teensies. Using `while`, however, means that the number of writes to the circular buffer tends to overtake the number of reads from it, thus there's occasional distortion as unread UDP bytes are overwritten before they can be used for output. So I have the peculiar situation where despite the server and Teensies reportedly running at 44.1 KHz (with a buffer size of 32, so UDP packets being exchange roughly every 726 µs), Teensy is receiving more packets from the server than it is sending. Indeed, JackTrip reports (via its -I flag) positive skew, i.e. the client falling behind the server.

Thinking that maybe all that UDP activitiy in update() (on an audio hardware interrupt, if I understand correctly) might be slowing Teensy down, I experimented with using TeensyTimerTool (see `USE_TIMER` above) to move the UDP reads/writes out of update() to a callback using GPT1; this tends to work for a few seconds, but eventually packet read/writes stop -- perhaps communicating with the ethernet shield on this sort of basis isn't viable. Might something less generic work, as per https://www.pjrc.com/teensy/interrupts.html? For sending, I'd need to be trying to call AudioStream::receiveReadOnly() via something other than the audio interrupt, which I sense wouldn't be straightforward.

I feel like synchorisation is a thorny issue, plus I'm a little out of my depth with the lower-level aspects of what I'm trying to achieve. That said, I'd be grateful to get people's views on:

  • whether setting up a hardware interrupt for the UDP reads/writes colud be viable (preferably checking every 50 µs or so for new packets)
  • and whether TeensyTimerTool could actually serve this purpose
  • replacing NativeEthernet with QNEthernet -- might this help with the fairly intensive network stuff I'm trying to achieve?
  • any other thoughts around synchronisation -- external clocking (NTP), nudging PLL4?

My hope is to get things as good as they can be while keeping the project accessible (so no PTP), interoperable (probably leaving Teensy's audio clock alone), then have the Teensies follow the jacktrip server, resampling audio buffers as necessary to keep things more-or-less in sync.

The full code lives on a development branch here.

Thanks in advance,

Tommy
 
Try this, I'm curious if it will work for you:
1. Change to QNEthernet
2. Change that `while` loop to an `if` (receivePackets())
3. Use the UDP packet buffering feature introduced in QNEthernet v0.16.0: you can pass an optional queue size to the EthernetUDP constructor, minimum 1. Play with some values here, for example 8 or something.

Note: Unfortunately, I made the `EthernetUDP` class final, so you'll have to use composition instead of inheritance. I'll look into removing `final` for the next release, if I think it's appropriate. (Or you can just remove the `final` keyword from the source and see if that works for you.)
Note: There's no need to specify a MAC address with QNEthernet (but you can if you really need to choose your own).
Note: You can use listeners to watch for link state and address changes, too.
Note: I’m not 100% certain how the library handles being called from an ISR.

This will enable internal receive buffering but without forcing the program to miss sending data because it now only processes one input packet at a time.
 
Last edited:
Thank you very much for your reply, shawn. I was looking at making the switch to QNEthernet prior to posting, and the packet-buffering feature looks like it'll come in very handy.

There's no need to specify a MAC address with QNEthernet (but you can if you really need to choose your own).

As you may have noticed, I've been using TeensyID to generate a MAC address; great that QNEthernet does the same thing under the hood.

I’m not 100% certain how the library handles being called from an ISR.

If you could take a guess at what complications might arise, I'd be very grateful! I feel like separating concerns with regard to audio and networking would be ideal, however if it emerges that doing everything from AudioStream::update() is the only viable approach then at least I know and can proceed with a degree of confidence.
 
The latest push makes EthernetClient, EthernetServer, and EthernetUDP non-final so you can derive from them.
 
Thanks shawn. Is that under a new tag, or just on the main branch?

There's every possibility that I'm doing something wrong, but so far QNEthernet is behaving quite unstably for me. In my JackTripClient::begin() method, I have:

Code:
uint8_t JackTripClient::begin(uint16_t port) {
    qn::Ethernet.onLinkState([this, port](bool state){
        Serial.printf("[Ethernet] Link %s\n", state ? "ON" : "OFF");

        if (state) {
            Serial.print("JackTripClient: IP is ");
            Serial.println(qn::Ethernet.localIP());

            Serial.printf("JackTripClient: Packet size is %d bytes\n", UDP_PACKET_SIZE);

            EthernetUDP::begin(port);
        }
    });

    if (!startEthernet()) {
        Serial.println("JackTripClient: failed to start ethernet connection.");
        return 0;
    }

    return 1;
}

bool JackTripClient::startEthernet() {
    qn::Ethernet.macAddress(clientMAC);

    Serial.print("JackTripClient: MAC address is: ");
    for (int i = 0; i < 6; ++i) {
        Serial.printf(i < 5 ? "%02X:" : "%02X", clientMAC[i]);
    }
    Serial.println();

    // Use the last byte of the MAC to set the last byte of the IP.
    // (May need a more sophisticated approach...)
    clientIP[3] += clientMAC[5];
    return qn::Ethernet.begin(clientIP, INADDR_NONE, INADDR_NONE);
}

And occasionally I see:


[Ethernet] Link OFF
JackTripClient: failed to send a packet
[Ethernet] Link ON


In JackTripClient::sendPacket() I'm calling qn::EthernetUDP::send(), so it seems it's returning false precisely once between the link being broken and re-established. This isn't a problem I encounter with NativeEthernet, with which I use beginPacket()/write()/endPacket() and have yet to witness a packet fail to send i.e. endPacket() returning 0 (of course this may have happened while I was not montioring serial output!).

I'm also finding that sometimes* a client connects to the jacktrip server and after a short while reports that it isn't receiving packets, which happens when parsePacket() returns 0 (or less) consistently for five seconds. Further investigation reveals it's returning -1, which I guess means either EthernetUDP's pcb_ is null or inBufSize_ is 0. I'm using an input queue size of 8 as you suggested; is there more I should do to effectively take advantage of the input queue? If you have any other suggestions for ways to mitigate (or investigate) the instability I'm seeing they'd be very much appreciated.

* and at other times it seems a client will sit and happily consume packets for as long as I'm willing to stare at serial output.
 
Ethernet only needs to be started once, and not from within the listener. Starting Ethernet is what calls the listeners; not the other way around. Can I see your whole program? Or at least the parts (whole files) that use Ethernet. A gist or GitHub or whatever link is fine too.

Since zero-length UDP packets are valid, parsePacket() returns < 0 if there’s no packet waiting. >=0 means there’s a packet waiting to be read. (Note that this differs slightly from how Arduino and other libraries do this.)

You also need to set the subnet mask and, I suggest, the gateway too.
 
Last edited:
What happens if you just do a direct conversion of your NativeEthernet code to QNEthernet, using only the small changes I suggest in post #2?
 
Thanks very much for following up on this, shawn. And apologies for not providing full code above -- I'll link to a gist at the bottom of this post. I'm juggling branches a bit at the moment, working on versions with NativeEthernet and QNEthernet in parallel, while also working on an app to send WFS parameters over OSC.

So, weirldy, what seems to be happening is:

- about two thirds of the time, when I boot Teensy it connects to the jacktrip server briefly but then reports that it isn't receiving packets.
- jacktrip also reports that it isn't receiving packets from the client.
- the remaining ~third of the time, Teensy stays connected to the jacktrip server, but after a few seconds the top level loop() function stops being called.

What happens if you just do a direct conversion of your NativeEthernet code to QNEthernet, using only the small changes I suggest in post #2?

The absolute most basic conversion results in an immediate failure, since at the end of JackTripClient::begin() I'm calling the following:

Code:
qn::Ethernet.begin(clientMAC, clientIP);

if (qn::Ethernet.linkStatus() != qn::EthernetLinkStatus::LinkON) {
    Serial.println("JackTripClient: Ethernet cable is not connected.");
} else {
    Serial.println("JackTripClient: Ethernet connected.");
}

return qn::Ethernet.linkStatus();

Which is fine for NativeEthernet, but with QNEthernet the link status doesn't update immediately (hence the onLinkState callback I suppose).

Switching to a non-deprecated version of qn::Ethernet.begin, and using the netmask and gateway I've specified in IPv4 config for the ethernet interface...
Code:
qn::Ethernet.begin(clientIP, {255, 255, 255, 0}, {192, 168, 10, 1});
...Teensy establishes a connection with the jacktrip server, but I see similar behaviour to that which I describe at the beginning of this post. Plus occasionally Teensy reports that it failed to send a packet, presumably because the ethernet link was momentarily broken.

Here's a gist of the minimal QNEthernet conversion. As ever, any suggestions would be gratefully received!
 
What about waiting for the link? Then it won’t die immediately. There’s a convenience Ethernet.waitForLink() function where you can specify a timeout, and it returns whether it was successful or not. When I have a chance, I will look at the code.

Note: in QNEthernet, link and other things that need polling are polled at an interval of about 125ms.
Note: polling doesn’t start until Ethernet is started. (For the vast majority of cases, you only need to start this once.)
Note: to change static IP, you can call Ethernet.setLocalIP(), setSubnetMask(), etc.
 
Last edited:
Ah, thanks for the tip re. waitForLink()! Gist updated accordingly.

As I mentioned above, it seems that after a while (and this while varies) top level loop() stops being called. Under such conditions, AudioSteam::update() continues to be called, however. I'm testing with two Teensy 4.1's and both are affected, so I think it's probably not a hardware issue. Some sort of interrupt conflict? Though I'm not currently using TeensyTimerTool, it's still part of the codebase, so that got me thinking... but judicious use of preprocessor directives to eliminate it and that particular issue persists.

Note: in QNEthernet, link and other things that need polling are polled at an interval of about 125ms.

I see that's in the call to EthernetClass::loop(), which is called from EthernetUDP::parsePacket(). Could calling parsePacket() at intervals of 0.7/1.4 ms (32/64 samples) be causing a problem?

Note: polling doesn’t start until Ethernet is started. (For the vast majority of cases, you only need to start this once.)

I'm calling Ethernet::begin(...) once, from JackTripClient::begin(). I've been using a private startEthernet() method to separate that operation, but I recently moved that back into begin().

Note: to change static IP, you can call Ethernet.setLocalIP(), setSubnetMask(), etc.

Does this not happen anyway when I call Ethernet::begin(...) with an IP address, netmask and gateway?
 
Calling parsePacket() often shouldn’t be much of a problem.

Yes. Those are useful after Ethernet has started if all you need to do is change the local address.
 
Back
Top