Hello,
I'm working on setting up multiple Teensy 4.1's as JackTrip clients for audio spatialisation (e.g. distributed WFS). Using NativeEthernet, I have a collection of ethernet-shield-equipped Teensies all receiving UDP packets of encoded audio from a JackTrip hub server running on a laptop, and sending packets back to the server. All that's required for this to work is that the sample rate and buffer size reported in the JackTrip packet headers match between server and client(s). So far so good.
Not so good, is trying to keep these clients in sync with the server. My JackTripClient object inherits from AudioStream (and EthernetUDP), and its update method is as follows:
As the name suggests receivePackets() sometimes finds more than one packet. That, plus sendPacket(), and doAudioOutput() look like this:
Using `if` instead of `while` in receivePackets() means the client drops a packet every so often. The time between dropouts varies a lot, perhaps more related to conditions on the server rather than the Teensies. Using `while`, however, means that the number of writes to the circular buffer tends to overtake the number of reads from it, thus there's occasional distortion as unread UDP bytes are overwritten before they can be used for output. So I have the peculiar situation where despite the server and Teensies reportedly running at 44.1 KHz (with a buffer size of 32, so UDP packets being exchange roughly every 726 µs), Teensy is receiving more packets from the server than it is sending. Indeed, JackTrip reports (via its -I flag) positive skew, i.e. the client falling behind the server.
Thinking that maybe all that UDP activitiy in update() (on an audio hardware interrupt, if I understand correctly) might be slowing Teensy down, I experimented with using TeensyTimerTool (see `USE_TIMER` above) to move the UDP reads/writes out of update() to a callback using GPT1; this tends to work for a few seconds, but eventually packet read/writes stop -- perhaps communicating with the ethernet shield on this sort of basis isn't viable. Might something less generic work, as per https://www.pjrc.com/teensy/interrupts.html? For sending, I'd need to be trying to call AudioStream::receiveReadOnly() via something other than the audio interrupt, which I sense wouldn't be straightforward.
I feel like synchorisation is a thorny issue, plus I'm a little out of my depth with the lower-level aspects of what I'm trying to achieve. That said, I'd be grateful to get people's views on:
My hope is to get things as good as they can be while keeping the project accessible (so no PTP), interoperable (probably leaving Teensy's audio clock alone), then have the Teensies follow the jacktrip server, resampling audio buffers as necessary to keep things more-or-less in sync.
The full code lives on a development branch here.
Thanks in advance,
Tommy
I'm working on setting up multiple Teensy 4.1's as JackTrip clients for audio spatialisation (e.g. distributed WFS). Using NativeEthernet, I have a collection of ethernet-shield-equipped Teensies all receiving UDP packets of encoded audio from a JackTrip hub server running on a laptop, and sending packets back to the server. All that's required for this to work is that the sample rate and buffer size reported in the JackTrip packet headers match between server and client(s). So far so good.
Not so good, is trying to keep these clients in sync with the server. My JackTripClient object inherits from AudioStream (and EthernetUDP), and its update method is as follows:
Code:
void JackTripClient::update(void) {
#ifdef USE_TIMER
// If using TeensyTimer, just do audio output here, as the timer's callback calls updateImpl()
doAudioOutput();
#else
updateImpl();
#endif
}
void JackTripClient::updateImpl() {\
// Read as many UDP packets as are available right now; place the contents in a circular buffer
receivePackets();
#ifndef USE_TIMER
// Copy the latest unread packet from the circular buffer to this object's audio outputs
doAudioOutput();
#endif
// Get the current audio input block, put it into a UDP packet and send that to the server
sendPacket();
if (showStats && connected) {
packetStats.printStats();
udpBuffer.printStats();
}
}
As the name suggests receivePackets() sometimes finds more than one packet. That, plus sendPacket(), and doAudioOutput() look like this:
Code:
void JackTripClient::receivePackets() {
if (!connected) return;
int size;
// Check for incoming UDP packets. Get as many packets as are available.
while ((size = parsePacket()) > 0) {
lastReceive = 0;
if (size == EXIT_PACKET_SIZE && isExitPacket()) {
// Exit sequence
Serial.println("JackTripClient: Received exit packet");
Serial.printf(" maxmem: %d blocks\n", AudioMemoryUsageMax());
Serial.printf(" maxcpu: %f %%\n\n", AudioProcessorUsageMax());
stop();
return;
} else if (size != UDP_PACKET_SIZE) {
Serial.println("JackTripClient: Received a malformed packet");
} else {
// Read the UDP packet and write it into a circular buffer.
uint8_t in[size];
read(in, size);
udpBuffer.write(in, size);
// Read the header from the packet received from the server.
memcpy(serverHeader, in, sizeof(JackTripPacketHeader));
if (showStats && packetStats.awaitingFirstReceive()) {
Serial.println("===============================================================");
Serial.printf("Received first packet: Timestamp: %" PRIu64 "; SeqNumber: %" PRIu16 "\n",
serverHeader->TimeStamp,
serverHeader->SeqNumber);
packetHeader.TimeStamp = serverHeader->TimeStamp;
packetHeader.SeqNumber = serverHeader->SeqNumber;
Serial.println("===============================================================");
}
packetStats.registerReceive(*serverHeader);
}
}
if (lastReceive > RECEIVE_TIMEOUT_MS) {
Serial.printf("JackTripClient: Nothing received for %.1f s. Stopping.\n", RECEIVE_TIMEOUT_MS / 1000.f);
stop();
}
}
void JackTripClient::sendPacket() {
// Might have received an exit packet, so check whether still connected.
if (!connected) return;
// Get the location in the UDP buffer to which audio samples should be
// written.
uint8_t packet[UDP_PACKET_SIZE];
uint8_t *pos = packet + PACKET_HEADER_SIZE;
// Copy audio to the UDP buffer.
audio_block_t *inBlock[NUM_CHANNELS];
for (int channel = 0; channel < NUM_CHANNELS; channel++) {
inBlock[channel] = receiveReadOnly(channel);
// Only proceed if a block was returned, i.e. something is connected
// to one of this object's input channels.
if (inBlock[channel]) {
memcpy(pos, inBlock[channel]->data, CHANNEL_FRAME_SIZE);
pos += CHANNEL_FRAME_SIZE;
release(inBlock[channel]);
}
}
packetHeader.SeqNumber++;
packetHeader.TimeStamp += packetInterval;
packetInterval = 0;
// Copy the packet header to the UDP buffer.
memcpy(packet, &packetHeader, PACKET_HEADER_SIZE);
// Send the packet.
beginPacket(serverIP, serverUdpPort);
size_t written = write(packet, UDP_PACKET_SIZE);
if (written != UDP_PACKET_SIZE) {
Serial.println("JackTripClient: Net buffer is too small");
}
auto result = endPacket();
if (0 == result) {
Serial.println("JackTripClient: failed to send a packet.");
}
packetStats.registerSend(packetHeader);
}
void JackTripClient::doAudioOutput() {
if (!connected) return;
// Copy from UDP inBuffer to audio output.
// Write samples to output.
audio_block_t *outBlock[NUM_CHANNELS];
uint8_t data[UDP_PACKET_SIZE];
// Read a packet from the input UDP buffer
udpBuffer.read(data, UDP_PACKET_SIZE);
for (int channel = 0; channel < NUM_CHANNELS; channel++) {
outBlock[channel] = allocate();
// Only proceed if an audio block was allocated, i.e. the
// current output channel is connected to something.
if (outBlock[channel]) {
// Get the start of the sample data in the packet. Cast to desired
// bit-resolution.
auto start = (const int16_t *) (data + PACKET_HEADER_SIZE + CHANNEL_FRAME_SIZE * channel);
// Copy the samples to the output block.
memcpy(outBlock[channel]->data, start, CHANNEL_FRAME_SIZE);
// Finish up.
transmit(outBlock[channel], channel);
release(outBlock[channel]);
}
}
}
Using `if` instead of `while` in receivePackets() means the client drops a packet every so often. The time between dropouts varies a lot, perhaps more related to conditions on the server rather than the Teensies. Using `while`, however, means that the number of writes to the circular buffer tends to overtake the number of reads from it, thus there's occasional distortion as unread UDP bytes are overwritten before they can be used for output. So I have the peculiar situation where despite the server and Teensies reportedly running at 44.1 KHz (with a buffer size of 32, so UDP packets being exchange roughly every 726 µs), Teensy is receiving more packets from the server than it is sending. Indeed, JackTrip reports (via its -I flag) positive skew, i.e. the client falling behind the server.
Thinking that maybe all that UDP activitiy in update() (on an audio hardware interrupt, if I understand correctly) might be slowing Teensy down, I experimented with using TeensyTimerTool (see `USE_TIMER` above) to move the UDP reads/writes out of update() to a callback using GPT1; this tends to work for a few seconds, but eventually packet read/writes stop -- perhaps communicating with the ethernet shield on this sort of basis isn't viable. Might something less generic work, as per https://www.pjrc.com/teensy/interrupts.html? For sending, I'd need to be trying to call AudioStream::receiveReadOnly() via something other than the audio interrupt, which I sense wouldn't be straightforward.
I feel like synchorisation is a thorny issue, plus I'm a little out of my depth with the lower-level aspects of what I'm trying to achieve. That said, I'd be grateful to get people's views on:
- whether setting up a hardware interrupt for the UDP reads/writes colud be viable (preferably checking every 50 µs or so for new packets)
- and whether TeensyTimerTool could actually serve this purpose
- replacing NativeEthernet with QNEthernet -- might this help with the fairly intensive network stuff I'm trying to achieve?
- any other thoughts around synchronisation -- external clocking (NTP), nudging PLL4?
My hope is to get things as good as they can be while keeping the project accessible (so no PTP), interoperable (probably leaving Teensy's audio clock alone), then have the Teensies follow the jacktrip server, resampling audio buffers as necessary to keep things more-or-less in sync.
The full code lives on a development branch here.
Thanks in advance,
Tommy