Teesny audio over UDP

That looks more like the packets are out of order does it not? When just viewing the raw audio file you shouldn't have this specific issue even if the clocks are different.
 
Add a sequence number to the sent packets and missing or out of orders packets will be easy to detect.
 
Thanks for the responses. I'll have a look at packet ordering and see whether that's what causing it.

On the use of UDP , there are already protocols that do this yes, I just wanted to explore pure UDP to see if it was fit for purpose given the fact that this was running over a LAN. I'm trying to avoid any unnecessary complexity, however, looks like I might not have a choice.
 
As far I know, RTSP is used for some compressed formats only. Maybe wrong...

With some work, a receiver could be written, with the codecs library. Transmitting needs encoding.. not sure if a library is available.
 
UDP is best thought of as Unreliable Datagram Protocol, even though it stands for User Datagram Protocol!

Reordering, packet loss, packet duplication, all can happen with UDP. Its not intended for streaming as-is.
Add packet numbering and a reconstruction buffer at the receiving end and you'll cope with most reordering and
duplication problems, but missing packets need some sort of ACK/NAK reply channel, or perhaps something
like the graceful degradation of audio CD format (every other sample in separate packets, so a missed packet
just drops the bandwidth for a while).
 
+1 on using the standard RTP. It does have a format for Linear PCM 16-bit Stereo audio.

Reordering of packets doesn't happen on most networks.
 
Not had a chance to get as far as I wanted tonight, however, I've added a sequence ID and I can see that the packets are arriving in the order they are sent. The really interesting behaviour is that after exactly 310 packets (each packet contains a 4 byte sequence id and a single audio block of 256 bytes) it starts consistently dropping 5-7 packets.

Audio Analysis.png

The image above shows the log file from reading the UDP Packets. I'll dig into why its drop so many packets consistently, it can't be a coincidence that its doing it every 310 packets every time. Obviously using audacity the odd shaped wave I shared earlier is being caused by the dropped packets.
 
Just to add that I think the above is a bug in my c# code, not an issue with the UDP packets. Looking at them in wireshark they are all nicely ordered, even past 310 packets. More investigation ahead!
 
Fixed the problem, the C# app wasn't reading fast enough from the stream and some packets got disposed. I've now got the Teensy audio coming through on the PC with zero packet loss or issues with ordering. I'll now replace the C# app with another Teensy to see if there are any issues that way.

Thanks for all the help so far.
 
The plot thickens! So I've now setup a second Teensy + Shield that reads the UDP packets being sent from the first Teensy. It then rebroadcasts these out over UDP and I've setup my PC C# app to read the rebroadcast packets and then save them to disk. Like the previous example, the packets are received in order with zero packet drops.

The resulting rebroadcast 1Khz wave viewed in audacity can be seen in the image below, it's the bottom wave. The top wave is the standard 1Khz tone:

Teensy to Teensy.jpg

So vaguely resembling the original but vastly different. I think my read/play code within the Teensy is suspect, I'm going to strip it right back and see if I can't find the issue. For anyone interested, the custom Audio Stream object I'm using is below:
Code:
#include "AudioPlayMemoryRaw.h"
#include <Arduino.h>
#include "play_sd_raw.h"
#include "spi_interrupt.h"
#include <queue>

IPAddress TARGET_IP(255, 255, 255, 255);
const uint8_t HEADER = 0xFE;

bool AudioPlayMemoryRaw::play(EthernetUDP UDP, const int port)
{
	stop();

	_UDP = UDP;
	_port = port;
	playing = true;
	return true;
}

void AudioPlayMemoryRaw::enqueue(int16_t* packet)
{
	if (playing)
		_OutputQueue.push(packet);
}

int AudioPlayMemoryRaw::queueSize()
{
	return _OutputQueue.size();
}

long packetNumber = 0;

void AudioPlayMemoryRaw::Send(audio_block_t* block)
{
	_UDP.beginPacket(TARGET_IP, _port);
	_UDP.write(HEADER);
	_UDP.write((uint8_t*)&packetNumber, sizeof(packetNumber));
	_UDP.write((uint8_t*)block->data, AUDIO_BLOCK_SAMPLES * 2);
	_UDP.endPacket();

	packetNumber++;

	if (packetNumber > 0x7FFFFFFF)
		packetNumber = 0;
}

void AudioPlayMemoryRaw::stop(void)
{
	if (playing)
	{
		playing = false;
	}
}

void AudioPlayMemoryRaw::update(void)
{
	unsigned int i;
	audio_block_t* block;

	// only update if we're playing
	if (!playing) return;

	// don't update or allocate memory if there is nothing to play
	if (_OutputQueue.size() <= 0) return;

	// allocate the audio blocks to transmit
	block = allocate();
	if (block == NULL) return;

	if (_OutputQueue.size() > 0)
	{
		int16_t* packet = _OutputQueue.front();
		_OutputQueue.pop();

		memcpy(block->data, packet, sizeof(block->data));

		transmit(block);
		Send(block);
	}

	release(block);
}

And how I am calling it:

Code:
#include "AudioPlayMemoryRaw.h"
#include <NativeEthernet.h>
#include <NativeEthernetUdp.h>
#include <Audio.h>
#include <Wire.h>
#include <SPI.h>
#include <SerialFlash.h>

AudioPlayMemoryRaw       rawPlayer;
AudioOutputI2S           headPhones;
AudioConnection          patchCord1(rawPlayer, 0, headPhones, 0);
AudioConnection          patchCord2(rawPlayer, 0, headPhones, 1);

AudioControlSGTL5000     sgtl5000_1;

IPAddress MY_IP(192, 168, 1, 223);
EthernetUDP UDP;
byte MYMAC[6];
const int PORT = 1153;

long lastPacketNumber = 0;

void setup()
{
    Serial.begin(115200);

    AudioMemory(30);

    digitalWrite(13, OUTPUT);
    LEDOn();

    sgtl5000_1.enable();
    sgtl5000_1.volume(0.5);

    SetupEthernet();

    LEDOff();

    rawPlayer.play(UDP, PORT);
}

void loop()
{
    Read();
}

void LEDOn()
{
    digitalWrite(13, HIGH);
}

void LEDOff()
{
    digitalWrite(13, LOW);
}

void LEDError()
{
    while (true)
    {
        LEDOff();
        delay(500);
        LEDOn();
    }
}

void Read()
{
    int packetsize = UDP.parsePacket();
    int16_t packet[AUDIO_BLOCK_SAMPLES];

    if (packetsize)
    {
        uint8_t packetHeader = 0x00;
        long packetNumber = 0;

        packetHeader = UDP.read();
        UDP.read((uint8_t*)&packetNumber, sizeof(long));
        UDP.read((uint8_t*)&packet, AUDIO_BLOCK_SAMPLES * 2);

        if (lastPacketNumber > packetNumber)
        {
            Serial.printf("%d - Out of order", packetNumber);
        }

        if (packetNumber != (lastPacketNumber + 1))
        {
            Serial.printf("%d - Packet dropped", packetNumber);
        }

        lastPacketNumber = packetNumber;

        rawPlayer.enqueue(packet);

        UDP.flush();
    }

}

void SetupEthernet()
{
    TeensyMAC(MYMAC);

    Ethernet.begin(MYMAC, MY_IP);

    Serial.println("Checking ethernet hardware");
    if (Ethernet.hardwareStatus() == EthernetNoHardware)
    {
        Serial.println("Ethernet shield was not found.  Sorry, can't run without hardware. :(");
        LEDError();
    }
    Serial.println("Checking ethernet link");
    if (Ethernet.linkStatus() == LinkOFF)
    {
        Serial.println("Ethernet cable is not connected.");
        LEDError();
    }

    UDP.begin(PORT);
}

void TeensyMAC(uint8_t* mac)
{
    for (uint8_t by = 0; by < 2; by++) mac[by] = (HW_OCOTP_MAC1 >> ((1 - by) * 8)) & 0xFF;
    for (uint8_t by = 0; by < 4; by++) mac[by + 2] = (HW_OCOTP_MAC0 >> ((3 - by) * 8)) & 0xFF;
    Serial.printf("MAC: %02x:%02x:%02x:%02x:%02x:%02x\n", mac[0], mac[1], mac[2], mac[3], mac[4], mac[5]);
}
 
Given the random nature of the wave I suspected something wasn't honouring the byte ordering. Turns out that was correct, by swapping the byte order we get a much more representative wave:

Teensy to Teensy mk2.jpg

The top is the standard 1Khz tone, the middle is the tone being received with the byte ordering wrong, the bottom is with the bytes swapped to the correct order.

If you look you can see that parts of the wave are coming through completely fine, however, there's a lot of random distortion happening. I'm taking a look at what's causing this now, any pointers or tips would be gratefully received. Could this be an issue with noise/interface from the Teensy?

The flow at the moment is Teensy 1 reads a 1Khz tone file from SD card and plays that over UDP. Teensy 2 reads the 1Khz UDP packets and then plays them using a custom Audio Stream object that also rebroadcasts the packets it reads from UDP over UDP. Looks like my read UDP function is probably where the issue lies.
 
Last edited:
For starters, I wouldn’t recommend using NativeEthernetUdp for this, due in part to its blocking nature. If you take the time to transfer the code directly to FNET you’ll more than likely have a better time if it is in your UDP read function. Especially once you start doing more than one stream like you plan on doing I don’t believe it will be able to keep up otherwise.
 
Thanks for the advice on FNET Vs NativeEthernetUdp. This issue was interesting, it was being caused by a combination of using std queue and also local variables in the read function. If I inspect the block after its read but before adding it to the std queue then it's absolutely fine. I'm replacing std queue with a much simpler data structure for buffering incoming packets.
 
Using RTP vs a proprietary format should just be a matter of adding a 12 byte header to the packets. Then lots of standard tools work to play and analyse the streams. For example, "RTP Player" in Wireshark.
 
Sorry for the lack of update. I'm yet to pursue the Circular_Buffer suggestion I've found the issue with the original code that was causing the clipped wave forms in my post above. Essentially I was using a local variable that, when it fell out of scope, caused some parts of the block to get changed (likely because I'm using std queue). By tidying up my memory management I've got it working really nicely sending audio from one Teensy to another using UDP.

Of note, I've been monitoring packet loss and ordering in my setup (single switch) and I've seen a few packet drops but nothing that has even been audible yet. I'm going to ramp up the number of channels which I suspect will require me to finally ditch the std queue and may introduce further packet drop and UDP issues. I'll keep you posted.

I've also managed to get a C# app pumping out raw audio over UDP. I had to manage the timing really carefully to avoid overflowing the Teensy's memory whilst ensuring the packets arriving quick enough to prevent audio distortion, but it's working nicely. This should allow me to use the PC to load up the network to see how the Teensy handles it.
 
Hopefully someday we will have RTP audio over ethernet/IP/UDP as a standard I/O device in the teensy audio library. This would be a building block towards eventual support of AES67 (which uses RTP).
 
So I've now got the Teensy receiving 7 x 44.1 Khz audio streams over UDP then mixing them together with the mic input to give a single output. Fundamentally it works really nicely without any perceivable delay and minimal noise/distortion, most of the distortion that does exist in the stream comes from the basic method I'm using for VOX (see my other post here). It currently doesn't handle multiple simultaneous streams for very long.

In it's current form, however, the Teensy runs out of memory quite easily which is largely due to the fact that the std::queue doesn't release memory. With a low number of UDP streams the buffers never hold more than 1-2 audio blocks, as they are received and processed very quickly, so all is good. When you introduce more streams the number of blocks held in the buffers increases, the queue allocates memory to hold the additional blocks but assumes you will likely need that memory again so doesn't release it. This way you end up with transient increases in buffer sizes stealing all of the memory. At the moment I'm simulating a really busy network with all 7 streams coming in continuously as a worse case scenario. Essentially the process that takes audio blocks from the buffers, plays them and then releases them isn't able to keep on top of the buffers and they run away in size. I'm going to do some more analysis of the memory usage today, however, I'm planning on swapping out the std::queue for the circular buffer suggestion (https://github.com/tonton81/Circular_Buffer).

Hopefully this will resolve the issue nicely.
 
I was working on a similar project a year or so ago and created a 2 channel UDP transport object for the Audio library.

Code is here: https://github.com/palmerr23/EtherAudio

Discussion here: https://forum.pjrc.com/threads/58660-Ethernet-audio-library-ready-to-beta-test

It works quite nicely with fairly low latency on SPI Wiznet adapters, and I haven't tried to convert it to native Teensy 4.1 ethernet as yet.

When I get back to it, I'll be converting the packet formats to Vincent Buren's VBAN protocol (very similar to mine), so that I can take advantage of all his neat gadgets for the PC/Mac/Mobile.

Hope this helps you as you move forward, but it seems like you're almost all the way there, anyway.

Richard
 
Keeping audio/video/midi streams in sync works really well with JACK which is open source and works in linux/windows/MacOS. This is used to keep multiple software in sync within the same OS, and also over the network.
I use this to sync video and audio across different applications. It's intended for POSIX compliant OS, but maybe it's possible to port, or at least inform a teensy build.

https://jackaudio.org/
https://jackaudio.org/api/
https://jackaudio.org/api/porting-guide.html

This might be of interest from https://github.com/jackaudio/jackaudio.github.com/wiki:

jack_transport> ?
activate Call jack_activate().
exit Exit transport program.
deactivate Call jack_deactivate().
help Display help text [<command>].
locate Locate to frame <position>.
master Become timebase master [<conditionally>].
play Start transport rolling.
quit Synonym for 'exit'.
release Release timebase.
stop Stop transport.
tempo Set beat tempo <beats_per_min>.
timeout Set sync timeout in <seconds>.
? Synonym for `help'.

echo play | jack_transport
# pass a command to execute. tempo change doesn't work via this method.

Also this:
https://github.com/jackaudio/jackaudio.github.com/wiki/JACK-Transport-limitations
 
Last edited:
Yes, I'm familiar with jack, but only at a single machine level.

Sadly, jack isn't not cross-platform, but that doesn't mean that there aren't some useful things to be learned from their mature approach to all things audio!

I'll have a look at its re-sync capabilities when I get back to UDP audio.

Most recently, I've been trying to work with WiFi as a UDP audio transport medium and have been stymied 802.11.x's inbuilt packet resend mechanism (below the TCP or UDP layer), which can do nasty things to a UDP stream (if packet re-ordering isn't available, with its inherent added delays). I was using ESP32's and they don't have enough WiFi throughput to reliably transport more than 2 x 24bit x 48kHz streams (where I was looking for 8 x 16bit x 48kHz minimum).
 
Can we use this to transfer audio over serial? I want to send and receive audio between two Teensies connected by the serial pins.
 
Back
Top