USB interface for multi channel outputs, not just stereo

[sonaben]

Hi folks,

I'd like to update the USB audio for Teensy 4 to handle more than 2 channels (starting with 4 but eventually 8) which it only currently supports, so that I can play 4/8 different channels from my computer and use them as outputs on the Teensy.

It seems that some people have worked on it: https://forum.pjrc.com/threads/60557...8i80-USB-Audio
But never got to the end of it.

I've been trying as well, but I'm a bit stuck as I don't really understand what the code does in some places.

Here's what I've done so far:

- Update to the AUDIO_INTERFACE descriptor
usb_desc.c line 1574 (USB 2 480mb), updated bNrChannels to 4 (was 2)
This change allows the Teensyduino interface in Windows to show 4 outputs
Now I'm not sure if anything else needs updating in the descriptor.

- Update to USB audio library
usb_audio.h and usb_audio.cpp:
Updated the size of rx_buffer to double it:

DMAMEM static uint8_t rx_buffer[AUDIO_RX_SIZE] __attribute__ ((aligned(64)));

Then I've updated the code to duplicate everywhere possible where right and left are being used.
Example:

void usb_audio_receive_callback(unsigned int len)
{
	unsigned int count, avail;
	audio_block_t *left, *right, *three, *four;
	const uint32_t *data;

	AudioInputUSB::receive_flag = 1;
	len >>= 4; // 1 sample = 4 bytes: 2 left, 2 right
	data = (const uint32_t *)rx_buffer;

	count = AudioInputUSB::incoming_count;
	left = AudioInputUSB::incoming_left;
	right = AudioInputUSB::incoming_right;
	three = AudioInputUSB::incoming_three;
	four = AudioInputUSB::incoming_four;

	if (left == NULL) {
		left = AudioStream::allocate();
		if (left == NULL) return;
		AudioInputUSB::incoming_left = left;
	}
	if (right == NULL) {
		right = AudioStream::allocate();
		if (right == NULL) return;
		AudioInputUSB::incoming_right = right;
	}
	if (three == NULL) {
		three = AudioStream::allocate();
		if (three == NULL) return;
		AudioInputUSB::incoming_three = three;
	}
	if (four == NULL) {
		four = AudioStream::allocate();
		if (four == NULL) return;
		AudioInputUSB::incoming_four = four;
	}
....
}

However I do not understand enough what happens in some functions e.g copy_to_buffers with the bitwise operations.

Also I am unsure if there are other places in the libraries that will need updating e.g usb.c

Any help would be MUCH appreciated, even just some hints to know if I'm going in the right direction!

I'd be happy to collaborate if you are interested in this update.

Thanks

Ben

[sonaben]

If someone who understands could explain what the bitwise operations do in the following function.... e.g why "left & 0x02"?!
Thanks!

static void copy_to_buffers(const uint32_t *src, int16_t *left, int16_t *right, unsigned int len)
{
	uint32_t *target = (uint32_t*) src + len; 
	while ((src < target) && (((uintptr_t) left & 0x02) != 0)) {
		uint32_t n = *src++;
		*left++ = n & 0xFFFF;
		*right++ = n >> 16;
	}

	while ((src < target - 2)) {
		uint32_t n2 = *src++;
		uint32_t n1 = *src++;
		uint32_t n = *src++;
		*(uint32_t *)left = (n1 & 0xFFFF) | ((n & 0xFFFF) << 16);
		left+=2;
		*(uint32_t *)right = (n1 >> 16) | ((n & 0xFFFF0000)) ;
		right+=2;
	}

	while ((src < target)) {
		uint32_t n = *src++;
		*left++ = n & 0xFFFF;
		*right++ = n >> 16;
	}
}

[kinesphere]

I made a custom board with the CS42448 for the same purpose and I'm also stuck at the same place as you. I managed to show more outputs in Windows too, but that was it. I still have hope this will be possible soon with Teensy. Understanding those bitwise operations would be a good new step and any progress on this will be very much appreciated!

[Stephanie Wallace]

I’m currently working on a project that requires at least 16 channels out over a single cable… and if anyone’s goals are similar—precisely synchronized timing to interface every individual channel with a DAW—USB 2.0 simply isn’t an option. The polling nature of USB 2.0 doesn’t allow for precision timing. IEEE 1588 compliance is necessary, which means either using NativeEthernet or going all out with an FPGA.

I’m very much hoping to avoid the FPGA route, especially as what are now next-gen MCUs by dint of current supply chain issues using the i.MX RT1176 offer a co-processor and multiple 1Gbps Ethernet interfaces make this exceptionally easy: AES67 implementation —> Ethernet —> drivers for a couple of common Ethernet to USB 3.2 chips used in adapters and USB hubs.

I’m currently working on an AES67 implementation for the Teensy 4.1, which can theoretically handle 48 bidirectional, 24 bit/48KHz audio channels over a 100Mbps connection, though due to some limitations of the 4.1 and i.MX RT1062, somewhere between 16 to 24 channels is more realistic. As long as I can achieve 16 channels while leaving plenty of memory/CPU overhead for other operations, I’ll be happy, and will write drivers for the two most common Ethernet to USB 3.2 chips used in existing devices.

If not, due to time constraints, there will be an FPGA involved, but regardless, all software/hardware will be open source.

[JayShoe]

Hi Stephanie, this is valuable work. Good luck.

[sonaben]

On my side I managed to get it "working" with 4 channels, but I'm getting some crackling which I haven't figured out yet.

So if anyone more experienced would like to spend one hour debugging with me, that would be much appreciated

[felipe_mello]

Hello, sonaben! Very nice to hear that you've managed to get 4 channels working (even if there's some crackling yet). I am, too, trying to make Teensy stream 4 ch (ideally 8) via USB. I've done a lot of digging here at the forum, and it seems that you have the most advanced code at the moment.

Could you share the modified files so we can try to come up with solutions, or at least try to understand what's going on?

Thank you for all your work!

[sonaben]

Hi Felipe, I'll PM you!

[Stephanie Wallace]

Thanks! Work is coming along swiftly follow a period of crunch at work, though this is still a bear of a project and I expect to release the first beta in 2 to 4 months depending upon the ol’ day job. I’m using some tricks with FlexIO, and currently writing custom Windows drivers. Linux and macOS users will have to wait until after release (or contribute!). 16 24bit channels at up to 96kHz will happen—likely no more.

This all started off as part of a much simpler project, and though I’m completing this, along with a few other projects, for both personal and professional reasons—I’d like to move from primarily desktop development to embedded design and DSP full time—I thought it worth noting that I’ve discovered XMOS produces a line of pro-quality, single-purpose microcontroller boards (that can be used along with a Teensy with very little work) that can send between 8 and 32 bidirectional channels at 16/24/32bits at up to 384kHz in various formats/with various interfaces over USB 2.0 for between $90 and $130USD. One particular virtue is that they are tiny. The 8i8o board measures only 50mm x 50mm.

ADC still has to happen externally, though they sell a roughly $300 board that handles ADC as well. I thought this might be useful information for folks who need this now and don’t want to shell out $2,000 for a pro rack. I had a link to the $130 board yesterday—I’ll find it in my history when I’m in front of my main dev machine soon.

[celoranta]

Very interested. I’m beginning work along similar lines: building preamps and A/D converters for hexaphonic guitar pickups to allow output of individual real-time audio from each discrete string over a network via a teensy 4.1. I too have looked at AES67 and XMOS. Tonight I’m finally starting the physical stuff, soldering some Cirrus Logic 8-channel A/D chips to some breakout boards, and soldering together the Teensy that had lain dormant for a year or so in a box.

My final goal is to upgrade my doubleneck guitar/bass and my chapman stick to output polyphonic audio and send/receive OSC to and from a custom-written JUCE plugin running as a virtual instrument in a DAW.

Any advice or guidance would be most lovely.

[mcginty]

Hi all!

I just noticed this thread and thought I'd post - I've been modifying the Teensy 4 core library to support 8 in/8 out USB Audio for an audio project I just started. It currently works as far as my computer being able to send 8-channel audio to the Teensy, and the Teensy then splitting those buffers into their respective channels and passing them on to TDM output to the CS42448 test board (https://oshpark.com/shared_projects/2Yj6rFaW).

Audio quality is fantastic to my ears, not noticeably worse at first listen than the nice DAC on my Allen&Heath mixer!

Finished:

• Modified usb_desc.* to announce itself as having 8 channels instead of 2 (and to calculate the correct packet sizes to announce, etc.) (done)
• Modified usb_audio.* to handle an arbitrary number of input channel buffers instead of hardcoded left/right (done)

Todo:

• Modify AudioOutputUSB to support >2ch as well
• Support 96kHz/24bit in both USB and the AudioStream system
• Get this code clean and stable enough to merge back into the main repository if that's desired.
• Get more confidence on the correctness of the USB Descriptors, what to do with USB HS is not negotiated, etc.
• USB Audio 2.0 (support for channel clusters seems nice)

As a side note, I had 5 PCBs made of that CS42448 test board, and have... 4 left, and am happy to mail them to anybody else in North America that want to try their hand at soldering one (can send my Digikey cart to show you what to order too). Feel free to message me.

Would be happy to collaborate if anybody else is working on similar projects. I'm doing this work to start on a DJ/DIY-focused mixer (kind of similar to the Teenage Engineering TX-6, but hackier of course . I made my first prototype with an STM32, and damn, so much respect to Paul and everybody who's worked on the Teensy and its libraries for creating such a fantastic playground.

[celoranta]

I’m handy with a soldering iron and wouldn’t mind collaborating…

[h4yn0nnym0u5e]

Would be more than happy to collaborate on this. I've got a pretty reasonable knowledge of the Audio library, much less so on the USB side. So, I could probably do the changes to AudioOutputUSB; supporting 96kHz is not too hard; 24-bit is much harder, as so much of the Audio library makes big assumptions about being 16-bit, and also uses DSP instructions which don't scale well to higher bit depths. Higher depths? You know what I mean...

Do you have github forks of cores and Audio?

[acumartini]

@mcginty
This is amazing! I am working on a custom board similar to the one it sounds like you designed and have trying to update the Teensy core libs to do exactly what you have accomplished. While I am not using the builtin Audio lib stuff it would be super helpful to see your modifications, especially to the descriptors. Would you be willing to post or share your modified usb_* files for 8 IN.

I will be working on all of the same TODO items (except 48kHz and no merge back to main as I am not using the Audio lib) so I can share my feedback and modifications as well.

[mcginty]

Here's what I've got so far, again just with USB multichannel *input* working so far, as well as code in main.cpp to route each of the stereo pairs to an output jack on the CS42448.

If it wasn't obvious, this is not cleaned up code and more like a "playground", which I made more obvious by changing the file structure from the parent "cores" repo, and copying in the Audio library to make it easier to modify in one repository.

I opted to use the Makefile method since it seemed easiest to tinker with compared to using the Arduino IDE, but it still depends on the Arduino libraries existing.

Basic usage (modifying your ARDUINOPATH as needed):

git clone https://github.com/mcginty/teensy4-studio-audio
cd teensy4-studio-audio
ARDUINOPATH=/usr/share/arduino make

@celoranta just PM'd you if you were interested in a board (anyone else feel free to PM me with an address if you want a remaining board)

[acumartini]

@mcginty
Was there meant to be an attachment on your last post? Not seeing one, but maybe I am missing it...

FYI I am using platformio for my project and they have a good mechanism for patching the teensy core files. Let me know if you would like me to share the INI etc.

[h4yn0nnym0u5e]

this is not cleaned up code and more like a "playground", which I made more obvious by changing the file structure from the parent "cores" repo, and copying in the Audio library to make it easier to modify in one repository.

I opted to use the Makefile method since it seemed easiest to tinker with compared to using the Arduino IDE, but it still depends on the Arduino libraries existing.

aaaand I'm out, at least for now ... separate cores and Audio repos may be a bit less convenient for a "playground", and using make might be easiest for you, but they make it seriously tedious to flip between different development branches while sticking with what I think of as the Joe Average standard Arduino IDE.

Good luck with your project, I'll be interested to give it a try out once it's back to the canonical Teensyduino structure.

[mcginty]

@mcginty
Was there meant to be an attachment on your last post? Not seeing one, but maybe I am missing it...

FYI I am using platformio for my project and they have a good mechanism for patching the teensy core files. Let me know if you would like me to share the INI etc.

https://github.com/mcginty/teensy4-studio-audio <- the repo is in the code block.

Feel free to share your PlatformIO config, would be interesting to make this usable in that environment.

[mcginty]

aaaand I'm out, at least for now ... separate cores and Audio repos may be a bit less convenient for a "playground", and using make might be easiest for you, but they make it seriously tedious to flip between different development branches while sticking with what I think of as the Joe Average standard Arduino IDE.

Good luck with your project, I'll be interested to give it a try out once it's back to the canonical Teensyduino structure.

Cool, give me a couple days to finish up my current experimentation and then I'll be re-canonicalizing it to make it easier to contribute to and see clean diffs for. I just posted this code as it lay on my FS (hacked together for my temporary convenience).

[acumartini]

Ah got it thanks! Here is the platformio.ini config (note i am using a micromod so set your board accordingly):

[platformio]
default_envs = release

[env]
platform = teensy@4.17.0
board = teensymm
board_build.mcu = imxrt1062
board_build.f_cpu = 600000000L
upload_protocol = teensy-gui
framework = arduino
monitor_speed = 115200
extra_scripts = pre:patch/apply.py
build_flags = 
	-std=c++0x
	-Wl,--print-memory-usage
	-g
	-D USB_MIDI_AUDIO_SERIAL
lib_deps = <your deps here>

[env:release]
build_flags = 
	${env.build_flags}

And patch/apply.py:

from os.path import join, isfile
import shutil

Import("env")

print("PATCHING...", end =" ")

FRAMEWORK_DIR = env.PioPlatform().get_package_dir("framework-arduinoteensy")

patchflag = ".patched"
patchflag_path = join("patch", patchflag)

# patch files only if the patchflag is not present
if not isfile(patchflag_path):
    original_file = join(FRAMEWORK_DIR, "cores", "teensy4", "usb_audio.h")
    patched_file = join("patch", "usb_audio.h")
    assert isfile(original_file) and isfile(patched_file)
    shutil.copyfile(patched_file, original_file)

    original_file = join(FRAMEWORK_DIR, "cores", "teensy4", "usb_audio.cpp")
    patched_file = join("patch", "usb_audio.cpp")
    assert isfile(original_file) and isfile(patched_file)
    shutil.copyfile(patched_file, original_file)

    original_file = join(FRAMEWORK_DIR, "cores", "teensy4", "usb_desc.h")
    patched_file = join("patch", "usb_desc.h")
    assert isfile(original_file) and isfile(patched_file)
    shutil.copyfile(patched_file, original_file)

    original_file = join(FRAMEWORK_DIR, "cores", "teensy4", "usb_desc.c")
    patched_file = join("patch", "usb_desc.c")
    assert isfile(original_file) and isfile(patched_file)
    shutil.copyfile(patched_file, original_file)

    with open(patchflag_path, "w") as fp:
        fp.write("")

    print("complete!")
else:
    print("skipped")

Which pulls in files local to the /patch directory if the /patch/.patched flag is not present. There is a bit of overhead remembering to copy changes over from teensy core files that you are modifying, but overall this works well. Once the changes are more or less done, I am going to remove the .patched flag and just have it copy happen with every compile so I can just forget about it.

[acumartini]

Excellent job with the AUDIO_CHANNELS iteration here btw. And these descriptor changes are awesome! This is non-trivial for sure.

[UHF]

... (kind of similar to the Teenage Engineering TX-6, but hackier of course ....

This is exactly what many people are looking for - a USB audio interface with multiple inputs (6+), preferably stereo in a small unit that doesn't cost over $400. This would sell very well in kit-form or complete.

[acumartini]

So I was able to get this working for 4 channels RX/TX. Have not tested more channels yet, but I think it will work just fine granted that I have enough room left in lower RAM for the buffers.

Thank you again @mcginty for sharing your progress, this was a huge help. I am not sure how useful this will be because I am not using the Audio lib to buffer data for TX, but here is my working usb_audio.cpp code:

int16_t usb_audio_buffer_out[2][AUDIO_CHANNELS][AUDIO_TX_BUFFER_SIZE]{{{0}}};
uint8_t usb_audio_buffer_out_idx[2]{0};
bool usb_audio_buffer_out_ping = true;

/*DMAMEM*/ uint16_t usb_audio_transmit_buffer[AUDIO_TX_SIZE / AUDIO_SAMPLE_BYTES] __attribute__ ((used, aligned(32)));

static void tx_event(transfer_t *t)
{
	uint32_t len = usb_audio_transmit_callback();
	// usb_audio_sync_feedback = usb_audio_feedback_acc >> usb_audio_sync_rshift; // Note: this was in the core code but I think it is unnecessary as it is handled in the sync callback 
	usb_prepare_transfer(&tx_transfer, usb_audio_transmit_buffer, len, 0);
	arm_dcache_flush_delete(usb_audio_transmit_buffer, len);
	usb_transmit(AUDIO_TX_ENDPOINT, &tx_transfer);
}

static void copy_from_buffers(uint32_t *dst, int16_t (&src)[AUDIO_CHANNELS][AUDIO_TX_BUFFER_SIZE], unsigned int len)
{
	uint32_t *target = (uint32_t *)dst + (len * AUDIO_CHANNELS / 2);
	uint16_t i = 0;

	while (dst < target) {
		for (uint8_t j = 0; j < AUDIO_CHANNELS; j += 2) {
			*dst++ = (src[j][i] << 16) | (src[j + 1][i] & 0xFFFF);
		}
		i++;
	}
}

// Called from the USB interrupt when ready to transmit another
// isochronous packet.  If we place data into the transmit buffer,
// the return is the number of bytes.  Otherwise, return 0 means
// no data to transmit
unsigned int usb_audio_transmit_callback(void)
{
	uint8_t buffer_idx = usb_audio_buffer_out_ping ? 1 : 0;
	uint8_t len = usb_audio_buffer_out_idx[buffer_idx]; // samples per channel

	copy_from_buffers(
		(uint32_t *)usb_audio_transmit_buffer,
		usb_audio_buffer_out[buffer_idx],
		len
	);

	usb_audio_buffer_out_idx[buffer_idx] = 0;
	usb_audio_buffer_out_ping = !usb_audio_buffer_out_ping;
	
	return len * AUDIO_CHANNELS * AUDIO_SAMPLE_BYTES;
}

Also, one thing to note, I had to rework the feedback mechanism to get the RX transfer rate to stabilize. I am pretty new to USB audio, but I am fairly certain you want to both increment and decrement the feedback accumulator to on underrun/overrun respectively to achieve a balanced state. You also want to provide preventive correction increment and decrement when the RX flag is set. Something like this:

if (diff < AUDIO_RX_BUFFER_SIZE / 2) { // prevent overrun
    usb_audio_feedback_acc -= AUDIO_RX_BUFFER_SIZE / 2 - diff;
} else { // prevent underrun
    usb_audio_feedback_acc += diff - AUDIO_RX_BUFFER_SIZE / 2;
}

[acumartini]

FYI I am maxing out at 4 channels. I can get 6 to load up but my process loop is not reading the data fast enough. This may not be an issue with the Audio library because of the block buffering is likely more efficient that what I am doing. 8 channels also loads up fine but leaves little to no room for extra processing (in my case UX display updates).

Update: Scratch that. With larger buffers I am able to keep up... still experimenting. Again not an issue with Audio lib because of the AudioStream buffer pool.

[celoranta]

Has anyone successfully maxed out the TDM capabilities of the Teensy 4? i.e. 32 channels of audio, with 16 on each SAI port?
Seems all the boards and code stop at 8...