USB MIDI and Audio

Status
Not open for further replies.
P

Phrewfuf

New member
Hi,

i am planning to build a MIDI controller with a teensy 3.0 and since i have some I2S capable audio-grade DACs laying around i thought why not include an audio interface aswell.

Now my question is: Is it possible to make the teensy 3.0 speak MIDI and Audio on USB?
 
Ah, that's cool, thanks.

Is there any info on how to make both run at the same time? Or, to make it a bit easier, what would i have to do to make the teensy be a MIDI and HID device, both at the same time?
I don't really want to use two teensies and two USB-ports of my machine for one single device.
 
recri said:
The basic Audio device will be the same kind of composite when ever Paul gets to it.

-- recri --
Click to expand...

Well, that's true, but it's not as simple as it sounds. There's a lot of job to do, first of all is getting a solid audio lib and documentation. Then, a restyling of the USB core lib may be needed (I actually think that's mandatory) to let audio objects communicate via USB. Right now, the USB core does not support packets bigger than 64 bytes (one audio block as defined by audio_block_t type contains 256 bytes), and there's also no support for real time streaming (isochronous synchronized endpoints). And the whole descriptors stuff should be heavily modified to let the user set the teensy to act as a specific USB audio device (which may range from a simple mono microphone, to a 4 channel in/out mixer with MIDI and eq and more). I'm planning to mod my currently working USB audio stuff to handle AudioStream objects, just like the audio lib. I'd like to get an object like AudioInputUSB or AudioOutputUSB that could be connected to any other audio object, or something like AudioMixer4USB that exposes a mixer's controls over USB. Will keep you up on the topic that Paul linked.
 
MickMad said:
I'd like to get an object like AudioInputUSB or AudioOutputUSB that could be connected to any other audio object,
Click to expand...

I'd really like to see this too. Soon I'll probably be able to actually help, after the audio library is at a 1.0 release.

Have you thought about how this might add or drop samples, or otherwise slightly adjust the incoming and outgoing USB streams to fit into the timing of the audio library?
 
Not yet actually. USB audio specifies synchronization endpoints that can be paired with isochronous one to give the host timing informations to adjust the rate of in/outgoing samples on the paired endpoint. I think that with some buffering (which could be made at block and at sample level) and using those synch endpoints, we could achieve seamless, lag-free usb streaming. The thing that worries me most right now is the routine that copies audio data and puts it in USB packets; I'd like to see a function that uses buffer pointers to create USB packets around existing buffers, rather than having to manually copy one buffer inside a packet. The USB module supports this kind of addressing with the Buffer Descriptor Tabel entries, and it already has DMA access to the table entries; having to manually copy data for every packet (which could happen every millisecond) is a big overhead. And, a new bdt handling could allow for dynamic, non fixed size packets. I'd really like to see something like a USB static class, with functions to create packets on the fly around a premade buffer (to avoid the copying overhead), to add these packets to in/out queues, but also with functions to set the contents of the descriptors to achieve the same degree of freedom given by the Audio lib inside the USB environment (to create specific audio functions on the fly). This is just a few of the things that should be made to get a nice USB audio lib, but this may take a very long time to get together.
 
An audio sink device has to give the host rate feedback on a separate endpoint so the host can adjust the number of samples per buffer up or down. I believe it sends one fixed point fraction of feedback for each usb buffer received and it's reporting how far ahead or behind the ideal number of samples it is.

An audio source device just adjusts the number of samples per buffer. It sends however many samples are available when the buffer needs to be sent and host figures out the ratio between its clock and the source clock from the samples received.

Googling "usb audio rate feedback" finds a variety of discussions and examples.

-- rec --
 
recri said:
An audio sink device has to give the host rate feedback on a separate endpoint so the host can adjust the number of samples per buffer up or down. I believe it sends one fixed point fraction of feedback for each usb buffer received and it's reporting how far ahead or behind the ideal number of samples it is.

An audio source device just adjusts the number of samples per buffer. It sends however many samples are available when the buffer needs to be sent and host figures out the ratio between its clock and the source clock from the samples received.

Googling "usb audio rate feedback" finds a variety of discussions and examples.

-- rec --
Click to expand...

There's a little neat formula in the USB Audio specs which defines the synchronization value that the device has to send in conjunction with the sink endpoint (which means IN towards the host, USB is host-centric). I was actually just too lazy to implement the synch endpoint in my last revision, also because I found that it was not really necessary ,since my library works even without it (at least it SEEMS to work nicely ;) ). An example is this: say we got a USB mono IN with 44.1 KHz sampling rate, and that we are slaved to the USB SOF clock (1 ms); we can't send 44.1 samples per ms, so we send 44 samples for 10 consecutive times, and the 10th time we signal on the synch endpoint that the 11th transaction will see a packet with a sample more than before, which means 45 samples.

I repeat though, this is not what concerns me for now. I mean, this could be the first thing I'd implement when I get to work on the lib again, but the real problems with this stuff is the conversion between audio_block_t data and usb_packet_t data, for the way usb_packet_t data is treated by the USB core lib. I got to find a way to directly fit the address of an existing buffer inside the BDT of the USB Module, that is I got to find a way to create a usb packet around a premade buffer, to avoid the big overhead of copying data from a buffer to another (there is a way, but it can become really trivial to fit an hack like this in the lib, without crashing every other existing USB lib).
 
I wouldn't worry about the memory-to-memory copy too much. Sure, avoiding it would be nice. But Cortex-M4 is very fast, so it's really not much CPU overhead if done well. All the input and output objects in the audio library do copies. Someday I might try to eliminate the copies using linked DMA transfers, but that costs 2 or 4 times as many DMA channels, which seemed like a really bad trade-off before Teensy 3.1 added 12 more channels. I'm still not convinced it would be worthwhile.

Cortex-M4 has a hardware optimization for successive similar 32 memory reads or writes, if you use code like this:

Code: 
  uint32_t tmp1, tmp2, tmp3, tmp4;
  tmp1 = *ptr++;  // 2 cycles
  tmp2 = *ptr++;  // 1 cycle
  tmp3 = *ptr++;  // 1 cycle
  tmp4 = *ptr++;  // 1 cycle

Normally a memory access takes 2 cycles, but the processor detects that the next upcoming instruction in the CPU's pipeline is a "similar" enough access that it can use some burst mode feature of the bus (a part of the ARM architecture I'm not as familiar with...) to speed up every successive read. The same happens for writes. So you can move 16 bytes in or out of the CPU in just 5 cycles. Of course this is only fast while you have more CPU registers available, so you probably only want to read 16 bytes at a time, especially if you're going to do any math or shuffling of bits around before writing them all back out. Even with looping overhead, you can get to about 1 byte per cycle this way, or about 96 Mbyte/sec.

16 bit stereo at 44.1 kHz is only 0.18 Mbyte/sec. A well optimized memory-to-memory copy costs about 0.2% CPU time. Even with only moderate optimization effort, it's easy to get under 1%. That's pretty acceptable overhead for an audio input or output process where the user can't create multiple instances.
 
Status
Not open for further replies.
Back
Top