Using MIDI. and usbMIDI. at the same time for lots of commands. Any potential problem

Status
Not open for further replies.

gatheround

Well-known member
Hey everyone,
I'm wanting to use the MIDI library on a Teensy LC to send regular serial (DIN connector) MIDI information at the same time I'm sending the same information through a USB port using the usbMIDI library.

My data is going to be a continuous stream of sampled analog values, so I'd be wanting to theoretically send them as fast as possible. I'd probably have a loop that does an analogRead and then bangs out the corresponding MIDI and usbMIDI commands.

Is there any potential problems doing this?

Some potential problems I'm worried about:

- Since I'd like this to be basically running as fast as possible, is sending the Serial MIDI data at the same time slowing things down compared to just doing usbMIDI?
- Is the timing already handled by the Teensy library for this? Like, if I bang out Serial MIDI data as fast as possible are the devices that are listening to this data going to freak out and choke because they can't receive this data as fast as it's being transmitted?
- Does the Teensy wait until each MIDI command has been sent (and received?) before it allows my program to move on to the next operation?

Thanks for your time.
 
I expect you’ll send each MIDI message to each of the two routes - going back and forth between the two.
You’ll likely wait for both to finish the first message before sending the second to either.
The serial MIDI operates at a unique defined baud rate (31250 bps) and a 3-byte message will take about a millisecond to transmit. The USB will be faster, I expect, so expect 1-2 ms per message.

Unless you implement some kind of multi-threading, each row in your sketch will finish before the next starts.

I think this will be plenty fast. Synthesizers expect not more that a message per millisecond, and many note envelopes are many milliseconds long. You’ll be able to sample faster that synths can accept (or ears can hear changes).

If you add a BLE (Bluetooth low energy) MIDI route, expect more per-message latency. 10-30 ms between packets.

Fun stuff!
 
I expect you’ll send each MIDI message to each of the two routes - going back and forth between the two.
You’ll likely wait for both to finish the first message before sending the second to either.
The serial MIDI operates at a unique defined baud rate (31250 bps) and a 3-byte message will take about a millisecond to transmit. The USB will be faster, I expect, so expect 1-2 ms per message.

Unless you implement some kind of multi-threading, each row in your sketch will finish before the next starts.

I think this will be plenty fast. Synthesizers expect not more that a message per millisecond, and many note envelopes are many milliseconds long. You’ll be able to sample faster that synths can accept (or ears can hear changes).

If you add a BLE (Bluetooth low energy) MIDI route, expect more per-message latency. 10-30 ms between packets.

Fun stuff!

Thank you David,

Ok so basically if I do the commands sequentially then I am slowing the USB and Serial transmission down to their combined transmission rate. I'm guessing that won't be a problem, but now I'm also curious if there's a way to execute the two commands in parallel, waiting for the slowest one to finish before moving on. I know that isn't the most optimized, but is logically simple and could speed things up a bit.
 
Threading is the only way I know how to do that, or perhaps dump the messages into a pair of queues, and have each MIDI deliverer check their queues for work and dispatch the message. They can check their work via an interrupt timer.
That said, I think the 1ms time for a MIDI message would be longer than an interrupt routine should take.
And also, one may get way ahead of the other.

Honestly, from a music standpoint, a millisecond is pretty fast.
It probably takes a second at your very fastest to drag a finger up the whole 88 keys on a piano... and that's about 12ms per note.

I do send messages every few milliseconds, but most are "expression" messages sent using MIDI continuous controller messages (crescendo, decrescendo). Note changes are comparatively slower.

I recommend that you see how it works alternating between the two pipes, and optimize for speed later, after you've "proven the concept". Simple is good!
 
Honestly, from a music standpoint, a millisecond is pretty fast.
It probably takes a second at your very fastest to drag a finger up the whole 88 keys on a piano... and that's about 12ms per note.

I do send messages every few milliseconds, but most are "expression" messages sent using MIDI continuous controller messages (crescendo, decrescendo). Note changes are comparatively slower.

I recommend that you see how it works alternating between the two pipes, and optimize for speed later, after you've "proven the concept". Simple is good!

Hi David, thank you - and you're 100% right that I should just try it before I start worrying about it.

FWIW, my application is like a hakken continuum type controller (but a lot simpler) where I am sending constant information regarding pitch on a continuous pitch instrument. So I guess one of the things I'm avoid is a noticeable "stair stepping" while moving pitch.
 
Threading is the only way I know how to do that, or perhaps dump the messages into a pair of queues, and have each MIDI deliverer check their queues for work and dispatch the message. They can check their work via an interrupt timer.
That said, I think the 1ms time for a MIDI message would be longer than an interrupt routine should take.
And also, one may get way ahead of the other.

Honestly, from a music standpoint, a millisecond is pretty fast.
It probably takes a second at your very fastest to drag a finger up the whole 88 keys on a piano... and that's about 12ms per note.

I do send messages every few milliseconds, but most are "expression" messages sent using MIDI continuous controller messages (crescendo, decrescendo). Note changes are comparatively slower.

I recommend that you see how it works alternating between the two pipes, and optimize for speed later, after you've "proven the concept". Simple is good!

Just running some timing tests. A USB Midi pitchbend command takes 6 microseconds and a serial MIDI pitchbend command takes 1800 microseconds, so about 300 times slower. If the commands are done sequentially, the USB sampling rate could be compromised as much as 1/300th of it's potential speed.
 
USB MIDI is incredibly fast compared to regular serial MIDI. But the underlying USB protocol is quite complex, so the overall timing depends on many factors which are challenging to measure.

With both serial and USB, when you send a MIDI message from Teensy, it goes into a buffer until Teensy's hardware can actually transmit the data. With serial, the buffer works in a simple way. Each byte is moved to the UART which transmits 31250 bits per second, including a start and stop bit for each byte. Some serial ports have a small FIFO, so if the FIFO wasn't already filled when your code sent the message, its bytes might get copied into the FIFO right away. Otherwise it happens as the hardware is ready. But the (slow) speed is easy to estimate, pretty much determined only by the baud rate.

USB is far more complicated. Like with serial, your message goes into a buffer, which lets your program keep running. At 12 Mbit/sec (Teensy 3.x, LC, 2.0), up to 16 MIDI messages can fit into a single packet, and at 480 Mbit/sec (Teensy 4.x) up to 128 message can fit. If you call usbMIDI.send_now(), whatever messages you've written are turned into a USB packet and given to the USB hardware to send to your PC. If your message fills up the buffer, it is also immediately turned into a USB packet and given to the hardware. But if you don't send enough messages to fill up a USB package, and you don't call send_now(), the USB MIDI code on Teensy waits until the next USB 1ms frame (12 Mbit) or 125us micro-frame (480 Mbit) to create a USB packet. This adds some latency, but is allows for much better USB bandwidth utilization, because packing more messages into the same packet is far more efficient.

So if you want lowest possible latency, call usbMIDI.send_now() after you've written your USB message(s). But know that you are using the USB bandwidth less efficiently by doing this. If your code will send several messages, only call send_now() after the last one.

When Teensy turns your MIDI message(s) into a USB packet, that's still not the end of the story. All USB bandwidth is managed by the USB host (your PC). Teensy can't decide when to transmit the packet. That packet sits waiting until the USB host controller chip in your PC sends an IN token. Teensy's USB device controller automatically responds to the IN token by transmitting the packet (or it responds with a NAK token if no packet is waiting to transmit on the Teensy side - during normal idle times your PC is rapidly sending INs and getting NAKs from Teensy). So there is an unavoidable delay that depends on how quickly your PC's host controller sends those IN tokens. That delay can vary. At 12 Mbit speed, times in the 40 to 250us range are typical. Things are much faster at 480 Mbit, not only because of the higher bitrate but also because timing margins are much tighter in the USB spec for highspeed mode. If other bandwidth hungry devices like high res USB webcams are active on the same USB host controller, they can hog as much as 80% of the 1ms frame (12 Mbit) or 125us microframe (480 Mbit), during which time your PC's host controller can't send IN tokens to Teensy. However, a 12 Mbit/sec Teensy is plugged into a hub, it actually communicates with a transaction translator inside the hub, which then communicates with your PC at 480 Mbit, which does allow the hub to send IN tokens while the upstream 480 Mbit is busy. Some hubs have only a single TT, so if 2 or more slower USB devices are plugged into the same hub, only 1 can use 12 MBit at a time. Other hubs have a TT on every downstream port. So there are a *lot* of complicated factors that go into how quickly Teensy will get an IN token that prompts it to actually transmit the USB packet.

But the really bad news is the delays that can happen on the PC side once the packet is received. Windows often has little delay before it schedules whatever software is waiting for the delay to actually run. But the operating system scheduling delay can be as high as 16 ms if your system is busy with other work, or running in a lower power mode. Linux and MacOS are better, but they can also have delays running the waiting program if other software is consuming a lot of resources.

Typically USB MIDI latency works out to be much lower than serial MIDI. Of course the bandwidth is much higher and Teensy does utilize it very efficiently if you don't call send_now(). But if you want lowest latency, you definitely should call usbMIDI.send_now(). Just know you're giving up efficient transmit to get lower latency.

With serial there's nothing you can do. The baud rate is fixed at 31250. But at least it's simple to estimate how much time is needed until your message is sent.
 
Thanks, @Paul.
At a very high-level...
I send MIDI through serial (at 31250 bd) to an internal synth chip (VLSI's vs1053b) and to wired headphones, and the feeling is immediate.
I simultaneously send through USB to a Mac with Garageband or Ableton, and the latency is fairly noticeable, though playable.
I send using BLE MIDI through and Adafruit BLE Friend UART module to a synth on my phone, and both latency and throughput are quite challenged.

So, for me, while serial is technically slow, its simplicity makes it practically fast.

I would love to be able to use BLE MIDI effectively.
At present, I need to "dumb-down" the message stream to make it more playable - e.g. sending only a subset of the available CC messages.
Still a work-in-progress
 
My guess is nearly all of that PC synth latency is happening on the PC. You can use usbMIDI.send_now() to eliminate the tiny amount of latency that is under Teensy's control, but that is at most only the amount of time 1 byte takes to transfer over serial MIDI.

On the PC side, some of the latency may be the operating system scheduling for the software to run. But my blind guess would the audio output driver & hardware. Perhaps it has some buffering? Especially if the output involves wireless headphones, any wireless audio stream could be adding quite a lot of buffering so it can deal with data loss.
 
That makes sense, Paul.
My challenge in producing a MIDI wind controller will be to enable a great musical experience without requiring the user to mess with their PC/Mac much at all. Plug and Play for my users (or in fact, just Play).

I've been considering the option to go directly to bluetooth earphones or speakers. I know BLE earbuds are the new leading edge (AirPods, certain hearing aids)
I searched for "bluetooth audio" on the forum and have many hits, but minimal success.
Has this been clearly demonstrated to good effect with some add-on module?
- a Teensy-generated audio signal wirelessly driving Bluetooth or BLE earphones/speakers?

[I may be thread-hijacking... I could start - or continue - another thread.]
 
When dealing Bluetooth, 5 is the way to low latency, because it has even faster negotiation than 4 or LE, especially in crowded WiFi coexistance situations.
I've done some BLE MIDI in the past, which resulted in a dedicated BLE MIDI receiver that gave MIDI over USB to the host to obtain reasonable latency, because any OS does not deal BLE connections as efficiently as a dedicated central MCU does. Also, with this trick, it is possible to show a wireless controller to any OS as class compliant USB MIDI, wich is supported everywhere. The only cosmetical flaw is the dongle, but still the controller can be used directly without this, but at higher latency due to OS overhead.
 
Status
Not open for further replies.
Back
Top