Using Serial and USB Audio Simultaniously

[benjaminkirkbride]

Hello all, thank you for considering my question.

I am trying to provide audio in to a Teensy 4.0, do some FFT to it, then send the output back over serial to the same host that is connected via USB.

I expected this to be relatively simple, but after some troubleshooting and getting the USB input to work (I was doing line-in prior), I see that now I (believe I) am unable to use serial. I assume this is due to having to put the board into usb mode "audio", but am not entirely certain.

I did some research on this topic and found a post here that seems to be about the same issue, but I found the advice given to be unclear: https://forum.pjrc.com/threads/61709...rial-and-Audio

The advice appears to be "use MIDI" or "Edit some TeensyDuino files". I tried including a basic MIDI implementation into my code, but that didn't seem to make any sort of difference. I am also not certain how to edit the board files exactly, and even if I was that is a solution I would really like to avoid as it would require modifying library files for anyone else in the future to be able to use my project.

First time using this forum and not particularly experienced with any of this in general, so please go easy on me

[PaulStoffregen]

If you're not yet familiar with audio on Teensy, start with this tutorial.

https://www.pjrc.com/store/audio_tutorial_kit.html

The tutorial covers Fourier Transform starting on page 24. There's also a full demo video, so you can see & hear it.

The first parts of section 2, starting on page 8, explain the design tool. Hopefully you can see how to create your own designs. Here's a very simple one with USB input and FFT. I just copied the one from the tutorial, but replaced the SD card, waveform and sample clip with the live USB input. (for the sake of a tutorial, known input data allows for a learning experience where the tutorial can show the expected results and not have the unknown issues of whether your PC transmits data).



Now there is one gotcha with USB input, which is documented in "Notes" in the design tool documentation, that you also need at least 1 other normal input or output to make audio work, since USB can't cause the library to update. Teensyduino 1.58 will adds detection for this and uses a timer, but if you have 1.57 or earlier, you'll need to have at least 1 I2S input or output (even if no audio shield is actually connected).

To actually use Serial and Audio, you need to select it in the Tools > USB Type menu. Here's a screenshot so you can see exactly what to click.



On that other thread, the question was how to do this without MIDI, since the only option with Serial and Audio in that menu also has MIDI. You don't actually have to use the MIDI interface, but it will be there (sitting unused).

Here is a quick program I tried just now, basically just adapted from Part 3-2 of the tutorial (eg, click File > Examples > Audio > Tutorial for that code).

#include <Audio.h>
#include <Wire.h>
#include <SPI.h>
#include <SD.h>
#include <SerialFlash.h>

// GUItool: begin automatically generated code
AudioInputUSB            usb1;           //xy=203,125
AudioMixer4              mixer1;         //xy=379,133
AudioAnalyzeFFT1024      fft1024_1;      //xy=552,194
AudioOutputI2S           i2s1;           //xy=556,120
AudioConnection          patchCord1(usb1, 0, mixer1, 0);
AudioConnection          patchCord2(usb1, 1, mixer1, 1);
AudioConnection          patchCord3(mixer1, 0, i2s1, 0);
AudioConnection          patchCord4(mixer1, 0, i2s1, 1);
AudioConnection          patchCord5(mixer1, fft1024_1);
AudioControlSGTL5000     sgtl5000_1;     //xy=394,241
// GUItool: end automatically generated code

void setup() {
  Serial.begin(9600);
  AudioMemory(10);
  sgtl5000_1.enable();
  sgtl5000_1.volume(0.5);
  mixer1.gain(0, 0.5);
  mixer1.gain(1, 0.5);
  mixer1.gain(2, 0.0);
  mixer1.gain(3, 0.0);
}

void loop() {
  // print Fourier Transform data to the Arduino Serial Monitor
  if (fft1024_1.available()) {
    Serial.print("FFT: ");
    for (int i = 0; i < 30; i++) { // 0-25  -->  DC to 1.25 kHz
      float n = fft1024_1.read(i);
      printNumber(n);
    }
    Serial.println();
  }
}

void printNumber(float n) {
  if (n >= 0.004) {
    Serial.print(n, 3);
    Serial.print(" ");
  } else {
    Serial.print("   -  "); // don't print "0.00"
  }
  /*
    if (n > 0.25) {
    Serial.print("***** ");
    } else if (n > 0.18) {
    Serial.print(" ***  ");
    } else if (n > 0.06) {
    Serial.print("  *   ");
    } else if (n > 0.005) {
    Serial.print("  .   ");
    }
  */
}

When I run this on my PC and open the serial monitor, it prints these blank lines rapidly.



And when I use the sound control panel to switch my PC's output to the Teensy audio device (many PCs switch automatically, but I must have disabled automatic switching at some point), this is the FFT output (or at least 1 quick moment as it prints a lot of stuff scrolling rapidly)



If you just copy this code into Arduino, set Tools > USB Type properly, and upload to your Teensy, and then when the audio device appears to your PC, play some music, you should get a similar FFT result in the serial monitor. Easy, right?

[benjaminkirkbride]

Thanks so much Paul!

For posterity, my issue was stemming from using arduino-cli and not using the right

usb

argument. Turns out I needed to be using

arduino-cli compile --export-binaries -b teensy:avr:teensy40:usb=serialmidiaudio .

did the trick!

In addition, I'm running into some issues now where I have to restart pulseaudio for the teensy to be detected as an output device. Guessing that this is a problem with my setup though, so I'll work through that on my own.

Thanks again!

[benjaminkirkbride]

This is what PulseAudio thinks the Teensy is prior to restarting PulseAudio. Restarting the Teensy does not change this.

Card #7
        Name: alsa_card.usb-Teensyduino_Teensy_MIDI_Audio_6705240-02
        Driver: module-alsa-card.c
        Owner Module: 28
        Properties:
                alsa.card = "5"
                alsa.card_name = "Teensy MIDI/Audio"
                alsa.long_card_name = "Teensyduino Teensy MIDI/Audio at usb-0000:0d:00.3-1, high speed"
                alsa.driver_name = "snd_usb_audio"
                device.bus_path = "pci-0000:0d:00.3-usb-0:1:1.2"
                sysfs.path = "/devices/pci0000:00/0000:00:08.1/0000:0d:00.3/usb3/3-1/3-1:1.2/sound/card5"
                udev.id = "usb-Teensyduino_Teensy_MIDI_Audio_6705240-02"
                device.bus = "usb"
                device.vendor.id = "16c0"
                device.vendor.name = "Van Ooijen Technische Informatica"
                device.product.id = "048a"
                device.product.name = "Teensy MIDI/Audio"
                device.serial = "Teensyduino_Teensy_MIDI_Audio_6705240"
                device.string = "5"
                device.description = "Teensy MIDI/Audio"
                module-udev-detect.discovered = "1"
                device.icon_name = "audio-card-usb"
        Profiles:
                input:analog-stereo: Analog Stereo Input (sinks: 0, sources: 1, priority: 65, available: yes)
                input:iec958-stereo: Digital Stereo (IEC958) Input (sinks: 0, sources: 1, priority: 55, available: yes)
                off: Off (sinks: 0, sources: 0, priority: 0, available: yes)
        Active Profile: input:analog-stereo
        Ports:
                analog-input: Analog Input (type: Analog, priority: 10000, latency offset: 0 usec, availability unknown)
                        Part of profile(s): input:analog-stereo
                iec958-stereo-input: Digital Input (S/PDIF) (type: SPDIF, priority: 0, latency offset: 0 usec, availability unknown)
                        Part of profile(s): input:iec958-stereo

This is what I get after:

Card #1
        Name: alsa_card.usb-Teensyduino_Teensy_MIDI_Audio_6705240-02
        Driver: module-alsa-card.c
        Owner Module: 7
        Properties:
                alsa.card = "5"
                alsa.card_name = "Teensy MIDI/Audio"
                alsa.long_card_name = "Teensyduino Teensy MIDI/Audio at usb-0000:0d:00.3-1, high speed"
                alsa.driver_name = "snd_usb_audio"
                device.bus_path = "pci-0000:0d:00.3-usb-0:1:1.2"
                sysfs.path = "/devices/pci0000:00/0000:00:08.1/0000:0d:00.3/usb3/3-1/3-1:1.2/sound/card5"
                udev.id = "usb-Teensyduino_Teensy_MIDI_Audio_6705240-02"
                device.bus = "usb"
                device.vendor.id = "16c0"
                device.vendor.name = "Van Ooijen Technische Informatica"
                device.product.id = "048a"
                device.product.name = "Teensy MIDI/Audio"
                device.serial = "Teensyduino_Teensy_MIDI_Audio_6705240"
                device.string = "5"
                device.description = "Teensy MIDI/Audio"
                module-udev-detect.discovered = "1"
                device.icon_name = "audio-card-usb"
        Profiles:
                input:analog-stereo: Analog Stereo Input (sinks: 0, sources: 1, priority: 65, available: yes)
                input:iec958-stereo: Digital Stereo (IEC958) Input (sinks: 0, sources: 1, priority: 55, available: yes)
                output:analog-stereo: Analog Stereo Output (sinks: 1, sources: 0, priority: 6500, available: yes)
                output:analog-stereo+input:analog-stereo: Analog Stereo Duplex (sinks: 1, sources: 1, priority: 6565, available: yes)
                output:analog-stereo+input:iec958-stereo: Analog Stereo Output + Digital Stereo (IEC958) Input (sinks: 1, sources: 1, priority: 6555, available: yes)
                output:iec958-stereo: Digital Stereo (IEC958) Output (sinks: 1, sources: 0, priority: 5500, available: yes)
                output:iec958-stereo+input:analog-stereo: Digital Stereo (IEC958) Output + Analog Stereo Input (sinks: 1, sources: 1, priority: 5565, available: yes)
                output:iec958-stereo+input:iec958-stereo: Digital Stereo Duplex (IEC958) (sinks: 1, sources: 1, priority: 5555, available: yes)
                off: Off (sinks: 0, sources: 0, priority: 0, available: yes)
        Active Profile: output:analog-stereo
        Ports:
                analog-input: Analog Input (type: Analog, priority: 10000, latency offset: 0 usec, availability unknown)
                        Part of profile(s): input:analog-stereo, output:analog-stereo+input:analog-stereo, output:iec958-stereo+input:analog-stereo
                iec958-stereo-input: Digital Input (S/PDIF) (type: SPDIF, priority: 0, latency offset: 0 usec, availability unknown)
                        Part of profile(s): input:iec958-stereo, output:analog-stereo+input:iec958-stereo, output:iec958-stereo+input:iec958-stereo
                analog-output: Analog Output (type: Analog, priority: 9900, latency offset: 0 usec, availability unknown)
                        Part of profile(s): output:analog-stereo, output:analog-stereo+input:analog-stereo, output:analog-stereo+input:iec958-stereo
                iec958-stereo-output: Digital Output (S/PDIF) (type: SPDIF, priority: 0, latency offset: 0 usec, availability unknown)
                        Part of profile(s): output:iec958-stereo, output:iec958-stereo+input:analog-stereo, output:iec958-stereo+input:iec958-stereo

[benjaminkirkbride]

Restarting the Teensy reverts back to Pulse thinking that it is only an input device.

[benjaminkirkbride]

Okay, two better workarounds have been established.

Again, restarting PulseAudio does work, the issue is that this also disconnects any playback sources, so they have to be restarted (browsers, Spotify, etc); very inconvenient.

Workaround 1 unloads every sound card and then reloads them all. This (for me at least) results in a second or so of interrupted audio, but all playbacks survive, and the sources and sinks appear to maintain their connections somehow. This may not be consistent though. Still better than having to restart every playback source:

pacmd unload-module module-alsa-card && pacmd load-module module-alsa-card && pacmd unload-module module-udev-detect && pacmd load-module module-udev-detect

Workaround 2 is much less disruptive, but is also harder to do (though it could certainly be scripted):

1. determine the "Module Owner" of the Teensy:

pactl list | grep -A 3 'Teensy' | grep 'Owner Module'

Then use that in place of unloading "module-alsa-card" like so:

pacmd unload-module $OWNER_MODULE && pacmd unload-module module-udev-detect && pacmd load-module module-udev-detect

Workaround 2 has one issue, it seems to bring along for the ride a bunch of extra duplicate instances of other cards. This could probably be worked around in other ways, but honestly having to do this is rare enough that I'll probably just use workaround 1.

Further consideration of a more holistic solution to this problem would be interesting to hear.

[benjaminkirkbride]

So it appears that PulseAudio will occasionally "forget"/reset the Teensy card config, necessitating running the above again. Very annoying.