Forum Rule: Always post complete source code & details to reproduce any issue!
Results 1 to 13 of 13

Thread: Teensy 4.0 As a Live Vocal Processor?

  1. #1

    Teensy 4.0 As a Live Vocal Processor?

    Assuming that Teensy 4.0 has audio processing capabilities that surpasses by far the capability of other Teensy versions, is anyone working on a vocal pitch shifter using non-granular formant methods? I'm looking for an ultra-compact pitch-correcting live harmonizing effects vocal processor for singing. Oh and let's throw in reverb in the mix as well.

  2. #2

    Lightbulb Teensy 4.0 As a Live Vocal Processor?

    Quote Originally Posted by ElectricBearSFO View Post
    Assuming that Teensy 4.0 has audio processing capabilities that surpasses by far the capability of other Teensy versions, is anyone working on a vocal pitch shifter using non-granular formant methods? I'm looking for an ultra-compact pitch-correcting live harmonizing effects vocal processor for singing. Oh and let's throw in reverb in the mix as well.
    Except for the reverb part, the rest is way beyond my skill set. Is anyone willing to take on this challenge?

  3. #3
    Junior Member
    Join Date
    Aug 2019
    Posts
    16
    Whilst the Teensy 4.0 is powerful, having tried something similar on an STM32H750 which is of similar performance, I think you will struggle. I've migrated to a Pi4 for high powered audio processing.

  4. #4
    Do you use the PI4 for real time audio processing? Is this through some OS, or bare metal? I feel like the teensy 4 is very well optimized for this (real time requires low latency). I wonder how the teensy stacks against the ADAU1701 or ADAU1452.

  5. #5
    Junior Member
    Join Date
    Aug 2019
    Posts
    16
    Quote Originally Posted by JayShoe View Post
    Do you use the PI4 for real time audio processing? Is this through some OS, or bare metal? I feel like the teensy 4 is very well optimized for this (real time requires low latency). I wonder how the teensy stacks against the ADAU1701 or ADAU1452.
    Hi Jay. Yes the Pi is used for real time audio. Two cores run Linux. I try to make one do all general housekeeping whilst the other focuses on input control - MIDI, HID, etc. The other two cores are (supposedly) isolated from Linux by the isolcpu command. I do see the occasional interrupt but nothing fatal. One core processes inputs whilst the other is reserved for output signal (re)construction.

    The original system used six STM32H750s is a 3 by 2 SPI connected mesh, but this had several problems so I was migrating to the STM32MP157 when the Pi4 came out so I changed to using that.

    Biggest advantage of the Pi is it's 64 bit so rather than constantly having to think whether to use single or double precision integer, or floating point, in the maths, everything is 64 bit integer. Each core is about 6 to 10 times faster than a H750 which is similar to a Teensy 4.0.

    However the thing missing from the Pi is SPDIF support and I did first use a WM8804 for this but I didn't like the clocking arrangements so was considering putting one H750 back in the system when I saw the Teensy4.0 release announcement. It hasn't arrived yet but I've already got a PCB respin done with one of these added to the system linked to the Pi by SPI to do all SPDIF and clock processing. It also has two more I2S outputs which are useful as secondary ports.

    Biggest problem with the Pi is its shutdown time - its excrutiatingly slow even by Linux standards. It simply hasn't been optimised as I assume nobody considers it important. But of course audio equipment is renowned for having the mains plugged pulled so working on the best way to get round this - either a UPS or some other form of allowing fast shutdown without data loss.

    Hope this helps whatever you are thinking about doing. I'd love to hear more details if you can.

    Mike

  6. #6
    Hi Mike,

    You've certainly peeked my interest with the PI4. I'm interested in all things audio. Live pro audio is my background, and I'm working on gadgets to create my own pro audio mixer. My goal is to create a series of DAC/ADC modules that can be plugged into a DSP (teensy 4? Pi4? FreeDSP) and create just the right combination of inputs and outputs for any given real time processing task. I'm desinging my modules to be DSP agnostic.

    I'm currently working on a Teensy 4 Audio Breakout that should help me towards this goal on the teensy. I also am playing with the FreeDSP Aurora.

    Jay

  7. #7
    Junior Member
    Join Date
    Aug 2019
    Posts
    16
    Quote Originally Posted by JayShoe View Post
    Hi Mike,

    You've certainly peeked my interest with the PI4. I'm interested in all things audio. Live pro audio is my background, and I'm working on gadgets to create my own pro audio mixer. My goal is to create a series of DAC/ADC modules that can be plugged into a DSP (teensy 4? Pi4? FreeDSP) and create just the right combination of inputs and outputs for any given real time processing task. I'm desinging my modules to be DSP agnostic.

    I'm currently working on a Teensy 4 Audio Breakout that should help me towards this goal on the teensy. I also am playing with the FreeDSP Aurora.

    Jay
    Yeah I've been interested in audio for about fifty years and was at Soundcraft for a time 20 years ago. Now I design stuff on contract but have been thinking about doing my own brand.

    Being DSP agnostic is good. Things move on constantly. Key thing is definition of the backplane to make sure it can do everything you need so make sure the backplane can handle far more traffic than you think you need.

    But I've never been keen on Xmos I'm afraid. A bit like Wolfson before them - more mass market and less pro audio. Also I prefer the microphone amplifiers right by the back of the XLRs, not at the end of a cable running from them where they can pick up digital noise, but maybe that's my own bias.

  8. #8
    Quote Originally Posted by MikeDB View Post
    Yeah I've been interested in audio for about fifty years and was at Soundcraft for a time 20 years ago. Now I design stuff on contract but have been thinking about doing my own brand.
    Well it's nice to meet you. I'm far from being a professional engineer. I'm self taught, just trying hard to get what it is I want. I'm glad we connected. Maybe we can help eachother out. We seem to have some similar projects. I'll post about my progress. I'm close to releasing the schematic and PCB images for the Audio Breakout. I might put one together first before I post. Not sure if I want to post it here, or on DiyAudio. Maybe both, I don't know yet! Trying to find the most helpful crew to bounce ideas off. :-)

  9. #9
    Senior Member Blackaddr's Avatar
    Join Date
    Mar 2017
    Location
    Canada
    Posts
    216
    I was working on a pitch shifter for the T3.6 that implements the Ocean's pitch shifting algorithm. It is a spin on the concept of the phase vocoder. The basic process is

    1) Use overlapping STFT to get the initial frequency/phase bins.
    2) Move each bin from it's original index to a new index based on the pitch scaling factor. For example, for octave up (scale factor 2). This will cause a sinusoid component to be created at the new desired frequency. Scale factors < 1.0 pitch down, > 1.0 pitch up.
    newBinIndex = round(oldBinIndex * 2.0f)
    3) The shifted sinusoid will have incorrect instantaneous phase so the phase is corrected based on the phase for the original bin, it's frequency and the scaling factor.
    4) Perform inverse STFT to get back to time domain, apply a synthesis windowing (hanning) function, then overlap add to the output.

    The main difference versus a traditional phase vocoder is you are not achieving pitch shifting by resampling, you are simply scaling the complex sinusoid components directly. Pitch shifting is very difficult, this method has serious drawbacks just like every other method.

    Here's why I paused the project.
    - The T3.6 is stuck using the old CMSIS library which has poor selection of FFT sizes. E.g. 512 and 2048 where 1024 is probably the best size for this application. My understanding is the T4 will use a much newer version of CMSIS that offers better FFT selection sizes.
    - The T3.6 is more than fast enough to perform the FFTs in real time. The problem is the phase correction steps require calculating a lot of cos() and sin() values which even using the CMSIS library is very slow. The T4 has a newer FPU with double precision that I'm hoping will allow far more acceleration of these computations.
    - The T3.6 code I had actually working was using > 90% of the T3.6 and the algorithm still need more tweaking (like IIR filtering to enhance tone, etc.).

    The T4 should provide more than enough power to implement the Ocean's algorithm with lots of processing power left to spare. I'm looking forward to completing the pitchshifter using the T4 once I get it working on myTGA Pro.

    My development branch for my pitch shfiter can be found in the BALibrary on a branch called 'feature/pitchshifter'. You can find the branch here. Take a look at the AudioEffectPitchShift class.

  10. #10
    Junior Member
    Join Date
    Aug 2019
    Posts
    16
    Quote Originally Posted by Blackaddr View Post
    I was working on a pitch shifter for the T3.6 that implements the Ocean's pitch shifting algorithm. It is a spin on the concept of the phase vocoder.
    Fair enough, but the OP did ask about running something far more complicated, probably using the Wavelet transform. The Teensy4.0 should be more than powerful enough for what you are doing and hopefully you'll get yours soon and make progress for which I hope you succeed.

    BTW, you mention having so calculate cos() and sin() values slowing you down. These are normally pre-calculated in an array as you need the same values time and time again if the Fourier transforms are all the same size. Or am I missing something about your algorithm ? I haven't looked at the library you mentioned yet.

  11. #11
    Senior Member+ MichaelMeissner's Avatar
    Join Date
    Nov 2012
    Location
    Ayer Massachussetts
    Posts
    3,085
    Quote Originally Posted by Blackaddr View Post
    I
    - The T3.6 is more than fast enough to perform the FFTs in real time. The problem is the phase correction steps require calculating a lot of cos() and sin() values which even using the CMSIS library is very slow. The T4 has a newer FPU with double precision that I'm hoping will allow far more acceleration of these computations.
    The Teensy 3.5 and 3.6 only support hardware single precision 32-bit floating point (i.e. float type). Double precision (i.e. double type) is done completely via software emulation. This means on the 3.5/3.6, you should:
    • Use float everywhere;
    • On floating point constants use the 'f' suffix. On the 3.5/3.6, teensydunio actually uses a compiler option to say all constants are single precision (which causes other problems), but it is a good habit to get into to use 'f' on any single precision floating point constant;
    • Use the single precision floating point libraries (i.e. cosf, not cos).


    Now, the Teensy 4.0 does have both double precision 64-bit and single precision 32-bit floating point hardware, so if you are moving exclusively to the Teensy 4, you can use the double precision math functions.

    However, in general, if you don't need more than 6 digits of accuracy, on most machines, it is faster to use single precision.

    However, there are machines (such as the PowerPC that I work on for my day job) where that is not the case, and double precision is faster than single precision for most operations (internally, the machine keeps everything in the 64-bit double format in registers, and the single precision instruction does the double precision instruction, and then rounds the value down to single precision, but keeps it in double precision format).

  12. #12
    Senior Member Blackaddr's Avatar
    Join Date
    Mar 2017
    Location
    Canada
    Posts
    216
    Quote Originally Posted by MikeDB View Post
    BTW, you mention having so calculate cos() and sin() values slowing you down. These are normally pre-calculated in an array as you need the same values time and time again if the Fourier transforms are all the same size. Or am I missing something about your algorithm ? I haven't looked at the library you mentioned yet.
    In this algorithm you are not using the same values over and over again so they can't be precomputed like for wave synthesis. You are taking the input complex phasor and multiplying in by another phase-shifed phasor where the argument of this complex sinusoid is a function of the current phase, the frequency and the scaling factor.

    Code:
    void AudioEffectPitchShift::m_ocean(float *inputFreq, float *outputFreq, float frameIndex, float pitchScale)
    {
        // zero the output buffer
        for (unsigned i=0; i<(2*SYNTHESIS_SIZE); i++) {
            outputFreq[i] = 0.0f;
        }
    
        float phaseAdjustFactor = -((2.0f*((float)(M_PI))*frameIndex)
                / (OVERLAP_FACTOR_F * FFT_OVERSAMPLE_FACTOR_F * SYNTHESIS_SIZE_F));
    
        for (unsigned k=1; k < SYNTHESIS_SIZE/2; k++) {
    
            float a = (float)k;
            // b = mka + 0.5
            // where m is the FFT oversample factor, k is the pitch scaling, a
            // is the original bin number
            float b = std::roundf( (FFT_OVERSAMPLE_FACTOR_F * pitchScale * a));
            unsigned b_int = (unsigned)(b);
    
            if (b_int < SYNTHESIS_SIZE/2) {
    
                // phaseAdjust = (b-ma) * phaseAdjustFactor
                float phaseAdjust = (b - (FFT_OVERSAMPLE_FACTOR_F * a)) * phaseAdjustFactor;
    
                float a_real = inputFreq[2*k];
                float a_imag = inputFreq[2*k+1];
    
                outputFreq[2*b_int]   = (a_real * arm_cos_f32(phaseAdjust)) - (a_imag * arm_sin_f32(phaseAdjust));
                outputFreq[2*b_int+1] = (a_real * arm_sin_f32(phaseAdjust)) + (a_imag * arm_cos_f32(phaseAdjust));
            }
        }
    }

  13. #13
    Junior Member
    Join Date
    Aug 2019
    Posts
    16
    Quote Originally Posted by Blackaddr View Post
    In this algorithm you are not using the same values over and over again so they can't be precomputed like for wave synthesis. You are taking the input complex phasor and multiplying in by another phase-shifed phasor where the argument of this complex sinusoid is a function of the current phase, the frequency and the scaling factor.
    Ok took me a while but try this. It won't be absolutely correct but should give you a strategy.

    For k=1 calculate phaseadjust as you are doing so you need a starting cos() and sin() values.

    Now if you rewrite the phaseadjust formula as

    phaseadjust = ((FFT_OVERSAMPLE_FACTOR_F * pitchScale * a) - (FFT_OVERSAMPLE_FACTOR_F * a)) * phaseAdjustFactor;

    This reduces to

    phaseadjust = ((FFT_OVERSAMPLE_FACTOR_F * (pitchScale - 1) * a) * phaseAdjustFactor;

    Hence you are actually calculating the cos() and sin() of a number that is stepping up
    .... 2C, 3C, 4C where C = FFT_OVERSAMPLE_FACTOR_F * (pitchScale - 1) * phaseAdjustFactor

    There are fast calculations for all of these that don't involve finding cos() or sin() again.
    For example
    cos(2C) = cos^2(C) - sin^2(C)
    sin(3C) = 3 * sin(C) - 4 * sin^3(C)
    .....

    There are derivable formulae for any multiplier that is always based on just the single cos(A) and sin(A) value. What you probably need to do is program in how these formulae are derived rather than a formula for each value of a (or k).

    Hope this helps - it will be much faster.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •