After seeing this wonderful tutorial video on the audio functionality of the teensy, I thought it might be a useful feature to expose the inverse FFT functions in the library. This potentially opens up the possibility to create frequency-based effects (such as the phase vocoder for example).
iFFT API in audio library?
- Thread starter johnty
- Start date
PaulStoffregen
Well-known member
I like this idea, but I have some practical questions.
Would these applications work with the magnitude-only output we currently have? Or do they require the full complex number FFT output?
For things like FFT-based filters & effect, should the entire implementation be inside the audio library? If we're going to provide data to Arduino sketches, there should be a good reason. Keeping the signal processing inside the library allows for predictable latency.
Would these applications work with the magnitude-only output we currently have? Or do they require the full complex number FFT output?
For things like FFT-based filters & effect, should the entire implementation be inside the audio library? If we're going to provide data to Arduino sketches, there should be a good reason. Keeping the signal processing inside the library allows for predictable latency.
For the first part, yes - we would need the complex numbers to properly resynthesize the result back into the time domain, as here's where the phase information is critical (indeed, this is one of the fundamental components of the phase vocoder effect.)
As to the second part, I think what you propose makes sense - so perhaps any effect would be written at the library level, without any need for direct manipulation of intermediate data in the Arduino sketch for the sake of performance.
adrian
Well-known member
I'm a first year maths kind of a person, but I'm getting the broad outline of things ... what you are talking about could allow convolution. I use guitar cabinet convolution all the time, on my beast i7, going direct from my amp head into into the adc.
I have a couple of nice cabinets as well (one marshall and one peavey) in 'the closet', but I like them only for lead guitar, where you need a bit of wide eyed speaker grit (not to mention power amp tube compression) .... anyway, to finish rambling, a mcu sized convolution engine would be like 'wow'! U could just have one impulse on there "chose your poison' (red wire impulses are great ... lilke, really great ...) and put a highish z input on front of your teensy, and bam ....
lots of amps have emulated line outs as they are called, and most of them suck ass, since most of them are passive resistor networks, i guess .... if you could 'realtime' fft / convolution, well, my goodness.... i think line six do use dsp for their cab outs, but they also use it for the 'valves' tooo (aka amp sim) .... that is yuccky (but I suppose you could even do that with convolution???!!!) .... convolving my Laney's line out ....SWEET .... mullards never tasted better than through my vst 'convolver' .... cabinets and air have their place, but in the studio, for 'clean' rythm .... give me dsp. At least as a flavour option if nothing else (plus its a bit quiter than 11 on the power tubes, with the marshall cab choking out barf).
And for bass, di is near mandatory, and for me (again) a speaker or amp sim convolver is pretty essential for bass ....
AND what about a REVERB... I've never been superimpressed with convolution reverbs, but lots of people like them.
What can I do, other than learn a lot of stuff!! to help. My C++ is pretty basic, and so is my maths ...maybe the best thing I can do is give you encouragement!!! Money?? first born??
I have a couple of nice cabinets as well (one marshall and one peavey) in 'the closet', but I like them only for lead guitar, where you need a bit of wide eyed speaker grit (not to mention power amp tube compression) .... anyway, to finish rambling, a mcu sized convolution engine would be like 'wow'! U could just have one impulse on there "chose your poison' (red wire impulses are great ... lilke, really great ...) and put a highish z input on front of your teensy, and bam ....
lots of amps have emulated line outs as they are called, and most of them suck ass, since most of them are passive resistor networks, i guess .... if you could 'realtime' fft / convolution, well, my goodness.... i think line six do use dsp for their cab outs, but they also use it for the 'valves' tooo (aka amp sim) .... that is yuccky (but I suppose you could even do that with convolution???!!!) .... convolving my Laney's line out ....SWEET .... mullards never tasted better than through my vst 'convolver' .... cabinets and air have their place, but in the studio, for 'clean' rythm .... give me dsp. At least as a flavour option if nothing else (plus its a bit quiter than 11 on the power tubes, with the marshall cab choking out barf).
And for bass, di is near mandatory, and for me (again) a speaker or amp sim convolver is pretty essential for bass ....
AND what about a REVERB... I've never been superimpressed with convolution reverbs, but lots of people like them.
What can I do, other than learn a lot of stuff!! to help. My C++ is pretty basic, and so is my maths ...maybe the best thing I can do is give you encouragement!!! Money?? first born??
Last edited:
adrian, i think you raise some interesting points and have made me rethink this from a performance issue: i had dome some rough and dirty tests (about a year ago, mind you) and seemed to have concluded that there was "enough power" to do a couple of FFTs at a time, which means these effects are feasible. however, it would be good to actually confirm this, especially since a lot of these effects would require one to overlap and add, which means doing on the order of 4-8 FFTs (and inverse) for every buffer.
the next thing i'd like to do is dig into the library, and do some more rigorous testing of the functions, and see if its even possible to do that many FFTs at a time. (gut feeling says yes... and compared doing convolutions in time for the effects you mentioned, frequency based ones are much much more computationally efficient. so this is a good area to get into.)
if only i had more time... what would i give for that? money?? first born??
*one more ninja edit*: one thing in particular i'm not certain about my initial test, was that i was simply synthesizing a few sine waves, and had 4 1024 point FFTs in the code. then in the loop i was checking for the .available() for each FFT, and then seeing if the whole thing could run without glitches. having a closer look at the library code this evening, my current interpretation is that if the FFT was being computed too slowly, we would simply miss chunks of data in between. therefore, my original concept of what was deemed "enough performance" seems totally off.
the next thing i'd like to do is dig into the library, and do some more rigorous testing of the functions, and see if its even possible to do that many FFTs at a time. (gut feeling says yes... and compared doing convolutions in time for the effects you mentioned, frequency based ones are much much more computationally efficient. so this is a good area to get into.)
if only i had more time... what would i give for that? money?? first born??
*one more ninja edit*: one thing in particular i'm not certain about my initial test, was that i was simply synthesizing a few sine waves, and had 4 1024 point FFTs in the code. then in the loop i was checking for the .available() for each FFT, and then seeing if the whole thing could run without glitches. having a closer look at the library code this evening, my current interpretation is that if the FFT was being computed too slowly, we would simply miss chunks of data in between. therefore, my original concept of what was deemed "enough performance" seems totally off.
Last edited:
adrian
Well-known member
Hmm, I'm going to have to dive into the audio library...
I found a really clear explanation of convolution here (I think its ok to link to it, copyright-wise) ....Hopefully its correct!!
http://www.dspguide.com/ch18/2.htm
I'm guess you know all this already ... but its new to me.
The whole process looks simple enough from a high level but I can see that the computational power needed is a big 'if'. It will take me some time to get up to speed on the audio library (but it doesn't look too hard, esp with the gui thing!!).
Obviously, the fft routine seems like a place to start! Like you say, the real and imaginary components appear to be needed (according to the link), and if I understand it, the teensy fft library only gives real / magnitude ... ?
Basically (at the high level) an input audio block is transformed, as is the impulse's samples (the impulses for guitar cabs are very short, mercifully) ...padding is applied to the transforms, and the two are multiplied together in the frequency domain, to give a new set of samples. Then, using ifft, the new set of samples is written back to the time domain ...But then a bit of add-overlap magic is necessary to properly 'reform' an output stream.
When you say that we would need 4 to 8 ffts and iffts per buffer, I guess what you are talking about is dicing up a 128 member audio block into 4 (lets say), and transforming / convolving each 32 samples with an impulse ...the reason for this is to have enough data on hand for overlap-adding ...BUT ...if the impulse was short relative to the audio block, would you need to dice into 3, rather than 4 (and lets forget about stereo!!). Obviously I don't get how to add-overlap, but wouldn't 3 suffice?
There is some code associated with the link above but I think it might be copyright.
Anyway, to do all this in audio 'real time' seems a big ask for the teensy
I think I'll take the first step of learning the audio library now .... see if I can transform a sine wave!! thats going to be hard enough for now! I was going to muck around with OSC (a bit more my level tbh) but I'll see how I fare. I have a teensy audio board somewhere, even!
I found a really clear explanation of convolution here (I think its ok to link to it, copyright-wise) ....Hopefully its correct!!
http://www.dspguide.com/ch18/2.htm
I'm guess you know all this already ... but its new to me.
The whole process looks simple enough from a high level but I can see that the computational power needed is a big 'if'. It will take me some time to get up to speed on the audio library (but it doesn't look too hard, esp with the gui thing!!).
Obviously, the fft routine seems like a place to start! Like you say, the real and imaginary components appear to be needed (according to the link), and if I understand it, the teensy fft library only gives real / magnitude ... ?
Basically (at the high level) an input audio block is transformed, as is the impulse's samples (the impulses for guitar cabs are very short, mercifully) ...padding is applied to the transforms, and the two are multiplied together in the frequency domain, to give a new set of samples. Then, using ifft, the new set of samples is written back to the time domain ...But then a bit of add-overlap magic is necessary to properly 'reform' an output stream.
When you say that we would need 4 to 8 ffts and iffts per buffer, I guess what you are talking about is dicing up a 128 member audio block into 4 (lets say), and transforming / convolving each 32 samples with an impulse ...the reason for this is to have enough data on hand for overlap-adding ...BUT ...if the impulse was short relative to the audio block, would you need to dice into 3, rather than 4 (and lets forget about stereo!!). Obviously I don't get how to add-overlap, but wouldn't 3 suffice?
There is some code associated with the link above but I think it might be copyright.
Anyway, to do all this in audio 'real time' seems a big ask for the teensy
I think I'll take the first step of learning the audio library now .... see if I can transform a sine wave!! thats going to be hard enough for now! I was going to muck around with OSC (a bit more my level tbh) but I'll see how I fare. I have a teensy audio board somewhere, even!
Last edited:
adrian
Well-known member
I'm actually looking at this myself ...sure am learning a lot, being a noob and all. I've got to the place where I'm thinking that I'll need 4 FFTs minimum, followed by 4 iffts, and a lot of copying this from that etc. So, big questions as to how teensy would cope ...But I'm learning lots of cool stuff anyway. The audio library objects are just great, and I'm making progress. I've found all the arm complex math functions, how the cfft function works (q15 etc etc the workspace struct / inplace processing) and I've worked out the analyse256 audio library. And I've started writing my own convolution object!
here is my plan ....
initially, read in a 512 sample impulse from SD, split it up into 4, window them, then pad them ... perform ffts on the 4 sets ....these are 'static global arrays' ..... then in a looop / update convolver, feed in live line audio from the audio library adc, 128 samples at a time. window and pad, and transform the 128 live audio and then complex mutliply each of the impulse static global arrays / transforms with each live audio line transform. Reassemble by adding (and decimate?!) and output a 128 sample convolved block. I'm still checking I have the theory right ... the programming doesn't look all that hard given I can stand on the shoulders of giants in the audio library area ....
Other than not knowing what I am doing, the big problem is the teensy's speed .... I had thought that a 128 sample live block could be convolved with a 128 sample impulse, but the impulses I have are a little longer, unfortunately. In fact it would be better to use 2048 length impulses ... Anyway, as a result, the splitting overlapping etc and the number of ffts etc is more complicated (possibly more complicated than my understanding) .... I wonder if buffering a bit of live audio data is a good idea??? BUt then, you start moving into latency issues.... Hmm. I might start working on two blocks at a time, like in the 256 analyse library
Anyway, its all fun until someone loses an eye
here is my plan ....
initially, read in a 512 sample impulse from SD, split it up into 4, window them, then pad them ... perform ffts on the 4 sets ....these are 'static global arrays' ..... then in a looop / update convolver, feed in live line audio from the audio library adc, 128 samples at a time. window and pad, and transform the 128 live audio and then complex mutliply each of the impulse static global arrays / transforms with each live audio line transform. Reassemble by adding (and decimate?!) and output a 128 sample convolved block. I'm still checking I have the theory right ... the programming doesn't look all that hard given I can stand on the shoulders of giants in the audio library area ....
Other than not knowing what I am doing, the big problem is the teensy's speed .... I had thought that a 128 sample live block could be convolved with a 128 sample impulse, but the impulses I have are a little longer, unfortunately. In fact it would be better to use 2048 length impulses ... Anyway, as a result, the splitting overlapping etc and the number of ffts etc is more complicated (possibly more complicated than my understanding) .... I wonder if buffering a bit of live audio data is a good idea??? BUt then, you start moving into latency issues.... Hmm. I might start working on two blocks at a time, like in the 256 analyse library
Anyway, its all fun until someone loses an eye
I guess you assume that
that is, scaling 1/N is only done once (usually only on ifft)
The arm integer FFTs, however, have to do scaling 1/N on both the forward AND inverse fft (to avoid overflow)
Consequence
for a 256 point fft or ifft, an input in Q15 (format 1.15 or 15 bit mantissa) becomes Q7 (format 9.7 or 7 bit mantissa)
so you get after the ifft(fft(x) operation a very small number.
while you could rescale before ifft, it is really the dynamic range that suffers
a Q31 point fft may be better but is obviously much slower, also because the cortex m4 has optimized instructions for 16 bit.
If you have not done yet, I suggest to play a little bit around with ifft(fft) for tonals embedded in different noise levels to see what happens to the tonal after the fft-ifft operation
Code:
x=ifft(fft(x))The arm integer FFTs, however, have to do scaling 1/N on both the forward AND inverse fft (to avoid overflow)
Consequence
for a 256 point fft or ifft, an input in Q15 (format 1.15 or 15 bit mantissa) becomes Q7 (format 9.7 or 7 bit mantissa)
so you get after the ifft(fft(x) operation a very small number.
Code:
x_out = x_in/256a Q31 point fft may be better but is obviously much slower, also because the cortex m4 has optimized instructions for 16 bit.
If you have not done yet, I suggest to play a little bit around with ifft(fft) for tonals embedded in different noise levels to see what happens to the tonal after the fft-ifft operation
Hello all, my first post. I know this is kind of an old thread, but I think, there is a solution for low power devices like this so far.
If someone can implement this for this board it will be awesome!. Is called "Partitioned Convolution", here is a white paper:
View attachment REAL-TIME PARTITIONED CONVOLUTION ALGORITHMS.pdf
Edit: Don't know much about it, but t looks like it processes one large impulse response, one block of samples at a time, with delays for every other block, so the whole process doesn't have to be computed in real time, only the first couple of blocks, anyway, go figure..
If someone can implement this for this board it will be awesome!. Is called "Partitioned Convolution", here is a white paper:
View attachment REAL-TIME PARTITIONED CONVOLUTION ALGORITHMS.pdf
Edit: Don't know much about it, but t looks like it processes one large impulse response, one block of samples at a time, with delays for every other block, so the whole process doesn't have to be computed in real time, only the first couple of blocks, anyway, go figure..
Last edited:
just scanned through that paper - my limited understanding is this: in general, convolutions are typically implemented in the frequency domain as its more efficient. as such it uses forward and inverse FFTs.
the paper presents a more efficient way to do a convolution by breaking the problem down into smaller, concurrent steps taking advantage of a multi-threaded processing environment. not sure if this makes it suitable for implementation on a single core embedded system, however...
the paper presents a more efficient way to do a convolution by breaking the problem down into smaller, concurrent steps taking advantage of a multi-threaded processing environment. not sure if this makes it suitable for implementation on a single core embedded system, however...