DD4WH
Well-known member
Realtime audio processing with floating point math & new CMSIS lib in the Teensy 3.5
I would like to use the floating point capabilities of the new Teensy 3.5 for faster and more accurate audio processing in real time. This will be the basis for audio-processing in a SoftwareDefinedRadio (like the Teensy SDR or mcHF) which involves a whole lot of FIR and IIR filters with decimation and interpolation and also some complex FFTs. [what I mean with "more accurate": with fixed point math for example, real time decimation and interpolation of audio with the Teensy 3.2 was not possible --> audio was distorted, same with FFT & IFFT]
However, the audio lib uses 16-bit fixed point for all the objects. This means, one cannot use the new FPU power of the Teensy 3.5 for audio processing. So, I looked for a way to use floating point math for all audio processing.
I tried the following script: it takes one audio block with 128 int16_t from the LINE INPUT by using the recordqueue-object, converts it from int16_t to float32_t, does all the desired audio processing in float32_t with the NEW ARM CMSIS lib (see Duffs description of how to use the new CMSIS lib for Teensyduino --> HERE and additionally add one file --> HERE), converts the audio to int16_t again and gives it back to the headphone output. It took me some hours to figure it out, but it works, even with quite a large FIR lowpass filter with 200 taps at 44118sps! I can not hear any decrease in audio quality, which is very promising, but requires rigorous testing! Ray has similar experience with nice audio using floating point FFT/IFFT pass thru --> THREAD.
My questions are:
- is my implementation of the recordqueue and playqueue objects correct and processor-efficient (lines 388-397)? Or do I need to alter something in order to be faster and accurate?
- the code takes one sample at a time (lines 388ff), processes it and plays the sample back (lines 426ff). Is there a way to take several samples, process them and then playback several samples (put them into the playqueue object one after the other)? That would be needed for a 1024point FFT, for example.
- the memory and processor usage of the audio objects only is displayed, so it stays at 0.8% all the time. How can I measure the processor usage for all that floating point processing?
Thanks in advance for your comments!
Frank DD4WH
I would like to use the floating point capabilities of the new Teensy 3.5 for faster and more accurate audio processing in real time. This will be the basis for audio-processing in a SoftwareDefinedRadio (like the Teensy SDR or mcHF) which involves a whole lot of FIR and IIR filters with decimation and interpolation and also some complex FFTs. [what I mean with "more accurate": with fixed point math for example, real time decimation and interpolation of audio with the Teensy 3.2 was not possible --> audio was distorted, same with FFT & IFFT]
However, the audio lib uses 16-bit fixed point for all the objects. This means, one cannot use the new FPU power of the Teensy 3.5 for audio processing. So, I looked for a way to use floating point math for all audio processing.
I tried the following script: it takes one audio block with 128 int16_t from the LINE INPUT by using the recordqueue-object, converts it from int16_t to float32_t, does all the desired audio processing in float32_t with the NEW ARM CMSIS lib (see Duffs description of how to use the new CMSIS lib for Teensyduino --> HERE and additionally add one file --> HERE), converts the audio to int16_t again and gives it back to the headphone output. It took me some hours to figure it out, but it works, even with quite a large FIR lowpass filter with 200 taps at 44118sps! I can not hear any decrease in audio quality, which is very promising, but requires rigorous testing! Ray has similar experience with nice audio using floating point FFT/IFFT pass thru --> THREAD.
Code:
// Use queue for input AND output
// to allow for floating point audio processing
//
// Frank DD4WH 2016_10_24
//
// this is a setup to test whether we can do real time audio processing
// with the Teensy 3.5 in floating point math
// it takes the LINE INPUT audio, converts it from int16_t to float32_t,
// does the audio processing in float32_t with the NEW ARM CMSIS lib (see https://forum.pjrc.com/threads/38325-Excellent-results-with-Floating-Point-FFT-IFFT-Processing-and-Teensy-3-6?p=119797&viewfull=1#post119797),
// and see here: https://forum.pjrc.com/threads/38325-Excellent-results-with-Floating-Point-FFT-IFFT-Processing-and-Teensy-3-6?p=120177&viewfull=1#post120177
// . . . converts the audio back to int16_t and gives it back to the headphone output
//
// QUESTIONS TO TEST
// - what is the processor load ?
// - how is the audio quality ?
// - can we do decimation / interpolation in 32-bit floating point in realtime without noticable audio loss ?
//
// MIT licence
#include <Audio.h>
#include <Wire.h>
#include <SPI.h>
#include <SD.h>
#include <Metro.h>
#include "font_Arial.h"
#include <ILI9341_t3.h>
#define BACKLIGHT_PIN 0
#define TFT_DC 20
#define TFT_CS 21
#define TFT_RST 32 // 255 = unused. connect to 3.3V
#define TFT_MOSI 7
#define TFT_SCLK 14
#define TFT_MISO 12
ILI9341_t3 tft = ILI9341_t3(TFT_CS, TFT_DC, TFT_RST, TFT_MOSI, TFT_SCLK, TFT_MISO);
Metro five_sec=Metro(500); // Set up a 0.5 second Metro
// this audio comes from the codec by I2S2
AudioInputI2S i2s_in;
//AudioInputI2S i2s_in_R;
AudioRecordQueue Q_in;
AudioPlayQueue Q_out;
AudioOutputI2S i2s_out;
AudioConnection patchCord1(i2s_in, 0, Q_in, 0);
AudioConnection patchCord3(Q_out, 0, i2s_out, 0);
AudioConnection patchCord4(Q_out, 0, i2s_out, 1);
AudioControlSGTL5000 sgtl5000_1; //xy=265.212
#define BUFFER_SIZE 256
const int myInput = AUDIO_INPUT_LINEIN;
int idx = 0;
float32_t float_buffer [256];
float32_t float_buffer2 [256];
int16_t int_buffer [256];
// 200 tap FIR filter
arm_fir_instance_f32 FIR_lowpass1;
float32_t FIR_lowpass1_state[200 + BUFFER_SIZE/2];
// 2-pole biquad IIR
arm_biquad_casd_df1_inst_f32 biquad_lowpass1;
float32_t biquad_lowpass1_state[4];
// pass-thru coefficients
float32_t biquad_lowpass1_coeffs[5] = {1,0,0,0,0};
// 10-pole biquad IIR
arm_biquad_casd_df1_inst_f32 biquad_lowpass2;
float32_t biquad_lowpass2_state[20];
// lowpass elliptic 2kHz, IIR biquad 5 stages = 10 pole
// fs 44118Hz
// IIR Filter designer Iowa Hills
// a1 and a2 negated
// order of coefficients: b0, b1, b2, a1, a2
float32_t biquad_lowpass2_coeffs[25] = {
0.290842061698378285,
-0.554693042178232898,
0.290842061698378285,
1.703434597368454600,
-0.730425678586978488,
0.449552103368312861,
-0.853559775858211078,
0.449552103368312861,
1.774090745134004670,
-0.819635176012419420,
0.490545749389036945,
-0.917438292195382643,
0.490545749389036945,
1.843867832401652420,
-0.907521038984343553,
0.307079065606046031,
-0.540112802032841843,
0.307079065606046031,
1.886261593263963920,
-0.960306922443214361,
0.062519618312348771,
-0.046493567918447658,
0.062519618312348771,
1.910667387543194320,
-0.989213056249444222
};
// FIR 200 taps, Raised Cosine 0.940
// Fc = 3.000kHz, 75dB stopband
// fs 44118Hz
// just to test if this works,
// a 200 tap FIR is ridicously big and lots of calculation work for the processor
// --> it works !
float32_t FIR_lowpass1_coeffs[200] =
{ 94.97870007611338390E-9,
-1.009924021425136600E-6,
-3.955509751038522200E-6,
-8.167421170036924140E-6,
-12.23385614931934380E-6,
-14.14794389888045070E-6,
-11.83601267362507410E-6,
-3.874707993346245160E-6,
9.784409190338859470E-6,
27.31190087730911390E-6,
44.93736028076724410E-6,
57.46570395053866780E-6,
59.37446254570318160E-6,
46.32396853632215540E-6,
16.75260847587800940E-6,
-26.87322881082111080E-6,
-77.58971029118251290E-6,
-124.5675198379420860E-6,
-154.9396363165045050E-6,
-156.6193065484440300E-6,
-121.5946328111847800E-6,
-48.95032523151913040E-6,
53.22076923312532420E-6,
167.7426457861406560E-6,
270.5249246949471740E-6,
334.8987951214899680E-6,
337.4870793426350130E-6,
264.4733746227138910E-6,
116.8225367007839280E-6,
-87.00810636093507360E-6,
-311.9857878058807610E-6,
-511.4939298316222110E-6,
-635.9492759051854590E-6,
-643.6567497319431370E-6,
-511.7505262783362240E-6,
-244.6910091391752930E-6,
122.0563006080398620E-6,
524.7970382117916870E-6,
881.2433438597264510E-6,
0.001105897264525711,
0.001128584366403902,
912.4295155936849820E-6,
467.1426447303733770E-6,
-146.0617101058849410E-6,
-820.2329826975067140E-6,
-0.001419395225860928,
-0.001804044124385529,
-0.001861021146414526,
-0.001531785067380846,
-832.6535950832496840E-6,
138.4931805272558170E-6,
0.001212301918764672,
0.002174825940792150,
0.002807530450404730,
0.002933442142689424,
0.002460262925219348,
0.001410796432743954,
-67.40198731776092700E-6,
-0.001717749520116288,
-0.003215444385842726,
-0.004227679123662836,
-0.004482787665470263,
-0.003834751661105775,
-0.002308921085773049,
-117.1613956177577340E-6,
0.002364261142304478,
0.004653686825192925,
0.006251853354833492,
0.006743484329240796,
0.005893195184744424,
0.003715295847640134,
500.2318515875483630E-6,
-0.003212572302564382,
-0.006715879276971413,
-0.009259324885670151,
-0.010197256146698041,
-0.009132578886926748,
-0.006027440073467311,
-0.001255112944564374,
0.004423632349767859,
0.009963382868447519,
0.014204955965723772,
0.016088984764790849,
0.014876395277668464,
0.010335310034903742,
0.002856460155015252,
-0.006531028294970668,
-0.016255904142096857,
-0.024406432741385542,
-0.029011490508947704,
-0.028359336686767097,
-0.021302866903510787,
-0.007499202509221305,
0.012460426961983004,
0.037058421514990315,
0.064002438051746699,
0.090503604392753637,
0.113642557166307487,
0.130768970678281582,
0.139873389683470517,
0.139873389683470517,
0.130768970678281582,
0.113642557166307487,
0.090503604392753637,
0.064002438051746699,
0.037058421514990315,
0.012460426961983004,
-0.007499202509221305,
-0.021302866903510787,
-0.028359336686767097,
-0.029011490508947704,
-0.024406432741385542,
-0.016255904142096857,
-0.006531028294970668,
0.002856460155015252,
0.010335310034903742,
0.014876395277668464,
0.016088984764790849,
0.014204955965723772,
0.009963382868447519,
0.004423632349767859,
-0.001255112944564374,
-0.006027440073467311,
-0.009132578886926748,
-0.010197256146698041,
-0.009259324885670151,
-0.006715879276971413,
-0.003212572302564382,
500.2318515875483630E-6,
0.003715295847640134,
0.005893195184744424,
0.006743484329240796,
0.006251853354833492,
0.004653686825192925,
0.002364261142304478,
-117.1613956177577340E-6,
-0.002308921085773049,
-0.003834751661105775,
-0.004482787665470263,
-0.004227679123662836,
-0.003215444385842726,
-0.001717749520116288,
-67.40198731776092700E-6,
0.001410796432743954,
0.002460262925219348,
0.002933442142689424,
0.002807530450404730,
0.002174825940792150,
0.001212301918764672,
138.4931805272558170E-6,
-832.6535950832496840E-6,
-0.001531785067380846,
-0.001861021146414526,
-0.001804044124385529,
-0.001419395225860928,
-820.2329826975067140E-6,
-146.0617101058849410E-6,
467.1426447303733770E-6,
912.4295155936849820E-6,
0.001128584366403902,
0.001105897264525711,
881.2433438597264510E-6,
524.7970382117916870E-6,
122.0563006080398620E-6,
-244.6910091391752930E-6,
-511.7505262783362240E-6,
-643.6567497319431370E-6,
-635.9492759051854590E-6,
-511.4939298316222110E-6,
-311.9857878058807610E-6,
-87.00810636093507360E-6,
116.8225367007839280E-6,
264.4733746227138910E-6,
337.4870793426350130E-6,
334.8987951214899680E-6,
270.5249246949471740E-6,
167.7426457861406560E-6,
53.22076923312532420E-6,
-48.95032523151913040E-6,
-121.5946328111847800E-6,
-156.6193065484440300E-6,
-154.9396363165045050E-6,
-124.5675198379420860E-6,
-77.58971029118251290E-6,
-26.87322881082111080E-6,
16.75260847587800940E-6,
46.32396853632215540E-6,
59.37446254570318160E-6,
57.46570395053866780E-6,
44.93736028076724410E-6,
27.31190087730911390E-6,
9.784409190338859470E-6,
-3.874707993346245160E-6,
-11.83601267362507410E-6,
-14.14794389888045070E-6,
-12.23385614931934380E-6,
-8.167421170036924140E-6,
-3.955509751038522200E-6,
-1.009924021425136600E-6,
94.97870007611338390E-9
};
void setup() {
// Audio connections require memory. and the record queue
// uses this memory to buffer incoming audio.
AudioMemory(200);
// Enable the audio shield. select input. and enable output
sgtl5000_1.enable();
sgtl5000_1.inputSelect(myInput);
sgtl5000_1.volume(0.6);
sgtl5000_1.adcHighPassFilterDisable(); // kills a lot of digital noise !
// Initialize the SD card
SPI.setMOSI(7);
SPI.setSCK(14);
if (!(SD.begin(10))) {
// stop here if no SD card. but print a message
while (1) {
Serial.println("Unable to access the SD card");
delay(500);
}
}
pinMode( BACKLIGHT_PIN, OUTPUT );
analogWrite( BACKLIGHT_PIN, 1023 );
tft.begin();
tft.setRotation( 3 );
tft.fillScreen(ILI9341_BLACK);
tft.setCursor(10, 1);
tft.setTextSize(2);
tft.setTextColor(ILI9341_WHITE);
tft.setFont(Arial_14);
tft.print("Floating point audio processing");
Q_in.begin();
biquad_lowpass1.numStages = 1;
for (idx = 0; idx < 4; idx++)
{
biquad_lowpass1_state[idx] = 0;
}
biquad_lowpass1.pState = biquad_lowpass1_state;
biquad_lowpass1.pCoeffs = biquad_lowpass1_coeffs;
biquad_lowpass2.numStages = 5;
idx = 0;
for (idx = 0; idx < 20; idx++)
{
biquad_lowpass2_state[idx] = 0;
}
biquad_lowpass2.pState = biquad_lowpass2_state;
biquad_lowpass2.pCoeffs = biquad_lowpass2_coeffs;
idx=0;
for(idx = 0; idx < (200+BUFFER_SIZE/2); idx++) // Initialize all filter state variables
{
FIR_lowpass1_state [idx]= 0;
}
FIR_lowpass1.pState = FIR_lowpass1_state;
arm_fir_init_f32((arm_fir_instance_f32 *)&FIR_lowpass1, 200, (float32_t *)&FIR_lowpass1_coeffs[0], &FIR_lowpass1_state[0],BUFFER_SIZE/2);
}
void loop() {
// begin audio collection
// packet already collected?
if (Q_in.available() >= 1)
{
byte buffer[BUFFER_SIZE];
// int16_t buffer[128];
memcpy(buffer,Q_in.readBuffer(), BUFFER_SIZE);
Q_in.freeBuffer();
// memcpy(buffer + 256,Q_in.readBuffer(), BUFFER_SIZE/2);
// Q_in.freeBuffer();
memcpy(int_buffer,buffer,BUFFER_SIZE); // copy audio block into int_buffer
// convert to float
arm_q15_to_float (int_buffer, float_buffer, BUFFER_SIZE/2); // convert int_buffer to float 32bit
/**************************************************************************
* From here, all the 32 bit float audio processing can start
* ************************************************************************
*/
// test filter 1 stage
arm_biquad_cascade_df1_f32 (&biquad_lowpass1, float_buffer,float_buffer2, BUFFER_SIZE/2);
// test filter 5 stages
arm_biquad_cascade_df1_f32 (&biquad_lowpass2, float_buffer2,float_buffer, BUFFER_SIZE/2);
// test FIR filter 200 taps
arm_fir_f32(&FIR_lowpass1,float_buffer, float_buffer2, BUFFER_SIZE/2);
/**************************************************************************
* END of 32 bit float audio processing
* ************************************************************************
*/
// convert to int
arm_float_to_q15 (float_buffer2, int_buffer, BUFFER_SIZE/2);
byte buffer2[BUFFER_SIZE];
memcpy(buffer2,int_buffer, BUFFER_SIZE); // copy bytewise into output buffer
int16_t *p = Q_out.getBuffer();
for (int i = 0; i < BUFFER_SIZE; i++)
{
*((char *)p + i) = buffer2[i];
}
Q_out.playBuffer(); // play it !
}
if (five_sec.check() == 1)
{
Serial.print("Proc = ");
Serial.print(AudioProcessorUsage());
Serial.print(" (");
Serial.print(AudioProcessorUsageMax());
Serial.print("), Mem = ");
Serial.print(AudioMemoryUsage());
Serial.print(" (");
Serial.print(AudioMemoryUsageMax());
Serial.println(")");
tft.fillRect(100,120,200,140,ILI9341_BLACK);
tft.setCursor(10, 120);
tft.setTextSize(2);
tft.setTextColor(ILI9341_WHITE);
tft.setFont(Arial_14);
tft.print ("Proc = ");
tft.setCursor(100, 120);
tft.print (AudioProcessorUsage());
tft.setCursor(180, 120);
tft.print (AudioProcessorUsageMax());
tft.setCursor(10, 150);
tft.print ("Mem = ");
tft.setCursor(100, 150);
tft.print (AudioMemoryUsage());
tft.setCursor(180, 150);
tft.print (AudioMemoryUsageMax());
AudioProcessorUsageMaxReset();
AudioMemoryUsageMaxReset();
}
}My questions are:
- is my implementation of the recordqueue and playqueue objects correct and processor-efficient (lines 388-397)? Or do I need to alter something in order to be faster and accurate?
- the code takes one sample at a time (lines 388ff), processes it and plays the sample back (lines 426ff). Is there a way to take several samples, process them and then playback several samples (put them into the playqueue object one after the other)? That would be needed for a 1024point FFT, for example.
- the memory and processor usage of the audio objects only is displayed, so it stays at 0.8% all the time. How can I measure the processor usage for all that floating point processing?
Thanks in advance for your comments!
Frank DD4WH
Last edited:
and (b) many years ago I converted the Fortran code in the Parks-McLellan paper to C for my Amiga and then added a graphical interface. I have since ported it to my PC.