Time division Multiplexing(TDM) for multilateration(TDOA) using mics from invensense

Status
Not open for further replies.

dreggory

Well-known member
I have a project I want to complete to help me solidify the things I'm learning as a physics student at the University of Utah. I made a program that runs on a raspberry pi and simulates solving the location of the source of a sound it completes in 0.1 seconds and solves in 3-D space.
z0_xyPlane.jpg
this picture shows the error if I have 7 decimal places of timing(with some tricky tricks I may even be able to get 9 decimals). each pixel location represents the simulated point and the color is how much error due to realistic timing round off.

So now I want to build it. I chose the ics-52000 mems mics from invensense because they are like a point sensor(small inlet port), they are preamplified, they are "omni-directional", they are fairly sensitive(to -26db), and they are 24 bit with a high signal to noise ratio of 65dBA.

Right now I'm scratching my head about the data protocol from the mics. They are calling it Time Division Multiplexing (TDM). the microcontroller (I chose the teensy 3.6 for it's speed) will initiate a pulse to the first mic and proceed to clock in 24 bits from the first mic then the first mic will initiate a pulse to the second mic and the microcontroller will read the 24 bits from the second mic and so on, all the mics data out pins are tied to one data line (I guess that's why they're calling it TDM). here is a picture of that timing diagram and wiring.
ics-5200 timing.png
Elsewhere in the data sheet https://www.invensense.com/wp-content/uploads/2016/05/DS-000121-ICS-52000-v1.2.pdf it says that if I want to sample at 48kHz with 5 to 8 mics I multiply the sampling frequency with 256 for my clock frequency which is: 12.288MHz I can do more or less clock speed to an extent but I can have a max sampling frequency of 52.8kHz which leads to a clock frequency of : 13.516MHz. This is faster than I have ever clocked data(by ~13 times). I've only ever used Arduinos. I purchased the Teensy because I could see the Arduino wasn't going to cut it. how can I do this? the wires also need to be about 2 feet, give or take a few inches (I was thinking coax or ethernet?). My end goal is to have a servo motor turret fire a marshmellow and hit the mouth or face of the loudest person in the room ...cause that would be hilarious:D.

So, how can I read in this data at these speeds? is there a hardware port on the Teensy that can do this? any suggestions?
 
Last edited:
Can I use the hardware SPI? then would I use SS for the WS signal and SCK and MISO for reading the bits?
 
The Teensy 3.6 I2S module is quite flexible and supports TDM. It has 8 word FIFOs for each of the 2 receive channels, so you could use it with either interrupts or DMA with reasonable overhead.

If you split the MICs across the 2 channels, you can run at half the clock frequency and your wiring will have better signal integrity.

There is I2S code floating around that you can use as a starting point - do a search.
 
I tried the code from: github link to i2s experimental library
I was able to get one microphone to work with this library but I really need 5 to work, I'm trying to get just two working to see if I can expand from there. But, I'm struggling to figure out buffers and switching between the 32 bit words.

My oscilloscope shows that WS goes high for 32 clock cycles and then low for another 32 cycles on a 50% duty. I think that's because the library was adapted to work with the InvenSense INMP441 I2S microphone, which is a stereo setup that toggles between mics. For my microphones I need one pulse for a clock cycle and then nothing until all the mics down the line have sent in their data (they prompt each other to start their transmission by daisy chaining WSO from the previous mic to the WS of the next mic)

ics-5200 timing2.png

I have been reading about the audio library, it doesn't support 24 bit or 32 bit words does it? I could use some more help. anyone?
 
Last edited:
I was able to get one microphone to work with this library but I really need 5 to work

For the problem you wanted to solve, you need only 4 mics. Maths is trivial. In that case you should be able to use the SAI protocol.
If you pass an inverted word clock to every even mics, you simulate stereo. the resulting half-cycle phase shift can be handled in SW.

For all configurations with more than 4 channels you need TDM (as mentioned by Paul).

As a sufficient high bit-clock rate may increase the power consumption, I tend to stick with 4 channels for all my TDOA work as much as possible.
 
I would like to use less mics, but the math led me to 3 non linear equations for 3 unknowns for 4 mics(enough to solve,) but I couldn't figure out how to get an explicit solution. My computer ran all night with no result for a solution. I was able to get pretty good results by monte carlo solving but not good enough for what I wanted to do, it took too long, it had various levels of accuracy depending on the location, and my solver would often miss the source entirely. By adding a 5th mic I was able to linearize the equations and get an explicit solution (subject to timing accuracy round off error) if you wouldn't mind sharing how you were able to get away with 4 mics only, I really would love to know, I asked many people in the math and physics departments at the University of Utah with only a few straight answers. I'm open to alternative methods.
If you pass an inverted word clock to every even mics, you simulate stereo. the resulting half-cycle phase shift can be handled in SW.
I'm not sure what this means, I haven't taken a signal processing class yet. What's an inverted word clock? And why would there be a half cycle phase shift because of it? And then how do I handle it in software?
 
Does anyone sell these ICS-52000 mics on a breakout board?

Looks like the soldering is all on the bottom side underneath the part.
 
Yes, but I just soldered directly to the pads with magnet wire and reinforced the connections with uv glue.
micSoldered.jpg
It works, but I may have to splice into twisted pairs for longer distance connections. I am trying to minimize sound shadows for better omnidirectional behavior anyway. The breakouts I saw are at:
https://www.notwired.co/ProductDetai...red-CO/605574/
But that's like 6 times the price of one mic.
I saw a ribbon cable version somewhere, I can't seem to find where I saw that one. but it didn't include the connectors that it needs. So I didn't go that route
 
Last edited:
I would like to use less mics, but the math led me to 3 non linear equations for 3 unknowns for 4 mics(enough to solve,) but I couldn't figure out how to get an explicit solution. My computer ran all night with no result for a solution. I was able to get pretty good results by monte carlo solving but not good enough for what I wanted to do, it took too long, it had various levels of accuracy depending on the location, and my solver would often miss the source entirely. By adding a 5th mic I was able to linearize the equations and get an explicit solution (subject to timing accuracy round off error) if you wouldn't mind sharing how you were able to get away with 4 mics only, I really would love to know, I asked many people in the math and physics departments at the University of Utah with only a few straight answers. I'm open to alternative methods.

OK, I misinterpreted the figure and thought they were angles and not location.
Yes, you only get a linear set of equations if you solve for x,y,z,R i.e. 4 variables, for which you need 5 sensors.

However, I you really wanted to localize with 5 mics then the sound source should be within 2 apertures (i.e twice x spacing, twice y spacing and twice z spacing). I want argue if you claim 10 apertures but the localisation error is then significant, or you have very high SNR. So, robust localization is only inside or close to the space covered by the mics. Best config would be to have 4 mics in the corner of the room and one in the middle. Having 2 feet max spacing as you suggested , may end up with very good direction estimate but not range. Now, for direction finding only, you need only 4 mics.

However, if you can get TDM working, simply play with multiple mics and see how error changes.
 
OK, I misinterpreted the figure and thought they were angles and not location.
I'm sorry, I should have explained the figure better, I went back and read what I wrote and it is confusing, sorry.
Each pixel represents a simulated point, and the color of each pixel represents how much error there was from the simulated point to the found point.

However, If you really wanted to localize with 5 mics then the sound source should be within 2 apertures (i.e twice x spacing, twice y spacing and twice z spacing). I wont argue if you claim 10 apertures but the localisation error is then significant, or you have very high SNR. So, robust localization is only inside or close to the space covered by the mics. Best config would be to have 4 mics in the corner of the room and one in the middle. Having 2 feet max spacing as you suggested , may end up with very good direction estimate but not range. Now, for direction finding only, you need only 4 mics.

The reason for the 10 apertures is the strict timing accuracy required of 7 decimal places, or more, of timing. Which is as if I were sampling at a rate of 10MHz. At first I was worried that I couldn't interpolate that much, even by spline interpolation because I'd only be sampling at 48kHz which is like 208 times slower than I needed. But I simulated interpolating two 48000Hz signals (with a known shift) composed of 100Hz to 3000Hz frequencies, with noise for both added and crazy enough, I was able to do it.

However, if you can get TDM working, simply play with multiple mics and see how error changes.
I simulated many configurations of the mics, I even simulated using 9 mics (since these mems mics can do up to 16) but because my algorithm chooses the first 3x3 matrix (instead of the best 3x3 matrix) it was actually worse with more mics and I couldn't figure out how to do Principal Component Analysis.
9MicErrors.png
So I settled on 5 mics in a tetrahedron with one in the center. With that arrangement the errors were radially dependent, instead of mixed. and 5 mics are faster/easier to process than 9. But you are right, I need to test it in reality, so I am dying(figuratively) to collect data from 5 mics. Thank you guys for your help by the way.
 
Last edited:
I simulated many configurations of the mics, I even simulated using 9 mics (since these mems mics can do up to 16) but because my algorithm chooses the first 3x3 matrix (instead of the best 3x3 matrix) it was actually worse with more mics and I couldn't figure out how to do Principal Component Analysis.

if you have more equations than variables, do simply a LMS approach: (A'*A)^-1*A', but you know that.
 
I even simulated using 9 mics (since these mems mics can do up to 16)

Unless you can get mics that only use 16 bits of the TDM frame, or you write your own audio lib & drivers, you will almost certainly be limited to only 8. The TDM frame is fixed at 256 bits length. Each of these mics takes 32 bits. The maximum number you can actually use in practice almost certainly will be no more than 8 mics.
 
if you have more equations than variables, do simply a LMS approach: (A'*A)^-1*A', but you know that.

I didn't know that. Thanks! I think I still want to keep it to five mics, but it would be fun to tinker with that in the future
 
Unless you can get mics that only use 16 bits of the TDM frame, or you write your own audio lib & drivers, you will almost certainly be limited to only 8. The TDM frame is fixed at 256 bits length. Each of these mics takes 32 bits. The maximum number you can actually use in practice almost certainly will be no more than 8 mics.
That is very good to know. So if I want to only use 160 bits out of the 256, for my 5 mics, how do I configure the WS signal to do a brief pulse and wait for 160 bits?
 
As I understand those ICS-52000 mics, admittedly from only a brief look at the datasheet, you don't need to configure anything.

Teensy creates the brief WS pulse to the first mic. Then each mic creates a new WS pulse to the next one. The remaining bits of the 256 bit frame will be random data. I believe they recommend a pulldown resistor, so the unused bits will be zeros.

The TDM object in the audio lib is fixed at 256 bits. If you want to receive 160 bits, you can. Just ignore the rest. But you do *not* somehow reconfigure TDM to a 160 bit frame. It's fixed at 256.
 
The TDM object in the audio lib is fixed at 256 bits. If you want to receive 160 bits, you can. Just ignore the rest. But you do *not* somehow reconfigure TDM to a 160 bit frame. It's fixed at 256.

Okay, that makes sense, it's a hardware limitation. I see that you added TDM to the Audio system design tool, this is excellent. Now I'm wondering how do I combine two 16 bit channels into one 32 bit channel? I think it's going to involve shifting bits but how do I store this info?

for my application I would like to be storing the data from each mic in a sliding array (oldest value pushed out each time the new value arrives). Then when a decaying threshold for the sound intensity is crossed, the Teensy will collect 60% (of the array size) more data and then start to analyze the time shift correlation between the first mic to trigger and all the rest.

I think for starters, what do I do to combine the channels into one 32 bit channel?
 
Last edited:
Okay so this will work. here is a video of my oscilloscope showing the WS signal on top and 64 bits out of 256 bits per TDM frame coming from two mics:

now I'm just wondering how to use this library, to maybe stream through serial to start testing methods of TDOA in Matlab or something

this is the code I used to get the signals going at least(but nothing for me to see, in the serial monitor or saving any of the mic data):
Code:
#include <Audio.h>
#include <Wire.h>
#include <SPI.h>
#include <SD.h>
#include <SerialFlash.h>


// GUItool: begin automatically generated code
AudioInputTDM            tdm1;           //xy=172,228
AudioRecordQueue         queue9;         //xy=428,202
AudioRecordQueue         queue7;         //xy=430,159
AudioRecordQueue         queue5;         //xy=434,117
AudioRecordQueue         queue1;         //xy=439,34
AudioRecordQueue         queue3;         //xy=439,78
AudioRecordQueue         queue6;         //xy=554,118
AudioRecordQueue         queue8;         //xy=554,159
AudioRecordQueue         queue10;        //xy=554,201
AudioRecordQueue         queue2;         //xy=559,35
AudioRecordQueue         queue4;         //xy=559,77
AudioConnection          patchCord1(tdm1, 0, queue1, 0);
AudioConnection          patchCord2(tdm1, 1, queue2, 0);
AudioConnection          patchCord3(tdm1, 2, queue3, 0);
AudioConnection          patchCord4(tdm1, 3, queue4, 0);
AudioConnection          patchCord5(tdm1, 4, queue5, 0);
AudioConnection          patchCord6(tdm1, 5, queue6, 0);
AudioConnection          patchCord7(tdm1, 6, queue7, 0);
AudioConnection          patchCord8(tdm1, 7, queue8, 0);
AudioConnection          patchCord9(tdm1, 8, queue9, 0);
AudioConnection          patchCord10(tdm1, 9, queue10, 0);
// GUItool: end automatically generated code


void setup() 
{
  // << nothing before the first delay will be printed to the serial
  Serial.begin(57600);
  delay(1500); 
  

}

void loop() 
{
  // put your main code here, to run repeatedly:
//  Serial.print(queue1); // this didn't work, I don't know what I'm doing -Greg
//  Serial.print(" ");
//  Serial.print(queue2);
//  Serial.print(" ");
//  Serial.print(queue3);
//  Serial.print(" ");
//  Serial.print(queue4);
//  Serial.print(" ");
//  Serial.print(queue5);
//  Serial.print(" ");
//  Serial.print(queue6);
//  Serial.print(" ");
//  Serial.print(queue7);
//  Serial.print(" ");
//  Serial.print(queue8);
//  Serial.print(" ");
//  Serial.print(queue9);
//  Serial.print(" ");
//  Serial.println(queue10);


}

what would I do to combine ten 16 bit channels to five 32 bit channels? and then print them out to the serial monitor for testing?
 
so this is what I'm doing to sample 2 mics over serial, but the sample rate is all over the place (in the range of 400-500 sps), and spotty.
Code:
#include <Audio.h>
#include <Wire.h>
#include <SPI.h>


// GUItool: begin automatically generated code
AudioInputTDM            tdm1;           //xy=172,228
AudioRecordQueue         queue9;         //xy=428,202
AudioRecordQueue         queue7;         //xy=430,159
AudioRecordQueue         queue5;         //xy=434,117
AudioRecordQueue         queue1;         //xy=439,34
AudioRecordQueue         queue3;         //xy=439,78
AudioRecordQueue         queue6;         //xy=554,118
AudioRecordQueue         queue8;         //xy=554,159
AudioRecordQueue         queue10;        //xy=554,201
AudioRecordQueue         queue2;         //xy=559,35
AudioRecordQueue         queue4;         //xy=559,77
AudioConnection          patchCord1(tdm1, 0, queue1, 0);
AudioConnection          patchCord2(tdm1, 1, queue2, 0);
AudioConnection          patchCord3(tdm1, 2, queue3, 0);
AudioConnection          patchCord4(tdm1, 3, queue4, 0);
AudioConnection          patchCord5(tdm1, 4, queue5, 0);
AudioConnection          patchCord6(tdm1, 5, queue6, 0);
AudioConnection          patchCord7(tdm1, 6, queue7, 0);
AudioConnection          patchCord8(tdm1, 7, queue8, 0);
AudioConnection          patchCord9(tdm1, 8, queue9, 0);
AudioConnection          patchCord10(tdm1, 9, queue10, 0);


// GUItool: end automatically generated code


int16_t* spot1 ; //I don't know what I'm doing - Greg
int16_t* spot2 ;
int16_t* spot3 ;
int16_t* spot4 ; 
int sig1;
int sig2;
bool sp1;
bool sp3;
bool newl;

void setup() 
{
  // << nothing before the first delay will be printed to the serial
  Serial.begin(250000);
  
  AudioMemory(256);
  queue1.begin();//I thought I should enable 
  queue2.begin();
  queue3.begin();
  queue4.begin();
 
  
}

void loop() 
{

  if(queue1.available() > 0)
  {
    spot1=queue1.readBuffer();
    //Serial.print(spot1[0]); //call of overloaded 'print(int16_t*&)' is ambiguous
    queue1.freeBuffer();
    sp1 = true;
  }

 
  
  
  if(queue2.available() > 0)
  {
    spot2=queue2.readBuffer();
    //Serial.print(spot2[0]);
    queue2.freeBuffer();
    if (sp1)
    {
      sig1 = spot1[0]*256 + spot2[0]/256;
      Serial.print(sig1);
      Serial.print("  ");
      newl= true;
      sp1=false;
    }
  }


  

  if(queue3.available() > 0)
  {
    spot3=queue3.readBuffer();
    //Serial.print(spot3[0]);
    queue3.freeBuffer();
    sp3 = true;
  }


  

  if(queue4.available() > 0)
  {
    spot4=queue4.readBuffer();
    //Serial.println(spot4[0]);
    queue4.freeBuffer();
    if (sp3)
    {
      sig2 = spot3[0]*256 + spot4[0]/256;
      Serial.print(sig2);
      Serial.println(" ");
      newl = false;
      sp3=false;
    }
    
  }
  if (newl)
    {
       Serial.println(" ");
       newl=false; 
    }

 

}
here's a picture of the two signals on top of eachother:

twoSignalsCropped.jpg

I'm wondering how I could sample at the 44.1kHz(sps) that my oscilloscope says the mics are sending at.
I would be happy if I could just get a window of 100 samples, sent to a file on my computer, from each mic at a sample rate of 44.1kHz from the same 100 TDM frames (it could even be delivered at the convenience of the Teensy, within reason).
how do I do this?
 
There are no 32 bit channels in the Teensy Audio Library. All connections are 16 bit data. To move 32 bit data, you need to use 2 connections. Currently, there is only 1 object in the entire library that uses 32 bit data (two 16 bit connections), the high res sine wave synthesis.

Perhaps the library will need a multiplier or mixer object that takes 32 bit inputs? Or maybe the TDM object might get a "gain" setting to multiply those 32 bit slots, so you can get a really useful 16 bit number and discard the lower 16 bits.

No matter what you do, consider that printing the live data stream to USB serial just isn't possible on a sustained basis. It's far too much data for 12 Mbit/sec USB (which is approx 1.0 to 1.1 Mbyte/sec speed on a lightly loaded USB bus, due to USB protocol overhead).
 
There are no 32 bit channels in the Teensy Audio Library. All connections are 16 bit data. To move 32 bit data, you need to use 2 connections.
I promise I was listening when you told me that before, and I did take that into account. Since the TDM data from the mics are 24 bit datum in 32 bit blocks, I multiplied the first 16 bit value in the TDM frame by 256(making the 16 bit value the MSBit's of a 24 bit number) and then added it to the second 16 bit value in the TDM frame divided by 256 (making the second 16 bit value the trailing 8 bits for the 24 bit number).

No matter what you do, consider that printing the live data stream to USB serial just isn't possible on a sustained basis. It's far too much data for 12 Mbit/sec USB (which is approx 1.0 to 1.1 Mbyte/sec speed on a lightly loaded USB bus, due to USB protocol overhead).
This makes sense, I guess I never took thought that the transfer rate says it all, thank you.

since I only care about the travel time of sound waves that propagate between my microphones for time difference of arrival calculations:
With my mics spaced at most 0.6 of a meter apart with, and the temperature at an extreme worst case(at -40C), the speed of sound would be around 307 meters/sec, this leads to a sound travel time of 0.002 seconds, divide that by the time between samples at a rate of
44,100 SPS (0.0000227) leads to--> 0.0018/0.0000227 = roughly 86 samples in the time it takes for the sound to go from one mic to the others, maybe I could make it an even 100 samples for good measure.

So,
- how would I send an array of 100 samples for each of the 5 mics over serial? (I want to recieve the data with a raspberry pi, that should only take around 0.00016 seconds at a transfer rate of 1 Mbit/sec right? or at 9600 bits/sec it would take about 0.016 sec )
- how do I fill an array? (I am struggling with the programming)
- will there be timing issues when trying to:
-combine the two 16 bit values into one 24 bit value,
-monitor a decaying threshold,
-fill an array of 100 values(maybe shifted, so that, a few older values before the threshold was tripped, are preserved),
-and then send them?

I know you are very busy Paul, thank you for answering my project questions. I am really impressed with the time and devotion you are spending on making sure that you are providing an excellent product... and it is excellent by the way. so, Thank you!
 
Sending all the data to a Raspberry Pi might not be such a great idea. While the Pi has a lot of RAM and a faster CPU, it's USB I/O performance isn't wonderful.
 
It's mostly so I can use the python program that I wrote for solving the sound source location and also to split up the tasks.(Teensy collects fast mic data, sends it when triggered by a sound event, Raspberry Pi processes the correlation time shift, solves location, sends location to Teensy, Teensy adjusts turret, and fires marshmellow).
The 5 mic's datum will only be sent every so often (like when the decaying threshold is crossed), I expect something like once every 5 seconds. And it would only be 500 samples at a time. However, you have made me consider writing this whole thing on the teensy though. I just think I will run into problems where I was using a python linear algebra package. I'm also a little more familiar with python.
 
Last edited:
Status
Not open for further replies.
Back
Top