Bat detector

Hi CorBee,

your progress is really fantastic! Unfortunately I do not have the time to really test all your improvements . . .

Just a few comments from my experience:

* I would use a max sample rate of 281kHz (not higher), then you have a freq response up to about 140kHz. Thats enough for almost all bat analysis purposes and -more important- I doubt that your cheap mic picks up freqs above 140k
* from that perspective, I am a little sceptical about your spectrogram showing three harmonics of P. pipistrellus. I think that at least the last harmonic (F end = 135k) is a pure artefact. I have never seen higher harmonics having higher energy (than lower harmonics) in P. pipistrellus (an exception are the Rhinolophus bats, where the second harmonic has the highest energy, even higher than the first). I have seen similar artefacts in my recordings in those cases where the ADC was at the overloading point and the distortion of the waveform was overemphazising the harmonics (because the waveform approaches a square ;-)), so I would like to see more spectrograms of your system with lower amplitude (to be sure that the ADC is not overloading): in those cases, I would predict that the harmonics would be much weaker and there would be no harmonics above 140k, maybe analog gain in the codec is a little too high

Frank
 
Hi Frank,

Thanks for commenting !
Here is a screenshot of the same bat at 281k, in this you can clearly see aliasing-effects of a harmonic turned upside down. Maybe its due to the fact that the bat was very nearby (a few meters) and leading to distortions. Maybe its time to add some AGC to prevent overloading ?

Cor

Screenshot from 2018-07-13 09-16-28.png
 
This shows a few calls and one of the issues I have with 281K, at marks A B and C there are disturbances in the file.

Screenshot from 2018-07-13 09-24-38.png
 
Hi,

Progress on this thread has been low for the last 2 weeks. That was mainly due to the fact that it was very warm overhere and I was also busy building a first real prototype.
IMG_20180730_113847983.png

cheers
Cor
 
Hi,

As stated in message #78 I do see regular spikes of noise when recording. I decided to do a simple test by recording 4 files (without any bats around) at different sampling rates. The screenshot below shows the spikes.
Screenshot from 2018-08-12 09-01-45.png

For 176k the time between the spikes is 93ms, for 192k it is 85ms, for 234k it is 70ms and for 281k it is 58ms. Next to this effect it also seems as if the average value of the noise is not the same for each recording and seems to increase with the increasing samplerate (see below).
Screenshot from 2018-08-12 09-10-33.png

From the 2nd screenshot it is clearly visible that the spikes are identical (same pattern) but that the noise around it changes level.

Anybody with a possible idea what can be the cause of this ? I am mainly interested in the possible source of the spike (clearly defined with a repeating pattern in time) or ways to find the source.

regards
Cor
 
Don't know what it is, but the spike is 16384 samples wide at each of the sampling rates which corresponds to 64 consecutive audio buffers. This suggests a problem in the code doing the recording rather than an external noise source.

Pete
 
Hi Pete,

The spike seems to be indeed the same width(thus the duration is fixed) for each samplerate, but that also means the number of samples is different for each samplerate.

Cor

Edit: Ive been counting the samples in the spike. For 176k its is 64, for 234k it is 48.
 
Last edited:
Ooops. I got that wrong. It is 16384 samples (64 audio buffers) between spikes.
But I don't think it is a coincidence that it is also the amount of data (32kB) that you write at one time to the SD card. I can't find anything overtly wrong with it but play with the buffer size and see what it effect it has on the spike.
Try BUFFSIZE of 16*1024 and 8*1024.

Pete
 
Hi,

Thanks for commenting.

Unfortunately it isnt by default 16384 between spikes also, at 176k there are 4032 samples between spikes and 64 in the spike,
at 234k it is 3040 between spikes and 48 in the spike.

I will still test if the BUFFSIZE-setting (currently 32*1024) plays a role in this.

Cor
 
I have tested using a 16*1024 BUFFSIZE setting.

176k 2046 between spikes, spike less apparent and therefore spikelength unclear
192k 1826 between spikes, spike length 58 samples
234k 1496 between spikes, spike length 48 samples

This test makes clear that the routine could indeed be part of the problem, I decreased BUFFSIZE and the time between spikes became shorter at the same samplerate.

Next test is to make the BUFFSIZE larger (64Kb) and see if that agrees.

EDIT: Ive tested with 48*1024 as BUFFSIZE (64 is not possible due to the amount of RAM needed). The result at 234k is as expected, the spikelength is still 48, between spikes is now 4583 samples.
Recap for 234k , spikelength=48samples.
Code:
Betweenspikes at BUFFsize 
                                          16*1024 = 1496
                                          32*1024 = 3040
                                          48*1024=  4583

Next step is to find out what happens in the code during recording as Pete seems to be right that the recording code is causing this
 
Last edited:
Just found out one more thing that hints towards issues with the recording code. The spikes in the records appear at exactly the same positions in two consecutive recordings. If the spikes would be due to another piece of the system this should not appear at exactly the same position in each recording. Ive enclosed the current recording procedure. This is set up to use a 48*1024 BUFFSIZE.

Code:
void continueRecording() {
  const uint32_t N_BUFFER = 2;
  const uint32_t N_LOOPS = 96; // !!! NLOOPS and BUFFSIZE ARE DEPENDENT !!! NLOOPS = BUFFSIZE/N_BUFFER
  // buffer size total = 256 * n_buffer * n_loops
  // queue: write n_buffer blocks * 256 bytes to buffer at a time; free queue buffer;
  // repeat n_loops times ( * n_buffer * 256 = total amount to write at one time)
  // then write to SD card

  if (recorder.available() >= N_BUFFER  )
  {// one buffer = 256 (8bit)-bytes = block of 128 16-bit samples
    for (int i = 0; i < N_BUFFER; i++) {
       memcpy(buffern + i*256 + nj * 256 * N_BUFFER, recorder.readBuffer(), 256);
       recorder.freeBuffer();
       } 
       
    if (nj >=  (N_LOOPS - 1)) 
    {
      nj = 0;
      // copy to 2nd buffer
      
      for (int ii = 0; ii < BUFFSIZE; ii++) {
        buffern2[ii] = buffern [ii];
        }
      rc = f_write (&fil, buffern2, N_BUFFER * 256 * N_LOOPS, &wr);
      
      }
    nj ++;
  }

}
 
Last edited:
Hi,

I have been measuring the samples etc using Audacity, this makes the results I have thusfar comparable.
One thing however I dont get, I measure around 4600 samples between the spikes at 234k samplerate. I was expecting that to be 4600/234000 around 20ms between spikes. When I however check the time between spikes in milliseconds I see 105ms, I would have guessed to see 24570(==24*1024) samples instead between the spikes. Maybe audicity does not show all data after changing the datarate of these raw-recordings.

EDIT: Ive checked raw data before conversion. And indeed there are 24570 samples between the spikes ! Audacity has no option to report the no of samples at 234k and I was reading it seems at a projected speed of 44k. Going to check the raw recordings once more without initial conversion.
 
Last edited:
And finally Pete was more right, the readouts in audacity have tricked me it seems(newbie user error !).
When I look at the raw data before conversion to the recorded datarate I indeed see 16384 samples at my original 32*1024BUFFSIZE and 24576 for the 48*1024BUFFSIZE from the start of each sample to the next sample (including the spike). This is independent of the samplerate. Therefore the spike is part of the recording and it is located in the first 256 samples.
 
Found the first issue with the code I was using thusfar. The lines of code that are bold/underlined result in the first 256 bytes of the buffer only to be filled at the start of the run. NJ is 0 at the start and fills a block in the buffer. After NLOOPS (in the code starting with if (nj>=(NLOOPS-1)) the code resets nj to 0. But immediately after that codeblock nj gets nj++ and goes to 1 and never becomes 0. Therefore the first 256 samples of each group are identical and never renewed.

Thinking about a solution


Code:
void continueRecording() {
  const uint32_t N_BUFFER = 2;
  const uint32_t N_LOOPS = 96; // !!! NLOOPS and BUFFSIZE ARE DEPENDENT !!! NLOOPS = BUFFSIZE/N_BUFFER
  // buffer size total = 256 * n_buffer * n_loops
  // queue: write n_buffer blocks * 256 bytes to buffer at a time; free queue buffer;
  // repeat n_loops times ( * n_buffer * 256 = total amount to write at one time)
  // then write to SD card

  if (recorder.available() >= N_BUFFER  )
  {// one buffer = 256 (8bit)-bytes = block of 128 16-bit samples
    for (int i = 0; i < N_BUFFER; i++) {
       memcpy(buffern + i*256 + nj * 256 * N_BUFFER, recorder.readBuffer(), 256);
       recorder.freeBuffer();
       } 
       
    if (nj >=  (N_LOOPS - 1)) 
    {
[U][B][SIZE=4]      nj = 0;[/SIZE][/B][/U]
      // copy to 2nd buffer
      
      for (int ii = 0; ii < BUFFSIZE; ii++) {
        buffern2[ii] = buffern [ii];
        }
      rc = f_write (&fil, buffern2, N_BUFFER * 256 * N_LOOPS, &wr);
      
      }[U][B][SIZE=4]
    nj ++;[/SIZE][/B][/U]
  }

}
 
just a guess:

* the first 256 bytes of the buffer will probably never be filled, because nj is never zero, but starts with 1
* also nj is never reaching N_LOOPS-1, but will always be set to zero when nj == (N_LOOPS - 1), so the last working condition is (N_LOOPS - 2) ! (so probably last part of the buffer will also never be filled)

But thats just a quick shot at your code.

All the best,

Frank

P.S.: sorry, our posts crossed ;-)
 
Initial fix. This makes sure the first block of 256 samples is updated. Instead of seeing the same pattern after each 24576 samples I now see a different starting block of 256 samples. Unfortunately this
block still has some extra noise. Maybe that noise gets trapped due to the fact that the procedure continueRecording is part of the loop(). In the loop (during recording) nothing much is getting done since I have aimed to minimize interaction with for instance the TFT. The only thing I check in the loop is if the user has pressed a button to stop recording. Maybe that still produces some unexpected noise.

Edit: thanks for chiming in Frank !

Code:
void continueRecording() {
  const uint32_t N_BUFFER = 2;
  const uint32_t N_LOOPS = 96; // !!! NLOOPS and BUFFSIZE ARE DEPENDENT !!! NLOOPS = BUFFSIZE/N_BUFFER
  // buffer size total = 256 * n_buffer * n_loops
  // queue: write n_buffer blocks * 256 bytes to buffer at a time; free queue buffer;
  // repeat n_loops times ( * n_buffer * 256 = total amount to write at one time)
  // then write to SD card
  
  if (recorder.available() >= N_BUFFER  )
  {// one buffer = 256 (8bit)-bytes = block of 128 16-bit samples
    for (int i = 0; i < N_BUFFER; i++) {
       memcpy(buffern + i*256 + nj * 256 * N_BUFFER, recorder.readBuffer(), 256);
       recorder.freeBuffer();
       } 

    nj++; 

    if (nj >  (N_LOOPS - 1)) 
    {
      nj = 0;
      // copy to 2nd buffer
      for (int ii = 0; ii < BUFFSIZE; ii++) {
        buffern2[ii] = buffern [ii];
        }
      rc = f_write (&fil, buffern2, N_BUFFER * 256 * N_LOOPS, &wr);
      
      }

    
  }

}
 
Last edited:
Another minor change, at the end of the void startRecording() a line nj=0; has to be added to make sure nj starts at zero at startup.
 
Just tested the new code outside (its daytime so no bats) on the crickets in our garden. The recording now sounds clean, the minor disturbance after each sample of 48k is not really audible. Using the original code the problem was that the 1st 256 samples were the same during all recordings. This either gave clicks or sudden lack of sound and most importantly lead to discontinued sounds.

Now back to further development
 
It seems that upping the BUFFSIZE to 48*1024 has a sideeffect. There is less memory available to for instance read the SDcard directory and thus ... I am back to BUFFSIZE=32*1024
 
Hi,

Still running test to dimish artifacts during recording to a minimum. As it stands the 1st batch of samples (32Kb buffer) is recorded free of artifacts, then at the beginning of the next recorded set an artifact can bee found lasting a few milliseconds. The source of this noise seems to be the line I have marked bold in the recording routine below.

Code:
void continueRecording() {
  const uint32_t N_BUFFER = 2;
  const uint32_t N_LOOPS = 96; // !!! NLOOPS and BUFFSIZE ARE DEPENDENT !!! NLOOPS = BUFFSIZE/N_BUFFER
  // buffer size total = 256 * n_buffer * n_loops
  // queue: write n_buffer blocks * 256 bytes to buffer at a time; free queue buffer;
  // repeat n_loops times ( * n_buffer * 256 = total amount to write at one time)
  // then write to SD card
  
  if (recorder.available() >= N_BUFFER  )
  {// one buffer = 256 (8bit)-bytes = block of 128 16-bit samples
    for (int i = 0; i < N_BUFFER; i++) {
       memcpy(buffern + i*256 + nj * 256 * N_BUFFER, recorder.readBuffer(), 256);
       recorder.freeBuffer();
       } 

    nj++; 

    if (nj >  (N_LOOPS - 1)) 
    {
      nj = 0;
      // copy to 2nd buffer
      for (int ii = 0; ii < BUFFSIZE; ii++) {
        buffern2[ii] = buffern [ii];
        }
     [B] rc = f_write (&fil, buffern2, N_BUFFER * 256 * N_LOOPS, &wr);[/B]
      
      }
}

I see less noise when I change the routine at that point but it looks as if the recording is discontinous due to this (odd changes). So the writing to the SD seems to be causing additional noise to the recording. I dont seem to get rid of that noise by for instance moving my audioinput wiring away from the teensy. Anybody with a guess at a possible solution ?

Code:
      AudioNoInterrupts();
      rc = f_write (&fil, buffern2, N_BUFFER * 256 * N_LOOPS, &wr);
      AudioInterrupts();

The line f_write points to a routine inside ff.c (I am using a non-altered version of these files) based on
Code:
/*----------------------------------------------------------------------------/
/  FatFs - Generic FAT Filesystem Module  R0.13a                              /
/-----------------------------------------------------------------------------/
/
/ Copyright (C) 2017, ChaN, all right reserved.

......

#if !FF_FS_READONLY
/*-----------------------------------------------------------------------*/
/* Write File                                                            */
/*-----------------------------------------------------------------------*/

FRESULT f_write (
	FIL* fp,			/* Pointer to the file object */
	const void* buff,	/* Pointer to the data to be written */
	UINT btw,			/* Number of bytes to write */
	UINT* bw			/* Pointer to number of bytes written */
)
{
	FRESULT res;
	FATFS *fs;
	DWORD clst, sect;
	UINT wcnt, cc, csect;
	const BYTE *wbuff = (const BYTE*)buff;


	*bw = 0;	/* Clear write byte counter */
	res = validate(&fp->obj, &fs);			/* Check validity of the file object */
	if (res != FR_OK || (res = (FRESULT)fp->err) != FR_OK) LEAVE_FF(fs, res);	/* Check validity */
	if (!(fp->flag & FA_WRITE)) LEAVE_FF(fs, FR_DENIED);	/* Check access mode */

	/* Check fptr wrap-around (file size cannot reach 4 GiB at FAT volume) */
	if ((!FF_FS_EXFAT || fs->fs_type != FS_EXFAT) && (DWORD)(fp->fptr + btw) < (DWORD)fp->fptr) {
		btw = (UINT)(0xFFFFFFFF - (DWORD)fp->fptr);
	}

	for ( ;  btw;							/* Repeat until all data written */
		btw -= wcnt, *bw += wcnt, wbuff += wcnt, fp->fptr += wcnt, fp->obj.objsize = (fp->fptr > fp->obj.objsize) ? fp->fptr : fp->obj.objsize) {
		if (fp->fptr % SS(fs) == 0) {		/* On the sector boundary? */
			csect = (UINT)(fp->fptr / SS(fs)) & (fs->csize - 1);	/* Sector offset in the cluster */
			if (csect == 0) {				/* On the cluster boundary? */
				if (fp->fptr == 0) {		/* On the top of the file? */
					clst = fp->obj.sclust;	/* Follow from the origin */
					if (clst == 0) {		/* If no cluster is allocated, */
						clst = create_chain(&fp->obj, 0);	/* create a new cluster chain */
					}
				} else {					/* On the middle or end of the file */
#if FF_USE_FASTSEEK
					if (fp->cltbl) {
						clst = clmt_clust(fp, fp->fptr);	/* Get cluster# from the CLMT */
					} else
#endif
					{
						clst = create_chain(&fp->obj, fp->clust);	/* Follow or stretch cluster chain on the FAT */
					}
				}
				if (clst == 0) break;		/* Could not allocate a new cluster (disk full) */
				if (clst == 1) ABORT(fs, FR_INT_ERR);
				if (clst == 0xFFFFFFFF) ABORT(fs, FR_DISK_ERR);
				fp->clust = clst;			/* Update current cluster */
				if (fp->obj.sclust == 0) fp->obj.sclust = clst;	/* Set start cluster if the first write */
			}
#if FF_FS_TINY
			if (fs->winsect == fp->sect && sync_window(fs) != FR_OK) ABORT(fs, FR_DISK_ERR);	/* Write-back sector cache */
#else
			if (fp->flag & FA_DIRTY) {		/* Write-back sector cache */
				if (disk_write(fs->pdrv, fp->buf, fp->sect, 1) != RES_OK) ABORT(fs, FR_DISK_ERR);
				fp->flag &= (BYTE)~FA_DIRTY;
			}
#endif
			sect = clst2sect(fs, fp->clust);	/* Get current sector */
			if (sect == 0) ABORT(fs, FR_INT_ERR);
			sect += csect;
			cc = btw / SS(fs);				/* When remaining bytes >= sector size, */
			if (cc > 0) {					/* Write maximum contiguous sectors directly */
				if (csect + cc > fs->csize) {	/* Clip at cluster boundary */
					cc = fs->csize - csect;
				}
				if (disk_write(fs->pdrv, wbuff, sect, cc) != RES_OK) ABORT(fs, FR_DISK_ERR);
#if FF_FS_MINIMIZE <= 2
#if FF_FS_TINY
				if (fs->winsect - sect < cc) {	/* Refill sector cache if it gets invalidated by the direct write */
					mem_cpy(fs->win, wbuff + ((fs->winsect - sect) * SS(fs)), SS(fs));
					fs->wflag = 0;
				}
#else
				if (fp->sect - sect < cc) { /* Refill sector cache if it gets invalidated by the direct write */
					mem_cpy(fp->buf, wbuff + ((fp->sect - sect) * SS(fs)), SS(fs));
					fp->flag &= (BYTE)~FA_DIRTY;
				}
#endif
#endif
				wcnt = SS(fs) * cc;		/* Number of bytes transferred */
				continue;
			}
#if FF_FS_TINY
			if (fp->fptr >= fp->obj.objsize) {	/* Avoid silly cache filling on the growing edge */
				if (sync_window(fs) != FR_OK) ABORT(fs, FR_DISK_ERR);
				fs->winsect = sect;
			}
#else
			if (fp->sect != sect && 		/* Fill sector cache with file data */
				fp->fptr < fp->obj.objsize &&
				disk_read(fs->pdrv, fp->buf, sect, 1) != RES_OK) {
					ABORT(fs, FR_DISK_ERR);
			}
#endif
			fp->sect = sect;
		}
		wcnt = SS(fs) - (UINT)fp->fptr % SS(fs);	/* Number of bytes left in the sector */
		if (wcnt > btw) wcnt = btw;					/* Clip it by btw if needed */
#if FF_FS_TINY
		if (move_window(fs, fp->sect) != FR_OK) ABORT(fs, FR_DISK_ERR);	/* Move sector window */
		mem_cpy(fs->win + fp->fptr % SS(fs), wbuff, wcnt);	/* Fit data to the sector */
		fs->wflag = 1;
#else
		mem_cpy(fp->buf + fp->fptr % SS(fs), wbuff, wcnt);	/* Fit data to the sector */
		fp->flag |= FA_DIRTY;
#endif
	}

	fp->flag |= FA_MODIFIED;				/* Set file change flag */

	LEAVE_FF(fs, FR_OK);
}
 
Hi,

progress-report:
As I have found no clear solution to the issue of noise trickling into the audiorecording when saving a set of samples I have increased the buffersize to 64Kb. To allow that I had to drop the usage of a double buffer (one was used for recording and one for saving to SD). The noise is not less but less frequent. Ive also tried many different locations to lead the the wires from my microphone to the audioboard within the boundaries of the enclosure I am using without a result. The noise in the recording is not audible at normal frequencies but becomes clearer when you slow down a recording at least 10x. I discovered it when listening to the wingbeats of a bumblebee ;)

Cor
 
Hi guys, I stumbled across this project a couple of weeks ago and thought I'd like to build one. I'm really impressed with what you have achieved and I can see that CorBee is taking it further again.
I have bought the Teensy 3.6 and the audio board and display. Getting ready to put it together but realised I've not got a MIC. Do I need to look for a particularly wide band one or is a standard one adequate?
 
Hi!

I use this microphone:

https://www.elv.de/elv-mikrofon-mems1-bausatz.html

coupled with this preamp

https://www.elv.de/smd-mikrofonvorverstaerker-komplettbausatz.html

which I modified like this:

https://forum.pjrc.com/threads/5267...nd-performance?p=182987&viewfull=1#post182987

I am not sure whether you really need the preamp if you use the audio shield (which has a programmable gain amplifier), but I guess for the best SNR (signal-to-noise-ratio) it is probably a good idea to use a preamp.

However, for first tests, you can use every standard electret mic. They all have some performance above 20kHz, but for maximum performance up to the 100kHz region, a dedicated ultrasonic mic is recommended.

Have fun with the Teensy,

Frank
 
Hi Frank

Thanks for chiming in. For those in the UK, micbooster.com also sells the SPU0410 on a tiny board. Its not on their website but you can inquire.
I have been testing my current setup without my preamp also, this has advantages if you want to hear a lot more in the frequencies below 30Khz. The preamp I use especially provides more gain at higher frequencies.

regards
Cor
EDIT: They even have added a smiling bat on the board ;)
 
Last edited:
Back
Top