Granular Synthesis with Teensy and Audio Adaptor

Status
Not open for further replies.
MacroMachines said:
The "thousands of very short sonic grains" is actually what is taking place in my test of using the 4 looper players playing very short randomized start and length loops, it equates to 4 times 20-100ms or so bits of audio per second so potentially up to a hundred or so maybe.
Click to expand...

a hundred or so? how would that work? isn't that "looper" basically just streaming files from SD? it won't scale much beyond that, as far as i can see. using the spi flash will work much better, i think.


I have a STM32F407 discovery board that is just sitting around as I had taken to the teensy for my first digital audio projects. Would that work? I looked at the OWL and I hadn't realized it was opensource, pretty cool! I just don't know where to start with moving from arduino which I code mostly in sublime text. What IDE should I get etc..
Click to expand...
mostly yes, you won't have the extra SRAM and the codec is different, of course, so at the very least you'd have to adjust the codec driver stuff. other than that, sublime text will do. and gcc.

RE: Pi based audio: I tried making something like that a year and a half ago, I think around when you were doing your's, I wasn't happy with the quality/power and couldn't find any good way to get quality audio I/O outside using a usb soundcard. Were you able to connect a DAC/codec to the PI for audio I/O?
Click to expand...

yep, there's drivers for the wm8731 ("rpi-proto") in the vanilla distro; that works/sounds pretty good and hassle-free. or you could use a fancier codec -- you've probably seen the Teensy SuperAudioBoard, it comes with everything you need for rpi, too, IIRC.
 
I picked up a copy of the Curtis Roads Microsound book a couple weeks ago. So far I've only read chapters 2 & 3 and skipped around through some other parts briefly. If only there were more hours in every day.....

I have been putting some thought into how I'd like to implement this in the Teensy Audio library. My main reason for reading Roads's book was to get a more general idea of the range of uses, rather than possibly going down a path with too narrow of a vision. It's also nice to get a feel for how other systems have done this.

My opinion so far is to provide a couple fairly low level but highly optimized functions in the audio library. Traditionally, the audio library objects provide a few simple Arduino style functions. But for granular synthesis (playing the grains), I've been toying with the idea of a more machine-oriented API meant for building companion libraries. Such a library would build a linked list of the grains it wants the audio library to play. You could do that yourself from an Arduino sketch, but the idea is other libraries would be made. One thing Roads's Microsound makes clear is there's a lot of different strategies for playing grains. My hope is to write an optimized low-level engine in the Teensy Audio library which does the high speed, low-level work efficiently. The actual decisions about which grains to play, when to mix each one, and what parameters each will use would happen in other libs, to be developed separately.

I understand a lot of the interest here is for live recording. There too, I'm imagining the Teensy Audio library would provide a fairly simple record function. In fact, it'll very likely be a companion to the memory player object. Decisions about how to chop up the recored waveform into grains would happen outside the audio library. You could choose regions of the recorded sound, put pointer to them into the grain specs, and tell the granular synthesis object to play them in pretty much any way you like.

Realistically, Teensy's internal RAM and Flash are probably going to be the only supported media. Perhaps in the distant future we'll expand to external SPI RAM or Flash, but there's no way I'm going to add that extra complexity and performance limit on the first attempt. So grain samples are going to be limited to about 50K of the extra RAM or about 200K of the extra internal Flash. When we get a bigger Teensy, those limits will expand to about 230K RAM and nearly a megabyte of Flash. Of course, if you have a segment of a waveform in memory, you can compose any number of grains using it.

So a big question at this early stage is what per-grain parameter should the low-level synthesis code support. So far, I have these six:

Time offset / when to begin
Play length / duration
Sample data
Window / envelope shape
Volume / gain / amplitude
Pitch stretch factor

Every extra feature supported in the low-level engine will come at the cost of extra overhead for all. So features play playing the sample data backwards would probably happen at a higher level, by simply creating a backwards copy of the samples in memory.
 
PaulStoffregen said:
My opinion so far is to provide a couple fairly low level but highly optimized functions in the audio library. Traditionally, the audio library objects provide a few simple Arduino style functions. But for granular synthesis (playing the grains), I've been toying with the idea of a more machine-oriented API meant for building companion libraries. Such a library would build a linked list of the grains it wants the audio library to play. You could do that yourself from an Arduino sketch, but the idea is other libraries would be made. One thing Roads's Microsound makes clear is there's a lot of different strategies for playing grains. My hope is to write an optimized low-level engine in the Teensy Audio library which does the high speed, low-level work efficiently. The actual decisions about which grains to play, when to mix each one, and what parameters each will use would happen in other libs, to be developed separately.
Click to expand...

I'm not sure if I understand fully. Are you saying you plan to create a separate library for granular? I agree with your assessment that there are many strategies, and I will look back at my previous outline of methods/properties/functions to see if I can think of any way to make them more capable. Really the most straightforward start would just be to create a single "grainPlay" object that is basically a windowed sample player that plays back a bit of a data from offset to length at a pitch ratio.

Alternately/additionally, and possibly more useful, would be to make a polyphonic voice management system in the library. A way of creating a section of a patch in the audio system tool that would be instantiated multiple times to create round-robin voice stealing as "note on" messages are sent into it, this way you could make your own grains (or synth voices) from any objects in the library and have very deliberate control over their playback. One could make grain delays by simply loading the delay object and using an envelope to window it. A granular sampler could just use the playRaw object. A "window" object that could be driven by phase could be useful.

I had thought the SD playback would be an unrealistic bottleneck to overcome but as you can see in my tests, if you set the clock to 96mhz it stabilizes at a reasonably low cpu level for 4 playback objects looping short grains (no window but sounded just fine on most source material). IF I can figure out how to buffer the start of the files it might work fantastically. I will post some video in a bit showing some examples.

PaulStoffregen said:
I understand a lot of the interest here is for live recording. There too, I'm imagining the Teensy Audio library would provide a fairly simple record function. In fact, it'll very likely be a companion to the memory player object. Decisions about how to chop up the recored waveform into grains would happen outside the audio library. You could choose regions of the recorded sound, put pointer to them into the grain specs, and tell the granular synthesis object to play them in pretty much any way you like.
Of course, if you have a segment of a waveform in memory, you can compose any number of grains using it.
Click to expand...

I think a buffer with read/write access would be ideal


PaulStoffregen said:
So a big question at this early stage is what per-grain parameter should the low-level synthesis code support. So far, I have these six:

Time offset / when to begin
Play length / duration
Sample data
Window / envelope shape
Volume / gain / amplitude
Pitch stretch factor
Click to expand...

is pitch stretch factor just pitch or something more?

PaulStoffregen said:
Every extra feature supported in the low-level engine will come at the cost of extra overhead for all. So features play playing the sample data backwards would probably happen at a higher level, by simply creating a backwards copy of the samples in memory.
Click to expand...

I posted a function that does this in another thread, Ill copy here:
 
mxxx said:
a hundred or so? how would that work? isn't that "looper" basically just streaming files from SD? it won't scale much beyond that, as far as i can see. using the spi flash will work much better, i think.
Click to expand...

Yes, this is not at all an optimal route, but I was just suprised it worked as well as it did, I really thought this route would sound horrible and not even come close to a "granular" sound, but it really suprised me:

https://www.instagram.com/p/BApXbT9hyu0/?taken-by=axiomcrux
(is it possible to embed instagram videos on here?)

mxxx said:
mostly yes, you won't have the extra SRAM and the codec is different, of course, so at the very least you'd have to adjust the codec driver stuff. other than that, sublime text will do. and gcc.
Click to expand...

so where would I start with the STM and the OWL thing?
 
Here's a granular effect I made a while ago for a forthcoming product. You can hear it used to pitch shift in the video. http://bleeplabs.com/store/thingamagoop-3000/
I made a lot of little changes and effects for the Thingamagoop3000 that I'll be cleaning up and posting when it starts shipping.

This is not too much different than what's been posted here but it works well enough for noise music!


.cpp:
Code: 
#include "effect_granular.h"
#include "arm_math.h"



void AudioEffectGranular::begin(int16_t *sample_bank_def, int16_t max_len_def)
{
  sample_req=1;
  length(max_len_def - 1);

  sample_bank=sample_bank_def;

}

void AudioEffectGranular::length(int16_t max_len_def)
{
  if (max_len_def<100)
  {
  max_sample_len = 100;
  glitch_len = max_sample_len/3;
  }
  else{
  max_sample_len = (max_len_def - 1);
  glitch_len = max_sample_len/3;
  }

  }


void AudioEffectGranular::freeze(int16_t activate,int16_t playpack_rate_def,int16_t freeze_length_def)
{
  if (activate==1){
  grain_mode = 1;
  }
  if (activate==0){
  grain_mode = 0;
  }

  rate(playpack_rate_def);
  if (freeze_length_def<50)
  {
    freeze_len=50;
  }
  if (freeze_length_def>=max_sample_len)
  {
    freeze_len=max_sample_len;
  }

  if (freeze_length_def>=50 && freeze_length_def<max_sample_len){
    freeze_len=freeze_length_def;
  }

}

void AudioEffectGranular::shift(int16_t activate,int16_t playpack_rate_def,int16_t grain_length_def)
{
  if (activate==1){
  grain_mode = 2;
  }
  if (activate==0){
  grain_mode = 3;
  }

  rate(playpack_rate_def);
  if (allow_len_change==1 )
  {
  //  Serial.println("aL");
      length(grain_length_def);

  }
}


void AudioEffectGranular::rate(int16_t playpack_rate_def)
{
  playpack_rate = playpack_rate_def;
}


void AudioEffectGranular::update(void)
{
  audio_block_t *block;

  if (sample_bank == NULL)return;

  block = receiveWritable(0);

  if (block) {


    if (grain_mode == 3) {
      //through

      for (int i = 0; i < AUDIO_BLOCK_SAMPLES; i++) {
        write_head++;

        if (write_head >= max_sample_len) {
          write_head = 0;
        }

        sample_bank[write_head] = block->data[i];

      }

      transmit(block);

    }


    if (grain_mode == 0) {
      //capture audio but dosen't output
      //this needs to be happening at all times if you want to use grain_mode = 1, the simple "freeze" sampler.

      for (int i = 0; i < AUDIO_BLOCK_SAMPLES; i++) {
        write_head++;

        if (write_head >= max_sample_len) {
          write_head = 0;
        }

        sample_bank[write_head] = block->data[i];

      }
    }


    if (grain_mode == 1) {
    //when activated the last 

      for (int j = 0; j < AUDIO_BLOCK_SAMPLES; j++) {

        if (playpack_rate >= 0)
        {
          accumulator += playpack_rate;
          read_head = accumulator >> 9;

        }

        if (read_head >= freeze_len) {
          accumulator = 0;
          read_head -= max_sample_len;
        }

        block->data[j] = sample_bank[read_head];

      }
      transmit(block);

    }


    if (grain_mode == 2) { 
      //GLITCH SHIFT
      //basic granular synth thingy 
      //the shorter the sample the max_sample_len the more tonal it is. Longer it has more definition. 
      //its a bit roboty either way which is obv great and good enough for noise music.

      for (int k = 0; k < AUDIO_BLOCK_SAMPLES; k++) {

        int16_t current_input = block->data[k];

        //we only want to start recodeing when the audio is crossing zero to minimize pops
        if ((current_input<0 && prev_input>0))
        {
          zero_cross_down=1;
        }

        else
        {
          zero_cross_down=0;
        }

        prev_input=current_input;


        if (zero_cross_down==1 && sample_req==1)
        {
          write_en=1;
        }


        if (write_en==1)
        {
            sample_req=0;
            allow_len_change=1; //reduces noise by not allowing the length to change after the sample has been recored. kind of not too much though 

          if (write_head >= glitch_len) {
            glitch_cross_len=glitch_len;
            write_head = 0;
            sample_loaded = 1;
            write_en=0;
            allow_len_change=0; 


          }

          sample_bank[write_head] = block->data[k];
          write_head++;

        }


        if (sample_loaded == 1) {
          //move it to the middle third of the bank. 
          //3 "seperate" banks are used
          float fade_len=20.00;
          int16_t m2=fade_len;

          for (int m = 0; m < 2; m++)  
          //I'm off by one somewhere? why is there a tick at the beginning of this only when it's combined with the fade out???? ooor am i osbserving that incorrectly
          //either wait it works enough
          {
            sample_bank[m + glitch_len] = 0;    
          }

          for (int m = 2; m < glitch_len-m2; m++)
          {
            sample_bank[m + glitch_len] = sample_bank[m];
          }

          for (int m = glitch_len-m2; m < glitch_len; m++)
            //fade out the end. You can just make fadet=0 but it's a little too daleky 
          {
            float fadet=sample_bank[m]*(m2/fade_len);
            sample_bank[m + glitch_len] = (int16_t)fadet;
            m2--;

          }
          sample_loaded = 0;
          sample_req=1;
        }

        accumulator += playpack_rate;
        read_head = (accumulator >> 9);
  
        if (read_head >= glitch_len) {
          read_head -= (glitch_len);

          accumulator=0;


          for (int m = 0; m < glitch_len; m++)
          {
            sample_bank[m + (glitch_len*2)] = sample_bank[m+glitch_len];
          //  sample_bank[m + (glitch_len*2)] = (m%20)*1000;
          }


        }


        block->data[k] = sample_bank[read_head+(glitch_len*2)];

      }
      transmit(block);

    }


    release(block);
  }

}

.h
Code: 
#include "AudioStream.h"

class AudioEffectGranular : 
public AudioStream
{
public:
  AudioEffectGranular(void): 
  AudioStream(1,inputQueueArray) { 
  }

  void begin(int16_t *sample_bank_def,int16_t max_len_def);
  void length(int16_t max_len_def);
  void rate(int16_t playpack_rate_def);

  void freeze(int16_t activate,int16_t playpack_rate_def,int16_t grain_length_def);
  void shift(int16_t activate,int16_t playpack_rate_def,int16_t grain_length_def);


  virtual void update(void);
  
private:
  audio_block_t *inputQueueArray[1];
  int16_t *sample_bank;
  int16_t max_sample_len;
  int16_t write_head,read_head,grain_mode,freeze_len,allow_len_change;
  int16_t playback_rate;
  int16_t capture_trigger,capture_index,current_mode,playpack_rate;
  int32_t accumulator,play_index,increment;
  int16_t sample_loaded,prev_input,zero_cross_up,zero_cross_down,sample_trigger,write_en,glitch_cross_len,load_req,glitch_len,glitch_min_len,sample_req;


};
 
john-mike said:
Here's a granular effect I made a while ago for a forthcoming product. You can hear it used to pitch shift in the video. http://bleeplabs.com/store/thingamagoop-3000/
I made a lot of little changes and effects for the Thingamagoop3000 that I'll be cleaning up and posting when it starts shipping.

This is not too much different than what's been posted here but it works well enough for noise music!
Click to expand...

Would you be willing to post an example sketch? I copied the two files but not entirely sure where to start with testing them, or how to use them. At least a general explanation of use?
 
Last edited:
MacroMachines said:
Would you be willing to post an example sketch?
Click to expand...

It's similar to the other time based effects. You need to define a bank and max length.

Code: 
  int16_t granular_bank_len = 3000;
  int16_t granular_bank[3000] = {};

  granular1.begin(granular_bank, granular_bank_len);

Then use one of these functions.

Code: 
      granular1.freeze(activate,playpack_rate,grain_length);

      granular1.shift(activate,playpack_rate,grain_length);
 
john-mike said:
It's similar to the other time based effects. You need to define a bank and max length.
Click to expand...

so is granular_bank an array that stores the buffer for the audio to be granulated? is activate a 1/0 boolean to turn it on/off? and I assume playback_rate is a ratio where 1 would be regular speed and 0 would be stopped, is it different in the freeze vs shift? like would shift use the playback rate to set the pitch shift ratio in that case?
 
MacroMachines said:
so is granular_bank an array that stores the buffer for the audio to be granulated? is activate a 1/0 boolean to turn it on/off? and I assume playback_rate is a ratio where 1 would be regular speed and 0 would be stopped
Click to expand...

Yes activate is 1/0.
Rate and length are integers but should be floats scaled like that in the final release.
Try rate is between 0-4095. Length can be 0 (though it's stops at 50) to the buffer length.


, is it different in the freeze vs shift? like would shift use the playback rate to set the pitch shift ratio in that case?
Click to expand...

Freeze loops the last "grain_length" of incoming samples when activated.
Shift is the simple granular pitch shift. It is the same thing as freeze really but it gets a new chunk of samples when the grain_length has passed (It's not just the prev sample being done as if the rate is faster than normal speed it has to loop).
 
Haha, I love how this thread escalated. This motivates me to continue to play with my code.

PaulStoffregen said:
My hope is to write an optimized low-level engine in the Teensy Audio library which does the high speed, low-level work efficiently. The actual decisions about which grains to play, when to mix each one, and what parameters each will use would happen in other libs, to be developed separately.
Click to expand...

That would be awesome! Could you post a imaginary example on how this API is used? That would make it easier for me to understand the general idea and comment on it.
 
Status
Not open for further replies.
Back
Top