Issues with custom audio object

toyosm

Active member
Hello, I'm working on a granular reverb composed by the Plate Reverb object from @Pio and a custom Granular object (alongside other stuff to control a custom board, etc). If I connect my granular object alone or pre reverb, the code uploads but doesn't work and the usb audio interface keeps rebooting. If I place the granular object after the reverb all works well. This puzzles me and don't know what can be happening. I tried to make another sketch with a custom object that only pases the input thru a buffer and then reads the buffer to the output and have the same behavior. I'm using the Plate Reverb object that outputs int16_t samples, not the 32 one.

Here's my granular code, may be a little untidy due to all tests I'm doing

.h
C++:
#include <sys/_stdint.h>
#ifndef AUDIO_EFFECT_MAX_GRANULAR_H
#define AUDIO_EFFECT_MAX_GRANULAR_H

#include <Arduino.h>
#include <Audio.h>
#include <AudioStream.h>
#include "arm_math.h"

#define BUFFER_SIZE     60000       // Size of the circular buffer
#define MAX_SPACING     127         // Maximum spacing between grains
#define MAX_GRAIN_SIZE 20000       // Maximum grain size
#define ENV_TRIANGULAR  0           // Triangular type of envelope for grains
#define ENV_RAMP        1           // Ramp type of envelope for grains
#define ENV_TRAPEZOID   2           // Trapezoidal type of envelope for grains
#define ENV_SIN         3           // Sinusoidal type of envelope for grains

class AudioEffectMaxGranular : public AudioStream {
public:
    AudioEffectMaxGranular();
  
    virtual void update(void);
    void triggerGrain(int speed, bool reverse = false);
    void setGrainSize(uint16_t size_ms);
    void setGrainPitch(int speed);
    void setReversePlayback(bool reverse);
    void setRandomizeGrains(bool randomize);
    void setGrainSpacing(uint16_t spacing_ms);  // Set spacing between grain triggers in milliseconds
    void setSweepSpeed(uint16_t speed_ms);
    void setRandomizeSpeed(bool randomSpeed);
    void setGrainEnvelope(byte type);
    void setGrainNumber(uint16_t number);

private:
    const static int MAX_GRAINS = 5;   

    struct Grain {
        bool active;                      // Grain active or inactive
        uint32_t readPosition;            // Grain read position
        uint32_t initialPosition;         // Grain initial position on the buffer
        uint32_t finalPosition;           // Grain final position on the buffer
        uint16_t playbackSpeed;              // Grain playback speed
        bool reverse;                     // Reverse playback active or inactive
        uint32_t age;                     // Age of the grain in samples
        uint16_t duration;                // Grain duration
    } grains[MAX_GRAINS];

    byte envelopeType;
    uint16_t grainDuration;
    int grainSpacing;  // Spacing between grains in samples
    int playbackSpeed;
    uint16_t grainBufferSpeed;
    bool reversePlayback;
    bool randomizeGrains;
    bool randomizeSpeed;
    int16_t bufferLeft[BUFFER_SIZE];
    int16_t bufferRight[BUFFER_SIZE];
    uint32_t bufferIndex;
    float sampleRate;
    uint32_t nextGrainTime;  // Time in samples when the next grain should be triggered
    int usedGrains;
    float inputAttenuation;
    uint32_t upperLimit;

    audio_block_t *inputQueueArray[2];

    float envelope(uint32_t age, float duration, byte type);
  
    int16_t processGrain(Grain &grain, int16_t *buffer);
    float windowHann(uint32_t age, uint32_t duration);
    void resetGrains(void);
  
};

#endif // AUDIO_EFFECT_MAX_GRANULAR_H

.cpp
C++:
#include <strings.h>
#include <stdint.h>
#include "AudioEffectMaxGranular.h"
#include <Arduino.h>

AudioEffectMaxGranular::AudioEffectMaxGranular()
: AudioStream(2, inputQueueArray),
  grainDuration(10000),
  grainSpacing(0),
  playbackSpeed(1),
  reversePlayback(false),
  randomizeGrains(false),
  bufferIndex(0),
  sampleRate(AUDIO_SAMPLE_RATE_EXACT),  // Initialize member variables
  nextGrainTime(0)
{
    for (int i = 0; i < MAX_GRAINS; i++) {
        grains[i].active = false;
    }

    for(int i = 0; i <= BUFFER_SIZE; i++)
    {
      bufferLeft[i] = 0;
      bufferRight[i] = 0;
    }

    randomizeSpeed = false;
    bufferIndex = 0;
    usedGrains = MAX_GRAINS;
    nextGrainTime = 0;
    grainBufferSpeed = 127;
    inputAttenuation = 0.75;
    randomSeed(532);
    envelopeType = ENV_TRAPEZOID;
    overlapping = 1000;
    Serial.begin(9600);
}

void AudioEffectMaxGranular::update(void)
{
    audio_block_t *blockLeft, *blockRight;              // Input Blocks
    audio_block_t *outblockL, *outblockR;               // Output Blocks
  
    blockLeft = receiveReadOnly(0);                     // Receive Left channel
    blockRight = receiveReadOnly(1);                    // Receive Right channel
    outblockL = allocate();                             // Allocate memory for Left channel output
    outblockR = allocate();                             // Allocate memory for Right channel output

    if (!outblockL || !outblockR)
    {
      if (outblockL) release(outblockL);
      if (outblockR) release(outblockR);
      if (blockLeft) release((audio_block_t *)blockLeft);
      if (blockRight) release((audio_block_t *)blockRight);
      return;
      }

    // Store incoming audio in the circular buffer
    for (int i = 0; i < AUDIO_BLOCK_SAMPLES; i++)
    {
        bufferLeft[bufferIndex] = blockLeft->data[i] * inputAttenuation;
        bufferRight[bufferIndex] = blockRight->data[i] * inputAttenuation;
        bufferIndex ++;
        if(bufferIndex > BUFFER_SIZE)
          bufferIndex = 0;
  
        // Trigger a grain if it's time
        if (nextGrainTime == 0)
        {
            triggerGrain(playbackSpeed, reversePlayback);
            nextGrainTime = grainBufferSpeed;
        }
        else
        {
            nextGrainTime--;
        }
    }
    release(blockLeft);
    release(blockRight);
  
    for(int i = 0; i < AUDIO_BLOCK_SAMPLES; i++)
    {
    // Process grains and output audio
        int16_t sampleLeft = 0;
        int16_t sampleRight = 0;

        for (int j = 0; j < usedGrains; j = j+2)
        {
            if (grains[j].active)
            {
               sampleRight += processGrain(grains[j], bufferLeft);
            }
        }
        for(int l = 1; l < usedGrains; l = l+2)
        {
            if (grains[l].active)
            {
                sampleLeft += processGrain(grains[l], bufferLeft);
            }

        }
        outblockL->data[i] = sampleLeft;
        outblockR->data[i] = sampleRight;
    }

    transmit(outblockL, 0);
    transmit(outblockR, 1);
    release(outblockL);
    release(outblockR);
}

  
// TRIGGER GRAINS
// FILL THE GRAIN STRUCTURE WITH ALL THE NECESSARY DATA
void AudioEffectMaxGranular::triggerGrain(int speed, bool reverse = false)
{
  __disable_irq();
  for(byte index = 0; index < usedGrains; index ++)
  {
    if(grains[index].active == false)                                                                   // If the grain in inactive, fill the data and activate
    {
      grains[index].playbackSpeed = speed;                                                              // Set grain playback speed
      grains[index].reverse = reverse;                                                                  // Set grain playback direction
      grains[index].age = 0;                                                                            // Reset grain age
    
      if(randomizeSpeed)
      {
        grains[index].playbackSpeed = random(1,3);                                                      // Randomize playback speed between 0 and +1 octave
      }
    
      grains[index].duration = (uint32_t)(grainDuration / grains[index].playbackSpeed);


      // Fill initial and final position of grains in buffer
      if(randomizeGrains)                                                                               // Random mode
      {
        upperLimit = BUFFER_SIZE - grains[index].duration - grainSpacing;
        grains[index].initialPosition = random(0, upperLimit);                       // Randomize the grain initial position
        grains[index].finalPosition = grains[index].initialPosition + grains[index].duration;                    // Set the final position
        grains[index].active = random(0, 2);                                                            // Randomize grain activation
      }
    
      else                                                                                              // Normal mode
      {
        upperLimit = BUFFER_SIZE - grains[index].duration - grainSpacing;
        grains[index].initialPosition = (index * (grains[index].duration  + grainSpacing));                   // Initial position: Number of the grain * (Duration of Grain + Spacing of Grain) + 1 margin sample for beggining of the buffer
        if(grains[index].initialPosition > upperLimit)             // Check if the initial position of the grain fits the buffer
        {
          grains[index].initialPosition = 0;                 // Fix the initial to the maximum available     
        }

        grains[index].finalPosition = grains[index].initialPosition + grains[index].duration ;                    // Final position: Initial + grain duration
        if(grains[index].finalPosition >= BUFFER_SIZE)                                                  // If the final position excedes the buffer
          grains[index].finalPosition = BUFFER_SIZE;                                                    // Set the final position to the end of the buffer (cropped grain)
      
        grains[index].active = true;                                                                    // Activate the grain
      }
    

      // Read position setting
      if(grains[index].reverse)
      {
        grains[index].readPosition = grains[index].finalPosition;                                       // Set the starting read position to the end of the grain
      }                                                                         // Reverse mode
      else                                                                                              // Normal mode
      {
        grains[index].readPosition = grains[index].initialPosition;                                     // Set the starting read position to the begining of the grain
      }
    }
  }
  __enable_irq();

}

void AudioEffectMaxGranular::setGrainSize(uint16_t size_ms)
{
  if(size_ms < MAX_GRAIN_SIZE)
  {
    grainDuration = size_ms;
  }
  else
  {
    grainDuration = MAX_GRAIN_SIZE;
  }
  resetGrains();
}

void AudioEffectMaxGranular::setGrainPitch(int speed) {
    playbackSpeed = speed;
    resetGrains();
}

void AudioEffectMaxGranular::setReversePlayback(bool reverse) {
    reversePlayback = reverse;
    resetGrains();
}

void AudioEffectMaxGranular::setRandomizeGrains(bool randomize) {
    randomizeGrains = randomize;
    resetGrains();
}

void AudioEffectMaxGranular::setSweepSpeed(uint16_t speed_ms){
  grainBufferSpeed = speed_ms * (sampleRate / 1000.0);                                  // mS to samples
  nextGrainTime = 0;                                                                    // Reset grain trigger counter
  resetGrains();
}

void AudioEffectMaxGranular::setRandomizeSpeed(bool randomSpeed)
{
  randomizeSpeed = randomSpeed;
  resetGrains();
}

void AudioEffectMaxGranular::setGrainSpacing(uint16_t spacing_ms)
{
  if(spacing_ms < MAX_SPACING)
  {
    grainSpacing = spacing_ms;
  }
  else
  {
    grainSpacing = MAX_SPACING;
  }
  resetGrains();
}

void AudioEffectMaxGranular::setGrainNumber(uint16_t number)
{
  usedGrains = number;                                                                  // Apply new grain number
  resetGrains();
 
}

void AudioEffectMaxGranular::resetGrains(void)
{
  for(int i = 0; i < MAX_GRAINS; i++)
  {
    grains[i].active = false;
  }
 
  for(int i = 0; i <= BUFFER_SIZE; i++)
  {
    bufferLeft[i] = 0;
    bufferRight[i] = 0;
  }
  nextGrainTime = 0;
}

void AudioEffectMaxGranular::setGrainEnvelope(byte type)
{
  envelopeType = type;
}

float AudioEffectMaxGranular::envelope(uint32_t age, float duration, byte type) {
    float fadeIn;                                                    // Fade In time
    float fadeOut;                                                   // Fade Out time
    uint16_t fadeInTime;
    uint16_t fadeOutTime;

    switch(type)
    {
      case ENV_TRIANGULAR:
        fadeInTime = (0.5 * duration);
        fadeIn = age / fadeInTime;
        fadeOutTime = 0.5 * duration;
        fadeOut = (duration - age) / fadeOutTime;
        break;

      case ENV_RAMP:
        //fadeIn = age / (0.9 * duration);
        //fadeOut = (duration - age) / (0.1 * duration);
        fadeInTime = 0;
        fadeIn = 1;
        fadeOutTime = 0;
        fadeOut = 1;
        break;

      case ENV_TRAPEZOID:
        fadeInTime = 0.3 * duration;
        fadeIn = age / fadeInTime;
        fadeOutTime = 0.3 * duration;
        fadeOut = (duration - age) / fadeOutTime;
        break;

      case ENV_SIN:
        fadeInTime = 0.1 * duration;
        fadeIn = sin(age*M_PI/20);
        fadeOutTime = 0.1 * duration;
        fadeOut = sin(age * (-M_PI/20) + PI/2);
        break;
    }

    if (age < fadeInTime)
    {
        return fadeIn;                                                            // Fade-in
    }
    else if (age > (duration - fadeOutTime))
    {
        return fadeOut;                                              // Fade-out
    }
    else
    {
        return 1.0;                                                                     // Sustained part of the grain
    }
}

int16_t AudioEffectMaxGranular::processGrain(Grain &grain, int16_t *buffer)
{
  int16_t sample = buffer[grain.readPosition];                                          // Take sample from actual readPosition
  int16_t nextSample = buffer[grain.readPosition + 1];                                  // Take sample from next position
  int16_t outSample;                                                                    // Output sample

  __disable_irq();
  if(grain.age >= grain.duration)                                                        // Check if the grain was completly reproduced
  {
    grain.active = false;                                                               // If so, deactivate
    return 0;                                                                           // Return 0
  }                                                                                  // +0 Octaves and above
  // Now move readPosition
  if(grain.reverse)                                                                   // Reverse mode
  {
    grain.readPosition = grain.readPosition - grain.playbackSpeed;          // Move the read position according to the playback speed (backwards)
  }
  else                                                                                // Normal mode
  {
    grain.readPosition = grain.readPosition + grain.playbackSpeed;          // Move the read position according to the playback speed
  }
  outSample = (sample) * windowHann(grain.age, grain.duration);
  grain.age ++;                                                                          // Increment the grain age                                                  // Apply envelope
  __enable_irq();
  return outSample;                                                                 
}

float AudioEffectMaxGranular::windowHann(uint32_t age, uint32_t duration) {
    return 0.5 * (1.0 - cos(2.0 * M_PI * age / duration));
}
 
Haven't had a chance to fully review the code but from the description can you check how your update() deals with null blocks i.e. same as you are checking the outblocks, check blockLeft & blockRight before processing them.
 
Also disabling the interrupts globally within the update/triggerGrain, which happens inside the audio interrupt is probably not a good idea.
 
I agree with both of the recommendations above.

A smaller problem is where you do 'if(bufferIndex > BUFFER_SIZE)'. As written, I think that you will overrun your buffer by one. To avoid this, use '>=' I stead of just '>'.
 
Thank you so much guys, that was the issue, dealing with the input null pointers.

I agree with both of the recommendations above.

A smaller problem is where you do 'if(bufferIndex > BUFFER_SIZE)'. As written, I think that you will overrun your buffer by one. To avoid this, use '>=' I stead of just '>'.
I don't think so, the sample is loaded first and then the index incremented. So, when you reach the BUFFER_SIZE number of sample, increment and fulfill the condition by being greater than the macro. Tried anyways, no change at all.

Another question that I have, maybe there's another thing I need to do. When using this with USB out, after a while (30/40/50 seconds, couldn't measure it) a "tic" starts to came out with the grains. That tick doesn't go out on the I2S signal (or at least couldn't notice it).
Tried to see if was some overflow of the buffer, any grain that fell outside boundaries, clipping, envelope of the grains, etc but no luck. Any thoughs? Also, there's some more documentation about this good practices (like the release of the null blocks), couldn't find it on the site.
 
Hi,

>>> I don't think so, the sample is loaded first and then the index incremented. So, when you reach the BUFFER_SIZE number of sample, increment and fulfill the condition by being greater than the macro.<<<

The problem is that your code allows the index to have the value of BUFFER_SIZE. It will write data to your arrays at this location. But, your arrays are not valid at this location. The last allowed value is at BUFFER_SIZE - 1. Therefore you're writing off the end of your array, which is very dangerous.

Sometimes, such overrun errors cause the system to crash. Or, if that one memory location that you're overwriting has not been allocated to another use, your overrun error might have no consequences. It's unpredictable.

Remember, in zero based languages like C/C++, when you allocate an array to have a length of BUFFER_SIZE, the allowable indices span 0 to BUFFER_SIZE-1.

So, while I do believe that this is a bug in your code, it does not appear to be the critical bug.

Chip
 
If you have tried fixes for the other two suggestions, the next thing I would try would be to strip out your entire update() and replace it with one that simply copies the input to the output. Then, I would slowly add in the pieces of the calculation. At some point, you'll add in enough of the calculation that your rebooting symptom will appear again. That'll help you find where the issue is.

Chip
 
Hi,

>>> I don't think so, the sample is loaded first and then the index incremented. So, when you reach the BUFFER_SIZE number of sample, increment and fulfill the condition by being greater than the macro.<<<

The problem is that your code allows the index to have the value of BUFFER_SIZE. It will write data to your arrays at this location. But, your arrays are not valid at this location. The last allowed value is at BUFFER_SIZE - 1. Therefore you're writing off the end of your array, which is very dangerous.

Sometimes, such overrun errors cause the system to crash. Or, if that one memory location that you're overwriting has not been allocated to another use, your overrun error might have no consequences. It's unpredictable.

Remember, in zero based languages like C/C++, when you allocate an array to have a length of BUFFER_SIZE, the allowable indices span 0 to BUFFER_SIZE-1.

So, while I do believe that this is a bug in your code, it does not appear to be the critical bug.

Chip
You're totally right!, a little burned out with all this that I forgotten that the index started at 0 and have BUFFER_SIZE elements. Thanks to explain again like a 5 yo
If you have tried fixes for the other two suggestions, the next thing I would try would be to strip out your entire update() and replace it with one that simply copies the input to the output. Then, I would slowly add in the pieces of the calculation. At some point, you'll add in enough of the calculation that your rebooting symptom will appear again. That'll help you find where the issue is.

Chip
The rebooting issue was fixed with releasing the null pointers at the input.
 
That's great you've found the problem - looks like an interesting effect.
A couple of minor additional comments
  • On the USB tick issue - I don't have much experience with the USB audio but I know it doesn't work well if you change AUDIO_BLOCK_SAMPLES away from the default - don't know if this is applicable here
  • BUFFER_SIZE - you might also want to just check processGrain() - you might need to check when moving the read position that grain.readPosition = grain.readPosition - grain.playbackSpeed; doesn't take you out of bounds ( same with the + grain.playbackSpeed;)
  • Finally, I noticed you might have a cut /paste error in update() - sampleRight += processGrain(grains[j], bufferLeft); , should this be bufferRight?
cheers, Paul
 
That's great you've found the problem - looks like an interesting effect.
A couple of minor additional comments
  • On the USB tick issue - I don't have much experience with the USB audio but I know it doesn't work well if you change AUDIO_BLOCK_SAMPLES away from the default - don't know if this is applicable here
  • BUFFER_SIZE - you might also want to just check processGrain() - you might need to check when moving the read position that grain.readPosition = grain.readPosition - grain.playbackSpeed; doesn't take you out of bounds ( same with the + grain.playbackSpeed;)
  • Finally, I noticed you might have a cut /paste error in update() - sampleRight += processGrain(grains[j], bufferLeft); , should this be bufferRight?
cheers, Paul

  • I didn't change the block samples macro from default, but I have to check the reverb's code for that.
  • Each grain has a variable that's grain age, should not exceed the boundaries with this. Prior to this have another check up to exit the processing if the read point reaches it's final destination (initial or final according to the direction of the buffer). I didn't noticed any changes with this.
  • It's not a cut/paste error, initially I had separated buffer for left and right channels, then I moved to a unique buffer to make it bigger to accommodate bigger grains. Then I output even grains on the left channel and odd grains on the right channel. It's like a stereo inout but with "mono" processing. Is not ideal, but since I cannot add extra RAM (using the micro module from sparkfun, not the teensy one) was a good trade off for bigger grains.
 
Back
Top