Teesny 4.1 + Audio Shield Crashing

o.w.varley

Well-known member
Hey all,

I'm trying to pin point an issue I've got that's causing the Teensy to crash. I've probably got too much code to share in full, however, I'll attempt to explain what I'm doing and that might help pin-point where the issue is. I'm building a program that allows audio to be transmitted and received via UDP. An AudioController class is defined with three 4 channel mixers, two take 8 inputs and combine them all into a single output to an I2C, the class definition for this with the Audio component definitions is below:

Code:
#pragma once
#include <SPI.h>
#include <Audio.h>
#include <Wire.h>
#include <SerialFlash.h>
#include <list>
#include "Constants.h"
#include "AudioVoiceOut.h"
#include "AudioVoiceIn.h"

class AudioController
{
	const int DEFAULT_GAIN = 40;
	static const int NUM_INPUT_CHANNELS = 8; 
	const int INTERCOM_MIX_CHANNEL = 0;
	const int MIC_MIX_CHANNEL = 1;
	const int VUHF1_MIX_CHANNEL = 2;
	const int VUHF2_MIX_CHANNEL = 3;
	const int TAC1_MIX_CHANNEL = 0;
	const int TAC2_MIX_CHANNEL = 1;
	const int DAS_MIX_CHANNEL = 2;
	const int AIRCRAFT_MIX_CHANNEL = 3;

public:
	AudioController() : _channelOut(PORT, PACKET_INTERCOM),

						_patchCord1(_intercomIn, 0, _preMix1, INTERCOM_MIX_CHANNEL),
						_patchCord2(_mic, 0, _preMix1, MIC_MIX_CHANNEL),
						_patchCord3(_VUHF1, 0, _preMix1, VUHF1_MIX_CHANNEL),
						_patchCord4(_VUHF2, 0, _preMix1, VUHF2_MIX_CHANNEL),
						_patchCord5(_TAC1, 0, _preMix1, TAC1_MIX_CHANNEL),
						_patchCord6(_TAC2, 0, _preMix2, TAC2_MIX_CHANNEL),
						_patchCord7(_DAS, 0, _preMix2, DAS_MIX_CHANNEL),
						_patchCord8(_Aircraft, 0, _preMix2, AIRCRAFT_MIX_CHANNEL),

						_patchCord9(_mic, 0, _channelOut, 0),
						_patchCord10(_preMix1, 0, _finalMix, 0),
						_patchCord11(_preMix2, 0, _finalMix, 1),
						_patchCord12(_finalMix, 0, _headset, 0),
						_patchCord13(_finalMix, 0, _headset, 1)
	{
		const float VOLUME = 0.65;

		AudioMemory(64);

		_sgtl5000.enable();
		_sgtl5000.volume(VOLUME);
		_sgtl5000.inputSelect(AUDIO_INPUT_MIC);
		_sgtl5000.micGain(DEFAULT_GAIN);

		delay(1000);

		InitChannels();
		Serial.println("Audio setup complete");
	}

	void AdjustVolume(uint8_t packetType, uint8_t volume);
	void Begin(bool isSimA);
	std::list<uint16_t> GetBufferSizes();
	AudioVoiceOut* GetVoiceOut() { return &_channelOut; }
	void ReadUDP();
	void StartTransmitting(uint8_t packetType);
	void StopTransmitting();

private:
	void InitChannels();
	void ReadAudioPacketFromUDP();
	void ReadChannelChangePacketFromUDP();

	AudioInputI2S _mic;

	AudioVoiceIn _intercomIn;
	AudioVoiceIn _VUHF1;
	AudioVoiceIn _VUHF2;
	AudioVoiceIn _TAC1;
	AudioVoiceIn _TAC2;

	AudioVoiceIn _DAS;
	AudioVoiceIn _Aircraft;

	AudioMixer4 _preMix1;
	AudioMixer4 _preMix2;
	AudioMixer4 _finalMix;

	AudioVoiceOut _channelOut;
	AudioOutputI2S _headset;

	AudioConnection _patchCord1;
	AudioConnection _patchCord2;
	AudioConnection _patchCord3;
	AudioConnection _patchCord4;
	AudioConnection _patchCord5;
	AudioConnection _patchCord6;
	AudioConnection _patchCord7;
	AudioConnection _patchCord8;
	AudioConnection _patchCord9;
	AudioConnection _patchCord10;
	AudioConnection _patchCord11;
	AudioConnection _patchCord12;
	AudioConnection _patchCord13;

	AudioControlSGTL5000 _sgtl5000;

	uint32_t _intercomChannel;
	EthernetUDP _client;

	AudioVoiceIn* _channels[NUM_INPUT_CHANNELS];

};

The AudioController reads UDP Packets and, when it finds packets that relate to audio data it enqueues these within buffers in the AudioVoiceIn objects it contains. From what I'm testing, the issue looks to be in the ReadUDP function, this checks the first byte of the UDP packet and if its an audio packet it calls ReadAudioPAcketFromUDP code for this below:

Code:
void AudioController::ReadAudioPacketFromUDP()
{
    uint8_t packetType = PACKET_NONE;
    uint32_t channelId;
    int16_t* packet = new int16_t[AUDIO_BLOCK_SAMPLES];

    // Packet format for Audio
    // Header - Byte
    // Packet Type - Byte
    // ChannelID - 4 Bytes (Unsigned Int32)
    // Packet Data - 256 Bytes

    _client.read((uint8_t*)&packetType, sizeof(packetType));
    _client.read((uint8_t*)&channelId, sizeof(channelId));
    _client.read((uint8_t*)packet, AUDIO_BLOCK_SAMPLES * 2);

    if (packetType < PACKET_MIN || packetType > PACKET_MAX)
    {
        Serial.printf("Invalid packetType (%d) ignoring packet.\n", packetType);
        delete[] packet;
        packet = NULL;

        return;
    }

    AudioVoiceIn* channel = _channels[packetType];

    // Special handling is included here for the GOD channel. This will allow an instructor to transmit on a single channel
    // and be received by both SIM A and SIM B.
    if (channel->GetChannel() == channelId || channelId == CHANNEL_INTERCOM_GOD)
    {
        channel->Enqueue(packet);
    }
    else
    {
        delete[] packet;
        packet = NULL;
    }

}

The AudioVoiceIn class contains within it a buffer into which audio blocks are enqueued. The memory for these blocks is allocated within the function above, as you can see I'm clearing the memory allocated to the packet in any of the cases that enqueue isn't called. Within the AudioVoiceIn class the Update function is responsible for releasing the memory allocated to packet. This is done within the Update function which is shown below:

Code:
void AudioVoiceIn::update(void)
{
	audio_block_t* block;

	// only update if we're playing
	if (!_enabled) return;

	// don't update or allocate memory if there is nothing to play
	if (_buffer.size() <= 0) return;

	// allocate the audio blocks to transmit
	block = allocate();
	if (block == NULL) return;

	if (_buffer.size() > 0)
	{
		int16_t* packet = _buffer.pop_front();

		memcpy(block->data, packet, sizeof(block->data));

		transmit(block);

		delete[] packet;
		packet = NULL;
	}

	release(block);
}

For _buffer I was initially using an std::queue, however, as I thought this might be the issue I swapped recently across to using TonTon's Circular Buffer but the problem remains. I've tried wrapping the Enqueue call with disabling/enabling interrupts but this doesn't make any difference either. I've got a bunch of crash reports, however, when using addr2line I get ??:?, Crash Reports are below:

Code:
  CrashReport:
  A problem occurred at (system time) 10:33:43
  Code was executing from address 0x42D94
  CFSR: 82
	(DACCVIOL) Data Access Violation
	(MMARVALID) Accessed Address: 0x9400A2
  Temperature inside the chip was 65.00 °C
  Startup CPU clock speed is 600MHz
  Reboot was caused by auto reboot after fault or bad interrupt detected

  CrashReport:
  A problem occurred at (system time) 10:36:50
  Code was executing from address 0x42D94
  CFSR: 82
	(DACCVIOL) Data Access Violation
	(MMARVALID) Accessed Address: 0xFFFEFFF0
  Temperature inside the chip was 65.77 °C
  Startup CPU clock speed is 600MHz
  Reboot was caused by auto reboot after fault or bad interrupt detected

  A problem occurred at (system time) 10:41:35
  Code was executing from address 0x42DE0
  CFSR: 82
	(DACCVIOL) Data Access Violation
	(MMARVALID) Accessed Address: 0xFFF4FFFB
  Temperature inside the chip was 65.00 °C
  Startup CPU clock speed is 600MHz
  Reboot was caused by auto reboot after fault or bad interrupt detected

In terms of diagnostic data, I'm monitoring RAM2, Audio Memory, CPU usage levels every second and none of these seem to be an issue. RAM2 reports around 300k free, Audio memory max is about 9 (With 64 blocks allocated as its max) and the CPU never gets above 2.0.

Any top tips on how to debug issues like this would be really appreciated. Even just being able to hone in on the suspect line of code would be a start as its frustrating that addr2line doesn't work!

Thanks,

Owen
 
Check every pointer for NULL - you don't check the result of pop_front() for instance.

Some of your functions do checking after allocating resources, then have to return the resources unused if
the checks fail. Cleaner to do as much checking as you can before allocating resources.

Code:
void AudioVoiceIn::update(void)
{
	audio_block_t* block;

Here you aren't initializing automatic pointer variables to NULL, _automatic variables aren't zeroed out_, its dangerous to have
a pointer variable with junk in it, because even checking for NULL won't help you then.

I'd suggest changing update like this:
Code:
void AudioVoiceIn::update(void)
{
	// only update if we're playing and there's data
	if (!_enabled || _buffer.size() <= 0) 
                return;

	int16_t* packet = _buffer.pop_front();
        if (packet == NULL)
                return;

        audio_block_t* block = allocate();
	if (block != NULL)
        {
		memcpy(block->data, packet, sizeof(block->data));
	        transmit(block);
                release(block);
        }
        delete[] packet;
}
Which has the advantage of being single-assignment as well as always checking before doing.

Though where the actual error is I don't know, but cleanliness with pointers is likely to help.
 
Thanks for the reply @MarkT, I'll make those changes and see if that makes a difference. It it helps, when I refer to having plenty of 'RAM' I'm referring to the heap on RAM2, the function I'm using to calculate it is below:

Code:
extern unsigned long _heap_start;
extern unsigned long _heap_end;
extern char* __brkval;

int GetFreeRAM2()
{
	return (char*)&_heap_end - __brkval;
}

Given I've just spent the morning trawling through some excellent forum posts that cover how Teensy 4.1's memory is allocated I'm going to look at this in more detail. I noticed that when I tried to upload the code optimised for debugging it fails to upload which makes me think I need to better understand my memory usage. Another interesting thing I've noticed is that after building (VS2019 + vMicro) I get this message:

Code:
   Opening Teensy Loader
mktinyfat*: Unable to parse C:\Users\owenv\AppData\Local\Temp\VMBuilds\Crew_Panel\teensy41\Debug\Crew_Panel.ino.elf
Uploading 'Crew_Panel' to 'Teensy 4.1' using 'COM9'
	The upload process has finished.

Previously (I can't remember how far back) this was correctly reporting the program's memory allocations.
 
Hey @MarKT, I've implemented those changes this morning and added NULL initialisation and NULL checking to every pointer I use and unfortunately it's still crashing.

I'm doing some testing now to look in more detail at the stack and heap usage.
 
I'm focusing on the fact that neither Teensy_Size.exe nor mktinyfat.exe can actually handle the .elf file. It doesn't look like the file is too big, the ELF is only 4632KB and the HEX is 1673KB. Looking at the error messages I'm getting a pretty consistent address of which part of the code is causing the failure, however, addr2line returns ??:? which feels like it's related to the issues with Teensy_Size and mktinyfat above.

When streaming UDP input to the AudioController (See code in original post) to all channels, I get a crash after about 2:20 seconds of playing. When I drop this down to a single channel it took about 2 hours to crash, however, it still crashed with the same error (CFSR 81 i.e. DACCVIOL with MMARVALID). That tells me it has to be some sort of small memory leak within the channel itself (AudioVoiceIn) but I'm finding it really frustrating that RAM2 seems to have plenty of capacity (300K+) at the point of crash.
 
I think I've got to the bottom of it, two issues that compounded each other.

Might have helped if, in the original post, I mentioned that I was dynamically allocating 256 bytes 7 times every ~3 milliseconds. The end result was that I was suffering from memory fragmentation caused by the significant number of dynamic allocations I was doing. This wasn't being helped by the fact that the function I'm using for reporting the RAM2 level just wasn't accurate. A simple bit of test code in which I allocate large chunks of RAM2 would result in no changes to the reported available RAM2 levels. I've not resolved the incorrect RAM2 reporting yet, but I'll look at that next.

The fix for the main problem was to swap to using static allocation. Instead of using std::queue or the CircularBuffer library I just wrote a simple circular buffer that grabs its memory when the Teensy starts up. No crashes yet...
 
Big thanks to @KurtE and @FrankB, I've just used imxrt-size and it's great:

Code:
estack:20028000 ebss:200172e0

FlexRAM section ITCM+DTCM = 512 KB
    Config : aabfffff (DDDDDIIIIIIIIIII)
    ITCM : 342144 B     (94.92% of  352 KB)
    DTCM :  94944 B     (57.95% of  160 KB)
    Available for Stack:  68896
OCRAM: 512KB
    DMAMEM:  30080 B    ( 5.74% of  512 KB)
    Available for Heap: 494208 B        (94.26% of  512 KB)
Flash: 588312 B ( 7.24% of 7936 KB)

Looks like I'm nearly maxing out the ITCM as well.
 
No, ITCM grows automagically in 32KB steps (and takes it's additional memory from DTCm which decreases in 32KB steps.. :)
They both share the same 512KB physical memory.

512KB = ITCM + DTCM

I could go more in detail here, but I think it's already too much here..
An little hint: You can save loads of ITCM if you use the keyword "FLASHMEM" for code that runs only once (inititialization code, etc..)
FLASHMEM void initme(void) {....} .. this will increase the available "DTCM" if it goes beyond the next 32KB step..
 
And if there are large segments of code used only on startup() - or rarely - and losing DTCM were a problem - that code can be kept in FLASH with FLASHMEM

If using current TD 1.54 the (near) equivalent info of imxrt-size is part of the IDE build
 
Also, you can use "PROGMEM" your large arrays of const data. It will placed in flash memeory then, not in the "expensive" RAM. Remember, there is a 32KB *really* fast cache for the flash....

PROGMEM const uint8_t myData[] = {.....}
 
And if there are large segments of code used only on startup() - or rarely - and losing DTCM were a problem - that code can be kept in FLASH with FLASHMEM

If using current TD 1.54 the (near) equivalent info of imxrt-size is part of the IDE build

Thanks @Frank and @defragster. I've now used FLASHMEM and PROGMEM to further optimise memory usage. It ran for 10 hours last night without crashing or leaking which is great.

I'm using 1.54, however, the Teensy_Size.exe that runs at the end of the build fails to run on the compiled file. It runs fine on other sketches I've got so I assumed this was due to the size and complexity of the program I'm working on at the moment. Interesting addr2line also fails on the compiled file if the addressed line of code is something I've written and not an external library.
 
There is something wrong.. this should not happen..
I'd try to delete Arduino & Teensyduino and do a fresh install.
 
Hey Frank,

It's only taken me 4 months but I finally got the time to re-install Arduino and Teensyduino and it fixed this problem straight away:

Code:
teensy_size*: Memory Usage on Teensy 4.1
teensy_size*: FLASH: code:482172, data:99144, headers:9044   free for files:7536104
teensy_size*: RAM1: variables:93380, code:343136, padding:17312   free for local variables:70460
teensy_size*: RAM2: variables:30080  free for malloc\new:494208
 
Back
Top