malloc/free for EXTMEM and DTCM

From top, this is the first commit which doesn't work

Commit: 7a458043ea6642c861425095526b24f567385b03 [7a45804]
Parents: 4da51c16fd
Author: PaulStoffregen <paul@pjrc.com>
Date: Sonntag, 25. Oktober 2020 17:49:34
Committer: PaulStoffregen
Add experimental interface

Think maybe the Oct 23rd or 24th commit was 1.54b2 just looking at the how long ago the commits were made?
Fits to this information
 
Think maybe the Oct 23rd or 24th commit was 1.54b2 just looking at the how long ago the commits were made?

I can confirm I see the same thing after sync.

I updated my install to use the cores
Code:
C:\arduino-1.8.13\hardware\teensy\avr\cores>ren teensy4 teensy4_release

C:\arduino-1.8.13\hardware\teensy\avr\cores>mklink /D teensy4 d:\GitHub\cores\teensy4
symbolic link created for teensy4 <<===>> d:\GitHub\cores\teensy4
And simple program did not print anything:
Code:
xtern "C" uint8_t external_psram_size;
void setup () {
  while (!Serial && millis() < 4000) ;
  Serial.begin(115300);
}

void loop() {
  Serial.println(F_BUS_ACTUAL, DEC);
  Serial.printf("PSRAM: %d\n", external_psram_size);
  delay(1000);
 }

Then I exited Arduino, restored back to using the released version of core:
Code:
C:\arduino-1.8.13\hardware\teensy\avr\cores>ren teensy4 teensy4_github

C:\arduino-1.8.13\hardware\teensy\avr\cores>ren teensy4_release teensy4

And it is printing...
 
if I undo your changes from 'Add experimental interface' in usb_desc.h:
I.e.,
Code:
#define NUM_INTERFACE		2   // back to 2 instead of 3  
...
//#define EXPERIMENTAL_INTERFACE 2  // commented

it works again.
 
I am now doing a comparison of the two directories

It sees these files are different:
screenshot.jpg
 
if I undo your changes from 'Add experimental interface' in usb_desc.h:
I.e.,
Code:
#define NUM_INTERFACE		2   // back to 2 instead of 3  
...
//#define EXPERIMENTAL_INTERFACE 2  // commented

it works again.

Confirmed. Working again
 
With that said just tried Triple_Serial_test and only seeing 1 Serial port in Windows 10 with T4.1 and t4.0
 
Ugh, looks like several minor descriptor problems have crept in. And my little experiment a few days ago wasn't such a good idea. Here's a fix.

https://github.com/PaulStoffregen/cores/commit/e599c1acc71ad84b33b6a3aafa7e2321d27c6b2e

With this, I'm seeing all 3 serial ports appear on Windows 7. Will need to re-image this machine to test Windows 10....

It is working now on T4.1 For simple Serial.

I also tried out my USBToSerialMulti sketch - Wonder if we should add something like this to examples:
Serial -> Serial1
SerialUSB1 -> Serial2
SerialUSB2 -> Serial3

Or which ever serial ports you define... And it appears to be working:
Code:
/* USB to Serial - Teensy becomes a USB to Serial converter
   http://dorkbotpdx.org/blog/paul/teensy_as_benito_at_57600_baud

   You must select Serial from the "Tools > USB Type" menu

   This example code is in the public domain.
*/

// set this to the hardware serial port you wish to use
#define HWSERIAL Serial1
#define HWSERIAL1 Serial2
#define HWSERIAL2 Serial3

unsigned long baud[3] = {19200, 19200, 19200};
unsigned char prev_dtr[3] = {0, 0, 0};

const int reset_pin[3] = {4, 3, 4};
const int led_pin = 13;  // 13 = Teensy 3.X & LC
// 11 = Teensy 2.0
//  6 = Teensy++ 2.0
void setup()
{
  pinMode(led_pin, OUTPUT);
  digitalWrite(led_pin, LOW);
  digitalWrite(reset_pin[0], HIGH);
  pinMode(reset_pin[0], OUTPUT);
  Serial.begin(baud[0]);	// USB, communication to PC or Mac
  HWSERIAL.begin(baud[0]);	// communication to hardware serial
#if defined(USB_DUAL_SERIAL) || defined(USB_TRIPLE_SERIAL)
  digitalWrite(reset_pin[1], HIGH);
  pinMode(reset_pin[1], OUTPUT);
  SerialUSB1.begin(baud[1]);
  HWSERIAL1.begin(baud[1]);
#endif
#if defined(USB_TRIPLE_SERIAL)
  digitalWrite(reset_pin[2], HIGH);
  pinMode(reset_pin[2], OUTPUT);
  SerialUSB2.begin(baud[2]);
  HWSERIAL2.begin(baud[2]);
#endif

}


long led_on_time = 0;
#define BUFFER_SIZE 512
byte buffer[BUFFER_SIZE];

void loop()
{

  // first do the simple stuff.
  processUSBToSerial(0, &Serial, &HWSERIAL, Serial.dtr(), Serial.baud());
#if defined(USB_DUAL_SERIAL) || defined(USB_TRIPLE_SERIAL)
  processUSBToSerial(1, &SerialUSB1, &HWSERIAL1, SerialUSB1.dtr(), SerialUSB1.baud());
#endif
#if defined(USB_TRIPLE_SERIAL)
  processUSBToSerial(2, &SerialUSB2, &HWSERIAL2, SerialUSB2.dtr(), SerialUSB2.baud());
#endif

  // if the LED has been left on without more activity, turn it off
  if (millis() - led_on_time > 3) {
    digitalWrite(led_pin, LOW);
  }
}

//=========================================================================
// Process for one USB to Serial connection.
// Wish USB Serial classes derived from common class other than Stream.
//=========================================================================
void processUSBToSerial(uint8_t serial_index, Stream* usbStream, HardwareSerial *hwserial,
                        unsigned char dtr, unsigned long usb_baud)
{
  int rd, wr, n;

  // check if any data has arrived on the USB virtual serial port
  rd = usbStream->available();
  if (rd > 0) {
    // check if the hardware serial port is ready to transmit
    wr = hwserial->availableForWrite();
    if (wr > 0) {
      // compute how much data to move, the smallest
      // of rd, wr and the buffer size
      if (rd > wr) rd = wr;
      if (rd > 80) rd = 80;
      // read data from the USB port
      n = usbStream->readBytes((char *)buffer, rd);
      // write it to the hardware serial port
      hwserial->write(buffer, n);
      // turn on the LED to indicate activity
      digitalWrite(led_pin, HIGH);
      led_on_time = millis();
    }
  }

  // check if any data has arrived on the hardware serial port
  rd = hwserial->available();
  if (rd > 0) {
    // check if the USB virtual serial port is ready to transmit
    wr = usbStream->availableForWrite();
    if (wr > 0) {
      // compute how much data to move, the smallest
      // of rd, wr and the buffer size
      if (rd > wr) rd = wr;
      if (rd > 80) rd = 80;
      // read data from the hardware serial port
      n = hwserial->readBytes((char *)buffer, rd);
      // write it to the USB port
      usbStream->write(buffer, n);
      // turn on the LED to indicate activity
      digitalWrite(led_pin, HIGH);
      led_on_time = millis();
    }
  }

  // check if the USB virtual serial port has raised DTR
  if (dtr && !prev_dtr[serial_index]) {
    digitalWrite(reset_pin[serial_index], LOW);
    delayMicroseconds(250);
    digitalWrite(reset_pin[serial_index], HIGH);
  }
  prev_dtr[serial_index] = dtr;

  // if the LED has been left on without more activity, turn it off
  if (millis() - led_on_time > 3) {
    digitalWrite(led_pin, LOW);
  }

  // check if the USB virtual serial wants a new baud rate
  if (usb_baud != baud[serial_index]) {
    baud[serial_index] = usb_baud;
    if (usb_baud == 57600) {
      // This ugly hack is necessary for talking
      // to the arduino bootloader, which actually
      // communicates at 58824 baud (+2.1% error).
      // Teensyduino will configure the UART for
      // the closest baud rate, which is 57143
      // baud (-0.8% error).  Serial communication
      // can tolerate about 2.5% error, so the
      // combined error is too large.  Simply
      // setting the baud rate to the same as
      // arduino's actual baud rate works.
      hwserial->begin(58824);
    } else {
      hwserial->begin(usb_baud);
    }
  }
}
 
If you've got the latest core installed, how's extmem_malloc() look?

Since I don't really play with malloc much I copied and example for malloc and converted for extram_malloc:
Code:
#include "Streaming.h"
#define cout Serial

void setup() {
  while(!Serial)
    ;
  int *ptr;
  ptr = (int*) malloc(5*sizeof(int));

  if(!ptr)
  {
    cout << "Memory Allocation Failed"  << endl;
    exit(1);
  }
  cout << "Initializing values..." << endl << endl;

  for (int i=0; i<5; i++)
  {
    ptr[i] = i*2+1;
  }
  cout << "Initialized values" << endl;

  for (int i=0; i<5; i++)
  {
    /* ptr[i] and *(ptr+i) can be used interchangeably */
    cout << *(ptr+i) << endl;
  }

  extmem_free(ptr);

}

void loop() {
  // put your main code here, to run repeatedly:

}
which gives:
Code:
Initializing values...

Initialized values
1
3
5
7
9
which matches the examples output that i copied. Tried this as well:
Code:
void setup() {
    while(!Serial) ;
  int i,n;
  char * buffer;

  Serial.printf ("How long do you want the string? ");
  i = 100;

  buffer = (char*) extmem_malloc (i+1);
  if (buffer==NULL) exit (1);

  for (n=0; n<i; n++)
    buffer[n]=rand()%26+'a';
  buffer[i]='\0';

  Serial.printf ("Random string: %s\n",buffer);
  extmem_free (buffer);


}
Seems like it works (example from C++ reference.

So now my question is how do you simply get the amount xmem not used?
 
Currently I find out how much PSRAM is installed with:
#ifdef ARDUINO_TEENSY41
extern "C" {
extern uint8_t external_psram_size;
}
#endif
and now to retrieve how much was allocated by EXTMEM variables we can do:
extern unsigned long _extram_start;
extern unsigned long _extram_end;

And to find out how much memory is left after that you can do the same calculation that the new heap code does:

Code:
external_psram_size * 0x100000 - ((uint32_t)&_extram_end - (uint32_t)&_extram_start)

Don't know if there is any way with this pool code to know how much of it was used or not... Like we can do with knowing how large the malloc heap has grown.
 
@KurtE
In smalloc.c did notice these functions:
Code:
/* Use these when you use just default smalloc_curr_pool pool */

vovoid sm_free(void *);

void *sm_realloc(void *, size_t);
void *sm_realloc_move(void *, size_t);
void *sm_calloc(size_t, size_t); /* calls zalloc internally */

int sm_alloc_valid(const void *p); /* verify pointer without intentional crash */

size_t sm_szalloc(const void *); /* get size of allocation */
/*
 * get stats: total used, user used, total free, nr. of allocated blocks.
 * any of pointers maybe set to NULL, but at least one must be non NULL.
 */
int sm_malloc_stats(size_t *, size_t *, size_t *, int *);id *sm_malloc(size_t);
void *sm_zalloc(size_t); /* guarantee zero memory allocation */
just not sure how to use them?
 
Sorry to say, but something is still strange with Serial.

It works stably with the IDE now. Unfortuantely, TyCommander somehow gets the wrong COM port now. E.g. the IDE reports COM3 which is the same as the device manager reports. If I connect to the teensy with TyCommander it reports (not always) COM4 and of course can't talk to it on that port.

Winding back to the commit right before the "experimental interface commit" everything works ok and as stably as it always did. Might of course be something on my side. Maybe someone finds time to check with their installation?

Edit In my setup the old (correct) version always generates COM4 (which works) the new one always COM3 (which only works with the IDE) if that helps.
 
Sorry to say, but something is still strange with Serial.

It works stably with the IDE now. Unfortuantely, TyCommander somehow gets the wrong COM port now. E.g. the IDE reports COM3 which is the same as the device manager reports. If I connect to the teensy with TyCommander it reports (not always) COM4 and of course can't talk to it on that port.

Winding back to the commit right before the "experimental interface commit" everything works ok and as stably as it always did. Might of course be something on my side. Maybe someone finds time to check with their installation?

Just checked for the T4.1 that I am testing with. Both TyCommander and IDE are both showing the same COM port number for the T4.1. Even checked with Triple Serial example.
 
So now my question is how do you simply get the amount xmem not used?

Good question. I didn't see anything in the smalloc code to query the largest allocation expected to succeed. Unless it has some functionality I missed, getting this capability might involve quite a project to dig into smalloc.
 
It works stably with the IDE now. Unfortuantely, TyCommander somehow gets the wrong COM port now. E.g. the IDE reports COM3 which is the same as the device manager reports. If I connect to the teensy with TyCommander it reports (not always) COM4 and of course can't talk to it on that port.

I'm struggling to understand what's wrong here. It kinda sounds like Windows decided to assign COM3 and the port is indeed working as COM3, but TyCommander is still believing it's COM4. Is that right?

Silly question... have you tried a cold reboot since the change?
 
Good question. I didn't see anything in the smalloc code to query the largest allocation expected to succeed. Unless it has some functionality I missed, getting this capability might involve quite a project to dig into smalloc.

There does seem to be some functions in smalloc.c that seem to do it: see post #40
 
The new version generates this device: USB\VID_16C0&PID_0483&MI_00\7&922067D&0&0000
The old version generates this device USB\VID_16C0&PID_0483\8077220

Looks like the standard Serial port is now a interface 0 from a compound device, where as the old version generates the usual stand alone device. Is this intended?
 
Don't know if there is any way with this pool code to know how much of it was used or not... Like we can do with knowing how large the malloc heap has grown.

The code in #11 shows how to get stats of the allocated memory.
 
The new version generates this device: USB\VID_16C0&PID_0483&MI_00\7&922067D&0&0000
The old version generates this device USB\VID_16C0&PID_0483\8077220

Looks like the standard Serial port is now a interface 0 from a compound device, where as the old version generates the usual stand alone device. Is this intended?

Dual Serial works here, probably because it always was a compound device.
 
I'm struggling to understand what's wrong here. It kinda sounds like Windows decided to assign COM3 and the port is indeed working as COM3, but TyCommander is still believing it's COM4. Is that right?
Silly question... have you tried a cold reboot since the change?

Sorry didn't see your question. Yes thats right and yes I did a reboot and even deinstalled all old COM ports. I'm quite sure the issue is related to the observation in #46 and 48
 
Looks like the stats about free space are a little optimistic.

Here's a quick test which queries the free space, then tries to allocate it all at once. Fails. :(

Code:
#include "smalloc.h"

void setup() {
  while (!Serial) ;
  Serial.println("EXTMEM Free Space Test");

}

void loop() {
  extmem_malloc(432);
  Serial.print("Free Space: ");
  size_t total = 0, freespace = 0;
  sm_malloc_stats_pool(&extmem_smalloc_pool, &total, NULL, &freespace, NULL);
  Serial.println(freespace);

  void *p = extmem_malloc(freespace);
  if (p) {
    Serial.println("  able to allocate claimed free space");
    extmem_free(p);
  } else {
    Serial.println("  can't allocate all the free space");
  }

  delay(500);
}
 
Back
Top