Teensy 4.1 Beta Test

Sounds great.

I am wondering, if the FS still lives after you reprogram the Teensy? For example by default does the Teensy loader clear all of the Flash or only the pages it needs to write the new image?

Assuming that the whole flash is not erased, Is the FS setup such that if the image size changes such that the starting address changes, will it survive.
Or another way of asking does SPIFFS start using the flash starting at the start address or does it or can it start its allocations at the end just below the EEPROM area and work it's way down.

Or optionally can we tell it to start at some specific start location like the 4mb mark?
 
Me too!

Again for extra Flash memory, I still wonder if we can or should be able to configure the 8MB flash that comes with T4.1 to maybe reserve something like 4MB to use as something like SPIFS?
Again I can see some interesting usage cases for allowing these devices with reasonably large storage to be able to have simple USB FS, where for example maybe for the Robot I am building, I could create new automated moves on my PC, simply plug in the USB port and copy them down to the T4.1 and then have them available...

FWIW, the Circuit Python on Teensy 4.0 provides a USB disk interface using the T4's flash memory. It provides a 1M filesystem. Paul did recently send a 4.1 beta to the Circuit Python developer.

Ultimately, it would be nice to replace Teensy Transfer with sketches that provide USB filesystems, rather than having to use special host programs that communicate via HID.

In terms of auto-setup, unless we can detect if the flash chips are there and size, it may be better to move this to a constructor/begin in the sketch, since there are some many variants. I.e. a Teensy 4.1 with no psram/flash, one with psram, one with flash, one with both soldered on, and the different sizes of psram/flash. As I suggested earlier, perhaps Paul would have a 4.1 variant with the chips soldered on.

For my usage, I would want to use all 8 megabytes of the external flash as a single filesystem. If I wanted to save data, I would use it as a file.

Being able to carve out a chunk of the existing flash would be nice also (both the unused areas, and possibly the top of the EEPROM area). But there it may be better to have it under program control.

I suspect it will be useful to get the RAW/WAV readers in the audio library to read from the SPIFFS filesystems. But there we need to get the libraries into the standard Teensy release.
 
Sounds great.

I am wondering, if the FS still lives after you reprogram the Teensy? For example by default does the Teensy loader clear all of the Flash or only the pages it needs to write the new image?

Assuming that the whole flash is not erased, Is the FS setup such that if the image size changes such that the starting address changes, will it survive.
Or another way of asking does SPIFFS start using the flash starting at the start address or does it or can it start its allocations at the end just below the EEPROM area and work it's way down.

Or optionally can we tell it to start at some specific start location like the 4mb mark?

For internal flash it looks like the loader does a reset of the internal flash. If I run Franks sketch as is it all works. If I comment out the write and just try and do a read you see the errors:
Code:
SPIFFS filesystem starts at: 0x60010000

4ce13d75, 60010000
mount res: 0
--> Test Start ...
--> Hello World! What a wonderful World :) <--
--> ... Test Middle ... 
--> --> xxxxxx <--
--> ... Test Middle ... 
--> errno -10010
--> xxxxxx <--
--> ... Test Middle ... 
--> [Bad #0==x|78][Bad #1==x|78][Bad #2==x|78][Bad #3==x|78][Bad #4==x|78][Bad #5==x|78] <--
--> ... Test End.
I would just use internal flash for temporary writes and reads using spiffs unless I am missing something in Franks code. Admittedly didn't go into it in depths.
 
FWIW, the Circuit Python on Teensy 4.0 provides a USB disk interface using the T4's flash memory. It provides a 1M filesystem. Paul did recently send a 4.1 beta to the Circuit Python developer.

Ultimately, it would be nice to replace Teensy Transfer with sketches that provide USB filesystems, rather than having to use special host programs that communicate via HID.
That would be good especially if we could extend that to USBHost

In terms of auto-setup, unless we can detect if the flash chips are there and size, it may be better to move this to a constructor/begin in the sketch, since there are some many variants. I.e. a Teensy 4.1 with no psram/flash, one with psram, one with flash, one with both soldered on, and the different sizes of psram/flash. As I suggested earlier, perhaps Paul would have a 4.1 variant with the chips soldered on.

For my usage, I would want to use all 8 megabytes of the external flash as a single filesystem. If I wanted to save data, I would use it as a file.
That's what is done with the extRAM_SPIFFS_t4 library. Right now there are three options you can use:
Code:
//INIT_PSRAM_ONLY
//INIT_FLASH_ONLY
//INIT_PSRM_FLASH

uint8_t config = INIT_FLASH_ONLY;
and then in the begin:
Code:
eRAM.begin(config);

Being able to carve out a chunk of the existing flash would be nice also (both the unused areas, and possibly the top of the EEPROM area). But there it may be better to have it under program control.

I suspect it will be useful to get the RAW/WAV readers in the audio library to read from the SPIFFS filesystems. But there we need to get the libraries into the standard Teensy release.
Not sure how to do this while keeping the data between the loads
 
A filesystem that is accessible through usb is planned since SerialFlash.h was developed. That was the time the Audionshield v1 was developed.
The TeensyTransfer was meant to live a few weeks only. It was never meant to "live" that long, and that's the reason why it has no gui or other nice features.
SO... looks like I have to update it a very last time. Maybe in a few weeks, when I feel like it.
I don't want to support that - I don't like to give support for users who use the commandline/terminal the very first time.
 
For QSPI pad setup - and Main Flash reserve notes to survive 'user data space {x_Python store, or other FS} to survive all but 15s Restore

Wondering if T_4.1 has a few spare 'fixed' words in EEPROM area to reserve and use? Rounding error or unused Padding? When it came to the PROP shield some bytes were commandeered for MPU usage?

The potential use/need for future T_1170 similar INFO even more likely with various storage options or other device settings possible.

If not in EEPROM a few words at FLASH top could be used - but more subject to overwrite and would consume a whole page to protect.

A 'safe static' Layout table could tell Programmer MCU how to treat Main Flash, and power on startup.c code about QSPI PSRAM, Flash:
uint32_t ssLayout[] = {
0xbeef0000, // sig and nibble or bits for each following entry [ FLASH, qspi PSRAM, qspi Flash{type,size} ]
0x0, // start reserved 8MB FLASH area
0x0, // end or length of FLASH reserve - ideally fills to TOP
0x0, // PSRAM present for init if non-zero
0x0, // PSRAM other? It is all runtime fresh and sketch controlled
0x0, // Flash present for init if non-zero - FS/other on Flash could define
0x0 // Flash other 'future'? - FS/other on Flash could define
};

Paul noted his PSRAM survey only finds 8MB RAM - ESP32's can only access 4MB directly to date
Future QSPI Flash support should evolve to include NAND that offers greater size and some speed tradeoffs, For AUTO access setup the Flash type would need to be known.

This would require a SKETCH to run and set DETAILS in ssLayout[] before first use after new FLASH usage or chips installed, or usage change.

ESP uses IDE build settings to define FLASH SPIFFS layout and PSRAM usage.

Paul: does programming MCU have write access to 1062 ( 128KB? ) BOOT FLASH ? Assuming that is runtime readable and otherwise idle?
 
You simply could add an include with this information - best via GCC commandline. Oh, I forgot....
Or an area before or after the EEPROM.
 
Another poster asked about SD logging - :: Teensy-3-6-SD-Card-missing-time-in-milliseconds

That inspired another to post a better buffered write solution with _isr() data collection.

It shows well on T_4.1 and especially with SDFat beta on exFAT SD on T_4.1. … may try more with PSRAM buffers to post here. Just moved code to a T_4.0 on the FRDM 4236 breakout and it works as well as the T_4.1.

For now some results in above post {where the code is} - very short general log times under 1 ms common, worst case was 13 ms for 16 KB writes - doing 1,000 16B samples per sec:
Code:
…
Writing dbuff1 to data file.  tmilli =  29697  took   0.72 mSec
Writing dbuff0 to data file.  tmilli =  30721  took   0.72 mSec
Writing dbuff1 to data file.  tmilli =  31745  took   0.72 mSec
Writing dbuff0 to data file.  tmilli =  32769  took   1.99 mSec
Writing dbuff1 to data file.  tmilli =  33793  took   1.23 mSec
Writing dbuff0 to data file.  tmilli =  34817  took   0.72 mSec
Writing dbuff1 to data file.  tmilli =  35841  took   0.72 mSec
Writing dbuff0 to data file.  tmilli =  36865  took   0.72 mSec
Writing dbuff1 to data file.  tmilli =  37889  took   0.72 mSec
Writing dbuff0 to data file.  tmilli =  38913  took   2.41 mSec
Writing dbuff1 to data file.  tmilli =  39937  took   0.72 mSec
Writing dbuff0 to data file.  tmilli =  40961  took   1.12 mSec
Writing dbuff1 to data file.  tmilli =  41985  took  12.68 mSec
Writing dbuff0 to data file.  tmilli =  43009  took   1.24 mSec
Writing dbuff1 to data file.  tmilli =  44033  took   0.72 mSec
Writing dbuff0 to data file.  tmilli =  45057  took   0.72 mSec
Writing dbuff1 to data file.  tmilli =  46081  took   0.72 mSec
Writing dbuff0 to data file.  tmilli =  47105  took   1.18 mSec
Writing dbuff1 to data file.  tmilli =  48129  took   1.95 mSec
Writing dbuff0 to data file.  tmilli =  49153  took   0.72 mSec
Writing dbuff1 to data file.  tmilli =  50177  took   0.72 mSec
Writing dbuff0 to data file.  tmilli =  51201  took   0.72 mSec
Writing dbuff1 to data file.  tmilli =  52225  took   0.72 mSec
Writing dbuff0 to data file.  tmilli =  53249  took   1.82 mSec
Writing dbuff1 to data file.  tmilli =  54273  took   1.31 mSec
Writing dbuff0 to data file.  tmilli =  55297  took   0.72 mSec
Writing dbuff1 to data file.  tmilli =  56321  took   0.72 mSec
Writing dbuff0 to data file.  tmilli =  57345  took   0.72 mSec
Writing dbuff1 to data file.  tmilli =  58369  took   1.99 mSec
Writing dbuff0 to data file.  tmilli =  59393  took   1.23 mSec
Writing dbuff1 to data file.  tmilli =  60417  took   0.72 mSec
Writing dbuff0 to data file.  tmilli =  61441  took   0.72 mSec
Writing dbuff1 to data file.  tmilli =  62465  took   0.72 mSec
Writing dbuff0 to data file.  tmilli =  63489  took   0.72 mSec
Writing dbuff1 to data file.  tmilli =  64513  took   2.48 mSec
Writing dbuff0 to data file.  tmilli =  65537  took   0.72 mSec
Writing dbuff1 to data file.  tmilli =  66561  took   0.72 mSec
Writing dbuff0 to data file.  tmilli =  67585  took   0.72 mSec
Writing dbuff1 to data file.  tmilli =  68609  took   1.19 mSec
Writing dbuff0 to data file.  tmilli =  69633  took   2.02 mSec
Writing dbuff1 to data file.  tmilli =  70657  took   0.72 mSec
Writing dbuff0 to data file.  tmilli =  71681  took   0.72 mSec
...

Upped sample rate to 10,000 16B samples /sec - some aberrations - now running at 0.72 ms generally
Code:
…
Writing dbuff1 to data file.  tmilli =  8908  took   0.72 mSec
Writing dbuff0 to data file.  tmilli =  9011  took   0.72 mSec
Writing dbuff1 to data file.  tmilli =  9113  took   1.88 mSec
Writing dbuff0 to data file.  tmilli =  9216  took   1.38 mSec
Writing dbuff1 to data file.  tmilli =  9523  took 202.48 mSec
Writing dbuff1 to data file.  tmilli =  9728  took 162.13 mSec
Writing dbuff0 to data file.  tmilli =  9625  took   0.72 mSec
Writing dbuff1 to data file.  tmilli =  9728  took   2.12 mSec
Writing dbuff0 to data file.  tmilli =  9830  took   1.23 mSec
Writing dbuff1 to data file.  tmilli =  9932  took   0.72 mSec
Writing dbuff0 to data file.  tmilli =  10035  took  13.01 mSec
Writing dbuff1 to data file.  tmilli =  10137  took   1.24 mSec
Writing dbuff0 to data file.  tmilli =  10240  took   0.72 mSec
…
Writing dbuff0 to data file.  tmilli =  201113  took   0.72 mSec
Writing dbuff1 to data file.  tmilli =  201216  took   0.72 mSec
Writing dbuff0 to data file.  tmilli =  201318  took  29.69 mSec
Writing dbuff1 to data file.  tmilli =  201420  took   0.72 mSec
Writing dbuff0 to data file.  tmilli =  201523  took   0.72 mSec
Writing dbuff1 to data file.  tmilli =  201625  took   1.80 mSec
Writing dbuff0 to data file.  tmilli =  201728  took   0.72 mSec
...

20K samples/sec with twin 64KB buffers:
Code:
Starting logging
Writing dbuff0 to data file.  tmilli =  0  took   4.10 mSec
Writing dbuff1 to data file.  tmilli =  205  took   7.89 mSec
Writing dbuff0 to data file.  tmilli =  409  took   6.14 mSec
Writing dbuff1 to data file.  tmilli =  614  took   2.88 mSec
Writing dbuff0 to data file.  tmilli =  819  took   4.71 mSec
Writing dbuff1 to data file.  tmilli =  1024  took   4.24 mSec
Writing dbuff0 to data file.  tmilli =  1229  took   4.71 mSec
Writing dbuff1 to data file.  tmilli =  1433  took   3.35 mSec
Writing dbuff0 to data file.  tmilli =  1638  took   3.77 mSec
…
Writing dbuff0 to data file.  tmilli =  86835  took   2.88 mSec
Writing dbuff1 to data file.  tmilli =  87040  took   2.88 mSec
Writing dbuff0 to data file.  tmilli =  87245  took   2.88 mSec
Writing dbuff1 to data file.  tmilli =  87449  took   2.88 mSec
Writing dbuff0 to data file.  tmilli =  87654  took   2.88 mSec
Writing dbuff1 to data file.  tmilli =  88678  took 650.24 mSec
Writing dbuff0 to data file.  tmilli =  88473  took   4.90 mSec
Writing dbuff1 to data file.  tmilli =  88678  took   3.93 mSec
Writing dbuff0 to data file.  tmilli =  88883  took   3.92 mSec
Writing dbuff1 to data file.  tmilli =  89088  took   5.04 mSec
Writing dbuff0 to data file.  tmilli =  89293  took   3.96 mSec
Writing dbuff1 to data file.  tmilli =  89497  took   3.96 mSec
Writing dbuff0 to data file.  tmilli =  89702  took   5.03 mSec
Writing dbuff1 to data file.  tmilli =  89907  took   3.96 mSec
Writing dbuff0 to data file.  tmilli =  90112  took   3.95 mSec
Writing dbuff1 to data file.  tmilli =  90317  took   5.04 mSec
Writing dbuff0 to data file.  tmilli =  90521  took   3.96 mSec
Writing dbuff1 to data file.  tmilli =  90726  took   3.96 mSec
Writing dbuff0 to data file.  tmilli =  90931  took   5.03 mSec
Writing dbuff1 to data file.  tmilli =  91136  took   3.96 mSec
Writing dbuff0 to data file.  tmilli =  91341  took   3.95 mSec
Writing dbuff1 to data file.  tmilli =  91545  took   5.03 mSec
Writing dbuff0 to data file.  tmilli =  91750  took   3.96 mSec
Writing dbuff1 to data file.  tmilli =  91955  took   3.95 mSec
Writing dbuff0 to data file.  tmilli =  92160  took   5.03 mSec
Writing dbuff1 to data file.  tmilli =  92365  took   3.97 mSec
Writing dbuff0 to data file.  tmilli =  92569  took  31.84 mSec
Writing dbuff1 to data file.  tmilli =  92774  took   3.73 mSec
Writing dbuff0 to data file.  tmilli =  93389  took 342.55 mSec
Writing dbuff1 to data file.  tmilli =  93184  took   2.88 mSec
Writing dbuff0 to data file.  tmilli =  93389  took   2.88 mSec
Writing dbuff1 to data file.  tmilli =  93593  took   2.88 mSec

<EDIT> Stopped above logging to transfer to T_4 breakout:
Code:
…
Writing dbuff0 to data file.  tmilli =  691814  took   2.88 mSec
Writing dbuff1 to data file.  tmilli =  692019  took   2.88 mSec
Writing dbuff0 to data file.  tmilli =  692224  took   2.88 mSec
Writing dbuff1 to data file.  tmilli =  692429  took   2.88 mSec
Writing dbuff0 to data file.  tmilli =  692633  took   2.88 mSec
Writing dbuff1 to data file.  tmilli =  692838  took   2.88 mSec
Writing dbuff0 to data file.  tmilli =  693043  took   2.88 mSec
Writing dbuff1 to data file.  tmilli =  693248  took   2.88 mSec
Writing dbuff0 to data file.  tmilli =  693453  took   2.88 mSec
Stopping logging
ADCTimer halted
Data file closed.
216780 KBytes written to file.
 
Last edited:
So the question I have would it be better to have separate libraries for FLASH with SPIFFS and use PSRAM to run something like what put together that mirrors FRAM_MB85_I2C lib? Easy enough to separate I think. Only challenge I had is when I wanted to use both at the same time with the initialization.

I'm planning to bring the PSRAM init stuff into the core library, so it will automatically get initialized at startup, if the chip is present. Had meant to do it a few days ago, but have been insanely busy here getting everything ready to release the product soonish.
 
For internal flash it looks like the loader does a reset of the internal flash.

Yes, confirmed, bootloader version 1.05 wipes the entire flash chip, except the eeprom area & restore program (which can't be erased) near the top.

For version 1.06, I am planning to change how flash erase works, specifically to avoid erasing large unused areas. I'm also considering a scheme to auto-detect if sections of the flash data are identical to the previously programed data and avoid erasing and reprogramming (and eventually a way to align certain data to flash block boundaries). If you try loading huge programs that use most of the 8 megs, you'll see why. The flash chip is slow to erase and slow to program.

flashspeed.png

The 2 highlighted specs are the ones we're using. A page is 256 bytes, so the writing speed is 640K/sec, which means writing all 8 megs takes 12.5 seconds (plus software overhead). Erasing is even worse, approx 430K/sec. I'm really not happy about these speeds, which were an inconvenient but tolerable wait with only 2 megs of flash, but turn into a terrible user experience with 8 megs.

But the reality of this covid-19 situation is I'm not going to be able to do substantial new work on the bootloader until we have our 2 regular employees back at PJRC. So far now, I had to go with 1.05 to get Teensy 4.1 released.
 
I used the current PSRAM init used for SPIFFS. Using the 8 MB for two buffers of 1,036,288 bytes.
One buffer can fill in an _isr() while loop monitors for need to write a filled PSRAM buffer to the SD card.

So this works well to write PSRAM while PSRAM writes to SD card - but the sketch dies when I make the buffer 1 GB, possibly a limit on single WriteBinary() write.

This started with the above noted code by @mborgerson using the SDFat beta on a SD card formatted exFAT, and a Pre-allocated 1GB file.

It is writing 50,000 samples per second (driven by an intervalTimer) of a struct of 4 uint32_t's into one buffer until full then swapping buffers to write the filled one in loop():
Code:
uint8_t config = INIT_PSRAM_ONLY;
struct datrec *dbuff0 = (datrec *)0x70050000;
struct datrec *dbuff1 = (datrec *)0x70090000;

This shows a 1,012 KB buffer fills in about 1.3 seconds, and the SD write of that buffer writes to SD in about 50ms, with the typical aberrations.
Code:
Twin buffers of size 1036288 Bytes
Enter command selected from (r, g, q, p)
Allocation succeeded.
Starting logging
0] 1393.26 mSec between 	Writing dbuff0 to data file.  tmilli =  0  took  78.55 mSec
1] 1489.34 mSec between 	Writing dbuff1 to data file.  tmilli =  1296  took 272.53 mSec
0] 1414.86 mSec between 	Writing dbuff0 to data file.  tmilli =  2591  took 392.04 mSec
1] 970.87 mSec between 	Writing dbuff1 to data file.  tmilli =  3887  took  67.56 mSec
0] 1305.29 mSec between 	Writing dbuff0 to data file.  tmilli =  5182  took  77.51 mSec
1] 1312.43 mSec between 	Writing dbuff1 to data file.  tmilli =  6477  took  94.59 mSec
0] 1251.09 mSec between 	Writing dbuff0 to data file.  tmilli =  7773  took  50.32 mSec
1] 1291.42 mSec between 	Writing dbuff1 to data file.  tmilli =  9068  took  46.40 mSec
0] 1299.12 mSec between 	Writing dbuff0 to data file.  tmilli =  10363  took  50.17 mSec
1] 1313.19 mSec between 	Writing dbuff1 to data file.  tmilli =  11659  took  68.02 mSec
0] 1307.98 mSec between 	Writing dbuff0 to data file.  tmilli =  12954  took  80.64 mSec
1] 1280.59 mSec between 	Writing dbuff1 to data file.  tmilli =  14249  took  65.88 mSec
0] 1307.13 mSec between 	Writing dbuff0 to data file.  tmilli =  15545  took  77.66 mSec
1] 1289.79 mSec between 	Writing dbuff1 to data file.  tmilli =  16840  took  72.10 mSec
0] 1272.90 mSec between 	Writing dbuff0 to data file.  tmilli =  18136  took  49.65 mSec
1] 1292.09 mSec between 	Writing dbuff1 to data file.  tmilli =  19431  took  46.40 mSec
0] 1298.61 mSec between 	Writing dbuff0 to data file.  tmilli =  20726  took  49.66 mSec
1] 1295.61 mSec between 	Writing dbuff1 to data file.  tmilli =  22022  took  49.92 mSec
0] 1295.13 mSec between 	Writing dbuff0 to data file.  tmilli =  23317  took  49.70 mSec
1] 1292.05 mSec between 	Writing dbuff1 to data file.  tmilli =  24612  took  46.40 mSec
0] 1298.65 mSec between 	Writing dbuff0 to data file.  tmilli =  25908  took  49.70 mSec
1] 1295.53 mSec between 	Writing dbuff1 to data file.  tmilli =  27203  took  49.88 mSec
0] 1291.74 mSec between 	Writing dbuff0 to data file.  tmilli =  28498  took  46.27 mSec
1] 1317.28 mSec between 	Writing dbuff1 to data file.  tmilli =  29794  took  68.20 mSec

Similar expected - same write time - twice as often - results doing 100,000 samples of 4 values per second:
Code:
Twin buffers of size 1048320 Bytes
0] 772.59 mSec between 	Writing dbuff0 to data file.  tmilli =  0  took  81.50 mSec
1] 642.32 mSec between 	Writing dbuff1 to data file.  tmilli =  655  took  68.64 mSec
0] 663.48 mSec between 	Writing dbuff0 to data file.  tmilli =  1310  took  76.92 mSec
1] 625.36 mSec between 	Writing dbuff1 to data file.  tmilli =  1966  took  47.10 mSec

Here is the code:
Code:
/**************************************************************
  Simple SD card timing datalogger for Teensy 3.6

  https://forum.pjrc.com/threads/60754-Teensy-3-6-SD-Card-missing-time-in-milliseconds?p=238352&viewfull=1#post238352

  This example shows how to log timing data to an SD card using the SD library
  and SDFat 2.0b
  As much as possible, the code uses the simplest Arduino functions, adding
  teensy-specific code only where necessary (such as IntervalTimer).

  The circuit:

   SD card attached to built-in SD Card slot on t3.6


  This example uses  buffering and interval timer to collect at regular
  intervals without collection timing jitter caused by SDC writes.
  A time stamp is saved with each record.

  A minimal user interface is added with three one-letter commands:
      'r'   Open file and start recording data with verbose output
      'g'   Open file and start recording data with only block write delay output
      'q'   Quit logging data and close file
      'p'   Play back data in CSV format


  NOTES:
    1. Unlike the Simple Logger, this version saves only the binary data.
       Conversion to strings is done on playback.
    2. Playback is done one record at a time.  This is less efficient,
       but output is going to the Serial port, so speed isn't as much
       of an issue.
    3. if you uncomment #define USE_EXFAT the program will use the EXFat
       file system of SDFat 2.0beta.  You will need an SD Card formatted
       for the EXFat file system.  That can be done with the tools
       in the SDFat examples




  Written for Teensy 3.6 by M. Borgerson  May 7, 2020
********************************************************************/

// SPI library not needed with Teensy built-in SD card

#define USE_EXFAT

#ifdef USE_EXFAT

#include "SdFat.h"
#include "sdios.h"
#include "ExFatlib\ExFatLib.h"
SdExFat sd;
SdioCard sdc;
ExFile dataFile;
#define SD_CONFIG SdioConfig(FIFO_SDIO)

#else

#include <SD.h>
File dataFile;

#endif

IntervalTimer ADCTimer;


#define SAMPLERATE 100000

// A simple structure to hold time stamp and three analog values
// The structure takes up 16 bytes to align millitime on 4-byte boundary
struct datrec {
	uint32_t millitime;
	uint32_t microtime;
	uint32_t DWTCount;
	uint32_t byteswritten;
};
// Allocate two buffers, each holding over 1 second of data.
// For efficiency, the total length should be a multiple of 512.
// Note that the T3.6 will run out of RAM at about 8000Hz sampling
// as each buffer would have to be 8000 * 16 or 128,000 bytes, leaving
// very little room for the stack, heap, and USB buffers.
//#define BUFFSIZE  1024

// allocate two buffers, each of which holds 1024 records
// each buffer is 1024 * 16 or 16KBytes

#define USE_PSRAM 1
#define BUFFSIZE  8190*8
#ifdef USE_PSRAM // 128KB apart
const char *logfilename = "TIMELG_R1.DAT";
#include <extRAM_t4.h>
extRAM_t4 eRAM;
//uint8_t config = 2; //0 - init eram only, 1-init flash only, 2-init both
//These have been replaced with defines for:
//INIT_PSRAM_ONLY
//INIT_FLASH_ONLY
//INIT_PSRM_FLASH
uint8_t config = INIT_PSRAM_ONLY;
struct datrec *dbuff0 = (datrec *)0x70050000;
struct datrec *dbuff1 = (datrec *)0x70090000;
#else
#define BUFFSIZE  1024
const char *logfilename = "TIMELOG1.DAT";
struct datrec dbuff0[BUFFSIZE];
struct datrec dbuff1[BUFFSIZE];
#endif


#define DEBUGPRINT true


// when graphout is true, during recording, only the block write
// interval is displayed.  This allows graphing with the Arduino
// Serial Plotter display.
bool graphout = false;

// these variables are declared volatile because they are changed or used in the
// interrupt handler
volatile uint16_t saveidx = 0;
volatile int16_t writebuffnum = -1;
volatile uint16_t savebuffnum = 0;
volatile uint32_t filestartmilli = 0;
volatile uint32_t byteswritten = 0;

#ifndef USE_EXFAT
const int chipSelect = BUILTIN_SDCARD;
#endif
const int ledpin = 13;

#define LEDON  digitalWriteFast(ledpin, HIGH);
#define LEDOFF digitalWriteFast(ledpin, LOW);

const char compileTime [] = "Timing Data Logger  Compiled on " __DATE__ " " __TIME__;
void setup() {
	if ( ARM_DWT_CYCCNT == ARM_DWT_CYCCNT ) {	  // activate ARM cycle counter
		ARM_DEMCR |= ARM_DEMCR_TRCENA; // Assure Cycle Counter active
		ARM_DWT_CTRL |= ARM_DWT_CTRL_CYCCNTENA;
	}

	pinMode(ledpin, OUTPUT);
	// Open serial communications and wait for port to open:
	Serial.begin(9600);
	while (!Serial && millis() < 4000 );
	Serial.println("\n" __FILE__ " " __DATE__ " " __TIME__);
	Serial.println(compileTime);
	// see if the card is present and can be initialized:
	Serial.print("Initializing SD card...");
#ifdef USE_EXFAT
	StartEXFat();
#else
	StartSD();
#endif

#ifdef USE_PSRAM // 128KB apart
	eRAM.begin(config);
	Serial.print("Start PSRAM for LOGGING :: ");
	Serial.println( logfilename );
#else
	Serial.print("LOGGING from RAM :: ");
	Serial.println( logfilename );
#endif

	Serial.printf("\nTwin buffers of size %u Bytes\n" , sizeof(datrec)*BUFFSIZE );
	Serial.println("Enter command selected from (r, g, q, p)");
}


#ifdef USE_EXFAT
void StartEXFat(void) {
	if (!sd.cardBegin(SD_CONFIG)) {
		Serial.print("cardBegin failed");
	}
	if (!sd.volumeBegin()) {
		Serial.print("  volumeBegin failed");
	}
	if (!sd.begin(SdioConfig(FIFO_SDIO))) {
		Serial.println("\nSD File  system initialization failed.\n");
		FastBlink();
	} else  Serial.print("EXFat initialization done.");

	if (sd.fatType() == FAT_TYPE_EXFAT) {
		Serial.println("  Card Type is exFAT");
	} else {
		FastBlink();
	}
	Serial.println("card initialized.");
}

#else
void StartSD(void) {
	if (!SD.begin(chipSelect)) {
		Serial.println("Card failed, not FAT32, or not present");
		// don't do anything more except fast blink LED
		FastBlink();
	}
	Serial.println("card initialized.");
}
#endif

void FastBlink(void) {  // blink forever
	while (1) {
		LEDON
		delay(50);
		LEDOFF
		delay(50);
	}
}

static uint32_t elapsedUS = 0;
void loop() {
	// put your main code here, to run repeatedly:
	char ch;
	uint32_t microselapsed;
	elapsedMicros wrtelapsed = 0;

	if (Serial.available()) {
		ch = Serial.read();
		if (ch == 'r') {
			graphout = false;
			StartLogging();
		}
		if (ch == 'g') {
			graphout = true;
			StartLogging();
		}
		if (ch == 'q') StopLogging();
		if (ch == 'p') PlaybackLog();
	}
	// Now check to see if a buffer is ready to be written to SD
	// writebuffnum will be set in the interrupt handler
	if (writebuffnum == 0) { //is dbuff0 ready?
		LEDON
		writebuffnum = -1;
		if (dataFile) { //if the file is open write dbuff0
			wrtelapsed = 0;
			WriteBinary(&dbuff0[0], BUFFSIZE);
			microselapsed = wrtelapsed;
			if (DEBUGPRINT) {
				if (graphout) {
					Serial.println((float)microselapsed / 1000.0);
				} else {
					Serial.printf("0] %6.2f mSec between \t", (float)(micros() - elapsedUS) / 1000.0);
					Serial.print("Writing dbuff0 to data file.  tmilli = ");
					Serial.printf(" %lu  took %6.2f mSec\n", dbuff0[0].millitime, (float)microselapsed / 1000.0);
					elapsedUS = micros();
				}
			}
		}
		LEDOFF
	}
	if (writebuffnum == 1) { // is dbuff1 ready?
		LEDON
		writebuffnum = -1;
		if (dataFile) { //if the file is open write dbuff0
			wrtelapsed = 0;
			WriteBinary(&dbuff1[0], BUFFSIZE);
			microselapsed = wrtelapsed;
			if (DEBUGPRINT) {
				if (graphout) {
					Serial.println((float)microselapsed / 1000.0);
				} else {
					Serial.printf("1] %6.2f mSec between \t", (float)(micros() - elapsedUS) / 1000.0);
					Serial.print("Writing dbuff1 to data file.  tmilli = ");
					Serial.printf(" %lu  took %6.2f mSec\n", dbuff1[0].millitime, (float)(microselapsed) / 1000.0);
					elapsedUS = micros();
				}
			}
		}
		LEDOFF
	}
	//delay(5);
}  // end of the loop() function

// write a buffer to the file as binary record. This  takes more time and file space than
// binary format, but can be directly read on PC.
// Note that we have to cast datrec pointer to a uint8_t pointer
// to keep the SD libraryhappy
void WriteBinary(volatile struct datrec *drp, size_t numstructs) {
	dataFile.write((uint8_t *)drp, numstructs * sizeof(datrec));

#ifdef USE_EXFAT
// sync() updates directory and puts SD card in low-power idle mode
// it increases the block write time to 6-10mSec, since the file system
// must wait for the long block write to finish before it can update the
// directory and put the SD card into idle mode
// Without the sync, the block write returns in 0.88mSec on T3.6 at 168MHz
// because all the multi-block write stuff happens in interrupt routines in
// the SDFat driver.
// Sync() will reduce power consumption by about 20 to 40mA in the ~1 second
// between block writes. That can be significant for low-power loggers.

//    dataFile.sync(); // uncomment for slower writes, but reduced SD power
#endif
}

// This is the interrupt handler for the interval timer.
// It saves the data in the data buffer structures.  The output is converted
// to CSV text when it is saved to the output file in the main loop.
// We don't want to do the conversion to a string in the interrupt handler
// since it takes a lot more time and many of the string routines are not
// reentrant. (Programmer talk for "They don't play nicely when interrupted")
void ADCChore(void) {
	uint32_t tmilli;
	tmilli = millis() - filestartmilli;
	byteswritten += sizeof(datrec); // update global bytes written count
	// save in  the proper buffer--defined by savebuffnum
	if (savebuffnum == 0) { // put data in dbuff0
		dbuff0[saveidx].millitime = tmilli;
		dbuff0[saveidx].microtime =  micros();
		dbuff0[saveidx].DWTCount = ARM_DWT_CYCCNT;
		//dbuff0[saveidx].byteswritten = byteswritten;
		dbuff0[saveidx].byteswritten = 0;
		saveidx++;
		if (saveidx >= BUFFSIZE) { // mark buffer for write to  SD
			writebuffnum = 0;
			savebuffnum = 1;   // start saving in other buffer on next interrupt
			saveidx = 0;  // start at beginning of next buffer
		}
	} else {  // must be saving to dbuff1
		dbuff1[saveidx].millitime = tmilli;
		dbuff1[saveidx].microtime =  micros();
		dbuff1[saveidx].DWTCount = ARM_DWT_CYCCNT;
		//dbuff1[saveidx].byteswritten = byteswritten;
		dbuff1[saveidx].byteswritten = 1;
		saveidx++;
		if (saveidx >= BUFFSIZE) { // mark buffer for write to  SD
			writebuffnum = 1;
			savebuffnum = 0;   // start saving in other buffer on next interrupt
			saveidx = 0;   // start at beginning of next buffer
		}
	}
}

void StartLogging(void) {
	if (dataFile) {
		Serial.println("Already collecting!");
		return;
	}
	// we open in a mode that creates a new file each time
	// instead of appending as in the Arduino example
	elapsedUS = micros();
#ifdef USE_EXFAT

	if (!dataFile.open(logfilename, O_WRITE | O_CREAT | O_TRUNC)) {
		Serial1.println("Open File failed");
		return;
	}
	// pre-allocate 1GB
	uint64_t alloclength = (uint64_t)(1024l * 1024l * 1024l);
	Serial1.print("Pre-allocating 1GB file space ");
	if (!dataFile.preAllocate(alloclength)) {
		Serial.println("Allocation failed. Proceeding anyway.");
	} else {
		Serial.println("Allocation succeeded.");
	}

#else

	if (SD.exists(logfilename)) SD.remove(logfilename);
	dataFile = SD.open(logfilename, FILE_WRITE);

#endif
	// if the file can't be opened, say so
	if (!dataFile) {
		Serial.println("Could not open output file!");
		return;
	}

	Serial.println("Starting logging");
	// initialize some variables for the buffers
	saveidx = 0;   // start saving at beginning of buffer
	savebuffnum = 0;  // start saving in dbuff0
	writebuffnum = -1;  // indicates no buffer ready yet
	//  start the interval timer to begin logging
	filestartmilli = millis();
	byteswritten = 0;
	ADCTimer.begin(ADCChore, 1000000 / SAMPLERATE); //begin() expects timer period in microseconds
}

void StopLogging(void) {
	Serial.println("Stopping logging");
	ADCTimer.end();
	if (DEBUGPRINT) Serial.println("ADCTimer halted");
	delay(10);
	if (dataFile) {
		Serial.println("Data file closed.");
#ifdef USE_EXFAT
		dataFile.truncate();  // Truncate down to data actually written
#endif
		dataFile.close();  // if file was open, close it.
	} else {
		if (DEBUGPRINT) Serial.println("dataFile was not open!");
	}
	writebuffnum = -1;
	// dont' write tottal bytes if graphing--it messes up the plot!
	if (!graphout) {
		Serial.printf("%lu KBytes written to file.\n", byteswritten / 1024);
	}
	// in the interest of simplicity, we ignore any partial buffer at the end
}

void PlaybackLog(void) {
	struct datrec temprec;
	if (dataFile) StopLogging(); // if still recording, stop and close file
#ifdef USE_EXFAT
	if (!dataFile.open(logfilename, O_READ)) {
		Serial1.println("Open File failed");
		return;
	}
#else
	dataFile = SD.open(logfilename, FILE_READ);
#endif
	if (!dataFile) {
		Serial.println("Could not open data file for reading.");
		return;
	}
	while (dataFile.available()) {
		dataFile.read(&temprec, sizeof(datrec));
		Serial.printf("%lu, %lu, %lu, %lu\n", temprec.millitime, temprec.microtime, temprec.DWTCount, temprec.byteswritten);
	}
	dataFile.close();
	Serial.println("\nPlayback complete.");
}
 
Looking more at p#287 code - the 'p' print code just dumps the file to USB so hard to observe. Added a 's' SPARSE print that watches millis in data change ( and what buffer data is from) and skips prints between and that cuts down the noise and changing skip counts can be annotated. Looking at those blocks there are segments out of place - about 512B worth or 32 entries - one disk sector.

Quick return to RAM writes don't show those. Not sure if it is PSRAM use - or size of buffer error in SDFat … RAM uses 1024 and PSRAM using 8190*8 with bad segments, quick test with only 8190 struct buffer doesn't show the error - more to follow.

Disk data is binary so no fun to read on PC - especially with up to or more than GB files.
 
I'm planning to bring the PSRAM init stuff into the core library, so it will automatically get initialized at startup, if the chip is present.

Please add a way for the application to get information about the soldered chips, then. Most important the size. Or the ID, if "size" needs too much code. Both would be great.
Its best to grab this information during initialization.
 
I'm planning to bring the PSRAM init stuff into the core library, so it will automatically get initialized at startup, if the chip is present. Had meant to do it a few days ago, but have been insanely busy here getting everything ready to release the product soonish.

Thanks Paul. Know you got a lot on your plate right now with getting ready for the release of the T4.1. Looking forward to it as well. In the meantime got a couple of other projects on the list to keep me busy :)
 
Yes, confirmed, bootloader version 1.05 wipes the entire flash chip, except the eeprom area & restore program (which can't be erased) near the top.

For version 1.06, I am planning to change how flash erase works, specifically to avoid erasing large unused areas. I'm also considering a scheme to auto-detect if sections of the flash data are identical to the previously programed data and avoid erasing and reprogramming (and eventually a way to align certain data to flash block boundaries). If you try loading huge programs that use most of the 8 megs, you'll see why. The flash chip is slow to erase and slow to program.
..
The 2 highlighted specs are the ones we're using. A page is 256 bytes, so the writing speed is 640K/sec, which means writing all 8 megs takes 12.5 seconds (plus software overhead). Erasing is even worse, approx 430K/sec. I'm really not happy about these speeds, which were an inconvenient but tolerable wait with only 2 megs of flash, but turn into a terrible user experience with 8 megs.

But the reality of this covid-19 situation is I'm not going to be able to do substantial new work on the bootloader until we have our 2 regular employees back at PJRC. So far now, I had to go with 1.05 to get Teensy 4.1 released.
Don't think I every appreciated how long it takes to erase the flash until now.

Agree, until the covid-19 situation is sorted out 1.05 it its.

Again thanks for the explanation
 
Looking more at p#287 code - the 'p' print code just dumps the file to USB so hard to observe. Added a 's' SPARSE print that watches millis in data change ( and what buffer data is from) and skips prints between and that cuts down the noise and changing skip counts can be annotated. Looking at those blocks there are segments out of place - about 512B worth or 32 entries - one disk sector.

Quick return to RAM writes don't show those. Not sure if it is PSRAM use - or size of buffer error in SDFat … RAM uses 1024 and PSRAM using 8190*8 with bad segments, quick test with only 8190 struct buffer doesn't show the error - more to follow.

Disk data is binary so no fun to read on PC - especially with up to or more than GB files.

@defragster - any chance you can update your code in post #287 with your Sparse option.
 
@Paul - I do very much like the idea of automatically initializing the memory stuff as mentioned, but again it will be curious on how we set it up such that suppose I have a T4.1 and I
use something like the castellated solder on extension.

Suppose I use pins 52 and 53 as Alternate pins for Serial1 and I hook up a display using SPI2 pins (49, 50, 54) with maybe 48 and 51 as CS/DC... Yes neither are hardware CS, but...
Selection in tools menu?
 
Selection in tools menu?

I would be with you if Arduino would save that.
But it does not.
Considering this, it is perhaps better to use the solder pads only for the intended purpose.
I don't like that much, but under these circumstances it might be the best.
 
I would be with you if Arduino would save that.
But it does not.
Considering this, it is perhaps better to use the solder pads only for the intended purpose.
I don't like that much, but under these circumstances it might be the best.
I hear you, and would agree if they were actually shipped with chips installed.

And we don't currently initialize other hardware, like SD cards or ... So just sort of wondering. The options I currently see for this include:
a) Automatic - More foolproof (maybe) - For RAM I understand for Flash, do you assume that the it will be used as a drive or do you expect that some other user init code will be called?

b) Keep it sort of like today, where maybe the user has to call some method to initialize external memory. We could keep it as external library or bring into core. Nice thing about this is you are flexible and your sketch only takes a code size init time hit if your program actually says it wants to use it.

c) Fault handling? - If the user sketch tries to use the external ram and has not called the init function, could we detect this in fault handler and do it for them...
 
... automatically initializing the memory stuff as mentioned, but again it will be curious on how we set it up such that suppose I have a T4.1 and I use something like the castellated solder on extension.

Yeah, automatic detection and initialization of the PSRAM is going to come with the downside that those 6 pins will get a flurry of activity at startup as the memory detection takes place. That pin activity is going to make those already hard-to-use-as-I/O pins even harder to use. But I feel it's a pretty good trade off, that those pads really are primarily meant for adding memory chips, especially the PSRAM.
 
@PaulStoffregen
On a related note have you given any thought to that special malloc for the PSRAM? Just curious.
 
Sadly, since packaging up 1.52-beta4, nearly every available hour this week has been prepping to release the product (soon-ish). Haven't had time to do any software stuff since beta4.

Planning to add the PSRAM auto-init and fix audio ADC on Teensy 3 (dunno how that slipped in... I tested it many times on both boards) on Saturday. Might get to the malloc thing, might not. I still have a long list of urgent things to complete before we can release the product.
 
@defragster - any chance you can update your code in post #287 with your Sparse option.

Opps - should have done that but got late and was going to add another check to make screen parsing faster - will do it soon - also since the SD write dies at 1MB - am suspicious the single huge write is the trouble - was pondering switching to more smaller buffers as most write fast - and that would allow the logging to be active longer.

Also added in an embarrassingly poor attempt at loop count.

Good Sparse notes look like:
Code:
skipped 19	6552, 9408030, 1349572094, 1	
skipped 7	6552, 9408430, 1349812090, 0	 	 ????
skipped 11	6553, 9409030, 1350172084, 0	 	 ????
skipped 19	6554, 9410030, 1350772084, 0	 	 ????
skipped 19	6555, 9411030, 1351372084, 0
for settings:
Code:
#define SAMPLERATE 20000
#define USE_PSRAM 1
#ifdef USE_PSRAM // 128KB apart
#define BUFFSIZE  8190

Current BAD sparse shows:
Code:
skipped 19	13920, 19040040, 2833835851, 0	
skipped 19	13921, 19041040, 2834435851, 0	
skipped 19	13922, 19042040, 2835035851, 0	
skipped 19	13923, 19043040, 2835635851, 0	
[B]skipped 9	[COLOR="#FF0000"]16380[/COLOR], 21500340, 15048562, 1	 	 ????
skipped 13	16381, 21501040, 15468549, 1	 	 ????
skipped 19	16382, 21502040, 16068549, 1	 	 ????
skipped 19	16383, 21503040, 16668549, 1	
skipped 19	16384, 21504040, 17268549, 1	
skipped 19	16385, 21505040, 17868549, 1	
skipped 19	16386, 21506040, 18468549, 1	
skipped 19	16387, 21507040, 19068549, 1	
skipped 19	16388, 21508040, 19668549, 1	
skipped 19	16389, 21509040, 20268549, 1	
skipped 19	16390, 21510040, 20868549, 1	
skipped 19	16391, 21511040, 21468549, 1	
skipped 19	16392, 21512040, 22068549, 1	
skipped 19	16393, 21513040, 22668573, 1	
skipped 19	16394, 21514040, 23268549, 1	
skipped 19	16395, 21515040, 23868549, 1	
skipped 19	16396, 21516040, 24468549, 1	[/B]
skipped 14	[COLOR="#FF0000"]13939[/COLOR], 19059990, 2845805851, 0	 	 ????
skipped 0	13940, 19060040, 2845835851, 0	 	 ????
skipped 19	13941, 19061040, 2846435851, 0	 	 ????
skipped 19	13942, 19062040, 2847035851, 0	
skipped 19	13943, 19063040, 2847635851, 0

when set to:
Code:
#define SAMPLERATE 20000
#define USE_PSRAM 1
#ifdef USE_PSRAM // 128KB apart
[B]#define BUFFSIZE  8190*8[/B]

That shows 17 lines (272 bytes misplaced) out of order from buffer 1 when buffer 0 was being written. That isn't a full 512B sector.

Will be back soon ...
 
Back
Top