Suggestions for recording data

A

ajc225

Active member
Hi, I'm trying to record data and figure out the best method to try to retain the most information possible. Currently, I have 2 teensy4.1 sending 4x16uint every 10kHz or a complete package of data (64bits) every 2kHz. I will be running the teensy for 1 hr but I could potentially split the hr up into many 1-5min chunks. Unfortunately, the current setup doesn't have a way to synchronize timestamps so they are exclude in the packages, but I was thinking of having a 3rd teensy "probe" the communication lines, add a timestamp (probably a 32bit unsigned long) to the package whenever the 3rd teensy receives data, and down sample the rate at which the package of data is being sent at about 100 Hz. I'm not sure if there is a method to record the data at the speed that I'm transferring information and space to store all the information, but I think down sampling will still give me a sufficient amount of data (If anyone does know of a way please let me know). All the information transfer is done using DMA.

So I will be storing a total of 2*(64+32)bits/pkg* 100Hz*3600s*1.25x10^-10Gb/bit = 8.64x10^-3 Gb of data when down sampled or 2*(64+32)bits/pkg* 2kHz*3600s*1.25x10^-10Gb/bit =0.1728 Gb of data when not down sampled correct me if I'm wrong. However, there are a couple of ways I found to record the data.

1. Save the data in a 32GB SDcard. However, I have to convert the data to a string increasing the needed storage size. If there's a way to send the direct int over the SDcard please let me know. Also, I would have write to the SDcard which I think takes about a few milliseconds and disrupt cpu stuff. Also unsure if I can use DMA to do this directly too.

2. Send the data over Serial and have code in python to read the port and store directly into my computer. Here, I'm unsure about the time it takes to send information through to serial port and the time it takes for python to write data to a file in which it might miss a package.

Sorry for the long post. Does anyone have insight on which one of these methods I should do or if there's another method that have overlooked?
 
Your overall rate of 173MB/hour is well within the capabilities of the T4.1 SD card, which can write ~ 20MBytes/second. There are lots of data logger examples you can use to put together a logger to handle that much data. You don't specify how the Teensies are sending the data. Are they using the hardware UART channels? If so, what baud rate are you using?

It's also not clear what exact rate you are using. I'm sure how you get from 4 x uint16_t at 10 KHz to 64 Bits at 2KHz? The first is 640,000 bits per second, the second is 64 x 2000, or 128,100 bits per second. What am I not understanding?

If you are going to write to the SD card, DO NOT convert the data to ASCII! Read and write binary data files to save SD card space and save a lot of CPU cycles. Binary data files also make buffering simpler and it is much easier to write to SD Card. If you use MTP-Teensy on the USB serial port, it will be simple to move the binary files to your PC, where you can analyze the data. If you really want to scroll through a few million lines of numbers in ASCII format, I hope you are a very patient person!
 
I'd suggest sampling the data interrupt driven and putting in a circular buffer - then the main code pulls data from the buffer to write out - this allows for slow SD card access etc. SD cards in particular can occasionally take a long time to respond, you need a buffer large enough for this.

How you write the data is independent of the buffering, just buffer raw data.
 
MarkT said:
I'd suggest sampling the data interrupt driven and putting in a circular buffer - then the main code pulls data from the buffer to write out - this allows for slow SD card access etc. SD cards in particular can occasionally take a long time to respond, you need a buffer large enough for this.

How you write the data is independent of the buffering, just buffer raw data.
Click to expand...
If, as I suspect, the data is arriving at one of the hardware Serial ports, the interrupts and buffering require only a few lines of code.
for 4 x uint16_t + 1 x uint32_t (Data + time stamp) at 10KHz, you need (12 bytes ) * 10000 or 120,000 bytes of buffer to handle SDC write delays up to one second. You can do something like this:

Code: 
// NOTE:  Example code to illustrate algorithm.   Many missing parts.
#define BUFFSIZE 120000
uint8_t bigbuffer[BUFFSIZE];
#define BAUDRATE  3000000 // 3MBits/second

void setup(){
 
    Serial1.addMemoryForRead(bigbuffer, BUFFSIZE);
    Serial1.begin(BAUDRATE);
    // initalize SD Card and open myfile.
}

#define BLOCKLEN  4096 // write 4KB chunks for efficiency
void loop(){
    uint16_t bytesread;
    uint8_t blockbuffer[BLOCKLEN];
    if(myfile){ // check for properly opened file
        if(Serial1.Available() > 4096){
            bytesread = Serial1.readBytes(blockbuffer, BLOCKLEN);
            myfile.write(blockbuffer, BLOCKLEN);
        }
    }
    //  add code to close file when desired
    //  Check MTP,etc. etc.
}

I'm still bothered by the discrepancy between the OP's data rates as being 10K or 2K readings per second.
If we expect to write 120,000 bytes per second, we need a baud rate at least 10 times that, which is why I specified 3MBits. That's a lot of data for a UART serial link and short wires or good drivers will be needed.
 
Are they using the hardware UART channels? If so, what baud rate are you using?
Click to expand...
Hi, thank you for the input and sorry for the confusion. I am using hardware serial to send the data at a baud rate of 2Mbaud.

I'm not sure how you get from 4 x uint16_t at 10 KHz to 64 Bits at 2KHz?
Click to expand...
For this, each teensy is sending an array of info [startByte, a, b, c, d] where elements a, b, c, d are the important data that I want to record. Each element in the array is sent at 100us (10kHz) after each other. Because there are five elements in the array it takes 500us (or 2kHz) to send each array of information.
If you use MTP-Teensy on the USB serial port, it will be simple to move the binary files to your PC, where you can analyze the data.
Click to expand...
After a bit of consideration, I am leaning towards sending all the data on the USB serial port. In my whole system, I'm using my computer as a central user interface between all the teensys so it makes more sense to keep it on my computer. Plus, if some data transfer thing goes wrong, I can kind of immediately see on my computer and stop the whole operation rather than unknowingly collect incorrect data for 1 hr.

My idea is that the 3rd probing teensy will compile the other 2 teensy's timestamps and important data into a byte type package with 24 bytes (192 bits) and use
Code: 
Serial.write(package, 24);
to send over USB Serial. From there, my python code can read, parse and store the data into a list. Once one of the 5min chunks of recording time is over, I will save that list into a csv file and rinse and repeat until the 1hr is up.

However, I did just realize that the the teensy website says:
Teensy USB Serial object always communicates at native USB speed, either 12 or 480 Mbit/sec
Click to expand...
That being said I don't know if the Serial to USB conversion happens through some FTDI type chip that the teensy might have or if it is done in software somewhere. At a rate of 480 Mbit/sec, I think I could actually record my data at a rate of 2khz rather than down sampling to 100Hz, but I don't know if there will be a chokehold at the point of Serial to USB conversion because I'm a little unclear of how fast that works and not super familiar with details of the conversion.
 
Last edited:
ajc225 said:
Hi, thank you for the input and sorry for the confusion. I am using hardware serial to send the data at a baud rate of 2Mbaud.


For this, each teensy is sending an array of info [startByte, a, b, c, d] where elements a, b, c, d are the important data that I want to record. Each element in the array is sent at 100us (10kHz) after each other. Because there are five elements in the array it takes 500us (or 2kHz) to send each array of information.

After a bit of consideration, I am leaning towards sending all the data on the USB serial port. In my whole system, I'm using my computer as a central user interface between all the teensys so it makes more sense to keep it on my computer. Plus, if some data transfer thing goes wrong, I can kind of immediately see on my computer and stop the whole operation rather than unknowingly collect incorrect data for 1 hr.

My idea is that the 3rd probing teensy will compile the other 2 teensy's timestamps and important data into a byte type package with 24 bytes (192 bits) and use
Code: 
Serial.write(package, 24);
to send over USB Serial. From there, my python code can read, parse and store the data into a list. Once one of the 5min chunks of recording time is over, I will save that list into a csv file and rinse and repeat until the 1hr is up.

However, I did just realize that the the teensy website says:

That being said I don't know if the Serial to USB conversion happens through some FTDI type chip that the teensy might have or if it is done in software somewhere. At a rate of 480 Mbit/sec, I think I could actually record my data at a rate of 2khz rather than down sampling to 100Hz, but I don't know if there will be a chokehold at the point of Serial to USB conversion because I'm a little unclear of how fast that works and not super familiar with details of the conversion.
Click to expand...
USB Serial on the T4.1 uses the built-in IMX-RT1062 USB controller to send and receive data. When you write bytes to the Serial device on the T4.1, the driver accepts the bytes, puts them into a circular queue of buffers and transmits the buffers as required. There can be a few microseconds delay if you write just 24 bytes, because the Serial driver would prefer to send more than just a few bytes per USB transaction. If you group several of your 24-byte packets together before transmission, you will get a higher effective transmission rate, but it probably won't make much difference as the T4.1 USB Serial will send the data in a single USBmicro-frame interval, which is 125uSec. There's probably more to it than that, and I am still learning the details of USB programming.

I did some tests long ago and found that the USB Serial can send data at more than 4MBytes/second. Your success at higher data rates will often depend on the capabilities of the receiving software on your PC. I was receiving images, and it didn't make any difference if the PC occasionally responded slowly while receiving and saving it to disk.
 
Thank you for the explanation! This was very helpful.
Your success at higher data rates will often depend on the capabilities of the receiving software on your PC
Click to expand...
Would you be able to expand on this statement a bit? I have USB 3.0 on my computer which, with a quick google search, says it can support rates up to 5Gbps. So is this more dependent on the driver I have installed on my computer or some possible Microsoft software that's hidden somewhere deep in my windows computer?
 
It's dependant upon the hardware making up the Processor/Processor Chipset on your PC and their appropriate drivers.
 
ajc225 said:
Thank you for the explanation! This was very helpful.

Would you be able to expand on this statement a bit? I have USB 3.0 on my computer which, with a quick google search, says it can support rates up to 5Gbps. So is this more dependent on the driver I have installed on my computer or some possible Microsoft software that's hidden somewhere deep in my windows computer?
Click to expand...
The T4.1 USB is limited to the 480MBits/second rate of USB 2.0. Even though your PC supports the higher rates of USB 3.0, it falls back to USB 2.0 rates when connected to a Teensy. The T4.1 hardware just cannot toggle bits on the data lines fast enough for USB 3.0 transfers, so you're limited to about 20MB/second maximum. (Five years ago, that would have seemed like an impossible goal for a low-cost, well-supported single-board computer.)
 
Mostly, it depends on how you call Serial.write(). If you use it to write individual bytes or words, Teensy 4.x tops out at about 4 Mbytes/sec. If you collect data into buffers (in binary) and write in chunks of 32 bytes or more (up to 512 makes sense), you can get 25-30 Mbytes/sec on typical desktop and laptop computers, using the USB Serial. The rate below 32 bytes per chunk varies, but 4 Mbytes/sec is the worst-case.
To avoid data loss, you do need to check that Serial.availableForWrite() returns a value at least as large as the chunk you want to write.

I posted a full practical Arduino sketch and Linux C program here. I only have Linux, so the host/receiver side program is written in C for Linux. It just sets the serial port to raw mode (no serial data conversions or text line buffering), then sends a random nonzero 64-bit (8-byte) seed, and receives the pseudorandom number sequence starting with that seed state, consisting of the 32 high bits of each Xorshift64* -generated consecutive value. It assumes the host is running on a little-endian machine, like Intel/AMD. This sequence is one of the rare ones that pass all tests in the BigCrunch statistical test, so it can be said to be "better" than even Mersenne Twister, and as such, is a suitable work load to measure real world expectations of the Teensy-to-Host transfer rates using USB Serial.

For Python, you'll want to read in chunks, and use pre-prepared struct.Struct beginning with < (indicating little-endian byte order in the data coming from Teensy) to .unpack() the data into values. For the seed value sent to Teensy, use struct.Struct("<Q") (for 64-bit little-endian unsigned integer), and struct.Struct("<I") (32-bit little-endian unsigned integer) for each of the generated values, noting they are only the 32 most significant bits of each 64-bit Xorshift* result. To stop, just close the serial device; Teensy will detect it, and stop generating the sequence, and wait for a new seed. When opening the serial device, drop/flush all buffered but not read data, before sending the seed, in case the OS cached the tail end of the sequence. Similar approach would work for your own application, perhaps with a dedicated end-of-data packet. Note that Teensy could generate timestamps relative to the point when the device was opened (Serial first evaluated as true) and when the packet was received, giving you relatively precise timestamps but with an unknown small constant delta to the actual wall clock.

In Python, integer type has no fixed bit size, and it will grow to sufficient size to hold the value, so it is important to keep only the 64 low bits of each value within the PRNG at the end of each step that may grow the value:
Python: 
state = self.state
state = state ^ (self.state >> 12)
state = (state ^ (state << 25)) & 18446744073709551615
state = state ^ (state >> 27)
self.state = state
highbits = ((state * 2685821657736338717) >> 32) & 4294967295
I would implement it as an iterator class, where the initializer takes the seed (self.state = int(seed) & 18446744073709551615, nonzero), and each __next__() returns the highbits updating self.state as shown above.
 
Last edited:
You must log in or register to reply here.
Back
Top