Transfer to PC

Oleum

Well-known member
Since I want to build a spectrometer, I also want to display the data nicely on my PC.
I don't want to burden the Teensy with such work, to control a display whose resolution and other performance would not be very comfortable anyway.
I find the normal serial baud rates quite antiquated.
My audio interface is also connected to the PC via USB, and can transmit several channels in parallel.
It is also possible to watch movies via WLAN.
So, what method I could use would be really fast, and work with Teensy 4.1/VB.Net?
 
I guess it depends on what data your Teensy-based spectrometer is going to send to the PC for displaying.
A Teensy 4.x has a 480Mbps USB interface so it can spit out data probably faster than your PC can handle.
Can you elaborate more on this data?

Paul
 
I find the normal serial baud rates quite antiquated
Check out this page which states: Unlike a Arduino Uno & Mega which use slow serial baud rates, the Teensy USB Serial object always communicates at native USB speed, either 12 or 480 Mbit/sec.

Paul
 
Sure, that would be great if I could transfer data that quickly, of course. But now it is often the case that you would also have to receive this data accordingly on the PC side. Are there any drivers that I could use from VB.Net. So far I have only worked with the serial interface on the PC. For this mostly virtual interfaces are opened, and also the data transfer was then managed over the parameters of a serial interface. I could still imagine to transfer data via LAN. At least I have done that before. So far not with the Teensy, but with other sources.
 
Long time ago I wrote a few programs using VB.Net And I used the serial port as well.
For my purposes at that time it was fast enough.
Perhaps it's fast enough for you now as well, but it depends on the amount of data you are planning to send to the PC.

On a side note: you mentioned VB.Net a few times. Is VB.Net a requirement? I can imagine there are better/faster tools nowadays to display external data.

Paul
 
Last edited:
It is possible that the speed will be quite sufficient for this project. However, my question is also to be understood in general. "So which would be the fastest method to transfer data?"
VB.Net is simply because I have been working with it for many years, at least at my workplace. At home, of course, it shouldn't matter to me which language I would use. But it's just a matter of getting used to it.
 
The fastest interface on a Teensy 4.1 is the USB interface at 480Mbps. The Ethernet chip on the T4.1 is has a max datarate of 100Mbps.
Question is I guess how fast compiled VB.Net code can handle USB data versus Ethernet data. That I do not know.
Perhaps other forum members can share their light on this.

Paul
 
And do you know how to receive this 480Mps on PC side? Do I have to go through Windows APIs to get it?
 
While USB is theoretically 4.8X faster, in practice they'll probably both work out similarly if you've using Windows and Visual Basic. Even though your PC is theoretically many times faster than 600 MHz Teensy, the reality of the software environment on the PC side will likely have your application on the PC ending up as the speed limiting factor. That is, if just moving data from memory of Teensy to the PC. Of course if you have a ADC with fixed sample rate, then the data source rate may end up the limiting factor. But if you just want to think about the overall speed, software overhead on the PC side matters quite a lot, especially if trying to update a GUI.

Ethernet with TCP/IP has a very nice advantage of well designed networking stack on the PC side. TCP and IP are designed around potentially long network packet latency. That won't be an issue for just a LAN between Teensy and your PC. But unless you go to pretty extraordinary effort on the software side (which might not even be possible with Visual Basic) you communication will almost certainly suffer from software latency on the PC. The networking stack features meant for packet latency will usually be helpful for software latency.

USB Serial, especially as implemented on Windows with Microsoft's default USBSER driver, leaves quite a lot of to be desired. It can perform very well, but only if your software is designed very carefully. Especially if your program deals with small chunks of data, it can become quite inefficient.

With either protocol, the absolute worst thing you can do is structure your communication is a query-response manner, where the PC transmits a command and waits for Teensy to reply before it transmits the next command. That absolutely kills any hope of good performance with either protocol. To get good performance, you need to use a streaming approach, where Teensy keeps transmitting. If you want acknowledgements or other checks, add tags or other ID to the data and design your communication so the PC can send them when possible and Teensy can receive and process them much later. If you code stops and waits for an ACK or other reply, you'll never get high performance.

So the long-winded point is with either 480 Mbit USB or 100 Mbit Ethernet, the overall speed will depend heavily on how your code is designed, and while Ethernet isn't theoretically as fast, the network stack design for high latency in global distance communication might ease some of the software effort needed on the PC side. Maybe.
 
Yes, you need to use Windows APIs.

I have personally never used Visual Basic, so can't give any VB specific help. I've always programmed Windows software in C / C++.

In C, you open the serial port with WIN32 CreateFile. If the port is COM10 or higher, special naming is needed.

Usually after opening, you configure it with GetCommTimeouts / SetCommTimeouts and GetCommState / SetCommState.

A key decision is whether you will use simple synchronous I/O or more complicated asynchronous I/O (Microsoft calls "Overlapped"). If you select Overlapped with CreateFile, then you are reqiured to do all the extra asynchronous stuff for every read and write. If you will access from more than 1 thread, you must use Overlapped, even if you don't want those features. The simple way only works from a single thread. Did I mention how USBSER driver leaves a lot to be desired, especially compared with TCP/IP networking stack?

Receive data with ReadFile, and transmit with WriteFile. If you opened with Overlapped, you'll probably use CreateEvent to make the event objects used to handle delayed completion.


(note to people who find this message in the future... Microsoft seems to "reorganize" their website every couple years, breaking all deep links. If these links no longer work, just use Google with these function names and "WIN32" to find their new locations)
 
And do you know how to receive this 480Mps on PC side? Do I have to go through Windows APIs to get it?

If you're not constrained to Visual Basic, then here is some code, for Windows, that you may use to receive binary data over Serial (port) from the Teensy, or any other Arduino. I used this program for many years to upload binary map & route data from a custom bike computer to PC (over serial).

Code:
/*
Compile:
gcc serialComm.c -Wall -O2 -std=gnu11 -s

Run example:
serialComm -port 7 -baud 115200
*/


#define WINVER 0x0700
#include <stdlib.h>
#include <string.h>
#include <stdio.h>
#include <wchar.h>
#include <inttypes.h>
#include <math.h>
#include <windows.h>



#define INITIALBUFFERLENGTH						(1*1024*1024)		// initial buffer size
#define CMD_THAT_INITIATES_DATA_TRANSFER		"send data"			// ask Teensy to send data. modify this to suit your requirements

static int baudRate = 115200;
static int scanMax = 32;
static int scanMin = 2;

static HANDLE hSerial = INVALID_HANDLE_VALUE;
static char dev_name[MAX_PATH] = "";
static int deviceNumber = -1;
static char *filename = NULL;



// recieved data is recieved here
void Process (char *buffer, size_t len)
{
	printf("Process(): Received %i bytes\n", (int)len);
}

static void serialClose ()
{
    if (hSerial != INVALID_HANDLE_VALUE){
        CloseHandle(hSerial);
        hSerial = INVALID_HANDLE_VALUE;
    }
}
 
static inline void exit_message (const char* error_message, int error)
{
    fprintf(stderr, error_message);
    fprintf(stderr, "\n");
 
    exit(error);
}

static int serialOpen (int port, const int baud)
{
 
 	if (port < 1){
        scanMax = port;
        scanMin = port;
 
    	for (int n = scanMax; n >= scanMin; --n){
        	sprintf(dev_name, "\\\\.\\COM%d", n);
        	hSerial = CreateFile(dev_name, GENERIC_READ|GENERIC_WRITE, 0, NULL, OPEN_EXISTING, 0, NULL);
        	if (hSerial != INVALID_HANDLE_VALUE){
        		port = n;
            	break;
            }
    	}
 	}else{
		sprintf(dev_name, "\\\\.\\COM%d", port);
		hSerial = CreateFile(dev_name, GENERIC_READ|GENERIC_WRITE, 0, NULL, OPEN_EXISTING, 0, NULL);
		if (hSerial == INVALID_HANDLE_VALUE)
			return 0;
	}

	if (port < 1) return 0;
	   
	DCB dcb;
 	GetCommState(hSerial, &dcb);
	dcb.fRtsControl = RTS_CONTROL_ENABLE;
	dcb.fDtrControl = DTR_CONTROL_ENABLE;
	dcb.fOutX = dcb.fInX = FALSE;
	dcb.fOutxCtsFlow = FALSE;
	dcb.fOutxDsrFlow = 0;
	dcb.BaudRate = baud;
	dcb.ByteSize = 8;
	dcb.Parity = NOPARITY;
	dcb.StopBits = ONESTOPBIT;
	SetCommState(hSerial, &dcb);
 
	COMMTIMEOUTS CommTimeouts;
 	GetCommTimeouts(hSerial, &CommTimeouts);
	CommTimeouts.WriteTotalTimeoutConstant = 5;
	CommTimeouts.WriteTotalTimeoutMultiplier = 1;
	CommTimeouts.ReadTotalTimeoutConstant = 100;
	CommTimeouts.ReadTotalTimeoutMultiplier = 0;
	CommTimeouts.ReadIntervalTimeout = -1;
	SetCommTimeouts(hSerial, &CommTimeouts);

	unsigned long comError = 0;
	COMSTAT comstat;
	PurgeComm(hSerial, PURGE_RXCLEAR | PURGE_RXABORT | PURGE_TXCLEAR | PURGE_TXABORT);
 	ClearCommError(hSerial, &comError, &comstat);
 	
 	return 1;
}

static int serialSendString (const char *str)
{
	size_t len = strlen(str);
	size_t bytesRead = 0;
		
	WriteFile(hSerial, str, (DWORD)len, (DWORD*)&bytesRead, NULL);
	return (bytesRead == len);
}

static int serialSendCmd (const char *cmd)
{
	char str[strlen(cmd)+4];
	snprintf(str, sizeof(str), "%s\r\n", cmd);
	
	return serialSendString(str);
}

static char *serialReadResponse (size_t bufferSize, const int allocDelta, size_t *len)
{
	*len = 0;
	size_t bytesRead = 0;
	size_t bytesReadTotal = 0;
	
	char *buffer = malloc(bufferSize);
	if (!buffer) return 0;
		

	do{
		bytesRead = 0;
		if (!ReadFile(hSerial, &buffer[bytesReadTotal], bufferSize-bytesReadTotal, (LPDWORD)&bytesRead, NULL))
			break;
			
		bytesReadTotal += bytesRead;
		//printf("%i %i %i '%c'\n", bytesRead, bytesReadTotal, bufferSize, *buffer);
		if (!bytesRead && bytesReadTotal){
			*len = bytesReadTotal;
			return buffer;
		}
		if (bytesReadTotal >= bufferSize-32){
			bufferSize += allocDelta;
			buffer = realloc(buffer, bufferSize);
			if (!buffer) return 0;
		}
	}while(1);

	free(buffer);
	return NULL;
}

int processFromSerial (const int deviceNumber, const int baudRate)
{
    atexit(serialClose);
 
	if (!serialOpen(deviceNumber, baudRate))
		return 0;

	if (serialSendCmd(CMD_THAT_INITIATES_DATA_TRANSFER)){					// ask for data
		size_t len;
		char *buffer = serialReadResponse(INITIALBUFFERLENGTH, 8192, &len);	// receive that data
		serialClose();
		if (buffer){
			if (len)
				Process(buffer, len);
			else
				printf("read failed\n");
			free(buffer);
		}
	}

	return 0;
}

uint64_t lof (FILE *stream)
{
	fpos_t pos;
	fgetpos(stream, &pos);
	fseek(stream,0, SEEK_END);
	uint64_t fl = ftell(stream);
	fsetpos(stream, &pos);
	return fl;
}

int main (int argc, char *argv[])
{
    int n = 1;
    while (n < argc){
        if (strcmp(argv[n], "-file") == 0){
        	if (++n >= argc)	
            	exit_message("filename not specified", 1);
            filename = (char*)argv[n];
            if (!filename)
            	exit_message("invalid filename", 1);

        }else if (strcmp(argv[n], "-port") == 0){
            if (++n >= argc)	
            	exit_message("no device number specified", 1);
            deviceNumber = atoi(argv[n]);
            
        	if (strcmp(argv[n], "-baud") == 0){
            	if (++n >= argc)
	            	exit_message("baud rate not specified", 1);
            	baudRate = atoi(argv[n]);
            }
		}else{ 
		//	printf("unrecognised option: %s\n", argv[n]);
        }
        n++;
    }
    
    if (filename){
    	//processFromFile(filename);
    }else if (deviceNumber >= 0 && baudRate){
    	processFromSerial(deviceNumber, baudRate);
    }else{
		exit_message("nothing to do\n", 1);
	}
		
	return 0;
}
 
Thank you, to all who have already participated here.
I think I could save a lot of transmission volume anyway, especially when it comes to data from a spectrometer, if I only send what has really changed.
So similar to the compression with films or photos, if one always sends only the address with the value of the date, which has just changed.
In spite of everything, it is an interesting topic how to quickly transport data into a PC.

@MichaelMC
I don't know how I'm going to do it yet. It all depends on how simple the hardware and software I build will be. I am a beginner when it comes to programming with C++.
I know a lot about spectrometers, but not so much about Teensy or Arduino hardware. That's why this will all develop very slowly and I will research in all possible directions what will be the best solution for me. Luckily I already own some really good optical gratings.
 
Your sensor has a typical readout rate of 500kHz. You will probably transmitt 16bit data which gives a required transfer speed of 1Mbyte per second. This is WAY slower than a T4 can tranmitt to a windows PC without any tricks by simply using USB-Serial. Here a thread where I tested this a couple of years ago: https://forum.pjrc.com/threads/58629-Win10-amp-T4-Serial-Communication-Tests Up to some 15MByte from a T4 to a dotNet PC program runs out of the box. Please note: There is a bug in the windows driver which limits the achivable speed but you will notice this only at ultra high transfer speeds.

I find the normal serial baud rates quite antiquated.
Nothing antiquated about using Serial over USB. High level languages support that nicely.

I usually use c#/wpf for windows GUI apps but both, c# and vb.net generate the same IL code which will ultimately be compiled to machine code. So, in praxis there will be no significant speed difference between those languages. Actually, there are a lot of online tools which will transfer c# code to vb.net code.

Here some more information about using serial to transfer data https://github.com/TeensyUser/doc/wiki/Serial.

All in all I'd say the transfer of your spectrometer data to the PC will be one of the smaller problems in your project. Getting those spectrometer data A/D converted at 500kHz with a reasonable accuracy seems to be more of a challenge.
What do you intend to measure with your spectrometer?
 
What do you intend to measure with your spectrometer?

First, of course, it would be great if I could get a spectrum at all. My dream is a Raman spectrometer, with which I would be able to analyze surfaces of plastics, for example. If such a device could be produced as a DIY in a small format, it would certainly be an interesting possibility for many people. I know there are such devices, but they are way too expensive. But imagine, one could build an IR or Raman spectrometer easily and cheaply, one could for example separate plastics already at home, or analyze the freshness of fruits while shopping.
 
First, of course, it would be great if I could get a spectrum at all. My dream is a Raman spectrometer, with which I would be able to analyze surfaces of plastics, for example. If such a device could be produced as a DIY in a small format, it would certainly be an interesting possibility for many people. I know there are such devices, but they are way too expensive. But imagine, one could build an IR or Raman spectrometer easily and cheaply, one could for example separate plastics already at home, or analyze the freshness of fruits while shopping.

Goodluck. You've set yourself quite the task for a beginner project. There are a number of us on this forum with a background in this field (scientific imaging instruments), both professionally and the home hobbyist maker. I'm sure we'll be more than happy to provide any assistance we can; Just ask!
 
My dream is a Raman spectrometer
Oh, I did my PhD in stimulated raman scattering some 30 years ago which was a lot of fun :)
Will be a challenge to squeeze out enough sensitivity for Raman spectroscopy from a line sensor. Did you think of using your gratings in a monochromator setup which would allow for much more sensitive detectors?

But imagine, one could build an IR or Raman spectrometer easily and cheaply, one could for example separate plastics already at home, or analyze the freshness of fruits while shopping.
I'm out of that field for many years, but wouldn't you need a powerfull (couple of 100mW, the "bluer" the better) excitation laser to get any DIY usable Raman signal? Nothing you want to point at fruit in the grocery store I guess.
 
Last edited:
Yes, I still think about that. As I said, it would be a dream to get a direct live measurement of the spectrum.
Using a stepper motor to rotate the grating would be possible, but the mechanical effort in that case would of course be much greater. I work in a chemical laboratory with different photometers. Raman, UV/Vis, OES. Our Raman microscope can pick up one section of the spectrum at a time and then rotates
the grating further. Of course, the question for me is whether I really need such a fine resolution as with this microscope, for example, to distinguish some common plastics. Further, the exposure time is often a big problem, as well as the incidence of extraneous light. If the LASERs are too powerful, you also have the problem of destroying the sample. This happens to us quite often, although we use a rather weak red laser.

For example, optical analysis of ethene on the surface of avocados might be a way to measure their ripeness. There are actually optical devices that determine the degree of ripeness of avocados. However, I am not sure what method they use. I don't think Raman is used here. Ethene absorbs in the near infrared, so that should be fairly easy to detect.
 
Until now, we had not talked about the performance of the laser at all. Whether 100mW is needed or not, we will see.
Imagine you are looking at a piece of fabric. For you it has the color green. That is what you can see with a 7341. And that's fine, of course. But if you want to know how the fabric was made, then you take a magnifying glass and see that it is made of yellow and blue threads. That's sort of what a spectrometer can do. With a slit and an optical grating, you can also build a spectrometer with a 7341, but then you have to rotate the grating with a motor so that the different wavelengths fall on the sensor afterwards. Again, you don't need a sensor that distinguishes colors, but only one that sees brightness. Normally you use a photomultiplier, but these are very sensitive and complicated to control. A diode array is unfortunately not as sensitive, but it has the advantage that you don't need any mechanical parts to rotate the wavelength.

https://publicspectra.com/Raman/Polypropylene
 
For those who want to know about Raman spectroscopy, here is an interesting video by Applied Science.

Paul

Thanks.

A bit surprised that de transfer speed to the pc would be a problem. If I understand it correctly a measurement would be an array of about 1000-2000 values and I suspect that you would be lucky with a real 6-bit resolution. This could be done with the usb-keyboard interface.

Keeping the setup rigid and lined up, would be another story.

Alain
 
6 bit would be too little for me. 10 bit could be enough for the time being.
I have 3648 active photodiodes. These are read 135 times analog within one second. But if I take the triple because of the sampling theorem, I come to 405Hz. So 3648 photodiodes with 405Hz results in about 1.5 million values in one second, which would have to be transmitted, if I could use the measured values completely. Of course I could already summarize the three values during the processing in the Teensy. Then there would be 492 thousand of 10 bit measured values. Am I calculating wrong here?
 
Good news is USB serial at 480 Mbit/sec is plenty fast enough to transmit 1.5 million 16 bit integers per second (3 Mbyte/sec).

I believe the default transmit buffer is 8K. It should be just enough to hold 1 whole frame of 3648 integers. If you suffer occasional latency on the PC side and want to keep it from holding up sustained sampling, you might want to use Serial.availableForWrite() to check how much space is available in Teensy's transmit buffer and perhaps temporarily keep a few frames in your programs's own buffers during those moments.

Of course, if the PC side just can't sustain the needed speed, no amount of buffering will help. Buffers on Teensy will all eventually fill up if the PC doesn't ingest fast enough in the long run. I'm mentioning this yet again because it's a problem that's come up on this forum many times, especially with Python using pySerial. Maybe your experience with Visual Basic will be much better than we often hear from people using Python? I just want to say we've heard over and over with these sorts of rapid sampling data collection projects about issues where the ultimate limiting factor turned out to be GUI-based software on the PC side.
 
Back
Top