Data from Teensy 4.0 to Python via serial

frohr

Well-known member
Hi all,
I want to sent data from Teensy 4.0 via serial (USB) to PC and read data by Pyrhon. I have code below but it is very slow. Any idea how to send for exmple 30000 float values and read in Python in 1 second? Now 30000 float values take about 6 seconds.
Many thanks
Michal

Teensy:
Code:
Serial.begin(2000000);
 for(int j = 1;j < 30000;j++) 
    {
      Serial.println(random(1, 500) / 100.0); 
     }

Python:
Code:
import serial
import time
ser = serial.Serial('COM4', 2000000, timeout=0.01)
data = []

def READ_SERIAL():
    time_start = time.process_time()
    data.clear()
    i = 0
    while i <= 30000:
        line = ser.readline()
        txthodnota = line.decode() 
        hodnota = float(txthodnota)
        data.append(hodnota)
        i += 1
    time_end = time.process_time()
    print("Time: " + str(time_end - time_start))
 
Not knowing Python, does this line actually limit the speed of the transfer data arriving?
Code:
ser = serial.Serial('COM4', [B][U]2000000[/U][/B], timeout=0.01)

That is only 2 MBaud and T_4.x USB runs at 480MBps, so that could take 2 more 0's.

That probably isn't it as the 30K floats (doubles?) with newline would at most be 1.2 MBits - maybe 1.44 Mbits if it is sending 'return' and 'newline' and 4 byte doubles - Opps - except it is sending text not packed byte values so maybe double that in either case

How long does it take to print directly to SerMon? It may be too fast for SerMon and show some garbage, or maybe the random(1, 500) is slowing it down?

Try something like this with desired byte count after the decimal: Serial.printf("%f\n", random(1, 500) / 100.0);

A few times over the years users have found Python to be SLOW at first attempt - 'they' did "something" and got the speed up, not sure if any posted what the solution was.
 
I tried this (my first program in python):

Code:
import serial
import time
ser = serial.Serial('COM5', 0)
dataRawLine = []
fifo = []
data = []

def READ_SERIAL_TASK():
    while ser.inWaiting() > 0:
        b = ser.read(1)
        if (b == b'\r'):
            continue
        if (b == b'\n'):
            fifo.append(dataRawLine)
            dataRawLine.clear()
        else:
            dataRawLine.append(b)

def READ_SERIAL_TEST():
    time_start = time.process_time()
    fifo.clear()
    dataRawLine.clear()
    i = 0
    while i <= 30000:
        READ_SERIAL_TASK()
        if len(fifo) > 1:
            txthodnota = b''.join(fifo.pop(0)).decode()
            if len(txthodnota) != 0 and txthodnota.count('.') == 1 and txthodnota.startswith(".") == False:
                hodnota = float(txthodnota)
                data.append(hodnota)
                i += 1
            else:
                print("can not convert to float: " + txthodnota)
            
    
    time_end = time.process_time()
    print("Rx Time: " + str(time_end - time_start))
    print(data)
    
READ_SERIAL_TEST()

as I was thinking the receive spends most of the time just waits for data in ser.readline()

but this code above still takes around 7 seconds
but It's at least non blocking


The baud rate don't matter as the teensy just ignore that "setting"-request and always operate at max speed possible
the baud rate is only used for hardware UART to be able to receive and time the asynchronous bit stream
see the python code where I have set the baudrate to 0, and it still works,
I even tried something standard available such as 300, and still same receive time.

I changed the Teensy code to:
Code:
void setup() {
  Serial.begin(300); // don't matter what it's set to while using USB serial
}

void loop() {
  Serial.println(random(1, 500) / 100.0); 
}
So that I could just run the python code over and over again
 
Sorry, that question was to the OP.

I'd probably just write a short C program that captures the data to a file.
That's way faster than python. He could just read the file in python, then.
Or probably do the whole thing in C...

Also note that print() does print 2 decimal places only by default.


I am very curious to see what else happens in this thread... we have had this some times before ... *wink*
 
If two decimal places are really sufficient, i'd just multiply the data by 100 and save 30000 bytes this way(no ".") ... also, print uses cr+lf..replacing this by "," saves another 30000 chars :)
Or, of course, transfer binary data.
 
I did try it in C#
and there the receive time was only 2s
Code:
namespace WindowsFormsApp2
{
    
    public partial class Form1 : Form
    {
        List<string> lines = new List<string>();
        System.Diagnostics.Stopwatch sw = new System.Diagnostics.Stopwatch();
        int count = 0;
        bool running = false;

        public Form1()
        {
            InitializeComponent();
        }

        private void button1_Click(object sender, EventArgs e)
        {
            serialPort1.Open();
            running = true;
            bgw.RunWorkerAsync();
        }

        private void button2_Click(object sender, EventArgs e)
        {
            serialPort1.Close();
            running = false;
            sw.Stop();
            sw.Reset();
        }

        private void bgw_ProgressChanged(object sender, ProgressChangedEventArgs e)
        {
            Report r = (Report)e.UserState;
            textBox1.Text = r.elapsedTime.ToString();
            rtxt.Lines = r.lines;
        }

        private void bgw_DoWork(object sender, DoWorkEventArgs e)
        {
            while (running)
            {
                try
                {
                    lines.Add(serialPort1.ReadLine());

                    if (count == 0) sw.Start();
                    count++;
                    if (count == 30000)
                    {
                        sw.Stop();
                        bgw.ReportProgress(0, new Report(lines.ToArray(), sw.ElapsedMilliseconds));
                        sw.Reset();
                        lines.Clear();
                        count = 0;
                    }
                }catch(Exception ex) { }
            }
        }
    }
    public class Report
    {
        public Report(string[] lines, long elapsedTime)
        {
            this.lines = lines;
            this.elapsedTime = elapsedTime;
        }
        public string[] lines;
        public long elapsedTime;
    }
}
 
binary data would not save anything if only one digit + two decimal places are printed
as a float is 4 bytes which is "equal" to 0.00 (same amount of bytes)
in the other hand skipping the \r\n would save 60000 bytes

it could also be sent in groups for example 1000 numbers at a time ending with a newline

(hehe I had my C# program running in the background while writing this printing 30000 chars every 2s,
and suddenly it was vey sluggish, task manager showed 5GB of memory usage, think there is a memory leak)
 
Assuming max 500 , it's two bytes unsigned integer.

Against "3.13crlf" = 6 bytes.

So it saves 66%


Edit: You could save even more by using the upper bits which are always zero

2 seconds is way behind the max theoretical USB speed. You're benchmarking your computer and program, not the Teensy and not USB... :)
 
To clear situation - I am reading data from accelerometer sensor via Teensy ADC, then convert to mV or g and send to PC/Python. Now as simulation I connected signal generator. See pictures what I see in Python. But for 30000 samples I need wait 6-7 seconds...

python - values from Teensy.jpeg
python FFT.jpeg
 
just of curiosity I did the test using javascript WebSerial API

there the receive time is only 27mS :cool:

Is there any possibility to use java to save teensy data to file and read in python? I am not familiar / never used Java/C
 
I did put in a bytes/s calculation
and it shows around 5Mbyte/s

the code in the teensy (to send 1M in one 'chunk'):
Code:
void setup() {
  // put your setup code here, to run once:
  Serial.begin(300); // don't matter what it's set to
}

void loop() {
  // put your main code here, to run repeatedly:
  for (int  i = 0; i < 1000000;i++) {
    Serial.print(random(1, 500) / 100.0);
    Serial.print(" ");
  }
  Serial.println();
}
 
I don't know how python implements arrays but this might be a performance issue:
Code:
 data.append(hodnota)

It can be that python needs to copy the array quite often if it runs out of preallocated memory. The larger the array gets the longer the copy takes. Maybe you can improve performance by preallocating a buffer with more than the expected data size before receiving.
 
I tried binary mode with python and seems to be fast. Python sends a character to Teensy 4.0 and Teensy sends then data in binary format to PC. 30k values takes about 30-50 ms. I am a hobbyist and don't know the internals. Tested in Ubuntu, some Intel core CPU. Teensy code is made with Platformio but should work similar way in Arduino IDE. Data is sent in one chunk which perhaps is not the best practice in a real application. Python version is 3.8.10 but most likely work in many version. Python program was started from the terminal.

Code:
#include <Arduino.h>
//testing sending float numbers from teensy to python (pyserial) using binary mode

#define ITEMS 30000
char inChar;
float data [ITEMS];

void setup() {
  Serial.begin(2000000);
  while (!Serial) {
  ; // wait for serial port to connect. Needed for native USB port only
  }
  //fill data array
  for(int i=0; i<ITEMS; i++) {
    data[i]=(float)i;
  }
}

void loop() {
  //wait that PC/python request data
  if (Serial.available() > 0) {
    inChar = Serial.read();// get incoming byte:
    //write to PC in binary format
    //https://arduino.stackexchange.com/questions/16765/how-to-cast-float-into-four-bytes
    for(int i=0; i<ITEMS; i++) {
      byte *b = (byte *)&data[i];
      Serial.write(b[0]);//low byte first
      Serial.write(b[1]);
      Serial.write(b[2]);
      Serial.write(b[3]);
    }
  }
}

Code:
import serial
import time
import struct
import pickle
 
ser = serial.Serial(port='/dev/ttyACM0', baudrate=1000000)
SAMPLES=30000
s = struct.Struct('<' + str(SAMPLES) + 'f')#struct for binary float data 

ser.reset_input_buffer()
start=time.time()
ser.write("r".encode())#just a char, doesn't matter which char
serial_data = ser.read(SAMPLES*4)#float is 4 bytes
unpacked_data = s.unpack(serial_data)
print(f"took {time.time()-start}")
print('Unpacked Values:', unpacked_data[0:20])
print('Unpacked Values:', unpacked_data[29980:30000])

ser.close()
 
@tsan
It takes ~15-20mS on windows 10 as well, using python 3.9.0

I did try to send all data 30000 items in one go ending with a newline
and receiving it in python with readLine()
still no boost

So the problems are while using ser.readLine() and just reading small amount of data using ser.read(NUM_OF_DATA)

but when reading big chunks with ser.read() is a lot faster
 
I did look at the pyserial source code
and when doing a read it calls the kernel32 (windows)
to get any data
I guess this takes a little time to do

so for every "request" this time adds up to a huge amount

30000 * (4-char float + 2 '/r/n') = 180000

7000mS/180000 ~ 38uS (request time)

and the readLine function is not efficient at all
it just gets the data bytes one at a time.


The conclusion of this is to
send data at known huge chunks

Alternative is when sending data from the teensy is to first send a line describing how much data
the 'python' program is to receive, that way you don't need to hardcode the size at both ends,
and you just need to change it at the teensy code.
 
It's been a couple years since I got everything working but you can definitely pull data into Python on Windows 10 at 10MB (yes bytes) per second from a T4. You'll need to scoop the data in large chunks and spend some time profiling your Python to find out where it slows down. Any real-time processing HAS to be in another thread/process (I forget which is the correct term here) and GUI updates have to be done on a schedule as they're very slow.

I know this is lacking in specifics but I just want to echo that a T4 can easily handle your task, and Python and Windows CAN do to it but you need to be on your game.
 
I tried binary mode with python and seems to be fast. Python sends a character to Teensy 4.0 and Teensy sends then data in binary format to PC. 30k values takes about 30-50 ms. I am a hobbyist and don't know the internals. Tested in Ubuntu, some Intel core CPU. Teensy code is made with Platformio but should work similar way in Arduino IDE. Data is sent in one chunk which perhaps is not the best practice in a real application. Python version is 3.8.10 but most likely work in many version. Python program was started from the terminal.

Code:
#include <Arduino.h>
//testing sending float numbers from teensy to python (pyserial) using binary mode

#define ITEMS 30000
char inChar;
float data [ITEMS];

void setup() {
  Serial.begin(2000000);
  while (!Serial) {
  ; // wait for serial port to connect. Needed for native USB port only
  }
  //fill data array
  for(int i=0; i<ITEMS; i++) {
    data[i]=(float)i;
  }
}

void loop() {
  //wait that PC/python request data
  if (Serial.available() > 0) {
    inChar = Serial.read();// get incoming byte:
    //write to PC in binary format
    //https://arduino.stackexchange.com/questions/16765/how-to-cast-float-into-four-bytes
    for(int i=0; i<ITEMS; i++) {
      byte *b = (byte *)&data[i];
      Serial.write(b[0]);//low byte first
      Serial.write(b[1]);
      Serial.write(b[2]);
      Serial.write(b[3]);
    }
  }
}

Code:
import serial
import time
import struct
import pickle
 
ser = serial.Serial(port='/dev/ttyACM0', baudrate=1000000)
SAMPLES=30000
s = struct.Struct('<' + str(SAMPLES) + 'f')#struct for binary float data 

ser.reset_input_buffer()
start=time.time()
ser.write("r".encode())#just a char, doesn't matter which char
serial_data = ser.read(SAMPLES*4)#float is 4 bytes
unpacked_data = s.unpack(serial_data)
print(f"took {time.time()-start}")
print('Unpacked Values:', unpacked_data[0:20])
print('Unpacked Values:', unpacked_data[29980:30000])

ser.close()

Hi,
I try now to use bluetooth instead serial. Maybe you can help me because I have stucked.

I have now very similar Teensy code:
Code:
#include <SoftwareSerial.h>
SoftwareSerial mySerial(0, 1); // RX, TX
char inChar;
void setup()  
{
  Serial.begin(38400);
  mySerial.begin(38400);
}
void loop() 
{
    if (Serial.available() > 0) {
    inChar = Serial.read();
    for(int i=0; i<30000; i++) {
      byte *b = (byte *)&i;
      mySerial.write(b[0]);
      mySerial.write(b[1]);
      mySerial.write(b[2]);
      mySerial.write(b[3]);
    }
  }
}

I have problem in Python:
Code:
import bluetooth
import serial
import struct
v_data = []
ser = ""
s = struct.Struct('<' + str(10) + 'f')
SAMPLES = 30000
bd_addr = "FC:A8:9A:00:22:33" #itade address
port = 1
sock=bluetooth.BluetoothSocket( bluetooth.RFCOMM )
sock.connect((bd_addr, port))
print('Connected')
#sock.settimeout(1.0)
 
sock.send("r")
print('Sent data')
 
while True:
    v_data = sock.recv(100)
    if len(v_data) == 0: break
    print("received [%s]" % v_data)
print(v_data)

Data seems like nonsense and it is very slow. Is there any way hot to improve it?
Thanks!
 
Hi,
I try now to use bluetooth instead serial. Maybe you can help me because I have stucked.

I have now very similar Teensy code:
Code:
#include <SoftwareSerial.h>
SoftwareSerial mySerial(0, 1); // RX, TX
char inChar;
void setup()  
{
  Serial.begin(38400);
  mySerial.begin(38400);
}
void loop() 
{
    if (Serial.available() > 0) {
    inChar = Serial.read();
    for(int i=0; i<30000; i++) {
      [COLOR="#FF0000"]byte *b = (byte *)&i;[/COLOR]
      mySerial.write(b[0]);
      mySerial.write(b[1]);
      mySerial.write(b[2]);
      mySerial.write(b[3]);
    }
  }
}

I have problem in Python:
Code:
import bluetooth
import serial
import struct
v_data = []
ser = ""
s = struct.Struct('<' + str(10) + 'f')
SAMPLES = 30000
bd_addr = "FC:A8:9A:00:22:33" #itade address
port = 1
sock=bluetooth.BluetoothSocket( bluetooth.RFCOMM )
sock.connect((bd_addr, port))
print('Connected')
#sock.settimeout(1.0)
 
sock.send("r")
print('Sent data')
 
while True:
    v_data = sock.recv(100)
    if len(v_data) == 0: break
    print("received [%s]" % v_data)
print(v_data)

Data seems like nonsense and it is very slow. Is there any way hot to improve it?
Thanks!

In the original code (from tsan) that you were using as your example, a float value was being stored in the "data[]" array, and the (float) members of that array were being translated into 4-byte components & being sent over the serial port as bytes.

In your code, the integer index "i" is (presumably) being translated into 4-byte components & being sent over bluetooth. Not sure if that's what you intended, but you may be reading neighboring values in your 4-byte translation if that integer index "i" occupies less than 4-bytes.

Mark J Culross
KD5RXT
 
Back
Top