Forum Rule: Always post complete source code & details to reproduce any issue!
Results 1 to 20 of 20

Thread: Data from Teensy 4.0 to Python via serial

  1. #1
    Member
    Join Date
    Nov 2020
    Location
    Czech Republic
    Posts
    62

    Data from Teensy 4.0 to Python via serial

    Hi all,
    I want to sent data from Teensy 4.0 via serial (USB) to PC and read data by Pyrhon. I have code below but it is very slow. Any idea how to send for exmple 30000 float values and read in Python in 1 second? Now 30000 float values take about 6 seconds.
    Many thanks
    Michal

    Teensy:
    Code:
    Serial.begin(2000000);
     for(int j = 1;j < 30000;j++) 
        {
          Serial.println(random(1, 500) / 100.0); 
         }
    Python:
    Code:
    import serial
    import time
    ser = serial.Serial('COM4', 2000000, timeout=0.01)
    data = []
    
    def READ_SERIAL():
        time_start = time.process_time()
        data.clear()
        i = 0
        while i <= 30000:
            line = ser.readline()
            txthodnota = line.decode() 
            hodnota = float(txthodnota)
            data.append(hodnota)
            i += 1
        time_end = time.process_time()
        print("Time: " + str(time_end - time_start))

  2. #2
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    15,532
    Not knowing Python, does this line actually limit the speed of the transfer data arriving?
    Code:
    ser = serial.Serial('COM4', 2000000, timeout=0.01)
    That is only 2 MBaud and T_4.x USB runs at 480MBps, so that could take 2 more 0's.

    That probably isn't it as the 30K floats (doubles?) with newline would at most be 1.2 MBits - maybe 1.44 Mbits if it is sending 'return' and 'newline' and 4 byte doubles - Opps - except it is sending text not packed byte values so maybe double that in either case

    How long does it take to print directly to SerMon? It may be too fast for SerMon and show some garbage, or maybe the random(1, 500) is slowing it down?

    Try something like this with desired byte count after the decimal: Serial.printf("%f\n", random(1, 500) / 100.0);

    A few times over the years users have found Python to be SLOW at first attempt - 'they' did "something" and got the speed up, not sure if any posted what the solution was.

  3. #3
    Senior Member manicksan's Avatar
    Join Date
    Jun 2020
    Location
    Sweden
    Posts
    504
    I tried this (my first program in python):

    Code:
    import serial
    import time
    ser = serial.Serial('COM5', 0)
    dataRawLine = []
    fifo = []
    data = []
    
    def READ_SERIAL_TASK():
        while ser.inWaiting() > 0:
            b = ser.read(1)
            if (b == b'\r'):
                continue
            if (b == b'\n'):
                fifo.append(dataRawLine)
                dataRawLine.clear()
            else:
                dataRawLine.append(b)
    
    def READ_SERIAL_TEST():
        time_start = time.process_time()
        fifo.clear()
        dataRawLine.clear()
        i = 0
        while i <= 30000:
            READ_SERIAL_TASK()
            if len(fifo) > 1:
                txthodnota = b''.join(fifo.pop(0)).decode()
                if len(txthodnota) != 0 and txthodnota.count('.') == 1 and txthodnota.startswith(".") == False:
                    hodnota = float(txthodnota)
                    data.append(hodnota)
                    i += 1
                else:
                    print("can not convert to float: " + txthodnota)
                
        
        time_end = time.process_time()
        print("Rx Time: " + str(time_end - time_start))
        print(data)
        
    READ_SERIAL_TEST()
    as I was thinking the receive spends most of the time just waits for data in ser.readline()

    but this code above still takes around 7 seconds
    but It's at least non blocking


    The baud rate don't matter as the teensy just ignore that "setting"-request and always operate at max speed possible
    the baud rate is only used for hardware UART to be able to receive and time the asynchronous bit stream
    see the python code where I have set the baudrate to 0, and it still works,
    I even tried something standard available such as 300, and still same receive time.

    I changed the Teensy code to:
    Code:
    void setup() {
      Serial.begin(300); // don't matter what it's set to while using USB serial
    }
    
    void loop() {
      Serial.println(random(1, 500) / 100.0); 
    }
    So that I could just run the python code over and over again

  4. #4
    Senior Member
    Join Date
    Apr 2014
    Location
    Germany
    Posts
    9,640
    Under windows?

  5. #5
    Senior Member manicksan's Avatar
    Join Date
    Jun 2020
    Location
    Sweden
    Posts
    504
    Windows 10

  6. #6
    Senior Member
    Join Date
    Apr 2014
    Location
    Germany
    Posts
    9,640
    Sorry, that question was to the OP.

    I'd probably just write a short C program that captures the data to a file.
    That's way faster than python. He could just read the file in python, then.
    Or probably do the whole thing in C...

    Also note that print() does print 2 decimal places only by default.


    I am very curious to see what else happens in this thread... we have had this some times before ... *wink*

  7. #7
    Senior Member
    Join Date
    Apr 2014
    Location
    Germany
    Posts
    9,640
    If two decimal places are really sufficient, i'd just multiply the data by 100 and save 30000 bytes this way(no ".") ... also, print uses cr+lf..replacing this by "," saves another 30000 chars
    Or, of course, transfer binary data.

  8. #8
    Senior Member manicksan's Avatar
    Join Date
    Jun 2020
    Location
    Sweden
    Posts
    504
    I did try it in C#
    and there the receive time was only 2s
    Code:
    namespace WindowsFormsApp2
    {
        
        public partial class Form1 : Form
        {
            List<string> lines = new List<string>();
            System.Diagnostics.Stopwatch sw = new System.Diagnostics.Stopwatch();
            int count = 0;
            bool running = false;
    
            public Form1()
            {
                InitializeComponent();
            }
    
            private void button1_Click(object sender, EventArgs e)
            {
                serialPort1.Open();
                running = true;
                bgw.RunWorkerAsync();
            }
    
            private void button2_Click(object sender, EventArgs e)
            {
                serialPort1.Close();
                running = false;
                sw.Stop();
                sw.Reset();
            }
    
            private void bgw_ProgressChanged(object sender, ProgressChangedEventArgs e)
            {
                Report r = (Report)e.UserState;
                textBox1.Text = r.elapsedTime.ToString();
                rtxt.Lines = r.lines;
            }
    
            private void bgw_DoWork(object sender, DoWorkEventArgs e)
            {
                while (running)
                {
                    try
                    {
                        lines.Add(serialPort1.ReadLine());
    
                        if (count == 0) sw.Start();
                        count++;
                        if (count == 30000)
                        {
                            sw.Stop();
                            bgw.ReportProgress(0, new Report(lines.ToArray(), sw.ElapsedMilliseconds));
                            sw.Reset();
                            lines.Clear();
                            count = 0;
                        }
                    }catch(Exception ex) { }
                }
            }
        }
        public class Report
        {
            public Report(string[] lines, long elapsedTime)
            {
                this.lines = lines;
                this.elapsedTime = elapsedTime;
            }
            public string[] lines;
            public long elapsedTime;
        }
    }

  9. #9
    Senior Member manicksan's Avatar
    Join Date
    Jun 2020
    Location
    Sweden
    Posts
    504
    binary data would not save anything if only one digit + two decimal places are printed
    as a float is 4 bytes which is "equal" to 0.00 (same amount of bytes)
    in the other hand skipping the \r\n would save 60000 bytes

    it could also be sent in groups for example 1000 numbers at a time ending with a newline

    (hehe I had my C# program running in the background while writing this printing 30000 chars every 2s,
    and suddenly it was vey sluggish, task manager showed 5GB of memory usage, think there is a memory leak)

  10. #10
    Senior Member
    Join Date
    Apr 2014
    Location
    Germany
    Posts
    9,640
    Assuming max 500 , it's two bytes unsigned integer.

    Against "3.13crlf" = 6 bytes.

    So it saves 66%


    Edit: You could save even more by using the upper bits which are always zero

    2 seconds is way behind the max theoretical USB speed. You're benchmarking your computer and program, not the Teensy and not USB...

  11. #11
    Senior Member
    Join Date
    Apr 2014
    Location
    Germany
    Posts
    9,640
    Quote Originally Posted by Frank B View Post
    not the Teensy
    Hm. Not that sure anymore. Is it faster by calculating with float instead double? (again, the /100.0) is superfluent anyway - the PC can do that much better.

  12. #12
    Member
    Join Date
    Nov 2020
    Location
    Czech Republic
    Posts
    62
    To clear situation - I am reading data from accelerometer sensor via Teensy ADC, then convert to mV or g and send to PC/Python. Now as simulation I connected signal generator. See pictures what I see in Python. But for 30000 samples I need wait 6-7 seconds...

    Click image for larger version. 

Name:	python - values from Teensy.jpeg 
Views:	1 
Size:	33.0 KB 
ID:	27313
    Click image for larger version. 

Name:	python FFT.jpeg 
Views:	0 
Size:	16.8 KB 
ID:	27314

  13. #13
    Senior Member
    Join Date
    Apr 2014
    Location
    Germany
    Posts
    9,640
    Yes and we confirmed that python is the reason.

  14. #14
    Senior Member manicksan's Avatar
    Join Date
    Jun 2020
    Location
    Sweden
    Posts
    504
    just of curiosity I did the test using javascript WebSerial API

    there the receive time is only 27mS

  15. #15
    Senior Member
    Join Date
    Apr 2014
    Location
    Germany
    Posts
    9,640
    that's pretty OK

  16. #16
    Member
    Join Date
    Nov 2020
    Location
    Czech Republic
    Posts
    62
    Quote Originally Posted by manicksan View Post
    just of curiosity I did the test using javascript WebSerial API

    there the receive time is only 27mS
    Is there any possibility to use java to save teensy data to file and read in python? I am not familiar / never used Java/C

  17. #17
    Senior Member manicksan's Avatar
    Join Date
    Jun 2020
    Location
    Sweden
    Posts
    504
    I did put in a bytes/s calculation
    and it shows around 5Mbyte/s

    the code in the teensy (to send 1M in one 'chunk'):
    Code:
    void setup() {
      // put your setup code here, to run once:
      Serial.begin(300); // don't matter what it's set to
    }
    
    void loop() {
      // put your main code here, to run repeatedly:
      for (int  i = 0; i < 1000000;i++) {
        Serial.print(random(1, 500) / 100.0);
        Serial.print(" ");
      }
      Serial.println();
    }

  18. #18
    Senior Member
    Join Date
    Apr 2014
    Location
    Germany
    Posts
    1,706
    I don't know how python implements arrays but this might be a performance issue:
    Code:
     data.append(hodnota)
    It can be that python needs to copy the array quite often if it runs out of preallocated memory. The larger the array gets the longer the copy takes. Maybe you can improve performance by preallocating a buffer with more than the expected data size before receiving.

  19. #19
    Junior Member
    Join Date
    Oct 2021
    Location
    Finland
    Posts
    1
    I tried binary mode with python and seems to be fast. Python sends a character to Teensy 4.0 and Teensy sends then data in binary format to PC. 30k values takes about 30-50 ms. I am a hobbyist and don't know the internals. Tested in Ubuntu, some Intel core CPU. Teensy code is made with Platformio but should work similar way in Arduino IDE. Data is sent in one chunk which perhaps is not the best practice in a real application. Python version is 3.8.10 but most likely work in many version. Python program was started from the terminal.

    Code:
    #include <Arduino.h>
    //testing sending float numbers from teensy to python (pyserial) using binary mode
    
    #define ITEMS 30000
    char inChar;
    float data [ITEMS];
    
    void setup() {
      Serial.begin(2000000);
      while (!Serial) {
      ; // wait for serial port to connect. Needed for native USB port only
      }
      //fill data array
      for(int i=0; i<ITEMS; i++) {
        data[i]=(float)i;
      }
    }
    
    void loop() {
      //wait that PC/python request data
      if (Serial.available() > 0) {
        inChar = Serial.read();// get incoming byte:
        //write to PC in binary format
        //https://arduino.stackexchange.com/questions/16765/how-to-cast-float-into-four-bytes
        for(int i=0; i<ITEMS; i++) {
          byte *b = (byte *)&data[i];
          Serial.write(b[0]);//low byte first
          Serial.write(b[1]);
          Serial.write(b[2]);
          Serial.write(b[3]);
        }
      }
    }
    Code:
    import serial
    import time
    import struct
    import pickle
     
    ser = serial.Serial(port='/dev/ttyACM0', baudrate=1000000)
    SAMPLES=30000
    s = struct.Struct('<' + str(SAMPLES) + 'f')#struct for binary float data 
    
    ser.reset_input_buffer()
    start=time.time()
    ser.write("r".encode())#just a char, doesn't matter which char
    serial_data = ser.read(SAMPLES*4)#float is 4 bytes
    unpacked_data = s.unpack(serial_data)
    print(f"took {time.time()-start}")
    print('Unpacked Values:', unpacked_data[0:20])
    print('Unpacked Values:', unpacked_data[29980:30000])
    
    ser.close()

  20. #20
    Senior Member manicksan's Avatar
    Join Date
    Jun 2020
    Location
    Sweden
    Posts
    504
    @tsan
    It takes ~15-20mS on windows 10 as well, using python 3.9.0

    I did try to send all data 30000 items in one go ending with a newline
    and receiving it in python with readLine()
    still no boost

    So the problems are while using ser.readLine() and just reading small amount of data using ser.read(NUM_OF_DATA)

    but when reading big chunks with ser.read() is a lot faster

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •