Teensy 3.6 to Python Serial Communication Optimization

Status
Not open for further replies.
X

Xerrorable

Member
Hi,
I realize that this question might have been asked multiple times in some other shape or form on this forum. I have read several of these threads, but I just can't figure out what is best. I would really appreciate the help!
My question regards the optimization of serial communication between a Teensy 3.6 and my Python (3) program. The Teensy is used as a data acquisition tool and the Python script is used to control the Teensy and visualize/store measured data.

My project is a sort of home-made scalar network analyzer, which outputs a voltage value for the Teensy ADC (at 13 bit resolution) to be read.
Each measurement consists of a frequency sweep from a certain starting frequency to a final frequency at a certain step size. (e.g. 15,000,000 Hz to 15,100,000 Hz with steps of 1 Hz) I need to record the ADC value (analogRead()) at each frequency and somehow communicate this to my Python script on my PC. As you can imagine this is quite a lot of data that has to be transferred and I wonder how I can transfer this in the most efficient way.

I have already noticed that allowing the Teensy to just Serial.print() the values will quickly overflow the buffer. Implementing a delay is possible of course, but I really need the data transfer to be as fast as possible.
Is there any other way to allow for faster communication without large delays or was my choice of Python just poor?

I am very much looking forward to any advice! Just general advice regarding the feasibility would already be very welcome. I could of course share my code, but at this point we would probably get lost in the details as I am a beginner programmer.
Cheers.
 
Sending the values in text form is a bit slower and uses a bit more bandwidth, but it is MUCH easier to handle and to monitor and debug any issues. The Serial over USB from T 3.6 to PC is real fast.

My suggestion is that you first get it running with ascii serial, that is using Serial.println(adcvalue) .

If it is the receiving (Python) buffer that overflows, and some delay on Teensy, try to optimize the python reading code, and then when everything works smoothly, even if a bit slower than you like, try to crank up the speed.

Its a better start with a working and slow solution than a nonworking and fast solution.
 
Thanks for the suggestion mlu!
I have tried this and it seems to work.
I tell my Teensy to send a begin character 'b' and stop character 's'. All the data is put in between with a Serial.print(analogRead(pin)) and a Serial.print(',') to serve as delimiter. (I do not use any delays in the Arduino code yet)
My python code looks like this (do note that a lot of libraries are not in use, yet):
Code: 
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
import time as time
import re
import serial
import csv
import struct
from drawnow import *


input_freq_start = '15500000'#input('Enter starting frequency: ')
input_freq_stop = '16500000'#input('Enter stopping frequency: ')
input_freq_step = '100'#input('Enter frequency stepsize: ')



ser = serial.Serial("COM4", 9600)
ser.flushInput()
ser.write((input_freq_start+';'+input_freq_stop+';'+input_freq_step+'\n').encode())

data = ''
Sweep = True
Input = False
while Sweep == True:
    if ser.inWaiting()>0:
        received = ser.read(ser.inWaiting()).decode('ascii').strip()
        data=data+received
        if data[-1]=='s':
            Sweep = False
ser.close()

data = data.split(",")
array = list(map(int, data[1:-1]))
fig = plt.figure()
ax = fig.add_subplot(111)
#ax.set_ylim(0,9000)
ax.plot(range(len(array)),array,color='red')

It seems to be working repeatedly always giving me the same amount of values.
The plot also looks good, it is a sweep of a quartz oscillator used on the Arduino. The x and y label are still a bit arbitrary, but at least it works.

freq_sweep_example.png

Do you think this is the fastest way, or could there potentially be a faster way?
I was also wondering if it is necessary to specify the baud rate for Teensy as I read somewhere that actually Teensy communicates at full USB speed?

Cheers!
 
Looks good :),

yes the USB serial runs at full USB if the sender and reader can handle this, I routinely sets the baurates at 115200. What is the effective transfer rate, in bytes/s or samples/s, have you added any delays on the Teensy side ?
 
Hey Mlu,
After some more testing, I found some more info.
For example when sending 10000 13 bit ADC values (minimum 2 bytes per value, correct?) I can reach about 2 scans per second, so 40000 bytes/s (correct?) I guess that with 12 bit resolution I could use 1.5 bytes and squeeze in some more data. Currently, I do not have any delays on the Teensy side.

On the python side I use serial.read(serial.inWaiting()) and it shows that it is constantly full (4096 bytes) when reading. Does this mean the buffer is overflowing and the Teensy is simply waiting for space?
Cheers!
 
In text format 13 bit unsigned values are 0 to 8192 so max four characters plus the ',' separator, so max 5 and min 2 characters per sample. Lets say 3 on average.

Anyway if the python in buffer is always full the limiting factor seems to be the python read and display handling. One possible way to improve speed is to not keep adding received values at the end of a string with +, this will constantly need time to expand the buffer allocation, but to preallocate a sufficient buffer or read up to ',' convert to value and save in a preallocated array of values.

The decode("ascii") is probably not needed since output from Teensy is "ascii"
 
Maybe I should have added this as well, but I changed the Arduino and Python code.
I am no longer using the comma separation and am now counting bytes on the python side.

Teensy Code that sends the data:
Code: 
  if (new_command == 1)
  { 
    Serial.write("b,");
    for (int i=freq_start; i<=freq_stop; i+= freq_step)
    {
      Set_Frequency(i);
      //delayMicroseconds(10);
      uint16_t reading = analogRead(AD8310_MAG);
      uint8_t byte_array[2] = {reading >> 8, reading & 0xFF};
      Serial.write(byte_array, 2);

      //Serial.print(test_counter);
      //Serial.print(",");
      //test_counter+=1;
    }
    
    Serial.write("s");
    digitalWrite(13,HIGH);
    new_command = 0;
  }

Python code that reads the serial data:
Code: 
ser = serial.Serial("COM4", 912600)
ser.flushInput()
ser.write((input_freq_start+';'+input_freq_stop+';'+input_freq_step+'\n').encode())

data = []
length = len(range(int(input_freq_start),int(input_freq_stop)+int(input_freq_step),int(input_freq_step)))
array= ''
c=int(0)
Sweep = True
try:
    while True:
        
        if Sweep == False:
            ser.write((input_freq_start+';'+input_freq_stop+';'+input_freq_step+'\n').encode())
            Sweep = True
            data = []
            c = 0
        while Sweep == True:
            if ser.inWaiting()>0:
                a_array = ser.read(ser.inWaiting())
                start_sequence = b'b,'

                if c>0:
                    array_pairs = zip(a_array[::2], a_array[1::2]) # (sample1, 2), (sample 3, 4)
                elif c==0:
                    array_pairs = zip(a_array[len(start_sequence)::2], a_array[len(start_sequence)+1::2]) # (sample1, 2), (sample 3, 4)
                stream = [i[0] << 8 | i[1] for i in array_pairs]
                # import pdb;pdb.set_trace()
                data = data+stream
                c+=1
                if len(data)==length:
                    Sweep = False
                    update(data)
except KeyboardInterrupt:
    print('interrupted!')
    close_all()
    print('yes.. really')

I will look into predefining the data arrays, since I know beforehand which amount of data points I am receiving.
 
I think you can read in CSV data directly using the pandas package, assuming it can handle serial in.
 
Hi MarkT,
I am not sure what you mean by this comment.

@Mlu
Could the serial.in_waiting that is constantly 4096 indeed point towards a bottleneck on the python side?
If so, can this be tackled? I am using just the regular pySerial library.

Edit: can this be tackled in the sense of increasing the buffer size?
 
I do not really get where you are going with this suggestion, MarkT.
I am having problems with my serial communication speed and possibly with the buffer size.

My main question is how I can move my data as fast as possible from the Teensy 3.6 to my Python (3) script.
Each sweep is 10000 13 bit ADC values, using 2 bytes per 13 bit value makes it 20000 bytes which are transfered (excluding a start and stop byte sequence).
Could anyone offer advice as to whether it should be possible to make this transfer faster than 40000 bytes/s (roughly 2 sweeps per second)?
 
I used a test from a post of Defragster (http://github.com/PaulStoffregen/USB-Serial-Print-Speed-Test) made by Paul, to test Teensy 3.6 speed.

The serial monitor shows that it prints roughly 36K lines per second (this is not really affected by baud rates higher than 115200).

I also tried this with a Teensy 4.0 which reaches roughly 110K lines per second on my Windows PC.

I read that the Teensy 3.6 should at least be capable of 12Mbit/s data transfer over its USB port. That should be more than fast enough to send my 40000 bytes at least 35-37 times per second.
Am I correct in assuming that the problem is at the python side of the program? I would really appreciate the help! It is likely due to my inexperienced programming.
 
To clearify some details:
- the speed that you set with Serial.begin() gets irgnored. For USB, it is always the max possible speed.
- 12Mbit does not mean it is the speed you can reach. There is pretty high overhead, and it depends on the host.
But yes, even this is way higher than your 40000 bytes. So, I'd assume the bottlenec is python.
Paul had a lot of trouble to get the serial monitor - which is written in java - fast enough to handle the high speed of the teensy.
But if you want to maximize it further, on the Teensy side, try to send send the data in large chunks (64 bytes perhaps)
 
Hi Frank,
Thanks for the input!
Good to know that the baud rate gets ignored. That explains why there is no difference.
I understand that the 12 Mbit/s cannot only be used for data. Some overhead would be expected.

In my python script I am trying narrow down where the slack comes from. I am using pyserial which appears to be using a 4K buffer. Everytime I examine Serial.in_waiting it yields 4K bytes. I tried manually changing this in the serialwin32.py file, but this had not effect (Serial.in_waiting still prints 4096).

I will look into the possibility of sending large chunks. Right now I am sending 2 bytes continuously.
 
Mhmm, that is an interesting find, Mark.
Are there any alternatives for PySerial that you know of?

I followed Mlu's advice of preallocating memory for storing data, but that did not have a significant impact. It takes 0.46s to send the command to receive the 10000 13 bit ADC values.
I will work on some more profiling to be able to narrow down more precisely where the time is lost.

Cheers,
Rens
 
Status
Not open for further replies.
Back
Top