Problem with getting maximum USB transfer speed

Status
Not open for further replies.

linuxgeek

Well-known member
I posted in another thread: http://forum.pjrc.com/threads/18852-Teensy-3-0-to-PC-communication-via-USB-serial-(-gt-256000-baud)
, but now that I have some test code this might be better to start a thread on this. I cannot get more than 128 kilobytes/second, when transferring. It's transferring a buffer in RAM as fast as possible for a 3MB transfer. It always takes about 20 - 30 seconds, regardless of buffer size, etc. Has anyone managed to get faster transfer rates?

teensy 3.0 sketch
Code:
uint8_t *data;
const int buf_length=512;
//a 3MB file                                                                                                   
const int chunks=3000320/buf_length;
long startTime=-1, elapsedTime=-1;

void setup() {
  Serial.begin(2000000);
  for (int i=0;i<8;i++) {
    delay(1000);
    Serial.print(i);
    Serial.println(" seconds");
    Serial.flush();
  }

  data = (byte*)malloc(buf_length);
  memset(data,1,buf_length);

  startTime=micros();
  for (int i=0;i<chunks;i++) {
    Serial.write(data,buf_length);
  }
  //  Serial.println();                                                                                        
  //Serial.print("elapsedTime: ");                                                                             
  //Serial.println(micros()-startTime);                                                                        
  //Serial.flush();                                                                                            
  //  Serial.end();                                                                                            

}

void loop() {

}


Processing Sketch
Code:
import java.nio.ByteBuffer;
import processing.core.*;
import processing.serial.Serial;

public class UsbReceiver extends PApplet {

	/**
	 * 
	 */
	private static final long serialVersionUID = 1L;

	public static void main(String args[]) {
		UsbReceiver usbReceiver = new UsbReceiver();
		Serial myPort = usbReceiver.setupSerial();
		usbReceiver.receiveFile(myPort, "usbTest.out");
	}
	
	public Serial setupSerial() {
		Serial serialPort;
		// List all the available serial ports:
		println(Serial.list());
		// Open the port you are using at the rate you want:
		System.out.println("serial length: " + Serial.list().length);
		serialPort = new Serial(this, Serial.list()[0], 2000000);
		System.out.println("myPort: " + serialPort.available());
		return serialPort;
	}
	
	public void receiveFile(Serial serial, String sOutputFilename) {
		int nBytes = -1, totalBytes = 0;
		int chunkSize = 512, fileSize=3000320;
		byte [] buffBytes = new byte[chunkSize];
		ByteBuffer tmpBuff = ByteBuffer.wrap(buffBytes);
		
		//try & wait until we have chunkSize available
		delay(50);
		System.out.println("avail: " + serial.available());
		long startTime=System.currentTimeMillis();
		
		for (int i=0;totalBytes<(fileSize-chunkSize);i++) {
			if (serial.available() >= chunkSize) {
				nBytes = serial.readBytes(buffBytes);
				totalBytes+=nBytes;
				//System.out.println("read" + i + " nBytes: " + nBytes + " totalBytes: " + totalBytes);
				tmpBuff.rewind();
			} else {
				//System.out.println("sleep: " + i);
				delay(1);
			}
		}
		delay(50);
		//read last chunk
		if (serial.available() > 0) {
			nBytes = serial.readBytes(buffBytes);
			totalBytes+=nBytes;
			//System.out.println(" last read, nBytes: " + nBytes + " totalBytes: " + totalBytes);
			tmpBuff.rewind();
		} else {
			System.out.println("nothing avail on last read");
		}
		System.out.println("Time elapsed: " + (System.currentTimeMillis() - startTime));
		System.out.println("done sending: " + totalBytes);
	}
}

Here's a shell script to make it easier to launch the java applet on linux. You'll need to put the processing sketch in a directory called "serialReceiver". I probably should have done this in Processing IDE, but I did it in eclipse. I'll try in Processing if someone says that will be helpful. You'll need to download Processing for this to work, and change the location of the various entries in the shell script to match your libraries, and classpath if need be.
Code:
#!/bin/sh                                                                                      

LIBRARY="/usr/local/processing/modes/java/libraries/serial/library/linux64"
CLASSPATH="/usr/local/processing/core/library/core.jar:/usr/local/processing/modes/java/libraries/serial/library/RXTXcomm.jar:/usr/local/processing/modes/java/libraries/serial/library/serial.jar:."

echo "java -Djava.library.path="$LIBRARY" -cp "$CLASSPATH" serialReceiver.UsbReceiver"
java -Djava.library.path="$LIBRARY" -cp "$CLASSPATH" serialReceiver.UsbReceiver

Ok, I just tested it and removed the 1st line that listed a package. You can now copy the Processing sketch into Processing and save it as "UsbReceiver", and it will run.
 
Last edited:
Here, this Processing Sketch is a little more useful. It outputs transfer rate:

Code:
import java.nio.ByteBuffer;
import processing.core.*;
import processing.serial.Serial;

public class UsbReceiver extends PApplet {

  /**
   * 
   */
  private static final long serialVersionUID = 1L;

  public static void main(String args[]) {
    UsbReceiver usbReceiver = new UsbReceiver();
    Serial myPort = usbReceiver.setupSerial();
    usbReceiver.receiveFile(myPort, "usbTest.out");
  }
  
  public Serial setupSerial() {
    Serial serialPort;
    // List all the available serial ports:
    println(Serial.list());
    // Open the port you are using at the rate you want:
    System.out.println("serial length: " + Serial.list().length);
    serialPort = new Serial(this, Serial.list()[0], 2000000);
    System.out.println("myPort: " + serialPort.available());
    return serialPort;
  }
  
  public void receiveFile(Serial serial, String sOutputFilename) {
    int nBytes = -1, totalBytes = 0;
    int chunkSize = 512, fileSize=3000320;
    byte [] buffBytes = new byte[chunkSize];
    ByteBuffer tmpBuff = ByteBuffer.wrap(buffBytes);
    long startTime = -1, endTime = -1, elapsedTime = -1;
    
    //try & wait until we have chunkSize available
    delay(50);
    System.out.println("avail: " + serial.available());
    startTime=System.currentTimeMillis();
    
    for (int i=0;totalBytes<(fileSize-chunkSize);i++) {
      if (serial.available() >= chunkSize) {
        nBytes = serial.readBytes(buffBytes);
        totalBytes+=nBytes;
        //System.out.println("read" + i + " nBytes: " + nBytes + " totalBytes: " + totalBytes);
        tmpBuff.rewind();
      } else {
        //System.out.println("sleep: " + i);
        delay(1);
      }
    }
    delay(50);
    //read last chunk
    if (serial.available() > 0) {
      nBytes = serial.readBytes(buffBytes);
      totalBytes+=nBytes;
      //System.out.println(" last read, nBytes: " + nBytes + " totalBytes: " + totalBytes);
      tmpBuff.rewind();
    } else {
      System.out.println("nothing avail on last read");
    }
      endTime=System.currentTimeMillis();
      elapsedTime = endTime - startTime;
      System.out.println("Time elapsed: " + elapsedTime);
      System.out.println("Bytes received: " + totalBytes);
      System.out.println("transfer rate: " + ((fileSize/1000.0)/(elapsedTime/1000)) + " KB/sec");
  }
}

Here's what the output should look like:

Code:
Stable Library
=========================================
Native lib Version = RXTX-2.1-7
Java lib Version   = RXTX-2.1-7
RXTX Warning:  Removing stale lock file. /var/lock/LCK..ttyACM0
[0] "/dev/ttyACM0"
[1] "/dev/ttyS0"
serial length: 2
myPort: 0
avail: 0
Time elapsed: 28818
Bytes received: 3000320
transfer rate: 107.15429 KB/sec

Oh, forgot to mention, there's an 8 second countdown on the arduino sketch, so it's not perfect as far as measuring time. I put that there so I have time to launch the processing sketch, and in case it hangs I can upload more easily by resetting and upload before the countdown is over.
 
Last edited:
I found some of Paul's old python serial script (and his early write ups on optimizing USB transfer rate), and tried that. I just blinked the led and I used a stopwatch, but it looked like it took 3 seconds to transfer 3MB (1MB/sec), so it looks like this is a problem with the java RXTX side. I'll see what I can find out on the Processing forum. Otherwise, I guess I can wrap a python script to do the transfer, or something like that. Good to know the t3 can actually pump it out.

For future reference here's the python script

Code:
# This script flushes every byte received on the serial port                    

import serial
import sys

ser = serial.Serial(
   port = '/dev/ttyACM0',
   baudrate = 115200,
   parity = serial.PARITY_NONE,
   stopbits = serial.STOPBITS_ONE,
   bytesize = serial.EIGHTBITS
)

if ser.isOpen() == False :
   ser.open()
   if ser.isOpen() == False :
      print "Can't open serial port"
      sys.exit(1)
 
I found some of Paul's old python serial script (and his early write ups on optimizing USB transfer rate), and tried that. I just blinked the led and I used a stopwatch, but it looked like it took 3 seconds to transfer 3MB (1MB/sec), so it looks like this is a problem with the java RXTX side.

Thanks! I was going to dig up that old script and give this a try.

1 Mbyte/sec is about the expected speed, though it can depend somewhat on the number and type of other USB devices connected.

You'd think a many-GHz 64-bit multi-core CPU with access to gigabytes of RAM using incredibly fast buses would not be limiting factor, but often it is. The software design matters greatly!
 
Sure.

I am going to try and recompile rxtx and get rid of the 20msec hard coded delay.
Do u think I can just remove the delay altogether?

I'll post the build of rxtx if it works.
I would have thought someone would have posted this by now,
But couldn't find it. Or better yet change it in the codebase.
 
The Arduino Team is now maintaining their own copy of librxtx. If you make a good fix and test it throughly, I'd suggest at least submitting a pull request.

https://github.com/arduino/RXTX

It's not always easy to get Arduino to accept fixes. In fact, it's almost never easy. But it has become much better over the last several months.

To get it accepted, you might need to start a discussion on the Arduino developers mail list. Usually the best path is to first work on the solution, so you're familiar with the problem and you've solved it or know how you will solve it. It's usually best to explain how the problem manifests for real users. Establishing the need for the fix is critical. Usually performance issues are a low priority. Many people make off-topic comments on that list, but ultimately the only person whose opinion matters is Christian Maglie. Well, Massimo of course has final say, but he's unlikely to get involved in a purely technical issue. Once you get Christian to agree, or at least give some positive input, then submit the pull request.
 
Ok, I'll try and do that when I get to that point.
I tried using the normal version of librxtx and compiling, but it doesn't recognize /dev/ttyACM*. Which there's probably that fix and some others in the arduino codebase, so I'll work off of that next.
 
BTW, the arduino rxtx codebase has it changed to usleep(1000), from usleep(20000).

transfer rate is now ~200KB/sec.

There's some other usleep() function calls in there, so might need to poke around.
 
Status
Not open for further replies.
Back
Top