movie2serial: image2data changes running very slowly

Status
Not open for further replies.

BuffaloFan32

Well-known member
I finally was able to change movie2serial so that it reads a bunch of pixel locations from a text file and then grabs those locations from an image and outputs them to my array. My problem is that my edits are causing the program to run very slowly. The biggest changes I made were to image2data:

Code:
// image2data converts an image to OctoWS2811's raw data format.
// The number of vertical pixels in the image must be a multiple
// of 8.  The data array must be the proper size for the image.
void image2data(PImage image, byte[] data) {
  int offset = 3;
  int i, j, counter, mask;
  int pixel[] = new int[8];
  
  for (i = 0; i < strLines.length/8; i++) {             //strLines is a list of pixels to find
    for (j = 0; j < 8; j++) {
      pixel[j] = image.pixels[int(strLines[i * 8 + j])];
      pixel[j] = colorWiring(pixel[j]);
    }
    // convert 8 pixels to 24 bytes                   
      for (mask = 0x800000; mask != 0; mask >>= 1) {
        byte b = 0;
        for (j=0; j < 8; j++) {
          if ((pixel[j] & mask) != 0) b |= (1 << j);
        }
        data[offset++] = b;
      }
  }
}

I also pass the entire screenshot to image2data now (I am only running one Teensy at the moment). I cannot figure out why it is running so much slower now when it seems to be running the same number of iterations on each frame as it was in the original version.
 
Maybe this question could be answered on Processing's forum? It's perfectly fine to ask here, but not many people here are Processing experts. I'm certainly not... and I wrote movie2serial.pde!

EDIT: first guess, probably wrong...

Ok, another guess, comparing this to the original movie2serial. Perhaps accessing the string as an array, like "strLines[i * 8 + j]", is having to jump through lots of Java's hoops of Unicode and international character handling layers?

A couple years ago I worked on the Arduino Serial Monitor's terrible performance. Back then, if Teensy send data at 1 Mbyte/sec, Arduino would completely freeze and eventually crash. Maybe some of that work will help?

Java has astounding bad performance for certain types of text processing, which you'd imagine would be really fast if you think in terms of C strings which are merely arrays of bytes.
 
Last edited:
I've noticed with Serial.write() if you don't write an entire buffer it's very slow. Writing byte by byte is not good. You're probably not doing this though.

Can you post your entire code?

Processing is really.. not that great. It's decent for images but I've always struggled to do much video processing with it. There's no hardware acceleration and it's very hard to tell what data types things like PImage are

PImage.pixels[] is just a standard 1D array though

But I have no idea if this is of type int so there may well be some typecasting going on

and I wrote movie2serial.pde
I was wondering where that came from. My current workplace were apparently using that for about a year to drive their digital displays
 
Last edited:
Here is the full code. Right now it is running an array of LEDs that is 32x113 but I plan to make that a lot bigger. Paul's original version ran the array just fine.

Code:
import processing.serial.*;
import java.awt.Rectangle;

float gamma = 1.7;

int numPorts=0;  // the number of serial ports in use
int maxPorts=24; // maximum number of serial ports

PImage img;  // Declare variable "a" of type PImage
PImage shot;
float x;  // horizontal location of headline
XML xml;
String[] headlines;
int index = 0;

Serial[] ledSerial = new Serial[maxPorts];     // each port's actual Serial port
Rectangle[] ledArea = new Rectangle[maxPorts]; // the area of the movie each port gets, in % (0-100)
boolean[] ledLayout = new boolean[maxPorts];   // layout of rows, true = even is left->right
PImage[] ledImage = new PImage[maxPorts];      // image sent to each port
int[] gammatable = new int[256];
int errorCount=0;
String[] strLines;

void setup() {
  String[] list = Serial.list();
  delay(20);
  println("Serial Ports List:");
  println(list);
  serialConfigure("COM4");  // change these to your port names
  if (errorCount > 0) exit();
  for (int i=0; i < 256; i++) {
    gammatable[i] = (int)(pow((float)i / 255.0, gamma) * 255.0 + 0.5);
  }
  size(480, 400);  // create the window
  
  strLines = loadStrings("C:/Documents and Settings/alissa/Desktop/positions2.txt");
  
  x = width;
  // The image file must be in the data folder of the current sketch 
  // to load successfully
  img = loadImage("test.JPG");  // Load the image into the program
  
  // The URL for the XML document
  String url = "http://news.google.com/news?cf=all&hl=en&pz=1&ned=us&output=rss";
  // Load the XML document
  xml = loadXML(url);
  // Grab the element we want
  XML[] headers = xml.getChildren("channel/item/title");
  headlines = new String[headers.length];
  for (int i = 0; i < headers.length; i++) {
     headlines[i] = headers[i].getContent();
  }
}

// image2data converts an image to OctoWS2811's raw data format.
// The number of vertical pixels in the image must be a multiple
// of 8.  The data array must be the proper size for the image.
void image2data(PImage image, byte[] data) {
  int offset = 3;
  int i, j, counter, mask;
  int pixel[] = new int[8];
  
  for (i = 0; i < strLines.length/8; i++) {
    for (j = 0; j < 8; j++) {
      pixel[j] = image.pixels[int(strLines[i * 8 + j])];
      pixel[j] = colorWiring(pixel[j]);
    }
    // convert 8 pixels to 24 bytes                    this is it
      for (mask = 0x800000; mask != 0; mask >>= 1) {
        byte b = 0;
        for (j=0; j < 8; j++) {
          if ((pixel[j] & mask) != 0) b |= (1 << j);
        }
        data[offset++] = b;
      }
  }
}  

// translate the 24 bit color from RGB to the actual
// order used by the LED wiring.  GRB is the most common.
int colorWiring(int c) {
  int green = (c & 0xFF0000) >> 16;
  int red = (c & 0x00FF00) >> 8;
  int blue = (c & 0x0000FF);
  red = gammatable[red/16];
  green = gammatable[green/16];
  blue = gammatable[blue/16];
  return (green << 16) | (red << 8) | (blue); // GRB - most common wiring
}

// ask a Teensy board for its LED configuration, and set up the info for it.
void serialConfigure(String portName) {
  if (numPorts >= maxPorts) {
    println("too many serial ports, please increase maxPorts");
    errorCount++;
    return;
  }
  try {
    ledSerial[numPorts] = new Serial(this, portName);
    if (ledSerial[numPorts] == null) throw new NullPointerException();
    ledSerial[numPorts].write('?');
  } catch (Throwable e) {
    println("Serial port " + portName + " does not exist or is non-functional");
    errorCount++;
    return;
  }
  delay(50);
  String line = ledSerial[numPorts].readStringUntil(10);
  if (line == null) {
    println("Serial port " + portName + " is not responding.");
    println("Is it really a Teensy 3.0 running VideoDisplay?");
    errorCount++;
    return;
  }
  String param[] = line.split(",");
  if (param.length != 12) {
    println("Error: port " + portName + " did not respond to LED config query");
    errorCount++;
    return;
  }
  // only store the info and increase numPorts if Teensy responds properly
  ledImage[numPorts] = new PImage(Integer.parseInt(param[0]), Integer.parseInt(param[1]), RGB);
  ledArea[numPorts] = new Rectangle(Integer.parseInt(param[5]), Integer.parseInt(param[6]),
                     Integer.parseInt(param[7]), Integer.parseInt(param[8]));
  ledLayout[numPorts] = (Integer.parseInt(param[5]) == 0);
  numPorts++;
}

void draw() {
  // Displays the image in same size as window at point (0,0)
  image(img, 0, 0, 480, 400);
  textSize(100);
  fill(255,255,255);
  text(headlines[index], x, 170);
  x = x - 10;
  float w = textWidth(headlines[index]);
  if (x < -w) {
    x = width; 
    index = (index + 1) % headlines.length;
  }
  if (keyPressed) {
    if (key == 'x') {
      clear();
    } else {
      //draw a bunch of squares below the pic
      loadPixels();
      for (int i=0; i < strLines.length; i++) {
        pixels[int(strLines[i])] = color(0, 0, 0);
      }
      updatePixels();
    }
  }
  
  for (int i=0; i < numPorts; i++) {    
    // copy a portion of the movie's image to the LED image
    int xoffset = percentage(width, ledArea[i].x);
    int yoffset = percentage(height, ledArea[i].y);
    int xwidth =  percentage(width, ledArea[i].width);
    int yheight = percentage(height, ledArea[i].height);
    //rect(10, 75, 10, 10);
    //shot = get(10, 75, 10, 10);
    shot = get(0, 0, width, height);
    //ledImage[i].copy(shot, xoffset, yoffset, xwidth, yheight,
    //                 0, 0, xwidth, yheight);
    ledImage[i] = shot;
    // convert the LED image to raw data
    byte[] ledData =  new byte[(ledImage[i].width * ledImage[i].height * 3) + 3];
    image2data(ledImage[i], ledData);
    if (i == 0) {
      ledData[0] = '*';  // first Teensy is the frame sync master
      int usec = (int)((1000000.0 / 30) * 0.75);
      ledData[1] = (byte)(usec);   // request the frame sync pulse
      ledData[2] = (byte)(usec >> 8); // at 75% of the frame time
    } else {
      ledData[0] = '%';  // others sync to the master board
      ledData[1] = 0;
      ledData[2] = 0;
    }
    // send the raw data to the LEDs  :-)
    ledSerial[i].write(ledData); 
  }
  
  if (keyPressed) {
    if (key == 'x') {
      exit();
    }
  }
}

// scale a number by a percentage, from 0 to 100
int percentage(int num, int percent) {
  double mult = percentageFloat(percent);
  double output = num * mult;
  return (int)output;
}

// scale a number by the inverse of a percentage, from 0 to 100
int percentageInverse(int num, int percent) {
  double div = percentageFloat(percent);
  double output = num / div;
  return (int)output;
}

// convert an integer from 0 to 100 to a float percentage
// from 0.0 to 1.0.  Special cases for 1/3, 1/6, 1/7, etc
// are handled automatically to fix integer rounding.
double percentageFloat(int percent) {
  if (percent == 33) return 1.0 / 3.0;
  if (percent == 17) return 1.0 / 6.0;
  if (percent == 14) return 1.0 / 7.0;
  if (percent == 13) return 1.0 / 8.0;
  if (percent == 11) return 1.0 / 9.0;
  if (percent ==  9) return 1.0 / 11.0;
  if (percent ==  8) return 1.0 / 12.0;
  return (double)percent / 100.0;
}
 
Don't resize the image in software using your percentage functions. Use copy() to create a new resized image. Then use the pixels[] data unchanged

Also I can't see where you're setting your baud rate. Would it not default to 9600?
 
For USB serial, the baud rate setting has no effect on the actual speed. Data is always sent at 12 Mbit/sec using the USB "bulk" protocol, which does automatically use flow control to adjust the speed to however fast both sides can actually work.

The baud rate setting is never actually used. It's merely stored on the Teensy side, so code on the Teensy can read it if necessary. OctoWS2811's VideoDisplay example (and most other code) just ignores the baud setting.
 
Well... I won't be needing this then :L
Code:
matrixPort = new Serial(this, "/dev/ttyACM1", 4000000);

I was wondering why there was mismatches between the USB serial and hardware serial
 
I don't I think I am actually using the percentage function anymore. I should remove it but I thought I might need it when I add more Teensy's. Right now, 'shot' is a copy of the whole frame and I pass the whole frame to image2data.
 
is there a version of movie2serial in written C++?

I'm not aware of anyone who's ported it all to another language and actually published usable code. Perhaps it's out there somewhere?

A few years ago I ported some of the code to C for this project.

http://www.dorkbotpdx.org/blog/paul/maker_faire_2013

This doesn't read a recorded movie, but it does read live webcam data (using the v4l2 library) and it overlays animated GIF. All the "2serial" part of movie2serial is present. Maybe you could add the "movie" part somehow?

If you do (or if anyone knows of anyone who has), I hope you'll post a followup and share the code.
 
I have seen that before but I'm afraid it was a little over my head. Maybe I should give it another go. The broad concept is similar to what I am trying to accomplish.
 
Status
Not open for further replies.
Back
Top