Serial communication problem: Magic numbers?

Status
Not open for further replies.

mqlo

Member
Hi,

I'm using (Py)CmdMessenger on OS X with Python 3 in conjunction with a Teensy 3.6 to control the intensity of a few LEDs. I do this by sending a series of 2x 7 integers as binaries to the Teensy who then turns on the output to the pins accordingly using analogWrite() with 12-bit precision.

This works fine except under two specific circumstances, namely when the at least 6 of the 7 binaries are 44 or 59 (out of 4095). I know this might sound bizarre but when six of the numbers are either 44 or 59, the command somehow gets lost. It is 100% repeatable.

When I hard code the values (in my Teensy sketch) to be 44 or 59, I can turn on the LEDs just fine, so I suspect it has to do with the serial communication.

Any suggestions at all?

Teensy sketch:

Code:
#include "CmdMessenger.h"

/* Define available CmdMessenger commands */
enum {
    get_valsS,
    get_valsC,
};


// Here, we'll different pins for the 2x 7 LEDs
// Define the first set
int C01 = 36;
int C02 = 37;
int C03 = 35;
int C04 = 10;
int C05 = 9;
int C06 = 6; 
int C07 = 17;

// Define the second set
int S01 = 2;
int S02 = 7;
int S03 = 8;
int S04 = 14;
int S05 = 20;
int S06 = 21;
int S07 = 22;

// Status LED
const int X00 = 13;

/* Initialize CmdMessenger -- this should match PyCmdMessenger instance */
const int BAUD_RATE = 115200;
CmdMessenger cmdMessenger = CmdMessenger(Serial,',',';','/');

void setup() {
  // put your setup code here, to run once:
  analogWriteResolution(12);// analogWrite value 0 to 4095 for high
  
  // Set all pins to output
  pinMode(C01, OUTPUT);
  pinMode(C02, OUTPUT);
  pinMode(C03, OUTPUT);
  pinMode(C04, OUTPUT);
  pinMode(C05, OUTPUT);
  pinMode(C06, OUTPUT);
  pinMode(C07, OUTPUT);
  pinMode(S01, OUTPUT);
  pinMode(S02, OUTPUT);
  pinMode(S03, OUTPUT);
  pinMode(S04, OUTPUT);
  pinMode(S05, OUTPUT);
  pinMode(S06, OUTPUT);
  pinMode(S07, OUTPUT);
  pinMode(X00, OUTPUT);


   int intensity0 = 4095;
   for (int i=0; i <= 38; i++){
        analogWrite(i, intensity0);
  }

  Serial.begin(BAUD_RATE);
  attach_callbacks();
}



/* callback */
void on_get_valsC(void){
    /* Grab seven integers
    int valueC01 = cmdMessenger.readBinArg<int>();
    int valueC02 = cmdMessenger.readBinArg<int>();
    int valueC03 = cmdMessenger.readBinArg<int>();
    int valueC04 = cmdMessenger.readBinArg<int>();
    int valueC05 = cmdMessenger.readBinArg<int>();
    int valueC06 = cmdMessenger.readBinArg<int>();
    int valueC07 = cmdMessenger.readBinArg<int>();

    analogWrite(C01, invv(valueC01));
    analogWrite(C02, invv(valueC02));
    analogWrite(C03, invv(valueC03));
    analogWrite(C04, invv(valueC04));
    analogWrite(C05, invv(valueC05));
    analogWrite(C06, invv(valueC06));
    analogWrite(C07, invv(valueC07));

}

/* callback */
void on_get_valsS(void){
    /* Grab seven integers
    int valueS01 = cmdMessenger.readBinArg<int>();
    int valueS02 = cmdMessenger.readBinArg<int>();
    int valueS03 = cmdMessenger.readBinArg<int>();
    int valueS04 = cmdMessenger.readBinArg<int>();
    int valueS05 = cmdMessenger.readBinArg<int>();
    int valueS06 = cmdMessenger.readBinArg<int>();
    int valueS07 = cmdMessenger.readBinArg<int>();
    
    analogWrite(S01, invv(valueS01));
    analogWrite(S02, invv(valueS02));
    analogWrite(S03, invv(valueS03));
    analogWrite(S04, invv(valueS04));
    analogWrite(S05, invv(valueS05));
    analogWrite(S06, invv(valueS06));
    analogWrite(S07, invv(valueS07));

}


void loop() {
    cmdMessenger.feedinSerialData();
}


/* Attach callbacks for CmdMessenger commands */
void attach_callbacks(void) { 
    cmdMessenger.attach(get_valsC, on_get_valsC);
    cmdMessenger.attach(get_valsS, on_get_valsS);
}

int invv(int input) {
return 4095-input;
}

Python code:

Code:
# ------------------------------------------------------------------------------
# Python program using the library to interface with the arduino sketch above.
# ------------------------------------------------------------------------------

import PyCmdMessenger
import time
import csv
import sys
import random

# Initialize an ArduinoBoard instance.  This is where you specify baud rate and
# serial timeout.  If you are using a non ATmega328 board, you might also need
# to set the data sizes (bytes for integers, longs, floats, and doubles).  
arduino = PyCmdMessenger.ArduinoBoard("/dev/cu.usbmodem3071981",baud_rate=115200,int_bytes=8, long_bytes=8, float_bytes=8, double_bytes=8)

# List of commands and their associated argument formats. These must be in the
# same order as in the sketch.
commands = [["get_valsS","iiiiiii"],["get_valsC","iiiiiii"]]

# Initialize the messenger
c = PyCmdMessenger.CmdMessenger(arduino,commands)

c.send("get_valsS", 44, 44, 44, 44, 44, 44, 1)
c.send("get_valsC", 100, 100, 100, 100, 100, 100, 100)
 
it may not help, but for the Teensy 3.6 i would change
Code:
arduino = PyCmdMessenger.ArduinoBoard("/dev/cu.usbmodem3071981",baud_rate=115200,int_bytes=8, long_bytes=8, float_bytes=8, double_bytes=8)
to
Code:
arduino = PyCmdMessenger.ArduinoBoard("/dev/cu.usbmodem3071981",baud_rate=115200,int_bytes=4, long_bytes=4, float_bytes=4, double_bytes=8)
 
44 is ascii ',' and 59 is ascii ';'
So I think those binary numbers are messing up the formatting or whatever this string in the object creation does:
CmdMessenger cmdMessenger = CmdMessenger(Serial,',',';','/'); ( this contains commas and semicolons )

My suspicion is that cmdMessager only handles ascii data transfer, but that is just a guess.
Edit: I now see the readBinArg commands so this suspicion appears to be incorrect, but I still bet it has something to do with commas and semicolons.
 
44 is ascii ',' and 59 is ascii ';'
So I think those binary numbers are messing up the formatting or whatever this string in the object creation does:
CmdMessenger cmdMessenger = CmdMessenger(Serial,',',';','/'); ( this contains commas and semicolons )

My suspicion is that cmdMessager only handles ascii data transfer, but that is just a guess.
Edit: I now see the readBinArg commands so this suspicion appears to be incorrect, but I still bet it has something to do with commas and semicolons.

Holy cow, that is probably related. I'll see if it has to do with the object creation. Thanks! :cool:
 
Can you try the example https://github.com/harmsm/PyCmdMessenger/tree/master/examples/arduino

Example worked for me with Teensy 3.6 and Mac python3 using
Code:
# ------------------------------------------------------------------------------
# Python program using the library to interface with the arduino sketch above.
# ------------------------------------------------------------------------------

import PyCmdMessenger

# Initialize an ArduinoBoard instance.  This is where you specify baud rate and
# serial timeout.  If you are using a non ATmega328 board, you might also need
# to set the data sizes (bytes for integers, longs, floats, and doubles).
arduino = PyCmdMessenger.ArduinoBoard("/dev/cu.usbmodem2056231",baud_rate=9600,int_bytes=4, long_bytes=4, float_bytes=4, double_bytes=8)

# List of commands and their associated argument formats. These must be in the
# same order as in the sketch.
commands = [["who_are_you",""],
            ["my_name_is","s"],
            ["sum_two_ints","ii"],
            ["sum_is","i"],
            ["error","s"]]

# Initialize the messenger
c = PyCmdMessenger.CmdMessenger(arduino,commands)

# Send
c.send("who_are_you")
# Receive. Should give ["my_name_is",["Bob"],TIME_RECIEVED]
msg = c.receive()
print(msg)

# Send with multiple parameters
c.send("sum_two_ints",4,1)
msg = c.receive()

# should give ["sum_is",[5],TIME_RECEIVED]
print(msg)
you'll need to edit the mac serial port in the python above. Notice i addjusted the data type sizes to match Teensy 3*

output
Code:
python3 tom.py
Connecting to arduino on /dev/cu.usbmodem2056231... done.
('my_name_is', ['Bob'], 1536407310.194747)
('sum_is', [5], 1536407310.19983)
example sends both strings and binary data.
 
44 is ascii ',' and 59 is ascii ';'
So I think those binary numbers are messing up the formatting or whatever this string in the object creation does:
CmdMessenger cmdMessenger = CmdMessenger(Serial,',',';','/'); ( this contains commas and semicolons )

My suspicion is that cmdMessager only handles ascii data transfer, but that is just a guess.
Edit: I now see the readBinArg commands so this suspicion appears to be incorrect, but I still bet it has something to do with commas and semicolons.

I think it might be related to having the wrong size for int's (needs to be 4 for Teensy 3), but python is configured to send 8 bytes, so teensy reads 4 binary bytes, then starts looking for separator, and it finds the data 44 or 59 ... try fixing the data sizes as noted in post #2
 
Alas, even with proper byte sizes, i experienced the same failures for data values 44,44, ..., using ubuntu python3 ... seems like a bug in the library for binary data.???

i added an OLED display to T3.6 to examine data reccieved from python
 
In my original Arduino sketch, the line

Code:
CmdMessenger cmdMessenger = CmdMessenger(Serial, ',', '; ','/');

defines , (comma) as field separator, ; (semicolon) as command separator, and / (slash) as escape separator.

When these are not passed, CmdMessenger, assumes the default, which happen to be the ones above.

I suspect that a good test would be to set the CmdMessenger with different separators, e.g.

! (33)
( (40)

and see if sending these numbers will cause the same issue.

I will try this next, to see what's going on.
 
Alas, even with proper byte sizes, i experienced the same failures for data values 44,44, ..., using ubuntu python3 ... seems like a bug in the library for binary data.???

i added an OLED display to T3.6 to examine data reccieved from python

@manitou: What is the data received when 44 is sent in?

And thanks for taking a look at this.
 
When 44's are sent, the read doesn't complete, nothing printed ??

i fear the problem is with python version it probably needs to "escape" separator bytes in the binary data stream. you could just send ascii/string values to avoid the python binary stream problems with whatever the field-separator byte is ...
 
I understood the original problem to be that it only fails with 6 44's sent but will work with 5 44's and two 1's as data.
If the thinking it is a library problem, which: the sending side or the receiving side. So divide and conquer strategy.
Replace the teensy program with a program that just prints out the bytes received and observe the passing and failing data streams.
With 44 as the field separator, 59 as the command separator and 47 as the escape character you should see something like:

command field whatever it is 44 00 00 00 47 44 44 00 00 00 47 44 44 00 00 00 47 44 44 00 00 00 47 44 44 00 00 00 47 44 44 00 00 00 47 44 44 00 00 00 01 59
 
With the python sending
c.send("get_valsS", 44, 44, 44, 44, 44, 44, 1)
c.send("get_valsC", 100, 100, 100, 100, 100, 100, 100)

here's a hex dump of bytes (122) received by teensy
Code:
30 2c 2f 2c 2f 00 2f 00 2f 00 2c 2f 2c 2f 00 2f |0,/,/?/?/?,/,/?/|
00 2f 00 2c 2f 2c 2f 00 2f 00 2f 00 2c 2f 2c 2f |?/?,/,/?/?/?,/,/|
00 2f 00 2f 00 2c 2f 2c 2f 00 2f 00 2f 00 2c 2f |?/?/?,/,/?/?/?,/|
2c 2f 00 2f 00 2f 00 2c 01 2f 00 2f 00 2f 00 3b |,/?/?/?,?/?/?/?;|
31 2c 64 2f 00 2f 00 2f 00 2c 64 2f 00 2f 00 2f |1,d/?/?/?,d/?/?/|
00 2c 64 2f 00 2f 00 2f 00 2c 64 2f 00 2f 00 2f |?,d/?/?/?,d/?/?/|
00 2c 64 2f 00 2f 00 2f 00 2c 64 2f 00 2f 00 2f |?,d/?/?/?,d/?/?/|
00 2c 64 2f 00 2f 00 2f 00 3b 00 00 00 00 00 00 |?,d/?/?/?;??????|

the python is escaping the 0's and the embedded 44. = 0x2c = ','
escape character (/) is 0x2f
seems like that ought to work??

44, 44, 44, 44, 44, 44, 257 works??
 
Last edited:
Here's what I looked at:

My send command is

Code:
c.send("get_valsS", 44, 44, 44, 44, 44, 44, 44)
c.send("get_valsS", 20, 20, 20, 20, 20, 20, 20)
c.send("get_valsS", 200, 200, 200, 200, 200, 200, 200)

PyCmdMessenger turns this into:

Code:
b'0,/,/\x00/\x00/\x00,/,/\x00/\x00/\x00,/,/\x00/\x00/\x00,/,/\x00/\x00/\x00,/,/\x00/\x00/\x00,/,/\x00/\x00/\x00,/,/\x00/\x00/\x00;'
b'0,\x14/\x00/\x00/\x00,\x14/\x00/\x00/\x00,\x14/\x00/\x00/\x00,\x14/\x00/\x00/\x00,\x14/\x00/\x00/\x00,\x14/\x00/\x00/\x00,\x14/\x00/\x00/\x00;'
b'0,\xc8/\x00/\x00/\x00,\xc8/\x00/\x00/\x00,\xc8/\x00/\x00/\x00,\xc8/\x00/\x00/\x00,\xc8/\x00/\x00/\x00,\xc8/\x00/\x00/\x00,\xc8/\x00/\x00/\x00;'

in line 170 in PyCmdMessenger.py: https://github.com/harmsm/PyCmdMessenger/blob/master/PyCmdMessenger/PyCmdMessenger.py



I think this is all a very big mystery to me.
 
>>> seems like that ought to work??

Yes, it looks like the python side encodes the two messages consistently sending the least significant byte first. (it seems strange to me it escapes the nulls but it is consistent ) I would say the issue is with the arduino side. What arduino library are you using?

And are you saying the data 44, 44, 44, 44, 44, 44, 257 works ok with the cmdmessenger code? This seems like a very subtle bug.
 
Doesn't work:

c.send("get_valsS", 44, 44, 44, 44, 44, 44, 44) -> b'0,/,/\x00/\x00/\x00,/,/\x00/\x00/\x00,/,/\x00/\x00/\x00,/,/\x00/\x00/\x00,/,/\x00/\x00/\x00,/,/\x00/\x00/\x00,/,/\x00/\x00/\x00;'
c.send("get_valsS", 44, 44, 44, 44, 44, 44, 59) -> b'0,/,/\x00/\x00/\x00,/,/\x00/\x00/\x00,/,/\x00/\x00/\x00,/,/\x00/\x00/\x00,/,/\x00/\x00/\x00,/,/\x00/\x00/\x00,/;/\x00/\x00/\x00;'

Works:

c.send("get_valsS", 44, 44, 44, 44, 44, 44, 257) -> b'0,/,/\x00/\x00/\x00,/,/\x00/\x00/\x00,/,/\x00/\x00/\x00,/,/\x00/\x00/\x00,/,/\x00/\x00/\x00,/,/\x00/\x00/\x00,\x01\x01/\x00/\x00;'
c.send("get_valsS", 59, 59, 59, 59, 257, 257, 257) -> b'0,/;/\x00/\x00/\x00,/;/\x00/\x00/\x00,/;/\x00/\x00/\x00,/;/\x00/\x00/\x00,\x01\x01/\x00/\x00,\x01\x01/\x00/\x00,\x01\x01/\x00/\x00;'

I can only imagine the parser somehow gets tripped up.
 
if you look at my hex dump, the first command line is 64 bytes long. The FIX is in CmdMessenger.h change
#define MESSENGERBUFFERSIZE 64
to
#define MESSENGERBUFFERSIZE 128
 
Status
Not open for further replies.
Back
Top