How to read bytes transmissions of various lengths ?

Status
Not open for further replies.

laptophead

Well-known member
Using Teensy 3.5 Serial1. Are all the Serial ports buffered? Is there an advantage in using 1,2.3 or another?

I am talking to motor with a built in encoder. I am sending commands and the motor controller replies in transmissions varying from 5 to 18 hexadecimal numbers, each made of 2 digits.

I found this method to work well:

HTML:
while (Serial1.available() >=7 ) // serial will wait for "Bytes_In_A"  before
  {
    for (int n = 0; n < 8; n++)
    { in_bytes[n] = Serial1.read();
    } // end of for loop

It is populating my array accurately.

The problem is that I get a 18 bytes, or 12 bytes or 10 bytes transmission and that is not working since I am aways waiting for 7 bytes.
If I always wait for 18 bytes, the shorter transmission don't get logged....

So ideally I would:
- Count the bytes
- execute the function to populate my array.
- clean the buffer for the next show.

I am not that sharp at this, please help.
(please note, each byte comes in a new line)

Thanks

Mitch
 
each byte comes in a new line
Each byte, or each response?
Which motor is it and can you give examples of the response?

If it sends each response with a CR and/or LF at the end, you can just store the response until the CR/LF occurs and then process it.

Pete
 
Serial1 and Serial2 have more hardware FIFO support so, if you receive or send lots of data, these will typically cause you to have to process fewer interrupts and as such are less likely to lose data, if your code can not process interrupt requests fast enough...

Again hard to say anything specific, but I often setup a sort of state-table for processing data. For example some Dynamixel Servo data might have the format something like:
<FF><FF><ID><CNT LOW><CNT HIGH><INST><data....><CR LOW><CR HIGH> (as finary data...

So code will simply start of in a state looking for an FF... If it receives a byte that is not FF it stays in this state, else goes into state looking for 2nd FF. If next byte is not FF goes back to first state else go for looking for ID... IF ID byte comes in and it is an FF (not allowed), it stays in this state, else it looks for next two bytes being length...

It may also also look at time between bytes and if a timeout go back to first state...

But obviously the easiest way to do this, is to maybe use a library that already is setup for your motor/controller. For example there is a library for RoboClaw controllers...
 
Using Teensy 3.5 Serial1. Are all the Serial ports buffered? Is there an advantage in using 1,2.3 or another?

I am talking to motor with a built in encoder. I am sending commands and the motor controller replies in transmissions varying from 5 to 18 hexadecimal numbers, each made of 2 digits.

I found this method to work well:

HTML:
while (Serial1.available() >=7 ) // serial will wait for "Bytes_In_A"  before
  {
    for (int n = 0; n < 8; n++)
    { in_bytes[n] = Serial1.read();
    } // end of for loop

It is populating my array accurately.

The problem is that I get a 18 bytes, or 12 bytes or 10 bytes transmission and that is not working since I am aways waiting for 7 bytes.
If I always wait for 18 bytes, the shorter transmission don't get logged....

So ideally I would:
- Count the bytes
- execute the function to populate my array.
- clean the buffer for the next show.

I am not that sharp at this, please help.
(please note, each byte comes in a new line)

Thanks

Mitch

IIRC Serial 1 and 2 have 8 byte FIFO hardware support - and Serial3 is a single byte.

Per post #2 - if there is a 'new line' terminating each set that would be a good character to break on and confirm.

Also the outer while() may be the trouble? It won't stop after expected bytes, but return to the inner for(). Perhaps:
Code:
[B]if (Serial1.available() >=7 )[/B] // serial will wait for "Bytes_In_A"  before
  {
    for (int n = 0; n < 8; n++)
    { in_bytes[n] = Serial1.read();
    } // end of for loop
 
thanks everyone,

Changing the while to if did not work.

The motor data sheet is at
http://dow.gyems.cn/RMD-servo motor control protocol (RS485) V1.6.pdf

Here is a sample transmission of 18 bytes.

I got it with:
if (Serial1.available()>16) { // this is necessary so all bytes get read and dont linger in the buffer
byte inByte = Serial1.read();
Serial.print("INCOMING:");
Serial.println(inByte, HEX);
}

Serial Output :

INCOMING:3E


INCOMING:A4


INCOMING:1
INCOMING:7
INCOMING:EA
INCOMING:C9
INCOMING:0
INCOMING:0
INCOMING:0
INCOMING:0
INCOMING:5
INCOMING:0
INCOMING:CE
INCOMING:3E
INCOMING:A4
INCOMING:2
INCOMING:7
INCOMING:EB
INCOMING:C9
INCOMING:0
INCOMING:0
INCOMING:F8
INCOMING:FF


There is no Arduino library or samples of how to do this, (Chinese tech support...)

But the product is good.

Please help
 
The pdf has a description of the protocol you are trying to read. The first byte that the motor sends is always the "head byte" which is 0x3E. You'll see that your example has one of these in it. The fourth byte of the message is the message length, in your example, it is 7. I haven't figured out the checksum yet but the basic process of reading this is to look for the 0x3E and read the following four bytes. The data length byte then tells you how many more bytes are going to follow in the data.

BTW. The T3.5 should have no problems handling the serial at 115,000 baud on any port.

Pete
 
Pete,

I have a lot more than seven bytes in my example, so is the fourth byte accurate? I don't see how.

So what to do?
read the 4 bytes always, and then how do I read the remaining bytes?
 
The checksum appears to be a simple sum of the bytes, 3E + A4 + 1 + 7 = EA, it could handle rollover in different ways though and since I don’t have an example to go off of I can just say it most likely adds the rollover back to the original, F0 + F0 = 1E0 -> 1 + E0 = E1, which is what I’ve seen used before.
 
Still, How do I solve my problem?
How to I receive transmissions of various lengths?

I tried just receiving the first 7 bytes. Worked fine for the first transmission , then the second transmission was all scrambled .
The remaining bytes (past 7) were assigned to in_bytes[0], in_bytes[1] etc. in a wrong way.
I wish I knew how to flush the incoming buffer after each timeout. The Serial1.flush(); command works only for the outgoing buffer.

Any ideas?
Thanks
 
Your example output is parts of two separate messages and you didn't read all of the data. The fourth byte indicates 7 data bytes but you stopped printing after the 6th byte.

Try incorporating this code into your sketch.
Code:
  char head_byte;

  // Look for the head byte 0x3E
  do {
    while(Serial1.available() == 0);
    head_byte = Serial1.read();
  } while(head_byte != 0x3E);

  // Read the next 4 header bytes
  char header[4];
  for(int i = 0; i < 4; i++) {
    while(Serial1.available() == 0);
    header[i] = Serial1.read();
  }

// At this point the checksum should be calculated to make sure that
// the header was received correctly.

  // The length of the data bytes which follow is specified
  // in header[2]
  char data_bytes[64];
  // The +1 is because the data length does NOT include the checksum
  // which we also must read.
  for(int i = 0; i < header[2]+1; i++) {
    while(Serial1.available() == 0);
    data_bytes[i] = Serial1.read();
  }

// And, again, we should check the checksum of the data bytes
// to make sure that they are correct as well.

  // Print the message for debugging.
  for(int i = 0; i < 4; i++) {
    Serial.printf("%02X ",header[i]);
  }
  Serial.printf("| ");
  for(int i = 0; i < header[2]+1; i++) {
    Serial.printf("%02X ",data_bytes[i]);
  }
  Serial.println();

I've declared data_bytes to be 64 chars so that it has room for up to 60 data bytes plus the checksum byte - plus 3 more to make sure :)
Pete
 
Last edited:
Regarding length, you don't want to read a fixed number of bytes with this since it can be variable and you want to be able to handle any data errors. So you have to start with only reading one byte to find the frame start header and if it equals 0x3E then it's safe to continue reading the remains 4 bytes of your header. Now that you have a possible header identified you want to test and make sure it's valid which is where the checksum comes into play, so you add the first 4 bytes of your header together and if it equals that 5th byte you can continue reading the rest of the message. Now the 4th byte tells you how long the data is which in your example message is 7, so at this point you can read in those 7 bytes, then there is one more byte that comes after the data in the form of the data checksum. Once you have all the data read you want to them test that against the data checksum and if it equals out then you can safely say that the message is valid and do what you need to with it.

Something like this should also work just as another example:
Code:
  while (Serial1.available()){
    uint8_t head = Serial1.read(); //Read one byte
    uint8_t command;
    uint8_t id;
    uint8_t data_length;
    uint8_t header_checksum;
    uint8_t data[60];
    uint8_t data_checksum;
    if(head == 0x3E){ //Test if byte is header start
      command = Serial1.read(); //Read remaing header
      id = Serial1.read();
      data_length = Serial1.read();
      header_checksum = Serial1.read();
      uint16_t temp_checksum = head + command + id + data_length; //Add temp checksum
      temp_checksum = (temp_checksum & 0xFF) + (temp_checksum >> 8); //Add temp checksum rollover
      if(temp_checksum == header_checksum){ //Test checksum
        uint16_t data_checksum_temp = 0;
        for(uint8_t i = 0; i < data_length; i++){ //Read data and add temp checksum
          data[i] = Serial1.read();
          data_checksum_temp += data[i];
        }
        data_checksum_temp = (data_checksum_temp & 0xFF) + (data_checksum_temp >> 8); //Add temp checksum rollover
        data_checksum = Serial1.read(); //Read data checksum
        if(data_checksum_temp == data_checksum){ //Test data checksum
          //Data is valid
        }
      }
    }
  }
 
Last edited:
VJ
Thanks a lot, so elaborate...

So I tried to put it to work:

Code:
while (Serial1.available()){
    uint8_t head = Serial1.read(); //Read one byte
    uint8_t command;
    uint8_t id;
    
    uint8_t data_length;
    uint8_t header_checksum;
    uint8_t data[60];
    uint8_t data_checksum;
    if(head == 0x3E){ //Test if byte is header start
      command = Serial1.read(); //Read remaing header
      id = Serial1.read();
      Serial.println (id);
      data_length = Serial1.read();
      uint8_t checksum = Serial1.read();
      uint16_t temp_checksum = head + command + id + data_length; //Add temp checksum
      temp_checksum = (temp_checksum & 0xFF) + (temp_checksum >> 8); //Add temp checksum rollover
      if(head + command + id + data_length == header_checksum){ //Test checksum
        uint8_t data_checksum_temp = 0;
        for(uint8_t i = 0; i < data_length; i++){ //Read data and add temp checksum
          data[i] = Serial1.read();
          data_checksum_temp += data[i];
        }
        data_checksum_temp = (data_checksum_temp & 0xFF) + (data_checksum_temp >> 8); //Add temp checksum rollover
        data_checksum = Serial1.read(); //Read data checksum
        if(data_checksum_temp == data_checksum){ //Test data checksum
          //Data is valid and can be used here
         // Serial.println (id);
          Serial.println (data [2]);
          
        }
      }
    }
    
  }

The Serial.println (data [2]); from the final "if " does not print.

The Serial.println (id); above does print, and it is accurate. Meaning the data flows.
I can also see the stream on the scope.

What are we missing?

Tnks
 
Still, How do I solve my problem?
How to I receive transmissions of various lengths?

For these sorts of protocols, I usually prefer to write code that reads 1 byte and decides what to do with it based on state variables and my own buffer to hold the largest possible incoming message. Often only 1 state variable is needed, to know the number of bytes you've stored into the buffer.

For these protocols where a reserved byte starts a new message and can never occur within the data, the very first thing to do is check if the incoming byte starts a new message.

Code:
int num_received = 0;
uint8_t mybuffer[32];

void loop() {
  if (Serial.available()) {
    uint8_t b = Serial.read();  // always read the next incoming byte
    if (b == 0x3E) {
      num_received = 0;  // if begin of message, forget everything previously received
    } else {
      mybuffer[num_received] = b;  // otherwise, put the byte into the buffer
      num_received = num_received + 1;

      // check here for a complete message
      if (num_received >= 4 and mybuffer[2] == num_received-1) {
         // check if the message is valid (known meaning, good checksum, etc)
         // actually do some thing with the message if it's good
      }

      // check if the buffer is full (shouldn't happen, right?)
      if (num_received >= 32) {
        num_received = 0;  // discard everything
      }
    }
  }
}

(I typed this code just now for this message... it's not tested in any way, so please consider is pseudo-code meant to illustrate a point, rather than known-good code)


If the code gets long & complicated, usually I'll put stuff like the checking for a valid message and the handling of various types of messages into functions.

There are a lot of other ways to solve this sort of problem. Receiving 1 byte at a time isn't the most efficient way, but it's plenty fast for enough for serial protocols where a moderate baud rate limits the maximum speed. Doing 1 byte at a time keeps things simpler, so you never need to deal with the case where you read more bytes than the current message.

I almost never use the approach of waiting for Serial.available() to be larger than some number. If it's anything more than 0, usually I read the data ASAP and get it into my own buffer.

Likewise, allocating a buffer for the maximum size message consumes more memory than other ways. But these messages are so small that Teensy has plenty of memory. Having the entire message in your own buffer, where you know the first byte in the buffer is the first byte of the message (or in the code above, the first byte after 0x3E) means you can much more easily write functions which take a buffer as an input parameter and do something with the message.
 
Likely the checksum rollover is different, I would need more examples to verify how it should be.
 
Actually try fixing the checksums so they are actually the right data type

Code:
[COLOR="#FF0000"]uint16_t[/COLOR] data_checksum_temp = 0;
 
Last edited:
Just in case this point wasn't clear, usually I prefer to have a single Serial.available() check and a single Serial.read() if anything is available, followed by code that decides what to do with that 1 byte. When another byte is available, it'll be detected and read on the next iteration.

(Again, this advice is for moderate speed serial protocols with small messages. Handling very high speed data is a different topic.)

I almost always avoid code like in msg #11 and #12 where a second Serial.read() is present in the code. While this sort of thing can be done, it's usually error prone. I prefer to avoid it. But if you really want to do that, consider the many ways the code in msg #11 and #12 could fail. First, Serial.available() isn't being checked again. So if the first Serial.available() was only 1 byte, then the rest of that code will read -1 (indicating no data) and use it as it if was actual 0xFF bytes received. Likewise, the rest of that code doesn't check each byte to see if it's 0x3E. Normally you shouldn't ever receive another 0x3E in those bytes, so that might work in all normal cases. But it could fail quite badly if in anomalous cases, like noise or a partial message. You can of course handle all those cases, but it makes the more-than-one-read approach get complicated quickly.

There are many possible ways to solve this problem. Much of the choice comes down to a matter of personal style. But I can tell you from many years of experience that the approach of exactly 1 check for whether data is available, and reading only 1 byte in only 1 location and the rest of the code decides what the proper course of action is for that 1 byte keeps things simple and makes the job of robust handling of errors and wrong data much simpler. Admittedly my opinion is biased, but I would strongly suggest you adopt that code structure. It may require a little more effort up front, but in the long run I believe you'll save yourself a lot of trouble.
 
Last edited:
VJ
I implemented the
uint16_t data_checksum_temp = 0;

But still no reading from the last if,
By the way, I am running at (115200). Is that relevant?

We still have a bug?
 
Like I said the checksum is probably being calculated wrong because I have no point of reference to go on as to how it handles rollover. If you can provide some more examples like you did in message #5 I can figure out what the checksum is supposed to do with the rollover.
 
Paul
Thanks so much, your advice makes a lot of sense.

I implemented your code and I am doing a printout. I used a flipping bool in order to print just once

Code:
  if (Serial1.available()) {
    uint8_t b = Serial1.read();  // always read the next incoming byte
    if (b == 0x3E) {
      num_received = 0;  // if begin of message, forget everything previously received
      Serial.println ("new message:");
      PrintOnce = 0;
    }

    else {
      mybuffer[num_received] = b;  // otherwise, put the byte into the buffer
      num_received = num_received + 1;


      // check here for a complete message
      if (num_received >= 4 and mybuffer[2] == num_received - 1) {

        //in_bytes[n] = Serial1.read();
        // check if the message is valid (known meaning, good checksum, etc)
        // actually do some thing with the message if it's good
      }

      // check if the buffer is full (shouldn't happen, right?)
      if (num_received >= 32) {
        num_received = 0;  // discard everything
      }
    }

  }
  else if (PrintOnce == 0)
  { Serial.print(" Byte 0:");
    Serial.print(mybuffer[0], HEX);
    Serial.print(" By1:");
    Serial.print(mybuffer[1], HEX);
    Serial.print(" By2:");
    Serial.print(mybuffer[2], HEX);
    Serial.print(" By3:");
    Serial.print(mybuffer[3], HEX);
    Serial.print(" By4:");
    Serial.print(mybuffer[4], HEX);
    Serial.print(" By5:");
    Serial.print(mybuffer[5], HEX);
    Serial.print(" By6:");
    Serial.print(mybuffer[6], HEX);
    Serial.print(" By7:");
    Serial.print(mybuffer[7], HEX);

    Serial.print(" By8:");
    Serial.print(mybuffer[8], HEX);
    Serial.print(" By9:");
    Serial.print(mybuffer[9], HEX);
    Serial.print(" By10:");
    Serial.print(mybuffer[10], HEX);
    Serial.print(" By11:");
    Serial.print(mybuffer[11], HEX);
    Serial.print(" By12:");
    Serial.print(mybuffer[12], HEX);
    Serial.print(" By13:");
    Serial.print(mybuffer[13], HEX);
    Serial.print(" By14:");
    Serial.print(mybuffer[14], HEX);
    Serial.print(" By15:");
    Serial.print(mybuffer[15], HEX);
    Serial.print(" By16:");
    Serial.print(mybuffer[16], HEX);
    Serial.print(" By17:");
    Serial.println(mybuffer[17], HEX);
    PrintOnce = 1;
  }

The numbers seem accurate, ( I will have to shift them by one since I am skipping the first read). Thanks, Yes

and now the problem:
I have several motors on the same Serial bus, being distinguished by the ID bit (third bit)

I have many commands that I sent fast one after another, ( all motors go, all motors stop, etc) but I still leave some delay for them to respond,

On the scope, there is no overlapping signal.

So when I send:

Code:
if (inChar == 'g') // from the keyboard
    {
      Angle_Speed (0, 200.03, 3); // this is 0º at 200º/sec , motor 1
      delay (2);
      Angle_Speed (0, 200.03, 2); // this is 0º at 200º/sec , motor 2
      delay (2);
      

    }

I get back in the ser monitor:

g
new message:




new message:
Byte 0:A4 By1:2 By2:7 By3:EB By4:C9 By5:DE By6:FF By7:0 By8:0 By9:2 By10:0 By11:A8 By12:0 By13:0 By14:0 By15:0 By16:0 By17:0

In the scope I see 2 distinct messages back.

Any ideas?
Thanks

This is relevant since I will need to read a Buch of encoders.
 
@vjmuzik: your code is not going to work because it reads from Serial1 several times in a row without checking to make sure that there's anything in the buffer to read.

Pete
 
Paul,

I figured it , thanks for your help.

Code:
  if (Serial1.available()) {

    uint8_t b = Serial1.read();  // always read the next incoming byte
    if (b == 0x3E) {
      num_received = 0;  // if begin of message, forget everything previously received
      //Serial.println ("new message:");
      PrintOnce = 0;
      //Show_Inc_String ('A');
    }

    else {
      mybuffer[num_received+1] = b;  // otherwise, put the byte into the buffer
      num_received = num_received + 1;
     // Serial.print (num_received);
       if (num_received == 11 && mybuffer[1] == 0x90 )
        { Enc_Calc(); // function to extract encoder readings from
        }

      }  // end of "if getting bytes in the array
      
      // check if the buffer is full (shouldn't happen, right?)
      if (num_received >= 20) {
        num_received = 0;  // discard everything
        PrintOnce = 0;
        Serial.print(" Overflow ");
      } // end of else iteration
     s
     
    } // end of ser available

 else if (PrintOnce==0)
      { //Show_Inc_String ('A');
      }

This line
if (num_received >= 4 and mybuffer[2] == num_received - 1)

didn't really work out since I have short transmissions of only 5 bytes.

But I can read the encoders, and I am happy,

Thanks a million, I was stuck on this problem

Mitch
 
Status
Not open for further replies.
Back
Top