Strtok output not as expected

Status
Not open for further replies.

J-Ro

Member
Hello,

I have used the code shown below in the code snippit on several projects for when I want to change variable values, control states, etc from a PC program.

For example, from the PC based user interface I can send over serial ---> $Data02, 300 and this will set the variable "PumpTime" to 300.

As I've mentioned I've used this code on several projects successfully with Arduino Mega boards.

Recently I started another project with a Teensy 3.2 and wanted to have a PC based interface using this same method but I don't get the same results out of this code on the Teensy 3.2 it seems.

Two things 1) TempString = String(strtok(SerialLine, ",")); results in the variable TempString being empty and 2) if (TempString != NULL) gives me the error "ambiguous overload for 'operator!=' (operand types are 'String' and 'int')"


Does StrTok need to be used in a different way with Teensy?

Code:
char SerialLine [64];
char SerialByte;
int cnt = 0;
String TempString = "";

void MySerialReader () {
  while (Serial.available()) {
      if (Serial.available() > 0) {
          SerialByte= Serial.read();
          SerialLine[cnt++] = char(SerialByte);
      }
  }
  TempString = String(strtok(SerialLine, ","));

  if (TempString == "$Data02") {
      TempString = String(strtok(NULL, ","));
      if (TempString != NULL) {
          PumpTime = TempString.toFloat();
      }
  }   
  if (TempString == "$Data03") {
      TempString = String(strtok(NULL, ","));
      if (TempString != NULL) {
          SetTemp = TempString.toFloat();
      }
  }
  if (TempString == "$Data04") {
      TempString = String(strtok(NULL, ","));
      if (TempString != NULL) {
          PowerON = TempString.toFloat();
      }
  }
    if (TempString == "$Data05") {
      TempString = String(strtok(NULL, ","));
      if (TempString != NULL) {
          LightTime = TempString.toFloat();
      }
  }
    if (TempString == "$Data06") {
      TempString = String(strtok(NULL, ","));
      if (TempString != NULL) {
          TimeZoneShift = TempString.toFloat();
      }
  }  
      if (TempString == "$Data07") {
      TempString = String(strtok(NULL, ","));
      if (TempString != NULL) {
          PumpSpeed = TempString.toFloat();
      }
  } 
  memset(SerialLine, 0, sizeof(SerialLine));
  cnt = 0;
}
 
strtok is from newlib, which would mean many microcontrollers would be affected.
I assume there is a problem with String.

Here is a simpler example:

Code:
char c[] = "test=123\0";

void setup() {
  while (!Serial);
  Serial.println(strtok(c, "="));
  Serial.println(strtok(NULL, ""));

  String s;
  s = strtok(c, "=");
  Serial.println(s);
  s = strtok(NULL, "");
  Serial.println(s);
  Serial.println("end.");
}

void loop() {
}
Output:
Code:
test
123
test

end.
So, without String involved, "123" gets printed - which is correct.
But the second call, s = strtok(NULL, ""); seems not to work. I assume the first line s = strtok(c, "="); does something wrong...

The Teensy String implementation seems to have a bug.
 
Last edited:
Nope...it's not String.

It indeed seems to be newlib:
Code:
char c[] = "test=123,456,789\0";


void setup() {
  while (!Serial);
  Serial.println("Pass1 :");
  Serial.println(strtok(c, "="));
  Serial.println(strtok(NULL, ","));
  Serial.println(strtok(NULL, ","));

  Serial.println("Pass2 :");
  Serial.println(strtok(c, "="));
  Serial.println(strtok(NULL, ","));
  Serial.println(strtok(NULL, ","));
}

void loop() {}
prints:
Code:
Pass1 :
test
123
456
Pass2 :
test

So.. time to search the net if there is a bugreport already. We use a quite old version of newlib.
 
I think its a read timing problem, the void MySerialReader () only reads what is currently in the serial buffer, and if this is not a compete message it is fed to strtok and then discarded. You cannot assume a complete message has been received just because Serial.available() is positive. There must be checks for end markers in messages, or setting a timeout that ensures all characters are received before trying to interpret them, otherwise strtok will be fed incomplete strings.
 
I am definitely a newb but I don't thing mlu is right.

I have verified that the line comeig in from Serial is what I expect by printing out SerialLine. Ive even hard coded SerialLine to what I want it to be and I get the same results ---> TempString = String(strtok(SerialLine, ",")); gives blank output.
 
serial.available() is true as soon the first byte is received.
So your string may be a single char.
 
first yes, but no assurance of a complete message, can you post the code with hard coded SerialLine values.
 
So I've been trying different things since Frank's initial reply indicating that maybe String was the issue.

if I replace:

Code:
  TempString = String(strtok(SerialLine, ","));

  if (TempString == "$Data02") {
      TempString = String(strtok(NULL, ","));
      if (TempString != NULL) {
          PumpTime = TempString.toFloat();
      }
  }

with:

Code:
  if(strcmp(strtok(SerialLine, ","), "$Data02") == 0){                                                                    
          PumpTime = atof(strtok(NULL, ","));  
  }

Then I get the functionality that I am after. $Data02 is correctly identified and the value after the comma is set into PumpTime.

So I don't know if this helps prove anything or not bug related?? It certainly proves that the whole string of characters is making it through the serial ok. Maybe this is the more legitimate way of doing this operation in general? Its certainly less lines of code anyways. Would love to hear thoughts/opinions on it.
 
I tried the code from original post on a 3.0 and a 3.6.

Code:
if (TempString != NULL)
refuses to compile, so I replaced it with
Code:
if (TempString != "")

then I sprinkled the code with some writes, and tested it with input "$Data02, 300" from serial monitor, everything seems ok.

What version of Teensyduino are you running ?

Code:
char SerialLine [64];
char SerialByte;
int cnt = 0;
String TempString = "";

float PumpTime, SetTemp, PowerON, LightTime, TimeZoneShift, PumpSpeed;
void MySerialReader () {
  while (Serial.available()) {
      if (Serial.available() > 0) {
          SerialByte= Serial.read();
          SerialLine[cnt++] = char(SerialByte);
      }
  }
  if (cnt>0) {
    Serial.println(SerialLine);
  }
  TempString = String(strtok(SerialLine, ","));
  if (cnt>0) {
    Serial.println(TempString);
  }

  if (TempString == "$Data02") {
      TempString = String(strtok(NULL, ","));
      Serial.println(TempString);
      if (TempString != "") {
          PumpTime = TempString.toFloat();
          Serial.printf("PumpTime %f\n",PumpTime);
      }
  }   
  if (TempString == "$Data03") {
      TempString = String(strtok(NULL, ","));
      if (TempString != "") {
          SetTemp = TempString.toFloat();
      }
  }
  if (TempString == "$Data04") {
      TempString = String(strtok(NULL, ","));
      if (TempString != "") {
          PowerON = TempString.toFloat();
      }
  }
    if (TempString == "$Data05") {
      TempString = String(strtok(NULL, ","));
      if (TempString != "") {
          LightTime = TempString.toFloat();
      }
  }
    if (TempString == "$Data06") {
      TempString = String(strtok(NULL, ","));
      if (TempString != "") {
          TimeZoneShift = TempString.toFloat();
      }
  }  
      if (TempString == "$Data07") {
      TempString = String(strtok(NULL, ","));
      if (TempString != "") {
          PumpSpeed = TempString.toFloat();
      }
  } 
  memset(SerialLine, 0, sizeof(SerialLine));
  cnt = 0;
}

void setup() {
  // put your setup code here, to run once:
  Serial.begin(115200);
  while (!Serial);

}

void loop() {
  // put your main code here, to run repeatedly:
  MySerialReader ();
}
 
mlu -- Thanks for your effort on this.

I am running Arduino IDE 1.8.7 and Teensyduino 1.44 which both seem to be quite old
 
Status
Not open for further replies.
Back
Top