Teensy 4.1 -- Serial.readStringUntil problem

Status
Not open for further replies.
I

ipeerbhai

Member
Howdy All,

I wrote an Arduino program and tested it on other Arduino boards. Now, I'm porting that working program to the Teensy 4.1. I'm running into odd serial port behavior that I don't understand.
Here's the function ( tested and works on both Arduino Uno and Sparkfun Artemis ).

This code seems to crash the serial port -- the Serial.println("ACK>") never gets called if I send a string like "abcdefg~" -- but if I send "abcdefgh", it does.
In both cases, the board stops sending any serial.Print strings after the readStringUntil function is called, and the board seems to be crashed.

I've tried replacing the readStringUntil function with a byte-by-byte read loop, and even that seems to crash the board. What am I doing wrong?

Code: 
//-----------------------------------------------------------------------------------------------------------------------------
// Function:
//  ReadSerialPortData keeps reading the port until it sees a ~ character, appends what it reads to the command
// buffer.
boolean CommandManager::ReadSerialPortData()
{
    boolean returnCode = false;
    int serialBytesRecieved = 0;
    serialBytesRecieved = Serial.available();
    Serial.print("Received data bytes::");
    Serial.println(serialBytesRecieved);

    // we can recieve data in fragments, and need to coalate it together.

    if (serialBytesRecieved > 0)
    {
        returnCode = true;
        Serial.println("ACK>");

        // we got a valid buffer but it's likely fragmented  Keep reading until we see the ~ character.
        m_commandBuffer += Serial.readStringUntil('~');
    }
    return (returnCode);
}
 
Last edited by a moderator:
Note: This one runs on T4.1
Code: 
//-----------------------------------------------------------------------------------------------------------------------------
// Function:
//  ReadSerialPortData keeps reading the port until it sees a ~ character, appends what it reads to the command
// buffer.
String m_commandBuffer;

boolean ReadSerialPortData()
{
  boolean returnCode = false;
  int serialBytesRecieved = 0;
  serialBytesRecieved = Serial.available();
  if (serialBytesRecieved > 0)
  {
    returnCode = true;
    Serial.println("ACK>");

    // we got a valid buffer but it's likely fragmented  Keep reading until we see the ~ character.
    m_commandBuffer += Serial.readStringUntil('~');
  }
  return (returnCode);
}

void setup() {
  while (!Serial && millis() < 4000) ;
  Serial.begin(115200);
  m_commandBuffer = "";
}

void loop() {
  if (ReadSerialPortData()) {
    Serial.print("Received:");
    Serial.println(m_commandBuffer);
    m_commandBuffer = "";
  }
}
With your function you had some Serial.print statements even when count was 0, which if you are calling constantly could overflow your Serial monitor program... As T4s can run serial real fast...
 
Agreed, we need a complete (and hopefully simple) program to help in any useful way. This function without the related code that manages m_commandBuffer is not good enough.

But just in case, I did a simple test to confirm Serial.readStringUntil() really is working.

Code: 
void setup() {
  Serial.setTimeout(10000);
}

void loop() {
  Serial.clearReadError();
  String s = Serial.readStringUntil('~');
  if (Serial.getReadError()) {
    //Serial.println("error or timeout waiting");
  } else {
    Serial.print("got string: ");
    Serial.println(s);
  }
}

I ran this on Teensy 4.1 and sent "abcdefg~" 3 times. I believe you can see in this screenshot that Teensy 4.1 kept running and was able to print the 2nd and 3rd time (and considers the newline received from the end of the last transmission to be the first character of the next line)

sc.png
 
The caller function has a delay(200); -- I added the prints for debugging. For a while, Serial.available was always returning 0. I made some random, unrelated, change, and that line started working without reason.
 
PaulStoffregen said:
Agreed, we need a complete (and hopefully simple) program to help in any useful way. This function without the related code that manages m_commandBuffer is not good enough.

But just in case, I did a simple test to confirm Serial.readStringUntil() really is working.

Code: 
void setup() {
  Serial.setTimeout(10000);
}

void loop() {
  Serial.clearReadError();
  String s = Serial.readStringUntil('~');
  if (Serial.getReadError()) {
    //Serial.println("error or timeout waiting");
  } else {
    Serial.print("got string: ");
    Serial.println(s);
  }
}

I ran this on Teensy 4.1 and sent "abcdefg~" 3 times. I believe you can see in this screenshot that Teensy 4.1 kept running and was able to print the 2nd and 3rd time (and considers the newline received from the end of the last transmission to be the first character of the next line)

View attachment 20891
Click to expand...

The caller function has a delay(200); The print is just there for debugging. This function continues to not work on Teensy, but work on Artemis and Arduino, and I have no idea why. It also works in isolation -- just not within the program. Once the chip crashes, I can no longer debug via prints. I do have a JTAG debugger available, but don't know how to connect it for the Teensy 4.1.
 
Sorry, I am a bit lazy here...

But in your sources, which by names on github is done on platform IO?

Setup is:
Code: 
void setup()
{
  // put your setup code here, to run once:
  pinMode(ledPin, OUTPUT);
  g_safetySystem = new SafetyManager(&g_TimerCounter);
  g_commandSystem = new CommandManager(g_RobotMotors, &g_TimerCounter, &g_PrevTimerCounter, g_sensorSystem, g_safetySystem);
  Serial.begin(500000);
  Serial.println("Ready>");
  RequestConfiguration();

  // Wait until we get a configuration before we do the rest
  while (!g_safetySystem->IsConfigured())
  {
    g_commandSystem->Dispatch();
    delay(200);
  } 
  
  // start the timer.
  g_mainTimer.begin(Dispatch, 1);
}
I don't see any call to ReadSerialPortData
Ok it is called in the Dispatch method of command...


As Paul and I also tried out the calls. Mine was more or less yours, other than being outside of the class and I removed your two outside prints that clog up the Terminal monitor waiting for data... But as you do it very 200ms would only be 5 per second, so Terminal monitor should work.

SO than it still gets me back to wondering how you are outputting data to this sketch? Through PlatfromIO? Does it work with Serial Monitor of Arduino?

Also in cases like this I would probably build a version of it that builds on Arduino and see if it works on it... Maybe it is a PlatformIO issue?

Sorry maybe not much help.
 
KurtE said:
Sorry, I am a bit lazy here...

But in your sources, which by names on github is done on platform IO?

Setup is:
Code: 
void setup()
{
  // put your setup code here, to run once:
  pinMode(ledPin, OUTPUT);
  g_safetySystem = new SafetyManager(&g_TimerCounter);
  g_commandSystem = new CommandManager(g_RobotMotors, &g_TimerCounter, &g_PrevTimerCounter, g_sensorSystem, g_safetySystem);
  Serial.begin(500000);
  Serial.println("Ready>");
  RequestConfiguration();

  // Wait until we get a configuration before we do the rest
  while (!g_safetySystem->IsConfigured())
  {
    g_commandSystem->Dispatch();
    delay(200);
  } 
  
  // start the timer.
  g_mainTimer.begin(Dispatch, 1);
}
I don't see any call to ReadSerialPortData
Ok it is called in the Dispatch method of command...


As Paul and I also tried out the calls. Mine was more or less yours, other than being outside of the class and I removed your two outside prints that clog up the Terminal monitor waiting for data... But as you do it very 200ms would only be 5 per second, so Terminal monitor should work.

SO than it still gets me back to wondering how you are outputting data to this sketch? Through PlatfromIO? Does it work with Serial Monitor of Arduino?

Also in cases like this I would probably build a version of it that builds on Arduino and see if it works on it... Maybe it is a PlatformIO issue?

Sorry maybe not much help.
Click to expand...

Thanks for trying. The prints are there for debugging, as I'm trying to figure out where this dies. After a bunch of random prints, I found it dies in this function, specifically.
Yes, it dies the same if I don't use platformIO. PlatformIO is just a build wrapper to make using VSCode easier. With PlatformIO and a cheap JTAG debugger off Aliexpress, you can do inline debugging on some boards right in VSCode. Some folks figured out how to do that on Teensy 4.0 -- but 4.1 changed the pins, and I don't know how to do that anymore.

The serial monitor in Arduino's IDE is how I test the code out.

The whole program works just fine on other Arduino platforms -- An Artemis specific variation of this program exists, and other than some mild refactoring, ambiq timer calls, and a call to ambiq's fast GPIO API in the sensor manager, is essentially the same as this one. That version works correctly.

This issue seems specific to the Teensy4.1, and specific to C++. If I use this code in isolated sketches in C, it just works. As you can see from the github, this code is really straight-forward. And since it kills Serial -- I have no idea how to debug this further.
 
Any chance I could talk you into condensing this down to a small example in a single .ino file which reproduces the problem when used with the Arduino IDE?

Sorry, I just can't put limited dev time into more investigation with the code as-is.
 
I try to help as I can... May take another look, but it is always easier to get help on things like, this if you had a self contained Arduino sketch... I do have VisualMicro setup and the like, and am working with the GDBStuff stuff that was setup recently...

And yes I know these types of issues can be frustrating. Often times it ends up being something like someone overwrite memory. or something similar... Sometimes it is something like something is set on an interrupt and tested in main code and the variable is NOT marked volatile and as such the compiler does not reload and you hang... And of the course there is always bugs you may happen to run into.

But in the early days of T4 beta stuff, we did not have any USB Serial support working so debugging was done using a different hardware Serial port. Example Serial4 is setup in kernel (under #ifdef) to do some low level debug outputs and the like. So I would hook up a USB to serial device (like another Teensy) to which ever serial port I wish to use and then debug that way.
 
PaulStoffregen said:
Any chance I could talk you into condensing this down to a small example in a single .ino file which reproduces the problem when used with the Arduino IDE?

Sorry, I just can't put limited dev time into more investigation with the code as-is.
Click to expand...

That would take a bit of time, and not really simplify much beyond this. If I can attach a JTAG debugger, then there's a chance I can figure it out myself. Is there a diagram of how to connect a segger for the 4.1?
 
ipeerbhai said:
That would take a bit of time, and not really simplify much beyond this.
Click to expand...

You could put it all into 1 file and remove all the C++ class stuff, so it's just simple code in setup() and loop(). Ideally this should be a short program which can be just copied into Arduino, like the code I posted in msg #4.


Is there a diagram of how to connect a segger for the 4.1?
Click to expand...

No, there is not. The JTAG pins are not exposed.
 
Note: I have it all as one Sketch folder. I downloaded the folder, renamed the main.cpp to USB_Hang.ino in the right name folder and have it building and dying.

So far I know that it is not hanging/dying where you think it is...
I updated your code:
Code: 
boolean CommandManager::ReadSerialPortData()
{
  boolean returnCode = false;
  int serialBytesRecieved = 0;
  serialBytesRecieved = Serial.available();
  //    Serial.print("Received data bytes::");
  //    Serial.println(serialBytesRecieved);

  // we recieve data in fragments, but need to coalate it together.

  if (serialBytesRecieved > 0)
  {
    returnCode = true;
    Serial.println("ACK>");Serial.flush();

    // we got a valid buffer but it's likely fragmented.
    m_commandBuffer += Serial.readStringUntil('~');
    Serial.println("after Serial.readStringUntil('~')"); Serial.flush();
  }
  return (returnCode);
}
One debug secret, when thing are hung/die like this, I often then do Serial.flush() after debug statements to force the serial to be done before continuing...

I also added one to start of CommandManager...
Code: 
void CommandManager::ProcessCommandBuffer()
{
  Serial.println("Call ProcessCommandBuffer()"); Serial.flush();
  // let's make a bunch of string object we can use to parse.
  String *subs = new String[SUBSTRINGS_LIMIT];
  for ( int i = 0; i < SUBSTRINGS_LIMIT; i++)
Ran the program and typed in something like ABCD~

And debug terminal output:
Code: 
ACK>

after Serial.readStringUntil('~')

Call ProcessCommandBuffer()
I may make a couple more outputs to localize, but that should give some more hints.
 
You are calling through a NULL ptr...

Code: 
void CommandManager::ProcessCommandBuffer()
{
  Serial.println("Call ProcessCommandBuffer()"); Serial.flush();
  // let's make a bunch of string object we can use to parse.
  String *subs = new String[SUBSTRINGS_LIMIT];
  for ( int i = 0; i < SUBSTRINGS_LIMIT; i++)
  {
    subs[i] = String("");
  }
  Serial.println("After String Subs..."); Serial.flush();
  //   ResetWatchDog(); // we know the PC is alive, it just said something.
  String Text = String(""); // our return string.
  Serial.printf("Reset WatchDog %x\n", (uint32_t)m_safetyManager); Serial.flush();
[COLOR="#FF0000"]  m_safetyManager->ResetWatchDog();[/COLOR]
  Serial.println("After watchdog"); Serial.flush();
  switch (m_commandBuffer[0])
  {
Code: 
ACK>

after Serial.readStringUntil('~')

Call ProcessCommandBuffer()

After String Subs...

Reset WatchDog 0
 
KurtE said:
You are calling through a NULL ptr...
Click to expand...

Ah, that makes sense.

Those other boards don't have a MPU that generates a program-stopping fault for illegal memory access (or if Artemis has one, I'm pretty sure Sparkfun wouldn't bother to use it like we do with Teensy 4). So they would execute whatever instructions are in that memory and if not too much goes wrong before a return instruction executes, it probably returns without any indication how wrong things went for a brief moment.
 
KurtE said:
You are calling through a NULL ptr...

Code: 
void CommandManager::ProcessCommandBuffer()
{
  Serial.println("Call ProcessCommandBuffer()"); Serial.flush();
  // let's make a bunch of string object we can use to parse.
  String *subs = new String[SUBSTRINGS_LIMIT];
  for ( int i = 0; i < SUBSTRINGS_LIMIT; i++)
  {
    subs[i] = String("");
  }
  Serial.println("After String Subs..."); Serial.flush();
  //   ResetWatchDog(); // we know the PC is alive, it just said something.
  String Text = String(""); // our return string.
  Serial.printf("Reset WatchDog %x\n", (uint32_t)m_safetyManager); Serial.flush();
[COLOR="#FF0000"]  m_safetyManager->ResetWatchDog();[/COLOR]
  Serial.println("After watchdog"); Serial.flush();
  switch (m_commandBuffer[0])
  {
Code: 
ACK>

after Serial.readStringUntil('~')

Call ProcessCommandBuffer()

After String Subs...

Reset WatchDog 0
Click to expand...
Thanks to all for the hints! When prints stopped working in ReadSerialPortData -- I got totally stuck. I would never have thought of flushing the buffer there to get further down the rabbit hole -- so I totally appreciate the assistance from all!
 
Why aren't the JTAG pins exposed? What about the SWD interface?
What can be done when you're getting random crashes? Which, BTW, sounds like uninitialized memory. Is there a place where memory can be wiped to zeroes as the part is coming up?
madGambol
 
There is a recent and active thread for GDB software debugging support that works on Teensy 4.x (and some others).

It will halt on faults and allow code examination as I saw it. Teensy CORES does have fault enumeration code that ships disabled after Beta - it is a crude dump - but does present when a Fault is detected. The above GDB interface when enabled is worlds better.

The Teensy is unique with the Bootloader chip using the debug lines to perform its magic. Teensy is designed to work with Arduino, and they have not yet defined a Debug platform - Paul has noted when Arduino does that Teensy will follow that (when/where possible).

Here is the link :: forum.pjrc.com/threads/61373-Using-GDB-with-Teensy-without-hardware-debugger-first-Beta
 
Status
Not open for further replies.
Back
Top