Teensy 3.6 requires 3 reprogram events to work properly

Status
Not open for further replies.

Piachnp

Member
Hello everyone,

I'm working on a robotics project using a Teensy 3.6 in Ubuntu 16.04. The teensy is connected to a custom shield to control 2 boards ("distal" and "proximal") which has been thoroughly validated.
I have been encountering some weird issues that can be hard to reproduce, even for me. But it's been happening in the same way for a while now, which is why I decided to post and see if anyone can shed some light into this behavior...

As the title says, my code works as intended, only after (usually) 3 "resets": I make changes to the code, upload the firmware using the arduino IDE and open the serial monitor to inspect the debug messages that I print there. On first upload, I get values for "Number of LEDs" and "Number of PDs" printed out that do not follow my code. Then I press the program/reset button on the physical Teensy 3.6 and the debug messages now show SOME correct values for one of the boards, but not both. Finally if I press the button once again, then the serial monitor shows all values initialized as per my code. I have attached 3 screenshots showing this happening - take into account just the reset button is being pushed in between these screenshots, no changes to the code at all.

First upload.jpg

Second Upload.jpg

Third Upload.jpg


Sometimes you get incorrect values that are zeros (as in the first screenshot) but I've also gotten random values in other occasions. Unfortunately the code base is large enough that I think attaching it is better than copying it in a /CODE block. For reference, the values shown in the 3rd screenshot are the correct ones (32 LEDs, 28 PDs). These values are constant member values declared in the derived class "Flexboardv21", which are passed directly into the base class constructor and saved into the base class member variables to hold this information (the reason for this structure is that there are other versions of the Flexboard with different number of LEDs and PDs and mappings which I have erased for clarity).

While there is use of "new" and dynamic memory allocation in the code, I still don't think that accounts for this kind of behavior.
Any ideas what could be causing this?

Thanks in advance for your time,
Pedro
 

Attachments

  • Sketch_and_libraries.zip
    9.5 KB · Views: 56
easy first guess - power and ground routing through Teensy and USB connecter to computer?

Either the USB connector is being abused for GND current and interfering with data transmission or perhaps code is being wonky and not responsive to the USB bootloader command.

Did this only start with the 'new' dynamic memory? Could that or other things that might be trashing memory be reduced/removed?
 
A quick test compile/upload a simple BLINK sketch. Assuming that runs when all the other lights and hardware are disabled - it should act normally given a good cable and healthy T_3.6

Not a Ubuntu user - not sure if there is anything in the system that might try to steal/interrupt the Teensy session. There has been an update to UDEV rules in prior months that should be in use?
 
Thanks for the suggestions! I will try tomorrow to provide power to the Teensy independently from the USB power.
I will try as well to remove the dynamic memory for now to see if that helps. I've actually reinstalled the arduino IDE and teensyduino last week and did re-download the udev rules again just in case.

BTW, the blink sketch always works at first try. Will report back soon with my findings
 
The serial is initialized as part of the finger controller class if the DEBUG flag is set to True (otherwise I wouldn't have gotten any output at all)
 
The setup function does not contain a Serial.begin(9600) statement.

Pete

To add to what Pete posted … Teensy when compiled with USB Always starts it - however depending on PC connection time and when SerMon is ready it can be in setup() and beyond before USN connect is actually established.

When USB Prints need to be seen reliably having something like this in setup() is best direction:
Code:
void setup() {
// …
	Serial.begin(115200);
	while (!Serial && millis() < 4000 );  // wait up to 4 seconds after Teensy wakes ...
	Serial.println("\n" __FILE__ " " __DATE__ " " __TIME__);
//…
}
 
If it were me, I would wonder about the work being done in the constructors...

I have been bit in the past with constructors that do much more than simply store away some settings that were passed in... Especially if the object is global...

I would probably see what happened if I moved most of the work of the initialization from FingerController contructor into its method initialize().

And of course double check all of the memory allocations, to make sure they were actually allocated, and that I did not have something like an off by one error...
 
The setup function does not contain a Serial.begin(9600) statement.

Pete

Sorry Pete,
from usb_serial.h
Code:
void begin(long) {
		//uint32_t millis_begin = systick_millis_count;
		//disabled for now - causes more trouble than it solves?
		//while (!(*this)) {
			// wait up to 2.5 seconds for Arduino Serial Monitor
			// Yes, this is a long time, but some Windows systems open
			// the port very slowly.  This wait allows programs for
			// Arduino Uno to "just work" (without forcing a reboot when
			// the port is opened), and when no PC is connected the user's
			// sketch still gets to run normally after this wait time.
			//if ((uint32_t)(systick_millis_count - millis_begin) > 2500) break;
		//}
	}
        void end() { /* TODO: flush output and shut down USB port */ };

Serial.begin(9600) does nothing, so it is not needed
 
Someone who knows way more than me about C++ seems to have found the problem.
My sketch defines the FingerController as a global variable, and the setup then calls the initialization method for it. Apparently the C++ compiler does not guarantee that the global variable is constructed by the time you reach the initialization call. The fix for this is to create a global getter function:

Initial code:
Code:
#include <FingerController.h>

#define ROS_BAUD_RATE 921600

/********************************************************************************
*****************************  GLOBAL VARIABLES  ********************************
********************************************************************************/
FingerController controller(ROS_BAUD_RATE);

/********************************************************************************
******************************  PROGRAM CODE  ***********************************
********************************************************************************/

void setup() {
    if(!controller.initialize()){
        Serial.println("Finger controller initialization failed");
        controller.error_led_blink(0);
        while(1);
    }
    Serial.println("Finger controller initialization done");
}

void loop(){  
    controller.spin();
}


Modified code:
Code:
#include <FingerController.h>

#define ROS_BAUD_RATE 921600

/********************************************************************************
*****************************  GLOBAL VARIABLES  ********************************
********************************************************************************/

FingerController* get_controller()
{
    static FingerController* controller = NULL;
    if (!controller) {
        controller = new FingerController(ROS_BAUD_RATE);
    }
    return controller;
}

/********************************************************************************
******************************  PROGRAM CODE  ***********************************
********************************************************************************/

void setup() {

    if(!get_controller()->initialize()){
        Serial.println("Finger controller initialization failed");
        get_controller()->error_led_blink(0);
        while(1);
    }
    Serial.println("Finger controller initialization done");
}

void loop(){  
    get_controller()->spin();
}
 
Someone who knows way more than me about C++ seems to have found the problem.
My sketch defines the FingerController as a global variable, and the setup then calls the initialization method for it. Apparently the C++ compiler does not guarantee that the global variable is constructed by the time you reach the initialization call. The fix for this is to create a global getter function:

Not sure if this is related, but if your code is in a .ino file, then the Arduino pre-processor shifts a lot of definitions around before passing the code to C++ compiler. You can check the generated cpp file in the working directory.
To avoid Arduino interferences, I use an empty ino file and put all code into a cpp file. Disadvantage is that code must be correct cpp syntax.
 
You can check the generated cpp file in the working directory.
To avoid Arduino interferences, I use an empty ino file and put all code into a cpp file. Disadvantage is that code must be correct cpp syntax.

Where can I find the generated cpp file? I could not find anything in the Arduino folder (that holds the sketches) or the installation directory (arduino-1.8.10 directory in Home for me in Ubuntu 16.04)

How do you leave the ino file blank and use just a normal cpp file? How does that cpp file gets executed?
 
If you activate verbose you will find the directory path to a temporary folder.

Arduino compiles all files in the sketch directory and links then together.
In addition, Arduino pre-processor combines all ino files into one big file, adds all required include files and where necessary generates forward declarations. This pre-processor may be from time to time the source of strange error messages.
 
It will be somewhere under your temp directory...

On Windows 10, you can bring up a browser window, and in address field type in: %temp%
In the resulting folder you may see directories with names like: arduino_build_213438

One of those is probably your build...

Note: As I was mentioning in previous post. You can probably solve some of this, by: simply moving some of the code out of
Code:
FingerController::FingerController(uint32_t rosserial_baud) {
  //Define hardware connections between flexboard and controller
  proximal_pins_.LED_A0 = 12;
  proximal_pins_.LED_A1 = 11;
  proximal_pins_.LED_A2 = 10;
  proximal_pins_.LED_A3 = 9;
  proximal_pins_.LED_A4 = 8;
  proximal_pins_.PD_A0 = 7;
  proximal_pins_.PD_A1 = 6;
  proximal_pins_.PD_A2 = 5;
  proximal_pins_.PD_A3 = 4;
  proximal_pins_.PD_A4 = 3;
  proximal_pins_.LED_OUT = A21;
  proximal_pins_.PD_IN = A14;

  distal_pins_.LED_A0 = 19;
  distal_pins_.LED_A1 = 20;
  distal_pins_.LED_A2 = 21;
  distal_pins_.LED_A3 = 22;
  distal_pins_.LED_A4 = 23;
  distal_pins_.PD_A0 =  34;
  distal_pins_.PD_A1 =  35;
  distal_pins_.PD_A2 =  36;
  distal_pins_.PD_A3 =  37;
  distal_pins_.PD_A4 =  38;
  distal_pins_.LED_OUT = A22;
  distal_pins_.PD_IN =  A20;
  
  //Define hardware connections for controller
  hw_pins_.TORQUE1 = A3;
  hw_pins_.TORQUE2 = A2;
  hw_pins_.TORQUE3 = A1;
  hw_pins_.LOAD_CELL = A0;
  hw_pins_.LED_TEENSY = 13;

  //Other initializations

  ros_baud_rate_ = rosserial_baud;
[COLOR="#FF0000"]  proximal_ = new FlexBoardv21(&proximal_pins_, "DIST17");
  distal_ = new FlexBoardv21(&distal_pins_, "DIST18");
  adc_ = new ADC();
  current_state_ = DECIDE_ACTION;[/COLOR]
  service_servos_ = false;
  prox_pair_idx_ = 0;
  dist_pair_idx_ = 0;
}
Into:
Code:
bool FingerController::initialize(){
  #ifdef DEBUG
    Serial.begin(SERIAL_BAUD);
    while(!Serial);
    Serial.println(F("Initializing Finger Controller..."));
  #endif

  //Setup ADC0 which handles sampling proximal flexboard and torque sensors
  adc_->adc0->setReference(ADC_REFERENCE::REF_3V3);
  adc_->adc0->setAveraging(1); // set number of averages
  adc_->adc0->setResolution(12); // set bits of resolution
  adc_->adc0->setConversionSpeed(ADC_CONVERSION_SPEED::HIGH_SPEED); // change the conversion speed
  adc_->adc0->setSamplingSpeed(ADC_SAMPLING_SPEED::VERY_HIGH_SPEED); // change the sampling speed
...
Example like try to move the lines in RED to the start of your initialize...

Problem with global objects, is there is no guarantee in what order constructors may get called. So if for example your code somehow depends on something being done before your code... It may or may not.

Good luck
 
An update on this issue. I am still having the same problem.
I've tried moving stuff out of the FingerController constructor into the initialization method, but still having issues. Not sure what the problem is. Sometimes I can't even get things to print out on the serial monitor, as if the program that I upload does absolutely nothing....

I'm at a lost on what's wrong with my code that SOMETIMES works, but sometimes doesn't.
 
Has post #7 been consulted or added? That is not indicated in p#10 code. Teensy will enter setup and appear to print nothing if that happens before USB Serial is established.

If pin 13 is free for LED usage perhaps set that to OUTPUT then set HIGH before the " while (!Serial && millis() < 4000 ); // wait up to 4 seconds after Teensy wakes … " then set it LOW after and the LED will flash on upload until Serial connects.
 
Should that always happen at the very beginning in the setup?? Because I have this section in the FingerController initialization method:


Code:
bool FingerController::initialize(){
  #ifdef DEBUG
    Serial.begin(SERIAL_BAUD);
    while(!Serial);
    Serial.println(F("Initializing Finger Controller..."));
  #endif

  ...

Which could explain why sometimes I don't see any output, since it might be getting caught in that while loop. I will modify with the timeout as suggested in post #7 to avoid this.
However, this whole code is originally designed for rosserial, and hence when I tried uploading the code without defining DEBUG (which gets rid of all Serial outputs using the preprocessor directives) it still wouldn't work...
Will incorporate #7 and keep testing...
 
I have noticed something else strange that might have to do with my current issues:

My current shield provides 5V independently from the USB voltage through the Vin pin. I would have expect that powering on the board would be enough to have my firmware start running, but it seems like this is not the case... When turning on the power, the board does nothing. If I connect the USB and reprogram, then the firmware works (most of the time, sometimes I have to reprogram as the title of the thread indicates...). In summary:
1) Upload a working firmware (based on an evolved version of the code provided in original thread)
2) Unplug USB
3) Remove 5V power to Teensy
4) Restore power to Teensy
5) Teensy does not seem to be executing the firmware as in step 1) :confused:

This behavior does not repeat if you upload the Blink example. In this case, after restoring the power the LED starts blinking again.
So there is definitely something fishy in my firmware....

Any ideas in light of this new clue?
I have uploaded here the latest firmware I am running (the FlexBoard class is now fully documented) in case that helps.
 

Attachments

  • Sketch_and_libraries_v2.zip
    14.8 KB · Views: 54
Having this line anywhere: while(!Serial);

When no USB Serial Monitor connects will sit forever.

It should be something like : while(!Serial && millis() < 2000 );
Where 2000 is replaced by an acceptable time. Maybe that was added regarding : "Will incorporate #7 and keep testing... "

gotta run now
 
I have already made that change, it happens as soon as the FingerController initialization is called, the first thing in there is this block of code:

Code:
  #ifdef DEBUG_PRINT
    Serial.begin(2000000);
    while (!Serial && millis() < 4000 );  // wait up to 4 seconds after Teensy wakes ...
    if(!Serial){
        error_led_blink(0);
        return false;
    } 
    else{
        Serial.println(F("Initializing Finger Controller..."));
    }
  #endif

BTW I also considered having this block in the setup() section of the sketch, instead of inside my FingerController initialization method, but - as expected - that didn't make any difference.
 
Would that : "return false;"

Prevent the 'FingerController initialization' process from completing when there was !Serial?

Usually makes sense to end up with the "Serial" detect and handing as needed in the primary : setup() so it doesn't get hidden or affect things later.
 
Again not sure what your code looks like now, but you earlier stuff was like:
Code:
bool FingerController::initialize(){
  #ifdef DEBUG
    Serial.begin(SERIAL_BAUD);
    while(!Serial);
    Serial.println(F("Initializing Finger Controller..."));
  #endif

  //Setup ADC0 which handles sampling proximal flexboard and torque sensors
  adc_->adc0->setReference(ADC_REFERENCE::REF_3V3);
  adc_->adc0->setAveraging(1); // set number of averages
  adc_->adc0->setResolution(12); // set bits of resolution
  adc_->adc0->setConversionSpeed(ADC_CONVERSION_SPEED::HIGH_SPEED); // change the conversion speed
  adc_->adc0->setSamplingSpeed(ADC_SAMPLING_SPEED::VERY_HIGH_SPEED); // change the sampling speed

  //Setup ADC1 which handles sampling distal flexboard
  adc_->adc1->setReference(ADC_REFERENCE::REF_3V3);
  adc_->adc1->setAveraging(1); // set number of averages
  adc_->adc1->setResolution(12); // set bits of resolution
  adc_->adc1->setConversionSpeed(ADC_CONVERSION_SPEED::HIGH_SPEED); // change the conversion speed
  adc_->adc1->setSamplingSpeed(ADC_SAMPLING_SPEED::VERY_HIGH_SPEED); // change the sampling speed

  //Setup DAC
  analogWriteResolution(12); //DAC working with 12bit resolution

  //Setup Status LED and Torque inputs
  pinMode(hw_pins_.LED_TEENSY, OUTPUT);
  pinMode(hw_pins_.TORQUE1, INPUT);
  pinMode(hw_pins_.TORQUE2, INPUT);
  pinMode(hw_pins_.TORQUE3, INPUT);
  pinMode(hw_pins_.LOAD_CELL, INPUT);

  //See if the SD card is present and can be initialized:
  if (!SD.begin(BUILTIN_SDCARD)) {
    #ifdef DEBUG
      Serial.println(F("Could not find SD card"));
    #endif
    return false;
  }
  
  //Initialize Flexboards
  if(!proximal_->initialize()){
    #ifdef DEBUG
      Serial.println(F("Proximal initialization failed"));
    #else
      error_led_blink(0);
    #endif
    return false;
  }
  if(!distal_->initialize()){
    #ifdef DEBUG
      Serial.println(F("Distal initialization failed"));
    #else
      error_led_blink(0);
    #endif
    return false;
  }

  //Now with flexboards initialized, get sampling pairs arrays
  prox_pairs_ = proximal_->get_sample_pairs();
  dist_pairs_ = distal_->get_sample_pairs();
  prox_num_pairs_ = proximal_->get_num_pairs();
  dist_num_pairs_ = distal_->get_num_pairs();

  //Proximal readings msg setup
  std_msgs::MultiArrayDimension prox_dim;
  std_msgs::MultiArrayLayout prox_layout;
  prox_dim.label = "prox_readings";
  prox_dim.size = prox_num_pairs_;
  prox_dim.stride = prox_num_pairs_;
  prox_layout.dim = (std_msgs::MultiArrayDimension *) malloc(sizeof(std_msgs::MultiArrayDimension) * 1);
  prox_layout.dim[0] = prox_dim;
  prox_layout.data_offset = 0;
  proximal_readings_msg_.layout = prox_layout;
  proximal_readings_msg_.data = (uint16_t *)malloc(sizeof(uint16_t) * prox_num_pairs_);
  proximal_readings_msg_.data_length = prox_num_pairs_;

  //Distal readings msg setup
  std_msgs::MultiArrayDimension dist_dim;
  std_msgs::MultiArrayLayout dist_layout;
  dist_dim.label = "dist_readings";
  dist_dim.size = dist_num_pairs_;
  dist_dim.stride = dist_num_pairs_;
  dist_layout.dim = (std_msgs::MultiArrayDimension *) malloc(sizeof(std_msgs::MultiArrayDimension) * 1);
  dist_layout.dim[0] = dist_dim;
  dist_layout.data_offset = 0;
  distal_readings_msg_.layout = dist_layout;
  distal_readings_msg_.data = (uint16_t *)malloc(sizeof(uint16_t) * dist_num_pairs_);
  distal_readings_msg_.data_length = dist_num_pairs_;

  //Verify that all memory was allocated succesfully
  if(distal_readings_msg_.data==NULL or proximal_readings_msg_.data==NULL){
    #ifdef DEBUG
      Serial.println(F("Memory allocation failed"));
    #endif
    return false;
  }

  // //Set up all ROS communication
  // nh_.getHardware()->setBaud(ros_baud_rate_);
  // nh_.initNode();
  // nh_.advertise(pub_prox_tactile_);
  // nh_.advertise(pub_dist_tactile_);
  // nh_.advertise(pub_loadcell_newtons_);
  // nh_.advertise(pub_loadcell_raw_);

  return true;
}
So if you return false early on... None of this stuff in there gets done?
Likewise returning false? Is your mainline code still like:
Code:
void setup() {
    if(!controller.initialize()){
        Serial.println("Finger controller initialization failed");
        controller.error_led_blink(0);
        while(1);
    }
    Serial.println("Finger controller initialization done");
}

void loop(){  
    controller.spin();
}
In which case it looks like your code will hang in the while(1); ...
 
Hey KurtE - that is what I was saying in p#22 - there is updated code in p#18 - but not time to pull it down and parse it just now.

But is as noted in p#22 that return false hits what is shown in p#23 - that would explain the failure to execute.
 
Status
Not open for further replies.
Back
Top