Forum Rule: Always post complete source code & details to reproduce any issue!
Results 1 to 24 of 24

Thread: Teensy 3.6 requires 3 reprogram events to work properly

  1. #1
    Junior Member
    Join Date
    May 2018
    Location
    NYC
    Posts
    16

    Teensy 3.6 requires 3 reprogram events to work properly

    Hello everyone,

    I'm working on a robotics project using a Teensy 3.6 in Ubuntu 16.04. The teensy is connected to a custom shield to control 2 boards ("distal" and "proximal") which has been thoroughly validated.
    I have been encountering some weird issues that can be hard to reproduce, even for me. But it's been happening in the same way for a while now, which is why I decided to post and see if anyone can shed some light into this behavior...

    As the title says, my code works as intended, only after (usually) 3 "resets": I make changes to the code, upload the firmware using the arduino IDE and open the serial monitor to inspect the debug messages that I print there. On first upload, I get values for "Number of LEDs" and "Number of PDs" printed out that do not follow my code. Then I press the program/reset button on the physical Teensy 3.6 and the debug messages now show SOME correct values for one of the boards, but not both. Finally if I press the button once again, then the serial monitor shows all values initialized as per my code. I have attached 3 screenshots showing this happening - take into account just the reset button is being pushed in between these screenshots, no changes to the code at all.

    Click image for larger version. 

Name:	First upload.jpg 
Views:	17 
Size:	49.8 KB 
ID:	18802

    Click image for larger version. 

Name:	Second Upload.jpg 
Views:	15 
Size:	79.3 KB 
ID:	18803

    Click image for larger version. 

Name:	Third Upload.jpg 
Views:	11 
Size:	87.7 KB 
ID:	18804


    Sometimes you get incorrect values that are zeros (as in the first screenshot) but I've also gotten random values in other occasions. Unfortunately the code base is large enough that I think attaching it is better than copying it in a /CODE block. For reference, the values shown in the 3rd screenshot are the correct ones (32 LEDs, 28 PDs). These values are constant member values declared in the derived class "Flexboardv21", which are passed directly into the base class constructor and saved into the base class member variables to hold this information (the reason for this structure is that there are other versions of the Flexboard with different number of LEDs and PDs and mappings which I have erased for clarity).

    While there is use of "new" and dynamic memory allocation in the code, I still don't think that accounts for this kind of behavior.
    Any ideas what could be causing this?

    Thanks in advance for your time,
    Pedro
    Attached Files Attached Files

  2. #2
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    12,175
    easy first guess - power and ground routing through Teensy and USB connecter to computer?

    Either the USB connector is being abused for GND current and interfering with data transmission or perhaps code is being wonky and not responsive to the USB bootloader command.

    Did this only start with the 'new' dynamic memory? Could that or other things that might be trashing memory be reduced/removed?

  3. #3
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    12,175
    A quick test compile/upload a simple BLINK sketch. Assuming that runs when all the other lights and hardware are disabled - it should act normally given a good cable and healthy T_3.6

    Not a Ubuntu user - not sure if there is anything in the system that might try to steal/interrupt the Teensy session. There has been an update to UDEV rules in prior months that should be in use?

  4. #4
    Junior Member
    Join Date
    May 2018
    Location
    NYC
    Posts
    16
    Thanks for the suggestions! I will try tomorrow to provide power to the Teensy independently from the USB power.
    I will try as well to remove the dynamic memory for now to see if that helps. I've actually reinstalled the arduino IDE and teensyduino last week and did re-download the udev rules again just in case.

    BTW, the blink sketch always works at first try. Will report back soon with my findings

  5. #5
    Senior Member
    Join Date
    Nov 2012
    Posts
    1,412
    The setup function does not contain a Serial.begin(9600) statement.

    Pete

  6. #6
    Junior Member
    Join Date
    May 2018
    Location
    NYC
    Posts
    16
    The serial is initialized as part of the finger controller class if the DEBUG flag is set to True (otherwise I wouldn't have gotten any output at all)

  7. #7
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    12,175
    Quote Originally Posted by el_supremo View Post
    The setup function does not contain a Serial.begin(9600) statement.

    Pete
    To add to what Pete posted … Teensy when compiled with USB Always starts it - however depending on PC connection time and when SerMon is ready it can be in setup() and beyond before USN connect is actually established.

    When USB Prints need to be seen reliably having something like this in setup() is best direction:
    Code:
    void setup() {
    // …
    	Serial.begin(115200);
    	while (!Serial && millis() < 4000 );  // wait up to 4 seconds after Teensy wakes ...
    	Serial.println("\n" __FILE__ " " __DATE__ " " __TIME__);
    //…
    }

  8. #8
    Senior Member+ KurtE's Avatar
    Join Date
    Jan 2014
    Posts
    7,400
    If it were me, I would wonder about the work being done in the constructors...

    I have been bit in the past with constructors that do much more than simply store away some settings that were passed in... Especially if the object is global...

    I would probably see what happened if I moved most of the work of the initialization from FingerController contructor into its method initialize().

    And of course double check all of the memory allocations, to make sure they were actually allocated, and that I did not have something like an off by one error...

  9. #9
    Senior Member
    Join Date
    Jul 2014
    Posts
    2,714
    Quote Originally Posted by el_supremo View Post
    The setup function does not contain a Serial.begin(9600) statement.

    Pete
    Sorry Pete,
    from usb_serial.h
    Code:
    void begin(long) {
    		//uint32_t millis_begin = systick_millis_count;
    		//disabled for now - causes more trouble than it solves?
    		//while (!(*this)) {
    			// wait up to 2.5 seconds for Arduino Serial Monitor
    			// Yes, this is a long time, but some Windows systems open
    			// the port very slowly.  This wait allows programs for
    			// Arduino Uno to "just work" (without forcing a reboot when
    			// the port is opened), and when no PC is connected the user's
    			// sketch still gets to run normally after this wait time.
    			//if ((uint32_t)(systick_millis_count - millis_begin) > 2500) break;
    		//}
    	}
            void end() { /* TODO: flush output and shut down USB port */ };
    Serial.begin(9600) does nothing, so it is not needed

  10. #10
    Junior Member
    Join Date
    May 2018
    Location
    NYC
    Posts
    16
    Someone who knows way more than me about C++ seems to have found the problem.
    My sketch defines the FingerController as a global variable, and the setup then calls the initialization method for it. Apparently the C++ compiler does not guarantee that the global variable is constructed by the time you reach the initialization call. The fix for this is to create a global getter function:

    Initial code:
    Code:
    #include <FingerController.h>
    
    #define ROS_BAUD_RATE 921600
    
    /********************************************************************************
    *****************************  GLOBAL VARIABLES  ********************************
    ********************************************************************************/
    FingerController controller(ROS_BAUD_RATE);
    
    /********************************************************************************
    ******************************  PROGRAM CODE  ***********************************
    ********************************************************************************/
    
    void setup() {
        if(!controller.initialize()){
            Serial.println("Finger controller initialization failed");
            controller.error_led_blink(0);
            while(1);
        }
        Serial.println("Finger controller initialization done");
    }
    
    void loop(){  
        controller.spin();
    }

    Modified code:
    Code:
    #include <FingerController.h>
    
    #define ROS_BAUD_RATE 921600
    
    /********************************************************************************
    *****************************  GLOBAL VARIABLES  ********************************
    ********************************************************************************/
    
    FingerController* get_controller()
    {
        static FingerController* controller = NULL;
        if (!controller) {
            controller = new FingerController(ROS_BAUD_RATE);
        }
        return controller;
    }
    
    /********************************************************************************
    ******************************  PROGRAM CODE  ***********************************
    ********************************************************************************/
    
    void setup() {
    
        if(!get_controller()->initialize()){
            Serial.println("Finger controller initialization failed");
            get_controller()->error_led_blink(0);
            while(1);
        }
        Serial.println("Finger controller initialization done");
    }
    
    void loop(){  
        get_controller()->spin();
    }

  11. #11
    Senior Member
    Join Date
    Jul 2014
    Posts
    2,714
    Quote Originally Posted by Piachnp View Post
    Someone who knows way more than me about C++ seems to have found the problem.
    My sketch defines the FingerController as a global variable, and the setup then calls the initialization method for it. Apparently the C++ compiler does not guarantee that the global variable is constructed by the time you reach the initialization call. The fix for this is to create a global getter function:
    Not sure if this is related, but if your code is in a .ino file, then the Arduino pre-processor shifts a lot of definitions around before passing the code to C++ compiler. You can check the generated cpp file in the working directory.
    To avoid Arduino interferences, I use an empty ino file and put all code into a cpp file. Disadvantage is that code must be correct cpp syntax.

  12. #12
    Junior Member
    Join Date
    May 2018
    Location
    NYC
    Posts
    16
    Quote Originally Posted by WMXZ View Post
    You can check the generated cpp file in the working directory.
    To avoid Arduino interferences, I use an empty ino file and put all code into a cpp file. Disadvantage is that code must be correct cpp syntax.
    Where can I find the generated cpp file? I could not find anything in the Arduino folder (that holds the sketches) or the installation directory (arduino-1.8.10 directory in Home for me in Ubuntu 16.04)

    How do you leave the ino file blank and use just a normal cpp file? How does that cpp file gets executed?

  13. #13
    Senior Member
    Join Date
    Jul 2014
    Posts
    2,714
    If you activate verbose you will find the directory path to a temporary folder.

    Arduino compiles all files in the sketch directory and links then together.
    In addition, Arduino pre-processor combines all ino files into one big file, adds all required include files and where necessary generates forward declarations. This pre-processor may be from time to time the source of strange error messages.

  14. #14
    Senior Member+ KurtE's Avatar
    Join Date
    Jan 2014
    Posts
    7,400
    It will be somewhere under your temp directory...

    On Windows 10, you can bring up a browser window, and in address field type in: %temp%
    In the resulting folder you may see directories with names like: arduino_build_213438

    One of those is probably your build...

    Note: As I was mentioning in previous post. You can probably solve some of this, by: simply moving some of the code out of
    Code:
    FingerController::FingerController(uint32_t rosserial_baud) {
      //Define hardware connections between flexboard and controller
      proximal_pins_.LED_A0 = 12;
      proximal_pins_.LED_A1 = 11;
      proximal_pins_.LED_A2 = 10;
      proximal_pins_.LED_A3 = 9;
      proximal_pins_.LED_A4 = 8;
      proximal_pins_.PD_A0 = 7;
      proximal_pins_.PD_A1 = 6;
      proximal_pins_.PD_A2 = 5;
      proximal_pins_.PD_A3 = 4;
      proximal_pins_.PD_A4 = 3;
      proximal_pins_.LED_OUT = A21;
      proximal_pins_.PD_IN = A14;
    
      distal_pins_.LED_A0 = 19;
      distal_pins_.LED_A1 = 20;
      distal_pins_.LED_A2 = 21;
      distal_pins_.LED_A3 = 22;
      distal_pins_.LED_A4 = 23;
      distal_pins_.PD_A0 =  34;
      distal_pins_.PD_A1 =  35;
      distal_pins_.PD_A2 =  36;
      distal_pins_.PD_A3 =  37;
      distal_pins_.PD_A4 =  38;
      distal_pins_.LED_OUT = A22;
      distal_pins_.PD_IN =  A20;
      
      //Define hardware connections for controller
      hw_pins_.TORQUE1 = A3;
      hw_pins_.TORQUE2 = A2;
      hw_pins_.TORQUE3 = A1;
      hw_pins_.LOAD_CELL = A0;
      hw_pins_.LED_TEENSY = 13;
    
      //Other initializations
    
      ros_baud_rate_ = rosserial_baud;
      proximal_ = new FlexBoardv21(&proximal_pins_, "DIST17");
      distal_ = new FlexBoardv21(&distal_pins_, "DIST18");
      adc_ = new ADC();
      current_state_ = DECIDE_ACTION;
      service_servos_ = false;
      prox_pair_idx_ = 0;
      dist_pair_idx_ = 0;
    }
    Into:
    Code:
    bool FingerController::initialize(){
      #ifdef DEBUG
        Serial.begin(SERIAL_BAUD);
        while(!Serial);
        Serial.println(F("Initializing Finger Controller..."));
      #endif
    
      //Setup ADC0 which handles sampling proximal flexboard and torque sensors
      adc_->adc0->setReference(ADC_REFERENCE::REF_3V3);
      adc_->adc0->setAveraging(1); // set number of averages
      adc_->adc0->setResolution(12); // set bits of resolution
      adc_->adc0->setConversionSpeed(ADC_CONVERSION_SPEED::HIGH_SPEED); // change the conversion speed
      adc_->adc0->setSamplingSpeed(ADC_SAMPLING_SPEED::VERY_HIGH_SPEED); // change the sampling speed
    ...
    Example like try to move the lines in RED to the start of your initialize...

    Problem with global objects, is there is no guarantee in what order constructors may get called. So if for example your code somehow depends on something being done before your code... It may or may not.

    Good luck

  15. #15
    Junior Member
    Join Date
    May 2018
    Location
    NYC
    Posts
    16
    An update on this issue. I am still having the same problem.
    I've tried moving stuff out of the FingerController constructor into the initialization method, but still having issues. Not sure what the problem is. Sometimes I can't even get things to print out on the serial monitor, as if the program that I upload does absolutely nothing....

    I'm at a lost on what's wrong with my code that SOMETIMES works, but sometimes doesn't.

  16. #16
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    12,175
    Has post #7 been consulted or added? That is not indicated in p#10 code. Teensy will enter setup and appear to print nothing if that happens before USB Serial is established.

    If pin 13 is free for LED usage perhaps set that to OUTPUT then set HIGH before the " while (!Serial && millis() < 4000 ); // wait up to 4 seconds after Teensy wakes … " then set it LOW after and the LED will flash on upload until Serial connects.

  17. #17
    Junior Member
    Join Date
    May 2018
    Location
    NYC
    Posts
    16
    Should that always happen at the very beginning in the setup?? Because I have this section in the FingerController initialization method:


    Code:
    bool FingerController::initialize(){
      #ifdef DEBUG
        Serial.begin(SERIAL_BAUD);
        while(!Serial);
        Serial.println(F("Initializing Finger Controller..."));
      #endif
    
      ...
    Which could explain why sometimes I don't see any output, since it might be getting caught in that while loop. I will modify with the timeout as suggested in post #7 to avoid this.
    However, this whole code is originally designed for rosserial, and hence when I tried uploading the code without defining DEBUG (which gets rid of all Serial outputs using the preprocessor directives) it still wouldn't work...
    Will incorporate #7 and keep testing...

  18. #18
    Junior Member
    Join Date
    May 2018
    Location
    NYC
    Posts
    16
    I have noticed something else strange that might have to do with my current issues:

    My current shield provides 5V independently from the USB voltage through the Vin pin. I would have expect that powering on the board would be enough to have my firmware start running, but it seems like this is not the case... When turning on the power, the board does nothing. If I connect the USB and reprogram, then the firmware works (most of the time, sometimes I have to reprogram as the title of the thread indicates...). In summary:
    1) Upload a working firmware (based on an evolved version of the code provided in original thread)
    2) Unplug USB
    3) Remove 5V power to Teensy
    4) Restore power to Teensy
    5) Teensy does not seem to be executing the firmware as in step 1)

    This behavior does not repeat if you upload the Blink example. In this case, after restoring the power the LED starts blinking again.
    So there is definitely something fishy in my firmware....

    Any ideas in light of this new clue?
    I have uploaded here the latest firmware I am running (the FlexBoard class is now fully documented) in case that helps.
    Attached Files Attached Files

  19. #19
    Junior Member
    Join Date
    May 2018
    Location
    NYC
    Posts
    16
    Anyone?
    I don't mind reprogramming until it works, but having it work when powering on is very important

  20. #20
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    12,175
    Having this line anywhere: while(!Serial);

    When no USB Serial Monitor connects will sit forever.

    It should be something like : while(!Serial && millis() < 2000 );
    Where 2000 is replaced by an acceptable time. Maybe that was added regarding : "Will incorporate #7 and keep testing... "

    gotta run now

  21. #21
    Junior Member
    Join Date
    May 2018
    Location
    NYC
    Posts
    16
    I have already made that change, it happens as soon as the FingerController initialization is called, the first thing in there is this block of code:

    Code:
      #ifdef DEBUG_PRINT
        Serial.begin(2000000);
        while (!Serial && millis() < 4000 );  // wait up to 4 seconds after Teensy wakes ...
        if(!Serial){
            error_led_blink(0);
            return false;
        } 
        else{
            Serial.println(F("Initializing Finger Controller..."));
        }
      #endif
    BTW I also considered having this block in the setup() section of the sketch, instead of inside my FingerController initialization method, but - as expected - that didn't make any difference.

  22. #22
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    12,175
    Would that : "return false;"

    Prevent the 'FingerController initialization' process from completing when there was !Serial?

    Usually makes sense to end up with the "Serial" detect and handing as needed in the primary : setup() so it doesn't get hidden or affect things later.

  23. #23
    Senior Member+ KurtE's Avatar
    Join Date
    Jan 2014
    Posts
    7,400
    Again not sure what your code looks like now, but you earlier stuff was like:
    Code:
    bool FingerController::initialize(){
      #ifdef DEBUG
        Serial.begin(SERIAL_BAUD);
        while(!Serial);
        Serial.println(F("Initializing Finger Controller..."));
      #endif
    
      //Setup ADC0 which handles sampling proximal flexboard and torque sensors
      adc_->adc0->setReference(ADC_REFERENCE::REF_3V3);
      adc_->adc0->setAveraging(1); // set number of averages
      adc_->adc0->setResolution(12); // set bits of resolution
      adc_->adc0->setConversionSpeed(ADC_CONVERSION_SPEED::HIGH_SPEED); // change the conversion speed
      adc_->adc0->setSamplingSpeed(ADC_SAMPLING_SPEED::VERY_HIGH_SPEED); // change the sampling speed
    
      //Setup ADC1 which handles sampling distal flexboard
      adc_->adc1->setReference(ADC_REFERENCE::REF_3V3);
      adc_->adc1->setAveraging(1); // set number of averages
      adc_->adc1->setResolution(12); // set bits of resolution
      adc_->adc1->setConversionSpeed(ADC_CONVERSION_SPEED::HIGH_SPEED); // change the conversion speed
      adc_->adc1->setSamplingSpeed(ADC_SAMPLING_SPEED::VERY_HIGH_SPEED); // change the sampling speed
    
      //Setup DAC
      analogWriteResolution(12); //DAC working with 12bit resolution
    
      //Setup Status LED and Torque inputs
      pinMode(hw_pins_.LED_TEENSY, OUTPUT);
      pinMode(hw_pins_.TORQUE1, INPUT);
      pinMode(hw_pins_.TORQUE2, INPUT);
      pinMode(hw_pins_.TORQUE3, INPUT);
      pinMode(hw_pins_.LOAD_CELL, INPUT);
    
      //See if the SD card is present and can be initialized:
      if (!SD.begin(BUILTIN_SDCARD)) {
        #ifdef DEBUG
          Serial.println(F("Could not find SD card"));
        #endif
        return false;
      }
      
      //Initialize Flexboards
      if(!proximal_->initialize()){
        #ifdef DEBUG
          Serial.println(F("Proximal initialization failed"));
        #else
          error_led_blink(0);
        #endif
        return false;
      }
      if(!distal_->initialize()){
        #ifdef DEBUG
          Serial.println(F("Distal initialization failed"));
        #else
          error_led_blink(0);
        #endif
        return false;
      }
    
      //Now with flexboards initialized, get sampling pairs arrays
      prox_pairs_ = proximal_->get_sample_pairs();
      dist_pairs_ = distal_->get_sample_pairs();
      prox_num_pairs_ = proximal_->get_num_pairs();
      dist_num_pairs_ = distal_->get_num_pairs();
    
      //Proximal readings msg setup
      std_msgs::MultiArrayDimension prox_dim;
      std_msgs::MultiArrayLayout prox_layout;
      prox_dim.label = "prox_readings";
      prox_dim.size = prox_num_pairs_;
      prox_dim.stride = prox_num_pairs_;
      prox_layout.dim = (std_msgs::MultiArrayDimension *) malloc(sizeof(std_msgs::MultiArrayDimension) * 1);
      prox_layout.dim[0] = prox_dim;
      prox_layout.data_offset = 0;
      proximal_readings_msg_.layout = prox_layout;
      proximal_readings_msg_.data = (uint16_t *)malloc(sizeof(uint16_t) * prox_num_pairs_);
      proximal_readings_msg_.data_length = prox_num_pairs_;
    
      //Distal readings msg setup
      std_msgs::MultiArrayDimension dist_dim;
      std_msgs::MultiArrayLayout dist_layout;
      dist_dim.label = "dist_readings";
      dist_dim.size = dist_num_pairs_;
      dist_dim.stride = dist_num_pairs_;
      dist_layout.dim = (std_msgs::MultiArrayDimension *) malloc(sizeof(std_msgs::MultiArrayDimension) * 1);
      dist_layout.dim[0] = dist_dim;
      dist_layout.data_offset = 0;
      distal_readings_msg_.layout = dist_layout;
      distal_readings_msg_.data = (uint16_t *)malloc(sizeof(uint16_t) * dist_num_pairs_);
      distal_readings_msg_.data_length = dist_num_pairs_;
    
      //Verify that all memory was allocated succesfully
      if(distal_readings_msg_.data==NULL or proximal_readings_msg_.data==NULL){
        #ifdef DEBUG
          Serial.println(F("Memory allocation failed"));
        #endif
        return false;
      }
    
      // //Set up all ROS communication
      // nh_.getHardware()->setBaud(ros_baud_rate_);
      // nh_.initNode();
      // nh_.advertise(pub_prox_tactile_);
      // nh_.advertise(pub_dist_tactile_);
      // nh_.advertise(pub_loadcell_newtons_);
      // nh_.advertise(pub_loadcell_raw_);
    
      return true;
    }
    So if you return false early on... None of this stuff in there gets done?
    Likewise returning false? Is your mainline code still like:
    Code:
    void setup() {
        if(!controller.initialize()){
            Serial.println("Finger controller initialization failed");
            controller.error_led_blink(0);
            while(1);
        }
        Serial.println("Finger controller initialization done");
    }
    
    void loop(){  
        controller.spin();
    }
    In which case it looks like your code will hang in the while(1); ...

  24. #24
    Senior Member+ defragster's Avatar
    Join Date
    Feb 2015
    Posts
    12,175
    Hey KurtE - that is what I was saying in p#22 - there is updated code in p#18 - but not time to pull it down and parse it just now.

    But is as noted in p#22 that return false hits what is shown in p#23 - that would explain the failure to execute.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •