Application code just... disappears?

[alexw]

Hey all, I'm brand new to using the Teensy 4.0 and I'm hoping some of you with more experience might be able to guide my debugging efforts better.

I've inherited a project that by all accounts seems to work just fine, but then if I leave it alone for a few days it's as if the flash just erases itself. I can still click the button and re-program the teensy, it's not bricked. And if I push and hold the button, it gets loaded with the blinky program. When I do re-program my own code, it goes right back to operating exactly as I expect it too, until after some number of days left alone and powered off it doesn't (or so seems to be the timeline). It certainly seems the problem must be software. The software is written in C++ using the Arduino port.

The project is quite simple: it's effectively just the teensy4.0, an accelerometer, a Pololu power switch, and a 18650 battery. It serves as a fancy timer to trigger some GPIO outputs with either a timeout or a bump of the accel. The host system wakes it up by GPIO'ing the Pololu switch "on" while some Teensy GPIOs are held high to put it in the "countdown" state, a state machine does its thing and watches for acceleration events, and at the end of operation, the Teensy basically offs itself by poking the Pololu switch "off" with one of its own GPIOs.

There's an interactive debug mode over USB serial such that I can watch the state machine in operation, and after it completes its tasks it'll enter a test mode where it responds to commands over serial. When I say the unit "just dies" and the application code never runs, what I mean is that I can power on the Teensy and it doesn't even enumerate any USB devices. Normally, it'll enumerate a few HID devices and the virtual COM port (which I can of course connect to).

The one thing I can think of: the project saves some state to flash using the EEPROM library every second. That way, if there's some power disruption or the teensy resets for another reason, it'll start back up with the timer already partly elapsed. Possibly, the code COULD be doing a flash write right before it cuts power, failing to give the write operation time to complete. But that wouldn't result in the application code failing to launch next boot, would it? And it doesn't agree with the "it's usually dead if I leave it alone a few days" timeline.

Hopefully I can just share some code, but I'm guessing for now I can't point you to the whole repo

Does this raise any obvious questions in your mind that I should chase down?

 

[alexw]

I THINK I've figured out what went wrong, and there are two possibilities, neither of which follows the theories above.

1) Sometimes, my application code is running but for some reason no USB devices enumerate. I can plug and replug, and no USB devices (at all, not just VCP) ever show up. But the code does run. It seemed that trying a different computer suddenly worked, then when I went back to the first computer it also worked. I haven't figured out root cause here or reliably reproduced, but at least the code is otherwise running correctly.
2) In my hardware application, the Prog pin is run out over a relatively long wire (~3ft) alongside USB so that the device can be programmed without local physical access to the teensy. This extra wire added enough capacitance to the node to dramatically increase the rise time on power-on. That is, my power supply comes up in <1ms, but the prog pin rise time is more like 100ms with the wire attached. This frequently resulted in the teensy powering on straight into bootloader mode as if I'd held the button during power-on. In combination with not being able to see LEDs, and perhaps in combination with USB issues external to the teensy from (1) (drivers? windows weirdness? Needed to restart the host machine?), that LOOKED LIKE a completely unresponsive device.

Adding a 10k pull-up resistor to Prog fixed (2), decreasing the rise time to more like 30ms. That seems to have sorted me out, at least until (1) comes up again. But now I know to immediately try a different computer first.

 

[PaulStoffregen]

On #1, a couple things you might try...

Add a LED which your code blinks early in startup. This can let you check if the code actually started running even when USB isn't working.

Try editing usb.c in the core library. First just add a syntax error and click Verify in Arduino IDE, to check that you're editing the right file. Then uncomment this line:

        //USB1_PORTSC1 |= USB_PORTSC1_PFSC; // force 12 Mbit/sec

If you have a dodgy USB cable, running at 12 Mbit/sec rather than 480 Mbit/sec might be more likely to work.

 

[alexw]

Now that is an extremely helpful suggestion! The LED is of course the best option but on further inspection and consideration I was able to determine that, yes, the code is operating nominally apart from the lack of a serial device appearing. It has some external outputs that clearly toggle on schedule as long as I'm clever enough to actually measure them while testing . I was much too quick to jump to the conclusion that the code wasn't running. I think a flat battery may have also been contributing to my confusion. I think I'd have noticed symptoms THAT severe, but it's possible there was a "now it's not responding at all! Throw it on charge and give up for the day" type of scenario.

As far as forcing full speed - yes, the cables the USB runs over can reasonably be described as dodgy. And I don't have enough data to say, but it seems very plausible that the bench test setup with slightly less dodgy cabling fails in this way less often than the as-installed setup that's much dodgier.