Please please please Consider a hardware debugging on the next Teensy revision

[birb]

I have dug through this forum a bit, and I know that this has already been brought up a few times, but I want to write my own plea. I have been using a teensy 4.1 for a formula SAE project to create a custom ECU for an electric vehicle. It was selected because the power & capability of the teensy was comparable to that of the highest end STM32's but with a much better developer experience in thanks to better platformio/arduino integration, and the arduino framework support, and has more pins and peripherals than an ESP-32-S3. So far working with it has been pleasant when things work, and if you are writing code that is going to operate in a single super-loop, then you will probably have no issues. However, my project uses freeRTOS, which makes debugging difficult without a hardware debugger for a few reasons:

• The software GDB project: https://github.com/ftrias/TeensyDebug just does not work in multi-threaded applications, not sure why but it seems to be a fundamental limitation as noted by a few of the issues on the project page.
• Serial printing is only an effective method of debugging **when** your serial print works. The nature of having a multi-threaded application meant I had multiple issues getting my Serial print to work at all, and it was very difficult to debug the issue with my Serial print when my only window into what the code is doing is said serial print.
  • sub-point on this, it takes the teensy quite a few seconds after upload for the com-port to reappear on the host PC, which means if you are trying to debug something in the early start-up, chances are you will have a hard time catching it (I found adding delays to set-up just delayed when the teensy connected to the host entirely which did not help much)
  • Serial prints also are not thread-safe meaning I would need to have 1 task printing debug messages on behalf of all my other threads which means the prints will not be happening in real-time compared to the thing I would be trying to inspect
• It's quite laborious to print out everything you might ever want to inspect when normally if I was working on another platform, I would simply add watches to all the variables I was interested in then run the debugger.
• The lack of JTAG/SWD limits the usefulness of the teensy for more complex projects, which is a true shame considering how capable of a platform it is, and if you want to mod it to expose the JTAG pins, you lose the convenience factor of the teensy bootloader, and you now have to go out and buy an additional hardware debugger.
  

My plea/suggestion is a system akin to what ESP-32 has on their most recent SOCs. On their current ESP-32-S3, there are 2 USB controllers, a USB Serial/Serial controller and a USB OTG controller providing the best of both worlds. On some devboards these are then fed into a USB hub, essentially giving the host computer access to all the functionality over 1 USB cable which really is a developer experience game changer. Now if that's too much, just having exposed/accessible JTAG pins to attach an external hardware debugger without losing your USB bootloader would probably be good enough for advanced users (even if it's not quite as nice of a developer experience).

 

[Stephanie Wallace]

I certainly can’t make any specific claims for PJRC, but I think it’s relatively safe to say ESP32’s new approach to JTAG is the future of the industry. The (relatively) straightforward and inexpensive implementation combined with an invaluable user experience is a development that would cause an outcry from enthusiasts and engineers… were it not already such an additional value by manufacturers that it’s nearly inevitable.

Imagine SSD’s without the heavy dedication to engineering, development time, and corresponding wait for prices to drop. Widespread adoption is nearly unthinkable.

 

[CollinK]

The Teensy boards do have hardware debugging. In fact, it's hooked up and everything. The problem is that the PRJC boot chip is the one connected so there is no prevision for another device to also be on the JTAG pins. What may be the path of least resistance is if PJRC can make their boot chip do JTAG passthrough on command. Since JTAG is there and wired this may or may not be pretty easy to pull off on things like the Teensy 4 / 4.1 / MM and future boards. I don't know the specifics but I think JTAG pass through could maybe be a possibility without hardware mods.

 

[birb]

yeah that would be nice as well. Honestly anything at this point would be a game changer. I'm currently developing a fairly large program for a teensy 4.1 with freeRTOS and the software debugger (https://github.com/ftrias/TeensyDebug) just doesn't work if you are using freeRTOS for reasons that are beyond my understanding (something to do with freeRTOS taking over the timer it needs), and I've had spent days trying to trace nearly impossible to diagnose issues through guess and check because that was essentially the only option available to me. I'm sure a more expert embedded developer could survive without a debugger but it really would make the development of much larger programs much more accessible, especially considering how powerful and versatile the teensy platform is.

 

[AndyA]

I'm sure a more expert embedded developer could survive without a debugger

Probably not, more experienced developers will be used to good debug tools. Once you get used to having a full IDE with debugger it's hard to go back.

We have hit similar issues. Using teensy makes it easy to get started and get the core functionality implemented quickly. And for simple projects where there is only one or two things going on it's great, it speeds development up massively. But when you then try to use it for a more complex project with lots of things going on the lack of debugging can make progress painfully slow at times. Because of the lack of JTAG support we'll probably go back to more conventional development methods for more complex projects in the future, the early benefits aren't worth the extra difficulty at later stages. Teensy or similar platforms will still get used for the quicker / simpler projects where there is less complexity and potential for tricky to find bugs.

Which is I suppose is a convoluted way of saying use the right tool for the job.

The lack of JTAG means Teensy isn't the right tool for some jobs. Whether that market (generally the more professional / complex projects) is something pjrc want to pursue is a business decision for them. I'd expect that if a "Teensy debugger" that connected to the bootloader chip and gave JTAG access and integration with an IDE was available then commercial developers would be happy to pay a reasonable amount for it.

 

[birb]

Probably not, more experienced developers will be used to good debug tools. Once you get used to having a full IDE with debugger it's hard to go back.

We have hit similar issues. Using teensy makes it easy to get started and get the core functionality implemented quickly. And for simple projects where there is only one or two things going on it's great, it speeds development up massively. But when you then try to use it for a more complex project with lots of things going on the lack of debugging can make progress painfully slow at times. Because of the lack of JTAG support we'll probably go back to more conventional development methods for more complex projects in the future, the early benefits aren't worth the extra difficulty at later stages. Teensy or similar platforms will still get used for the quicker / simpler projects where there is less complexity and potential for tricky to find bugs.

Which is I suppose is a convoluted way of saying use the right tool for the job.

The lack of JTAG means Teensy isn't the right tool for some jobs. Whether that market (generally the more professional / complex projects) is something pjrc want to pursue is a business decision for them. I'd expect that if a "Teensy debugger" that connected to the bootloader chip and gave JTAG access and integration with an IDE was available then commercial developers would be happy to pay a reasonable amount for it.

It's an interesting point to say that perhaps the teensy simply isn't the right tool for the job. Looking at the current "hobbyist"/prototype board landscape, the new arduinos and esp32's all have JTAG debugging capabilities directly accessible over USB, which puts the teensy in a really weird place in the market. It is more powerful and capable than an arduino/esp32, yet simultaneously, has a less capable set of dev tools due to the lack of JTAG debugging. Seems like if PJRC wants to remain competitive, it's something they have to do.

For my future projects of this level of complexity, I'll probably have to look into ESP32's as I can keep all the same "ease of use" benefits from being able to tap into the Arduino ecosystem, without leaving behind the developer tools I would need to work efficiently. The number of problems that I spent days solving rather than minutes because of the lack of JTAG has been mindboggling. I just wish ESP32s had more I/O and maybe slightly faster clock speeds.

 

[AndyA]

For my future projects of this level of complexity, I'll probably have to look into ESP32's as I can keep all the same "ease of use" benefits from being able to tap into the Arduino ecosystem

We're already using ESP32's in a number of places, generally using the espressif environment rather than arduino. A lot of the places we've used them we don't even have wifi / bluetooth, one of their big selling points. They are cheap, relatively fast and more importantly supply chain / availability isn't an issue.
We've used teensy when we need the processing power, more UARTs or a wired ethernet. All of these applications so far have been logically simple even if they did need the raw power. But then we tried using it on a more complex application and it's been painful. If we were to start again from scratch right now we'd probably either go raw i.MX RT or a higher end STM32.

There is no such thing as a perfect tool for all jobs, if anyone claims they have one then they are selling something. Or delusional. Or simply an idiot. Teensy certainly has it's place. But without real debugging that place is not more complex applications.