Using GDB with Teensy without hardware debugger, first Beta

[ftrias]

I put together a library that allows GDB (included in Teensyduino) to perform source-level debugging on the Teensy without debugging hardware (no need for JTAG, SWD, etc). It uses GDB's Remote Serial Protocol to communicate with Teensy over a Serial connection (physical or USB). No extra hardware or software is required.

This is a beta release for comments and testing. It works on Teensy 4. Tested primarily on Mac.

See https://forum.pjrc.com/threads/61262...d=1#post243070 for previous discussion and this for background: https://forum.pjrc.com/threads/26358...Debugger-Stack.

Highlights:

* Set/remove breakpoints at runtime in most places in your code
* Examine and change memory and registers
* View call stack
* Halt code at any time
* Next/step one instruction at a time

Since real breakpoints are permanently disabled on the Teensy, in order to emulate breakpoints the library uses the SVC interrupt (like I did in TeensyThreads). Since Teensy 4 places most code in RAM, activating a breakpoint replaces the original instructions with SVC calls (and puts the original back later). On Teensy 3.2, setting breakpoints uses the Cortex-M4's features to "patch" parts of flash by pointing it to RAM.

The library is enabled by including a header, `TeensyDebug.h`. There is a program that adds a menu option that configures Arduino to open GDB automatically after uploading your program. You can also run GDB manually and use `target remote` to connect to the serial port used by the GDB interface.

To read more and try it out visit: http://github.com/ftrias/TeensyDebug

Any comments or code would be greatly appreciated!

[KurtE]

Sounds like fun!

I tried running the Install exe for windows, but it failed as I use the portable version of Arduino (zip file), where instead of c:\program files(...) I instead install
c:\Arduino-1.8.12 (also have 11 and 10...)

[ftrias]

Sounds like fun!

I tried running the Install exe for windows, but it failed as I use the portable version of Arduino (zip file), where instead of c:\program files(...) I instead install
c:\Arduino-1.8.12 (also have 11 and 10...)

Run the install program with the -i option as in: teensy_debug -i=c:\Arduino-1.8.12

[jonr]

It's works well - a debugger saves time and using one should be standard practice. Unfortunately, teensy doesn't support hardware debuggers.

On linux, start gdb with something like ~/arduino/arduino-1.8.12/hardware/tools/arm/bin/arm-none-eabi-gdb /tmp/arduino_build_361456/gdbstub-test.ino.elf -ex="target remote /dev/ttyACM1"

I see no need for python or install scripts. I just copied TeensyDebug.cpp, TeensyDebug.h and gdbstub.cpp into the same directory as my test program. Then compile and load with the Arduino IDE. Then run gdb in a terminal window.

I discovered that gdb can be scripted with the "define" command. So I could make some commands to conveniently read/clear/set teensy pins.

[KurtE]

Run the install program with the -i option as in: teensy_debug -i=c:\Arduino-1.8.12

Thanks, it is still not happy

D:\GitHub>cd TeensyDebug

D:\GitHub\TeensyDebug>teensy_debug -i=c:\arduino-1.8.12
Do you want to install GDB support for Teensyduino?
[y/n] ? y
Install GDB in Teensyduino
Teensyduino app found, but contents not found in c:\arduino-1.8.12! Try using -i
Traceback (most recent call last):
  File "\\Mac\Home\Documents\Source\TeensyDebug\teensy_debug.py", line 331, in <module>
  File "\\Mac\Home\Documents\Source\TeensyDebug\teensy_debug.py", line 235, in runGDB
NameError: name 'exit' is not defined
[17872] Failed to execute script teensy_debug

D:\GitHub\TeensyDebug>dir c:\arduino-1.8.12
 Volume in drive C is Windows
 Volume Serial Number is B2F0-35F3

 Directory of c:\arduino-1.8.12

02/13/2020  01:32 PM    <DIR>          .
02/13/2020  01:32 PM    <DIR>          ..
02/13/2020  11:32 AM        16,354,304 arduino-builder.exe
02/13/2020  11:32 AM           404,480 arduino.exe
02/13/2020  11:32 AM                71 arduino.l4j.ini
02/13/2020  11:32 AM           401,920 arduino_debug.exe
02/13/2020  11:32 AM                71 arduino_debug.l4j.ini
02/13/2020  01:31 PM    <DIR>          drivers
02/13/2020  01:33 PM    <DIR>          examples
02/13/2020  01:33 PM    <DIR>          hardware
02/13/2020  01:31 PM    <DIR>          java
05/16/2020  05:49 AM    <DIR>          lib
02/13/2020  01:31 PM    <DIR>          libraries
02/13/2020  11:32 AM            43,520 libusb0.dll
02/13/2020  11:32 AM           421,200 msvcp100.dll
02/13/2020  11:32 AM           770,384 msvcr100.dll
02/13/2020  01:31 PM    <DIR>          reference
02/13/2020  11:32 AM            94,379 revisions.txt
02/13/2020  01:32 PM    <DIR>          tools
02/13/2020  01:31 PM    <DIR>          tools-builder
02/13/2020  11:32 AM               331 wrapper-manifest.xml
              10 File(s)     18,490,660 bytes
              11 Dir(s)  752,807,575,552 bytes free

Note: I am not sure if this helps or not, but If I type python I see

D:\GitHub\TeensyDebug>python
Python 3.7.1 (v3.7.1:260ec2c36a, Oct 20 2018, 14:57:15) [MSC v.1915 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.

[jonr]

I look forward to more features. Perhaps "write register" and workarounds for calling target functions from the gdb command line?

[ftrias]

Thanks, it is still not happy

All it does is:

1. copy the source files (*.cpp, *.h) to a directory named TeensyDebug in your library directory
2. customize and copy boards.local.txt and platforms.local.txt to ...Arduino/hardware/teensy/avr
(a) change platforms.local.txt to use "teensy_debug.exe" instead or "run.command"
3. copy teensy_debug.exe to ..Arduino/hardware/tools

[defragster]

SUCCESS!!! Got the latest from github -

problem was using the included :: debug.begin(Serial1); ... needed to be debug.begin(SerialUSB1);

Using library included example :: ...\libraries\TeensyDebug\examples\breakpoint_test \breakpoint_test.ino

Reading symbols from T:\TEMP\\arduino_build_breakpoint_test.ino\breakpoint_test.ino.elf...done.
(gdb) target remote \\.\COM21
Remote debugging using \\.\COM21
0x00000000 in _stext ()
(gdb) p mark
$1 = 5
(gdb) b level1
Breakpoint 1 at 0x108: file T:\tCode\libraries\TeensyDebug\examples\breakpoint_test/breakpoint_test.ino, line 30.
(gdb) c
Continuing.

Breakpoint 1, level1 () at T:\tCode\libraries\TeensyDebug\examples\breakpoint_test/breakpoint_test.ino:30
30      void level1() {
(gdb)

This is on Windows and works on T_4.1 ( IDE 1.82 w/TD 1.52 ) compileing from SublimeText using TSET

In this case::
> run TSET.cmd ( with edited user paths ) in sketch folder enter these 5 keys at prompts :: "711md"
> run Compile.cmd from the sketch folder to build and start GDB with symbols loaded.
> find the Debug COM_PORT and enter :(gdb) target remote \\.\COM21

Using the code from GITHUB readme::

Reading symbols from T:\TEMP\\arduino_build_GDBgitSample.ino\GDBgitSample.ino.elf...done.
(gdb) target remote \\.\COM21
Remote debugging using \\.\COM21
setup () at T:\tCode\GDB_DBG\GDBgitSample/GDBgitSample.ino:23
23      }
(gdb) p mark
$1 = 0
(gdb) b test_function()
Breakpoint 1 at 0xb8: file T:\tCode\GDB_DBG\GDBgitSample/GDBgitSample.ino, line 6.
(gdb) c
Continuing.

Breakpoint 1, test_function () at T:\tCode\GDB_DBG\GDBgitSample/GDBgitSample.ino:6
6       void test_function() {
(gdb) c
Continuing.

Breakpoint 1, test_function () at T:\tCode\GDB_DBG\GDBgitSample/GDBgitSample.ino:6
6       void test_function() {
(gdb) p mark
$2 = 1
(gdb) c
Continuing.

Breakpoint 1, test_function () at T:\tCode\GDB_DBG\GDBgitSample/GDBgitSample.ino:6
6       void test_function() {
(gdb)

AWESOME NOTE - GDB has AUTOCOMPLETE! It has the symbol list so typing "b tes" then "TAB" will complete to "b test_function()"

[defragster]

Note sure what the installer does? Tie in Python for 'add in commands' that don't seem to work now ? - digitalRead, digitalWrite, ...

The above works without Python or anything else 'installed' except putting the github code into (sketchbook)\libraries.

Now to learn useful GDB commands:
some here:

c or continue: Debugger will continue executing until the next break point.
n or next: Debugger will execute the next line as single instruction.
s or step: Same as next, but does not treats function as a single instruction, instead goes into the function and executes it line by line.

l – list
p – print
c – continue
s – step
ENTER: pressing enter key would execute the previously executed command again.
l command: Use gdb command l or list to print the source code in the debug mode. Use l line-number to view a specific line number (or) l function to view a specific function.
bt: backtrack – Print backtrace of all stack frames, or innermost COUNT frames.
quit – Exit from the gdb debugger.

And from:

GDB offers a big list of commands, however the following commands are the ones used most frequently:
b main - Puts a breakpoint at the beginning of the program
b - Puts a breakpoint at the current line
b N - Puts a breakpoint at line N
b +N - Puts a breakpoint N lines down from the current line
b fn - Puts a breakpoint at the beginning of function "fn"
d N - Deletes breakpoint number N
info break - list breakpoints
r - Runs the program until a breakpoint or error
c - Continues running the program until the next breakpoint or error
f - Runs until the current function is finished
s - Runs the next line of the program
s N - Runs the next N lines of the program
n - Like s, but it does not step into functions
u N - Runs until you get N lines in front of the current line
p var - Prints the current value of the variable "var"
bt - Prints a stack trace
u - Goes up a level in the stack
d - Goes down a level in the stack
q - Quits gdb

And another list :: yolinux.com/TUTORIALS/GDB-Commands.html

[jonr]

Does anyone know how gdb determines where the IVT (vector table) is located? gdb thinks 0x60001000, not knowing that it has been relocated.

[defragster]

On Windows ran :: teensy_debug.exe ... BAD IDEA? ... as used here it with Command Line build from TSET.

It asked for Arduino folder as it is a ZIP install

It completed and Nothing would Build! Errors and what did build failed upload?

Rebooted computer and deleted that IDE folder and copied over a saved IDE 1.8.12 folder, Reinstalled TD 1.52 to assure current, Re-Integrated TyCommander and all is well.

Not sure what the EXE is doing in edit of the IDE menu system - IDE file edit seemed to fail as used here it with Command Line build from TSET.

It didn't throw build errors I saw - but resulted in unusable HEX for T_4 and 4.1 as I saw it.

Assume edits are in platform.txt? Not sure what edits as I trashed that before looking. Any details might help a fix.

[defragster]

Edited github sketch to try to write a value to Address 0x0 and it faults when I send over Serial USB characters.

Was running the sketch normally - then I hit SerMon 'Send' and it stops and shows the line number and 'l' lists the code - very handy.

Just compiled with the #include, started GDB, pointed to the COM21 port, 'c' to continue started the code that stopped on Halt in setup, then his 'Send' and it tops and tells me where, and 'l' gives context code:

Reading symbols from T:\TEMP\\arduino_build_GDBgitSample.ino\GDBgitSample.ino.elf...done.
(gdb)  target remote \\.\com21
Remote debugging using \\.\com21
setup () at T:\tCode\GDB_DBG\GDBgitSample/GDBgitSample.ino:23
23      }
(gdb) c
Continuing.

Program received signal SIGSEGV, Segmentation fault.
0x00000172 in loop () at T:\tCode\GDB_DBG\GDBgitSample/GDBgitSample.ino:32
32              *f0=0;
(gdb) l
27        Serial.println(mark);
28        delay(1000);
29        while ( Serial.available() ) {
30              int*f0=0;
31              Serial.read();
32              *f0=0;
33        }
34      }(gdb)

[jonr]

Nice, ftrias implemented gdb commands like "monitor digitalWrite(13,1)" (this turns on the LED). Also "monitor call" to call any function in your code.

[defragster]

Nice, ftrias implemented gdb commands like "monitor digitalWrite(13,0)" (this turns on the LED). Also "monitor call" to call any function in your code.

That's with Python version running right? Tried that on Windows where Python not native or right with included xxxGDB-py.exe and they are not known.

<EDIT> :: It does work with proper syntax >> (gdb) monitor digitalWrite(13,1)

[jonr]

I don't use the python stuff, but here is proof:

(gdb) monitor digitalWrite(13,1)
(gdb) monitor foo()
Target does not support this command.
(gdb)

[defragster]

I don't use the python stuff, but here is proof:

(gdb) monitor digitalWrite(13,1)
(gdb) monitor foo()
Target does not support this command.
(gdb)

It does work! I didn't put 'monitor' on the front - I worked from the summary list of commands - not reading the leading example in the text before

[defragster]

@ftrias - very good work indeed! That's what multimon is for - running on one screen and debugging on the other. Very nice for catching faults and showing where and even having the code on hand - even CORES code!

Just glanced at the .h file - adding an ASSERT() would be a nice 'selective halt()' and the name() is free :: T:\tCode\TLAjr\testOut\testOut.ino:12:12: error: 'ASSERT' was not declared in this scope

#define ASSERT(expr)   if (!(expr)) halt()

Started a T_4 and a 4.1 working together - can compile both for debug and each connects to a working GDB session on the same Windows machine. TyCommander great for giving the COM#'s {from UI} and disable serial on debug ports and giving a SerMon for each active default SerialSpew. Would be nicer if it reported a Com# to command line - but it programs by user selection - then auto programs based on filename ( including Teensy model ) for subsequent uploads - and doesn't return a COM#.

Compiling with :#pragma GCC optimize ("O0") reduced the T_4's trivial loop() toggle to 5.8M from 9.2M - that was without debugger library included or activated in setup().

That T_4 code is just putting out pulses the T_4.1 is looking at, the T_4.1 under GDB had no trouble counting them as before.

A breakpoint was placed on loop() - so only one count per 'c' > continue. Using the counted "ignore 1 10000" to have it skip 10K hits on the breakpoint. It ran for some short time - sending out only 10 counts per second instead of 5M because of the breakpoint testing overhead - as seen by the T_4.1. Then it stopped cycling and sending counts at all as the T_4.1 - under GDB but running freely. When the T_4 was restarted it resumed showing expected counts. So the 10,000 counts to ignore were too much for something and what would have taken the small part of a second {5,800,000/10,000} instead would have been over 1,000 seconds.

[ftrias]

A breakpoint was placed on loop() - so only one count per 'c' > continue. Using the counted "ignore 1 10000" to have it skip 10K hits on the breakpoint. It ran for some short time - sending out only 10 counts per second instead of 5M because of the breakpoint testing overhead - as seen by the T_4.1. Then it stopped cycling and sending counts at all as the T_4.1 - under GDB but running freely. When the T_4 was restarted it resumed showing expected counts. So the 10,000 counts to ignore were too much for something and what would have taken the small part of a second {5,800,000/10,000} instead would have been over 1,000 seconds.

Breakpoints are slow because TeensyDebug only processes GDB commands every 5ms. When a breakpoint is hit, Teensy notifies GDB right away. GDB sends a number of commands (clear breakpoints, get registers, set breakpoints, continue). Each one is processed every 5 ms. Thus, we're looking at at least 20 ms or more.

The reason it processes on a timer is because Teensy doesn't have interrupt or event driven serial. So we must do a poll and this could slow down the program.

Maybe since T4 is so fast, I can cut down the interval from 5ms.

[KurtE]

Sound good,

I will get back to this soon, right now busy debugging something

Wonder how your stuff would work with VisualGDB? https://visualgdb.com/
I was using this a long while ago when I was debugging code on RPIs and Odroid

[defragster]

Just installed IDE 1.8.13 and Beta 1 of TD 1.53 and same GDB on Windows working.

I commented the :: //#pragma GCC optimize ("O0")
>> And it still 'seems' to work - without the drop from 9M to 5M loop()'s per second. Is that optimize OFF really helpful if not looking at the assemebly? It only affects the INO - all the CORES and libs are still compile FASTEST in my case - also changing default optimization can change the problem.

@ftrias - indeed on the 1062's at 600 MHz it might be nice to try faster than 5ms interval.
> Note: {KurtE just improved this for TD 1.53} But every exit of loop() or call to delay() calls yield() which ( used to ) iterate over Serial USB and ALL Serial#'s for a .available() check to call the serialEvent()'s. This does add overhead to standard loop() processing - in fact often to avoid this I've added { like in current sketches } a "void yield(){}" to override that "weak" yield required by Arduino for those Event()'s.
>> So adding: if ( GDB_Serial.available() ) polling might not show much effect over normal operation - but would require code added to loop()?

*Note: @luni has a TeensyTimerTool library that can add timer based timers, or loop() polled TCK timers. For the loop() version he has an option to redefine the default yield() so that can happen without adding the poll code directly into loop() - that code keys off of this :: luni64/TeensyTimerTool/blob/master/src/defaultConfig.h#L61 So when a sketch doesn't use any Event()'s yield() can be replaced perhaps to do GDB polling.

[ftrias]

I updated Github today with these fixes:

1. Reduce serial polling to 0.5 milliseconds. This should increase responsiveness. Will look at alternatives @defragster suggested.
2. Fix `p func()` that allows you to call most functions from GDB when it is stopped.
3. Improve install script.
4. Fix numerous bugs with SP and stack handling. It now steps into and over function calls more consistently.

It only really works with Teensy 4. I'm not going to bother getting Teensy 3 working.

[defragster]

Wow - Fresh Code! ... it was 8 minutes old.

Put it on the T_4/4.1 combo - each under GDB

Broke into the T_4 and did

b loop
ignore 1 1000
c

And the count was now passing 60/sec instead of just 10 { though full speed is 9,821,585/sec } , and it worked to complete 1,000

Just started repeat of prior ignore 10,000 ... it ended like this:

(gdb) b loop
Breakpoint 3 at 0xec: file T:\tCode\TLAjr\testOut/testOut.ino, line 18.
(gdb) ignore 3 10000
Will ignore next 10000 crossings of breakpoint 3.
(gdb) c
Continuing.
Remote connection closed
...
(gdb) c
The program is not being run.
(gdb)  target remote \\.\com26
Remote debugging using \\.\com26
loop () at T:\tCode\TLAjr\testOut/testOut.ino:18
18      void loop() {
(gdb) c
Continuing.
Remote connection closed
/home/build/work/GCC-5-build/src/gdb/gdb/target.c:2676: internal-error: Can't determine the current address space of thread process 42000

A problem internal to GDB has been detected,
further debugging may prove unreliable.
Quit this debugging session? (y or n) n

This is a bug, please report it.  For instructions, see:
<http://www.gnu.org/software/gdb/bugs/>.

/home/build/work/GCC-5-build/src/gdb/gdb/target.c:2676: internal-error: Can't determine the current address space of thread process 42000

A problem internal to GDB has been detected,
further debugging may prove unreliable.
Create a core file of GDB? (y or n) y
Command aborted.
(gdb)  target remote \\.\com26
Remote debugging using \\.\com26
millis () at T:\arduino-1.8.13\hardware\teensy\avr\cores\teensy4/core_pins.h:1566
1566            return systick_millis_count;
(gdb) monitor restart
Remote connection closed
(gdb)

Didn't quit GDB, but reconnected and did restart and both Teensy are running fine.

[wwatson]

OK I am really embarrassed here I was using gdb with the Motorola 68332 back in the 90's and cannot remember how it all worked. I was part of the EFI332 forum which used the 68332 for a fuel injection system that they were developing. I still have an old computer with all of the software and hardware for it. It used a BDM interface to communicate with the 68332 CPU board. That is what led me to the Teensy. In those days they were using the PC parallel printer port through an interface board to upload and debug a engine control system. I bought a Teensy 2.0 to do the same thing as the parallel printer port interface did. It actually worked. But I never got past using it with the CPU board and the IO board. I still have not assembled the driver boards I have. Never had the time to completely set this up on my 1980 Dodge Mirada.

Then things changed to the Megasquirt system, which is awesome.

To make a long story short I need to know if the software is uploaded into the T4.x by Arduino or GDB?

Silly me

[defragster]

Arduino ... TeensyLoader (or TyCommander, other) load the hex onto the Teensy as normal and it runs monitoring the port indicated in the sketch - sketch starts and runs as normal/programmed ... watching for connect from GDB.

[wwatson]

Thanks @defragster - I understand now. I am seeing the 'mark' variable changing values. This brings back cool memories