runsisi's

technical notes

gdb 基础功能使用

2019-01-21 runsisi#debug#gdb

Ubuntu 下的 gdb 可以通过 show configuraion 查看 gdb 自身编译时的配置(CentOS 7.x 下的 gdb 竟然没有这个命令):

~$ gdb -n -batch -ex 'show configuration' | grep python
             --with-python=/usr (relocatable)

上述命令实际上只是利用了 -ex 选项可以让 gdb 执行指定的命令而已,因此也可以在 gdb 交互终端中执行 show configuration 命令:

(gdb) show configuration
This GDB was configured as follows:
   configure --host=x86_64-linux-gnu --target=x86_64-linux-gnu
             --with-auto-load-dir=$debugdir:$datadir/auto-load
             --with-auto-load-safe-path=$debugdir:$datadir/auto-load
             --with-expat
             --with-gdb-datadir=/usr/share/gdb (relocatable)
             --with-jit-reader-dir=/usr/lib/gdb (relocatable)
             --without-libunwind-ia64
             --with-lzma
             --with-python=/usr (relocatable)
             --without-guile
...

调用函数:

$ sudo gdb -p <pid> -ex 'call malloc_trim(0)' -ex 'set confirm off' -ex 'quit'

或:

$ sudo gdb -p <pid> -batch -ex 'call malloc_trim(0)'

调试 coredump 文件:

~# file core-sig6-pid3599883-uid167
core-sig6-pid3599883-uid167: ELF 64-bit LSB core file x86-64, version 1 (SYSV), SVR4-style, from '/usr/bin/ceph-osd -f --cluster ceph --id 1 --setuser ceph --setgroup ceph'
~# gdb -c core-sig6-pid3599883-uid167 /usr/bin/ceph-osd

c++ mangled 名字到原始名字(gdb 8.0+):

~$ c++filt  _ZN8librados5IoCtx3cctEv
librados::IoCtx::cct()

(gdb) demangle -l c++ _ZN8librados5IoCtx3cctEv
librados::IoCtx::cct()

去掉让人讨厌的 ---Type <return> to continue, or q <return> to quit--- 提示:

(gdb) set height 0

或者

(gdb) set pagination off

设置程序参数:

(gdb) set args ARGS
(gdb) show args

设置程序参数并(重)启动:

(gdb) r ARGS

设置程序环境变量:

(gdb) set environment KEY VALUE
(gdb) show environment

设置汇编语法类型(默认为 att 语法):

(gdb) set disassembly-flavor
(gdb) show disassembly-flavor

命令行上反汇编函数:

$ gdb /path/to/exe -batch -ex 'disassemble main'
$ gdb /path/to/exe -batch -ex 'disassemble 0x6c0'

在指定地址处设置断点:

(gdb) b *0x55555555484d

字符串条件断点:

(gdb) b object_property_find if strcmp(name, "drive") == 0

或者

(gdb) b object_property_find if $_streq(name, "drive")

将已有断点设置为条件断点:

(gdb) info b
Num     Type           Disp Enb Address            What
1       breakpoint     keep y   0x00007ffff7a0101d in PyImport_ImportModule at Python/import.c:1262
        breakpoint already hit 1 time
(gdb) cond 1 $_streq(name, "x")   
(gdb) info b
Num     Type           Disp Enb Address            What
1       breakpoint     keep y   0x00007ffff7a0101d in PyImport_ImportModule at Python/import.c:1262
        stop only if $_streq(name, "x")
        breakpoint already hit 1 time
(gdb) cond 1 strcmp(name, "x") == 0   
(gdb) info b
Num     Type           Disp Enb Address            What
1       breakpoint     keep y   0x00007ffff7a0101d in PyImport_ImportModule at Python/import.c:1262
        stop only if strcmp(name, "x") == 0
        breakpoint already hit 1 time

将条件断点重置为普通断点:

(gdb) cond 1
Breakpoint 1 now unconditional.
(gdb) info b
Num     Type           Disp Enb Address            What
1       breakpoint     keep y   0x00007ffff7a0101d in PyImport_ImportModule at Python/import.c:1262

当值改变时暂停程序执行:

(gdb) watch EXPR

查看、禁用、启用、删除断点(包括普通断点、条件断点、watch):

(gdb) info b
(gdb) disable [NUM]
(gdb) enable [NUM]
(gdb) d [NUM]

执行至指定行:

(gdb) adv LINENO

源代码级单步调试:

(gdb) n
(gdb) s

汇编级单步调试:

(gdb) ni
(gdb) si

查看寄存器:

(gdb) info r
(gdb) info r rip

执行完当前函数并返回至调用者:

(gdb) fin

终止当前函数执行并返回至调用者(可以设置返回值):

(gdb) ret [VAL]

当程序暂停执行时打印值:

(gdb) disp/fmt EXPR
(gdb) info display
(gdb) undisplay [NUM]

查看变量值:

(gdb) p/fmt VAR

查看内存内容:

(gdb) x/fmt ADDR

查看类型定义:

(gdb) ptype EXPR

基本计算:

(gdb) p/x 0x5555555546c6 + 0x20090a
(gdb) p/d 2 * 5

打印 gdb 工作目录:

(gdb) pwd

改变 gdb 工作目录:

(gdb) cd DIR

加载 gdb 脚本:

(gdb) cd DIR/TO/FILE
(gdb) source FILE

设置 verbose:

(gdb) set verbose on

查看符号搜索路径:

(gdb) show debug-file-directory 
The directory where separate debug symbols are searched for is "/usr/lib/debug".

设置被调试程序工作目录(程序启动之前设置):

(gdb) set cwd DIR

查看被调试程序工作目录:

(gdb) show cwd

进入 shell 或执行 shell 命令:

(gdb) shell [CMD]

进入 Python 或运行 Python 代码:

(gdb) python [CODE]

查看 C++ 模板参数类型(实际上就是打印当前栈帧):

(gdb) f
#0  boost::fast_pool_allocator<std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::pair<librbd::xImageInfo, int>
 > >, boost::default_user_allocator_new_delete, std::mutex, 32u, 0u>::allocate (n=1) at /usr/include/boost/pool/pool_alloc.hpp:446

打印结构体大小:

(gdb) p sizeof(std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::pair<librbd::xImageInfo, int>> >)
A syntax error in expression, near `const, std::pair<librbd::xImageInfo, int>> >)'.
(gdb) p sizeof(std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::pair<librbd::xImageInfo, int > > >)
$10 = 376
(gdb) p sizeof(std::pair<librbd::xImageInfo, int>)
$11 = 312

查看被调试进程当前加载的动态库:

(gdb) info sharedlibrary
From                To                  Syms Read   Shared Object Library
0x00007ffff7dd5f10  0x00007ffff7df4b20  Yes         /lib64/ld-linux-x86-64.so.2
0x00007ffff78c2380  0x00007ffff7a91207  Yes         /opt/runsisi/python3/lib/libpython3.6d.so.1.0
0x00007ffff74902d0  0x00007ffff7608c3c  Yes         /lib/x86_64-linux-gnu/libc.so.6
0x00007ffff7255bb0  0x00007ffff72640f1  Yes         /lib/x86_64-linux-gnu/libpthread.so.0
0x00007ffff704ce50  0x00007ffff704dbde  Yes         /lib/x86_64-linux-gnu/libdl.so.2
0x00007ffff6e49e70  0x00007ffff6e4a93a  Yes         /lib/x86_64-linux-gnu/libutil.so.1
0x00007ffff6ab6a80  0x00007ffff6b752f5  Yes         /lib/x86_64-linux-gnu/libm.so.6
0x00007ffff5fb1820  0x00007ffff614d5de  Yes         /home/runsisi/working/test/pyleak/build/x.cpython-36d-x86_64-linux-gnu.so
0x00007ffff5c72490  0x00007ffff5d21b4e  Yes (*)     /usr/lib/x86_64-linux-gnu/libstdc++.so.6
0x00007ffff59d0ac0  0x00007ffff59e13fd  Yes (*)     /lib/x86_64-linux-gnu/libgcc_s.so.1
(*): Shared library is missing debugging information.

查看符号定义:

(gdb) info sym operator new
operator new(unsigned long, std::nothrow_t const&) in section .text of /home/runsisi/working/test/pyleak/build/jemalloc/artifacts/lib/libjemalloc.so.2
(gdb) info sym operator delete
operator delete(void*, unsigned long) in section .text of /home/runsisi/working/test/pyleak/build/jemalloc/artifacts/lib/libjemalloc.so.2

需要特别注意的是,随着动态库的加载 info sym 的信息可能会解析到最新的符号,但实际上程序运行实际上仍然使用的是符号最初解析得到的定义:

(gdb) info sym operator new
operator new(unsigned long)@plt in section .plt of /home/runsisi/working/test/pyleak/build/test_crash
(gdb) info sym operator delete
operator delete(void*, unsigned long)@plt in section .plt of /home/runsisi/working/test/pyleak/build/test_crash
(gdb) n
(gdb) n
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
(gdb) info sym operator new
operator new(unsigned long, std::nothrow_t const&) in section .text of /home/runsisi/working/test/pyleak/build/jemalloc/artifacts/lib/libjemalloc.so.2
(gdb) info sym operator delete
operator delete(void*, unsigned long) in section .text of /home/runsisi/working/test/pyleak/build/jemalloc/artifacts/lib/libjemalloc.so.2

给被调试进程发送信号:

help signal
Continue program with the specified signal.
Usage: signal SIGNAL
The SIGNAL argument is processed the same as the handle command.

An argument of "0" means continue the program without sending it a signal.
This is useful in cases where the program stopped because of a signal,
and you want to resume the program while discarding the signal.

In a multi-threaded program the signal is delivered to, or discarded from,
the current thread only.
(gdb) signal SIGINT
Continuing with signal SIGINT.
[Thread 0x7f45fc7ee700 (LWP 20897) exited]
[Thread 0x7f45fcfef700 (LWP 20896) exited]
...

查看源代码:

(gdb) info sources
(gdb) info source

很多时候,我们允许的程序我们有源代码,但是是在另外一台机器执行,或者在同一台机器执行,但是源代码位置相对编译时的路径已经发生了变化,这时候可以使用 set substitute-path 命令更改源代码的路径:

(gdb) help set substitute-path
Usage: set substitute-path FROM TO
Add a substitution rule replacing FROM into TO in source file names.
If a substitution rule was previously set for FROM, the old rule
is replaced by the new one.
(gdb) info source
Current source file is rados_threadsafe.cc
Compilation directory is /home/runsisi
Located in /home/runsisi/rados_threadsafe.cc
Source language is c++.
Compiled with DWARF 2 debugging format.
Includes preprocessor macro info.
(gdb) set substitute-path /home/runsisi/ /home/tecs/runsisi/src

查找函数(支持标准的正则表达式过滤):

~# gdb ceph-mgr
(gdb) info functions .*shutdown.*
All functions matching regular expression ".*shutdown.*":

File /usr/src/debug/ceph-12.2.9/src/common/ceph_crypto.cc:
void ceph::crypto::shutdown(bool);

File /usr/src/debug/ceph-12.2.9/src/mgr/DaemonServer.cc:
void DaemonServer::shutdown();

File /usr/src/debug/ceph-12.2.9/src/mon/MonClient.cc:
void MonClient::shutdown();
...
(gdb) info functions AdminSocket::shutdown.*
All functions matching regular expression "AdminSocket::shutdown.*":

File /usr/src/debug/ceph-12.2.9/src/common/admin_socket.cc:
void AdminSocket::shutdown();

反汇编函数(如果安装了 debuginfo 包,可以先用 layout split 命令同时打开源代码窗口和汇编窗口,然后再用 disassemble 命令进行反汇编):

(gdb) disassemble AdminSocket::shutdown
Dump of assembler code for function AdminSocket::shutdown():
   0x000000000044cac0 <+0>:     push   %r15
   0x000000000044cac2 <+2>:     push   %r14
   0x000000000044cac4 <+4>:     push   %r13
   0x000000000044cac6 <+6>:     push   %r12
   0x000000000044cac8 <+8>:     push   %rbp
   0x000000000044cac9 <+9>:     push   %rbx
   0x000000000044caca <+10>:    mov    %rdi,%rbx
   0x000000000044cacd <+13>:    sub    $0x1b8,%rsp
   0x000000000044cad4 <+20>:    mov    0x7861dd(%rip),%rbp        # 0xbd2cb8
   0x000000000044cadb <+27>:    mov    %fs:0x28,%rax
   0x000000000044cae4 <+36>:    mov    %rax,0x1a8(%rsp)
   0x000000000044caec <+44>:    xor    %eax,%eax
   0x000000000044caee <+46>:    lea    0x18(%rbp),%rax
   0x000000000044caf2 <+50>:    mov    %rax,0x20(%rsp)
   0x000000000044caf7 <+55>:    mov    0x40(%rdi),%eax
   0x000000000044cafa <+58>:    test   %eax,%eax
   0x000000000044cafc <+60>:    js     0x44cdb4 <AdminSocket::shutdown()+756>
   0x000000000044cb02 <+66>:    mov    0x28(%rdi),%rcx
...

通过函数名获取地址:

(gdb) info address AdminSocket::shutdown
Symbol "AdminSocket::shutdown()" is a function at address 0x44cac0.

从地址获取函数:

(gdb) info line *0x44cac0
Line 597 of "/usr/src/debug/ceph-12.2.9/src/common/admin_socket.cc" starts at address 0x44cac0 <AdminSocket::shutdown()> and ends at 0x44cad4 <AdminSocket::shutdown()+20>.
(gdb) info line AdminSocket::shutdown
Line 597 of "/usr/src/debug/ceph-12.2.9/src/common/admin_socket.cc" starts at address 0x44cac0 <AdminSocket::shutdown()> and ends at 0x44cad4 <AdminSocket::shutdown()+20>.
(gdb) info line *(0x44cac0 + 5)
Line 597 of "/usr/src/debug/ceph-12.2.9/src/common/admin_socket.cc" starts at address 0x44cac0 <AdminSocket::shutdown()> and ends at 0x44cad4 <AdminSocket::shutdown()+20>.

虽然 addr2line 也可以通过地址得到函数信息,但是它只能处理没有经过 strip 处理的可执行文件,对于 CentOS/Ubuntu 这种一般都进行 strip 处理然后生成独立的 debuginfo/dbg 安装包的系统,addr2line 基本没有什么用:

~# addr2line -e /usr/bin/ceph-mgr 0x44cac0
??:0

读取 ELF 文件(包括 elf 可执行文件和动态库)的 build-id(in ELF .note section):

~$ eu-readelf -n /usr/bin/osdmaptool

Note section [ 2] '.note.ABI-tag' of 32 bytes at offset 0x254:
  Owner          Data size  Type
  GNU                   16  VERSION
    OS: Linux, ABI: 2.6.32

Note section [ 3] '.note.gnu.build-id' of 36 bytes at offset 0x274:
  Owner          Data size  Type
  GNU                   20  GNU_BUILD_ID
    Build ID: 08c29ee8586fdbe3d0958c5df3167cab5d8e9eec

~$ ll /usr/lib/debug/.build-id/08/c29ee8586fdbe3d0958c5df3167cab5d8e9eec*
lrwxrwxrwx. 1 root root 26 Dec  1 15:07 /usr/lib/debug/.build-id/08/c29ee8586fdbe3d0958c5df3167cab5d8e9eec -> ../../../../bin/osdmaptool
lrwxrwxrwx. 1 root root 30 Dec  1 15:07 /usr/lib/debug/.build-id/08/c29ee8586fdbe3d0958c5df3167cab5d8e9eec.debug -> ../../usr/bin/osdmaptool.debug

~$ ll /usr/lib/debug/usr/bin/osdmaptool.debug
-r--r--r--. 1 root root 5115216 Nov 30 22:39 /usr/lib/debug/usr/bin/osdmaptool.debug

~$ rpm -qf /usr/lib/debug/.build-id/08/c29ee8586fdbe3d0958c5df3167cab5d8e9eec
ceph-debuginfo-12.2.9-2.5.g00b0f3b.el7.x86_64
~$ rpm -qf /usr/lib/debug/.build-id/08/c29ee8586fdbe3d0958c5df3167cab5d8e9eec.debug
ceph-debuginfo-12.2.9-2.5.g00b0f3b.el7.x86_64
~$ rpm -qf /usr/lib/debug/usr/bin/osdmaptool.debug
ceph-debuginfo-12.2.9-2.5.g00b0f3b.el7.x86_64

关于 backtrace 的地址:

I’ve had a look at files backtrace.c and backtracesyms.c files in glibc source code (git://sourceware.org/git/glibc.git, commit 2482ae433a4249495859343ae1fba408300f2c2e). Assuming I haven’t misread/misunderstood things: backtrace() itself looks like it will only give you symbol addresses as they are at runtime, which I think means you need the library load address as it was from pmap or similar. However, backtrace_symbols() recalculates things so that the addresses are relative to the shared library ELF, and not the process at runtime, which is really convenient. It means you don’t need information from pmap.

如果当前调试进程存在 fork 调用,设置是否继续调试当前进程,还是调试子进程:

(gdb) help set follow-fork-mode
Set debugger response to a program call of fork or vfork.
A fork or vfork creates a new process.  follow-fork-mode can be:
  parent  - the original process is debugged after a fork
  child   - the new process is debugged after a fork
The unfollowed process will continue to run.
By default, the debugger will follow the parent process.

同时,还有一个对 exec 调用的设置:

(gdb) help set follow-exec-mode
Set debugger response to a program call of exec.
An exec call replaces the program image of a process.

follow-exec-mode can be:

  new - the debugger creates a new inferior and rebinds the process
to this new inferior.  The program the process was running before
the exec call can be restarted afterwards by restarting the original
inferior.

  same - the debugger keeps the process bound to the same inferior.
The new executable image replaces the previous executable loaded in
the inferior.  Restarting the inferior after the exec call restarts
the executable the process was running after the exec call.

By default, the debugger will use the same inferior.

打开或关闭源代码窗口(tui):

ctrl - x - a

切换源代码窗口与命令交互窗口:

ctrl -x -o

刷新终端显示:

ctrl - l

窗口的更多控制可以使用 layout 命令:

(gdb) help layout
Change the layout of windows.
Usage: layout prev | next | <layout_name> 
Layout names are:
   src   : Displays source and command windows.
   asm   : Displays disassembly and command windows.
   split : Displays source, disassembly and command windows.
   regs  : Displays register window. If existing layout
           is source/command or assembly/command, the 
           register window is displayed. If the
           source/assembly/command (split) is displayed, 
           the register window is displayed with 
           the window that has current logical focus.

参考资料

[1] How do I set a conditional breakpoint in gdb, when char* x points to a string whose value equals “hello”?

https://stackoverflow.com/questions/4183871/how-do-i-set-a-conditional-breakpoint-in-gdb-when-char-x-points-to-a-string-wh

[2] Convenience Functions

https://sourceware.org/gdb/current/onlinedocs/gdb/Convenience-Funs.html#Convenience-Funs

[3] 100-gdb-tips

https://github.com/hellogcc/100-gdb-tips

[4] gdb Debugging Full Example (Tutorial)

http://www.brendangregg.com/blog/2016-08-09/gdb-example-ncurses.html

[5] Tracing and Disassembling a Program

https://www.usna.edu/Users/cs/aviv/classes/si485h/s17/units/02/unit.html

[6] How to map function address to function in *.so files

https://stackoverflow.com/questions/7556045/how-to-map-function-address-to-function-in-so-files

[7] Have GDB show the type of a template argument

https://stackoverflow.com/questions/37616367/have-gdb-show-the-type-of-a-template-argument

[8] gdb: size of a struct that isn’t in context?

https://stackoverflow.com/questions/28982534/gdb-size-of-a-struct-that-isnt-in-context

[9] Debugging with GDB

https://sourceware.org/gdb/current/onlinedocs/gdb/index.html