Ubuntu 下的 gdb 可以通过 show configuraion 查看 gdb 自身编译时的配置（CentOS 7.x 下的 gdb 竟然没有这个命令）：
~$ gdb -n -batch -ex 'show configuration' | grep python
--with-python=/usr (relocatable)
上述命令实际上只是利用了 -ex 选项可以让 gdb 执行指定的命令而已，因此也可以在 gdb 交互终端中执行 show configuration 命令：
(gdb) show configuration
This GDB was configured as follows:
configure --host=x86_64-linux-gnu --target=x86_64-linux-gnu
--with-auto-load-dir=$debugdir:$datadir/auto-load
--with-auto-load-safe-path=$debugdir:$datadir/auto-load
--with-expat
--with-gdb-datadir=/usr/share/gdb (relocatable)
--with-jit-reader-dir=/usr/lib/gdb (relocatable)
--without-libunwind-ia64
--with-lzma
--with-python=/usr (relocatable)
--without-guile
...
调用函数：
$ sudo gdb -p <pid> -ex 'call malloc_trim(0)' -ex 'set confirm off' -ex 'quit'
或：
$ sudo gdb -p <pid> -batch -ex 'call malloc_trim(0)'
调试 coredump 文件：
~# file core-sig6-pid3599883-uid167
core-sig6-pid3599883-uid167: ELF 64-bit LSB core file x86-64, version 1 (SYSV), SVR4-style, from '/usr/bin/ceph-osd -f --cluster ceph --id 1 --setuser ceph --setgroup ceph'
~# gdb -c core-sig6-pid3599883-uid167 /usr/bin/ceph-osd
c++ mangled 名字到原始名字（gdb 8.0+）：
~$ c++filt _ZN8librados5IoCtx3cctEv
librados::IoCtx::cct()
(gdb) demangle -l c++ _ZN8librados5IoCtx3cctEv
librados::IoCtx::cct()
去掉让人讨厌的 ---Type <return> to continue, or q <return> to quit--- 提示：
(gdb) set pagination off
打开 pretty printing：
(gdb) set print pretty on
设置程序参数：
(gdb) set args ARGS
(gdb) show args
设置程序参数并（重）启动：
(gdb) r ARGS
设置程序环境变量：
(gdb) set environment KEY VALUE
(gdb) show environment
设置汇编语法类型（默认为 att 语法）：
(gdb) set disassembly-flavor
(gdb) show disassembly-flavor
命令行上反汇编函数：
$ gdb /path/to/exe -batch -ex 'disassemble main'
$ gdb /path/to/exe -batch -ex 'disassemble 0x6c0'
在指定地址处设置断点：
(gdb) b *0x55555555484d
字符串条件断点：
(gdb) b object_property_find if strcmp(name, "drive") == 0
或者
(gdb) b object_property_find if $_streq(name, "drive")
将已有断点设置为条件断点：
(gdb) info b
Num Type Disp Enb Address What
1 breakpoint keep y 0x00007ffff7a0101d in PyImport_ImportModule at Python/import.c:1262
breakpoint already hit 1 time
(gdb) cond 1 $_streq(name, "x")
(gdb) info b
Num Type Disp Enb Address What
1 breakpoint keep y 0x00007ffff7a0101d in PyImport_ImportModule at Python/import.c:1262
stop only if $_streq(name, "x")
breakpoint already hit 1 time
(gdb) cond 1 strcmp(name, "x") == 0
(gdb) info b
Num Type Disp Enb Address What
1 breakpoint keep y 0x00007ffff7a0101d in PyImport_ImportModule at Python/import.c:1262
stop only if strcmp(name, "x") == 0
breakpoint already hit 1 time
将条件断点重置为普通断点：
(gdb) cond 1
Breakpoint 1 now unconditional.
(gdb) info b
Num Type Disp Enb Address What
1 breakpoint keep y 0x00007ffff7a0101d in PyImport_ImportModule at Python/import.c:1262
当值改变时暂停程序执行：
(gdb) watch EXPR
查看、禁用、启用、删除断点（包括普通断点、条件断点、watch）：
(gdb) info b
(gdb) disable [NUM]
(gdb) enable [NUM]
(gdb) d [NUM]
执行至指定行：
(gdb) adv LINENO
源代码级单步调试：
(gdb) n
(gdb) s
汇编级单步调试：
(gdb) ni
(gdb) si
查看寄存器：
(gdb) info r
(gdb) info r rip
执行完当前函数并返回至调用者：
(gdb) fin
终止当前函数执行并返回至调用者（可以设置返回值）：
(gdb) ret [VAL]
当程序暂停执行时打印值：
(gdb) disp/fmt EXPR
(gdb) info display
(gdb) undisplay [NUM]
查看变量值：
(gdb) p/fmt VAR
查看内存内容：
(gdb) x/fmt ADDR
查看类型定义：
(gdb) ptype EXPR
基本计算：
(gdb) p/x 0x5555555546c6 + 0x20090a
(gdb) p/d 2 * 5
打印 gdb 工作目录：
(gdb) pwd
改变 gdb 工作目录：
(gdb) cd DIR
加载 gdb 脚本：
(gdb) cd DIR/TO/FILE
(gdb) source FILE
设置 verbose：
(gdb) set verbose on
查看符号搜索路径：
(gdb) show debug-file-directory
The directory where separate debug symbols are searched for is "/usr/lib/debug".
设置被调试程序工作目录（程序启动之前设置）：
(gdb) set cwd DIR
查看被调试程序工作目录：
(gdb) show cwd
进入 shell 或执行 shell 命令：
(gdb) shell [CMD]
进入 Python 或运行 Python 代码：
(gdb) python [CODE]
查看 C++ 模板参数类型（实际上就是打印当前栈帧）：
(gdb) f
#0 boost::fast_pool_allocator<std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::pair<librbd::xImageInfo, int>
> >, boost::default_user_allocator_new_delete, std::mutex, 32u, 0u>::allocate (n=1) at /usr/include/boost/pool/pool_alloc.hpp:446
打印结构体大小：
(gdb) p sizeof(std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::pair<librbd::xImageInfo, int>> >)
A syntax error in expression, near `const, std::pair<librbd::xImageInfo, int>> >)'.
(gdb) p sizeof(std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::pair<librbd::xImageInfo, int > > >)
$10 = 376
(gdb) p sizeof(std::pair<librbd::xImageInfo, int>)
$11 = 312
查看被调试进程当前加载的动态库：
(gdb) info sharedlibrary
From To Syms Read Shared Object Library
0x00007ffff7dd5f10 0x00007ffff7df4b20 Yes /lib64/ld-linux-x86-64.so.2
0x00007ffff78c2380 0x00007ffff7a91207 Yes /opt/runsisi/python3/lib/libpython3.6d.so.1.0
0x00007ffff74902d0 0x00007ffff7608c3c Yes /lib/x86_64-linux-gnu/libc.so.6
0x00007ffff7255bb0 0x00007ffff72640f1 Yes /lib/x86_64-linux-gnu/libpthread.so.0
0x00007ffff704ce50 0x00007ffff704dbde Yes /lib/x86_64-linux-gnu/libdl.so.2
0x00007ffff6e49e70 0x00007ffff6e4a93a Yes /lib/x86_64-linux-gnu/libutil.so.1
0x00007ffff6ab6a80 0x00007ffff6b752f5 Yes /lib/x86_64-linux-gnu/libm.so.6
0x00007ffff5fb1820 0x00007ffff614d5de Yes /home/runsisi/working/test/pyleak/build/x.cpython-36d-x86_64-linux-gnu.so
0x00007ffff5c72490 0x00007ffff5d21b4e Yes (*) /usr/lib/x86_64-linux-gnu/libstdc++.so.6
0x00007ffff59d0ac0 0x00007ffff59e13fd Yes (*) /lib/x86_64-linux-gnu/libgcc_s.so.1
(*): Shared library is missing debugging information.
查看符号定义：
(gdb) info sym operator new
operator new(unsigned long, std::nothrow_t const&) in section .text of /home/runsisi/working/test/pyleak/build/jemalloc/artifacts/lib/libjemalloc.so.2
(gdb) info sym operator delete
operator delete(void*, unsigned long) in section .text of /home/runsisi/working/test/pyleak/build/jemalloc/artifacts/lib/libjemalloc.so.2
需要特别注意的是，随着动态库的加载 info sym 的信息可能会解析到最新的符号，但实际上程序运行实际上仍然使用的是符号最初解析得到的定义：
(gdb) info sym operator new
operator new(unsigned long)@plt in section .plt of /home/runsisi/working/test/pyleak/build/test_crash
(gdb) info sym operator delete
operator delete(void*, unsigned long)@plt in section .plt of /home/runsisi/working/test/pyleak/build/test_crash
(gdb) n
(gdb) n
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
(gdb) info sym operator new
operator new(unsigned long, std::nothrow_t const&) in section .text of /home/runsisi/working/test/pyleak/build/jemalloc/artifacts/lib/libjemalloc.so.2
(gdb) info sym operator delete
operator delete(void*, unsigned long) in section .text of /home/runsisi/working/test/pyleak/build/jemalloc/artifacts/lib/libjemalloc.so.2
给被调试进程发送信号：
help signal
Continue program with the specified signal.
Usage: signal SIGNAL
The SIGNAL argument is processed the same as the handle command.
An argument of "0" means continue the program without sending it a signal.
This is useful in cases where the program stopped because of a signal,
and you want to resume the program while discarding the signal.
In a multi-threaded program the signal is delivered to, or discarded from,
the current thread only.
(gdb) signal SIGINT
Continuing with signal SIGINT.
[Thread 0x7f45fc7ee700 (LWP 20897) exited]
[Thread 0x7f45fcfef700 (LWP 20896) exited]
...
查看源代码：
(gdb) info sources
(gdb) info source
很多时候，我们允许的程序我们有源代码，但是是在另外一台机器执行，或者在同一台机器执行，但是源代码位置相对编译时的路径已经发生了变化，这时候可以使用 set substitute-path 命令更改源代码的路径：
(gdb) help set substitute-path
Usage: set substitute-path FROM TO
Add a substitution rule replacing FROM into TO in source file names.
If a substitution rule was previously set for FROM, the old rule
is replaced by the new one.
(gdb) info source
Current source file is rados_threadsafe.cc
Compilation directory is /home/runsisi
Located in /home/runsisi/rados_threadsafe.cc
Source language is c++.
Compiled with DWARF 2 debugging format.
Includes preprocessor macro info.
(gdb) set substitute-path /home/runsisi/ /home/runsisi/src
查找函数（支持标准的正则表达式过滤）：
~# gdb ceph-mgr
(gdb) info functions .*shutdown.*
All functions matching regular expression ".*shutdown.*":
File /usr/src/debug/ceph-12.2.9/src/common/ceph_crypto.cc:
void ceph::crypto::shutdown(bool);
File /usr/src/debug/ceph-12.2.9/src/mgr/DaemonServer.cc:
void DaemonServer::shutdown();
File /usr/src/debug/ceph-12.2.9/src/mon/MonClient.cc:
void MonClient::shutdown();
...
(gdb) info functions AdminSocket::shutdown.*
All functions matching regular expression "AdminSocket::shutdown.*":
File /usr/src/debug/ceph-12.2.9/src/common/admin_socket.cc:
void AdminSocket::shutdown();
反汇编函数（如果安装了 debuginfo 包，可以先用 layout split 命令同时打开源代码窗口和汇编窗口，然后再用 disassemble 命令进行反汇编）：
(gdb) disassemble AdminSocket::shutdown
Dump of assembler code for function AdminSocket::shutdown():
0x000000000044cac0 <+0>: push %r15
0x000000000044cac2 <+2>: push %r14
0x000000000044cac4 <+4>: push %r13
0x000000000044cac6 <+6>: push %r12
0x000000000044cac8 <+8>: push %rbp
0x000000000044cac9 <+9>: push %rbx
0x000000000044caca <+10>: mov %rdi,%rbx
0x000000000044cacd <+13>: sub $0x1b8,%rsp
0x000000000044cad4 <+20>: mov 0x7861dd(%rip),%rbp # 0xbd2cb8
0x000000000044cadb <+27>: mov %fs:0x28,%rax
0x000000000044cae4 <+36>: mov %rax,0x1a8(%rsp)
0x000000000044caec <+44>: xor %eax,%eax
0x000000000044caee <+46>: lea 0x18(%rbp),%rax
0x000000000044caf2 <+50>: mov %rax,0x20(%rsp)
0x000000000044caf7 <+55>: mov 0x40(%rdi),%eax
0x000000000044cafa <+58>: test %eax,%eax
0x000000000044cafc <+60>: js 0x44cdb4 <AdminSocket::shutdown()+756>
0x000000000044cb02 <+66>: mov 0x28(%rdi),%rcx
...
通过函数名获取地址：
(gdb) info address AdminSocket::shutdown
Symbol "AdminSocket::shutdown()" is a function at address 0x44cac0.
从地址获取函数：
(gdb) info line *0x44cac0
Line 597 of "/usr/src/debug/ceph-12.2.9/src/common/admin_socket.cc" starts at address 0x44cac0 <AdminSocket::shutdown()> and ends at 0x44cad4 <AdminSocket::shutdown()+20>.
(gdb) info line AdminSocket::shutdown
Line 597 of "/usr/src/debug/ceph-12.2.9/src/common/admin_socket.cc" starts at address 0x44cac0 <AdminSocket::shutdown()> and ends at 0x44cad4 <AdminSocket::shutdown()+20>.
(gdb) info line *(0x44cac0 + 5)
Line 597 of "/usr/src/debug/ceph-12.2.9/src/common/admin_socket.cc" starts at address 0x44cac0 <AdminSocket::shutdown()> and ends at 0x44cad4 <AdminSocket::shutdown()+20>.
虽然 addr2line 也可以通过地址得到函数信息，但是它只能处理没有经过 strip 处理的可执行文件，对于 CentOS/Ubuntu 这种一般都进行 strip 处理然后生成独立的 debuginfo/dbg 安装包的系统，addr2line 基本没有什么用：
~# addr2line -e /usr/bin/ceph-mgr 0x44cac0
??:0
读取 ELF 文件（包括 elf 可执行文件和动态库）的 build-id（in ELF .note section）：
~$ eu-readelf -n /usr/bin/osdmaptool
Note section [ 2] '.note.ABI-tag' of 32 bytes at offset 0x254:
Owner Data size Type
GNU 16 VERSION
OS: Linux, ABI: 2.6.32
Note section [ 3] '.note.gnu.build-id' of 36 bytes at offset 0x274:
Owner Data size Type
GNU 20 GNU_BUILD_ID
Build ID: 08c29ee8586fdbe3d0958c5df3167cab5d8e9eec
~$ ll /usr/lib/debug/.build-id/08/c29ee8586fdbe3d0958c5df3167cab5d8e9eec*
lrwxrwxrwx. 1 root root 26 Dec 1 15:07 /usr/lib/debug/.build-id/08/c29ee8586fdbe3d0958c5df3167cab5d8e9eec -> ../../../../bin/osdmaptool
lrwxrwxrwx. 1 root root 30 Dec 1 15:07 /usr/lib/debug/.build-id/08/c29ee8586fdbe3d0958c5df3167cab5d8e9eec.debug -> ../../usr/bin/osdmaptool.debug
~$ ll /usr/lib/debug/usr/bin/osdmaptool.debug
-r--r--r--. 1 root root 5115216 Nov 30 22:39 /usr/lib/debug/usr/bin/osdmaptool.debug
~$ rpm -qf /usr/lib/debug/.build-id/08/c29ee8586fdbe3d0958c5df3167cab5d8e9eec
ceph-debuginfo-12.2.9-2.5.g00b0f3b.el7.x86_64
~$ rpm -qf /usr/lib/debug/.build-id/08/c29ee8586fdbe3d0958c5df3167cab5d8e9eec.debug
ceph-debuginfo-12.2.9-2.5.g00b0f3b.el7.x86_64
~$ rpm -qf /usr/lib/debug/usr/bin/osdmaptool.debug
ceph-debuginfo-12.2.9-2.5.g00b0f3b.el7.x86_64
关于 backtrace 的地址：
I’ve had a look at files backtrace.c and backtracesyms.c files in glibc source code (git://sourceware.org/git/glibc.git, commit 2482ae433a4249495859343ae1fba408300f2c2e).
Assuming I haven’t misread/misunderstood things:backtrace()itself looks like it will only give you symbol addresses as they are at runtime, which I think means you need the library load address as it was from pmap or similar. However,backtrace_symbols()recalculates things so that the addresses are relative to the shared library ELF, and not the process at runtime, which is really convenient. It means you don’t need information from pmap.
如果当前调试进程存在 fork 调用，设置是否继续调试当前进程，还是调试子进程：
(gdb) help set follow-fork-mode
Set debugger response to a program call of fork or vfork.
A fork or vfork creates a new process. follow-fork-mode can be:
parent - the original process is debugged after a fork
child - the new process is debugged after a fork
The unfollowed process will continue to run.
By default, the debugger will follow the parent process.
同时，还有一个对 exec 调用的设置：
(gdb) help set follow-exec-mode
Set debugger response to a program call of exec.
An exec call replaces the program image of a process.
follow-exec-mode can be:
new - the debugger creates a new inferior and rebinds the process
to this new inferior. The program the process was running before
the exec call can be restarted afterwards by restarting the original
inferior.
same - the debugger keeps the process bound to the same inferior.
The new executable image replaces the previous executable loaded in
the inferior. Restarting the inferior after the exec call restarts
the executable the process was running after the exec call.
By default, the debugger will use the same inferior.
打开或关闭源代码窗口（tui）：
ctrl - x - a
切换源代码窗口与命令交互窗口：
ctrl -x -o
刷新终端显示：
ctrl - l
调整窗口高度：
(gdb) winheight src -2
(gdb) winheight src +3
(gdb) winheight cmd 12
窗口的更多控制可以使用 layout 命令：
(gdb) help layout
Change the layout of windows.
Usage: layout prev | next | <layout_name>
Layout names are:
src : Displays source and command windows.
asm : Displays disassembly and command windows.
split : Displays source, disassembly and command windows.
regs : Displays register window. If existing layout
is source/command or assembly/command, the
register window is displayed. If the
source/assembly/command (split) is displayed,
the register window is displayed with
the window that has current logical focus.
如果 gdb 打印变量时出现如下错误，则说明 gdb 版本太低，比如在 CentOS 下使用 devtoolset 构建的版本，然后使用系统自带的 gdb 进行调试：
(gdb) p func
Python Exception <class 'gdb.error'> There is no member or method named _M_head_impl.:
$1 =
参考资料
[1] How do I set a conditional breakpoint in gdb, when char* x points to a string whose value equals “hello”?
[2] Convenience Functions
https://sourceware.org/gdb/current/onlinedocs/gdb/Convenience-Funs.html#Convenience-Funs
[3] 100-gdb-tips
https://github.com/hellogcc/100-gdb-tips
[4] gdb Debugging Full Example (Tutorial)
http://www.brendangregg.com/blog/2016-08-09/gdb-example-ncurses.html
[5] Tracing and Disassembling a Program
https://www.usna.edu/Users/cs/aviv/classes/si485h/s17/units/02/unit.html
[6] How to map function address to function in *.so files
https://stackoverflow.com/questions/7556045/how-to-map-function-address-to-function-in-so-files
[7] Have GDB show the type of a template argument
https://stackoverflow.com/questions/37616367/have-gdb-show-the-type-of-a-template-argument
[8] gdb: size of a struct that isn’t in context?
https://stackoverflow.com/questions/28982534/gdb-size-of-a-struct-that-isnt-in-context
[9] Debugging with GDB
https://sourceware.org/gdb/current/onlinedocs/gdb/index.html
最后修改于 2019-01-21