runsisi's

technical notes

Go 程序调试

2019-04-10 runsisi#golang#debug

Go 程序的调试可以使用 gdb 或者 dlv,但 dlv 在打印变量内容、调试 goroutine 等方面对 Go 的支持比 gdb 要好很多。

dlv

dlv v1.2.0 版本好像存在 bug,dlv exec 加载程序后不会出现 dlv 的交互终端(这里不考虑 dlv debug 这种基本无意义的使用方式),而是直接运行本身,因此如果有必要可能需要使用老版本:

~$ go get github.com/derekparker/delve/cmd/dlv@v1.1.0  
go: finding github.com/derekparker/delve/cmd/dlv v1.1.0
go: finding github.com/derekparker/delve/cmd v1.1.0
go: finding golang.org/x/arch/x86/x86asm latest
go: finding golang.org/x/sys/unix latest
go: finding golang.org/x/arch/x86 latest
go: finding golang.org/x/sys latest
go: finding golang.org/x/arch latest
~$ dlv version
Delve Debugger
Version: 1.1.0
Build: $Id: 1990ba12450cab9425a2ae62e6ab988725023d5c $

dlv 的使用基本上与 gdb 类似:

~$ dlv exec terraform -- init
Type 'help' for list of commands.
(dlv) funcs discovery.*Get
github.com/hashicorp/terraform/plugin/discovery.(*ProviderInstaller).Get

条件断点:

(dlv) l
> github.com/hashicorp/terraform/terraform.loadProviderSchemas() /home/runsisi/workingcopy/src/hcl/terraform/terraform/schemas.go:95 (PC: 0xfe5fd0)
    90: }
    91:
    92: func loadProviderSchemas(schemas map[string]*ProviderSchema, config *configs.Config, state *states.State, components contextComponentFactory) tfdiags.Diagnostics {
    93:         var diags tfdiags.Diagnostics
    94:
=>  95:         ensure := func(typeName string) {
    96:                 if _, exists := schemas[typeName]; exists {
    97:                         return
    98:                 }
    99:
   100:                 log.Printf("[TRACE] LoadSchemas: retrieving schema for provider type %q", typeName)
(dlv) b 96
Breakpoint 2 set at 0x103372f for github.com/hashicorp/terraform/terraform.loadProviderSchemas.func1() /home/runsisi/workingcopy/src/hcl/terraform/terraform/schemas.go:96
(dlv) cond 2 typeName == "linode"
(dlv) bp
Breakpoint unrecovered-panic at 0x432bf0 for runtime.fatalpanic() /usr/lib/go-1.12/src/runtime/panic.go:690 (0)
        print runtime.curg._panic.arg
Breakpoint 1 at 0xfe5f8b for github.com/hashicorp/terraform/terraform.loadProviderSchemas() /home/runsisi/workingcopy/src/hcl/terraform/terraform/schemas.go:92 (1)
Breakpoint 2 at 0x103372f for github.com/hashicorp/terraform/terraform.loadProviderSchemas.func1() /home/runsisi/workingcopy/src/hcl/terraform/terraform/schemas.go:96 (0)
        cond typeName == "linode"

其他命令可以在交互终端中输入 help 进行查找:

(dlv) help
The following commands are available:
    args ------------------------ Print function arguments.
    break (alias: b) ------------ Sets a breakpoint.
    breakpoints (alias: bp) ----- Print out info for active breakpoints.
    call ------------------------ Resumes process, injecting a function call (EXPERIMENTAL!!!)
    clear ----------------------- Deletes breakpoint.
    ...
(dlv) help funcs
Print list of functions.

        funcs [<regex>]

If regex is specified only the functions matching it will be returned.

dlv 好像不支持调试子进程,因此对于 terraform 这种以子进程形式运行的程序毫无意义,不过 terraform 支持不以子进程的形式运行:

~$ export TF_FORK=0

这样才能继续使用 dlv 进行调试。

gdb

gdb 好像对 Go 程序的 fork 处理存在问题,当然也有可能是 terraform 自身使用的 panicwrap 带来的问题:

(gdb) set follow-fork-mode child
(gdb) r
Starting program: /home/runsisi/go/bin/terraform init
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7ffff55b3700 (LWP 179982)]
[New Thread 0x7ffff4db2700 (LWP 179983)]
[New Thread 0x7fffeffff700 (LWP 179984)]
[New Thread 0x7fffef7fe700 (LWP 179985)]
[New Thread 0x7fffeeffd700 (LWP 179986)]
[New Thread 0x7fffee7fc700 (LWP 179987)]
[New Thread 0x7fffed7fa700 (LWP 179990)]
[New Thread 0x7fffedffb700 (LWP 179989)]
[New process 179988]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
/build/gdb-GT4MLW/gdb-8.1/gdb/nat/x86-linux-dregs.c:146: internal-error: void x86_linux_update_debug_registers(lwp_info*): Assertion `lwp_is_stopped (lwp)' failed.
A problem internal to GDB has been detected,
further debugging may prove unreliable.
Quit this debugging session? (y or n) 

与 dlv 类似,同样同样需要设置环境变量:

~$ export TF_FORK=0

此时就可以不需要设置 follow-fork-mode:

~$ gdb terraform
GNU gdb (Ubuntu 8.1-0ubuntu3) 8.1.0.20180409-git
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from terraform...sdone.
Loading Go Runtime support.
(gdb) set args init
(gdb) info functions discovery.*Get
All functions matching regular expression "discovery.*Get":

File /home/runsisi/workingcopy/src/hcl/terraform/plugin/discovery/get.go:
void github.com/hashicorp/terraform/plugin/discovery.(*ProviderInstaller).Get;
(gdb) b github.com/hashicorp/terraform/plugin/discovery.(*ProviderInstaller).Get
Breakpoint 1 at 0xb382d0: file /home/runsisi/workingcopy/src/hcl/terraform/plugin/discovery/get.go, line 109.
(gdb) r
Starting program: /home/runsisi/go/bin/terraform init
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7ffff55b3700 (LWP 179406)]
[New Thread 0x7ffff4db2700 (LWP 179407)]
[New Thread 0x7fffeffff700 (LWP 179408)]
[New Thread 0x7fffef7fe700 (LWP 179409)]
[New Thread 0x7fffeeffd700 (LWP 179410)]
[New Thread 0x7fffee7fc700 (LWP 179411)]
[New Thread 0x7fffedffb700 (LWP 179412)]
2019/04/10 10:51:37 [INFO] Terraform version: 0.12.0 dev 
2019/04/10 10:51:37 [INFO] Go runtime version: go1.12
2019/04/10 10:51:37 [INFO] CLI args: []string{"/home/runsisi/go/bin/terraform", "init"}
...

参考资料

Delve

https://github.com/go-delve/delve/tree/master/Documentation