runsisi's

technical notes

从源码构建 Python

2019-08-28 runsisi#python

最近在定位一个 Python 模块内存异常释放导致段错误的问题,发现 Ubuntu 18.04 的 Python dbg 包竟然没有包含源码(可能必须用 python-dbg/python3-dbg ?),gdb 调试非常不方便,因此只得自己构建一个。

构建 Python2.7 版本

$ tar xvzf Python-2.7.16.tgz
$ cd Python-2.7.16/
$ export CXX=/usr/bin/g++
$ ./configure --prefix=/opt/runsisi/python2 --enable-shared --without-computed-gotos --with-pydebug --without-pymalloc --with-valgrind LDFLAGS="-Wl,--rpath=/opt/runsisi/python2/lib"

如果不 export CXX,本身并没有什么影响,只是在 configure 执行过程中有一个告警而已; 选项 --with-pydebug --without-pymalloc --with-valgrind 仅用于定位内存问题,正式版本不需要; 选项 --enable-shared 会单独构建一个 libpython 动态库,用于其他 C/C++ 程序嵌入 Python 解释器; 选项 --without-computed-gotos 禁用 bytecode 指令执行优化(computed goto)而直接使用 switch..case; 如果指定构建选项 --with-ensurepip=install,那么默认会安装 Python 源代码中自带的 pip,只是这个版本会比较低。

Ubuntu18.04 自带的 Python2/Python3 在构建时没有使能 --enable-shared 选项,因此它们的运行不依赖 libpython 动态库,但是为了提供上面提到的嵌入的 Python 解释器,又单独提供了 libpython 的安装包。

CentOS 7.x 的 Python2 在构建时使能了 --enable-shared 选项,因此它的运行依赖 libpython 动态库。

如果不指定 rpath,Linux 下的程序默认会从系统目录下搜索动态库,因此如果使能了 --enable-shared 选项,则必须要定义 rpath(当然也可以后期通过 LD_LIBRARY_PATH 等方式来修改这一行为)。

编译过程中如果出现如下的打印一般都是因为编译机器上缺少相应的 dev 包,如无需要忽略即可:

$ make -j16
...
Python build finished, but the necessary bits to build these modules were not found:
_bsddb             _sqlite3           _tkinter        
bsddb185           dbm                dl              
gdbm               imageop            readline        
sunaudiodev                                           
To find the necessary bits, look in setup.py in detect_modules() for the module's name.
...

安装:

$ sudo make install

设置环境变量:

$ export PATH=/opt/runsisi/python/bin:$PATH
$ which python
/opt/runsisi/python/bin/python

使能了 --enable-shared 选项的情况下,我们能明显看到 Python 主程序依赖 libpython 动态库:

$ ldd /opt/runsisi/python2/bin/python
	linux-vdso.so.1 (0x00007ffed9d50000)
	libpython2.7.so.1.0 => /opt/runsisi/python2/lib/libpython2.7.so.1.0 (0x00007fca53bf2000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fca53801000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fca535e2000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fca533de000)
	libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007fca531db000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fca52e3d000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fca5428d000)

没有使能的情况下,是这样的:

$ ldd /opt/runsisi/python2s/bin/python
	linux-vdso.so.1 (0x00007ffe97feb000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f4753447000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f4753243000)
	libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007f4753040000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f4752ca2000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f47528b1000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f4753ae6000)

安装 pip:

$ curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
$ sudo /opt/runsisi/python2/bin/python get-pip.py
$ which pip
/opt/runsisi/python2/bin/pip

使用 sudo pip 安装软件需要指定 pip 全路径:

$ sudo /opt/runsisi/python2/bin/pip install conda
$ which conda
/opt/runsisi/python2/bin/conda

构建 Python3.7 版本

整个过程与构建 2.7 版本类似,构建选项根据需要选择即可。

$ tar xvzf Python-3.7.4.tgz
$ cd Python-3.7.4/
$ export CXX=/usr/bin/g++
$ ./configure --prefix=/opt/runsisi/python3 --enable-shared --without-computed-gotos --with-pydebug --without-pymalloc --with-valgrind LDFLAGS="-Wl,--rpath=/opt/runsisi/python3/lib"
$ make -j16
$ sudo make install
...
Looking in links: /tmp/tmpfltvavjz
Collecting setuptools
Collecting pip
Installing collected packages: setuptools, pip
Successfully installed pip-19.0.3 setuptools-40.8.0

默认会安装 pip(Python3 configure 脚本默认配置了 --with-ensurepip=install),因此通常不需要再手工进行安装。

$ export PATH=/opt/runsisi/python3/bin:$PATH
$ which python3
/opt/runsisi/python3/bin/python3
$ which pip3
/opt/runsisi/python3/bin/pip3

由于使能了 --enable-shared 选项,因此会依赖 libpython 动态库:

$ ldd /opt/runsisi/python3/bin/python3
        linux-vdso.so.1 (0x00007ffc797cf000)
        libpython3.7dm.so.1.0 => /opt/runsisi/python3/lib/libpython3.7dm.so.1.0 (0x00007ffbcb8ab000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007ffbcb4ba000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007ffbcb29b000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007ffbcb097000)
        libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007ffbcae94000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007ffbcaaf6000)
        /lib64/ld-linux-x86-64.so.2 (0x00007ffbcc056000)

使用 sudo pip 安装软件需要指定 pip 全路径:

$ sudo /opt/runsisi/python3/bin/pip install conda
$ which conda
/opt/runsisi/python3/bin/conda

参考资料

Python executable not finding libpython shared library

https://stackoverflow.com/questions/7880454/python-executable-not-finding-libpython-shared-library

What exactly does ./configure --enable-shared do during python install?

https://stackoverflow.com/questions/38772946/what-exactly-does-configure-enable-shared-do-during-python-altinstall