Alpine 混用 apk 源导致 Ceph 加载失败

制作镜像保留复现环境

~$ docker commit 87 runsisi:ceph
sha256:33900137ac2d2b90968876c7589eca0dde0b15df294fd57eca137d2b6cf09326
~$ docker images
REPOSITORY                                      TAG                 IMAGE ID            CREATED             SIZE
runsisi                                         ceph                33900137ac2d        3 seconds ago       84.19 MB
ceph                                            v1                  dcd4d43ff959        5 minutes ago       84.18 MB
apk-ceph-fuse                                   v12.2.12            8ea49652b319        6 minutes ago       277.3 MB

复现问题

~$ docker exec -it 87 sh
/ # ceph -v
Traceback (most recent call last):
  File "/usr/bin/ceph", line 127, in <module>
    import rados
ImportError: Error relocating /usr/lib/libnspr4.so: secure_getenv: symbol not found
/ # ldd /usr/lib/libnspr4.so
        /lib/ld-musl-x86_64.so.1 (0x55d56a547000)
        libc.musl-x86_64.so.1 => /lib/ld-musl-x86_64.so.1 (0x55d56a547000)
Error relocating /usr/lib/libnspr4.so: secure_getenv: symbol not found

查看已安装的 nspr、musl 版本

/ # apk list -I | grep nspr
nspr-4.23-r0 x86_64 {nspr} (GPL-3.0-only AND LGPL-2.1-only AND LGPL-3.0-only AND MPL-2.0) [installed]
/ # apk list -I | grep musl
musl-1.1.22-r3 x86_64 {musl} (MIT) [installed]
musl-utils-1.1.22-r3 x86_64 {musl} (MIT BSD GPL2+) [installed]

安装 readelf 工具

/ # apk add binutils
fetch http://mirrors.example.com.cn/alpine/v3.10/main/x86_64/APKINDEX.tar.gz
fetch http://mirrors.example.com.cn/alpine/v3.10/community/x86_64/APKINDEX.tar.gz
fetch http://mirrors.example.com.cn/alpine/edge/community/x86_64/APKINDEX.tar.gz
fetch http://mirrors.example.com.cn/alpine/edge/main/x86_64/APKINDEX.tar.gz
fetch http://mirrors.example.com.cn/alpine/edge/testing/x86_64/APKINDEX.tar.gz
(1/1) Installing binutils (2.33.1-r0)
Executing busybox-1.30.1-r2.trigger
OK: 100 MiB in 51 packages

确认问题仍然存在

/ # ldd /usr/lib/libnspr4.so
        /lib/ld-musl-x86_64.so.1 (0x5609f554f000)
        libc.musl-x86_64.so.1 => /lib/ld-musl-x86_64.so.1 (0x5609f554f000)
Error relocating /usr/lib/libnspr4.so: secure_getenv: symbol not found

通过 readelf 读取动态链接符号表

/ # readelf --dyn-syms /usr/lib/libnspr4.so | grep getenv
     2: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND secure_getenv
    24: 0000000000000000     0 FUNC    GLOBAL DEFAULT  UND getenv
/ # readelf --dyn-syms /lib/ld-musl-x86_64.so.1 | grep getenv
   456: 000000000001c343   100 FUNC    GLOBAL DEFAULT    8 getenv
   888: 000000000001cd53     0 FUNC    GLOBAL DEFAULT    8 fegetenv

显然，nspr 动态链接所需的符号（UND） secure_getenv 在 musl 中未定义。

列出 nspr、musl 可用版本

/ # apk list | grep musl
musl-dbg-1.1.22-r3 x86_64 {musl} (MIT)
musl-dbg-1.1.24-r0 x86_64 {musl} (MIT)
musl-1.1.22-r3 x86_64 {musl} (MIT) [installed]
musl-1.1.24-r0 x86_64 {musl} (MIT) [upgradable from: musl-1.1.22-r3]
musl-obstack-1.1-r0 x86_64 {musl-obstack} (GPL-2.0)
libc6-compat-1.1.22-r3 x86_64 {musl} (MIT)
libc6-compat-1.1.24-r0 x86_64 {musl} (MIT)
musl-utils-1.1.22-r3 x86_64 {musl} (MIT BSD GPL2+) [installed]
musl-utils-1.1.24-r0 x86_64 {musl} (MIT BSD GPL2+) [upgradable from: musl-utils-1.1.22-r3]
musl-nscd-dev-1.0.2-r0 x86_64 {musl-nscd} (MIT)
musl-dev-1.1.22-r3 x86_64 {musl} (MIT)
musl-dev-1.1.24-r0 x86_64 {musl} (MIT)
musl-nscd-1.0.2-r0 x86_64 {musl-nscd} (MIT)
musl-nscd-openrc-1.0.2-r0 x86_64 {musl-nscd} (MIT)
musl-nscd-doc-1.0.2-r0 x86_64 {musl-nscd} (MIT)
musl-obstack-dev-1.1-r0 x86_64 {musl-obstack} (GPL-2.0)
musl-libintl-1.1.24-r0 x86_64 {musl} (MIT)
/ # apk list | grep nspr
nspr-dev-4.20-r0 x86_64 {nspr} (MPL-1.1 GPL-2.0 LGPL-2.1)
nspr-dev-4.23-r0 x86_64 {nspr} (GPL-3.0-only AND LGPL-2.1-only AND LGPL-3.0-only AND MPL-2.0)
nspr-4.23-r0 x86_64 {nspr} (GPL-3.0-only AND LGPL-2.1-only AND LGPL-3.0-only AND MPL-2.0) [installed]
nspr-4.20-r0 x86_64 {nspr} (MPL-1.1 GPL-2.0 LGPL-2.1)

升级 musl 至新版本

/ # apk add musl=1.1.24-r0
(1/1) Upgrading musl (1.1.22-r3 -> 1.1.24-r0)
OK: 100 MiB in 51 packages
/ # readelf --dyn-syms /lib/ld-musl-x86_64.so.1 | grep getenv
   456: 000000000001c342   102 FUNC    GLOBAL DEFAULT    8 getenv
   888: 000000000001cd91     0 FUNC    GLOBAL DEFAULT    8 fegetenv
  1459: 000000000001c4ec    17 FUNC    GLOBAL DEFAULT    8 secure_getenv
/ # ldd /usr/lib/libnspr4.so
        /lib/ld-musl-x86_64.so.1 (0x561bd9054000)
        libc.musl-x86_64.so.1 => /lib/ld-musl-x86_64.so.1 (0x561bd9054000)
/ # ceph -v
ceph version 12.2.12-2-44-ge654eed76e (e654eed76ecf1aec63df8c5dae07910169b8b37c) luminous (stable)

降级 musl 至原来的版本

/ # apk add musl=1.1.22-r3
(1/1) Downgrading musl (1.1.24-r0 -> 1.1.22-r3)
OK: 100 MiB in 51 packages
/ # ceph -v
Traceback (most recent call last):
  File "/usr/bin/ceph", line 127, in <module>
    import rados
ImportError: Error relocating /usr/lib/libnspr4.so: secure_getenv: symbol not found

降级 nspr 至老版本

/ # apk add nspr=nspr-4.20-r0
ERROR: 'nspr=nspr-4.20-r0' is not a valid world dependency, format is name(@tag)([<>~=]version)
/ # apk add nspr=4.20-r0
(1/1) Downgrading nspr (4.23-r0 -> 4.20-r0)
OK: 100 MiB in 51 packages
/ # ceph -v
ceph version 12.2.12-2-44-ge654eed76e (e654eed76ecf1aec63df8c5dae07910169b8b37c) luminous (stable)

显然，新版本的 nspr 需要动态链接 resolve 符号 secure_getenv，而这个符号只在新版本的 musl 中有定义，因此不管是通过升级 musl 还是降级 nspr 都能解决该问题。