制作镜像保留复现环境

1
2
3
4
5
6
7
~$ docker commit 87 runsisi:ceph
sha256:33900137ac2d2b90968876c7589eca0dde0b15df294fd57eca137d2b6cf09326
~$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
runsisi ceph 33900137ac2d 3 seconds ago 84.19 MB
ceph v1 dcd4d43ff959 5 minutes ago 84.18 MB
apk-ceph-fuse v12.2.12 8ea49652b319 6 minutes ago 277.3 MB

复现问题

1
2
3
4
5
6
7
8
9
10
~$ docker exec -it 87 sh
/ # ceph -v
Traceback (most recent call last):
File "/usr/bin/ceph", line 127, in <module>
import rados
ImportError: Error relocating /usr/lib/libnspr4.so: secure_getenv: symbol not found
/ # ldd /usr/lib/libnspr4.so
/lib/ld-musl-x86_64.so.1 (0x55d56a547000)
libc.musl-x86_64.so.1 => /lib/ld-musl-x86_64.so.1 (0x55d56a547000)
Error relocating /usr/lib/libnspr4.so: secure_getenv: symbol not found

查看已安装的 nspr、musl 版本

1
2
3
4
5
/ # apk list -I | grep nspr
nspr-4.23-r0 x86_64 {nspr} (GPL-3.0-only AND LGPL-2.1-only AND LGPL-3.0-only AND MPL-2.0) [installed]
/ # apk list -I | grep musl
musl-1.1.22-r3 x86_64 {musl} (MIT) [installed]
musl-utils-1.1.22-r3 x86_64 {musl} (MIT BSD GPL2+) [installed]

安装 readelf 工具

1
2
3
4
5
6
7
8
9
/ # apk add binutils
fetch http://mirrors.example.com.cn/alpine/v3.10/main/x86_64/APKINDEX.tar.gz
fetch http://mirrors.example.com.cn/alpine/v3.10/community/x86_64/APKINDEX.tar.gz
fetch http://mirrors.example.com.cn/alpine/edge/community/x86_64/APKINDEX.tar.gz
fetch http://mirrors.example.com.cn/alpine/edge/main/x86_64/APKINDEX.tar.gz
fetch http://mirrors.example.com.cn/alpine/edge/testing/x86_64/APKINDEX.tar.gz
(1/1) Installing binutils (2.33.1-r0)
Executing busybox-1.30.1-r2.trigger
OK: 100 MiB in 51 packages

确认问题仍然存在

1
2
3
4
/ # ldd /usr/lib/libnspr4.so
/lib/ld-musl-x86_64.so.1 (0x5609f554f000)
libc.musl-x86_64.so.1 => /lib/ld-musl-x86_64.so.1 (0x5609f554f000)
Error relocating /usr/lib/libnspr4.so: secure_getenv: symbol not found

通过 readelf 读取动态链接符号表

1
2
3
4
5
6
/ # readelf --dyn-syms /usr/lib/libnspr4.so | grep getenv
2: 0000000000000000 0 FUNC GLOBAL DEFAULT UND secure_getenv
24: 0000000000000000 0 FUNC GLOBAL DEFAULT UND getenv
/ # readelf --dyn-syms /lib/ld-musl-x86_64.so.1 | grep getenv
456: 000000000001c343 100 FUNC GLOBAL DEFAULT 8 getenv
888: 000000000001cd53 0 FUNC GLOBAL DEFAULT 8 fegetenv

显然,nspr 动态链接所需的符号(UNDsecure_getenv 在 musl 中未定义。

列出 nspr、musl 可用版本

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
/ # apk list | grep musl
musl-dbg-1.1.22-r3 x86_64 {musl} (MIT)
musl-dbg-1.1.24-r0 x86_64 {musl} (MIT)
musl-1.1.22-r3 x86_64 {musl} (MIT) [installed]
musl-1.1.24-r0 x86_64 {musl} (MIT) [upgradable from: musl-1.1.22-r3]
musl-obstack-1.1-r0 x86_64 {musl-obstack} (GPL-2.0)
libc6-compat-1.1.22-r3 x86_64 {musl} (MIT)
libc6-compat-1.1.24-r0 x86_64 {musl} (MIT)
musl-utils-1.1.22-r3 x86_64 {musl} (MIT BSD GPL2+) [installed]
musl-utils-1.1.24-r0 x86_64 {musl} (MIT BSD GPL2+) [upgradable from: musl-utils-1.1.22-r3]
musl-nscd-dev-1.0.2-r0 x86_64 {musl-nscd} (MIT)
musl-dev-1.1.22-r3 x86_64 {musl} (MIT)
musl-dev-1.1.24-r0 x86_64 {musl} (MIT)
musl-nscd-1.0.2-r0 x86_64 {musl-nscd} (MIT)
musl-nscd-openrc-1.0.2-r0 x86_64 {musl-nscd} (MIT)
musl-nscd-doc-1.0.2-r0 x86_64 {musl-nscd} (MIT)
musl-obstack-dev-1.1-r0 x86_64 {musl-obstack} (GPL-2.0)
musl-libintl-1.1.24-r0 x86_64 {musl} (MIT)
/ # apk list | grep nspr
nspr-dev-4.20-r0 x86_64 {nspr} (MPL-1.1 GPL-2.0 LGPL-2.1)
nspr-dev-4.23-r0 x86_64 {nspr} (GPL-3.0-only AND LGPL-2.1-only AND LGPL-3.0-only AND MPL-2.0)
nspr-4.23-r0 x86_64 {nspr} (GPL-3.0-only AND LGPL-2.1-only AND LGPL-3.0-only AND MPL-2.0) [installed]
nspr-4.20-r0 x86_64 {nspr} (MPL-1.1 GPL-2.0 LGPL-2.1)

升级 musl 至新版本

1
2
3
4
5
6
7
8
9
10
11
12
/ # apk add musl=1.1.24-r0
(1/1) Upgrading musl (1.1.22-r3 -> 1.1.24-r0)
OK: 100 MiB in 51 packages
/ # readelf --dyn-syms /lib/ld-musl-x86_64.so.1 | grep getenv
456: 000000000001c342 102 FUNC GLOBAL DEFAULT 8 getenv
888: 000000000001cd91 0 FUNC GLOBAL DEFAULT 8 fegetenv
1459: 000000000001c4ec 17 FUNC GLOBAL DEFAULT 8 secure_getenv
/ # ldd /usr/lib/libnspr4.so
/lib/ld-musl-x86_64.so.1 (0x561bd9054000)
libc.musl-x86_64.so.1 => /lib/ld-musl-x86_64.so.1 (0x561bd9054000)
/ # ceph -v
ceph version 12.2.12-2-44-ge654eed76e (e654eed76ecf1aec63df8c5dae07910169b8b37c) luminous (stable)

降级 musl 至原来的版本

1
2
3
4
5
6
7
8
/ # apk add musl=1.1.22-r3
(1/1) Downgrading musl (1.1.24-r0 -> 1.1.22-r3)
OK: 100 MiB in 51 packages
/ # ceph -v
Traceback (most recent call last):
File "/usr/bin/ceph", line 127, in <module>
import rados
ImportError: Error relocating /usr/lib/libnspr4.so: secure_getenv: symbol not found

降级 nspr 至老版本

1
2
3
4
5
6
7
/ # apk add nspr=nspr-4.20-r0
ERROR: 'nspr=nspr-4.20-r0' is not a valid world dependency, format is name(@tag)([<>~=]version)
/ # apk add nspr=4.20-r0
(1/1) Downgrading nspr (4.23-r0 -> 4.20-r0)
OK: 100 MiB in 51 packages
/ # ceph -v
ceph version 12.2.12-2-44-ge654eed76e (e654eed76ecf1aec63df8c5dae07910169b8b37c) luminous (stable)

显然,新版本的 nspr 需要动态链接 resolve 符号 secure_getenv,而这个符号只在新版本的 musl 中有定义,因此不管是通过升级 musl 还是降级 nspr 都能解决该问题。