cargo package 时间戳错误

问题现象

构建 proxmox-backup deb 包时总是出现如下的 lintian 错误:

E: proxmox-backup-server: package-contains-ancient-file etc/apt/sources.list.d/pbs-enterprise.list 1973-11-29
E: proxmox-backup-server: package-contains-ancient-file lib/systemd/system/proxmox-backup-daily-update.timer 1973-11-29

问题分析

解压 deb 之后确认问题真实存在:

$ stat etc/apt/sources.list.d/pbs-enterprise.list
  File: etc/apt/sources.list.d/pbs-enterprise.list
  Size: 70              Blocks: 8          IO Block: 4096   regular file
Device: 810h/2064d      Inode: 310438      Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1000/ runsisi)   Gid: ( 1000/ runsisi)
Access: 2021-07-05 18:34:50.100000000 +0800
Modify: 1973-11-30 05:33:09.000000000 +0800
Change: 2021-07-05 18:34:50.100000000 +0800
 Birth: -
$ stat lib/systemd/system/proxmox-backup-daily-update.timer
  File: lib/systemd/system/proxmox-backup-daily-update.timer
  Size: 184             Blocks: 8          IO Block: 4096   regular file
Device: 810h/2064d      Inode: 310444      Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1000/ runsisi)   Gid: ( 1000/ runsisi)
Access: 2021-07-05 18:34:50.100000000 +0800
Modify: 1973-11-30 05:33:09.000000000 +0800
Change: 2021-07-05 18:34:50.100000000 +0800
 Birth: -

回头看一下原始文件的时间戳:

$ ll etc/
total 44
-rw-r--r-- 1 runsisi runsisi 480 Nov 30  1973 Makefile
-rw-r--r-- 1 runsisi runsisi  70 Nov 30  1973 pbs-enterprise.list
-rw-r--r-- 1 runsisi runsisi 401 Jul  5 18:06 proxmox-backup-banner.service
-rw-r--r-- 1 runsisi runsisi 375 Nov 30  1973 proxmox-backup-banner.service.in
-rw-r--r-- 1 runsisi runsisi 233 Jul  5 18:06 proxmox-backup-daily-update.service
-rw-r--r-- 1 runsisi runsisi 220 Nov 30  1973 proxmox-backup-daily-update.service.in
-rw-r--r-- 1 runsisi runsisi 184 Nov 30  1973 proxmox-backup-daily-update.timer
-rw-r--r-- 1 runsisi runsisi 408 Jul  5 18:06 proxmox-backup-proxy.service
-rw-r--r-- 1 runsisi runsisi 407 Nov 30  1973 proxmox-backup-proxy.service.in
-rw-r--r-- 1 runsisi runsisi 315 Jul  5 18:06 proxmox-backup.service
-rw-r--r-- 1 runsisi runsisi 302 Nov 30  1973 proxmox-backup.service.in
$ stat etc/pbs-enterprise.list
  File: etc/pbs-enterprise.list
  Size: 70              Blocks: 8          IO Block: 4096   regular file
Device: 810h/2064d      Inode: 306444      Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1000/ runsisi)   Gid: ( 1000/ runsisi)
Access: 2021-07-05 18:06:29.900000000 +0800
Modify: 1973-11-30 05:33:09.000000000 +0800
Change: 2021-07-05 18:06:29.830000000 +0800
 Birth: -

显然,原始文件的时间戳就是有问题的,之所以 lintian 只报错其中的两个文件,是因为只有这两个文件是原样拷贝的。

proxmox-backup 的源代码包是使用 debcargo 打包的,底层实际上也是调用的 cargo 的接口,因此,尝试创建一个最简的 cargo 工程用 cargo package 打包,看看是不是也存在类似的问题。tar xf 解压 cargo package 创建的 crate 文件(需要先重命名为 tar.gz 文件),然后查看时间戳:

$ stat Cargo.lock
  File: Cargo.lock
  Size: 150             Blocks: 8          IO Block: 4096   regular file
Device: 810h/2064d      Inode: 310545      Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1000/ runsisi)   Gid: ( 1000/ runsisi)
Access: 2021-07-06 08:28:36.570000000 +0800
Modify: 1970-01-01 08:00:01.000000000 +0800
Change: 2021-07-06 08:28:36.570000000 +0800
 Birth: -
$ stat Cargo.toml
  File: Cargo.toml
  Size: 717             Blocks: 8          IO Block: 4096   regular file
Device: 810h/2064d      Inode: 310546      Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1000/ runsisi)   Gid: ( 1000/ runsisi)
Access: 2021-07-06 08:28:36.570000000 +0800
Modify: 1970-01-01 08:00:01.000000000 +0800
Change: 2021-07-06 08:28:36.570000000 +0800
 Birth: -
$ stat Cargo.toml.orig
  File: Cargo.toml.orig
  Size: 295             Blocks: 8          IO Block: 4096   regular file
Device: 810h/2064d      Inode: 310547      Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1000/ runsisi)   Gid: ( 1000/ runsisi)
Access: 2021-07-06 08:28:36.570000000 +0800
Modify: 1973-11-30 05:33:09.000000000 +0800
Change: 2021-07-06 08:28:36.570000000 +0800
 Birth: -

显然,cargo 自身是有问题的,而且从 cargo package 就得到两个特殊的时间戳,其中一个与 debcargo 得到的一致(回过头去看 debcargo 打包的源代码,其实所有文件的时间戳都是有问题的,而且这两个错误的时间戳都):

Modify: 1973-11-30 05:33:09.000000000 +0800
Modify: 1970-01-01 08:00:01.000000000 +0800

我们回到 cargo 的源代码,从源代码分析问题所在。

源码分析

源码打包的代码是在 src/cargo/ops/cargo_package.rs 中的 tar 函数实现的,首先从 build_ar_list 可以看到 Cargo.toml, Cargo.lock, Cargo.toml.orig 以及其它文件是如何归类的:

Cargo.toml -> FileContents::Generated
Cargo.toml.orig -> FileContents::OnDisk
Cargo.lock -> FileContents::Generated
others -> FileContents::OnDisk

然后从 tar 函数来看看各种分类的文件是如何被处理的:

match contents {
	FileContents::OnDisk(disk_path) => {
		let mut file = File::open(&disk_path).with_context(|| {
			format!("failed to open for archiving: `{}`", disk_path.display())
		})?;
		let metadata = file.metadata().with_context(|| {
			format!("could not learn metadata for: `{}`", disk_path.display())
		})?;
		header.set_metadata_in_mode(&metadata, HeaderMode::Deterministic);
		header.set_cksum();
		ar.append_data(&mut header, &ar_path, &mut file)
			.with_context(|| {
				format!("could not archive source file `{}`", disk_path.display())
			})?;
	}
	FileContents::Generated(generated_kind) => {
		let contents = match generated_kind {
			GeneratedFile::Manifest => pkg.to_registry_toml(ws)?,
			GeneratedFile::Lockfile => build_lock(ws)?,
			GeneratedFile::VcsInfo(s) => s,
		};
		header.set_entry_type(EntryType::file());
		header.set_mode(0o644);
		header.set_size(contents.len() as u64);
		header.set_cksum();
		ar.append_data(&mut header, &ar_path, contents.as_bytes())
			.with_context(|| format!("could not archive source file `{}`", rel_str))?;
	}
}

对于 FileContents::OnDisk 类型,header.set_metadata_in_mode 根据第二个参数来决定其行为(具体实现可参考 fill_platform_from),比如在这里的 HeaderMode::Deterministic 参数下,最终会调用 self.set_mtime(123456789);,看看这个时间戳是什么时候:

$ date -d @123456789
Fri Nov 30 05:33:09 CST 1973

是不是很熟悉?

而对于 FileContents::Generated 类型,可以看到并没有显式设置时间戳,没有显式设置就是 0,那我们上面看到的时间戳为什么是:

Modify: 1970-01-01 08:00:01.000000000 +0800

而不是:

Modify: 1970-01-01 08:00:00.000000000 +0800

还记不记得之前给 cargo 打过补丁:

https://github.com/rust-lang/cargo/pull/9517

显然,这个 1s 的差异就是这里来的,那如果回过去看 debcargo 生成的文件:

$ stat Cargo.lock
  File: Cargo.lock
  Size: 57136           Blocks: 112        IO Block: 4096   regular file
Device: 810h/2064d      Inode: 310551      Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1000/ runsisi)   Gid: ( 1000/ runsisi)
Access: 2021-07-06 09:07:10.660000000 +0800
Modify: 1970-01-01 08:00:00.000000000 +0800
Change: 2021-07-06 09:07:10.660000000 +0800
 Birth: -

由于 debcargo 依赖的 cargo 是没有打过补丁的,所以就没有那个多出来的 1s。

解决思路

给 cargo 中的 tar 函数打补丁,其中 FileContents::OnDisk 类型直接设置成原始文件的时间戳(由于各平台差异,需要参考 fill_platform_from 的实现,这里简单起见默认用的 unix 平台实现),而 FileContents::Generated 类型使用当前时间戳。

--- cargo_package.rs.orig       2021-07-06 10:51:26.760000000 +0800
+++ cargo_package.rs    2021-07-06 10:44:23.900000000 +0800
@@ -509,6 +509,8 @@
                     format!("could not learn metadata for: `{}`", disk_path.display())
                 })?;
                 header.set_metadata_in_mode(&metadata, HeaderMode::Deterministic);
+                use std::os::unix::fs::MetadataExt;
+                header.set_mtime(metadata.mtime() as u64);
                 header.set_cksum();
                 ar.append_data(&mut header, &ar_path, &mut file)
                     .with_context(|| {
@@ -524,6 +526,8 @@
                 header.set_entry_type(EntryType::file());
                 header.set_mode(0o644);
                 header.set_size(contents.len() as u64);
+                use std::time::{SystemTime, UNIX_EPOCH};
+                header.set_mtime(SystemTime::now().duration_since(UNIX_EPOCH).unwrap().as_secs());
                 header.set_cksum();
                 ar.append_data(&mut header, &ar_path, contents.as_bytes())
                     .with_context(|| format!("could not archive source file `{}`", rel_str))?;

需要注意的是 debcargo 引用的 cargo 要在 vendor 目录下打补丁,打补丁之前要在 Cargo.toml 中添加 patch 配置,否则下一次 cargo vendor 会覆盖补丁版本:

[patch.crates-io]
cargo = { path = "vendor/cargo" }

参考资料

cfg!(linux) should work
https://github.com/rust-lang/rust/issues/26726

Conditional compilation
https://doc.rust-lang.org/reference/conditional-compilation.html

Need a way to easily modify vendored crates for local testing and try pushes
https://bugzilla.mozilla.org/show_bug.cgi?id=1323557


最后修改于 2021-08-29