最近陆续有 "Linux Lab" (http://tinylab.org) 用户报告偶发 firefox 跑不起来，偶发执行 sync 命令后不返回的情况，本文是这个 bug (https://gitee.com/tinylab/linux-lab/issues/I1GGG1) 的分析报告。
有部分 Linux Lab 开发者参与了现象收集和初步分析工作，最后由我汇总如下文。在此感谢他们的付出。
我们陆续收到的现象汇总如下：
Product: Gigabyte、ASUSTeK 、Hasee
ARCH: x86_64
CPU: i5-6500、i5-4210U
RAM: 7915 MiB
System: Deepin 20 Beta、 Ubuntu 18.04 LTS、Ubuntu 20.04 LTS
Kernel: 5.3、5.4
Docker: 19.03
Shell: 5.0Product: VMware、Lenovo Y50-70、X555LI、VBox
ARCH: x86_64
CPU: i7-8550U、i5-4210H、i7-6700K、i7-8550U
RAM: 3934 MiB、 2906 MiB
System: Ubuntu 16.04、Ubuntu 18.04、CentOS Linux release 7.7.1908、Deepin 15.11
Kernel: 4.4、4.15、4.19、5.0、5.2、5.3
Docker: 18.09、19.03
Shell: 4.2、4.4、5.0启动 lab
登录 lab
stop lab
systemctl stop docker
sync 卡死用下面的命令分别抓取了正常情况和异常情况下 sync 的调用信息
sudo perf ftrace -t function_graph -G ksys_sync sync > 1.txt通过分析，发现异常情况下，最后停在了这个位置：
0) 0.364 us | _raw_spin_lock(); | down_read() {根据多年的嵌入式研发经验，简单的命令卡死，第一反应当然是看 cpu、mem、io、/proc/<pidof sync>/status、/proc/<pidof sync>/stack、gdb attach 上去看看。发现 cpu、mem、IO 消耗都很低，sync 卡死在 ksys_sync 里面的 iterate_supers 中，状态是 disk sleep，gdb 也卡死了。
void ksys_sync(void)
{
int nowait = 0, wait = 1;
wakeup_flusher_threads(WB_REASON_SYNC);
iterate_supers(sync_inodes_one_sb, NULL);
iterate_supers(sync_fs_one_sb, &nowait);
iterate_supers(sync_fs_one_sb, &wait);
iterate_bdevs(fdatawrite_one_bdev, NULL);
iterate_bdevs(fdatawait_one_bdev, NULL);
if (unlikely(laptop_mode))
laptop_sync_completion();
}此时感觉和 Linux 内核还是有一定关系的（毕竟卡死在内核了）。
configs/linux-lab/tools/container-run 中加载的模块：loop minix nfsd 9pnet_virtio kvm $kvm_dri。发现加载 nfsd 的时候有异常。apt upgrade 方式升级的，nfsd 模块是 upgrade 的时候自带的，不是 Linux Lab 中的。因此，可以先确认一下在 host 里面开启 nfs 服务并且挂载 nfs 文件系统，看有没有问题。如果没有问题，再看 Linux Lab 中的配置和 host 有什么区别。/etc/exports 和 host 中的 exports 有差异。统一配置后，在 Linux Lab 中挂载 nfsd 还是有问题。只能在 Linux Lab 中一步步的执行、排查。luo@luo:~/linux/cloud-lab@luo:E126ElinuxcloudE45Elab<annotation encoding="application style="font-size: inherit;line-height: inherit;color: rgb(91, 218, 237);overflow-wrap: inherit !important;word-break: inherit !important;" span="" encoding=""application"><span class="katex-html" aria-hidden="true" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;"><span class="strut" style="height:1em;vertical-align:-0.25em;" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;"><span class="mord mathit" style="margin-right:0.01968em;" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">l<span class="mord mathit" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">s<span class="mord mathit" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">m<span class="mord mathit" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">o<span class="mord mathit" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">d<span class="mord mathit" style="margin-right:0.03588em;" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">g<span class="mord mathit" style="margin-right:0.02778em;" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">r<span class="mord mathit" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">e<span class="mord mathit" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">p<span class="mord mathit" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">n<span class="mord mathit" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">s<span class="mord mathit" style="margin-right:0.10764em;" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">f<span class="mord mathit" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">d<span class="mord mathit" style="margin-right:0.01968em;" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">l<span class="mord mathit" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">u<span class="mord mathit" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">o@<span class="mord mathit" style="margin-right:0.01968em;" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">l<span class="mord mathit" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">u<span class="mord mathit" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">o<span class="mspace" style="margin-right:0.2777777777777778em;" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;"><span class="mspace" style="margin-right:0.2777777777777778em;" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;"><span class="strut" style="height:1em;vertical-align:-0.25em;" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;"><span class="mord mathit" style="margin-right:0.05764em;" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">E<span class="mord mathit" style="margin-right:0.05764em;" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">E<span class="mord mathit" style="margin-right:0.01968em;" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">l<span class="mord mathit" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">i<span class="mord mathit" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">n<span class="mord mathit" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">u<span class="mord mathit" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">x<span class="mord mathit" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">c<span class="mord mathit" style="margin-right:0.01968em;" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">l<span class="mord mathit" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">o<span class="mord mathit" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">u<span class="mord mathit" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">d<span class="mord mathit" style="margin-right:0.05764em;" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">E<span class="mord mathit" style="margin-right:0.05764em;" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">E<span class="mord mathit" style="margin-right:0.01968em;" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">l<span class="mord mathit" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">a<span class="mord mathit" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">b ./tools/docker/run linux-lab
LOG: Current Lab is linux-lab
LOG: Open the running linux-lab
LOG: Current Lab is linux-lab
LOG: Available login methods:
bash ssh vnc webssh webvnc
LOG: Switch to another method:
.//home/luo/linux/cloud-lab/tools/docker/login LOGIN_METHOD
LOG: Running 'tools/docker/bash'
LOG: Current Lab is linux-lab
LOG: Running ' docker exec -it -w //labs/linux-lab -u ubuntu 'linux-lab-10410' //bin/bash'
ubuntu@linux-lab:/labs/linux-lab@linuxE45Elab:labslinuxE45Elab<annotation encoding="application style="font-size: inherit;line-height: inherit;color: rgb(91, 218, 237);overflow-wrap: inherit !important;word-break: inherit !important;" span="" encoding=""application"><span class="katex-html" aria-hidden="true" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;"><span class="strut" style="height:1em;vertical-align:-0.25em;" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;"><span class="mord mathit" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">s<span class="mord mathit" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">u<span class="mord mathit" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">d<span class="mord mathit" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">o<span class="mord mathit" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">m<span class="mord mathit" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">o<span class="mord mathit" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">d<span class="mord mathit" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">p<span class="mord mathit" style="margin-right:0.02778em;" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">r<span class="mord mathit" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">o<span class="mord mathit" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">b<span class="mord mathit" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">e<span class="mord mathit" style="margin-right:0.05764em;" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">E<span class="mord mathit" style="margin-right:0.05764em;" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">E<span class="mord mathit" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">d<span class="mord mathit" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">m<span class="mord mathit" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">n<span class="mord mathit" style="margin-right:0.10764em;" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">f<span class="mord mathit" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">s<span class="mord mathit" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">d<span class="mord mathit" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">u<span class="mord mathit" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">b<span class="mord mathit" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">u<span class="mord mathit" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">n<span class="mord mathit" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">t<span class="mord mathit" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">u@<span class="mord mathit" style="margin-right:0.01968em;" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">l<span class="mord mathit" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">i<span class="mord mathit" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">n<span class="mord mathit" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">u<span class="mord mathit" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">x<span class="mord mathit" style="margin-right:0.05764em;" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">E<span class="mord mathit" style="margin-right:0.05764em;" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">E<span class="mord mathit" style="margin-right:0.01968em;" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">l<span class="mord mathit" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">a<span class="mord mathit" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">b<span class="mspace" style="margin-right:0.2777777777777778em;" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;"><span class="mspace" style="margin-right:0.2777777777777778em;" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;"><span class="strut" style="height:1em;vertical-align:-0.25em;" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;"><span class="mord mathit" style="margin-right:0.01968em;" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">l<span class="mord mathit" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">a<span class="mord mathit" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">b<span class="mord mathit" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">s<span class="mord mathit" style="margin-right:0.01968em;" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">l<span class="mord mathit" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">i<span class="mord mathit" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">n<span class="mord mathit" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">u<span class="mord mathit" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">x<span class="mord mathit" style="margin-right:0.05764em;" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">E<span class="mord mathit" style="margin-right:0.05764em;" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">E<span class="mord mathit" style="margin-right:0.01968em;" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">l<span class="mord mathit" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">a<span class="mord mathit" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">b echo @linuxE45Elab:labslinuxE45Elab<annotation encoding="application style="font-size: inherit;line-height: inherit;color: rgb(91, 218, 237);overflow-wrap: inherit !important;word-break: inherit !important;" span="" encoding=""application"><span class="katex-html" aria-hidden="true" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;"><span class="strut" style="height:0.69444em;vertical-align:0em;" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">137<span class="mord mathit" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">u<span class="mord mathit" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">b<span class="mord mathit" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">u<span class="mord mathit" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">n<span class="mord mathit" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">t<span class="mord mathit" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">u@<span class="mord mathit" style="margin-right:0.01968em;" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">l<span class="mord mathit" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">i<span class="mord mathit" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">n<span class="mord mathit" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">u<span class="mord mathit" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">x<span class="mord mathit" style="margin-right:0.05764em;" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">E<span class="mord mathit" style="margin-right:0.05764em;" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">E<span class="mord mathit" style="margin-right:0.01968em;" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">l<span class="mord mathit" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">a<span class="mord mathit" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">b<span class="mspace" style="margin-right:0.2777777777777778em;" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;"><span class="mspace" style="margin-right:0.2777777777777778em;" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;"><span class="strut" style="height:1em;vertical-align:-0.25em;" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;"><span class="mord mathit" style="margin-right:0.01968em;" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">l<span class="mord mathit" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">a<span class="mord mathit" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">b<span class="mord mathit" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">s<span class="mord mathit" style="margin-right:0.01968em;" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">l<span class="mord mathit" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">i<span class="mord mathit" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">n<span class="mord mathit" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">u<span class="mord mathit" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">x<span class="mord mathit" style="margin-right:0.05764em;" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">E<span class="mord mathit" style="margin-right:0.05764em;" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">E<span class="mord mathit" style="margin-right:0.01968em;" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">l<span class="mord mathit" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">a<span class="mord mathit" style="font-size: inherit;color: inherit;line-height: inherit;overflow-wrap: inherit !important;word-break: inherit !important;">b
</span class="mord mathit"></span class="mord mathit"></span class="mord mathit" style="margin-right:0.01968em;"></span class="mord mathit" style="margin-right:0.05764em;"></span class="mord mathit" style="margin-right:0.05764em;"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit" style="margin-right:0.01968em;"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit" style="margin-right:0.01968em;"></span class="strut" style="height:1em;vertical-align:-0.25em;"></span class="mspace" style="margin-right:0.2777777777777778em;"></span class="mspace" style="margin-right:0.2777777777777778em;"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit" style="margin-right:0.01968em;"></span class="mord mathit" style="margin-right:0.05764em;"></span class="mord mathit" style="margin-right:0.05764em;"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit" style="margin-right:0.01968em;"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit"></span class="strut" style="height:0.69444em;vertical-align:0em;"></span class="katex-html" aria-hidden="true"></annotation encoding="application></span class="mord mathit"></span class="mord mathit"></span class="mord mathit" style="margin-right:0.01968em;"></span class="mord mathit" style="margin-right:0.05764em;"></span class="mord mathit" style="margin-right:0.05764em;"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit" style="margin-right:0.01968em;"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit" style="margin-right:0.01968em;"></span class="strut" style="height:1em;vertical-align:-0.25em;"></span class="mspace" style="margin-right:0.2777777777777778em;"></span class="mspace" style="margin-right:0.2777777777777778em;"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit" style="margin-right:0.01968em;"></span class="mord mathit" style="margin-right:0.05764em;"></span class="mord mathit" style="margin-right:0.05764em;"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit" style="margin-right:0.01968em;"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit" style="margin-right:0.10764em;"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit" style="margin-right:0.05764em;"></span class="mord mathit" style="margin-right:0.05764em;"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit" style="margin-right:0.02778em;"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit"></span class="strut" style="height:1em;vertical-align:-0.25em;"></span class="katex-html" aria-hidden="true"></annotation encoding="application></span class="mord mathit"></span class="mord mathit"></span class="mord mathit" style="margin-right:0.01968em;"></span class="mord mathit" style="margin-right:0.05764em;"></span class="mord mathit" style="margin-right:0.05764em;"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit" style="margin-right:0.01968em;"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit" style="margin-right:0.01968em;"></span class="mord mathit" style="margin-right:0.05764em;"></span class="mord mathit" style="margin-right:0.05764em;"></span class="strut" style="height:1em;vertical-align:-0.25em;"></span class="mspace" style="margin-right:0.2777777777777778em;"></span class="mspace" style="margin-right:0.2777777777777778em;"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit" style="margin-right:0.01968em;"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit" style="margin-right:0.01968em;"></span class="mord mathit"></span class="mord mathit" style="margin-right:0.10764em;"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit" style="margin-right:0.02778em;"></span class="mord mathit" style="margin-right:0.03588em;"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit"></span class="mord mathit" style="margin-right:0.01968em;"></span class="strut" style="height:1em;vertical-align:-0.25em;"></span class="katex-html" aria-hidden="true"></annotation encoding="application>执行 modprobe nfsd，返回值137，说明执行失败。查看 dmesg，发现如下异常信息：
[ 304.086144] Installing knfsd (copyright (C) 1996 okir@monad.swb.de).
[ 304.086228] BUG: kernel NULL pointer dereference, address: 0000000000000058
[ 304.086231] #PF: supervisor write access in kernel mode
[ 304.086233] #PF: error_code(0x0002) - not-present page
[ 304.086234] PGD 0 P4D 0
[ 304.086238] Oops: 0002 [#1] SMP PTI
[ 304.086241] CPU: 3 PID: 9151 Comm: modprobe Tainted: G O 5.3.0-51-generic #44~18.04.2-Ubuntu
[ 304.086242] Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
[ 304.086257] RIP: 0010:nfsd_fill_super+0x71/0x90 [nfsd]
[ 304.086260] Code: 85 c0 89 c3 74 09 89 d8 5b 41 5c 41 5d 5d c3 49 8b 7c 24 68 31 f6 48 c7 c2 70 84 a0 c0 e8 97 fe ff ff 48 3d 00 f0 ff ff 77 0d <49> 89 45 58 89 d8 5b 41 5c 41 5d 5d c3 89 c3 eb cb 0f 1f 40 00 66
[ 304.086261] RSP: 0018:ffffba6945edfaa8 EFLAGS: 00010287
[ 304.086263] RAX: ffff9256d671e600 RBX: 0000000000000000 RCX: 0000000000000002
[ 304.086265] RDX: 0000000000000000 RSI: 0000000000000100 RDI: ffff9256cf8309e0
[ 304.086267] RBP: ffffba6945edfac0 R08: ffff9256d671e620 R09: 0000000000000000
[ 304.086268] R10: 0000000000000000 R11: fefefefefefefeff R12: ffff925796085000
[ 304.086269] R13: 0000000000000000 R14: ffffffffc09cf4d0 R15: ffff92579b344680
[ 304.086272] FS: 00007fb55e4b9740(0000) GS:ffff92579bb80000(0000) knlGS:0000000000000000
[ 304.086273] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 304.086275] CR2: 0000000000000058 CR3: 0000000056ab4000 CR4: 00000000000406e0
[ 304.086281] Call Trace:
[ 304.086288] vfs_get_super+0x5b/0xe0
[ 304.086292] ? vfs_parse_fs_param+0xdc/0x1c0
[ 304.086302] nfsd_fs_get_tree+0x2c/0x30 [nfsd]
[ 304.086305] vfs_get_tree+0x2a/0x100
[ 304.086308] fc_mount+0x12/0x40
[ 304.086310] vfs_kern_mount.part.31+0x76/0x90
[ 304.086312] vfs_kern_mount+0x13/0x20
[ 304.086321] nfsd_init_net+0x101/0x140 [nfsd]
[ 304.086325] ops_init+0x44/0x120
[ 304.086327] register_pernet_operations+0xed/0x200
[ 304.086339] ? trace_event_define_fields_nfsd_stateid_class+0xb3/0xb3 [nfsd]
[ 304.086341] register_pernet_subsys+0x28/0x40
[ 304.086352] init_nfsd+0x22/0xcbc [nfsd]
[ 304.086355] do_one_initcall+0x4a/0x1fa
[ 304.086357] ? _cond_resched+0x19/0x40
[ 304.086361] ? kmem_cache_alloc_trace+0x15c/0x210
[ 304.086365] do_init_module+0x5f/0x227
[ 304.086368] load_module+0x1aa4/0x2140
[ 304.086372] __do_sys_finit_module+0xfc/0x120
[ 304.086375] ? __do_sys_finit_module+0xfc/0x120
[ 304.086379] __x64_sys_finit_module+0x1a/0x20
[ 304.086381] do_syscall_64+0x5a/0x130
[ 304.086383] entry_SYSCALL_64_after_hwframe+0x44/0xa9
[ 304.086386] RIP: 0033:0x7fb55dfd93c9
[ 304.086388] Code: 01 00 48 81 c4 80 00 00 00 e9 f1 fe ff ff 0f 1f 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 8f 9a 2c 00 f7 d8 64 89 01 48
[ 304.086390] RSP: 002b:00007ffd8171ba08 EFLAGS: 00000246 ORIG_RAX: 0000000000000139
[ 304.086392] RAX: ffffffffffffffda RBX: 000056544ce0eee9 RCX: 00007fb55dfd93c9
[ 304.086393] RDX: 0000000000000000 RSI: 000056544ce0eee9 RDI: 0000000000000003
[ 304.086395] RBP: 0000000000000000 R08: 0000000000000000 R09: 000056544d238150
[ 304.086396] R10: 0000000000000003 R11: 0000000000000246 R12: 000056544d230250
[ 304.086398] R13: 000056544d233b20 R14: 000056544d2382f0 R15: 0000000000040000
[ 304.086400] Modules linked in: nfsd(+) auth_rpcgss nfs_acl lockd grace kvm irqbypass 9pnet_virtio 9pnet minix xt_nat xt_tcpudp veth xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype iptable_filter iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c bpfilter br_netfilter bridge stp llc nls_utf8 isofs vboxsf(O) aufs overlay intel_rapl_msr snd_intel8x0 snd_ac97_codec ac97_bus snd_pcm snd_seq_midi snd_seq_midi_event snd_rawmidi vboxvideo snd_seq snd_seq_device snd_timer drm_vram_helper snd ttm drm_kms_helper drm fb_sys_fops soundcore syscopyarea joydev sysfillrect intel_rapl_common sysimgblt crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_64 crypto_simd cryptd glue_helper vboxguest(O) input_leds intel_rapl_perf serio_raw mac_hid binfmt_misc sch_fq_codel cuse parport_pc ppdev sunrpc lp parport ip_tables x_tables autofs4 hid_generic usbhid hid ahci i2c_piix4 psmouse libahci video e1000 pata_acpi
[ 304.086436] CR2: 0000000000000058
[ 304.086439] ---[ end trace 0deb363709117b2c ]---
[ 304.086449] RIP: 0010:nfsd_fill_super+0x71/0x90 [nfsd]
[ 304.086451] Code: 85 c0 89 c3 74 09 89 d8 5b 41 5c 41 5d 5d c3 49 8b 7c 24 68 31 f6 48 c7 c2 70 84 a0 c0 e8 97 fe ff ff 48 3d 00 f0 ff ff 77 0d <49> 89 45 58 89 d8 5b 41 5c 41 5d 5d c3 89 c3 eb cb 0f 1f 40 00 66
[ 304.086453] RSP: 0018:ffffba6945edfaa8 EFLAGS: 00010287
[ 304.086454] RAX: ffff9256d671e600 RBX: 0000000000000000 RCX: 0000000000000002
[ 304.086456] RDX: 0000000000000000 RSI: 0000000000000100 RDI: ffff9256cf8309e0
[ 304.086457] RBP: ffffba6945edfac0 R08: ffff9256d671e620 R09: 0000000000000000
[ 304.086459] R10: 0000000000000000 R11: fefefefefefefeff R12: ffff925796085000
[ 304.086460] R13: 0000000000000000 R14: ffffffffc09cf4d0 R15: ffff92579b344680
[ 304.086462] FS: 00007fb55e4b9740(0000) GS:ffff92579bb80000(0000) knlGS:0000000000000000
[ 304.086463] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 304.086465] CR2: 0000000000000058 CR3: 0000000056ab4000 CR4: 00000000000406e0刚开始加载模块的时候，kernel NULL pointer dereference
supervisor write access in kernel mode，error_code(0x0002) - not-present page，PGD 0 P4D 0
Oops：backtrace 中有 init_nfsd，初始化 nfsd 失败。
sync 卡死可能和在 docker 容器中加载 nfsd 模块失败有关系。
ksys_sync 下面的第一个 iterate_supers 中。相关代码如下： void ksys_sync(void)
{
int nowait = 0, wait = 1;
wakeup_flusher_threads(WB_REASON_SYNC);
iterate_supers(sync_inodes_one_sb, NULL);
iterate_supers(sync_fs_one_sb, &nowait);
iterate_supers(sync_fs_one_sb, &wait);
iterate_bdevs(fdatawrite_one_bdev, NULL);
iterate_bdevs(fdatawait_one_bdev, NULL);
if (unlikely(laptop_mode))
laptop_sync_completion();
}
void iterate_supers(void (*f)(struct super_block *, void *), void *arg)
{
struct super_block *sb, *p = NULL;
spin_lock(&sb_lock);
list_for_each_entry(sb, &super_blocks, s_list) {
if (hlist_unhashed(&sb->s_instances))
continue;
sb->s_count++;
spin_unlock(&sb_lock);
down_read(&sb->s_umount);
if (sb->s_root && (sb->s_flags & SB_BORN))
f(sb, arg);
up_read(&sb->s_umount);
spin_lock(&sb_lock);
if (p)
__put_super(p);
p = sb;
}
if (p)
__put_super(p);
spin_unlock(&sb_lock);
}iterate_supers 遍历 super_blocks 链表，执行 f(sb, arg)。perf ftrace 最后结果显示卡死在 down_read(&sb->s_umount) 下。也就是说在拿这个读写信号量锁的时候卡死了，肯定是有人 down_write 了，一直没有 up_write （写者一直持有锁，读者只能一直等）。 void __sched down_read(struct rw_semaphore *sem)
{
might_sleep();
rwsem_acquire_read(&sem->dep_map, 0, 0, _RET_IP_);
LOCK_CONTENDED(sem, __down_read_trylock, __down_read);
}
EXPORT_SYMBOL(down_read);
#define LOCK_CONTENDED(_lock, try, lock) \
lock(_lock)
inline void __down_read(struct rw_semaphore *sem)
{
if (!rwsem_read_trylock(sem)) {
rwsem_down_read_slowpath(sem, TASK_UNINTERRUPTIBLE);
DEBUG_RWSEMS_WARN_ON(!is_rwsem_reader_owned(sem), sem);
} else {
rwsem_set_reader_owned(sem);
}
}
struct rw_semaphore {
atomic_long_t count;
/*
* Write owner or one of the read owners as well flags regarding
* the current state of the rwsem. Can be used as a speculative
* check to see if the write owner is running on the cpu.
*/
atomic_long_t owner;
#ifdef CONFIG_RWSEM_SPIN_ON_OWNER
struct optimistic_spin_queue osq; /* spinner MCS lock */
#endif
raw_spinlock_t wait_lock;
struct list_head wait_list;
#ifdef CONFIG_DEBUG_LOCK_ALLOC
struct lockdep_map dep_map;
#endif
};rwsem_down_read_slowpath。用 trace 抓取 rwsem_down_read_slowpath，找到 sem 的 owner。 luo@luo:/usr/share/bcc/tools$ sudo ./trace 'rwsem_down_read_slowpath(struct rw_semaphore *sem, int state) "count=0x%lx owner=%s", sem->count.counter, ((struct task_struct *)((sem->owner.counter)&~0x7))->comm'
/virtual/main.c:44:66: warning: comparison of array '((struct task_struct *)((sem->owner.counter) & ~7))->comm' not equal to a null pointer is
always true [-Wtautological-pointer-compare]
if (((struct task_struct *)((sem->owner.counter)&~0x7))->comm != 0) {
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~ ~
1 warning generated.
PID TID COMM FUNC -
10127 10127 sync rwsem_down_read_slowpath count=0x103 owner=modprobe从 down_read(&sb->s_umount) 可以发现，sem 是 super_block 的成员，可以通过 sem 获取 super_block 相关的信息。
luo@luo:/usr/share/bcc/tools$ sudo ./trace 'rwsem_down_read_slowpath(struct rw_semaphore *sem, int state) "count=0x%lx fs name=%s", sem->count.counter, (((struct super_block*)((void *)sem-(void*)(&(((struct super_block*)0)->s_umount))))->s_id)'
PID TID COMM FUNC -
10144 10144 sync rwsem_down_read_slowpath count=0x103 fs name=nfsd到此，已经找到了 sync 卡死和挂载 nfsd 失败的两个直接联系。
当 Linux >= v5.3 时，在 docker 镜像中加载 nfsd 模块失败，并且 modprobe 退出后 nfsd 模块的读写锁被写者 modprobe 持有，导致 sync 命令申请获取该锁时失败，一直等待。
这个 bug 目前只有一个 workaround 的方案：
针对 Linux >= 5.3 的，在 host 侧插入 nfsd 模块。
针对 win 和 macosx，暂时没有比较好的策略，保留原有策略，后续需要确认 win 和 macosx docker 所用版本，如果不超过 5.3，应该是没问题。
后续找到在 docker 中加载 nfsd 失败的原因了，会写一篇对应的文章。
sudo apt update -y && sudo apt upgrade -y
sudo apt install -y linux-tools-common linux-tools-`uname -r` linux-cloud-tools-`uname -r`docker pull zlim/bcc
docker run -it --rm \
--privileged \
-v /lib/modules:/lib/modules:ro \
-v /usr/src:/usr/src:ro \
-v /etc/localtime:/etc/localtime:ro \
--workdir /usr/share/bcc/tools \
zlim/bcc本文由哈喽比特于4年以前收录，如有侵权请联系我们。
文章来源：https://mp.weixin.qq.com/s/FrudEz2CPrlqeaO5o_kdBg
京东创始人刘强东和其妻子章泽天最近成为了互联网舆论关注的焦点。有关他们“移民美国”和在美国购买豪宅的传言在互联网上广泛传播。然而，京东官方通过微博发言人发布的消息澄清了这些传言，称这些言论纯属虚假信息和蓄意捏造。
日前，据博主“@超能数码君老周”爆料，国内三大运营商中国移动、中国电信和中国联通预计将集体采购百万台规模的华为Mate60系列手机。
据报道，荷兰半导体设备公司ASML正看到美国对华遏制政策的负面影响。阿斯麦（ASML）CEO彼得·温宁克在一档电视节目中分享了他对中国大陆问题以及该公司面临的出口管制和保护主义的看法。彼得曾在多个场合表达了他对出口管制以及中荷经济关系的担忧。
今年早些时候，抖音悄然上线了一款名为“青桃”的 App，Slogan 为“看见你的热爱”，根据应用介绍可知，“青桃”是一个属于年轻人的兴趣知识视频平台，由抖音官方出品的中长视频关联版本，整体风格有些类似B站。
日前，威马汽车首席数据官梅松林转发了一份“世界各国地区拥车率排行榜”，同时，他发文表示：中国汽车普及率低于非洲国家尼日利亚，每百户家庭仅17户有车。意大利世界排名第一，每十户中九户有车。
近日，一项新的研究发现，维生素 C 和 E 等抗氧化剂会激活一种机制，刺激癌症肿瘤中新血管的生长，帮助它们生长和扩散。
据媒体援引消息人士报道，苹果公司正在测试使用3D打印技术来生产其智能手表的钢质底盘。消息传出后，3D系统一度大涨超10%，不过截至周三收盘，该股涨幅回落至2%以内。
9月2日，坐拥千万粉丝的网红主播“秀才”账号被封禁，在社交媒体平台上引发热议。平台相关负责人表示，“秀才”账号违反平台相关规定，已封禁。据知情人士透露，秀才近期被举报存在违法行为，这可能是他被封禁的部分原因。据悉，“秀才”年龄39岁，是安徽省亳州市蒙城县人，抖音网红，粉丝数量超1200万。他曾被称为“中老年...
9月3日消息，亚马逊的一些股东，包括持有该公司股票的一家养老基金，日前对亚马逊、其创始人贝索斯和其董事会提起诉讼，指控他们在为 Project Kuiper 卫星星座项目购买发射服务时“违反了信义义务”。
据消息，为推广自家应用，苹果现推出了一个名为“Apps by Apple”的网站，展示了苹果为旗下产品（如 iPhone、iPad、Apple Watch、Mac 和 Apple TV）开发的各种应用程序。
特斯拉本周在美国大幅下调Model S和X售价，引发了该公司一些最坚定支持者的不满。知名特斯拉多头、未来基金（Future Fund）管理合伙人加里·布莱克发帖称，降价是一种“短期麻醉剂”，会让潜在客户等待进一步降价。
据外媒9月2日报道，荷兰半导体设备制造商阿斯麦称，尽管荷兰政府颁布的半导体设备出口管制新规9月正式生效，但该公司已获得在2023年底以前向中国运送受限制芯片制造机器的许可。
近日，根据美国证券交易委员会的文件显示，苹果卫星服务提供商 Globalstar 近期向马斯克旗下的 SpaceX 支付 6400 万美元（约 4.65 亿元人民币）。用于在 2023-2025 年期间，发射卫星，进一步扩展苹果 iPhone 系列的 SOS 卫星服务。
据报道，马斯克旗下社交平台𝕏（推特）日前调整了隐私政策，允许 𝕏 使用用户发布的信息来训练其人工智能（AI）模型。新的隐私政策将于 9 月 29 日生效。新政策规定，𝕏可能会使用所收集到的平台信息和公开可用的信息，来帮助训练 𝕏 的机器学习或人工智能模型。
9月2日，荣耀CEO赵明在采访中谈及华为手机回归时表示，替老同事们高兴，觉得手机行业，由于华为的回归，让竞争充满了更多的可能性和更多的魅力，对行业来说也是件好事。
《自然》30日发表的一篇论文报道了一个名为Swift的人工智能（AI）系统，该系统驾驶无人机的能力可在真实世界中一对一冠军赛里战胜人类对手。
近日，非营利组织纽约真菌学会（NYMS）发出警告，表示亚马逊为代表的电商平台上，充斥着各种AI生成的蘑菇觅食科普书籍，其中存在诸多错误。
社交媒体平台𝕏（原推特）新隐私政策提到：“在您同意的情况下，我们可能出于安全、安保和身份识别目的收集和使用您的生物识别信息。”
2023年德国柏林消费电子展上，各大企业都带来了最新的理念和产品，而高端化、本土化的中国产品正在不断吸引欧洲等国际市场的目光。
罗永浩日前在直播中吐槽苹果即将推出的 iPhone 新品，具体内容为：“以我对我‘子公司’的了解，我认为 iPhone 15 跟 iPhone 14 不会有什么区别的，除了序（列）号变了，这个‘不要脸’的东西，这个‘臭厨子’。
