BYR Achieve · 镜像论坛

公司新组装了一台训练集来微调模型用，配置的环境如下 OS: Ubuntu 24.04.4 LTS x86_64 Kernel: 6.17.0-14-generic CPU: Intel Ultra 9 285K (24) @ 5.700GHz GPU: NVIDIA 5090d 24G * 2 MEMORY: 192488MiB (48G * 4) GPU驱动版本 Driver Version: 580.126.09 CUDA Version: 13.0 使用conda虚拟环境 python=3.12 torch=2.10.0 我现在遇到了一个非常诡异的问题，即运行部分软件会报错segfault：（1）uv：无报错（2）apt：极低概率报错（3）swift：极高概率报错，启动后训练过程中也极高概率报错（4）python：import torch会大概率报错使用dmesg抓取最近的segfault内核日志如下 [Tue Mar 17 16:09:48 2026] python3[530826]: segfault at 71b67da9a22d ip 000000000050723e sp 00007ffd20b29d70 error 6 in python3.12[10723e,420000+259000] likely on CPU 0 (core 0, socket 0) [Tue Mar 17 16:15:53 2026] swift[545813]: segfault at 1 ip 0000000000576c9f sp 00007ffe8b364080 error 6 in python3.12[176c9f,420000+259000] likely on CPU 0 (core 0, socket 0) [Tue Mar 17 16:16:01 2026] python3[546273]: segfault at 39 ip 0000000000528a78 sp 00007ffc4b976870 error 6 in python3.12[128a78,420000+259000] likely on CPU 0 (core 0, socket 0) [Tue Mar 17 16:21:13 2026] pt_autograd_0[558829]: segfault at 105887d8 ip 00007b78e17d4134 sp 00007b77bc7e6480 error 4 in libcuda.so.580.126.09[1d4134,7b78e1766000+f84000] likely on CPU 0 (core 0, socket 0) [Tue Mar 17 16:24:01 2026] python3[566928]: segfault at 78 ip 000072c656aca182 sp 00007ffc467945f0 error 4 in libtorch_cpu.so[1cca182,72c655e22000+10f4a000] likely on CPU 0 (core 0, socket 0) [Tue Mar 17 16:24:07 2026] python3[567275]: segfault at ffffffff85e30ff0 ip 000000000053837d sp 00007ffce23a1230 error 5 in python3.12[13837d,420000+259000] likely on CPU 0 (core 0, socket 0) [Tue Mar 17 16:24:15 2026] swift[567630]: segfault at 6d0307f ip 000000000052177d sp 00007fff46d2efe0 error 4 in python3.12[12177d,420000+259000] likely on CPU 0 (core 0, socket 0) [Tue Mar 17 16:24:21 2026] python3[567973]: segfault at 0 ip 000072c91c6ca1cb sp 00007ffff3b1cf20 error 6 in libtorch_cpu.so[1cca1cb,72c91ba22000+10f4a000] likely on CPU 0 (core 0, socket 0) [Tue Mar 17 16:24:26 2026] swift[568209]: segfault at 76ba53003fd0 ip 000076ba53003fd0 sp 00007ffc4830cca8 error 15 in libc.so.6[202fd0,76ba53003000+2000] likely on CPU 0 (core 0, socket 0) [Tue Mar 17 16:48:01 2026] python[673744]: segfault at 89ee8a0 ip 0000000000576c7b sp 00007ffd13ad3760 error 4 in python3.12[176c7b,420000+259000] likely on CPU 0 (core 0, socket 0) [Tue Mar 17 16:48:18 2026] python[675642]: segfault at 9880ff4 ip 0000000000536e80 sp 00007ffc1f8aa288 error 6 in python3.12[136e80,420000+259000] likely on CPU 0 (core 0, socket 0) [Tue Mar 17 16:50:09 2026] python[687279]: segfault at ffffffffff9fe8a0 ip 0000000000576c7b sp 00007fff4ae451c0 error 5 in python3.12[176c7b,420000+259000] likely on CPU 0 (core 0, socket 0) [Tue Mar 17 16:56:14 2026] python[722763]: segfault at 1 ip 000000000055dab3 sp 00007ffc5c6d9728 error 6 in python3.12[15dab3,420000+259000] likely on CPU 0 (core 0, socket 0) [Tue Mar 17 17:03:01 2026] swift[746742]: segfault at 7e770e4ac248 ip 00007e770d1894c0 sp 00007ffff1cc8348 error 4 in libc.so.6[1894c0,7e770d028000+188000] likely on CPU 3 (core 20, socket 0) 为了检查内存健康程度，我执行了一些机器压力测试（1）memTest86，结果为全部通过（2）cpp程序：随机分配内存，验证内存数据一致性，全部通过（3）cpu：使用numpy进行大矩阵乘法运算，全部通过未报错（4）CUDA脚本：进行大矩阵乘法运算，全部通过未报错我还做过以下尝试（1）拔掉三根，只保留一根内存，无法解决问题（2）内存关闭XMP，频率调到4400M - 4800M，无法解决问题（3）主板BIOS更新到最新，无法解决问题现在不知道机器究竟是什么问题，有大佬能帮看下吗，感激不尽

大量随机segfault求助