Requesting Help with `txg_sync` Error I Can't Diagnose
First, before this rather large infodump, I want to thank anyone that takes the time to read through this to offer any information or advice on trying to resolve this issue.
Here’s the issue I’ve been struggling with.
I keep getting this error on my server:
<span style="color:#323232;">INFO: task txg_sync:1615 blocked for more than 241 seconds.
</span><span style="color:#323232;">Tainted: P O 5.15.0-112-generic #122-Ubuntu
</span>
Background:
I’ve got a homeserver running Ubuntu server 22.04 (no DE) with an Intel Core i7-6700K CPU @ 4.00GHz, 32GB RAM, and two ZFS pools. The OS is installed on its own 128GB SSD, and the two ZFS pools consist of a 128GB SSD in its own pool for the server’s cache, and a 4x8TB HDD RAIDZ1 pool that is my main data/server storage (much more detailed system info below).
I have a bunch of services running in Docker containers, and overall everything is great, except for when that error rears up.
The error seems random, but occurs most reliably, but not consistently, when I’m trying to write larger media files to the RAIDZ1 pool. I am aware that this an IOPS issue, but so far I have not been able to diagnose it.
Back in February, I was carrying some boxes down to the basement where the server rack is, and I accidentally kicked a stool into the server which knocked the shit out of it (it’s a tower pc that I built back in 2016 or 2017 and repurposed in 2020ish to server use). When I turned on the monitor it was in total panic mode. The screen was gibberish colors and flickering madness.
I had to force shutdown with the power button. I waited a good couple of minutes, and on reboot, everything seemed fine. Until I noticed the new txg_sync error a week or so later when I went down to add some media to the server.
After a lot of searching and reading that didn’t turn up pertinent info, I ran across a comment that said this error is almost always hardware related. Like a loose connection or a failing disk or something. With me having knocked the shit out of the server, I realized I should have opened the case up and checked it all out. I shut it down, opened it up, and found a loose connector on the motherboard. I reseated it, checked everything else (though not thoroughly enough, which we’ll get to), and rebooted hoping I had found the problem.
It seemed fine for a bit, but no luck. The error returned.
More searching with no luck, and then about a month ago, a friend he suggested I check all the SATA connectors by disconnecting each one and reconnecting to insure a good, solid connection. I had previously checked if they were seated when I opened the case, but didn’t disconnect and reconnect. While doing this, I found a SATA cable with a busted clip and thought again I had found the problem. I replaced the cable, and went about a week before the error resurfaced.
It continues to occur, as I mentioned inconsistently. Most reliably, but not always, when writing data to the RAIDZ1 pool.
I have run a thorough memtest, and there were no errors or issue with the RAM, and as far as I can tell, there are no errors/failures with the HDDs.
Below is a lot of system info, and an example of what I find in dmesg for the error.
System Info
OS & Kernel
<span style="color:#323232;">Ubuntu 22.04.4 LTS
</span><span style="color:#323232;">Linux [redacted user] 5.15.0-113-generic #123-Ubuntu SMP Mon Jun 10 08:16:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
</span>
CPU
<span style="color:#323232;">Architecture: x86_64
</span><span style="color:#323232;"> CPU op-mode(s): 32-bit, 64-bit
</span><span style="color:#323232;"> Address sizes: 39 bits physical, 48 bits virtual
</span><span style="color:#323232;"> Byte Order: Little Endian
</span><span style="color:#323232;">CPU(s): 8
</span><span style="color:#323232;"> On-line CPU(s) list: 0-7
</span><span style="color:#323232;">Vendor ID: GenuineIntel
</span><span style="color:#323232;"> Model name: Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz
</span><span style="color:#323232;"> CPU family: 6
</span><span style="color:#323232;"> Model: 94
</span><span style="color:#323232;"> Thread(s) per core: 2
</span><span style="color:#323232;"> Core(s) per socket: 4
</span><span style="color:#323232;"> Socket(s): 1
</span><span style="color:#323232;"> Stepping: 3
</span><span style="color:#323232;"> CPU max MHz: 4200.0000
</span><span style="color:#323232;"> CPU min MHz: 800.0000
</span><span style="color:#323232;"> BogoMIPS: 7999.96
</span><span style="color:#323232;"> Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dno wprefetch cpuid_fault invpcid_single pti ssbd ibrs ibpb stibp fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp md_clear flush_l1d arch_capabilities
</span><span style="color:#323232;">Caches (sum of all):
</span><span style="color:#323232;"> L1d: 128 KiB (4 instances)
</span><span style="color:#323232;"> L1i: 128 KiB (4 instances)
</span><span style="color:#323232;"> L2: 1 MiB (4 instances)
</span><span style="color:#323232;"> L3: 8 MiB (1 instance)
</span><span style="color:#323232;">NUMA:
</span><span style="color:#323232;"> NUMA node(s): 1
</span><span style="color:#323232;"> NUMA node0 CPU(s): 0-7
</span><span style="color:#323232;">Vulnerabilities:
</span><span style="color:#323232;"> Gather data sampling: Vulnerable: No microcode
</span><span style="color:#323232;"> Itlb multihit: KVM: Mitigation: VMX unsupported
</span><span style="color:#323232;"> L1tf: Mitigation; PTE Inversion
</span><span style="color:#323232;"> Mds: Mitigation; Clear CPU buffers; SMT vulnerable
</span><span style="color:#323232;"> Meltdown: Mitigation; PTI
</span><span style="color:#323232;"> Mmio stale data: Mitigation; Clear CPU buffers; SMT vulnerable
</span><span style="color:#323232;"> Retbleed: Mitigation; IBRS
</span><span style="color:#323232;"> Spec rstack overflow: Not affected
</span><span style="color:#323232;"> Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
</span><span style="color:#323232;"> Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
</span><span style="color:#323232;"> Spectre v2: Mitigation; IBRS; IBPB conditional; STIBP conditional; RSB filling; PBRSB-eIBRS Not
</span><span style="color:#323232;"> affected; BHI Not affected
</span><span style="color:#323232;"> Srbds: Mitigation; Microcode
</span><span style="color:#323232;"> Tsx async abort: Mitigation; TSX disabled
</span>
RAM
<span style="color:#323232;">Memory Device
</span><span style="color:#323232;"> Array Handle: 0x004A
</span><span style="color:#323232;"> Error Information Handle: Not Provided
</span><span style="color:#323232;"> Total Width: 64 bits
</span><span style="color:#323232;"> Data Width: 64 bits
</span><span style="color:#323232;"> Size: 8 GB
</span><span style="color:#323232;"> Form Factor: DIMM
</span><span style="color:#323232;"> Set: None
</span><span style="color:#323232;"> Locator: DIMM_A2
</span><span style="color:#323232;"> Bank Locator: BANK 1
</span><span style="color:#323232;"> Type: DDR4
</span><span style="color:#323232;"> Type Detail: Synchronous
</span><span style="color:#323232;"> Speed: 2133 MT/s
</span><span style="color:#323232;"> Manufacturer: Corsair
</span><span style="color:#323232;"> Serial Number: 00000000
</span><span style="color:#323232;"> Asset Tag: 9876543210
</span><span style="color:#323232;"> Part Number: CMK16GX4M2A2400C16
</span><span style="color:#323232;"> Rank: 2
</span><span style="color:#323232;"> Configured Memory Speed: 2133 MT/s
</span><span style="color:#323232;"> Minimum Voltage: Unknown
</span><span style="color:#323232;"> Maximum Voltage: Unknown
</span><span style="color:#323232;"> Configured Voltage: 1.2 V
</span>
<span style="color:#323232;">Handle 0x004D, DMI type 17, 40 bytes
</span><span style="color:#323232;">Memory Device
</span><span style="color:#323232;"> Array Handle: 0x004A
</span><span style="color:#323232;"> Error Information Handle: Not Provided
</span><span style="color:#323232;"> Total Width: 64 bits
</span><span style="color:#323232;"> Data Width: 64 bits
</span><span style="color:#323232;"> Size: 8 GB
</span><span style="color:#323232;"> Form Factor: DIMM
</span><span style="color:#323232;"> Set: None
</span><span style="color:#323232;"> Locator: DIMM_B1
</span><span style="color:#323232;"> Bank Locator: BANK 2
</span><span style="color:#323232;"> Type: DDR4
</span><span style="color:#323232;"> Type Detail: Synchronous
</span><span style="color:#323232;"> Speed: 2133 MT/s
</span><span style="color:#323232;"> Manufacturer: Corsair
</span><span style="color:#323232;"> Serial Number: 00000000
</span><span style="color:#323232;"> Asset Tag: 9876543210
</span><span style="color:#323232;"> Part Number: CMK16GX4M2A2400C16
</span><span style="color:#323232;"> Rank: 1
</span><span style="color:#323232;"> Configured Memory Speed: 2133 MT/s
</span><span style="color:#323232;"> Minimum Voltage: Unknown
</span><span style="color:#323232;"> Maximum Voltage: Unknown
</span><span style="color:#323232;"> Configured Voltage: 1.2 V
</span>
<span style="color:#323232;">Handle 0x004E, DMI type 17, 40 bytes
</span><span style="color:#323232;">Memory Device
</span><span style="color:#323232;"> Array Handle: 0x004A
</span><span style="color:#323232;"> Error Information Handle: Not Provided
</span><span style="color:#323232;"> Total Width: 64 bits
</span><span style="color:#323232;"> Data Width: 64 bits
</span><span style="color:#323232;"> Size: 8 GB
</span><span style="color:#323232;"> Form Factor: DIMM
</span><span style="color:#323232;"> Set: None
</span><span style="color:#323232;"> Locator: DIMM_B2
</span><span style="color:#323232;"> Bank Locator: BANK 3
</span><span style="color:#323232;"> Type: DDR4
</span><span style="color:#323232;"> Type Detail: Synchronous
</span><span style="color:#323232;"> Speed: 2133 MT/s
</span><span style="color:#323232;"> Manufacturer: Corsair
</span><span style="color:#323232;"> Serial Number: 00000000
</span><span style="color:#323232;"> Asset
</span>
Disks
<span style="color:#323232;">NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
</span><span style="color:#323232;">loop0 7:0 0 63.9M 1 loop /snap/core20/2264
</span><span style="color:#323232;">loop1 7:1 0 63.9M 1 loop /snap/core20/2318
</span><span style="color:#323232;">loop2 7:2 0 87M 1 loop /snap/lxd/27948
</span><span style="color:#323232;">loop3 7:3 0 87M 1 loop /snap/lxd/28373
</span><span style="color:#323232;">loop4 7:4 0 38.7M 1 loop /snap/snapd/21465
</span><span style="color:#323232;">loop5 7:5 0 38.8M 1 loop /snap/snapd/21759
</span><span style="color:#323232;">sda 8:0 0 223.6G 0 disk
</span><span style="color:#323232;">├─sda1 8:1 0 1M 0 part
</span><span style="color:#323232;">├─sda2 8:2 0 2G 0 part /boot
</span><span style="color:#323232;">└─sda3 8:3 0 221.6G 0 part
</span><span style="color:#323232;"> └─ubuntu--vg-ubuntu--lv 253:0 0 100G 0 lvm /
</span><span style="color:#323232;">sdb 8:16 0 223.6G 0 disk
</span><span style="color:#323232;">├─sdb1 8:17 0 223.6G 0 part
</span><span style="color:#323232;">└─sdb9 8:25 0 8M 0 part
</span><span style="color:#323232;">sdc 8:32 0 7.3T 0 disk
</span><span style="color:#323232;">├─sdc1 8:33 0 7.3T 0 part
</span><span style="color:#323232;">└─sdc9 8:41 0 8M 0 part
</span><span style="color:#323232;">sdd 8:48 0 7.3T 0 disk
</span><span style="color:#323232;">├─sdd1 8:49 0 7.3T 0 part
</span><span style="color:#323232;">└─sdd9 8:57 0 8M 0 part
</span><span style="color:#323232;">sde 8:64 0 7.3T 0 disk
</span><span style="color:#323232;">├─sde1 8:65 0 7.3T 0 part
</span><span style="color:#323232;">└─sde9 8:73 0 8M 0 part
</span><span style="color:#323232;">sdf 8:80 0 7.3T 0 disk
</span><span style="color:#323232;">├─sdf1 8:81 0 7.3T 0 part
</span><span style="color:#323232;">└─sdf9 8:89 0 8M 0 part
</span>
ZFS
version
<span style="color:#323232;">zfs-2.1.5-1ubuntu6~22.04.4
</span><span style="color:#323232;">zfs-kmod-2.1.5-1ubuntu6~22.04.3
</span>
list
<span style="color:#323232;">NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
</span><span style="color:#323232;">srvrcache 222G 13.8G 208G - - 17% 6% 1.00x ONLINE -
</span><span style="color:#323232;">srvrpool 29.1T 17.5T 11.6T - - 1% 60% 1.00x ONLINE -
</span>
status
<span style="color:#323232;">pool: srvrcache
</span><span style="color:#323232;"> state: ONLINE
</span><span style="color:#323232;">status: Some supported and requested features are not enabled on the pool.
</span><span style="color:#323232;"> The pool can still be used, but some features are unavailable.
</span><span style="color:#323232;">action: Enable all features using 'zpool upgrade'. Once this is done,
</span><span style="color:#323232;"> the pool may no longer be accessible by software that does not support
</span><span style="color:#323232;"> the features. See zpool-features(7) for details.
</span><span style="color:#323232;"> scan: scrub repaired 0B in 00:00:35 with 0 errors on Sun Jun 9 00:24:36 2024
</span><span style="color:#323232;">config:
</span><span style="color:#323232;">
</span><span style="color:#323232;"> NAME STATE READ WRITE CKSUM
</span><span style="color:#323232;"> srvrcache ONLINE 0 0 0
</span><span style="color:#323232;"> sdb ONLINE 0 0 0
</span><span style="color:#323232;">
</span><span style="color:#323232;">errors: No known data errors
</span><span style="color:#323232;">
</span><span style="color:#323232;"> pool: srvrpool
</span><span style="color:#323232;"> state: ONLINE
</span><span style="color:#323232;">status: Some supported and requested features are not enabled on the pool.
</span><span style="color:#323232;"> The pool can still be used, but some features are unavailable.
</span><span style="color:#323232;">action: Enable all features using 'zpool upgrade'. Once this is done,
</span><span style="color:#323232;"> the pool may no longer be accessible by software that does not support
</span><span style="color:#323232;"> the features. See zpool-features(7) for details.
</span><span style="color:#323232;"> scan: scrub repaired 0B in 11:41:27 with 0 errors on Sun Jun 9 12:05:31 2024
</span><span style="color:#323232;">config:
</span><span style="color:#323232;">
</span><span style="color:#323232;"> NAME STATE READ WRITE CKSUM
</span><span style="color:#323232;"> srvrpool ONLINE 0 0 0
</span><span style="color:#323232;"> raidz1-0 ONLINE 0 0 0
</span><span style="color:#323232;"> sdd ONLINE 0 0 0
</span><span style="color:#323232;"> sde ONLINE 0 0 0
</span><span style="color:#323232;"> sdf ONLINE 0 0 0
</span><span style="color:#323232;"> sdc ONLINE 0 0 0
</span><span style="color:#323232;">
</span><span style="color:#323232;">errors: No known data errors
</span>
iostat
<span style="color:#323232;">capacity operations bandwidth
</span><span style="color:#323232;">pool alloc free read write read write
</span><span style="color:#323232;">---------- ----- ----- ----- ----- ----- -----
</span><span style="color:#323232;">srvrcache 13.8G 208G 0 22 48.0K 291K
</span><span style="color:#323232;"> sdb 13.8G 208G 0 22 48.0K 291K
</span><span style="color:#323232;">---------- ----- ----- ----- ----- ----- -----
</span><span style="color:#323232;">srvrpool 17.5T 11.6T 33 59 2.14M 1.03M
</span><span style="color:#323232;"> raidz1-0 17.5T 11.6T 33 59 2.14M 1.03M
</span><span style="color:#323232;"> sdd - - 8 15 554K 275K
</span><span style="color:#323232;"> sde - - 7 13 536K 252K
</span><span style="color:#323232;"> sdf - - 8 15 562K 275K
</span><span style="color:#323232;"> sdc - - 8 14 539K 252K
</span><span style="color:#323232;">---------- ----- ----- ----- ----- ----- -----
</span>
PCI
<span style="color:#323232;">00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor Host Bridge/DRAM Registers (rev 07)
</span><span style="color:#323232;">00:01.0 PCI bridge: Intel Corporation 6th-10th Gen Core Processor PCIe Controller (x16) (rev 07)
</span><span style="color:#323232;">00:02.0 VGA compatible controller: Intel Corporation HD Graphics 530 (rev 06)
</span><span style="color:#323232;">00:14.0 USB controller: Intel Corporation 100 Series/C230 Series Chipset Family USB 3.0 xHCI Controller (rev 31)
</span><span style="color:#323232;">00:16.0 Communication controller: Intel Corporation 100 Series/C230 Series Chipset Family MEI Controller #1 (rev 31)
</span><span style="color:#323232;">00:17.0 SATA controller: Intel Corporation Q170/Q150/B150/H170/H110/Z170/CM236 Chipset SATA Controller [AHCI Mode] (rev 31)
</span><span style="color:#323232;">00:1b.0 PCI bridge: Intel Corporation 100 Series/C230 Series Chipset Family PCI Express Root Port #17 (rev f1)
</span><span style="color:#323232;">00:1c.0 PCI bridge: Intel Corporation 100 Series/C230 Series Chipset Family PCI Express Root Port #1 (rev f1)
</span><span style="color:#323232;">00:1d.0 PCI bridge: Intel Corporation 100 Series/C230 Series Chipset Family PCI Express Root Port #9 (rev f1)
</span><span style="color:#323232;">00:1f.0 ISA bridge: Intel Corporation Z170 Chipset LPC/eSPI Controller (rev 31)
</span><span style="color:#323232;">00:1f.2 Memory controller: Intel Corporation 100 Series/C230 Series Chipset Family Power Management Controller (rev 31)
</span><span style="color:#323232;">00:1f.3 Audio device: Intel Corporation 100 Series/C230 Series Chipset Family HD Audio Controller (rev 31)
</span><span style="color:#323232;">00:1f.4 SMBus: Intel Corporation 100 Series/C230 Series Chipset Family SMBus (rev 31)
</span><span style="color:#323232;">00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (2) I219-V (rev 31)
</span><span style="color:#323232;">01:00.0 VGA compatible controller: NVIDIA Corporation GP106 [GeForce GTX 1060 3GB] (rev a1)
</span><span style="color:#323232;">01:00.1 Audio device: NVIDIA Corporation GP106 High Definition Audio Controller (rev a1)
</span><span style="color:#323232;">03:00.0 USB controller: ASMedia Technology Inc. ASM1142 USB 3.1 Host Controller
</span>
dmesg
<span style="color:#323232;">[321540.060243] INFO: task txg_sync:1615 blocked for more than 120 seconds.
</span><span style="color:#323232;">[321540.060330] Tainted: P O 5.15.0-112-generic #122-Ubuntu
</span><span style="color:#323232;">[321540.060408] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
</span><span style="color:#323232;">[321540.060495] task:txg_sync state:D stack: 0 pid: 1615 ppid: 2 flags:0x00004000
</span><span style="color:#323232;">[321540.060498] Call Trace:
</span><span style="color:#323232;">[321540.060500] <TASK>
</span><span style="color:#323232;">[321540.060502] __schedule+0x24e/0x590
</span><span style="color:#323232;">[321540.060509] schedule+0x69/0x110
</span><span style="color:#323232;">[321540.060512] schedule_timeout+0x87/0x140
</span><span style="color:#323232;">[321540.060515] ? zio_issue_async+0x12/0x20 [zfs]
</span><span style="color:#323232;">[321540.060653] ? __bpf_trace_tick_stop+0x20/0x20
</span><span style="color:#323232;">[321540.060657] io_schedule_timeout+0x51/0x80
</span><span style="color:#323232;">[321540.060661] __cv_timedwait_common+0x12c/0x170 [spl]
</span><span style="color:#323232;">[321540.060669] ? wait_woken+0x70/0x70
</span><span style="color:#323232;">[321540.060672] __cv_timedwait_io+0x19/0x20 [spl]
</span><span style="color:#323232;">[321540.060679] zio_wait+0x116/0x220 [zfs]
</span><span style="color:#323232;">[321540.060799] dsl_pool_sync+0xb6/0x400 [zfs]
</span><span style="color:#323232;">[321540.060890] ? __mod_timer+0x214/0x400
</span><span style="color:#323232;">[321540.060894] spa_sync_iterate_to_convergence+0xe0/0x1f0 [zfs]
</span><span style="color:#323232;">[321540.060997] spa_sync+0x2dc/0x5b0 [zfs]
</span><span style="color:#323232;">[321540.061098] txg_sync_thread+0x266/0x2f0 [zfs]
</span><span style="color:#323232;">[321540.061206] ? txg_dispatch_callbacks+0x100/0x100 [zfs]
</span><span style="color:#323232;">[321540.061314] thread_generic_wrapper+0x61/0x80 [spl]
</span><span style="color:#323232;">[321540.061324] ? __thread_exit+0x20/0x20 [spl]
</span><span style="color:#323232;">[321540.061332] kthread+0x127/0x150
</span><span style="color:#323232;">[321540.061336] ? set_kthread_struct+0x50/0x50
</span><span style="color:#323232;">[321540.061339] ret_from_fork+0x1f/0x30
</span><span style="color:#323232;">[321540.061344] </TASK>
</span>
Thank you again to anyone who takes the time to offer any info or advice on resolving this.