You might want to start by rebalancing by percentages and not all at once. If nothing else it’ll tell you much sooner if you’re on the right track or not. Something like sudo btrfs balance start -dusage=20 -musage=20 /mnt/disk3 to work on only blocks that are 20% full or less. That should coaleace them into single data blocks and free up some others.