Note that this is of course a very theoretical attack vector.
Wouldn’t it then decrypt to gibberish data unless they already had the encryption keys?
Depends. I don’t know the situation of LUKS and its commonly used ciphers in particulare but even some commonly used ciphers are vulnerable to things like bitflip attacks.
This is usually “fixed” by authenticating them but that’s not easily possible at the block layer.
If it decrypts incorrectly, shouldn’t BTRFS checksumming then return an I/O error to user space as well?
Note that btrfs usually uses CRCs, not cryptographic checksums. They’re designed to catch “naturally” occuring corruption, not crafted corruption. Naturally, it’d still be extremely hard to break them when working with encrypted data but it’s a “uh, sounds pretty hard” situtation, not a “we can prove you’d need billions of years to do it” one.
You can use cryptographic checksums but note again here that the attacker could be able to modify the checksum aswell.
I don’t know how feasible this really is a but a possible attack could be to tell btrfs that the extent you modified is a nochecksum extent (you can turn off checksums in btrfs) which would make btrfs simply not check the checksum.