I don’t see how swap has much chance to fragment. A swapfile has to be fully allocated up front and cannot be CoW. If it’s allocated well in the first place, it will stay that way.
The swap code doesn’t really do I/O through the filesystem. AIUI, it locks the file, gets the disk block #s from the FS, and after that it accesses those blocks directly.