It is the “Torrent Address detection” magnet link I am using, and it is this that reveals my real IP when the VPN is disabled. The traffic in qBittorrent stops though.
EDIT: And as I mentioned in an earlier post, it works as intended if I open the client when not connected to the VPN. It is just if the client is running while I disconnect that this problem occurs, as far as I can tell.