You will almost certainly be able to use KDE Connect under Xfce. However, some dependencies will most likely be installed during the installation, which in turn have their own dependencies. With a bit of bad luck, you will install half of Plasma, so to speak.