It makes it so that downvotes just won’t be accepted on that Federated server. You can still downvote the content from servers that do but it won’t count on the original servers.
Also to answer your question no they can’t downvote your content from the other servers, on instances that have downvotes disabled downvotes won’t be able to go through because they are disabled across that instance and therefore can’t be used by users of that instance.