But say (simplying greatly) the public key tells my computer to multiply my text by a prime number
If the prime number is already known from the public key, then why is any computation required? To decrypt it can’t I (or anyone else) just divide by the prime? Even with a significantly more complex calculation, can’t you just work the steps back in reverse using the instructions from the public key?