It’ll be much faster the next time. It has to make sure all the data is out of the space to be freed. Assuming it moves it as close to the start of the partition as it can (and you’re shrinking it from the end) then it’ll be faster.
If you’re shrinking it from the start, yeah, it’s going to take forever because it will always have to move a lot of data.