My one addition is to consider how you will handle this as time goes on. Will you laugh it off to mutual friends with a “Why would I care what he says? There are a million people who’s opinions of me matter more.”? When would you consider it actual harassment? What impact would it have to have on your life to effect your mental well being enough to take action, and what would that action be?
I pose these questions because proactively answering them can put you in a much better situation in that happy life. Most likely, it will also mature your opinion of the situation over time, allowing you to be just the right balance of firm but level-headed on how you alter those plans when and if the time comes.