As a straight man and while I also don’t celebrate Thanksgiving, I don’t think your gayness should be a reason to not celebrate a holiday with straight people. Your gayness would be as welcome as my straightness on my table.
Edit: I honestly don’t understand why I am being down voted. Op expressed how they are straight and he is gay, as if that is the issue. It is not. It sounds to me as if they were bigots and pushy with their sexuality. They could be gay like him and they still could be bigots and making him uncomfortable with being too pushy with their sexuality. In other words, I tried to express that he should draw the line at sexuality but at bigots.