I echo the other commenter and recommend speaking with a “registered dietician” (RD degree) about your personal nutrition goals.
Calories in/out, physical activity levels, and genetics are three of the biggest factors with blood cholesterol levels. Would you overall eat fewer calories if you switched to low fat dairy? Maybe then it’s a decent strategy for you.
Harvard’s Nutrition Source is a great educational resource about nutrition that is science based and uses accessible language.