I was finally able to fix the washed out issue when enabling HDR.
Turns out my TV has known HDR issues with devices like the AppleTV Apple DiscussionI reset the picture to default. Opened the picture menu where it displays the signal. Enabled HDR on the Display Configuration and this time instead of washed out, it was an actual proper HDR desktop.