Many years ago when I was still doing my undergrad I had a cyber security prof talk about side channels:
”There’s no way to prevent side-channels. As long as two components are sharing the same physical resource there will be side channels. The only problem is that these side channels are leaking way more bits than we expected.”
So the question here is how big does the side channel need to be to leak something sensitive from memory? Turning off mitigations will almost certainly lead to larger side channels. Whether that is worth the risk is up to you.