That server sounds a bit older in the teeth… Has new thermal paste been applied to the cpu? Even if the reported temps are under 90c, you might be getting hot spots causing glitches inside the package.
Worth trying a couple of different generations of kernel as well, both newer and older. You might be hitting a regression somewhere.