What you say will absolutely be held against you, especially at work. If you can’t say anything nice, don’t say anything at all. If you’re cornered and specifically asked that question, keep the reply short and sweet.
Last time I was cornered for an assessment I replied “I’m a happy worker” and left it at that.