Religions in general are not a symbol but actual oppression and not just women but people in general.
That said, have you ever seen anyone at the olympics wearing a cross? I don’t recall. Definitely not as visible as a hijab though. Likely already banned for safety reasons (that’s definitely a thing for football, not sure about other sports).