Is it possible to go too far in one direction? You’ll never know unless you find out, and there’s no guarantee you’ll ever find out.
I think in his later years Feynman did LSD and started feeling more comfortable with the potential philosophical ramifications of his work. We have to draw a line somewhere as a starting point for our understanding of reality, but it’s important to always be willing to reconsider that line. That’s what good science is all about.