I want to mention that one can set the pager for man to be Vim too. Then it would load the document in Vim instead in less for display and navigation. This can be set with option man -P pager or with the environmental variable $MANPAGER or $PAGER . I had set this up in the past with original Vim, but it required some special options for Vim as well. It was nice, but ultimately not needed; so I went back to less. Sometimes less is more.
Edit: Here is how one can use Neovim as the pager: