Humans need to eat. The land needed for agriculture already covers significant percentage of the habitable land. About half based on our world in data [1]. Yes most of this is due to livestock so this can be significantly reduced but still.
Other species also need space to live. Even if you look at it in s strictly selfish fashion and disregard the right of other species to exist - we are part of the ecosystem so if it dies we die.