You’re not doing it wrong. It’s just a bit like that at the moment.
What Peertube needs is for other Fediverse platforms to build in a filter so only posts which contain video are displayed.
Peertube posts do federate but because there are way less videos produced than text and image posts (for obvious reasons) they are drowned out. Filtering by content type seems like the obvious solution. This is how Pixelfed works (but for still images), so it’s clearly possible.
My feeling is that we’re still in the first iteration of the Fediverse, where it’s mostly trying to replace the functionality of corporate platforms. Which I think has mostly been achieved now. Hopefully it will start leaning into its unique strengths now and become really innovative.
Of course there is also the issue of adoption and Peertube has had less publicity than Mastodon and Lemmy, so it’s understandable that there is less content. If you can, contact your favourite youtubers and ask them to start a Peertube channel. If enough of us do this it starts to become a thing :)