My experience is similar. I don’t play YouTube videos on my 4B with 8GB of RAM very often. When I do, I make sure it’s well less than a quarter of my 1920x1080 screen. (I use a tiling window manager, so I usually just make my browser window the top-left quadrant of my screen and don’t theater-mode or anything.) And I often reduce the quality to 480p or whatever.
If I’m going to watch something longer than a few minutes and want to be doing other things on my Raspberry Pi while the video is running, I’ll just pull it up on my phone propped next to my monitor.