The BubbleUPnP app can cast to Chromecast (and DLNA) and has built-in support for Jellyfin as a media source (you can browse your Jellyfin content from the Bubble app). It’s basically the “remote” you want.
Bubble can also act as a media player so if you launch a video file from a file manager and tell it to play on Bubble while it’s connected to a Chromecast, it will cast it.
Bubble can also directly access local files on the phone, Samba shares, and various types of cloud storage accounts.
Its only shortcoming is no SSH support but there’s a workaround for that: the Solid Explorer file manager has a built-in relay that will pipe a video file over SSH to a local video player… pick Bubble as that player and you can play to Chromecast from SSH.