I have an Anbernic RG353M & it sounds like it fits your needs. Emulates up to PSP. Moddable: I run ArkOS which is Debian-based - so I can SSH in, apt install programs, run custom bash scripts I wrote, etc. I also ported a desktop game to run on it. So moddability points are high! Also dual-boots into Android, so Android games are available if that’s your thing. It comes with the Android dual-boot right out of the box!
Downsides:
It can be a little small in my hands for long play sessions. I have a custom 3D printed handle extension which works great and solves this problem.
There’s a hardware problem involving the 3.5mm audio jack output where chip noise gets in the audio. Annoying, but ignorable. Switching to Bluetooth headphones is a fine workaround too.
In long play sessions when the device gets hot horizontal bar artifacts show up on the screen. Like the audio thing, they are annoying but ignorable. When they show up I normally interpret it as a sign I’ve been playing too long and take a break while the device cools down.
Overall, it’s got those hardware quirks above but I still like it and I don’t regret it for the price (something like ~$100 a couple of years ago I think). Battery life is very good, even for graphics intensive games. It can go many hours without needing a charge, and generally it has better stamina than I do!