I believe the Pictrs is a hard dependency and Lemmy just won’t work without it, and there is no way to disable the caching. You can move all of the actual images to object storage as of v0.4.0 of Pictrs if that helps.
Other fediverse servers like Mastodon actually (can be configured to) proxy all remote media (for both privacy and caching reasons), so I imagine Lemmy will move that way and probably depend even more on Pictrs.