I have 1.8TB of pics and videos on my server and it all runs fine in browsers or the app. Perhaps you could look at disabling the machine learning portion of it to save some overhead? For reference, my setup is:
Server - HP Micro server G8 Phone - Samsung Galaxy Note 10+ Tablet - Samsung Galaxy S7+
None of these are high end anymore, hell the server is ancient by computer standards. But I did put a GPU in it for ex hardware transcoding as well as machine learning for Immich.