I’m sorry but I think that’s just the way Java Edition goes mate, lol.
You see a modpack that recommends 6GB allocated and you think “oh, I’m fine, I have 16”, next thing you know you’re almost going OOM.
I have recently upgraded to 32GB solely because of ‘All The Mods 9’