Yeah, I was very dissapointed by that as well. As far as I’m aware, the only time it matters is when
Act 3 spoilerThe Emperor offers you the Astral tadpole, if you used any worms you have to pass a check that grows harder the more worms you consumed to resist using it, if you abstained you can just say no.