Heavy metals like lead accumulate in the body, so exposure over time is a concern too.
In the US, childhood lead poisoning cases are opened by public health investigators when the kid’s blood lead level is reported over 3.5 micrograms of lead per deciliter of blood. That’s not much - and if the kid’s eating contaminated food everyday for multiple sittings, I can see how it can accumulate over that threshold quickly.
For example, lead contaminated soil is considered a hazard if measured to have over 400 ppm lead. For things actually meant to be eaten, a fraction of that ppm is all it takes.