I mean that's the classic slippery slope fallacy you're employing here. The answer is, sometimes it's a more clear cut situation and other times it isn't.
But just because the next rung down your logical ladder is more of a gray area than smoking does not mean that smoking is now also as much of a gray area. That's not how this works.
This is the same style of argument people make when debating against gay marriage. Well if gay people can get married does that mean people can marry dogs now? Why not? Where do we draw the line?