It gets easier, but I think a mix of the two is more proper. For example, fantasy books may use the words hauberk or tabard a lot, but not necessarily you will know exactly what it is, as it would entail that you had encyclopedic knowledge about it. Besides dictionaries, you would have to refer to Wikipedia or other encyclopedias, which is fine and can be interesting if you are curious.
You have to balance your curiosity with the need to read along the text in a rhythm that you like. I also find those estimates used in many internet articles (indicated by x min read) to be much longer than what I usually take.
It gets better and you do not have to be perfect at it, as nobody is.