I came to the same conclusion about my usual disinterest in books stemming from me having Aphantasia. The only kinds of books I’ve been able to consistently get through are very comedic in their writing style (e.g. Douglas Adams, Terry Pratchett, Dennis E. Taylor, etc.). I think the focus on humor instead of visualizing the story and its world is what helps me when it comes to reading books.