I think the main drawback for me was how unlikable I found Booker and Elizabeth to be. It didn't help that Elizabeth would start talking every time I tried to listen to a voxophone. I remember having to take a break from the game after attempting to listen to one 3 or 4 times, but she wouldn't stop talking.
I really like the gameplay though. I'll boot up Infinite every once in a while to play Clash in the Clouds.