Star Trek really has 2 different genres, there's action/adventure and there's real hard sci-fi where philosophy is at the forefront. Voyager generally appeals more to the action/adventure fans, whereas the previous iterations appeared like the entire series was heading in a more philosophical direction with TOS to TNG to DS9 increasing in their thoughtfulness. VOY was seen as a huge backslide to people who were tuning in largely for the philosophical aspect of the show.
Considering there was and still are very few popular philosophical and thought provoking shows that challenge the viewer's world view and biases, I think it's fair to be upset that the new direction of the show is to dumb down everything and focus more on the action.
Of course, that's not to say that Voyager was completely devoid of any philosophical debate, but I don't think anyone can make the case that it's equally as intelligent as TNG and DS9.