My personal problem with Voyager was that it didn't really advance the series. We had The Original Series which was innovative for its time and The Animated Series which was more of the same. Then we had The Next Generation which really modernized Star Trek and made it fit into the 80's and then we got Deep Space Nine which was Star Trek finally really growing up and leaving its campy roots. For me, Voyager was a step backward, it was a lot more like The Next Generation in it's style and I felt that it failed to bring anything new to the table, as did Enterprise which, angering many fans, wen't backwards in the timeline.
The two hour long pilot of TNG was full of political and social commentary and groundbreaking stuff in general. The pilot of DS9 changed the face of Trek from the very first scene with shaky cameras, amazingly large and detailed sets, and dark, dramatic lighting. Voyager should have been the next step in the path but I think in the long run, both Voyager and Enterprise felt like they early 90's Trek and I think that's why they are less accepted among fans, even though Voyager did well.
I still enjoy both Voyager and Enterprise, but I feel that Star Trek XI was the first of the Star Trek franchise to truly leave the 20th century.