Well the problem is, if I only have had three votes on my story, and two people vote one star, then it significantly lowers the apparent quality of my work to people.
To be fair, if you have only three ratings on your story and two of them are one-stars, chances are at least one of them is genuine. Unless you had the misfortune of encountering two trolls early on, or someone who bothers with multiple accounts just to bomb stories.
There should be a system like on RR, which would let only rate the book after reading a certain amount of chapters.
This would be bad for one-shots or stories with few chapters, a requirement to leave a review with a rating would be better I think. Sure, you would have less ratings in general, but it would be much easier to recognise both 1- and 5-star trolls.
Because let's face it, unjustified 5-stars are just as bad 1-stars, except authors especially seem to only care about the latter while ignoring the former entirely and some people go as far as rating 5-stars just because they see a single 1-star rating on a story without even reading it - so as to "balance out the trolls".