Open spaces in the dialog: why "I didn't like it" and "bad" aren't the same.

Overall, I think textual reviews are far more useful than star systems, for the reasons you state: A good textual review will state, not a single yes/no judgment about the work, but the reviewer's perception about where the work succeeded, where it failed, and what's good about it and let the reader make up their own mind.

That said, I'm kind of in love with collaborative filtering systems -- where a large number of consumers give numerical reviews to a large number of things they can make judgments about -- and the system then compares their ratings to other people who have rated other things, adjusts/weights other people's ratings based on the comparison, and can then hand back expected ratings of things the person is heretofore unfamiliar with.

The thing is, at least theoretically, a well written collaborative filtering system should be able to deal with both of the major issues with numerical ratings. People who generally like big fancy shoe stores will have their ratings largely ignored when compared against people who generally like small practical shoe stores, while people who like the same kind of shoe stores will have greater weight. People who have the same basic shape of ratings but generally rate lower can have their ratings increased when trying to calculate an "expected" rating, while the same thing can happen in reverse for people who generally rate higher (obviously, you've got compression at the top and bottom, as someone who gives a "5" for both "I liked it and would read it again" and "this was the best book I ever read, and I'm going to make a blog extolling the properties of this book" is going to have less info in their ratings than someone who rates one of those a 3 and the other a 5, but it can still extract what info is available). In theory (I don't know if there are any collaborative filtering algorithms that do this, but hey, science!), it can even extract differences across categories -- so someone who usually rated high fantasy books similarly to how I rated them but had completely varying views of horror would have a better weight with respect to me for fantasy than horror.

Of course, even this doesn't save stuff if your ratings are inconsistent -- but such a system rewards consistent ratings. (also, there's no reason in such a system to ever show an individual person's ratings to anyone except that person; it's pretty useless to anyone else!)

So...in terms of rating individual views, numeric ratings are actively harmful while detailed analysis from a good reviewer can be really useful -- but in aggregate, with good statistics (and good statistics that favor the subjective views of the reader), I think they can be really useful.

Well said (as usual).

*nodnod*

I recently found out about review sites, and spent an interesting afternoon reading about places I knew. And I found that, no matter how much I liked a place, someone will have given it a one-star rating. The diner where I've eaten breakfast every Saturday for thirty-something years got one star from someone because they wouldn't customize his home fries. I take bad review with many grains of salt.

I think the only thing I'd ever give one star -- and maybe shave off two or three points -- is Armageddon. (In the entire movie, they got one thing right -- the Moon has less gravity than the Earth. But asteroids have a nice comfortable 1g.)

This. Very much this.

I find writing reviews (properly, I mean, with the intent to be fair) helps crystallise one's own preferences - which then means one spends less time reading (and rating) things which aren't one's cup of tea. (F'rex, finally nailing down a bad review to: 'I didn't enjoy this because actually I just don't really enjoy YA stories' has meant I now know to check (when people recommend me things) whether a book is YA before I buy it. Had I not been writing it down and trying to put my finger on it, I would never have made the connection).

This is why I don't hand out scores with my reviews. I prefer to address the subject matter and try to figure out what's good and bad and what worked for me. Read me long enough, you'll get a sense of my calibration. I know I'm not as profound as some people, I don't look at things in the same way as others, I don't bring the same viewpoints or experiences to a review. I just hope that when I say something about a book or movie or whatever, it's from enough of an informed position to make what I say useful. Either in what I say and how I say it, or what I don't say or do.

I dislike narrowing it all down to numbers or letters. I wish Netflix had half-star options, so I could give things 2 1/2 stars to say that while it was an okay movie, I neither liked nor disliked it, it left me kind of "meh." Instead of the rather more definitely "liked" or "disliked". :>

THIS!! on the Netflix their system is a love or hate and nothing in between.

I started writing book reviews mostly to enhance my own reading experience by forcing me to put into words something intelligible about what I was reading. If other people read and comment on what I have to say, that's just an extra side dish of Awesome.

I'm pretty careful to explain what I think the book means and what struck me as great or not so great about it. I don't think I've ever given out stars. I agree with you--stars don't help people evaluate whether they'd feel the same way about the book as the reviewer does.

Are you leaving GR, then? Sad :( you don't have to rate to stay on the site... I keep an eye on your journal entries over there more than I do here...

Using Twisp and Catsby strips to illustrate a point is always, ALWAYS a valid strategy.


Ominous!

When I started reviewing books on my blog I made a very conscious decision that I wasn't going to use a star rating system. I find it so limiting, and I want my words to tell someone whether or not I liked a specific work, not how many stars I awarded it. Then you have the inconsistency of the ranking and how it matches up against what you said. I've seen reviews that give something say 3 out of 5, but when you read the review they didn't really like the book that much, on the other hand I've seen something where the review indicates that the book was for them better than the star rating would suggest. I think you also run the risk of people just looking at the stars and not bothering with the actual review. Having said all that I do still do my 5 favourite books of the year. I think the number 1 spot is locked down by The Girl Who Circumnavigated Fairyland etc... (I love that book so hard!), but One Salt Sea is also in there.

When I am reviewing books and they are just not hitting my sweet spots at all I usually go along the angle of "who would enjoy this" to get a decent write-up going on. Unless the book is made of fail in which case I just never ever speak of them again.

we both hate and fear clowns

Ever seen Mystery Science Theater's "Here Comes the Circus"?
http://www.youtube.com/watch?v=OnyvA2pdQvo

I have sometimes thought that, if I were to use a "star" system it would be, not out of five, but out of ten stars. Each star would represent ten percent of the total work, and the number of stars would indicate the point at which I first put the book down/tried to leave the theatre/reached for the remote control.

But that would require watching myself as much as the work, and really it's just too much work.

That was a really interesting post - and has generated so much comment that you will be busy till Christmas replying to it all. But here I am adding to it...!!

First, I agree with jslinder that your overall point holds in other spheres. I recently had a really serious example of this. I needed hospital treament and was asked to rate my pain on a ranking system. Because I had personally, in the past, suffered something even more life-threatening, I didn't go for the 'top' ranking, and as a result got a misdiagnosis - temporary but terrifying and potentially lethal. So general understanding of how ranking works is important - very very important.

Secondly, you have made me think seriously about coming out of my f'locked closet (my LJ got hacked and I am paranoid) and doing more reviews - I usually only do them for friends and then I don't rank anything, just 'unpack' the book for them. But I do rank things in my own notes/lists for my own benefit.
*5 is fabulous (within its genre) and I will say why because mindless squee is as unhelpful to the reader as mindless denigration
*4 is great with the odd flaw
*3 is good but unmemorable and I have to re-read the start and end before reviewing it to see what on earth it was about
*2 has lots of flaws which I consider carefully
*1 is dire but I actually got to the end.
*0 is for if I had to give up, or just skim
(Incidentally October rates a consistent 5).
There is huge gap in my head between 2 and 1 or 0.

Finally, the coconut problem is not solved. I wants some!!

This post is very at for me today ;-) Thanks for posting!

Which just goes to show, doesn't it? Because I quite enjoyed Moon Flash, and I don't even like SF all that much. And I'm a picky reader. On the other hand, language trumps just about everything else for me, and McKillip's language is never less than gorgeous, even when her plot or worldbuilding is less inspired than usual.

When I expanded to Word Press, I broke down and created a 10 point number system. However, it's not just 10/10 or 3/10: there's a descriptor: A rating of 10 is actually "10 - My Precious" and a 3 is actually (don't laugh) "3 - Not My Cup of Tea" and before anyone accuses me of not wanting to say something is bad, 2 is "2 - Below Standard" and 1 is "1 - A Waste of Time & Money".

I use the categories but not the numbers on LJ. The thing I like about number systems, so long as they're accompanied by a brief explanation AND a relatively meaty review, is that it helps me rank how the reader liked one book over another. When it's just a category, or when the review isn't very meaty, it's hard to tell what the reviewer thought, and I always want something quick and easy to look at to give me some idea of what someone thought in the end.

That said, I know when I see a "bad" rating but I'm curious about the book, I read to find out why the reviewer found it bad. If it's legit, like characters don't make sense, or not legit, which is someone ranting and raving about crap that doesn't matter. And when I give ratings, I always try to make sure people know it's how I felt about the book, and I always try to think of who might like the book better than I so I can recommend it to them. :)

At any rate, not sure I'm making sense, but I'm throwing my two cents out there anyway. :)

When I was 11, a bunch of people around me were reading Rebecca by Daphne du Maurier so I picked it up from the library. This taught me two very important things: 1) I do not like the gothic novel, and 2) the difference between what I like and what is good. I could see that Rebecca deserved its place as a classic, despite the fact that I Hated. It. (This also led me later in life to avoid Victorian lit classes like the plague.)

It can be interesting watching what people value in a novel: I tried a few books by a very popular genre author and was utterly perplexed at her popularity. The plots were predictable, the pacing was atrocious, and the dramatis personae were two-dimensional stock characters. I asked a friend about it, and she pointed out that a lot of people loved the books for the world-building, which was excellent. I don't feel the positive differential between this author's world-building and the average makes up for the negative diferential for all those other metrics, but apparantly a lot of people disagree with me. And I'm perfectly happy to have that discussion, but there's no way it could be summed up by a numbered rating system.

Of course, ine of my most humorous experiences on Yelp was when I gave a four-star review (per Yelp's own textual explanation of ratings, this means "Yay! I'm a fan!") to a place that was very new, and noted in the writeup that it verged on three stars BUT they were very new and clearly working on addressing the service issues. I praised the food. I praised the manager. I did not praise some other things, but I identified the specific issues and that I thought they might be resolved in a few months.

I got a NASTY message from someone basically telling me I was wrong to complain about the things I did. ("It's called mood lighting" - okay, I didn't say it was a little dark. I said it was SO DARK I couldn't identify my server when she was two tables away, which really sucked because she was not good at checking on us and I needed to flag her down umpty zillion times during the meal. There's a difference between "mood lighting" and "oops, we should have lights?")

Now, see, I remember when Kennedy was shot (I was in first grade) -- I even remember seeing Oswald getting shot on live tv. I can't separate out the historic Jack Kennedy from the Camelot Kennedy in my thoughts about him because he is forever gilded in my mind by the moment I was told of his death. Now, I know he wasn't Mr. Perfect, and I also have a feeling that he'd have made enough mistakes as president that the world may or may not have been better than it is today -- but I still don't want to read King's alt. history revision. I just. Don't. I used to like his writing, but I can't read it now. I tried "Under the Dome" and bounced so hard. I looked into this big tome and went. Nope. Just not my cuppa. Doesn't mean I don't go "yay!" to folks who do like it. I'm happy for them.

Hmmm. I wonder if I'm just having the alt. history heeebie jeebies about things in my own lifetime? Is unsure.

Thanks for this post. Just... I needed this.

I know it's not anywhere near the point but I just wanted to say that any time I hear/read "that's just how [I/we/whatever] roll" I want to fall to the floor and roll over and over yelling "Well this is how I roll!"

I hope for a future where something like Amazon's recommendation system that actually works will be created.

Where it will be "Based on the 20 items you liked, here's a 21st item for you to consider" as opposed to "Based on the one item you liked, here another 20 for you to consider."

I'm perpetually flummoxed by their choices.
I'll have gotten something perfectly reasonable, and then they will throw the weirdest stuff at me because of it.
I find myself wondering whether it is totally random.
(Like the college roommate questionaries, which I am convinced are all just for LOLs while they go ahead and match people at random.)

Criticism is one of the hardest, most complex skill sets to learn; the more I think I know about it, the more I find out I could have done better.

Rating systems are, by and large, nonsense, I agree. Articulating what works and what doesn't, what works for me and what doesn't, and why - that's the challenge, and that's what I like to see in other critics' work, too.

It's also worth doing right.

This is brilliant, Seanan. I've often thought this, but never written about it, and now I don't have to: I just have to point to your post.

And I don't like bell peppers either. Shoes, though. . .