The Rating Game

Like a perennial weed, the subject of reviews and ratings for books has come up for discussion in several fora recently, and I reckon it merits collecting into a post here. Even more timely, as news reports that Amazon is suing over 1000 users of the marketplace site Fiverr over paid fake reviews, something that isn’t new in the book world and many suspect still goes on. Amazon, like most other marketplaces and also non-selling sites like Goodreads, connects their massive database of books with a rating and reviewing system that public users contribute to. At the end of the day, what both readers and writers are looking for is a system that allows fair evaluation of a work in order to help it connect with more readers. Problem is, I’m not sure that’s what we’ve got.

The Problem

What’s wrong with the current system? In a lot of ways, it generally works. Reviews on Amazon or Goodreads have been very helpful over the years in helping me choose books (and other things), and I’ve written a fair few myself. I can’t imagine going back to the pre-internet days when all you had was a cover and a blurb (ok, and the first however many pages you wanted to read) to go on. Yes, we also had our friends, family and traditional media, but these days even my “word-of-mouth” recommendations come from Twitter or Facebook…upon which I promptly check out the reviews!

From the start, however, I’ve had frustrations with one aspect of the reviewing system that most of these sites adopt: the star rating. Star ratings have a long tradition, and are there for a clear purpose: to allow a quick comparison of different work by a simple scoring metric. In the digital, internet age, tens, hundreds or thousands of public ratings are usually combined to give an average score. This then allows you to rank every book in the database on this average and let the best books come out on top, right?* With such a broad sampling of opinions, surely that will give a reasonable, useful answer?

Well, not exactly. For a start, books at the extremes in terms of numbers of reviews can often skew towards the top end. Both the uber-popular bestseller with legions of one-book-a-year fans and the self-pubber with reviews from mom, dad, best friend and the writer will skew towards the 5-star average. Any book with a hint of controversial content will crash down the scale – heck there’s even rating bias against women authors (in Fantasy, from my own research). There’s also confusion between whether the ratings represent how ‘good’ a book is, or how much you liked it – and the meaning of stars differ from site to site. Finally, there are the frustrating reviews which rate the service (“book was never delivered”) or the price (“why is the ebook more than the paperback?”) or something else entirely (“great toaster!”).

What this all boils down to is a rating system that isn’t all that useful but causes all sorts of consternation for readers and authors alike. Written reviews themselves are usually more useful (more later), but are still often overshadowed by the scores. Further, Goodreads allows ratings without a written review, so then you don’t have anything else to go on, which is hardly helpful. (I’m not sure about Amazon, because I know you can as a user rate something without writing a review but I’m not sure where those ratings go. If you do, get in touch!)

And this problem isn’t unique to books. I won’t go into detail here, but it’s interesting that another massive ratings site, IMDb, has a 10-point scale and doesn’t use simple averages in an obvious acknowledgement of the problems here. The same is true for popular travel and restaurant review site TripAdvisor. And even when academics use such ratings systems in peer review, the results are rarely reliable.

What’s the big deal?

There are several reasons why star ratings are important for readers, and even moreso for authors.

First, average scores remain an instant and very visible appraisal of the quality of a book (whatever that means). For a reader, this can mean doubting the worth of reading something with less than 4-star average. For a writer, it can mean crushing rejection of your hopes and dreams (have a look at my book on GR, if you want). Further, when sites let you browse by ranking (like Amazon), visibility depends on a high average rating. It also may impact recommendations, but I haven’t a clue how those work.

Second, and slightly more sinister, is that the complex algorithms running sites like Amazon apparently use these scores in even more complicated ways. Somewhat galling, Amazon counts anything below a 4-star review as “negative” (even though the definition is “It’s ok”) and this can be damaging to an author. Apparently, it used to be worse than it is now, whereas lately I’ve heard that any review helps to get the attention of the Amazon elder gods. When visibility on lists or charts is a huge factor in success, no wonder people are obsession over and even trying to game these systems.

Now you get a situation where readers and authors want different things. Readers still want critical reviews so they can choose how to spend their hard-earned cash (not to mention reading time). Writers and publishers, however, want a deluge of positive reviews upon which to surf to fame and fortune. Potential reviewers are pressured into only leaving positive comments, and the old adage of “if you can’t say anything nice, don’t say anything at all” seems the new golden rule. Authors publicly wail and gnash their teeth at “bad” reviews, knowing that more than their feelings are being hurt. Readers are left to try to sort one apparently amazing book from another, without really knowing what to expect once they get past the sample chapter(s).

The New World Order

With traditional publishing, reviews and ratings are often more helpful because reviewers are more critical. This would seem natural when reviewing a professional product that you’ve paid a decent amount of money for, and when you are often a small voice in a crowd. Yes, the rating scale gets skewed by popularity/controversy (or just gender) as mentioned above, and everything bunches around the top marks. For example, when I collected the ratings for 155 fantasy authors on Goodreads, the average book rating was 3.97 (comparing book to book regardless of number of ratings) and the average rating given was 4.09 (comparing all book ratings at once). Still, you can usually find a few good reviews (i.e. considered, critical ones, not necessarily positive) on which to make a decision.

For indie/self-publishing it can be very different. Some of this is probably readers going a little easier on people they know have done all the work themselves, for whom they feel it will matter a lot more, and often after having paid very little or nothing for the product. That’s natural and understandable, and to be honest I have done it myself. You can pick these ones out, if they have helpful lines like “not perfect, but…” or “pretty good for a free book!”.

Then there are the less trustworthy reviews. Whether these come from family, friends, fanbois or (*gasp*) paid reviewers, they offer almost nothing to the prospective reader and only serve as sacrifices to the Elder Gods of the Algorithm. “Best book ever!” they say. “Stephen King meets Tolkien meets Aristotle!” they shout. “Can’t wait to buy the next one!” they inevitably conclude. And of course the average rating is 4.9 and even then the author ain’t happy.

For the reader, all this just leads to an evaporation of trust. I’ve gotten to the point where I simply ignore the 5-star reviews and dig down to the “negative” ones in search of critical appraisal. Since you almost inevitably find one saying damning things about any number of things, I mostly end up not purchasing the book. So, on one hand it may be helping the all-powerful visibility algorithm, but I can’t see how it’s necessarily translating into massive sales (especially with all the typos and bad grammar in the sample!). Worse, a slew of uncritically glowing reviews tends to provoke a 1-star from someone either suckered into buying the unworthy product, or someone who has rage-read it just to be able to share the truth with the world.

The Point, or How to Get to It

Of course, we still have the old reliable methods of personal/internet recommendations, word of mouth, review sites and blogs. And if you don’t know who to believe, you can always pick a book off a shelf or open the sample chapters and see if you like what you read. But, with so many more books out there these days, the chances are ever increasing that you may just never come across a book that could be exactly what you are looking for – the very idea gives me cold sweats!

Clearly, readers would benefit from a system that more accurately represented the merits of different books. Some authors might cling to their precious 5-star average, but this will eventually devalue as more and more people realize the system is broken. In the long run, I think a more refined and useful system will benefit everyone. Better tools would help readers find what they really want and help books find the right audience. Whether this is a tweak to Amazon’s algorithms, a revamp of Goodreads’ scoring system, or something completely new, I don’t know, but I hope it’s coming. Soon.

* Goodreads, to my knowledge, has no big lists you can rank by average rating. Amazon does.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s