Odyssey Comparative Rating System
Posted: Mon Dec 19, 2011 10:33 pm
I've thought for some time about the issue of providing accurate ratings for Odyssey episodes and the problems that the current systems have.
I think both the five-star system that is commonly used, the percentile system (as seen on AIOWiki), and, and as much as I hate to admit it, my personal rating system (the one you can see in many of my reviews, next to the traditional five-star rating) all leave something to be desired.
The symptom of this problem appears when you realize that very few episodes have truly bad ratings. Even episodes that are almost universally considered to be bad (such as "Fairy Tal-E-Vision" and "Idol Minds"( have come off relatively strong in the percentile ratings, and the others are all rated at 90%. This might not sound like a problem, until you consider that "Karen" (which won the first Odyssey episode competition, and I think is considered a great episode) is rated slightly lower than the first two Mulligan episodes.
The five-star system isn't much better, as just about any episode that's considered to be good or average will usually hit 4 stars ("above average"), and it has to be unthinkably awful (or have a character that someone doesn't like) to get reviewed as a 1.
As far as my rating system goes, well, my ratings are directly based on the rating system that's used in 2K hockey games. The less said about that the better.
The Odyssey episode competitions, I think, are a step in the right direction, but they are very slow-paced, don't permit for many episodes to be ranked at a time, and have a limited set of episode comparisons (plus a limited number of reviewers, which skews the results).
What I want to do is to create like the Kitten War page, or hockey-reference's NHL Elo rating system. By having every episode ever created in one long list, we could determine which is the best by people just going on and click which one they think is better, over and over again until there's a stable list of rankings.
What I'd like to know is what sorts of ratings you think should make up this list, though. Not all episodes are made alike; for instance, some are serious, and some are not. Some are "easy listening" and some are a bit deeper.
I have three questions that I think I would like to ask in each matchup between episodes:
"Which is better written?"
"Which is more entertaining?"
"Which is more applicable to your life?"
I'm not sure these are the best questions to ask, though, to rate episodes, so I'd like to get input on this.
Thanks for your time, and for trudging through my difficult writing style.
I think both the five-star system that is commonly used, the percentile system (as seen on AIOWiki), and, and as much as I hate to admit it, my personal rating system (the one you can see in many of my reviews, next to the traditional five-star rating) all leave something to be desired.
The symptom of this problem appears when you realize that very few episodes have truly bad ratings. Even episodes that are almost universally considered to be bad (such as "Fairy Tal-E-Vision" and "Idol Minds"( have come off relatively strong in the percentile ratings, and the others are all rated at 90%. This might not sound like a problem, until you consider that "Karen" (which won the first Odyssey episode competition, and I think is considered a great episode) is rated slightly lower than the first two Mulligan episodes.
The five-star system isn't much better, as just about any episode that's considered to be good or average will usually hit 4 stars ("above average"), and it has to be unthinkably awful (or have a character that someone doesn't like) to get reviewed as a 1.
As far as my rating system goes, well, my ratings are directly based on the rating system that's used in 2K hockey games. The less said about that the better.
The Odyssey episode competitions, I think, are a step in the right direction, but they are very slow-paced, don't permit for many episodes to be ranked at a time, and have a limited set of episode comparisons (plus a limited number of reviewers, which skews the results).
What I want to do is to create like the Kitten War page, or hockey-reference's NHL Elo rating system. By having every episode ever created in one long list, we could determine which is the best by people just going on and click which one they think is better, over and over again until there's a stable list of rankings.
What I'd like to know is what sorts of ratings you think should make up this list, though. Not all episodes are made alike; for instance, some are serious, and some are not. Some are "easy listening" and some are a bit deeper.
I have three questions that I think I would like to ask in each matchup between episodes:
"Which is better written?"
"Which is more entertaining?"
"Which is more applicable to your life?"
I'm not sure these are the best questions to ask, though, to rate episodes, so I'd like to get input on this.
Thanks for your time, and for trudging through my difficult writing style.