[9/15/2012, 2:50 PM UPDATE] ||| We are going to turn off the Justice Reviews for now while we work on some fixes to common issues. Thanks for your patience and apologies on the inconvenience. We'll also be addressing some of the feedback about the Tribunal being too lenient.
With the new Tribunal features, we've finally unlocked full transparency on both ends of the Tribunal--punished players now receive Reform Cards and reviewers now receive Justice Reviews. I want to take some time to consolidate threads and talk to you about the current status of the Tribunal.
When the Tribunal first launched, many players felt it was this blackbox that was too harsh. Players would often complain on the forums that they were 'banned for no reason.' With the Reform Card launch, players began to see who exactly was being punished and felt the Tribunal was extremely fair. Players began scraping the Reform Cards and finding "Pardoned" Reform Cards from players that actually deserve to be punished. Now with the Justice Reviews, players are seeing more and more that the Tribunal is actually quite lenient.
Just because the Tribunal is lenient does not mean it is ineffective or broken.
The Tribunal is an iterative process--a collaboration between Team PB&J and the community. Now that we have full transparency, it's time for us to work together to determine where the standards should be. Team PB&J has been reviewing the recent 2 days of cases and the community feedback and I'll address some issues here.
1) Reviewers may be biased when they only view the 1st game
a) In science, we know about 'order effects.' These are biases where the order in which you view evidence can greatly skew your decision. We've implemented a solution where game orders are randomized for every reviewer which will reduce the bias of any one game.
2) Tribunal is too lenient!
a) We're going to be reviewing 3 days of data, then determining the first tweaks to the Tribunal that will allow it to more aggressively catch toxic players and punish them appropriately. This should happen next week.
3) EU Tribunals are hindered by multiple major languages
a) We're going to take serious looks at the EU Tribunals and determine some solutions to tackle the issues specific to those Tribunals. For now, every user gets 5 free Skips per day so if cases are in a non-native language, players should "Skip" instead of taking a wild guess (and typically using "Pardon.") Previously, players tended to guess instead of "Skip" because there was at least a 50% chance of IP; however, when we remove IP at the end of September we should see a more liberal use of "Skips."
4) There's some funky stuff going on with Streaks and Accuracy
a) We're be fixing these and are aware of most of the issues
We're going to keep an eye on these things far better now that we've implemented all the feature foundations for the Tribunal. We need players to use the Tribunal more than ever because Team PB&J needs your feedback and needs your data. Together, we actually now have the power to change our own community. In the next few weeks, you should see dramatically better accuracy in the Tribunal and if you don't, tell Team PB&J and we'll work with the community to fix it.
Got questions? Let's do it.