[9/15/2012, 2:50 PM UPDATE] ||| We are going to turn off the Justice Reviews for now while we work on some fixes to common issues. Thanks for your patience and apologies on the inconvenience. We'll also be addressing some of the feedback about the Tribunal being too lenient.

---

Summoners,

With the new Tribunal features, we've finally unlocked full transparency on both ends of the Tribunal--punished players now receive Reform Cards and reviewers now receive Justice Reviews. I want to take some time to consolidate threads and talk to you about the current status of the Tribunal.

When the Tribunal first launched, many players felt it was this blackbox that was too harsh. Players would often complain on the forums that they were 'banned for no reason.' With the Reform Card launch, players began to see who exactly was being punished and felt the Tribunal was extremely fair. Players began scraping the Reform Cards and finding "Pardoned" Reform Cards from players that actually deserve to be punished. Now with the Justice Reviews, players are seeing more and more that the Tribunal is actually quite lenient.

Just because the Tribunal is lenient does not mean it is ineffective or broken.

The Tribunal is an iterative process--a collaboration between Team PB&J and the community. Now that we have full transparency, it's time for us to work together to determine where the standards should be. Team PB&J has been reviewing the recent 2 days of cases and the community feedback and I'll address some issues here.

1) Reviewers may be biased when they only view the 1st game
a) In science, we know about 'order effects.' These are biases where the order in which you view evidence can greatly skew your decision. We've implemented a solution where game orders are randomized for every reviewer which will reduce the bias of any one game.

2) Tribunal is too lenient!
a) We're going to be reviewing 3 days of data, then determining the first tweaks to the Tribunal that will allow it to more aggressively catch toxic players and punish them appropriately. This should happen next week.

3) EU Tribunals are hindered by multiple major languages
a) We're going to take serious looks at the EU Tribunals and determine some solutions to tackle the issues specific to those Tribunals. For now, every user gets 5 free Skips per day so if cases are in a non-native language, players should "Skip" instead of taking a wild guess (and typically using "Pardon.") Previously, players tended to guess instead of "Skip" because there was at least a 50% chance of IP; however, when we remove IP at the end of September we should see a more liberal use of "Skips."

4) There's some funky stuff going on with Streaks and Accuracy
a) We're be fixing these and are aware of most of the issues

We're going to keep an eye on these things far better now that we've implemented all the feature foundations for the Tribunal. We need players to use the Tribunal more than ever because Team PB&J needs your feedback and needs your data. Together, we actually now have the power to change our own community. In the next few weeks, you should see dramatically better accuracy in the Tribunal and if you don't, tell Team PB&J and we'll work with the community to fix it.

Got questions? Let's do it.

How many names have you written down?

Attached Thumbnails

 

I rescind my question due to the OP update.

I guess I should have expected this.

Everyone was really up in arms about how strict the Tribunal was before reform cards came out. Should have seen that people would say the opposite when people saw how lenient some of the pardoned cases were.

My question is: Do you have any plans to make viewing more than one case a requirement or make more restrictions like requiring you to view/scroll through all of the chat log before making a decision, or would these be considered too harsh? I personally don't think it should come to that, but I want to know if it's at all on the radar.

i think the work that you personally and the rest of your team puts into this gets over shadowed by the misconception of those three topics.. i do not agree with any of them honestly since you have rioters that speak almost every language on earth.. and can understand what is being done and said.. i thank you personally as someone who has had rage issues in the past to help me change how and who i am as a player. and cleaning up our community

I have questions!

I know there are false reports in the tribunal as a way to test the integrity of the judgments being rendered, and I have a few quick questions about that.

1. Are these 'false-positives' placed there by Riot? (The PB&J team)

2. If they are placed there on purpose, what happens if the majority actually happens to punish the case? Do you guys 'protect' the case and therefor take no action?

3. If someone judges one of the 'false-positive' cases as "Pardon", but the majority of reviewers incorrectly judge "Punish", how does that impact the Justice Rating of the person that actually judged it correctly (Pardon)? It seems it would be unfair if the Justice Rating was dinged in these cases since, in fact, they reviewed the case correctly (despite the incorrect majority judgement). And the same holds true in the opposite direction I suppose.

4. How do 'Skips' affect your 'Longest Streak' rating?

Thanks in advance for any Riot clarifications!

Thanks Lyte!

Do you have any plans now that it's widely known (as opposed to only known to people who pay attention) that the people who do the Tribunal are way too lenient? If so, what ideas do you currently have that you plan on discussing?

And the Justice review needs some tweaks on how they update. Apparently, I have 116% accuracy in Tribunal.

Is there any sort of a rough ETA on the leaderboard for the Justice Review, perchance?