As an additional measure to weed out people who don't read tribunal cases before hitting punish, I propose that a few fake tribunal cases are thrown into the lot. The cases first game shown should be something that on the surface looks like an actionable offence, but a closer look reveals it as harmless. Then the rest of the games can be more obvious as pardonable.
Whenever anyone punishes said fake cases, a few things should happen:
-Tribunal voting weight should be severely decreased.
-An automated email is sent out on the next batch to inform the player about the problems of auto-punishing and informing them that repeated misuse of the system could lead to actions taken against their account.
-All cases reviewed by this player that day have this players vote invalidated.
A few notes to increase usefulness: The cases should have randomized elements (names, order of events, chat logs) to prevent quickly recognizing it as a fake case. There should be a large variety of cases.
Once Riot was reasonably sure players were actually reviewing cases, they could even increase IP rewards without worry of farmers coming in without contributing.
For those who complain about the popular opinion being wrong sometimes, this isn't a tool to change honest voters opinions on punishable offenses, this is to weed out those not even trying to review the cases. They could add in less obvious cases if they wanted to test the public's feelings on what is an actionable offense, but that is another can of worms all together.
tldr: Fake obviously-pardonable tribunal cases to weed out trolls/IP farmers.