You can and will get banned for being a ****** in these games. It ruins the experience for 9/10 players and is reportable/bannable. The problem is determining the legitimacy of the infraction, but rest assured if it is obvious you were being an ass you can get in trouble for it.

Ignore anyone who tells you that being a jerk in custom games is ok. It isn't. The Summoner's Code isn't restricted to "Normal" and "Ranked" game types. Ruining the game for other players is ALWAYS wrong.

I report people who leave. I also report those who troll and don't follow the rules. You just use your better judgement to determine what category to stick them in. I make people agree to follow the rules before I start a game, so that way if they do break them, that means it was completely intentional. Also, only allowing levels 10+, 20+ or some other level restriction prevents a lot of trolls, because I've found most of them are at the lower levels.