I'm 100% positive custom games don't count towards the Leaver Penalty. A few things that may be happening here: if you spend an excessive amount of time idle (afk/disconnected/asleep on the keyboard/etc), the exact amount not having been released due to reasons mentioned in my previous post, you'll eventually get hit by the 'leaver buster' system; this is to prevent players from just afking games while alt tabbed, and to punish players who can't manage their time effectively ('but I had dinner!' isn't a valid excuse, know when you're going to eat, or don't eat until after the game). Another possibility is Co-op games; a lot of people don't realize, but they _are_ watched by the leaver buster system, as it's a matched game and one you don't have any say in your teammates (if you're solo queueing).
I believe they removed it, but the 'warning' for leaving custom games was just a product of how 'quitting' a game worked; there wasn't any actual punishment though.
Your leaver level will be lowered after playing games without leaving, though again, the exact mechanics haven't been released.