Well then don't choose those masteries...
The bottom line is, that 4% CDR will help you for the first 20-30mins of the game...
That's usually 75% of the game...
You're complaining that 4% CDR doesn't help you during... What... The last 10mins of the game?
It's a small sacrifice to make, considering how much that 4% CDR helps for the rest of the game, early game is far more important anyway...
You don't HAVE to take that mastery, or that item...
It just happens that taking both has a VERY MILD DRAWBACK...
It's like taking Sunfire Cape, and then complaining that you agro'd a jungle mob while fleeing and died...
SunFire Cape has a VERY MILD DRAWBACK in some situations... It doesn't make it a bad item, it just means you very rarely have to be slightly more careful than usual...