I guess one of the primary reason is the loss of resources.
Without a jungler all the free gold and xp in your jungle goes to waste. If top is tanky enough to stay in lane then that means the two top champs will be under-leved.
But I think the real reason people get angry is because no one taking jungle is a sign that players are not working as a team, ie everyone is so set on having their lane and only their lane that they won't sacrifice that for the team.
BTW are these ranked games? In normal games people should really not be that uptight imo, its just practice really.