Not to mention point advantage.
What OP is trying to do is ask to not be consistent and adapt to changing environments. However many points you gain is your advantage. It means you have to hold on and be consistent for less time than the enemy.
Screwing up and allowing your enemy to four or even five cap you means you deserved to loose that advantage.
It's not a matter of "Only endgame matters." Its "Only teamwork and adaptability matters." It doesn't matter WHAT stage of the game it is. If you're good at the beginning and your enemy is bad, you can shut them down and have a quick game. It's possible, I've seen and done it and had it done to me.
Every moment of the game matters, from beginning to the end. That's what makes this game so good. Slip up less, learn to anticipate counter builds and learn to counter build yourself. And learn the strengths and weaknesses of your chosen champions and team builds.