It depends on the comps. If its something like a Yi, WW, Rengar and you are not defending with some serious AoE CC or super burst, that tower is going to be smashed in the blink of an eye, then they turn around and obliterate your defenders. Suddenly you are all alone, up top probably, and theres a chance not even wards will save you. ACE, then push to nexus for gg.

Conversely as ZehNetoBR said you got a Cho'Gath or an annie, and suddenly that tower might as well be a stationary adc and you guys pick up a kill or two, in which case yeah farming or pushing far away might be a good idea.