I just played a game with Maokai against a team that was completely tanky dps.
My team was Maokai, Vayne, Gangplank, Fiddlesticks, Jax.
Their team was Renekton, Nocturne, Cho'Gath, Jax, Skarner.
I went mid against Cho'Gath, which I found was a huge mistake, as Cho'Gath has equal burst, 10 times more sustenance, and a fair bit more health with Feast. I realise that Vayne would have been better against Cho'Gath, but their entire team was just essentially unkillable, yet able to do solid damage.
I mean, their Nocturne was able to survive for 3 seconds under the outer mid turret when he used Paranoia on me (he was on full health, I was on about 300 - managed to W>Q, surviving with Magic Sap, but he STILL made it out and killed me).
Admittedly it was a 4v5, but in this situation, what would I do? Or did we just get outplayed?

Are you serious? Was your maokai ap? cuz if he was then thats the reason why you lost.

My main is Maokai, and i never build ap on him, he is the best pure tank their is. Better than rammus cuz of Maokais passive. Did one of you buy a thornmail? I mean that solves the problem. I bet your enemy bought a thornmail cuz u had vayne and gp in your team.

anyway if fought with a tanky team before, Tank Garen, Tank Maokai, Tank Rammus, Allistar, and chogath. I played as xin, u know what i did? told my team to tower dive. pffft it solved the problem, we were 20-49 and we won that cuz we tower dived. it doesnt hurt to get killed by a turret, luckily my team had 2 fortifys haha. btw xin is a turrets worst nightmare.

If Vayne didn't have a Black Cleaver has her first big item, then that's not good. Her silver bolts scales so well with attack speed that it'd be silly not to get damage, attack speed and armor penetration in one item. BC combined with Starks should give her plenty of armor penetration, damage, and survivability from the lifesteal.

And the guy above me is right, never build ap on Maokai. His ratios are good, but a tanky/cdr build is ever better. It lets you live longer, while being able to spam you cc's more often.

The way to defeat a tanky dps team is to have everyone on your team tank up a little themselves.

Given that tanky dps is especially good at killing squishies, it should be obvious that the squishies on your team can't play like they normally do. Instead, they just need to bulk up and outdps the tanky dps via superior damage scaling.

If it was a 4v5 then there was little you could have done, but in a 5v5 your team should have been able to hold out to the point where most of their tanky dpsers fell off (Renekton in particular, is not nearly as dangerous late game as he is early-mid game).

As to the two previous comments...Mao works best as tanky CDR with a splash of AP. There's no sin in getting an early rod of ages on him at all, provided you grab a glacial shroud afterward and then maybe work on an abyssal on the side. And Vaynes stacking attack speed are useless. Yes, silver bolts is a good passive skill. No, it's not so good that you should base your entire build around it. Unless the enemy team is stacking armor and health like crazy, I suppose.

Well... admittedly I was doing a premade with some lower levelled friends, and the rest of my team refused to buy a Thornmail, or any armor at all, despite how AD-based their team was. I wasn't going to buy a Thornmail, I was stuck in a mid stalemate against Cho'Gath!
So... you're saying that my team just didn't counter them very well?

It sounds like none of you really built to counter the enemy team at all really. When you see the enemy team lineup you need to decide what sort of items you will buy mid-late game to get the most effectiveness. If you're having a hard time with your lane opponent, then for sure you want to make your first items with that in mind. But mid-late game you need to assess the threats. If the threat is 'the enemy team is too tanky to just burst down', then the correct response is either to poke and kite to wear them down (inapplicable with your team comp), or have the squishies build enough survivability to not get squished, meaning you can fight with an advantage after your aoe ults have been thrown out.

Ideally, you'd have fought inside a Mao ult, while Fiddle ults in and Gangplank drops his ult. Then because of the tanky item/s your team has, combined with the Mao ult, none of them die to the otherwise relatively low damage dealers on the enemy team, and can simply beat them down with their superior damage scaling. Some lifesteal/spell vamp wouldn't hurt either.

I'm guessing what actually happened is your Fiddle and Vayne (and maybe the Gangplank depending on his gear) got destroyed due to being too fragile and you didn't have enough dps alive to do anything significant.