Maokai's ult scales in utility directly based upon the number of people (enemy team, and allied) inside of it. In threes it is significantly less useful than in 5s. When you add in the mobility of the champions favored in 3s, it becomes virtually useless. If you're looking for an AP tank I suggest Amumu or Singed, both scale well off of AP and tank items.
Really I have found that the utility of tanks depends upon your team composition. The best meta I've found so far is Teemo + 2 bruisers. Shrooms are better than saplings for replacing wards, and Teemo can carry very well without a full tank, the tank meta is less useful. Also, most tanks build a lot of gp10 items and then camp a lane rather than farming constantly. Given the Shrine buff to farm income, and the short nature of the game, tanks don't get tanky enough to matter unless they are very aggressive (Malphite, Amumu, and Singed seem to be doing best from what I gather on the forums).
If for some reason your team thinks that carries are viable in 3s, then you are looking at an adc in one lane (ezreal/vayne seem most prominent), and some form of AP carry in the other (hope for teemo). Because carries can be engaged upon very easily in the new map (no wards unless you have teemo, lots of bush, poor line of sight outside of lane, no walls to hide behind, etc), they need a very strong front line that can force the enemy team to deal with someone other than the carry. Alistar comes to mind as someone with the cc to do this, but his lack of damage would be extremely detrimental in 3s.
I think your best bets are Amumu, Singed, or Malphite. They all do great burst damage, can peel, are resistant to being bursted down by bruisers, and can force/respond to fights instantly (Maokai requires some strategic planning for best use).