magic pen definately.
maokai is an ap tank not an ad tank. 85% of your damage is from spells, maokai's auto attack frames are bad, and his spells interrupt his auto attack...
for dominion i'd use these runes:
magic pen marks
dodge seals
ap/level or magic resist/level glyphs.
not sure about quints. maybe ap? certainly not attack speed though. edit: just read morphio's post and i totally forgot about movement speed. that would be a good idea since it's dominion.
For items... for dominion, who knows, but max cdr is really nice...
In dominion I start with a catalyst because you level so fast... the hp/mana on level up procs crazy fast. That really gives you crazy sustain in bot "lane"... i always go bot with maokai... although if you're not going bot I guess you should take boots so you get to windmill faster.
as for other items... frozen heart is really nice, definately rush that... i mean it's got everything you need, CDR, mana, armor and that awesome slow aura vs annoying auto attackers of which there are lots on dominion.
I always build shurelya's in summoner rift but not sure if it's worth it on dominion as the extra gold from early philo stone is wasted. But still, you'd want to max your CDR quick, maokai is really good with max cdr... maybe grab a kindlegem after the frozen heart and build it into something late game if ever.
after that you'll probably want some magic resist, so negatron cloak --> abyssal scepter?