If you're dominating as Swain early game, you should be able to 1v1 pretty much anyone on their team, often staying alive 1v2 or 1v3 until the rest of your team gets there to help. Spirit Visage is a great item on him, as are other defensive AP items (Zhonya's, Rylai's, Abyssal). You don't have to build all of these, but throwing one or two into it based on the enemy team could really help you out. If you're not getting Golem often, I would probably grab a Chalice (oh, and mana runes are a must on him, get scaling mp5 seals); this lets you keep your ultimate up for absurd amounts of time, while still casting your other spells.
Work on your last hitting, if you're harassing well, or even out killing the enemy team, but they're getting more minion kills than you, something is wrong.