I just played a game as Graves and we were getting worked. The opposing Graves made a comment that if I knew how to build a better Graves I could have carried the team.

He may have been referring to the fact that I skipped the Doran's blades and went straight for big items such as Blood Thirster and Phantom Dancers, but he was wearing a Wiggle's Lantern so I am in the dark.

Is it because the armor that Wriggles gives or are the Doran blades all that key?
(or because my damage game isn't up to snuff yet :/ )

Any incite would be appreciated

The problem with skipping Doran's is that unless you're really fed and the enemy has died a bunch of times, you hit a period where you're going to be behind in damage/survivability if the enemy has built them. During that period they can trade damage with you and win, and then use that advantage to snowball and put you farther behind.

Wriggle's isn't bad for AD carries since it gives you AD, armor, and lifesteal, it's up to you whether you prefer that or Doran's Blade x2 or x3.

As the person above has said, it's mostly because you haven't had a chance to build mid-game survivability. Wriggles is a decent early-mid game item, and gives you the sustain and slight tankiness in lane that helps out a lot. If you go straight for the Bloodthirster, you lack the consistency during the mid-game. Considering mid-game is where early team fights, dragons, and objectives are secured, it is more often safer to build survivability and damage at an equal pace for the first bit.

I believe Graves' item build is something like this:
Start: D.Blade/Boots3
Early: D.Bladex2/3 or Wriggle's, AS Boots
Mid: Inf Edge/BT, Vamp Scepter*, Zeal
Late: PD, Survivability

Last Whisper depends on runes/masteries and overall tankiness of the other team.

I usually go 2-3x Doran's Blade then I shift into Infinity Edge. Bloodthirster is good if the laning phase goes on for a while, but it doesn't seem to add half the damage that IE does. If I'm the only one doing well at this point, I'll go for atmog's for stockiness, if the team is doing well, I go for PD.

I play graves a lot, and so far I've had the most success with this combo:
full page crit runes, start with dorans or boots if you farm well. Rush IE and zeal. get vamp scept for sustain. Depending on the other team, Last Whisper is usually better to get than the other expensive items, it provides more damage against most teams.