As the person above has said, it's mostly because you haven't had a chance to build mid-game survivability. Wriggles is a decent early-mid game item, and gives you the sustain and slight tankiness in lane that helps out a lot. If you go straight for the Bloodthirster, you lack the consistency during the mid-game. Considering mid-game is where early team fights, dragons, and objectives are secured, it is more often safer to build survivability and damage at an equal pace for the first bit.
I believe Graves' item build is something like this:
Early: D.Bladex2/3 or Wriggle's, AS Boots
Mid: Inf Edge/BT, Vamp Scepter*, Zeal
Late: PD, Survivability
Last Whisper depends on runes/masteries and overall tankiness of the other team.