That's the problem with playing too passive. You let your lane opponent roam. You need to understand how to respond to a roaming lane opponent, especially if they're successfully ganking.
First off, invest in a ward or two yourself. It may set you back some gold, but if the end result is that your team wins, then what does it matter?
Second, shove the lane. I'm not familiar with Veigar myself, but generally speaking, if your lane opponent is leaving lane to roam and gank, you need to shove your lane under his turret. Last hit what you can, but what's most important is to get the enemy turret to start hitting your minions. If this happens, your lane opponent is losing gold and experience.
And there are different solutions to a roaming laner, so it's somewhat situational. Sometimes the best strategy is straight pressuring turrets (this is typically best in bot lane or as top lane as AD where you can actually dish a lot of damage to the turret). Sometimes the best strategy is to match up with them in the same lane. Sometimes you counter gank somewhere else. Sometimes go steal their buffs. Whatever you do, it's important to respond either by evening the odds where the gank is happening or by pressuring somewhere else.