I think you're asking, which is better, AP or Magic Penetration? If it's situational, when should I get one versus the other?
Unfortunately, there is no easy answer to your question. I should develop a calculator (maybe I'll do that this weekend).
Really, you need to know the variables involved and how the mechanics work. Really, what you should be concerned with is this question:
"Given the choice among several possible items I can buy next, which one will increase my damage the most in terms of % damage over what I'm currently doing per gold piece spent?"
I can sum them up the answer into some basic points:
-AP scales in a linear way. Remember that your spells already do a certain amount of damage, so add that to the AP adjustment. For example, if you are Sion and your shield burst is maxed at level 5, you are doing 320 + 1.0 * your AP. So if your AP is 80, your base damage is now 400. If you were to pick up another 40 AP from somewhere, you have added 10% damage. As you can see, the more AP you have, the "less" more AP gives you when expressed as a percentage, because it scales in a linear way. If you had 420 AP, your base damage would be 800. Another 40 AP now only represents a 5% increase in damage.
-Magic Penetration (Sorc Boots) also scales in a linear way, but it is dependent upon factors you do not directly control. It really matters how much armor your target has. Example:
A normal champion has 30 magic resist. Getting the Sorc Boots brings this down to 10. This means you're doing around 18% more damage, regardless of what your base damage was. Now, let's figure in a guy that has 150 magic resist. Getting Sorc Boots brings this down to 130. This means you're doing 15% more damage. So, as you can see, the damage increase has nothing to do with your base damage, and everything to do with the MR of your target.
Let's look at the % decrease now for Void Staff. Once again we'll compare 30 vs 150 MR. At 30 MR, Void Staff knocks this down to 18. This is about a 7% damage increase. (meh). Against the 150 MR, Void Staff makes it 90.... that means now you're doing 31.6% more damage!
TL;DR conclusions:
- As you increase your AP, adding more AP is less and less impressive in terms of increased % damage.
- If your target has low Magic Resist, Sorc Boots are going to be more handy than Void Staff. Conversely, if your target has high Magic Resist, Void Staff is going to allow you to inflict more damage.
- If you are comparing AP vs. Magic Resistance Penetration/Reduction/Whatever, you need to consider both your base damage and your target's MR. If know or can predict both, you can decide which item you should get.
Math:
Base Damage = Ability base damage + AP * (Ability Scaling Ratio)
Effective Magic Resistance = Target MR - 20 for Abyssal * (1 - .4 for Void Staff - .15 for Mastery) - 20 for Sorc Boots, in that order.
Actual Damage = Base Damage * 100 / (100 + Effective Magic Resistance)
Example (compare Zhonya's Ring to Void Staff--you don't have either yet but you do have the 15% mastery, abyssal scepter, and the boots.). Target has 100 MR, you currently have 180 AP:
Base Damage = 320 + 180 * 1.0 = 500
Effective magic Resistance = 100 - 20 * 0.85 - 20 = 48
Actual Damage = 500 * 100 / (148) = 338 Damage
With Zhonya's Ring: AP now 350. New Actual Damage = 453
With Void Staff: AP now 230. New Actual Damage = 474
So it this particular situation, it was better to get the much cheaper Void Staff. But every situation is different.
General Tips:
- If you're trying to decide what boots you should get, the new Sorc Boots are a good choice if you want to maximize damage. It's not a piddly amount, unless your target just has godlike MR (only tanks will).
- Void Staff gives you +50 AP. A typical item like Abyssal or Rod of Ages gives +80 but costs over 3000 gold. So, in a sense, you're almost getting the 40% penetration for free. If at least one enemy has over 100 magic resist, this is worth it.
- If you're trying to figure out if you should get more survivability or more AP, you should know that a pure caster in LoL is almost always better off getting at least some kind of survivability item (Rod of Ages, Frozen Heart, LichBane/Wit's End, etc.) This is because that, after running the numbers, health, armor and MR are generally more powerful % wise during the most critical points in the game, versus more AP. You will find it rare that a truly good team wants you to just max damage (see my section on diminishing returns for +AP).
Hope this helps.