If I have high armor penetration, say 30, and attack someone with low armor, say 10, what would happen?

Would my attacks:
1. Count as though the enemy had 0 armor, dealing full damage.
or
2. Count as though the enemy had -20 armor, dealing extra damage as a percent (negative armor)
or
3. Count as though the enemy had taken extra flat 20 damage, as if my attack was 20 damage higher.

I'm trying to compare armor penetration vrs. armor reduction.

2. Count as though the enemy had -20 armor, dealing extra damage as a percent (negative armor)

The _reduced_ damage is calculated by Armor/(100+Armor). If it is -(minus) it is amplified damage.
So -20/80 = 25% amplified damage.

And what happens when your ArP = their armor + 100...or even close to that? There's more to the equation then that or else some servers will be crashing.

ok, so option 3 is slashed out like I thought

But I still dont know if it's 1 or 2, requiescat brings a good point, the point I thought, but it still doesn't seem confirmed

Debuffs can take armor below 0. For example, black cleaver.

Armor Penetration cannot take armor below 0. If someone's Armor is already equal or less than zero, Armor Pen is ignored. So if you already have someone sundered with black cleaver stacks, that Brutalizer is not doing any additional good.

Negative armor increases damage taken by a flat percent. -20 Armor means you take +20% damage. -100 Armor means you take +100% damage.

Er, I was under the impression that negative values worked just like positive ones -- that is, -50 armor would be +33% damage, while 50 armor is -33% damage