I'll brake the formula for you =)
A fictional example:
I have 1000 hp and 100 armor.
My opponent has 80 attack damage, 20 armor penetration, and 10% armor penetration.
1. Flat pen is applied first. Your armor goes down by 20. You have 100-20=80 armor left.
2. % pen is applied second. Your armor goes down by 10%80=8. You have 80-8 = 72 armor left.
72 armor increases your effective hp by 72%. This means that your oponent would need to do 1720 damage to kill you. You can find the % from here. 1720* X% = 1000
72 armor provides 42% damage reduction.
He hits you for 80, you take 80*58% = 46,5 damage.
Works exactly the same way for magic resistence/penetration.
If the armor is negative, it is calculated by the same formula, but instead it amplifies the damage (in place of reducing it).