For the most part, anything that scales with attack damage or states that it does physical damage will benefit from armor penetration, as well as any auto attacks. Such skills include Pantheon's Spear Shot and Heartseeker Strike, Ashe's Volley, Caitlyn's Piltover Peacemaker, Gangplank's Parrrley, Nasus's Siphoning Strike, and a few others. The majority of damaging abilities however deal magic damage and are best used with magic penetration.

For Kog'Maw, I believe his Bio-Arcane Barrage does magic damage. For a short while it was actually doing physical damage; I assume they reverted it back, but I could be wrong. If you wish to test, you could just get a friend to 1v1 you with teleport. Take the skill at level 1 and have them run mid with no items. Activate it and fire off a shot, and check the damage. Have them go back and buy some magic resist, and see if the damage done is altered.

Also, these penetration runes apply to items, too. If you have a Bloodrazor, you can improve the damage done by using magic penetration runes, as the 4% per hit it deals is done in magic damage.

Magic pen pierces Magic Resist, Armor pen pierces Armor. What's better usually depends on the champion.
AD is physical, AP is magic. General physical champs want AD and Casters want AP.
Kog can go either way, I think. It mostly depends on your team and the enemy team.

Oh man, Koggy is not a fun example for explanation. I'll go over a few runes in general and see if that helps, and there are champ-specific guides that will help you find runes for specific champions.

First, Attack Damage (AD) is for physical-style moves and auto-attacks. Urgot's Acid Hunter skill uses AD. Ability Power (AP) is applied to magical damage moves, which make up the majority of the moves in LoL. Kog'Maw's Living Artillery uses AP. You can see whether or not a skill is modified by AP or AD by the color of the number by the base number: red is AD, green is AP.

Armor pen: Used for primarily physical champions, or hybrid champions that utilize physical attacks often, such as Kayle. She, Tryndamere, Ashe, or Master Yi would use these.

Magic pen: Used for many casters, especially those with lower AP ratios like Fiddlestix. AP is plentiful on a variety of items, whereas mpen is less available, which is why it's often runed for.

There's the two runes you mentioned, and a little bit about AP and AD.

Well...armor penetration works with physical damage (produced by your right click on Kog and none of his other abilities), and magic penetration works with magic damage (produced by all of Kog's abilities, except his ult does true damage I think?).

Attack damage boosts physical damage and ability power boosts magical damage.

When you use Bio-arcane Barrage on Kog, your autoattack gets upgraded with bonus range and a special effect of dealing % based magic damage to the target. It's still doing the same thing as it always did, it just adds the effect of BaB to the attack.

For Kog, you want to prioritise magic penetration over armor pen, since the bulk of your damage is magic. For your glyphs, seals and quints you might want to look at mana regeneration (for seals/glyphs) or runes that help make you less squishy (health, armor or magic resist).

Try to buy runes that play to the strengths of your champion, or ones that help cover the weaknesses up.

Armor penetration as the name implies, penetrates armor. The more Armor penetraton you have, the less effective is the damage mitigation of items that give armor like Chain Mail. Magic penetration is the same but for items that give magic resist like Negatron Cloak or Null Cape.

Attack damage gives your auto attacks more power, while ability power gives your skills more power, so in Bio Arcane you're using your ability power, same goes for Living Artillery

Kog'Maw fires artillery from a great distance. After a short delay, the artillery falls dealing 80/120/160 (+0.3) magic damage and reveals targets for 4 seconds (deals 150% bonus damage to Champions).

Do you see that 0.3? that means that for every point in ability power you have, your Living Artillery will do an extra 0.3 damage. So at 100 ability power, Living Artillery is gonna deal 30 extra damage.

When you see the (+ number) on a skill it means it ability power will give it more damage unless the skill states that it will deal more damage the more Attack damage you have like with Pantheon's Spear Shot or Ashe's Volley.

if you want to know what the skill needs ap or ad, just look at the color of the number. If its green its ap, if red ad. By number I mean where the added dmg is.

First, sorry for my english.
question: My auto attack deal physical damage and bio-arcane add magical damage. So my target defend my attaque with armor defence, magical defence or both??