Can you make someone go below 0? Like a negative number.

i believe you can with MP..

but Armor pen i have no clue.. also small golems already have negative MR on SR.

Only if you can somehow do it with "reduction" effects, such as black cleaver/stark's fervor or abyssal scepter/amumu passive(for MR) etc.

Alright cause I was wondering for Melee type characters.

Corki's Gatling Gun + Black Cleaver.

Armor pen lets you ignore armor but you cannot cause them to have "negative armor"
while Armor reduction lets you do so

same with magic pen and MR reduction

black cleaver