Hey, I'm a Jax player. I used to play him frequently, but over the months his nerfs have been numerous, until he reached the point of not-viable (now). I'm wondering whether I should pick Malphite as my new main. Any tips?

You do know Malph is a tank and possibly a DPS/tank hybrid right? Jax is more of a tanky DPS, their playstyle's are totally different so might be a little something to think about.
Regarding tips for Malphite:

Carefully place your ult and wait for the enemy to group up.
1 vs 1 the R>E>W combo works pretty well, then auto-attack, use Q when they start running and E when you are close enough. If your ult is on CD use Q>E>W, auto-attack and then Q again when they run.
You are pretty much an anti DPS-melee who can overpower most of these champs early to mid game. A fed DPS late game will usually kill you 1 vs 1!
Going against AP bursters is a lot harder, building def for this will cost you damage output.
Like the MF Cho build (yes I know) try to scare your opponents early game by eating any enemy melee champ stupid enough to confront you and reap the psychological rewards later in the game

Play rock songs while playing Malphite because you are the ROCK. Thats what sets the mood for being an unstoppable force that Malphite is.

TANK
0/21/9 Smite(if jungling) Exhaust, Ghost.. Notables (Flash, Tele, Heal, Clarity)
Doran shield, health pot, Boots, Sunfire, Aegis, Mercthreads, Frozenheart, Omen, sell Dorans, and replace with either a Thornmail/Sunfire if there are alot of Dps or a GA/B.Veil if there are lots of mage.