Arcane Smash - Maokai knocks back nearby enemies with a shockwave, dealing magic damage and slowing them.
Maokai releases an arcane shockwave. Nearby enemies are knocked back and all affected enemies take 70/115/160/205/250 (+0.4) magic damage and are slowed by 20/27/34/41/48% for 2 seconds.

As the title says, maokai's description for his ability and how it works (sometimes) is to push enemies back. I've play maokai non-stop lately in every ranked game and I noticed some issues with it. Sometimes the Q does it's job and actually pushes the person backwards or whatever way your aiming to push. Other times it just does the damage and slows without knocking them back. Even if it's the same champion from the same distance it's just hit/miss on whether it will push or not. I think riot should look at this and possibly fix the bug so it pushes all the time.

There is a specified damage range and it's longer than the knockback range. Notice that your Q also knocks back things BEHIND you and does damage. It's considered more of an attack with a small radius all the way around your body consisting of knockback + damage, and then the extended part, which is the skillshot, consisting of just damage.

Basically: The knockback-aoe has a shorter range than the skillshot distance you see. It's meant to be that way.

If you read the description on Maokai's wiki page:

http://leagueoflegends.wikia.com/wiki/Maokai

It says that Arcane Smash has an aoe knockup diameter of 200, while the range for the actual attack is 600.

Think of the whole attack as a circle with a stick coming out of it. The stick being just damage.

You ever notice that when a person is playing Fiddlesticks and casts 'Fear', the enemy champ tends to move away from Fiddlesticks, but whenever Fiddlesticks-BOT casts 'Fear' 100% of the time the enemy champ moves back towards Fiddlesticks-Bot???

You ever notice that everyone, friendly and enemy champs, move at the same speed on Trundle's 'Pillar of Filth', and the only difference is that the 'Slow Animation' doesn't appear on friendly champs???

You ever notice that Heimerdinger's Turrets target whatever is closest and that the auto-attacks of most champions especially ranged champions like Ashe, Teemo, etc, do the same thing???

I bet Riot couldn't fix these things if they tried.

Could it be possibly because you used Arcane smash at the same time of Riven's Q?

Even if you use it at the same time as her Q it pushes her in the air.