I just tested it. As you can see, the golem is debuffed with a Slow effect, I have a Rylai's, and I just hit it with Spearshot (which is on CD now).
If it's just applying a ghost debuff for some idiotic reason, then that's one thing, but I AM applying slow debuffs on use of skills with purely physical damage components. I'm fairly certain at this point that the only physical damage spells which won't work with Rylai's or spell vamp are those that specifically enable on-hit effects (Parrley and Mystic Shot) and that pure-physical damage spells will proc it.
EDIT: Wow, low-res pic. Sorry. Point still stands though.