Connection speed is a factor, but not the only one. It all depends on:
- Your ISP (Some can use traffic shaping to make browsing quicker than gaming/downloading)
- Your line quality (as mentioned above)
- Your distance from the server (as mentioned above)
- Your local connection (EG: Faulty wireless might result in high ping)
- Your speed
- How many people are using your connection
There's probably other factors I'm missing, but the above sounds good. Pingtest.net also, to a CA server, to see how you get on.