This sounds like the symptoms of Bufferbloat.
Basically, what happens is that somewhere on your network link, there is a large buffer which is holding your outgoing packets. When a ping packet is sent out, it ends up in this buffer and has to wait a long time before all the packets in front of it can be sent first. The actual network speed is not affected by this, meaning that the rate of packets going through the buffer is the same as normal.
Imagine this like a thin pipe with a fat bulge in the middle. As small amounts of water flow through, no effect is noticed. Only when a large amount of water is flowing, will the bulge fill with water. The rate of water going in and coming out is the same, but water spends more time inside the pipe than normal.
The cure for this is to find out where the problem lies, and reduce the size of this buffer to limit the amount of packets that slow down the ping traffic. This may be easier said than done. If the problem is in your router, then you could try using CeroWRT and adjust the buffer sizes. If the problem is in your ISP, you can adjust your router to send data slightly slower than your available bandwidth, which prevents the buffer from ever being filled.
More advice can be found on this question: How to mitigate BufferBloat on an ADSL connection?
Check your connection with The ICSI Netalyzr (Java required). It will let you know what is going on with your network connection.