Does the problem go away temporarily if you stop all your traffic for a while without rebooting? If so, then what you've described sounds like the classic symptoms of buffer bloat, which is increasingly becoming a problem on the Internet, especially with poorly-designed home gateways.
Basically, poorly-designed home gateways often buffer too many packets when the network gets congested, which prevents the TCP/IP stacks of the endpoints of the connection from detecting the congestion, which prevents TCP's congestion-control mechanisms from kicking in, exacerbating the problem.
The fix is to get a better router, or install aftermarket third-party open source firmware distribution like DD-WRT, OpenWRT, Tomato, etc., which may do a better job and may give you access to tuning parameters that will let you fix this for yourself. Products sold as "gaming" routers are often optimized for low-latency, and thus are less likely to hit this problem. They are also more likely to give you access to buffer tuning parameters using factory firmware.
Jim Gettys is the Internet researcher credited with diagnosing the buffer bloat problem bringing it to the attention of the wider Internet protocol research community. He writes about it at BufferBloat.net. His audience is other researchers, so unless you're really getting into this stuff, that site might not be for you.
Iljitsch van Beijnum is a writer for Ars Technica who follows IETF proceedings and Internet protocol stuff closely. Here's an article he published a few months ago on the subject of Buffer Bloat: Understanding bufferbloat and the network buffer arms race
