I think I've found an answer after some more digging around. When you are uploading to YouTube, it most likely takes up 100% of your upload speed. I can't say for sure, since you didn't reply to my comment yet, but upload speeds are generally much slower than download speeds.
You would think that uploading shouldn't affect downloading but it does. TCP protocol requires acknowledgement packets to be sent back when downloading. It's difficult to send these ACK packets when your internet's upload is being taken up by the YouTube upload.
Source: http://www.tomshardware.com/forum/43869-42-internet-slow-uploading-something