You should try some more constellations. What if you and your brother are both watching the same video or playing the same game? The cause of this may be that the different applications require different QoS. That is, indeed, very useful. Image you download a file, you can wait a bit longer. But if your brother is having a call with a friend on skype and it lags just to complete your download a little bit faster, that would make no sense. That is why different QoS are used. So maybe everything is pretty good the way it is. Or at least as it is intended.
"Fixing" - or better changing - this is eventually difficult. First you should be sure that this is the cause of the issue, so test more as mentioned above. Then maybe install wireshark and check your tcp/ip packets. And if you're sure that is the cause and you're sure you really want to change it, you'll have to change the application QoS settings manually. Have a look at this http://technet.microsoft.com/en-us/library/cc771283.aspx But it can become really nasty.
In my opinion, the best way to solve this problem is to order a better internet connection. if available, of course. otherwise you'll have to ask your brother not to use the connection with a high-priority QoS application when you're playing a game.
Hope this helps you a bit. Generally, there won't be much to fix..