I assume you are using the latest version of Google Chrome.
In the bottom right of the task bar, you will find the audio icon, right click on that icon and select 'Open Volume Mixer'
Youtube will be shown as 'Adobe Flash Player'
and Google Hangouts will be listed there as 'Hangouts'
You can change the volume of either with the shown volume selectors.