Your Mac remembers loudness settings for each device individually. This means you can do the following:
- Turn down the volume when you are on loudspeakers to 0 (or mute).
- Plug in your headphones.
- Turn up the volume.
- Plug out the headphones. Your volume will be changed to the loudspeaker levels again (i.e., muted).
These settings should actually be remembered, even when rebooting your Mac.