Killing buffers and quitting your emacs frame are totally independant as you mention : the server keeps everything you don't close open, independently of what you display in how many windows. (I often have the same buffers open on 2 or 3 of my virtual desktops, for example not at the same point.)
To kill a buffer, that's C-x k.