LustyJuggler works almost exactly as you describe.
Hit <leader>b
to open a list of buffers with the most recent first, type a couple of letters to narrow the list down to the one you want and finally <CR>
to open it.
I personnaly use its sibling LustyExplorer which offers the same feature plus a similar one for the filesystem and a similarly designed "search in buffers" as well.