Direct Memory Access (DMA) is the missing piece of your puzzle. DMA lets I/O device controllers directly read and write memory without CPU intervention.
Without DMA, a buffer may still be employed as the data needs to live somewhere while it is being retrieved and it may be desired to only give data to the requesting program if the I/O completes successfully.