Largely taken from Wikipedia - Memory hierarchy and my own experience:
It is inevitable that, the lower down the memory hierarchy you go, the slower the data transfer rates will be. So "matching" the data transfer rates will not be possible. However, for optimal performance, you need to make sure that data which must change or be accessed more frequently is kept in the "higher layers" of the pyramid. For instance, if you access a piece of data 12 times per second, you definitely don't want that data to reside at the bottom of the pyramid, in the tape backup level! You would at least want it to be in the hard drive level, although personally I think it would belong in RAM.
An optimal performance design (of software, services, systems, etc) is designed to take the limitations and realities of the memory hierarchy into account. In other words, if the software you run and the requirements of the data you're transferring are designed in such a way so that the largest and least-intensively-used data is stored in lower stages of the memory hierarchy, and the smallest and most-frequently-used data is stored in the highest stages of the memory hierarchy, you will experience "good performance".
This has many possible ways of implementation:
Programs that access lots of data very frequently, such as graphics device drivers, need to constantly read and write pixel data. This pixel data is displayed on the screen at least 60 times per second. For such frequent use, this data is almost always residing either at the RAM level or the processor cache level, which is very fast but the amount of storage is very limited.
Programs that let you play with large amounts of data that comes and goes, such as word processors, need to somewhat frequently store moderate amounts of data. For example, as you type up a Word document or a post in a Web browser, immediately storing every character to the hard disk or tape would be very slow, so instead they are stored in RAM until you save the file. There are many programs and operating system subsystems that "cache" data in RAM, which means they temporarily store it in RAM or in the processor cache for short periods until you decide to either throw the data away, or save it permanently to disk.
Users who store large files, such as music and video collections, need to manage that data wisely. When you are archiving your data for long term storage, you might back it up to DVDs, BluRay discs, or even to magnetic tape -- but these media are not ideal for fast retrieval due to the need to physically set up tapes and discs, and due to the relatively slow read and write performance (burning a disc takes several minutes). So the ideal situation is that you will store most of the media that you need/want to access most frequently on your hard drive. It is then temporarily copied into memory when it is decoded and played back in real time. So as you go from having the media "around" (archived on DVD), to "immediately accessible" (on hard disk), to "currently playing" (in RAM), it moves up the pyramid.