It uses special versions of games that are adapted for low RAM conditions. Adaptation includes less detailed models, loading textures directly into video memory (that is separated from RAM) and so on. Also, game that is running on playstation is in better position compared to it's PC cousin: it is the only application running on the hardware, so it can better utilize memory by assuming that it don't need to share it with other applications.
PS3 has 256 megabytes of memory for OS and game, and another 256 megabytes of video memory that is used by game graphics. Of course, 256 megabyes of video memory is less than 512mb - 1024mb on PC video cards, but PS3 game is developed for known hardware, so textures can be efficiently repacked. That reduces visual quality slightly but they takes much less space.
Many games has "really high quality" textures only on PC version, so the game that is running on PC with maximum graphics settings will have slightly better image quality compared to same game that is launched on PS3.