I found the following response while Googling:
Photoshop keeps the memory for future uses to avoid memory fragmentation.
See this thread for explanations by one of its engineers.
Look also at this kb item or search for 'memory and performance" in the help files.