How does Windows deal with process memory or resource exhaustion?
How a program behaves under "stress conditions" depends on how the particular program has been written (what privileges it has, does it contain bugs, etc) and this behaviour will determine how the rest of the system is affected when something "goes wrong".
It is possible that data can be lost. It is possible for the system to crash with a BSOD or just be unable to write data because of a delayed write failed error.
Pushing the Limits of Windows
The blog post series by Mark Russinovich (starting at Pushing the Limits of Windows: Physical Memory) goes into this topic in some detail.
In particular, in Pushing the Limits of Windows: Paged and Nonpaged Pool, he illustrates the effect of running a test program:
If you want to witness first hand how a system behaves when pool runs low, use the Notmyfault tool. It has options that cause it to leak either nonpaged or paged pool in the increment that you specify. You can change the leak size while it’s leaking if you want to change the rate of the leak and Notmyfault frees all the leaked memory when you exit it
On one test system, he said about the results:
On one test system, I eventually saw this error message indicating that data had potentially been lost. I hope you never see this dialog on a real system!
Further Reading
Windows Internals, 6th edition by Mark Russinovich contains a lot more information on how Windows works under the hood.