The monitor may not be able to handle that high a resolution and goes out of synch. Do you hear any high-pitched whistling and clicking as it tries to synch up to the signal? That's usually a sign that the monitor doesn't work at that resolution, and why it can only display a much lower resolution.
You don't say the vintage of the display, but my hunch it probably can't go much more than 1024 x 768 which was high-resolution way back when except on expensive CAD/CAM and graphics systems which used special video cards and monitors capable of the resolutions we use every day now.