My recommendation would be a desktop machine.
At a previous place I worked, we had a real time 3D graph of commits/breaks into a version control server (This is back in 2009) and we quickly found then, that there was a big difference to the stability of 3D graphics on a desktop system as opposed to a server system.
Desktop systems are designed for end users, and end users are expected to want to do things like gaming and fancy graphics, so the graphics subsystems in a desktop build has more safeguards in place to prevent issues occurring.
Many desktop systems are also not designed to be headless, so a lot of the graphics drivers out there will actively monitor their hardware for crashes and overheating, and reload/recover the driver so the user of the system can see what their doing in order to take remedial action.
Server builds on the other hand, are designed to run long periods, often without any kind of display attached, so software to recover a faulting display driver may not be as important, especially since most windows servers have their graphics output layer tuned for access from a workstation using remote desktop.
Lastly, desktop builds have the graphics subsystem removed out of the kernel code, and while this is also largely true of server builds, there is more risk of a graphics driver crash on a server build causing more problems that require a machine re-boot than there is on a desktop build.
The one problem you might find with a desktop build however is accumulation of crud. Temporary files building up, windows indexing it's search index etc, and yes dare I say it desktop builds do need to be rebooted at least occasionally.
That said, my office PC (A 64 bit windows 7 machine) get's left on pretty much 24/7, and it's in constant heavy use, but I usually get away with manually doing a controlled re-boot about once a month.
If there's no other activity on the system and only your 3D application running, and as long as you turn off non essential services such as windows search, then you should be ok.
Implement a manual policy to reboot the thing once a month to install updates and such like and you should be fine.
As for network performance, well again on my office PC I have a number of IIS7 sites that I use for monitoring my other machines, many of these I access remotely from things like my laptop, and it all works perfectly fine for internal tools that have at most 100 users.