There should be no danger in turning on an ATX power supply at zero load.
That being said, the outputs may be out of regulation, hunting (overshooting and undershooting their setpoints), or absent. None of these things are 'dangerous' per the definitions of safety used by regulatory agencies like UL.
What this means is unless you apply some load, you won't be able to determine anything useful by probing it while disconnected from the PC. How much "some load" is will likely vary from model to model, but if you don't want to shell out for a full-blown PSU tester as Kruug suggested, try loading each rail with power resistors to 10% of its label-rated maximum, then test with your multimeter. If the behaviour is still bad, you may indeed have a defective power supply.