If you're erasing all the data on the drive anyway, (this is one of the things a format does, after all) the validity of any file handles open on it is fairly moot. The only thing to be concerned about at this point is if you were backing up the data before the format, and there are still processes working on that. To be safe, close all other programs before retrying and double-check to be sure that any backups you had going have fully completed. If it still gives issue about open file handles, try a reboot.
If that doesn't work, try rebooting in safe mode. If that doesn't work, pull out your toolkit of choice (e.g.: SysInternals, NirSoft, etc.) and start troubleshooting to find out what keeps opening handles on that drive so you can kill it. Or just reformat the whole machine, because at this point there's definitely something extra-special going on.