regsvr32 is a very simple program that simply loads a DLL and calls DllRegisterServer. It's the DLL that's taking the time to register not the regsvr32 process.
If this is a build process under your control on a server under your control you may be able to handle this a little bit smarter. If the list of classes (or other registration data) hasn't changed then you probably don't even need to call regsvr32. As long as you haven't unregistered the data the COM data will remain in the registry pointing to a DLL in a particular location. Just because the DLL may change doesn't mean that any of COM data associated with the DLL has changed.