You can Safely Remove after pausing the USB data transfer so there is no chance of corrupting your device partition or your data.
Edit: Sorry I forgot to mention I disconnected the device and reconnected it and resumed the copy process without any problem.
Update: After starting the transfer I paused the copy process, ejected the USB drive, inserted another USB drive with different files, disconnected that drive as well, reconnected the first one again and finally restarted the copy process and it got resumed without any problem.
As you asked in the comments about how the data transfer might be affected if the drive letter is changed, in that case when you try to resume the process Windows prompts you that the source file is missing and displays the following three options:
Try again
Skip
Cancel
If you press the Try again
button it restarts the process from the beginning.
I have only USB 2.0 ports so I can't tell you about any change when USB 3.0 ports are used.