If you do a normal delete it will remain in the recycle bin location. Doing a shift delete will remove it so it doesnt appear in the recycle bin.
However even after a shift delete the file remains physically on the disk until something else overwrites it. This is the premise for data recovery. Even if you blow away the partition scheme the files are still there provided you don't overwrite them, or use a scrubbing tool to wipe the data.
EDIT: I will use NTFS as an example, which is usually what windows uses. In NTFS, all file, directory and metafile data—file name, creation date, access permissions (by the use of access control lists), and size—are stored as metadata in the Master File Table (MFT). So in my example, the file location is stored in the MFT. So /temp/file.txt when cut then pasted to /temp2/file.txt the record for that file is changed in the MFT. The actual file itself is not moved. There is a bit more to it, but thats the general gist of it.