It's good practice to always eject the USB disk properly. In certain (rare) scenarios, data can become corrupted if you just yank it out. One reason for this is that the OS's write cache may not have been flushed completely. See Lifehacker article.
The risk of physical damage is not significant, unless you have a SSD drive.