In order to detect if there's a floppy disk in the drive, the drive seeks. This is the light and noise you are experiencing. When you navigate from your local disks to your computer, Windows Explorer checks to see if you inserted a disk.
Unfortunately, there's no way to prevent this behaviour short of disabling the floppy drive. Some drives are quieter than others, though I'm not sure many companies still sell drives these days.
If you are willing to disable the drive, you can often do so in your BIOS. This is a lot easier than physically unplugging the drive. Alternatively, you can (or at least, used to be able to) buy external floppy drives. You could leave this unplugged until you needed to use a floppy disk.