Considering the drive itself contains a magnet that I would expect to be far stronger than these are, I would doubt there to be any issues from them coming into contact. It's largely a myth that hard drives are easily wiped by a magnet (VHS, tapes and floppy media are far more likely to). In fact, a test was done here involving huge neodymium magnets, far more powerful than most magnets you will have, which caused zero data loss at all (more of a chance of mechanically damaging the drive due to the power of them if they'd been closer to the read heads).
I wouldn't state it's impossible for some form of damage to occur, but it's very unlikely. Don't worry too much about it, but keep them separate if you can.