I have the same router and had the same problem. The fix for me was to do a site-survey and set my channels as far away from my neighbors as possible.
If you think latency is the cause, ping your appleTV from your desktop (or better yet your router) several hundred times and make sure the ping is consistent. With wireless N it should be always under 30ms or so. Even a consistent 200 ms latency should not interfere with streaming music.
Another thing you may want to check is that another device has a static IP set in your DCHP pool range. This would create an annoying conflict that could cause this type of problem.