I'm experiencing this issue as well. I'd hazard a guess this is a bug with the application itself and has to be filed and solved as such.
I've contacted Slack and they sent me this reponse (emphasis mine):
Sorry to see you have been experiencing this issue with excessive instances of Slack. This issue came to light after we updated to version 2.0, and while the engineers are working on a full resolution for that at the moment, they've put together a support build in the meantime which should fix this up for you. You can access that here: https://slack-ssb-updates.s3.amazonaws.com/support-test/SlackSetup.exe
I've installed the quick fix which minimizes the problem by collapsing all the items in the alt-tab menu down to only 1 lingering notification.
Until then, the only workaround to get rid of all lingering items seems to be to close the application and restart it.
Apart from that we'll just have to wait for a new version.