Really this should not have been possible as the PCIe 8-pin connector was designed to have a completely different arrangement of pin shapes than the existing EPS12V connector. These are shown in the image below (notice the difference in bevelling on the corners):
It appears from the picture you posted that the cable that when you pushed it in, the corners of the square pins got scraped away allowing insertion - the moral is don't force connectors in (I'm not trying to be condescending, but it's worth keeping in mind).
In this situation you have probably been saved by virtue of the card in question having both a PCIe 6-pin and a PCIe 8-pin connector. By having had both the 6-pin connector plugged in at the same time as the EPS12V connector, from the diagram above it is clear that basically all the 12V's and all the GND's are shorted together. While not a very good thing at all, it means that the PSU will not have been able to turn on, nor apply any voltage on the 12V lines.
Essentially this means your graphics card will never have seen any damaging voltages from having the polarity reversed on one of its connectors. So in theory the card should be absolutely fine.
For reference, if the card has just an 8-pin connector, then if you powered it with an EPS12V, it would have been exposed to -12V which would have almost certainly have caused it to fry - and would probably also have taken the motherboard out along with it!
As a side note, it would actually be possible to easily plug an 6+2-pin PCIe connector (these are the ones that can be separated or combined to be either 6 or 8 pin) into an EPS12V socket - in fact I did this once, but fortunately realised before turning it on. The true 8-Pin connectors (the ones which cannot be split off to go into a 6-pin connector) have a flat bar which joins across two of the pins to prevent doing this.