Check you mail log. It should show all deliveries and their status. I would expect most failures to be at send time, which should result in a bounce message being sent to the sending address on your server.
Your mail queue may contain a number of messages which are being temporarily refused. This can be for cases such as mail box full, server outages and other situations which can be resolved.
Bounce messages have an empty sender so you may want to filter bounce messages to a process rather than a mailbox. (The empty sender address prevents bounce messages from being bounced.) It can then parse the message to verify it is a bounce message, and record the status of that address. Bounces may be temporary (mailbox full, etc.) so you should processes bounces messages accordingly
Many users disable delivery status notifications, so they are not a reliable indicator that a message was not delivered. The DSN response is a reliable indicator of delivery. DSN responses are sent to the sending address as an email message. I believe that like bounce messages, the envelope sender address is empty to prevent them from generating bounces. If you are processing the messages with a program it will need to process these messages differently than bounce messages.
In the end you should end up with three classes of email addresses:
- addresses known to bounce (bounces may be permanent or temporary);
- addresses known to work (sending DSN messages); and
- addresses of unknown status (neither bouncing nor sending DSN).
You should actively remove addresses that bounce with a permanent status.