Possible scenarios:
If the multiple recipients were all in the TO or CC fields, any Reply All message would be seen by all of them. Any Reply message would only be seen by the original sender.
If some of the recipients were in the BCC field and others were in the TO and CC fields, if someone in the BCC field replied all, it would go to the sender and the TO and CC recipients. Anybody else in the BCC field would not receive the replies.
If all the recipients were in the BCC field originally, Reply All would act the same as Reply because none of the recipients would have the other recipients addresses.
The Answer
In short, the only way people would have seen others responses they were not supposed to would be if someone mistakenly clicked Reply All instead of Reply (which happens WAY too often in my experience as a professional IT person).
Recommendations
As a precaution, if the responses are supposed to be private, the teacher should ALWAYS place ALL the recipients in the BCC field in the original message.
Actually, email best practices dictate that, unless there is a specific reason that recipients should see each other and collaborate in an email conversation, you should default to putting all your recipients in the BCC fields.
UPDATE: Info on BCC and Exceptions
BCC is handled by the original sender's email host. Generally, once the email has left the senders email server or host there is no way for other servers and systems to discover or recover the other addresses on the BCC list.
However, some email software is poorly designed and will leak this information. See the "Security Considerations" section on the BCC Wikipedia page: http://en.wikipedia.org/wiki/Blind_carbon_copy
I was unable to find a list of email host software that handles BCC in this way and thus would leak the BCC list, but the following considerations apply:
Because the headers are still private, even if the sender's email host failed to remove this private header, your own email software would not have a built-in method of displaying these BCC addresses. You'd have to actually manually view the message headers in order to see the BCC list. Email clients don't usually have a field for viewing the BCC recipients in a received message, even if that information were sent.
Because this is actually a security flaw, most mainstream email hosts and server software would prioritize fixing this vulnerability, and so it would not be very common, and it would be even less likely to find it on a host used by a major organization such as a college or university.
Summary
So, theoretically, yes the professor COULD be using insecure email systems that leaked that information, and yet, students COULD be manually finding the other addresses in the BCC, IF that were the case.
However, these are outside the norm and therefore unlikely, and both dependent upon the professor using BCC in the first place, which we haven't yet determined.
The best answer is usually the most likely, and that is that the classmates hit Reply All instead of simply Reply. And the solution is that the professor ought to be more intelligent in their use of email if they need these messages to be private.
Bonus Analogy
Someone programmatically "revealing" hidden messages is not simply unlikely, it is impossible, especially, as you mention, the recipients are on different servers, different hosts, different services altogether.
Think of email like a postal system: Acme Corp sends a mass mailing to 100 people and includes stamped and addressed envelopes inside EACH message for EVERY other person that original message was sent to.
Reply All would mean one of those original recipients photocopied their response 100 times, and took EACH stamped envelope and put them back in the mail with the response. In this case Acme Corp and EVERY other person get the same response.
Reply (not Reply All) would mean a recipient creates their response and puts it ONLY in the envelope addressed back to Acme Corp.
None of the other recipients could possibly receive that message unless they actually hijacked the mail carrier or robbed the postal box.
Email works in more or less the same way: It's not that you can just write some code and now you see all the Replies as if they were Reply All, because those Replies never came by your mail server, or to your "internet stop", as it were.
So, if we assume these aren't super-hackers from Hollywood, the only reason people are seeing reply messages is because Reply All was used mistakenly.