Yes, you will be constrained by your upload speed. Your calculations don't take into account packet overhead, for which you will need to add about 10kbit of bandwidth, meaning about 80k per call (depending on a number of things, like number of samples per second, compression) - so a total of 25 calls is probably reasonable.
I do not know much about Teamspeak (but I am familiar with VOIP in general). I believe that Teamspeak is not a codec, rather an encapsulation like SIP. If this is the case, I wonder if the codec you are using is ALAW or ULAW. This is relevant because those codecs provide "phone call quality" voice with very little processing overhead, meaning your server can handle a lot of them. If that is the case, you could look at a Speex based codec which will give you even better quality at a lower bandwidth utilization - but with higher CPU utilization.
This link might also be usefull to you.