"How can I have encryption work and have one public key that people can use to communicate with me? Do I export my private key and import it into my mail client on computer B?"
Yes. Like you & DragonLord said. One public key only has one private key, if you want to decrypt your messages on a (any) device, you need the private key on that device too.