Yes, it's okay to just copy them over: Git for Windows (what you called "msysgit") ships with a build of an OpenSSH client, and since Ubuntu ships OpenSSH as its SSH client as well, the keys require no conversion.
Note that both keys should be placed under the ~/.ssh directory on the Ubuntu machine (note that if this directory does not yet exist, and you're about to create it by hand, it must be owned by your primary user and group and have 0600 access mode). Then the id_rsa file — the private key — must be set to have 0600 access mode. The public key (id_rsa.pub) should use 0644 access mode — that is, write access for you only and read access for everyone.
Note that it's okay to lose id_rsa.pub as it can be regenerated from your private key at any time.