While most computers nowadays have USB3, you will really notice a speed differenc if you're limited to USB2 speeds. Boot and load times are primarily the affected areas, so if you can live with that, this won't really be an issue.
There is also the generally increased wear and tear that comes with a portable device, so take the posibility of data theft and loss into consideration when making your choice.
Outside of that, I would have a hard time living with only 64Gb nowadays (although I used to make due with 43Mb on my first PC), but scaling it up to a bigger drive when available shouldn't be a problem.
In short: The downsides aren't that big, especially if you're going to use it as a secondary OS, or just to try it out. I'm sure you already know this, but many linux distros are made to be compact and run this way, so if the limitations don't bother you, then I'm sure you'll like it.