Yes you can (seen that before?).
You will need first to convert the vdi file into a raw format,
VBoxManage internalcommands converttoraw Ubuntu.vdi Ubuntu.raw
which is the image of a whole drive; then you will need to dd this to the USB stick, for instance in *Nix this would be:
dd if=/path/to/rawfile/Ubuntu.raw of=/dev/sdb
asssuming the USB stick is /dev/sdb
(remember, if you misidentify it, you will wipe out the whole content of device /dev/sdb
, so make sure there are no mistakes). There is also a dd version for Windows, see its sourceforge page.