First, you should know that higher class SD cards will not be appropriate if you want to use them for ReadyBoost because higher classes tend to compromise random access times for better sequential reads and writes which are great for digital photography and large file transfer but useless for ReadyBoost. What you are looking for is better random access times and with classification of SD cards by classes you could tell which would be appropriate for your requirements.
Secondly, when it comes to durability of both USB flash drives and SD cards, the chance of failure is higher than of normal use due to limited write cycles and constant activity which keeps device under stress, resulting in higher temperatures. Considering the type of NAND flash memory they have, such devices are not designed to be used as caching devices.
Thirdly, if I had laptop with integrated SD card reader and were to choose between USB flash drive and SD card, I would choose SD card due to advantage of keeping it constantly inside card reader without it sticking out of laptop and breaking if struck by something and freeing me USB slot. Additionally, class rating that SD cards have, helps me choose lower class SD card that would be appropriate for use with ReadyBoost. Provided I never used SD card for ReadyBoost, I would start with Class 4 and capacity of 4GB.
I am not sure whether more than 4GB will bring noticeable results but if you intend to use more, then keep in mind that starting from Windows 7, you can use more than 4GB of capacity. Just make sure your SD card is formatted in NTFS filesystem.
If you still want to buy USB flash drive for use with ReadyBoost, make sure the one you get will be accepted by ReadyBoost.
There's ReadyBoost compatibility chart that you may find useful: http://www.grantgibson.co.uk/misc/readyboost/
You can read about requirements of ReadyBoost here: http://en.wikipedia.org/wiki/ReadyBoost#Requirements
You can also download CrystalDiskMark to check if your drive matches ReadyBoost's requirements: http://crystalmark.info/software/CrystalDiskMark/index-e.html