Its hard to say without some testing. I find it hard to believe its ReadyBoost thats causing the issue. My gut feeling is that there is something wrong with the drive, or the USB controller.
One of the advantages of ReadyBoost is that it uses the SuperFetch to predict what file you are going to read next and write it to the ReadyBoost cache. If the cache is busy or the file is not in the cache, the system reads the file from the drive normally. ReadyBoost can only help your system, even if the increase in performance is so minimal you wouldnt see it.
I would scan the disk for errors using Scandisk. I bet you will find bad sectors.