There are four five possible solutions:
Truncate the files to a length within UDF1.02 standard, e.g. use a script to automate this, do it intelligently, i.e. append an incrementing an index number to the file if the same filename exists from a previous truncation. Remembering to ensure that appending the index number still keeps the overall file length within the standard. This is a possible solution but I would have to be aware of files that reference other files e.g. html, the references within files would be broken if the filename is truncated.
Use a later UDF standard. Not favourite as UDF1.02 is most broadly compatible, lowest common denominator
There is a beyond compare thread that discusses use of regex within the tool but the thread has not been concluded to provide a solution that definitely works yet: http://www.scootersoftware.com/vbulletin/showthread.php?p=29639
Use MindGems Fast Duplicate File Finder to compare 2 folders and set the config so that one of the folders is excluded from having its duplicates automarked, this means that the other folder can have its duplicates marked (and removed) leaving just the files not in the first folder: http://www.mindgems.com/products/Fast-Duplicate-File-Finder/Fast-Duplicate-File-Finder-FAQ.htm#EXCLUDE_FOLDER_FROM_AUTOMARK
EDIT - new: Turn off unicode in UDF if only ascii needed. This doubles the available characters so truncation situations will be reduced or eliminated. unicode off is still a valid option for the UDF standard, including UDF1.02, apparently: http://forum.imgburn.com/index.php?showtopic=17822&st=0&p=131270&#entry131270