xterm's scrollback is limited because it allocates an array once(actually a FIFO) to keep track of the scrolled lines. Doing this once is much faster than periodically reallocating.
You could set the scrollback limit to a very large number (even a million), which is probably long enough for practical purposes. There is no such thing as "unlimited" when the data is stored in memory, in any case.
Before a rewrite in 2009, the practical limit for scrollback was on the order of 10,000 lines due to the way it updated the buffer. Now, it is essentially dependent upon how much memory you choose to use (with the caveat that resizing a window with a million lines of scrollback may be slow).