My guess would be that the instance of bash, in which you are executing the xterm command, already resolves the $ANSWER
variable before passing it to xterm.
Either escape the dollar symbol or use single quotes around the expressions. This appears to work in my case.
Update - syntax:
xterm -e "echo 123;read ANSWER;echo answer=\$ANSWER;sleep 5"
or
xterm -e 'echo 123;read ANSWER;echo answer=$ANSWER;sleep 5'
This is equivalent of your syntax if ANSWER is empty before xterm call:
xterm -e "echo 123;read ANSWER;echo answer=;sleep 5"
As you can see the variable is resolved to the value and never enters the xterm. Also the bash running inside your xterm command is completly separated, variables or their values from inside cannot be easily passed outside it's scope.