I'd like to emphasize the fact that VNC is simply a horrible comparison point.
Between Linux systems, a properly configured xpra provides much, much better performance - mostly thanks to supporting H.264 based lossy compression.
The only lossy compression supported by TeamViewer seems to be reducing the colour depth and some kind of interlaced updates which help little with animated content as you noted. A good system would be able to smoothly sacrifice quality to keep a reasonable rate of updates. The mentioned xpra comes somewhat close to it, but honestly it still does a crappy job.
Another issue is TCP vs. UDP based. TCP based protocols are quite vulnerable to for example high packet loss, and the fact that once you sent some data, it has to be transmitted - thus a single large update means your screen will hang until it arrived. Particularly on slow connections that makes VNC unusable since any big change on the screen means you'll not see any changes until minutes later. Compare also ssh vs. mosh for pure text.
It seems nobody has managed to invent a really good protocol yet though, one that combines high compression ratios of advanced video codecs, use of UDP with a good way to deal with lost packets, perceptive coding that detects and favours readability of text over "useless" details, good automated balancing of update/transmission speeds with quality and encode speeds, gradual and smooth quality refinement over time for non-updated areas, client side caching and differencing (while video coding can reference past frames they are not really quite a good match to normal desktop content, and the fact that you could easily store 100s of MB of reference data client side). In other words: All that TeamViewer seems to do is to not do completely stupid things (though VNC has the excuse of mostly showing its age), I know of no solution that implements even a fraction of clever tricks possible even today, not to mention all those that one could imagine.