In GTK, an engine changes the look of the widgets drawn, often to match the native widget set on a particular OS, to control properties such as widget curvature or animations. An engine is software running on your machine.
Themes are a general term for all of the artwork & the color schemes used when rendering everything you see. It's not a GTK-unique term. Two different engines could use the same theme files as input, but the final result would be rendered a bit differently. Themes are a (mostly) static collection of files.