Right-click VBoxWindowsAdditions-amd64.exe, choose properties and set the compatibility to Windows 7. Then "Direct 3D Support (Experimental)" won't be checked and you can proceed without install error.
Source: https://4sysops.com/archives/install-virtualbox-guest-additions-on-windows-10/