I figured out how to get it to work.
If you have the RDP-session windowed instead of fullscreen, the keys work the same as in the Citrix session, and in OSX:
shift+alt+8 or 9 --> {} alt+8 or 9 --> [] alt+7 --> |
I edited the rdp file so that the resolution is as close to fullscreen as possible:
desktopwidth:i:1920 desktopheight:i:1150