GNOME Gmail configuration information is stored in gconf, soon to be dconf. Tab/new window behavior is not a configuration parameter. There is no other configuration file.
You could modify the /usr/bin/gnome-gmail script to open the compose screen as a new window.
Change
webbrowser.open_new_tab( gmailurl )
to
webbrowser.open_new( gmailurl )
At the bottom of the script.