I’m not 100% certain what’s going on, but I suspect you need to empty your trash while the DMG is mounted.
OS X stores the trash in a hidden directory in the root of each mounted drive. If you drag files from the mounted DMG to the trash, they won’t actually be deleted; they’ll remain in that hidden directory (inside the DMG) and take up additional space. If you empty your trash, that will delete the hidden directory, and you’ll be able to store more data in the encrypted DMG.