Finally found the right combination. Change the key mapping from Seil [1]. In the keyboard preferences - Leave modifiers as it is (without changing the default behaviors for Caps Lock). I also installed Karabiner [2] but I am not whether it's really needed.
[1] - https://github.com/tekezo/Seil [2] - https://pqrs.org/osx/karabiner/