You should not be relying on OOM killer to manage your processes. OOM killer is a measure of last resort, when the only other alternative is a system crash. E.g. all cache and disk buffers memory was flushed and committed, everything that can be swapped out/discarded is processed, and you still don't have enough memory... Obviously you don't want your running system to ever reach this state.
Because of the strict constrains that OOM killer operates under (cannot allocate more memory, cannot swap in other processes, etc.), it will kill processes you don't want to be killed to relieve memory pressure.
I think if your system just doesn't have enough memory, you need to add more memory or swap space, depending whether you're running out of physical memory or total virtual memory.
If, on the other hand, you have a few runaway processes that consume too much memory from time to time, due to a memory leak or some other bug, you can control that via other means:
Set
ulimit -mbefore starting the offending process and limit how much memory that process can allocate.Restart the offending process gracefully on a schedule via cron, if there is a memory leak and it is fairly predictable.
In any case, OOM killer is not your friend, it's a loose cannon :-/