It doesn't have to be a memory leaking fork bomb - even, for example, a make -j
(or with a too high j
factor) on a moderate code size or any process spawning a pile of descendents (fewer than the reasonable limit for an active user), each chewing an amount of memory significant by itself but too small to be targeted by the OOM killer (or to offer a significant relief when nailed by the OOM killer) can have have a similar effect.
It's possible to write a customized monitoring script/tool (to be executed by root at high priority) which could watch for such process spawning patterns and, if necessary, kill them by pgid or userid (i.e simultaneouly, not one by one like the OOM killer) before they become fatal to the system. Would work for reasonable spawning/resource draining rates, but I'm not sure if possible for just any rates.