This is rather inconsistent, other times the program is killed due to running out of memory just fine or hits some assertion in memory manager.

Here's a demo:

The taskbar and system server are killed. For some reason, the browser doesn't disappear and isn't killed. Which processes are killed is really inconsistent.

From this state, you'll notice the taskbar doesn't restart and the resource graphs stop ticking along. Other times when the taskbar isn't killed, you won't be able to start anything from quick launch. In other runs, I tried opening a program from another terminal and it doesn't work. When allocate finishes, the terminal will lock up.

Then, going to hover over themes completely locks the system, with zero debug output.

I ran this in VirtualBox to try and get some more data. A bunch of the EIPs I got was from servicing interrupts. It seems to be spinning in the kernel.

One particular EIP I got stood out: c0163ef1. This is in Thread::SelectBlocker::should_unblock, which is only called by Scheduler::consider_unblock. So seemingly some thread is blocked in this state. (This may not match up for you since I have a slightly modified CPU.h and CPU.cpp)

Here's an example of where programs stop responding: