Ran into an issue like this on relatively modern embedded Linux platform. It was a driver bug and it’s been fixed, but here’s the scenario:
- heavy disk access because we were writing real-time images to an SSD at about 500MB/s
- our application was steady-state about 4GB of RAM and we had 32GB available on the platform
- the serial port that we received data from was, under the hood, using DMA
In certain cases, Linux would completely run out of free pages (28GB of it being used for cache on files we were never going to read again). These were all available pages but just occupied at the exact moment. The serial driver would request a page for DMA when it received an interrupt and being inside an interrupt context would request that page with NOBLOCK. That meant that kmalloc would return NULL instead of giving a page, since it would need to evict one of the cache pages before one was available. The serial driver would then blow up and never retry the DMA transaction.
DMA pages are fun. Some devices have special needs for DMA buffers, so maybe you've got something ancient that can only use memory under 32-bit, or maybe you have something really ancient that can only use memory under 16 MB; or maybe the disk controller is fine for regular disk access, but administrative commands need to use a limited range. I didn't really finish debugging that one, I got close enough and said well --- we can just measure SMART status a lot less frequently on boxes with that controller and called it a day. :)
- heavy disk access because we were writing real-time images to an SSD at about 500MB/s
- our application was steady-state about 4GB of RAM and we had 32GB available on the platform
- the serial port that we received data from was, under the hood, using DMA
In certain cases, Linux would completely run out of free pages (28GB of it being used for cache on files we were never going to read again). These were all available pages but just occupied at the exact moment. The serial driver would request a page for DMA when it received an interrupt and being inside an interrupt context would request that page with NOBLOCK. That meant that kmalloc would return NULL instead of giving a page, since it would need to evict one of the cache pages before one was available. The serial driver would then blow up and never retry the DMA transaction.
Fun to debug that one!