Start with the debug entries in the notes.ini
CONSOLE_LOG_ENABLED=1
DEBUG_THREADID=1
DEBUG_SHOWLEAKS=1
DEBUG_TRAPLEAKS_SHOWSTACK=1
DEBUG_DUMP_FULL_HANDLE_TABLE=1
DEBUG_SHOW_MEMORY=1
DEBUG_TRAPLEAKS=826d
The DEBUG_TRAPLEAKS=826d entry is for the NSF directory manager and the DEBUG_THREADID=1 may show you what process is triggering the low memory event and you can go from there.
From the OS - you may want to run performance manager and monitor the disk queue length on the disk with the Domino data. If Domino is having to wait on the disk so it can process NSF files this could have an impact on long open handles on a large number of NSF files.
Additional reading: http://www-01.ibm.com/support/docview.wss?uid=swg27036159&aid=1