When the server is reporting pool full errors, as listed below, it is experiencing a possible memory leak in the block for the listed pool. The top 5 commonly reported memory blocks with this condition is listed below:
Insufficient memory - NSF directory manager pool is full
Insufficient memory - Index pool is full
Insufficient memory - Event pool is full
Insufficient memory - NSF Monitor pool is full
Insufficient memory - Network pool is full
Once the errors are reported on the server console, the Domino administrator should take the following actions:
1. Capture two manual NSDs:
2. Bring down the Domino server
3. Enable the following debug in the server's notes.ini:
CONSOLE_LOG_ENABLED=1
DEBUG_THREADID=1
DEBUG_SHOWLEAKS=1
DEBUG_TRAPLEAKS_SHOWSTACK=1
DEBUG_DUMP_FULL_HANDLE_TABLE=1
DEBUG_SHOW_MEMORY=1
DEBUG_TRAPLEAKS=[block_type]
4. Bring the Domino server back up
5. Set up memory dumps for collection every 4 hours to capture memory allocations
6. Submit the manually executed NSD(s) and console log file from the incident to support.
7. If the issue reoccurs with the debug enabled then perform step 1 and restart server. Afterwards submit the NSD(s), console log and memory dumps from the incident to Lotus support.
Proactive Actions
The Domino administrator can monitor for any of the above error messages reported to the Domino console by using Domino Domain Monitoring (DDM) probes to alert them once they occur so that the listed actions can be performed.
The Domino administrator can also dynamically enable the following through the Domino server console to monitor for the error messages:
set config DEBUG_CONWRITE=pool is full
set config DEBUG_CONWRITE_NOBREAK=1
Additional Information on the listed blocks
When entering the block types for any of the listed blocks in the DEBUG_TRAPLEAKS parameter, you will use the below hexadecimal type IDs (dropping 0x) for the corresponding block.
Example: To set up trapleaks on the NSF directory manager pool block BLK_NSF_DIRMANPOOL, the parameter will look like the following:
DEBUG_TRAPLEAKS=826d
NSF directory manager pool corresponds to memory block: BLK_NSF_DIRMANPOOL (Type: 0x826d)
Index pool corresponds to memory block: BLK_NIF_POOL (Type:0x8311)
Event pool corresponds to memory block: BLK_EVENT_POOL (Type: 0x9506)
NSF Monitor pool corresponds to memory block: BLK_NSF_MONITOR_POOL (Type:0x82df)
Network pool corresponds to memory block: BLK_NETBUFFER (Type:0x8a03)
Debug Parameters in use
DEBUG_TRAPLEAKS prints out information about private memory that is still allocated when a process terminates. It also includes some information about shared memory as well, which is allocated by a process but accessible to all Domino processes, when the last process terminates. Therefore, to get information about a leak, the process (if the leak is on private handles/blocks) or the server (if the leak is on shared handles/blocks) must quit cleanly because trap_leaks compares what was allocated to what is left at program end. If there is a crash, trap_leaks will not be able to report leaks.
By setting DEBUG_CONWRITE, the Domino console checks console message and a stack trace is printed to the console when the specified message is reported. By default, the Domino server halts when the message is reported; therefore, the parameter, DEBUG_CONWRITE_NOBREAK=1, will need to be set to keep the Domino server running normally. The value of DEBUG_CONWRITE will never display in a console log. This is to intentionally stop the server from generating a stack/NSD at the wrong time. You will need to check the notes.ini, NSD or the sysinfo file if you need to confirm the value of this setting.
Related Documents
"Insufficient Memory - NSF Directory Manager Pool is Full" or "The Buffer Pool is Too Full to Bring in Another Page" (Technote #1209395)
Error: "Insufficient memory - index pool is full" results in server hang (Technote #1405762)
Error: 'Warning: Cannot record event - cannot keep up with event occurrence rate!" (Technote #1097225)
Domino server panic with BLK_NETBUFFER memory leaks (Technote #1407257)
BMI Causes a Memory leak on BLK_NSF_POOL (Technote #14008593)
INTERMITTENT MEMORY LEAK ISSUE IN BLOCK BLK_NSF_DIRMANPOOL (APAR #LO47470)
Server Crash with error message PANIC OSBBLOCKADDR: BAD BBLOCK HANDLE (0) Domino Crash when calling function Nifviewregisterrebuild (Technote #1501337)
Server crashed with the panic message " PANIC: OSBBlockAddr: Bad BBlock handle (0)" (Technote #1408586)
About the Author
Shermeker Sanders is an IBM Certified IT Specialist currently working as an Advisory Software Engineer Lead in the IBM Collaborative Solutions (ICS) division on the Domino UNIX Crash & Performance team in Atlanta, GA. Previously she worked on the AIX pSeries Kernel and Crash teams in Dallas, TX.