This article will describe areas in an NSD log to gather information on memory related crash on HTTP process.
Memory crash Memory exhaustion defined by “Insufficient memory” messages
Memory Handles defined by “Out of Handles” messages
java.lang.OutOfMemory errors
NSD log contains memory summary for Domino processes. Start by reviewing the Process Memory Mappings section for HTTP process. For NSDs generated from Unix OS, use “Process Heap” size under “Process Heap Memory Stats” instead.
In the example below, Process Memory Mapping shows that HTTP private memory is over 1G in size:
HTTP overall memory is comprised of the following areas:
1. Private Static-DPools
2. MBlocks
3. Thread stack Space
4. JVM
5. LotusScript
Next step is to identify which of these areas is making up majority of HTTP overall memory.
Private Static-DPools
Memory allocations performed by Notes using various DPools (4 MB to 8 MB). These pools are not freed back to the OS after being used.
NSD or Memcheck log look at “Process Heap Memory Stats”
DPools are found within an NSD or Memcheck log under “Process Heap Memory Stats” section:
Check Top 10 section for HTTP in NSD log to identify any memory blocks that are large in size or have high handle count.
DPool debug for HTTP process will generate call stacks in Console log when DPool is allocated. Note, DPool debug can also be used for Shared memory which does not apply in this case.
Example of DPool debug for HTTP on Windows OS uses 'nhttp' as value. In this example, debug will track DPool allocations for memory block 0x044
debug_private_pools=1
debug_private_pools_process_name=nhttp
debug_private_pools_blocktype=044f
Memcheck logs can be collected to track memory usage and identify time frame memory spike occurs or correlate high memory usage with activity on server. The memcheck logs can be automated by setting up Program document to run "-memcheck -perf" over period of time.
MBlocks
MBlocks are memory blocks allocated directly from the OS and released back to the OS when freed by the caller.
From NSD log search for “Directly Allocated Memory” under memcheck for HTTP process.
If Directly Allocated Memory is showing high memory usage with blk_local or blk_local_block, add Notes.ini parameter inotesuseNotesMemory=2 as this will help manage direct memory allocation more efficiently. In some cases when using this parameter, Admins may notice high handle count reported. It is then recommended to change setting to 0.
| Notes ini Parameter | Memory allocation |
|---|
| inotesusenotesmemory=0 | Memory is allocated directly from the OS |
| inotesusenotesmemory=1 (default) | Memory is allocated through BLK_LOCAL or BLK_LOCAL_BLOCK |
| inotesusenotesmemory=2 | Memory is allocated through BLK_HTTPINPROCESSMEM |
Thread stack Space
Generally, each HTTP thread uses about 1 MB of memory per thread. Check NSD stat section for number of HTTP worker threads configured on Domino server:
Using more than 40 worker threads for HTTP may not be necessary and taking up overhead memory that can be used for processing requests.
Domino HTTP Active Thread recommendations
Setting "Optimize HTTP performance based on the following primary activity:" can also impact number of HTTP worker threads configured. Check that this is set to => "Advanced (Custom Settings)" in Server document – Internet Protocols tab – HTTP tab.
For Lotus Notes Traveler server the number of worker threads are going to be higher.
Tuning_http_threads_for_Lotus_Notes_Traveler
JVM
java.lang.OutOfMemoryError messages can be reported in Console when JVM memory has been exhausted.
Factors causing high memory usage in JVM could be:
Java heap size
Servlets
Java agents
Java heap size for JVM used by HTTP process can be increased through Notes.ini parameter JavaMaxHeapSize or HTTPJVMMaxHeapSize new in Domino 8.5.1. HTTP memory will be impacted in cases when Java heap size has been increased.
NSD log has Notes.ini section to check current settings for HTTP JVM.
Default for 32-bit of Domino:
HTTPJVMMaxHeapSizeSet=1
HTTPJVMMaxHeapSize=64M
Note: Domino (32-bit) on iSeries uses 1024M as default for HTTPJVMMaxHeapSize
Default for 64-bit of Domino:
HTTPJVMMaxHeapSizeSet=1
HTTPJVMMaxHeapSize=1024M
HTTPJVMMaxHeapSizeSet=1 new in Domino 8.5.2 maintains HTTPJVMMaxHeapSize value.
Troubleshooting Java agents, Admins can implement Java agent debug:
What is the significance of using the notes.ini parameter, debugshowjavaagenttid=1
Memcheck logs can be generated over period of time to track memory allocations and correlate with Java agent debug.
LotusScript
LotusScript memory currently found in a memory.dmp file. The “HTTP” process section has “LotusScript Memory Usage” section
Memory dumps can be generated over period of time to track memory allocations and correlate with agent debug.
How to automate the collection of memory dumps