The goal of this series of articles is to provide information on commonly seen Notes and Domino memory errors, along with ways to proactively take action. This may allow you to take action and avoid potential crashes.

Links to detailed information on common memory related errors

• AccessAllprotected
• Max # memory segments / shared or private memory , Out of handles
• JVM – out of memory
• Pool is full – various pools can run into this, a pool of memory is fully utilized
• Insufficient system resources
• LDAP BPool Full

Why do we see Memory related issues in Notes/Domino?

When it comes to memory usage in Notes/Domino, the topic can get complex very quickly. There are many variables which affect Domino's memory usage such as: custom coding/applications/addins, amount of simultaneous db/document access, server load. Simultaneous db/document access is complex in itself and has increased greatly over the years. This could refer to users accessing documents, 3rd party applications accessing dbs/documents (like backup, archiving applications), mass mailings, a huge increase today is due to mobile device access to Notes/Domino. Many mobile devices communicate with servers which probe and access the Domino Server every couple seconds for each user, this behavior drives large amounts of Server load which translates to memory usage. As you can see, there are many variables, and every system can be different.

With technology continually improving, we continue to push limits; especially older 32-bit OS limits, which limit memory usage, overall and per process memory. Mail and attachment sizes get bigger, our social contacts have expanded greatly. Social and mobile multiply loads, sharing with more people, devices and communities.

Of course, Domino also provides a complete application development platform, supporting many languages; for example, Java, Lotuscript, as well as other APIs. Many times we have seen custom applications which have been developed and tested, and function fine, only to start having memory issues as the application is rolled out into the production environment, or as the applications usage scales up to lots of users.

All of the above lead to higher loads and thus higher memory usage. Common preventative actions that should be taken to avoid these issues

• Consider upgrading to 64-bit OS and 64-bit Domino
  
  The main advantage of Domino 64-bit is that it dramatically increases memory address accessibility over Domino 32-bit. Many of the 32-bit limitation have been increased, allowing more memory usage.
  
FAQ: 64-bit version of Domino (Technote #1296452)
Home > IBM Redbooks: Lotus Notes and Domino version 8.5 Deployment Guide > Domino 64 Bit
Steps to upgrade a 32-bit Domino server to 64-bit server on Windows platform (Technote #1506430)
.
• Run Domino Configuration Tuner (DCT) periodically
  
  IBM recommends you run DCT in your environment on a periodic basis. We publish updates to DCT regularly. DCT performs a configuration review of your Domino servers, checking against best practices and rules which IBM have determined important. There are several memory setting related rules which are recommended to increase Domino server stability. Run DCT against your Domino servers, review the recommendations and make the changes to your servers.

Domino Configuration Tuner (DCT)
IBM Redbooks: Optimizing Lotus Domino Administration > 4.2 Domino Configuration Tuner (DCT)
.
• Monitor and Alert for the error conditions
  
  If you are having these issues in your environment, Domino provides the ability to pro-actively monitor and alert for these conditions. In many cases proactive actions can be taken which may avert an outage. Minimally the administrator can collect diagnostic data. Additionally consider a pro-active restart (see below). Memory statistics are also available, and can be monitored.


Troubleshooting IBM Lotus Domino monitoring tools (Section 6, Figure 17 shows how to setup an event where you can alert Administrator(s) if some log message/error.
How to receive mail notification when a statistic exceeds a threshold (Technote #1232603)

• Ensure testing of customer applications, test scaling up number of users and documents
  
  It is very important to test your applications with the expected user loads you and real data similar to what you will see in production. Some of the most common memory-related problems are due to a high use of BLK_OPENED_NOTE memory. BLK_OPENED_NOTE is the memory block the Notes memory manager utilizes to hold body content of documents as they are opened. There are many technotes involving this memory block. Over time, with additions of mobile devices accessing documents and increases in simultaneous access to larger documents, system load and memory use has increased. Custom applications and 3rd party applications can still cause memory usage to be higher than available. Later versions of Domino better handles these conditions. There are times when the task/application causing the problem may need to be adjusted, especially if running in a 32-bit environment. Upgrading to 64-bit Domino is one possible solution.

Managing 64-bit memory in IBM Lotus Domino 8.x
Example BLK_OPENED_NOTE Application issue (Technote #1252299)

• Possible proactive restart
  
  There are times when some memory conditions occur, where the problem is due to a cumulative effect. Restarting the Domino server or a particular task, may temporarily relieve the situation. This would not be normally considered a good solution, but it may avert a worse issue, and give time to troubleshoot and follow-up on the issue. See the Monitor and Alert section above to see how to monitor for a condition. If alerted about a condition, consider collecting diagnostic data and followed by proactive restart.

.

Common troubleshooting methods (to be expanded)

• Collect diagnostic data - ISA-Lite
• Common debug - enabling console log, how to run NSDs, how to create memory dumps
• Start to identify the culprit - If crash, id fatal.. look at top 10, find process crashed
  
  Run the LND (Lotus Notes Diagnostic) tool on your diagnostic data. LND provides a way to quickly analyze your NSDs. If the system has crashed, LND will automatically identify the process and crashing stack. Additionally it will provide information about the crashing process, ie: memory, databases being processed, etc. LND will also search the IBM technote database and provide links to possible related documents.
  Check the following resource for details on how to use LND and where to get it: Tutorial on using the LND tool