The error message "Insufficient system resources exist to complete the requested service" is reported to Domino by the Operating System and it indicates that an Operating System resource is nearly or fully exhausted. The most common resource that message refers to on a machine running a Domino Server is the Paged Pool (a section of kernel memory). The error is often preceded by the phrase "Hardware/OS error." This problem is occasionally, but not always, reported in the Windows Event Viewer Logs.
The Operating System allocates paged pool resources for multiple reasons, but on a Domino server with many databases, a large percentage of the paged pool will be allocated for open databases and files on the machine. The reason for this is because for every 1 MB of file open on the OS it consumes 1 KB of paged pool memory. The maximum size of the paged pool on Windows 2003 32-bit can vary from roughly 250 MB – 525 MB depending on certain settings on the OS (refer to the table below for more details). Given the way the paged pool is used, it is important to note that this problem is caused by the total size of the currently accessed files on the OS and is not related to the number of files accessed. So after several hundred GB of files are opened on the machine the OS must clear space from the paged pool by closing files.
By default, the OS will try to trim the paged pool whenever 80% or more of it is consumed. If Windows fails to trim the paged pool, and it becomes exhausted. widespread database corruption typically occurs on the server as well as issues with Domino server performance, hangs and/or crashes.
None of the following changes can guarantee that we will prevent the OS from hitting the paged pool limit. These are only attempts to get around the problem. In some cases the OS cannot keep the paged pool usage low enough to prevent the problem.
Domino memory dumps will not show paged pool usage as this has nothing to do with Domino memory usage. There is one section of NSD that includes some OS/Kernel memory breakdowns that might, in some cases, provide information about the paged pool, but it is not always useful. Otherwise, NSD does not provide any details about the OS paged pool and its usage.
To start, the following documents provide important background information on Pool Resources:
This is an article from the Microsoft TechNet that goes into detail about pool resources to provide a better understanding of the paged and nonpaged pools. It also provides information about a Microsoft tool that can be used to confirm the maximum paged pool usage on a server.
This technote explains that for every 1 MB of file opened on the OS it consumes 1 KB of paged pool memory. It also references two registry settings that can be used to alter paged pool usage on the server. One registry change (
PoolUsageMaximum) will cause the OS to trim the paged pool more rapidly and the other change (
PagedPoolSize) will raise the maximum size of the paged pool. It mentions a workaround where you disable the DBCACHE on the Domino Server entirely. This causes Domino to stop buffering a file after the users are no longer active in the database. This method is rarely recommended since it can cause a fairly significant performance impact on the server because every new DB access will cause disk access instead of the access of the DB in cache, like it did before. This is also just a workaround because the act of opening the DB still uses paged pool resources when it is accessed and will only clear them out of the paged pool more quickly since they won't be cached for use later. This just makes it so the DB is not cached over time and the paged pool resources will be returned when the database is closed. Some spikes of accessing many DBs over a short period of time could still cause paged pool spikes. For more information, here is a Microsoft Knowledge Base article,
Server is unable to allocate memory from the system paged pool,
references the
PoolUsageMaximum and
PagedPoolSize registry keys and how they alter the paged pool on the OS.
These two registry keys are vital in any attempt to resolve the problem with "Insufficient system resources" errors.
These are the values you can expect to be the maximum for your Page Pool. These numbers may vary depending on your environment and your own testing:
Tests were done on machines running 32-bit Windows 2003 Standard Edition SP2 with 2 GB RAM and 4 GB of RAM
/3GB Set | /PAE Set | PagedPoolSize Key Value | Observed Paged Pool Maximum (KB) | Observed Paged Pool Maximum (MB) |
Yes | No | 0 | 258,048 | 252 |
Yes | Yes | 0 | 260,096 | 254 |
Yes | No | FFFFFFFF | 262,144 | 256 |
Yes | Yes | FFFFFFFF | 264,192 | 258 |
No | No | 0 | 360,448 | 352 |
No | Yes | 0 | 362,496 | 354 |
No | No | FFFFFFFF | 536,576 | 524 |
No | Yes | FFFFFFFF | 536,576 | 524 |
Notes:
- Lotus Domino support for the Windows /3GB switch (Technote #1257005) discusses the /3GB and /PAE switches in detail. These are both boot.ini switches available on some versions of 32-bit Windows. The /3GB switch effects the virtual user address space of each process and the /PAE switch effects the amount of RAM that can be managed by the OS. Lotus recommends against using the /3GB switch, and the table above shows the negative effect it has on the expected maximum value for the paged pool on Windows 2003 32-bit.
You can do some additional things outside of the registry to try to combat the overall amount of data that is accessed/cached over a period of time. Here is information on several helpful Notes.ini parameters which can be changed and monitored:
- Along with lowering the UBM you can change the default dbcache maxentries value with an INI parameter NSF_DbCache_Maxentries. This parameter can alter the number of databases allowed in the dbcache at one time and could lessen the paged pool usage. This setting does not have an "across the board" recommendation. The size of the DBs is not taken into account, so you could have a small number of very large DBs cached at one time leading to this problem. It is important to note that the dbcache is allowed to grow to 1.5 times the value for NSF_DbCache_Maxentries. For example, if you set NSF_DbCache_Maxentries=1000 then the dbcache can store up to 1500 databases. This is something that you can monitor on the server to decide a comfortable number in relation to common DB sizes. Here is a document that discusses this in more detail as well: How to set the number of databases cached simultaneously in Domino
- On Domino 7.x and 8.x, the maximum DBCACHE hold time is much higher. DBs can be stored in the cache for up to 10,000 minutes (about 6.6 days) after they are closed. This can be significantly shorter depending on whether the max entries is hit and DBs need to be trimmed from the cache to make room for new entries. This hold time was increased to allow for much quicker access to DBs on the server even if they hadn't been opened recently. The maximum hold time can be controlled in Domino 7.x and above by adding the notes.ini parameter NSF_DBCACHE_CLEAN_HOLD_TIME=n (where n is a value in minutes). A value of 0 causes the default (10,000) value to be in effect. A value of -1 causes the cache behavior to revert to Domino 6.x behavior. A very small value may cause undesirable performance issues. The suggested setting is -1 so that these servers use the R6 method of trimming DBs. In Domino 6.x, the database remains in the cache until it is opened again or for about 15 to 20 minutes (generally). For more information refer to the technote: Database cache hold time changes in Domino 7.x and above
- You can also use the Server_MaxSessions and Server_Session_Timeout parameters to make sure sessions are not staying open for much longer than necessary on the server and also to trim older idle sessions if Domino needs to. These also do not have an across the board recommendation. Generally, 30 minutes is enough time that it does not cause too many new server connections and it also does not let stale connections stay open for too long. The maxsessions parameter does NOT restrict users from hitting the server and if that limit is hit it will only drop the most inactive connection. Here is a technote that discusses these in more detail: Description of "Server_MaxSessions=" and "Server_Session_Timeout=" Parameters
Server_MaxSessions=x <-This is the number of sessions
Server_Session_Timeout=30 <- This is a value in minutes
Additional steps that can be tried:
- You can also periodically perform a 'dbcache flush' command on the server to flush the databases out of the dbcache temporarily and to try to prevent the paged pool usage from getting too high. As users continue to access the server they will be placed back in the cache, but it can be something you use if you see the paged pool suddenly growing or if you catch the start of the 'Insufficient system resources' errors.
- You can also monitor Paged Pool usage in Windows Performance Monitor (PerfMon) by adding the "Pool Paged Bytes" counter that is listed under the Memory performance objects. That way you can see how the paged pool usage looks over time and get an idea if there are certain times where you need to be more cautious and try to restrict any of these values further.
- If you do not have PerfMon setup and you want to see Paged Pool usage, at that point in time, look at Windows task manager -> Performance tab -> Kernel Memory section -> Paged value.
- Be aware that other things outside of Domino can lead to files remaining open on the machine: Backup systems, Operating System level anti-virus programs, Disk Defragmenter, etc.
** Please Note ** None of these changes guarantees that you will not see this problem in the future as we are still limited by how much we can increase the Paged Pool and how few databases we can have opened on an active server. There are other, more drastic, changes that may need to be considered in the future (especially as these servers continue to grow in total data size). Some of these include:
- Re-platform to 64-bit Windows where there are significantly higher address space limits. It is much less likely to exhaust the paged pool because we go from about a 350-525 MB limit to a 128 GB limit.
- Re-platform to a non-Windows 32-bit environment (for example, some form of Unix)
- Spread the load across more Domino servers so the total Data footprint is smaller on each server
- Implement an archiving solution to control the database sizes that the users are consistently accessing. Then the archive databases that are large and accessed much less frequently are stored elsewhere so the mail servers are significantly less likely to have this problem