This forum is closed to new posts and responses. Individual names altered for privacy purposes. The information contained in this website is provided for informational purposes only and should not be construed as a forum for customer support requests. Any customer support requests should be directed to the official HCL customer support channels below:

HCL Software Customer Support Portal for U.S. Federal Government clients
HCL Software Customer Support Portal


~Alexis Quetjipyter
Jul 14, 2014, 2:11 PM
46 Posts

Insufficient memory - server executive pool is full

  • Category: Domino Server
  • Platform: Windows
  • Release: 9.0.1
  • Role: Administrator
  • Tags: insufficient memory,pool is full
  • Replies: 6

Since I updated two Domino servers to 9.0.1FP1, they sometimes hangs only when they're running the archiving process (compact -a) with this message:

 

13/07/2014 10:55:15 a.m.  Database server task exited: Insufficient memory - server executive pool is full.
ti="004C7842-03257D14" sq="0014F62D" THREAD [0B60:0241-096C] WAITING FOR WRITE LOCK ON FRWSEM 0x4245 open database queue semaphore (@02ED1C30)  (R=0,W=1,WRITER=0B60:0834,1STREADER=0000:0000) FOR 30000 ms
13/07/2014 10:55:15 a.m.  Unable to create Server task
13/07/2014 10:55:15 a.m.  Database server task exited: Insufficient memory - server executive pool is full.

 

This never ends, the server stop providing service until I finally stop it, remove the transaction logs and then start the server again. Doing this, next time it runs the archive process it works fine, but two weeks later it hangs again with same error.

I opened a PMR some time ago and they recommend me to split a user list and run the archiving on each list on different days. I didn't do that yet, but what I've done is to run the archiving process for each server on different days (the two servers were archiving at the same time before). Each server has 250 users give or take, they are not too much for an archiving process, and when the servers was running on Domino 8.5.3 they worked fine.

Now one server runs archiving on Saturdays and the other one on Sundays. Last weekend one of them hanged, and the other one hanged the weekend before. 

 

I'm going to try to add enable_collection_clean_queue=0 parameter to notes.ini. I've read that it's a workaround for memory leaks problems.

Did anyone has this problem using server based archive on Domino 9.0.1FP1?

 

~Alexis Quetjipyter
Jul 14, 2014, 2:15 PM
46 Posts
One last thing

I forgot to mention that each server has 2 cores and 6 Gb RAM. Running on Windows Server 2008 R2

 

Resources are enough for the amount of users that I have on each server. I didn't see any excess of resource consumption. 

~James Revelulyader
Jul 14, 2014, 9:09 PM
145 Posts
What's the PMR #?
Did you include NSD information and matching console log data?
~Ethan Minnilylen
Jul 15, 2014, 10:43 AM
2 Posts
Make sure to collect NSDs and console log
Based on the description of the issue you are having, I would like you to set the debug parameters below in your server notes.ini.  I go from the premises that you have no debug set, please if already exist disregard the one here.  To troubleshoot hang or performance issues, IBM Support requires a complete set of debug data from the same occurrence; otherwise identifying the root cause and resolution can be delayed.
Proposed actions:

To create this data, use these steps:

1. Enable the appropriate debug parameters in the notes.ini file, then restart the Domino server or use the 'set config' command

2. Wait for the next occurrence or recreate it if possible, then issue the following Domino server console commands:

- show stat ( create a program document to run it every 5 minutes )

3. Run the NSD debugger manually back-to-back (that is, multiple times 3 times at least ) during the problem state. To do so, refer to the instructions in "How to run a manual NSD for Notes/Domino on Windows" (#1204263).


DEBUG_CAPTURE_TIMEOUT=1
DEBUG_SHOW_TIMEOUT=1

DEBUG_THREADID=1
CONSOLE_LOG_ENABLED=1

DEBUGSIGCHILD=1
DEBUG_INITTERM=1

The semaphore timeout if any, messages will be written to SEMDEBUG.TXT, which gets created under the IBM_TECHNICAL_SUPPORT directory. The next time the problem reoccurred, and data captured with the NSD utility and send it in along by zipping the  IBM_TECHNICAL_SUPPORT directory.
Please open a PMR with collected information since it is as stated above difficult to solve a memory related issue without seeing actual debugs output.  If you already has a PMR# do provide as per John entry above.
~Alexis Quetjipyter
Jul 16, 2014, 3:28 PM
46 Posts
Answers

Thanks everyone for your responses. I wanted to answer this yesterday but I didn't have the time to do so.

 

Ok, the PMR number was 27035,000,613. It's closed now because I had to try to split the process, and since the problem doesn't occur every weekend, it could took a while to see another occurrence. The NSD file was sent on that occasion. I can open a new one after I collect the data Hamed said in his previous post.

I already have some debug parameters on the notes.ini, but not everyone listed there, so I'll set them and wait. 

 

Regarding Steve question, the first time this problem occurs, I restarted the server and it crashed again after a few minutes. Removing transaction logs (and creating new ones) was the solution. I confess I didn't try this last time, I simply remove them to save time since I don't mind the server running consistency checks on all databases. It happened on sunday, our databases are controlled in size so the check is relatively quick, and very very few users may check mails that day.

~Tanita Zeklutexakol
Jul 16, 2014, 8:34 PM
32 Posts
if you have the nsd from the crash shortly after restart
I'd like to see it. Yes I'm particular about having the transaction logs removed. If there is an issue w/ them I'd like to know so it can be fixed.

--Steve
swatts@notesdev.ibm.com

This forum is closed to new posts and responses. Individual names altered for privacy purposes. The information contained in this website is provided for informational purposes only and should not be construed as a forum for customer support requests. Any customer support requests should be directed to the official HCL customer support channels below:

HCL Software Customer Support Portal for U.S. Federal Government clients
HCL Software Customer Support Portal