HCL Notes and Domino 9.0 Social Edition Forum - Error: "Unable To Extend an ID Table - Insufficient Memory" (new compact -REPLICA parameter)

Oct 30, 2014, 11:10 AM

48 Posts

re Error: "Unable To Extend an ID Table - Insufficient Memory"

Hello David ,
Regarding the technote > its waiting to be published. Basically Re Error: "Unable To Extend an ID Table - Insufficient Memory"

Cause :
The ID Tables are an in memory structure used to hold lists of notes. Currently there is no way to display how full these would be. The ID Tables contain ranges of ID's so the size is variable. As the allocated and deleted notes become more fragmented the tables take up more memory.

Resolving the problem
This issue has been reported to IBM Quality Engineering under SPR # JPAI6W8KUJ and it is fixed in IBM Domino 9.x series.

For Domino 8.5.x version , if customer upgrades to 8.5.3 FP3 , then hotfix can be requested to IBM Technical Support.
New functionality is added to compact task as below

Compact -REPLICA (aka ID Table Re-org Feature)

A mechanism to create a new replica in-place in order to internally reorganize the IDs in a database to avoid ID table fragmentation that results in database access failure. This is accomplished via the compact utility with a new set of switches to control the new functionality.

hope this helps !

TomOC.

~Ethan Prejipyvitchettu

Nov 1, 2014, 2:01 AM

7 Posts

Technote 1635439 not published

Thanks, I've pieced together that information and have been using compact -replica. This command is great, but I was hoping for more formal details from the Technote.

What is wrong with the Technote that it is in an unpublished status for more than a year? Also, I am unable to see SPR JPAI6W8KUJ to determine what the actual fix is for this one.

Any info on accessing the actual details of these records? Apologies if I not looking in the right place for this information.

~Fritz Zekjumizenli

Nov 3, 2014, 8:07 PM

10 Posts

technote - id table

I saw no reason why the technote should not have been published - unless the author was planning to add additional information.
The SPR appears to be fixed on Unix - AIX on Domino 9.0.0.
An SPR is only published after it has been tested.

Regarding the main issue - the id table of the custom database.

The basic issue is that if the database is very volatile - you create and delete an extremely large number of documents - you can fragment the ID Table, and the database can not generate a document ID for any new documents. What you need to do is redesign how the custom database is used.
1. Do not delete old documents - remove most of their fields to save space and set a flag so they do not appear in any of your views.
2. Periodically create a new replica of the database.

Here is a technote that goes into the details of the ID Table.

How many documents will cause a database to show the "Unable To Extend an ID Table - Insufficient Memory" error message when Replicating or Compacting?

Product:	IBM Domino > Replication > Version 8.5, 8.0, 7.0, 6.5
Platform(s):	Windows
Doc Number:	1413921
Date:	10/27/2013

Abstract

The "Unable To Extend an ID Table - Insufficient Memory" error message can happen in databases with as few as 100,000 documents, but may not appear in databases with over 1 million documents. As a result, this leads to confusion about a "safe" number of documents in a database.

Problem

What causes this error to appear, and how many documents can an ID Table hold before encountering this error?

Cause

The ID table uses

Content

The limit on documents that an ID table can accommodate is indeterminate. The ID table data structure compresses consecutive runs of NoteIDs (RRVs) into a single entry in fixed-size table. As such, the total number of notes that can be stored in an ID table is dependent on how fragmented the NoteIDs are in the database. Here's where it starts to get confusing, so let me provide an example to illustrate.

Let's say we have a database with the five following NoteIDs:

0x0004
0x0008
0x0010
0x0014
0x001C

NoteIDs are always allocated with an offset of four from the last allocated NoteID. Based on that fact, we know that these are the total possible slots for notes in this database:

0x0004
0x0008 ---> 0x0004 + 4
0x000C ---> 0x0008 + 4
0x0010 ---> 0x000C + 4
0x0014 ---> 0x0010 + 4
0x0018 ---> 0x0014 + 4
0x001C ---> 0x0018 + 4

The unused slots in the database have been greyed out to highlight the concept of "gaps" in the map of allocated RRVs. This is of critical importance for ID tables. To restate the earlier point, an ID table compresses consecutive runs of NoteIDs into a single entry in a fixed-size table.

If we build an ID table from the example notes, the following will occur:

<Example 1: ID table with non-consecutive notes>
Offset, Repeat
0x0004, 1
0x0010, 1
0x001C, 0

Because NoteIDs are always allocated with an offset of four, we can represent consecutive NoteIDs (where each NoteID an offset of four from the previous NoteID) with only two pieces of information:

1. The beginning offset
2. The number of repeats (this means how many times we can add four to the last NoteID and land on another NoteID)

If you look at our Example 1 ID table, we start at offset 0x0004. The next ID is 0x0008. Because this is four greater than 0x0004, we can use the ID table entry for 0x0004 to represent this ID by incrementing the repeat value. The next ID is 0x0010 (remember that 0x000C isn't a note in the database). This is eight greater than 0x0008, so we cannot use a repeat value to represent this document. Therefore, it gets a new entry in the ID table. The next note is 0x0014, which is four greater than 0x0010. Thus we can use a repeat to represent it. The last ID, 0x001C, is eight greater than 0x0014, so it needs a new entry in the table.

Now let's look at the same table if the two unused slots were instead actual documents. The total slots now look like this:

0x0004
0x0008 ---> 0x0004 + 4
0x000C ---> 0x0008 + 4
0x0010 ---> 0x000C + 4
0x0014 ---> 0x0010 + 4
0x0018 ---> 0x0014 + 4
0x001C ---> 0x0018 + 4

Every NoteID is four greater than the last. If we build an ID table from this example, it looks like this:

<Example 2: ID table with consecutive notes>
Offset, Repeat
0x0004, 6

In example 2, every note in the database can be represented by a single entry in the ID table because they are all consecutive. As a result, this table only needs one entry, where the previous non-consecutive example had an ID table that had three entries in it.

The key point of understanding is that databases with a higher concentration of consecutive NoteIDs will have much more efficient ID tables in which single entries represent many IDs. If the database has a highly fragmented map of allocated NoteIDs, then the ID table will be very inefficient, and single entries will represent few IDs. In the worst case, a single entry represents just a single NoteID, and you will exhaust the ID table's memory limitation with relatively few IDs.

Because each database is different, we cannot know how fragmented the customer's map of allocated NoteIDs is, so we have no way of even estimating how many documents they can have before encountering the problem. All we can do are speak to generalities like these:

1. The higher the document count, the greater the chance to encounter problems.
2. Volatile databases (high rate of creates and deletes) are most susceptible to the problem, especially if deletes are random with respect to time.
3. Creating a new database helps by partially reordering the slots for NoteIDs. This can be done by creating a new replica or creating a new copy.