|
|
| The
"Unable To Extend an ID Table - Insufficient Memory" error message
can happen in databases with as few as 100,000 documents, but may not appear
in databases with over 1 million documents. As a result, this leads
to confusion about a "safe" number of documents in a database. |
| What
causes this error to appear, and how many documents can an ID Table hold
before encountering this error? |
The
limit on documents that an ID table can accommodate is indeterminate. The
ID table data structure compresses consecutive runs of NoteIDs (RRVs) into
a single entry in fixed-size table. As such, the total number of notes
that can be stored in an ID table is dependent on how fragmented the NoteIDs
are in the database. Here's where it starts to get confusing, so let me
provide an example to illustrate.
Let's say we have a database with the
five following NoteIDs:
0x0004
0x0008
0x0010
0x0014
0x001C
NoteIDs are always allocated with an
offset of four from the last allocated NoteID. Based on that fact, we know
that these are the total possible slots for notes in this database:
0x0004
0x0008 ---> 0x0004 + 4
0x000C ---> 0x0008 + 4
0x0010 ---> 0x000C + 4
0x0014 ---> 0x0010 + 4
0x0018 ---> 0x0014 + 4
0x001C ---> 0x0018 + 4
The unused slots in the database have
been greyed out to highlight the concept of "gaps" in the map
of allocated RRVs. This is of critical importance for ID tables. To restate
the earlier point, an ID table compresses consecutive runs of NoteIDs into
a single entry in a fixed-size table.
If we build an ID table from the example
notes, the following will occur:
<Example 1: ID table with non-consecutive
notes>
Offset, Repeat
0x0004, 1
0x0010, 1
0x001C, 0
Because NoteIDs are always allocated
with an offset of four, we can represent consecutive NoteIDs (where each
NoteID an offset of four from the previous NoteID) with only two pieces
of information:
1. The beginning offset
2. The number of repeats (this means
how many times we can add four to the last NoteID and land on another NoteID)
If you look at our Example 1 ID table,
we start at offset 0x0004. The next ID is 0x0008. Because this is four
greater than 0x0004, we can use the ID table entry for 0x0004 to represent
this ID by incrementing the repeat value. The next ID is 0x0010 (remember
that 0x000C isn't a note in the database). This is eight greater than 0x0008,
so we cannot use a repeat value to represent this document. Therefore,
it gets a new entry in the ID table. The next note is 0x0014, which is
four greater than 0x0010. Thus we can use a repeat to represent it. The
last ID, 0x001C, is eight greater than 0x0014, so it needs a new entry
in the table.
Now let's look at the same table if
the two unused slots were instead actual documents. The total slots now
look like this:
0x0004
0x0008 ---> 0x0004 + 4
0x000C ---> 0x0008 + 4
0x0010 ---> 0x000C + 4
0x0014 ---> 0x0010 + 4
0x0018 ---> 0x0014 + 4
0x001C ---> 0x0018 + 4
Every NoteID is four greater than the
last. If we build an ID table from this example, it looks like this:
<Example 2: ID table with consecutive
notes>
Offset, Repeat
0x0004, 6
In example 2, every note in the database
can be represented by a single entry in the ID table because they are all
consecutive. As a result, this table only needs one entry, where the previous
non-consecutive example had an ID table that had three entries in it.
The key point of understanding is that
databases with a higher concentration of consecutive NoteIDs will have
much more efficient ID tables in which single entries represent many IDs.
If the database has a highly fragmented map of allocated NoteIDs, then
the ID table will be very inefficient, and single entries will represent
few IDs. In the worst case, a single entry represents just a single NoteID,
and you will exhaust the ID table's memory limitation with relatively few
IDs.
Because each database is different,
we cannot know how fragmented the customer's map of allocated NoteIDs is,
so we have no way of even estimating how many documents they can have before
encountering the problem. All we can do are speak to generalities like
these:
1. The
higher the document count, the greater the chance to encounter problems.
2. Volatile
databases (high rate of creates and deletes) are most susceptible to the
problem, especially if deletes are random with respect to time.
3. Creating
a new database helps by partially reordering the slots for NoteIDs. This
can be done by creating a new replica or creating a new copy.
|
| |